How to queue a UNIX command — The Task Spooler

We always talk about queues in modern development: RabbitMQ, AWS SQS and other available software. That’s the best way and the best practice to give the ability to scale up to your application; nevertheless, sometimes you manage legacy applications, and for different reasons, you are not able to use a queue or have time to refactor the codebase properly;

Or you want to run commands in your UNIX terminal in a queue-fashioned way with the ability to run it even in parallel; this article is definitely for you!

Going back to the main problem, not so long ago, I found a situation where a client had this setup:

ECS Cluster and Background Process

Taking the image as an example: An actor can do a POST to a RESTFul endpoint, the request is redirected to an ECS CLUSTER where you have an automatic scaling up/down. The endpoint is responsible for spawning in the background a “/bin/command that” executes in a background job.

Problems

ECS Cluster with resource Exthenuation

Peak traffic / Resource Exthenuation
Even having an autoscaling configuration, what would happened if you have a spike in traffic? Imagine ten thousand users calling your POST endpoint and generating another…