How to queue a UNIX command — The Task Spooler
We always talk about queues in modern development: RabbitMQ, AWS SQS and other available software. That’s the best way and the best practice to give the ability to scale up to your application; nevertheless, sometimes you manage legacy applications, and for different reasons, you are not able to use a queue or have time to refactor the codebase properly;
Or you want to run commands in your UNIX terminal in a queue-fashioned way with the ability to run it even in parallel; this article is definitely for you!
Going back to the main problem, not so long ago, I found a situation where a client had this setup:
Taking the image as an example: An actor can do a POST to a RESTFul endpoint, the request is redirected to an ECS CLUSTER where you have an automatic scaling up/down. The endpoint is responsible for spawning in the background a “/bin/command that” executes in a background job.
Problems
Peak traffic / Resource Exthenuation
Even having an autoscaling configuration, what would happened if you have a spike in traffic? Imagine ten thousand users calling your POST endpoint and generating another ten thousand background process in the underlying infrastructure. The ECS Cluster will not scale up so quickly, and you will get some EC2 instance behind the scene running hundreds of background job, causing CPU/MEMORY clogging. This means that the system will be impaired!
Instance/Container failure
As each of the EC2 instances runs a container, and each of the containers is responsible for running the background job, what happens if one of this job fails? You will lose it forever.
Task Spooler
As anticipated, this blog post is only showing a solution using the Task Spooler. This doesn’t mean to be a best practice; it’s only another way to solve a problem when dealing with a legacy system, and you don’t have the time/resource to refactor it.
