You have an asynchronous processing application using an Auto Scaling Group and an SQS Queue. The Auto Scaling
Group scales according to the depth of the job queue. The completion velocity of the jobs has gone down, the Auto
Scaling Group size has maxed out, but the inbound job velocity did not increase. What is a possible issue?
Some of the new jobs coming in are malformed and unprocessable.
The routing tables changed and none of the workers can process events anymore.
Someone changed the IAM Role Policy on the instances in the worker group and broke permissions to access the queue.
The scaling metric is not functioning correctly.
The IAM Role must be fine, as if it were broken, NO jobs would be processed since the system would never be able to
get any queue messages. The same reasoning applies to the routing table change. The scaling metric is fine, as instance
count increased when the queue depth increased due to more messages entering than exiting. Thus, the only reasonable
option is that some of the recent messages must be malformed and unprocessable.