This bug took us a while to solve as we weren’t really sure what caused it as it came and went. The PHP consumers processing the messages had all the correct rights. Resources weren’t exhausted. At first, we thought this was coming from an API service we called. But we weren’t violating the rate limit in place. After monitoring the program more closely, we could rule this out.
The cause of this error was a request to another server made by one of the consumers. The firewall settings weren’t the problem, but fail2ban was. It interpreted the flood of requests as a threat and blocked the RabbitMQ IP for several minutes to hours.
Solution
Whitelisting the RabbitMQ server IP in fail2ban solved this issue for us.