Here at hapily, we utilize AWS Lamba in most of our apps. User connections tend to be closed by the Lambda if they go unused for too long. But they can stay open depending on a variety of different factors. One of which I'll cover in this post!
When your app functions this way, keeping your connection number down is a priority 1 task. If you reach the limit, it can prevent other users from accessing the app or just make it crash entirely.
Of course, if you do get the dreaded “Too many connections” error, you may not know or understand what’s causing it. As a programmer myself, I can tell you that an elusive bug such as this can make you want to chuck your computer out the window and swear eternal vengeance.
That’s why it’s important to understand how your database works from the get go. The better your understanding, the better you can prevent issues like the one documented below.
How to fix skyrocketing idle connections in AWS
Code can be VERY temperamental. One misaligned comma or a variable that’s just one digit too long and your program will explode (hopefully not literally).
While that can be mildly amusing the first couple of times it occurs, it can result in a headache if left unchecked.
One such explosion (or bug in layman’s terms) occurred while we were using AWS. We implemented an RDS Proxy to facilitate communication with our MySQL RDS. This wasn’t a problem at first. But as our user base grew, our number of idle connections began piling up so high that it started to attract tourists.
Despite the fact that we only had around 50 simultaneous connections, there were 700 connections open. That's reasonable (it's not).
This resulted in us reaching our database limit of 1000 connections. And with our limit reached, our project crashed. What's worse, managing database connectivity for Lambda based applications wasn't the most straightforward:.png)
Our development team tried a variety of different solutions to fix the problem. They:
1. *Adjusted the Pool Connection Limit:* They lowered the pool connection limit in all our Lambda functions as well as in our backend.
2. *Modified Lambda Concurrency:* To limit the number of pools opened, they adjusted the concurrency settings on our Lambdas.
3. *ORM Framework Monitoring:* Our backend pool, managed by an ORM framework, was also closely monitored for irregularities.
4. *Auto Scaling Group:* Even though they had set up an auto-scaling group, it wasn't scaling up enough to justify the 1000 connections being utilized.
5. *User Segmentation:* They created separate database users for each project, Lambda functions, the backend, and any other application in an attempt to isolate the issue.
No dice. The issue persisted. After taking another look at the code, the development team discovered what was causing all of these extra connections.
Remember what I said about code being very particular? That’s exactly what was going on here. Turns out, there was a parameter within our RDS Proxy configuration called MaxConnectionsPercent. The name itself wasn’t the problem, but the fact that it was set to 100% was.
You see, this parameter determines how often the proxy closes idle database connections. The higher the percentage, the higher the number of connections that can get away with being idle.
You can probably see how it being set to 100% was a problem. The proxy wasn’t closing ANY idle connections.
The solution was obvious: Lower the percentage. After dropping it down to 15%, the idle connection number plummeted.
With the problem solved, the team implemented a new process to manage not just our Lambda connections, but other connection types as well..png)
The moral of the story is that whether you’re using a program or a programming platform, understanding its inner workings is crucial to your success! A few trips to the docs to read up on any variables you don’t understand never hurts either.
As one of our developers put it “You may not know how things work behind the scenes. Using something when you don't know how it works may be costly down the road. Understanding how it works is the baseline for us to effectively manage and optimize our applications.”
If you enjoyed this post, why not share it with your friends? And if you want to see one of our apps, check out this blog post on Associ8!
