We had same problem with our architecture, use case was to process 10000 message per second and we end up using direct connections with Postgres and we failed with too many connection error.Later we updated the architecture with Pg Pool as a connection pool. It was amazing experience.
I would have gone for this solution much later actually. Following are my priority sorted optimizations if I have to remain on lambda. 1. Api gateway caching 2. Create db connection in a different file, keep configuration such that handler doesn't close connection. 3. Increase connection pool, keep around total no of connections possible / no of concurrent lambdas allowed. Keep in mind all the apps as well as 20% overhead 4. Provisioned concurrency of 1-2 5. If requests increases increase RAM. Try to scale vertically as well. More powerful cpu will be able to handle multiple requests with same dB connection. 6. Create a db connection manager ec2 As you can see, there are much easier optimizations you can create before going for a pg bouncer setup.
Mehul, it has been some good time since you switched from mongodb to postgres on Codedamn, why did you switch in the first place and what are lessons you learn't from both databases?? Am just curious to know. Making a video about it would be highly appreciated 🙏.
Hi Mehul, could you please create a video explaining why you chose AWS RDS over AWS Aurora and managing a database manually with an EC2 instance? Thanks!
Hi,Mehul Are you using provisioned concurrency for your backend because lambda has a limit of 1000 concurrent invocation per account so how is it getting managed ?
What is prisma accelarator , it claims to dec query timing (using caching ) , i have tried it and the time for query went from 1200ms to literally 120ms , but i want some sort of detail video on this by experts
@@uday4717 writing sql directly in node will make the queries vulnerable to sql injection and even if we try to cover all aspects which leads to sql injection problem we wont be able to cover alll of them And if someone has already a solution for it like orms and query builder why to do the hassle And the most important thing is readability Writing raw queries will make code messy And someone else reading and understanding the same queries
can someone explain why mehul is using serverless for this product? if number of req and traffic is high why not host the app in some instance rather then lambda?
Nice video. How do you serve multiple requests from a lambda instannce. I mean, in aws, for each request one lambda instance is spun up right? Are you using any package or optimizations in lambda. I mean i heard that you can manaully open up a tcp connection in lambda to stream data like vercel does.
@@codedamn It shows only how to get the context information of the lambda and function. You mean you are keeping event loop non empty programmatically until the lambda times out.
how do you manage web socket in serverless because i have seen in dashboard that there are features that needs web socket and all ,so how do you manage socket in serverless ,i am new at this backend and serverless
we use aws aurora serverless v1 only. this is same problem there. autoscaling does not mean you shouldn't put pooler in front of it. managed databases are extremely expensive and just scaling them unnecessarily increases bill a lot.
@@codedamn so what you would say is, even though aurora scales, putting a connection pooling mechanism in front of the lambda helps in cost and performance. the only question I have is how do you connect the lambda and ec2? Also are you using any ORM? If yes, then how does that look with the setup that you have? thanks.
@@ChadSigma111 bro i do understand english but many people dont and they also deserve this level of content and if am not able to understand english then why would i be wasting my time watching the video
Bro , genuinely thanks for starting again
We had same problem with our architecture, use case was to process 10000 message per second and we end up using direct connections with Postgres and we failed with too many connection error.Later we updated the architecture with Pg Pool as a connection pool. It was amazing experience.
pgbouncer is amazing
@@codedamn yes ..keep sharing such content which people are solving daily..Thanks
@@codedamn hi what do you think if Supavisor instead of pgbouncer
I've tried pgBouncer myself on linux, and connected my backend with pgBouncer port instead of postgres. It scale amazingly
tutorial?
I would have gone for this solution much later actually. Following are my priority sorted optimizations if I have to remain on lambda.
1. Api gateway caching
2. Create db connection in a different file, keep configuration such that handler doesn't close connection.
3. Increase connection pool, keep around total no of connections possible / no of concurrent lambdas allowed. Keep in mind all the apps as well as 20% overhead
4. Provisioned concurrency of 1-2
5. If requests increases increase RAM. Try to scale vertically as well. More powerful cpu will be able to handle multiple requests with same dB connection.
6. Create a db connection manager ec2
As you can see, there are much easier optimizations you can create before going for a pg bouncer setup.
5:27 You can send multiple parallel queries on the same connection, no need for multiple connections
Unless you start “transaction”
Why dont have a slack channel and bring the community together. These discussions are very valuable
Really great insignt and quality content.
Great video i wanted this for so long
Mehul keep those videos coming learning a lot from it ❤.
Mehul, it has been some good time since you switched from mongodb to postgres on Codedamn, why did you switch in the first place and what are lessons you learn't from both databases?? Am just curious to know. Making a video about it would be highly appreciated 🙏.
would make one soon
@@codedamn Great🤝, will be waiting
Hi Mehul, could you please create a video explaining why you chose AWS RDS over AWS Aurora and managing a database manually with an EC2 instance?
Thanks!
Can you please make a complete system design/ architecture video on fermion..lot we can learn..Thanks for the works
Please keep making these kind of videos
Hi,Mehul
Are you using provisioned concurrency for your backend because lambda has a limit of 1000 concurrent invocation per account so how is it getting managed ?
the limit can be increased. we have a much higher limit for our account.
What is prisma accelarator , it claims to dec query timing (using caching ) , i have tried it and the time for query went from 1200ms to literally 120ms , but i want some sort of detail video on this by experts
Prisma Accelerate is literally what Mehul showed in the video: A connection pooler that sits between your DB and Serverless functions.
first ❤️🔥
does ec2 instances also relies on connection pooler or just serverless instances only need it ?
We also have RDS Proxy for achieving something similar if I'm not wrong.
did you try the pg bouncer extension?
is n't this a very common classic case of load balancer ... or is this really a new trick
does your backend connected directly to lambda or your lambda gets trigger from api gateway?
It’s a good problem to have. If you need pgBouncer then congratulations 🎉🎉
Very informative
One video about using sql with node
something about query builders and orms what are the best ones and best practices
will do
Use directly SQL why abstraction
@@uday4717 writing sql directly in node will make the queries vulnerable to sql injection and even if we try to cover all aspects which leads to sql injection problem we wont be able to cover alll of them
And if someone has already a solution for it like orms and query builder why to do the hassle
And the most important thing is readability
Writing raw queries will make code messy
And someone else reading and understanding the same queries
If its more of a read problem, why not seperate the read and write queries sperately (master/slave). more costly? not there yet?
Nice solution, can we have more videos on how to optimise cloud services costing.?
can someone explain why mehul is using serverless for this product? if number of req and traffic is high why not host the app in some instance rather then lambda?
Just because this you have to go serverless.
Serverless just use in case, you have traffict increase in short time. 1-2h/day.
What is Mongo db equivalent for pgbouncer
Nice video. How do you serve multiple requests from a lambda instannce. I mean, in aws, for each request one lambda instance is spun up right? Are you using any package or optimizations in lambda. I mean i heard that you can manaully open up a tcp connection in lambda to stream data like vercel does.
you can keep lambda alive like this docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html
@@codedamn It shows only how to get the context information of the lambda and function.
You mean you are keeping event loop non empty programmatically until the lambda times out.
that was a really good video
How can one achieve this in AWS RDS??
do u tech this in courses also???
Hi Mehul,
Any reason for not using RDS proxy instead of managing it yourself with PGBoucer.
we use Aurora Serverless v1 that doesn't support RDS proxy
how do you manage web socket in serverless because i have seen in dashboard that there are features that needs web socket and all ,so how do you manage socket in serverless ,i am new at this backend and serverless
which feature do you think requires websocket? we don't use websockets anywhere except a few things like playgrounds
Why not used RDS proxy - Aws managed, instead of a EC2 instance + pgBouncer which is user managed ?
not supported on aws aurora serverless v1
@@codedamn is shifting to V2 costlier than using EC2 ?
Which bouncer have you used here? Can you name the software?
pgbouncer
In your bussiness, why just simple create EC2 instance (backend) and connect to postgres ?
With endpoint call regular, lambda is very expensive.
video coming soon
AWS should give 1st class support for “pgbouncer” with their db service. So we don't need the extra headache with ec2 setup & scaling it separately
does ec2 instances also relies on connection pooler or just serverless instances only need it ?
how do you keep lambda container up after returning the response?
basically `callbackWaitsForEmptyEventLoop`. read this docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html
@@codedamn ah, interesting, never played with this param. Thanks for sharing!
rds proxy does same on aws
why not use AWS Aurora instead of this?
we use aws aurora serverless v1 only. this is same problem there. autoscaling does not mean you shouldn't put pooler in front of it. managed databases are extremely expensive and just scaling them unnecessarily increases bill a lot.
@@codedamn so what you would say is, even though aurora scales, putting a connection pooling mechanism in front of the lambda helps in cost and performance.
the only question I have is how do you connect the lambda and ec2? Also are you using any ORM? If yes, then how does that look with the setup that you have?
thanks.
@@codedamn also what's your thought on RDS Proxy?
What do you guys use? Node Js or anyother tech?
node.js
So let's say if the 10,000 connections are not disconnected, all the connections are still active. Will that be a problem or the bouncer manages that?
bouncer manages that easily
@@codedamn thanks for the answer.
Make a video on Prisma Accelerate
will check it out
bro looks like gautam gambheer when clean shave
hey mehul, why not create tutorials on these so we can learn actually how to implement these.
Screwed up the cost and made unnecessary load on DB by using lambda 😂😂😂😂
Why are u not active on your hindi channel like u have became here
no point, the audience is the same.
the content is pretty high level, so if you can't understand english how are you even working?
@@ChadSigma111 bro i do understand english but many people dont and they also deserve this level of content and if am not able to understand english then why would i be wasting my time watching the video
@@Guru-mf3ew you = anyone interested in tech. This wasn't specifically meant for you.
@@ChadSigma111 ohh sorry I misunderstood it
why dont you teach this stuff in codedamn?