Scaling to 10,000+ PostgreSQL connections with Node.js and one simple trick

Поделиться
HTML-код
  • Опубликовано: 22 дек 2024

Комментарии • 88

  • @adityakumarsingh7648
    @adityakumarsingh7648 2 месяца назад +18

    Bro , genuinely thanks for starting again

  • @shreneekpatil220
    @shreneekpatil220 2 месяца назад +29

    We had same problem with our architecture, use case was to process 10000 message per second and we end up using direct connections with Postgres and we failed with too many connection error.Later we updated the architecture with Pg Pool as a connection pool. It was amazing experience.

    • @codedamn
      @codedamn  2 месяца назад +5

      pgbouncer is amazing

    • @shreneekpatil220
      @shreneekpatil220 2 месяца назад +1

      @@codedamn yes ..keep sharing such content which people are solving daily..Thanks

    • @rohan7011
      @rohan7011 2 месяца назад

      @@codedamn hi what do you think if Supavisor instead of pgbouncer

  • @mustafahere01
    @mustafahere01 2 месяца назад +7

    I've tried pgBouncer myself on linux, and connected my backend with pgBouncer port instead of postgres. It scale amazingly

  • @pritamsinha5479
    @pritamsinha5479 2 месяца назад +2

    I would have gone for this solution much later actually. Following are my priority sorted optimizations if I have to remain on lambda.
    1. Api gateway caching
    2. Create db connection in a different file, keep configuration such that handler doesn't close connection.
    3. Increase connection pool, keep around total no of connections possible / no of concurrent lambdas allowed. Keep in mind all the apps as well as 20% overhead
    4. Provisioned concurrency of 1-2
    5. If requests increases increase RAM. Try to scale vertically as well. More powerful cpu will be able to handle multiple requests with same dB connection.
    6. Create a db connection manager ec2
    As you can see, there are much easier optimizations you can create before going for a pg bouncer setup.

  • @nsuid7499
    @nsuid7499 2 месяца назад +1

    5:27 You can send multiple parallel queries on the same connection, no need for multiple connections

    • @howhello354
      @howhello354 2 месяца назад

      Unless you start “transaction”

  • @Mr.T999
    @Mr.T999 2 месяца назад +5

    Why dont have a slack channel and bring the community together. These discussions are very valuable

  • @AlokSingh-rj1gt
    @AlokSingh-rj1gt 25 дней назад

    Really great insignt and quality content.

  • @sumedhpawar3769
    @sumedhpawar3769 2 месяца назад +1

    Great video i wanted this for so long

  • @debjyotisarkar9578
    @debjyotisarkar9578 2 месяца назад

    Mehul keep those videos coming learning a lot from it ❤.

  • @tibzdankan
    @tibzdankan 2 месяца назад +9

    Mehul, it has been some good time since you switched from mongodb to postgres on Codedamn, why did you switch in the first place and what are lessons you learn't from both databases?? Am just curious to know. Making a video about it would be highly appreciated 🙏.

    • @codedamn
      @codedamn  2 месяца назад +11

      would make one soon

    • @tibzdankan
      @tibzdankan 2 месяца назад +3

      @@codedamn Great🤝, will be waiting

  • @deepakgunpal4584
    @deepakgunpal4584 14 дней назад

    Hi Mehul, could you please create a video explaining why you chose AWS RDS over AWS Aurora and managing a database manually with an EC2 instance?
    Thanks!

  • @JagdishAdusumalli
    @JagdishAdusumalli 2 месяца назад +2

    Can you please make a complete system design/ architecture video on fermion..lot we can learn..Thanks for the works

  • @rahulsaini7409
    @rahulsaini7409 2 месяца назад +1

    Please keep making these kind of videos

  • @mitrajanumishra8071
    @mitrajanumishra8071 2 месяца назад +2

    Hi,Mehul
    Are you using provisioned concurrency for your backend because lambda has a limit of 1000 concurrent invocation per account so how is it getting managed ?

    • @codedamn
      @codedamn  2 месяца назад +1

      the limit can be increased. we have a much higher limit for our account.

  • @bytebytego-1
    @bytebytego-1 2 месяца назад +2

    What is prisma accelarator , it claims to dec query timing (using caching ) , i have tried it and the time for query went from 1200ms to literally 120ms , but i want some sort of detail video on this by experts

    • @nikolasburk
      @nikolasburk 2 месяца назад

      Prisma Accelerate is literally what Mehul showed in the video: A connection pooler that sits between your DB and Serverless functions.

  • @molviwithskills
    @molviwithskills 2 месяца назад +2

    first ❤️‍🔥

  • @MadMax-in9xz
    @MadMax-in9xz Месяц назад

    does ec2 instances also relies on connection pooler or just serverless instances only need it ?

  • @619dixitrock
    @619dixitrock 2 месяца назад

    We also have RDS Proxy for achieving something similar if I'm not wrong.

  • @rtorcato
    @rtorcato 2 месяца назад

    did you try the pg bouncer extension?

  • @philips8065
    @philips8065 2 месяца назад +1

    is n't this a very common classic case of load balancer ... or is this really a new trick

  • @paljain01
    @paljain01 2 месяца назад

    does your backend connected directly to lambda or your lambda gets trigger from api gateway?

  • @chinmoykr
    @chinmoykr 2 месяца назад

    It’s a good problem to have. If you need pgBouncer then congratulations 🎉🎉

  • @anirudhcodes
    @anirudhcodes 2 месяца назад

    Very informative

  • @sumedhpawar3769
    @sumedhpawar3769 2 месяца назад +1

    One video about using sql with node
    something about query builders and orms what are the best ones and best practices

    • @codedamn
      @codedamn  2 месяца назад +2

      will do

    • @uday4717
      @uday4717 2 месяца назад +1

      Use directly SQL why abstraction

    • @sumedhpawar3769
      @sumedhpawar3769 2 месяца назад

      @@uday4717 writing sql directly in node will make the queries vulnerable to sql injection and even if we try to cover all aspects which leads to sql injection problem we wont be able to cover alll of them
      And if someone has already a solution for it like orms and query builder why to do the hassle
      And the most important thing is readability
      Writing raw queries will make code messy
      And someone else reading and understanding the same queries

  • @ooogabooga5111
    @ooogabooga5111 2 месяца назад

    If its more of a read problem, why not seperate the read and write queries sperately (master/slave). more costly? not there yet?

  • @atanusamanta3835
    @atanusamanta3835 2 месяца назад

    Nice solution, can we have more videos on how to optimise cloud services costing.?

  • @bhargavpurohit1478
    @bhargavpurohit1478 2 месяца назад +1

    can someone explain why mehul is using serverless for this product? if number of req and traffic is high why not host the app in some instance rather then lambda?

    • @FranciscoMarcosMilhomemAbreu
      @FranciscoMarcosMilhomemAbreu 2 месяца назад +3

      Just because this you have to go serverless.

    • @minhhieple6483
      @minhhieple6483 2 месяца назад +1

      Serverless just use in case, you have traffict increase in short time. 1-2h/day.

  • @anantgupta3285
    @anantgupta3285 2 месяца назад

    What is Mongo db equivalent for pgbouncer

  • @SathishKannan-kg4mj
    @SathishKannan-kg4mj 2 месяца назад

    Nice video. How do you serve multiple requests from a lambda instannce. I mean, in aws, for each request one lambda instance is spun up right? Are you using any package or optimizations in lambda. I mean i heard that you can manaully open up a tcp connection in lambda to stream data like vercel does.

    • @codedamn
      @codedamn  2 месяца назад

      you can keep lambda alive like this docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html

    • @SathishKannan-kg4mj
      @SathishKannan-kg4mj 2 месяца назад +1

      @@codedamn It shows only how to get the context information of the lambda and function.
      You mean you are keeping event loop non empty programmatically until the lambda times out.

  • @ManishRachakonda
    @ManishRachakonda 2 месяца назад

    that was a really good video

  • @userasd360
    @userasd360 2 месяца назад

    How can one achieve this in AWS RDS??

  • @KapilSharma-mi8tx
    @KapilSharma-mi8tx 2 месяца назад

    do u tech this in courses also???

  • @adityaguptai
    @adityaguptai 2 месяца назад

    Hi Mehul,
    Any reason for not using RDS proxy instead of managing it yourself with PGBoucer.

    • @codedamn
      @codedamn  2 месяца назад +1

      we use Aurora Serverless v1 that doesn't support RDS proxy

  • @parshvsheth-rj2vq
    @parshvsheth-rj2vq 2 месяца назад

    how do you manage web socket in serverless because i have seen in dashboard that there are features that needs web socket and all ,so how do you manage socket in serverless ,i am new at this backend and serverless

    • @codedamn
      @codedamn  2 месяца назад

      which feature do you think requires websocket? we don't use websockets anywhere except a few things like playgrounds

  • @adesh_codemonk
    @adesh_codemonk 2 месяца назад

    Why not used RDS proxy - Aws managed, instead of a EC2 instance + pgBouncer which is user managed ?

    • @codedamn
      @codedamn  2 месяца назад +1

      not supported on aws aurora serverless v1

    • @adesh_codemonk
      @adesh_codemonk 2 месяца назад

      @@codedamn is shifting to V2 costlier than using EC2 ?

  • @DK-ox7ze
    @DK-ox7ze 2 месяца назад

    Which bouncer have you used here? Can you name the software?

  • @minhhieple6483
    @minhhieple6483 2 месяца назад

    In your bussiness, why just simple create EC2 instance (backend) and connect to postgres ?
    With endpoint call regular, lambda is very expensive.

    • @codedamn
      @codedamn  2 месяца назад +1

      video coming soon

  • @howhello354
    @howhello354 2 месяца назад

    AWS should give 1st class support for “pgbouncer” with their db service. So we don't need the extra headache with ec2 setup & scaling it separately

    • @MadMax-in9xz
      @MadMax-in9xz Месяц назад

      does ec2 instances also relies on connection pooler or just serverless instances only need it ?

  • @rsroyalrahul5
    @rsroyalrahul5 2 месяца назад

    how do you keep lambda container up after returning the response?

    • @codedamn
      @codedamn  2 месяца назад

      basically `callbackWaitsForEmptyEventLoop`. read this docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html

    • @rsroyalrahul5
      @rsroyalrahul5 2 месяца назад

      @@codedamn ah, interesting, never played with this param. Thanks for sharing!

  • @shreyaan
    @shreyaan Месяц назад

    rds proxy does same on aws

  • @AsishGeorgeTech
    @AsishGeorgeTech 2 месяца назад

    why not use AWS Aurora instead of this?

    • @codedamn
      @codedamn  2 месяца назад

      we use aws aurora serverless v1 only. this is same problem there. autoscaling does not mean you shouldn't put pooler in front of it. managed databases are extremely expensive and just scaling them unnecessarily increases bill a lot.

    • @AsishGeorgeTech
      @AsishGeorgeTech 2 месяца назад

      @@codedamn so what you would say is, even though aurora scales, putting a connection pooling mechanism in front of the lambda helps in cost and performance.
      the only question I have is how do you connect the lambda and ec2? Also are you using any ORM? If yes, then how does that look with the setup that you have?
      thanks.

    • @AsishGeorgeTech
      @AsishGeorgeTech 2 месяца назад

      @@codedamn also what's your thought on RDS Proxy?

  • @dataisstunning1912
    @dataisstunning1912 2 месяца назад

    What do you guys use? Node Js or anyother tech?

  • @harrypotter22347
    @harrypotter22347 2 месяца назад

    So let's say if the 10,000 connections are not disconnected, all the connections are still active. Will that be a problem or the bouncer manages that?

    • @codedamn
      @codedamn  2 месяца назад +1

      bouncer manages that easily

    • @harrypotter22347
      @harrypotter22347 2 месяца назад

      @@codedamn thanks for the answer.

  • @buildwithharshit
    @buildwithharshit 2 месяца назад +1

    Make a video on Prisma Accelerate

    • @codedamn
      @codedamn  2 месяца назад +1

      will check it out

  • @DEV4NSH_1
    @DEV4NSH_1 2 месяца назад

    bro looks like gautam gambheer when clean shave

  • @sujjee
    @sujjee 2 месяца назад

    hey mehul, why not create tutorials on these so we can learn actually how to implement these.

  • @albincr
    @albincr 2 месяца назад

    Screwed up the cost and made unnecessary load on DB by using lambda 😂😂😂😂

  • @Guru-mf3ew
    @Guru-mf3ew 2 месяца назад +1

    Why are u not active on your hindi channel like u have became here

    • @ChadSigma111
      @ChadSigma111 2 месяца назад +4

      no point, the audience is the same.
      the content is pretty high level, so if you can't understand english how are you even working?

    • @Guru-mf3ew
      @Guru-mf3ew 2 месяца назад

      @@ChadSigma111 bro i do understand english but many people dont and they also deserve this level of content and if am not able to understand english then why would i be wasting my time watching the video

    • @ChadSigma111
      @ChadSigma111 2 месяца назад

      @@Guru-mf3ew you = anyone interested in tech. This wasn't specifically meant for you.

    • @Guru-mf3ew
      @Guru-mf3ew 2 месяца назад +1

      @@ChadSigma111 ohh sorry I misunderstood it

  • @ManishRachakonda
    @ManishRachakonda 2 месяца назад

    why dont you teach this stuff in codedamn?