How I generated 10,000 PDFs in seconds using SST
HTML-код
- Опубликовано: 30 сен 2024
- 📘 T3 Stack Tutorial: 1017897100294....
🤖 SaaS I'm Building: www.icongenera...
💬 Discord: / discord
🔔 Newsletter: newsletter.web...
📁 GitHub: github.com/web...
📺 Twitch: / webdevcody
🤖 Website: webdevcody.com
🐦 Twitter: / webdevcody
It would be cool to see how you go through the process of creating a PDF template, using something like an invoice or a receipt page, where you use things like mustache or handlebars for templating etc. I know there are some libraries out there like reactPDF and things like tailwindPDF. I'm curious to see how you would approach something like that. I haven't really seen any good videos around this topic.
did he ever make it?
Love this real world work
Interesting video, but the "spiciest" part is missing. Fanning out is trivial, how do you fan in afterwards and signal that the job is over? The latter requirement introduces state that one needs to handle and managing state is always PITA. We had similar stuff with PDF recently, same mess with puppeteer. In our case Azure Durable Functions was a pleasure to work with. I'm sure Amazon has same capabilities.
Aws has step functions. You can also just set a number to 10,000 in a data store and decrement that strongly consistent number and when it hits zero invoke your final fan in function
lashonta
I've started to like SST a lot, too! Love your SST content. Keep 'em coming.
Why are you using puppeteer I can’t understand that?
Puppeteer can be used to take html and convert to a pdf. This allows us to write our pdfs using react and css which is easier than any existing pdf library I know of
Great video!
I'm going through a similar path. I'm currently using sam-cli and it's alright for most use cases, but I found a blocker and thought SST could help.
I'm now trying to define in SST quite a similar flow you have in this video: customer -> lambda -> sqs -> lambda -> external API
My problem is I can't figure out how to link the pusher lambda to an existing ALB and VPC. I want the lambda to be linked and add a route in the ALB the rest of services are integrated in already.
Any idea on how to define an ALB on SST?
Thank you!
Nice I got this to work. I went with the event bus instead of the SQS queue, it fires a single ID to the api which triggers an event to start and immediately replies 200. You came in really clutch with this video because I have made PDF using python on a running server but using puppeteer in a server less mode is really great because my new project is full server-less. THANKS!
I’ve never used event bus, I need to read about iy
this is just what i was looking for 2months ago, thanks for these kind of videos very few provide these kind of videos where most of them are only showing basic of the systems
Can you do a video about caching data? (like jsons etc) using Redis vs Memcached, with aws (sst). So basically let's say you are airbnb and you want to cache your listings on the landing page so the users don't hit the database every time
I have a video about caching using redis. Published it a few weeks ago
Thank you so much for this. Your PDF videos are a god send!
Great video thanks!!!! this helps me a lot
btw, how did you understand `chromium.setGraphicsMode = false;`
haven't find anywhere online where it says you need this in order the lambda function to work
without this line I was getting timeouts on my lambda 🤯
It’s a bug. Without it the close method just hangs. There are open git issues that talk about it. All of this code makes me concerned honestly, but idk how else to easily run puppeteer inside lambda without using this Spartacus package 😂
You should start a series off of this video where you try to do performance enhancements to it, trying to make it faster or ways of optimizing and such, I think that would be fun, maybe even try it in different languages.
Yes pls
What is that rest client extension?
Slim shady lurkin
I noticed you were using chunk([], 10) in order to limit the number of messages in the queue to 10, and therefore limiting lambda invocations to 10, is there another way of doing this? Is there like a lambda invocation maximum you can set perhaps using SST config?
SendBatchCommand can only accept 10 entries at a time, so you need to batch them
@@WebDevCody Oh ok, so how exactly are you limiting the lambdas to 10 at a time?
@@ComfyCosi when you setup a sqs consumer you can specify how many events it’ll read from the queue at the same time. I think it defaults to 10
What's the reason for using SQS to batch these pdf request events? Why not just do 10000 requests in your lambda 1(api) to lambda 2(pdf generator) directly?
Sqs has built in retry’s and error handling. If for whatever reason a pdf fails to generate for one of those events, the event will be pushed back into the queue automatically and retried multiple times until it’s successful. So for example, if s3 is having issues causing the pdf lambda to fail, or if the lambda runs out of memory due to generating a certain pdf, it’ll be retried. You can also setup a dead letter queue so that if the event fails X amount of times it gets placed in another queue you can use to further process or inspect. Btw, I do think if you invoke a lambda directly from another lambda using the async invoke sdk method, aws uses sqs behind the scenes
@@WebDevCody cool!
Correct me if I'm wrong here, but Lambda limits 1,000 concurrent in flight requests per region at which the additional requests will be throttled. Just thinking this through, since there's a headless chrome browser opening and closing then the throttling could affect the run time more drastically than batching the requests using the queue. Would be interesting for you to test and bring the results to us @Huge Letters :)
Hey, cool tutorial, you could speed up the processing times by not using await inside the for loop as you are essentially doing it synchronously which takes time. Instead use map and Promise.all the result. Probably will get you a big speedup
Oh in the api where I send off sqs events? Yeah that would speed stuff up assuming I don’t hit sqs rate limits or anything
@@WebDevCody exactly, it's not as important I just wanted to mention it. Thanks a lot for these, SST seems like a really cool way of defining infrastructure in one language and codebase
sir, do you recommend any course for learning what market needs when it comes to AWS ? ec2, s3, etc..
I am still tuing my dequeue lambda and queue, how much memory / timeout / queue batchSize?
Nice job love! 👑
Great solution. I use this exact same setup for a similar task, but to update all my lambda functions. I create a JSON file of all the functions I need to update, then using Lambda, push them into queues to update env vars, save a version, update the alias, etc.
Great video again dude! Really enjoying these type of vids :)
Two things: why are you calling your api function file "lambda" considering you named the other lambda function more descriptive "generator"? Is there something similar to SST for GCP?
I don’t remember, probably just lazy
How much did that cost? from an AWS billing perspective?
my billing dashboard says $0.17 for yesterday, and I ran this scenario twice.
Can you use SST but provide lambda handles in different language ? to only replace aws cdk with SST
not sure, I haven't tried, but I assume you could potentially change the runtime to use go or something?
how do you manage templates? I have the HTML code in my dequeue lambda but that's kinda messy when I have 20+ more templates.
(pdf content = input data + template)
we have a bunch of react .jsx template which we compile to html and send to puppeteer
Do u have a repo for this?
I think I lost the code, I’ll try to look for it.
Babe, are you planning to learn the next-gen frameworks like SvelteKit and Solid-Start?
nah I'm sticking to next. I'm not too interested in switching to use something new just to render a form to a page
Keep up the great videos!
Thank you 🙏!
Or you could just generate PDFs at client side and save cost on resources.
sure, assuming your users don't have a shitty mobile phone
What is the name of the file icons extension?
bearded icons
I've never seen an easier setup using AWS
It’s pretty legit
SST looks great at first glance, but if you try to build a real-world application with a few services it's almost impossible to connect all things together in a normal way. It's good for pet projects but nothing more. They also have the worst documentation ever, a few basic examples and that's it. I try to build a simple next app with graphql (appsync) + dynamodb + ses for emails - the worst 9 days of my life. Im switching back to Amplify.
It is a bit lower level than amplify, and yes the onboarding could be a bit better. Maybe more tutorials videos could help it a bit
@@WebDevCody Tutorials videos would be perfect. Also, give it a challenge, and try to build next app using SSG consuming graphql Api and maybe add some auth on top of it :D I'm really curious about your option about DX. Have a good one!
Why not just do it concurrently in one process (lambda or fargate container)? Is it memory constraints? I feel like Rust could crush this speed.
Puppeteer and chromium is what is used under the hood to generate these pdfs, not JavaScript or rust, so ultimately your bottleneck will be how many chrome tabs can you have open at once generating pdfs, and if chromium crashes, do you have enough error handling to reprocess all the failed processes?
@@WebDevCody Would be interesting to test what these limits are, maybe a fun video idea?
Please could you make a video about how to do all this stuff without using sst, meaning create lambdas and sqs manually on aws and then link them and then call them etc like a general thing so even those who're not using sst can take advantage of this setup. Anyway thanks for this, these kinda video that show real life implementations really help alot. keep it up.
Very very cool… as someone that doesn’t know much about aws, should I learn more about the core of aws before trying sst
Yeah you probably need to understand what lambda is at the very least, probably also api gateway