thanks for the great video! In some days we'll start implementing a solution for automatically charging money to users (we will store a token of their credit/debit cards which will allow us to call an external API to charge them). A couple of alternatives that we've considered are: a. Running a periodic cron job (with Cloud Scheduler), which triggers just one Cloud Function that travels through all the pending payments, try to complete them, and finally writes to an emails collection (which is configured to use the Firebase Extension that automatically sends emails) for notifying administrators with a summary of success and failure payments. b. Running a periodic cron job (with Cloud Scheduler), but then using Pub/Sub, like you explain in the video. In our case, we will probably only write the results to Firestore, and also notify administrators via email, with a summary of success and failure payments. Regarding scalability, I suspect that a single Cloud Function could handle hundreds or thousands of pending payments (by sending requests to the external payment service in parallel), although we'll obviously do some research and testing. In case our business scales to a point which can't be handled by, e.g, a daily cron job with a single Firebase Function, we could fine-tune the periodicity of the cron job, to decrease each function execution load. This is a very important workload for us, so we're glad that you've explained this practical use case, as that made us consider more carefully some possible issues that we could face by using only one Cloud Function, and the advantages of the more event based alternative.
Awesome design session...My mind map is Martin Omander's videos always more talks about Cloud Run...I have seen nice Cloud Run sessions from Martin Omander which was missing this is design session. If I want to include Martin Omander's favorite Cloud Run then replace just *App Engine with Global Load Balancer with Serverless NEG pointing to a Cloud Run service as the backend service...*
Would I be correct in thinking that Cloud Task would be a realistic alternative to pub sub in this scenario. I kind of half expected the choice to be Cloud Tasks at first mention of emailing and then refactor the design to use pub sub when you realized there would be > 1 consumer (??). Just checking my understanding here...
I think Cloud Tasks is meant more for tasks you want to be rate-limited. It'd probably be fine for a task count under a certain amount, but if there are many thousands or millions of tasks, it's going to bottleneck. The docs say Cloud Tasks have a maximum delivery rate of "500 qps/queue", whatever that means.
Thank you for creating and sharing this info on design, great to see the the thought process around identity, data storage, scheduling, messaging/pub-sub and retries 🧠
And how you recommend implementing writing all that info in BigQuery? Generate a CSV file and import it, or stream records directly in BQ from cloud run?
As a teacher, I feel like I am obligated to point out blind spots. I follow/subscribe to every RUclips channel to Google (Google Cloud, Firebase/Firestore, Workspace, Chrome Dev, etc.) And I keep seeing the same thing in all the videos. The info when your instructional videos finally get to the point (teaching) it is great! But I wish "Google" would stop making them read horrible, inauthentic, and not funny scripts that are obviously being read. I do not need to be sold or "entertained". I watch movies for entertainment with professional actors. I hate to be harsh, but it is painful to watch the videos. We are here because we like if not LOVE your product. Just teach us. Don't try and sell us and don't make your awesome tech teams act. We just want to learn. By the way, I am an instructor and teach globally so hopefully, my advice will be heard. Keep making great products and teaching us how to use them. Leave the acting to Netflix. Thanks again.
I love the session, but I thing these design session should be also focus on the cost of architecture because that is the first thing a developer think of first.😊
Hi Shashank, good question! First, if you are planning for millions of subscriptions eventually, but have far fewer now, quickly build something now that can handle your current load and worry about scalability later. Focus on getting your product to market quickly. I would probably build it similar to what's shown in this video: Cloud Scheduler triggers a Cloud Function that reads all the records from Firestore and publishes one Pub/Sub message per subscription. Another Cloud Function would handle each of these messages. The function would be simple because it would only handle one message. If you like containers, consider replacing Cloud Functions with Cloud Run. If you want to have greater control over the rate at which subscriptions are processed, consider replacing Pub/Sub with Cloud Tasks. Best of luck with your app!
Great question! Currently the "most serverless" database in Google Cloud is Firestore, a NoSQL/non-relational database. The SQL/relational databases (CloudSQL, AlloyDB, Spanner) are "less serverless" as they require a fixed monthly fee or maintenance.
![Seungmin "그렇게, 천천히, 우리(As we are)" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/kAzmhLHePqU/mqdefault.jpg)
I'm loving these design sessions, please keep doing them because they are gold. Thanks!
really love Dina and Martin's session. It feels real.
Server is amazing. None of the app and website can't be made without it😘😘
Exactly
Great job Dina !
This is so helpful. Waiting for more sessions.! 👏🏻
thanks for the great video!
In some days we'll start implementing a solution for automatically charging money to users (we will store a token of their credit/debit cards which will allow us to call an external API to charge them). A couple of alternatives that we've considered are:
a. Running a periodic cron job (with Cloud Scheduler), which triggers just one Cloud Function that travels through all the pending payments, try to complete them, and finally writes to an emails collection (which is configured to use the Firebase Extension that automatically sends emails) for notifying administrators with a summary of success and failure payments.
b. Running a periodic cron job (with Cloud Scheduler), but then using Pub/Sub, like you explain in the video. In our case, we will probably only write the results to Firestore, and also notify administrators via email, with a summary of success and failure payments.
Regarding scalability, I suspect that a single Cloud Function could handle hundreds or thousands of pending payments (by sending requests to the external payment service in parallel), although we'll obviously do some research and testing. In case our business scales to a point which can't be handled by, e.g, a daily cron job with a single Firebase Function, we could fine-tune the periodicity of the cron job, to decrease each function execution load.
This is a very important workload for us, so we're glad that you've explained this practical use case, as that made us consider more carefully some possible issues that we could face by using only one Cloud Function, and the advantages of the more event based alternative.
Happy to hear the video was helpful!
Excellent session
Awesome design session...My mind map is Martin Omander's videos always more talks about Cloud Run...I have seen nice Cloud Run sessions from Martin Omander which was missing this is design session. If I want to include Martin Omander's favorite Cloud Run then replace just *App Engine with Global Load Balancer with Serverless NEG pointing to a Cloud Run service as the backend service...*
That's a fair point :-)
Would I be correct in thinking that Cloud Task would be a realistic alternative to pub sub in this scenario. I kind of half expected the choice to be Cloud Tasks at first mention of emailing and then refactor the design to use pub sub when you realized there would be > 1 consumer (??). Just checking my understanding here...
I think Cloud Tasks is meant more for tasks you want to be rate-limited. It'd probably be fine for a task count under a certain amount, but if there are many thousands or millions of tasks, it's going to bottleneck. The docs say Cloud Tasks have a maximum delivery rate of "500 qps/queue", whatever that means.
Thank you for creating and sharing this info on design, great to see the the thought process around identity, data storage, scheduling, messaging/pub-sub and retries 🧠
Glad you enjoyed it!
Very helpful, thanks!
We're glad to hear it, Andy!
It would be nice to get the codebase for this app great session
I'm afraid we never wrote the code to implement this design. It would have been fun, though!
Can you do a video on Adaptive video streaming using serverless and cloud storage
Very cool!!
Please do these kind of sessions more frequently.
And how you recommend implementing writing all that info in BigQuery? Generate a CSV file and import it, or stream records directly in BQ from cloud run?
Imagine if there is a tool online which can help you visualize these design like they are doing. Is there something like this already out?
It would be important to close sequencing to permit ports or |serial|~
4(importance);:/list could be runas{log}
As a teacher, I feel like I am obligated to point out blind spots. I follow/subscribe to every RUclips channel to Google (Google Cloud, Firebase/Firestore, Workspace, Chrome Dev, etc.) And I keep seeing the same thing in all the videos.
The info when your instructional videos finally get to the point (teaching) it is great! But I wish "Google" would stop making them read horrible, inauthentic, and not funny scripts that are obviously being read. I do not need to be sold or "entertained". I watch movies for entertainment with professional actors.
I hate to be harsh, but it is painful to watch the videos. We are here because we like if not LOVE your product. Just teach us. Don't try and sell us and don't make your awesome tech teams act. We just want to learn. By the way, I am an instructor and teach globally so hopefully, my advice will be heard.
Keep making great products and teaching us how to use them. Leave the acting to Netflix.
Thanks again.
I love the session, but I thing these design session should be also focus on the cost of architecture because that is the first thing a developer think of first.😊
Amazing
If I have an application where I have to renew the user monthly subscription which could be in millions then how do we design that in Google cloud?
Also my data is saved in firestore because I want to go serverless
Hi Shashank, good question! First, if you are planning for millions of subscriptions eventually, but have far fewer now, quickly build something now that can handle your current load and worry about scalability later. Focus on getting your product to market quickly.
I would probably build it similar to what's shown in this video: Cloud Scheduler triggers a Cloud Function that reads all the records from Firestore and publishes one Pub/Sub message per subscription. Another Cloud Function would handle each of these messages. The function would be simple because it would only handle one message.
If you like containers, consider replacing Cloud Functions with Cloud Run. If you want to have greater control over the rate at which subscriptions are processed, consider replacing Pub/Sub with Cloud Tasks.
Best of luck with your app!
Love the Video's
"Yeah. No."
No.
she is a good actor, but script is already prepared.
"We try to keep it serverless, which means it is nosql" - why is that?
Great question! Currently the "most serverless" database in Google Cloud is Firestore, a NoSQL/non-relational database. The SQL/relational databases (CloudSQL, AlloyDB, Spanner) are "less serverless" as they require a fixed monthly fee or maintenance.