Distributed Job Scheduler - System Design | Basic to Expert | On Whiteboard
HTML-код
- Опубликовано: 22 дек 2022
- Job Scheduler is very common use case for any organization to schedule their configurable job in cloud distributed environment.
How to design Job Scheduler?
From basic design to expert distributed design.
Distributed Job Scheduler - System Design | Basic to Expert | On Whiteboard
#designinterview #systemdesign #distributeddesign #javainterviewquestions
#distributedjobscheduler #systemdesign #whiteboardclass #beginnersguide
Kindly Subscribe, share and like this video to grow this channel, #thankyou
Nice step by step explanation. Thanks Binod.
very well explained!
In the video it is mentioned (around 14:30), Kafka have 'n' partitions then we will have 'm' consumer groups.
As each consumer group reads from all partition, how we will handle reading of the same meesage by different consumer group ?
very detailed and amazing explanation !
Good video, explanation.
Good explanation bhai helped alot.
Usually in system design diagrams, the API gateway comes first then the load balancer and then the application servers. Here I am little confused since the LB comes before Api servers.
when multiple exexutors are connected to one partition and select jobs to run by using mysql select, how could you make sure jobs in the same partition will not be executed multiple time? If you use lock, there would be performance issues.
If each job does the same maintain the final state if final state is reached no more update, even the second job try to update (some state machine logic)
What exactly is a task? How does one actually execute this task? can you get to the details about executor about what exactly it does to execute the job?
Task is something that you want the computer to execute either by a specific time (ie: every 30 minutes) or execute once at some point in the future.
Take RUclips as an example. When a user uploads a video. RUclips will create a task for converting the video and distributing the video to CDNs. However, it's not possible for RUclips to process all of these uploaded videos at once. So, it will create a task in the database and whenever a video processor (executor) is free. It will pick up the task to convert the video. If something happens, the failed task will be picked up to execute again in the future.
@@varshard0 Thanks for your response. I got what a task is from a theoretical standpoint, but I'm still trying to find what does it look like practically. For example, in your example above, what exactly will be stored so that executor will know it is video processor which need to do some video conversion. I'm assuming this system will need to do multiple types of tasks and not just video conversion, so the executor should somehow know how to execute the task. Basically some example of what is stored in the job detail which helps to execute the job will be good.
@@SP-yf1ib The task can be anything you like. If can be a code uploaded by the customer, which needs to scheduled and run. For example, I can upload a python script which checks the dollar vs inr and emails me when it reaches a certain threshold. But I need a scheduler to run it once a day for me.
@@rishabhjain2404 Right, but that is one important part of the scheduler which I think is missing here. How does the scheduler run it when it can be anything? For eg. in case of a python script, it need to know how to store and execute the python script. So one or two examples of how it is getting stored and executed would've helped.
How do you shard RDBMS ?
Like he mentioned, we will hash the job_id. If we have 5 master DBs, the hash will generate a number upto 5 and that job will be saved on that DB.
were you shooting during an earthquake ?
Why is the video shaking so much