Exactly. i'd love to learn, what if a new task comes, which is between the time of 2 already scheduled tasks, how to handle the insertion/ordering in Kafka.
I'm curious about when are tasks put in Kafka to be executed. Are tasks that were scheduled to run now (or in past) the only ones pushed to kafka. If so, how can you reduce the frequency of cassandra queries. On the other hand, if tasks that are scheduled to be executed in near future are also pushed to kafka then how does insertion of new tasks in between two already scheduled work, since Kafka will order the messages?
Few questions regarding The Old Solution: 1) What was the reason for choosing Cassandra as the database ? 2) Why was WorkQueue slow ? Because of too many I/O operations ? Few questions regarding The New Solution: 1)You said we cannot scale once we have one partition per broker and increasing the partitions is costly. So what is the alternative ? What if i want to scale further ? 2)When you say logical queues will be stuck, you mean to say logical queues in the stuck logical queue instance will be stuck right ? and not all the logical queues across the service instances right ?
Well explained but one thing seems to be missing here is how scheduling is implemented. It'd be nice to have that explained as well.
Exactly. i'd love to learn, what if a new task comes, which is between the time of 2 already scheduled tasks, how to handle the insertion/ordering in Kafka.
I'm curious about when are tasks put in Kafka to be executed. Are tasks that were scheduled to run now (or in past) the only ones pushed to kafka. If so, how can you reduce the frequency of cassandra queries.
On the other hand, if tasks that are scheduled to be executed in near future are also pushed to kafka then how does insertion of new tasks in between two already scheduled work, since Kafka will order the messages?
Very good talk. Well-prepared and well-presented.
Few questions regarding The Old Solution:
1) What was the reason for choosing Cassandra as the database ?
2) Why was WorkQueue slow ? Because of too many I/O operations ?
Few questions regarding The New Solution:
1)You said we cannot scale once we have one partition per broker and increasing the partitions is costly. So what is the alternative ? What if i want to scale further ?
2)When you say logical queues will be stuck, you mean to say logical queues in the stuck logical queue instance will be stuck right ? and not all the logical queues across the service instances right ?
I need to implement same in java. Is there any place I can get the architecture documents of this ?
I worked on a similar project a while back. We built a system off quartz, how did you proceed? just curious :)