Lesson 171 - Producer Control Flow Pattern

In lesson 46 I gave an introduction to reactive architecture patterns. In this lesson I’ll talk about one of those pattern-the Producer Control Flow pattern, one used to create self-healing systems by programmatically monitoring downstream systems and applying backpressure to upstream systems so that systems can heal themselves.
Reference Links:
Reactive Architecture: www.developert...
Software Architecture Monday: bit.ly/3dadEe3
Fundamentals of Software Architecture: amzn.to/3rgFLjY
Software Architecture: The Hard Parts: amzn.to/3BjMMF2

[@vishwas13485]

Hello Mark , I would really love the way you simply explain things and provide insights on architecture decisions. Keep it up :-)

[@markrichards5014]

Thanks! 🙂

[@mahdi5796]

Thank you. I was in the assumption that scalability is job of the event channel. Given cloud scalable messaging services such as Amazon Kinesis, do you think this pattern is still relevant?

[@markrichards5014]

Many message brokers have this pattern built-in as a self-protection mechanism, but in most cases it's not a slowdown, but rather a SHUTDOWN-in other words, rather than slowing down producers, you'll simply get an exception trying to send the stream or write to the queue. Keep in mind though - it's less about the messaging service, and more about how fast the consumers can process the messages. Imagine if you shut down all consumers and started sending 5000 log or metrics messages a second. How long before the messaging service would say "Enough!"? Not long. That said, it depends on the message or streaming service you are using. Bottom line-don't assume the messaging service or stream can handle an infinite amount of messages. Check to see if theres a top limit, and work down from there...

[@cybernetic100]

I have a few questions:
- How would this system scale up when you have multiple consumers and multiple instances of each producer? Is the flow monitor going to be created per producer-consumer (or queue), pair? or is there only one FM to rule 'em all?
- The flow control queue is probably shared amongst multiple instances of the same producer (which act as competing consumers of the flow control queue), in which case only one of the instances will be able to reduce its message rate, the other instances will still continue pumping messages at the same rate resulting in only a marginal "back pressure". How will I work around this problem to make sure all instances of the producers are able to slow down? Might a centralised storage for the delay (per producer) be an option, where, instead of pushing a control flow message to the producer, I store it in a database of some sort (perhaps keyed by some kind of id for the producer) and all the instances of the producer can read the delay from this store and eventually slow down? Ofcourse disabling the control flow will mean removing the record for that producer from the database.

[@mqtt07]

You make a good point. I would not use queues in the path from the flow monitor to the consumer set. I would use a topic instead. Queues is for decoupling systems and generally has persistence, so that when consumer is offline the queue piles up with messages from the flow monitor. We don't want that in this case. With topic we can have fanout real-time broadcasting which is probably more appropriate in this case. My two cents

[@cybernetic100]

@@mqtt07 I am not sure that solves the problem of not all publishers responding to the flow control messages evenly and promptly. I did a little proof of concept with Kafka for flow control topic (3 partitions) and with 3 publisher instances. Each instance will have been assigned a partition. Then from the flow monitor I published flow control enable/disable messages (key less messages) onto the topic and only 1 of the 3 instances responded at any time.
When I killed one of the instances, which triggered a partition rebalance amongst the remaining publishers, then I saw both responding to flow control messages. This is due to the fact that key less messages are just round robined across partitions but since the flow monitor code is idempotent i.e. it only publishes an enable/disable message once per cycle, the messages will not always end up on all the partitions. Kafka only allows one consumer per consumer group to read from a partition.
Then I tried MySql as the "broker" for flow control messages instead of Kafka by simply UPSERTING a record with a boolean flag, and low and behold all publishers responded immediately as you'd expect. May be I'm missing something and a response to my scaling question might reveal additional insights, but given what I have understood so far and experimented with, I am not sure if async messaging for flow control is the most effective or reliable way to do this (regardless of queue or topic). Its also a question of how reliable we want this flow control to be.

[@cybernetic100]

I am also not sure where this pattern will really help or how will it really work.
Assuming flow control is needed when you have a consumer that cannot simply scale out to deal with the backlog of the messages (may be due to external dependencies that limit concurrency), you want a flow monitor per consumer queue and given most of the time you are using pub/sub (e.g. one SNS topic + many SQS queues), how do you selectively reduce the message rate for specific consumers without slowing it every other consumer as well? Or without making the producer aware of all the consumers listening to the messages so it can only selectively reduce the rate?
I am not sure!

[@markrichards5014]

In that case you can broadcast the slow down and resume messages through a topic rather than a queue...

[@marioppavlov]

Hello, and thanks for explaining this pattern. However, I'm having some thoughts and questions. First I didn't manage to find further explanation to that pattern in neither of the books: Fundamentals of Software Architecture or Software Architecture: The Hard Parts.
Proceeding to my question, I wonder if this pattern can be used to only limit the producers. Definitely I can see a lot of use-cases to limit the producer, but in some cases this is simply not possible or not preferable. Lets take for example the Trading system that has been used in the video. I guess that the explanation most probably is related to logging or some kind of monitoring process for the system where events are send for further analysis, but what will happen if we implement such a control pattern to the trading core engine. Usually the trade should occur as soon as possible so that certain price can be met or at least the best possible price is met. In this case however, the delay can be quite significant.
So my question is how applicable is to apply the control pattern on top of the consumers, and for example to scale up the system with multiple consumers so that the system can handle the volume faster? (I know that there is a threshold to the scaling factor also and it may produce to other issues at that end)
P.S.
I believe that I found out the further explanation in the Fundamentals book inside the Chapter 14. Event-Driven Architecture Style, Error Handling.

[@markrichards5014]

This pattern didn't make it into our books (not enough room). At times I've introduced a service between the producer and consumer that manages the control flow. when I don't have control over producers (such as 3rd party trading systems like Front Arena or Charles River) You are right - there are always tradeoffs-would you rather have trades take longer to place, or have them fail? Because those are really the two options...