Message delivery semantics and Hinted Handoff pattern in distributed systems.
hey Fernando, thank you so much for the guest post opportunity in your awesome newsletter.
Super interesting read. Thanks both Fernando and NK!
The article came at a perfect time, I was just going through the Amazon Dynamo paper. It was interesting to learn the durability concern of the hinted handoff and that it can be remediated with periodic replica synchronization to reconcile them
About the message queues for exactly-once delivery, I think idempotent messages solve part of the problem, but may introduce an extra problem of message ordering. x=5 is idempotent as I can execute it multiple times in a row, but when interacting with another message, if the order on the consumer side is x=5, x=3, x=5 (e.g. a retry of the x=5 message), then the outcome is not expected.
I'd love to read more about message ordering in distributed systems. Looking forward for the next articles!