Jumping into the new year
Jumping into the new year
My plans and resolutions for the new year. Plus: how Uber handles Push notifications and how to improve your test suite while bug-fixing.
Hi Friends,
Welcome to the 8th edition of the Polymathic Engineer newsletter. I hope you all had a great time with your loved ones and you had a fantastic start to the new year.
I spent the last two weeks in Italy, visiting my family. It was the first time after many years that I could spend the Christmas break with my parents at the place where I grew up, and I felt so grateful and happy for this.
During this period, I rested and enjoyed my family as much as possible to recharge my batteries. Still, I also tried to clarify what I’d like to focus on next year.
Last year I mainly focused on creating technical content. Still, this year, I’d like to dedicate more time to improve my software engineering skills and mentoring other developers to improve their algorithmic and distributed system skills.
Regarding the first point, the main action is to read more technical books, ideally one per month. Regarding the second point, the idea is to open a YouTube channel and create some Github repositories to share more knowledge. I’ll give more details in the coming issue, but if somebody would like to contribute, feel free to let me know.
Let’s go with some content now:
How Uber eats handle push notifications
Test-driven bug-fixing
Coding challenge
Interesting Tweets
Uber Eats Push Notification
Uber Eats extensively uses push notifications to communicate with its customers. More than 1B push notifications are sent each month. A dedicated Consumer Communication Gateway system handles such a high volume of messages.
Different Uber's internal teams send messages about new restaurants, promotions, or offerings. This initially created a variety of quality and timing issues.
Notifications were conflicting, duplicated, and without a custom frequency for each user. This forced the marketing team to manually check notifications without any automation. To solve this issue, Uber built a system called Consumer Communication Gateway.
The Gateway receives all the notifications that the various teams at Uber want to send to a user. Each notification is then inserted into a buffer, scored according to its usefulness, scheduled, and delivered.
The Gateway is implemented by 4 main components:
The Persistor receives notifications and metadata via gRPC, storing them in non-volatile storage called inbox. This is an array of MySQL data stores sharded by user-UUID for horizontal scaling. The inbox abstracts any detail allowing access to data as document-like structures.
The Schedule Generator fetches all the notifications for a user every time the Persistor writes into the inbox. Notification scheduling is then computed using ML and linear programming. Finally, a Machine Learning model assigns a score to each notification. Each score represents a user's probability of making an order after receiving the notification. Next, the scores, the possible delivery times, and other constraints are given to a linear program solver. This solver determines the optimal scheduling.
The Scheduler is a distributed cron-like system that can be triggered 10K+ times per second. It receives the notification and the delivery time, beginning the delivery process at the right time. This is done using Cadence as an orchestration engine in combination with Kafka.
The Push Delivery component does some last quality checks before sending a notification. The notification is then sent downstream to 3rd party services like FCM or APN. This component works asynchronously behind a Kafka buffer to provide retries if the delivery fails.
For more details about the ML algorithms used by the Schedule generator, please refer to the original article in Uber’s engineering blog.
Test Driven Bug-fixing
I spend most of my working time prototyping software and I usually find it hard to use Test Driven Development. The main reasons is that I don't know the final result of my code and I continuously experiment and change specifications.
Nevertheless, I try to write tests as much as possible. Bug-fixing offers a perfect chance to do this. This is the procedure I follow:
find a bug
write a test replicating the bug
run the test
check that it fails as expected
fix the bug
run the test and check that it passes
The second step is the most important. The test needs to be written before even trying to fix the bug. Somebody could argue that it is inefficient if the bugfix is 2 lines of code and the writing of the test takes 50 lines.
But I always find the effort of creating and maintaining these tests more than acceptable. So first, the overall number of tests will increase, and second, the test will remain in my test suite to avoid regressions.
Coding Challenge
The coding challenge of the last issue was Middle of the linked list. Given the head of a linked list, the problem asks to find the middle node. In case the length of the list is even, and there are two middle nodes, the problem asks for the 2nd middle node.
The problem is relatively easy but requires a good understanding of how pointers work and the ability to manipulate them. For example, the brute force approach would be to iterate the first time the list, find the list's length, and then iterate a second time to find the middle node.
A more elegant and efficient approach is to use 2 pointers moving at different speeds. On each step, the fast pointer advance by 2 nodes and the slow pointer by 1 node.
This way, the slow pointer reaches the middle node while the fast one reaches the end. This is because the slow pointer moves at half the speed.
The time complexity is O(N), where N is the length of the list, and the space complexity is O(1) since no additional memory is required. Here is the code for the solution.
The coding challenge for the next week is Unique Number of Occurrences.
Interesting Tweets
Seniority in software engineering is much more than money and promotions. Becoming a senior means taking on more responsibilities like making the right trade-offs, adding value to the business, giving reasonable estimations, mentoring, and sharing knowledge. This thread greatly summarizes many traits of senior engineers.
Understanding time and space complexity is essential not only for interviews but also for actual development. This thread shows that having deep math or technical knowledge is unnecessary to estimate your code's complexity.
