Senior engineers always find their own work
Thank you so much for resonating with what I share in this newsletter. I hope the content I share and the resources I list are making you a better engineer and accelerating your career growth. If you want me to share something else, let me know by dropping me a message or leaving a comment on the post.
If you find this newsletter helpful, do spread the word about my newsletter on social media. It would mean the world to me :-)
Senior engineers always find their own work
Beyond a certain level, you are required to come up with new ideas and do something innovative to get promoted, but how exactly?
At some point in your career, you are required to propose some potential projects that your team or your organization can take up. Coming up with project ideas is a sign of a good senior engineer.
It indicates that you are actively looking at potential gaps in your product and thus solving them can lead to an increase in revenue, cost reduction, or an improvement in user experience. To come up with new projects, here are a few things you should actively do
talk to your customers and understand potential gaps
keep exploring new things, every single day
prototype to build a practical understanding
watch conference talks, explore new GitHub projects
understand the nuances by going deeper through source code and papers
It takes one paper, one blog, one talk, or one conversation, to come up with one insight that will change your trajectory. So, instead of waiting for your manager to give you that magical project, pitch yours.
Remember, a senior engineer finds their own work.
You can find this post on my LinkedIn and Twitter; do leave a like.
📹 Video I posted this week
This week I posted Everything you need to know about CockroachDB w/ Ben Darnell
Why is it impossible to kill CockroachDB?
I had this question for a long time and who better to answer this than the co-founder himself? hence I hosted Ben Darnell for a deep-tech conversation on CockroachDB and Distributed Databases.
We went into details of all three layers - distributed storage, distributed execution, and the language interface of Cockroach. Two things that blew my mind were how they
built SQL layer on top of distributed ordered KV pairs
provide strong consistency even when the DB spans the entire world
ps: while discussing the future of databases, Ben suggested a really interesting idea that will not only blow your mind but could inspire you to kick off an amusing open-source project.
By the way, they do have offices in Bangalore, so if you are looking to work on databases, check out their open positions. I can vouch for them :)
🧠 Paper I read this week
This week I spent reading Epoxy: ACID Transactions Across Diverse Data Stores
Given the rise in complexity of applications, it is very common to use heterogeneous data stores like a combination of ElasticSearch and MySQL, one for transactional data while the other powering search. But what if we wanted to maintain transactions across them?
This week I read a paper titled "Epoxy" which is a protocol that uses PostgreSQL as the central coordinator and its MVCC to ensure consistency. After skimming the paper once, I found the concept interesting, so putting this out as a recommendation.
You can download this and other papers I recommend from my papershelf.
How do updates work under the hood in Elasticsearch?
Elasticsearch uses Lucene to power the full-text search and lookups, and because Lucene stores data in immutable segments, ES cannot do in-place updates which makes the updates expensive.
Whenever an update reaches the engine, it has to reindex the entire document, regardless of which API is used (full or partial updates). Upon receiving the request, new Lucene segments are created and older versions of the affected documents (if any) are soft deleted. An update operation is thus an insert with some soft deletes.
Soft deletion means that the data would be lying around bloating the storage and memory of the index. Merge and compaction would spike up the garbage collection operations, putting a toll on the memory, CPU, and cluster performance.
So, if you an update heavy ES use case, do tune your cluster properly; some parameters to consider are refresh interval, shard allocation, and gc parameters like max memory, pool size, and algorithm. One thing that worked for me was to rotate ES infrastructure periodically.
You can find this post on my LinkedIn and Twitter; do leave a like.
📰 Interesting articles I read this week
I read a few engineering blogs almost every single day, and here are the three articles I would recommend you to read.
Thank you so much for reading this edition of the newsletter 🔮 If you found it interesting, you will also love my courses
I keep sharing no fluff stuff across my socials, so, if you resonate do give me a follow on Twitter, LinkedIn, YouTube, GitHub
Senior or senior+?