Lessons I learned growing from an IC to an director of engineering
I’m Jinpeng, the engineering director and head of TiDB storage team. I started my engineer career in 2009 when I graduated from college, and worked in software industry for more than 14 years since then. I mainly worked in the infra domain(NLP, Full Text Search, Large Scale Distributed Storage, SQL Databases, etc.). The following diagram shows the rough timeline of my career, it consists of 4 stages, two stages in big companies and the other two in startup. If consider the total time, I spent 10 years in startups and 4 years in big companies.
Today I will share lessons learned during my journey from an IC(individual contributor) to a first-line team leader, and then to an engineering director. BTW, all these happened in PingCAP, my current employer. PingCAP is a startup started from 2015, the main product of PingCAP is TiDB, an open source distributed SQL database. PingCAP also provide TiDB Cloud, a fully managed cloud database service.
Personal output VS Team output
For the first 7 years of my engineer career, I was an individual contributor(IC) for most of the time. As an IC, my thinking logic was “what project I need to maintain, what feature I need to deliver, what online issues I need to resolve, what is the difference between solution A and solution B, what new technical skill I need to learn, etc”. All these thinking logic are focus on what output I can make directly, what problem I can resolve directly.
When I joined PingCAP at July 2016, there were around 10+ engineers at that time, including 3 founders(Max, Ed, Dylan) who also were engineers and still wrote code then. So I started my PingCAP journey as an IC since there were just around 10 engineers in total. I kept focusing on personal output and did pretty good, and was top 10% performer for the 2017 annual year. I performed well because I was already an experienced IC role before.
As our product TiDB used by more and more customers, PingCAP employed more engineers, 50 in 2018, 100 in 2019, … So we need more team leaders to do management works beside engineering works. And I started to manage a 4~5 engineers team as a team leader for my 2nd year in PingCAP, focused on performance and RocksDB engine. I followed IC working style and focused on my personal output. Personally I done a lot of engineering works and had good output, but if we consider the whole team, I did bad. Team had a weak talent ladder because I thought “hiring people is HRs’ responsibility”, I did too many oncalls and occupied the grow chance of other team members. I got a “B” which means “lower than expected” for my 2nd annual year’s performance review, although personally I has good output (from IC’s perspective).
The “B” gave me a shock, let me rethought what is the difference between an IC and a team leader. And then I read dozens of management books, from Peter Drucker to Fredmund Malik, and followed by many practices. Today I can’t clearly remember what these books have said, but I can say these books as well as practices changed my thinking logic. I cared more about “the whole team” rather than myself since then.
- What are the top priority things for the whole team? If we don’t do something in advance, will it become a big debt in the future?
- What business value our team should deliver? Are our team goals aligned with the company’s goals?
- How to guarantee each team member has expected output and growth?
- Does my team need to hire new talents? What differentiate value we hope these new members to deliver?
- …
Build a team with diversity
I made a mistake that preferred to hire engineers have similar knowledge and experience as myself in the first year of my team leader career. It is easier to understand each other and communicate with each other. But as time went on, I realized that with too many engineers have highly similarity may limit the ceiling and possibilities of the whole team.
Let me show a concrete example. As more and more users and customers used our database, we need to provide rich enterprise tools as well as a stable database kernel to our users. So there were some new projects like backup&restore and change data capture were incubating. These projects required a lot of knowledge of our database kernel mechanisms like storage engine, transaction mechanism and GC as well as knowledge and experience of ecosystem. When these new projects started, there were few engineers in kernel team can participate with these important projects, so we hired new engineers and constructed a few new teams. And we had a lesson learned that these new engineers were frustrated to make good design when lacking of knowledge of kernel mechanisms. It delayed these projects and introduced some technical debts that today we are still repaying.
Focus and listen to your real customers
Most engineers tend and like to build new features from scratch, because in this progress they can feel the thrill of creation and building, from 0 to 1. In the early days of TiDB, because it was a new database and lacked of many database features, so our engineers added more and more features and functions. But in this way, we lost our focus. Losing focus is very dangerous for a startup, the only way to achieve excellent result and high ROI is focus. TiDB supported JSON even 5 years ago, but it was just GA last year, 4 years after it was first supported. It was not GA for the past 5 years, so it created less value for our real customers even we had invested many resources on it 5 years ago. It also took some resources to restart this legacy feature. The feedback of this feature was delayed a lot since no concrete user used it after it was delivered. Even the original engineer who have implemented this feature has left the team.
When start a new project or feature, we should clearly answer following questions:
- Who are your real target customers for current stage?
- What problem does this new project / feature resolve?
- Which user scenario need this feature?
- What the whole user journey looks like?
- Any other sibling features also are must to have to make the user journey fluent for this specific scenario?
Focus and planning development works by scenario for your real customers, not by feature and for unclear customers.
Quick win VS big plan
Many engineer managers or tech leaders are prefer to make big plan but failed in execution. Comprehensive architecture refactoring, building a new unified store, a totally different test platform, etc.
Things can change and be different than when we made the plan. Better to split a big plan into several milestones, at lease quarterly milestones. Each milestone has clear goals and user observations part. So we can focus on the execution of the next milestone, and do regular review based on milestones. And regular reviews are a good opportunity to make some changes to our original plan, as things may have changed and we need to adjust our knobs in time.
I was not doing well with the Partitioned Raft KV project. It was a big architecture change for the whole storage layer. I made the whole plan 2 years ago and started the execution. I did break it down into milestones, but these milestones were not followed well, nor were the regular reviews done well. It was originally planned to be delivered in one year, but it actually took nearly two years in total.
