Ask the CTO: Balancing development desire with product goals
We can’t build systems in a vacuum--architecture is a multi-year vision requiring an iterative delivery.
Teeter-totter. (source: Laura Bernhardt on Flickr)
The problem: How do I balance the team’s desire to build quality (and new) systems with our need to hit product milestones with what we have?
I’m managing a team of developers who are working in an old codebase. It’s not a great system, the technology is pretty out of date, but it works OK for now, and we can still build new features into it. We’ve hired several new senior members of the team over the past few months, and they are agitating to move out of this old system into a new architecture. They presented a proposal for what should change and a high-level timeline that extends out a couple of years to completion. In the meantime, the product team has some features that they are betting on to grow the business, and I’m getting asked to estimate the time to build out these features. In the old system, they can be built in a few months, but the developers hate working in the old system and want to hold off until we get moved to the new architecture. Every time I ask for a compromise, the team says that breaking the new architecture down in order to deliver new features is a hack and not worth doing. What do I do?
The solution: Remember, this is evolution, not revolution
It’s important to remember that when your team brings you a long-term plan that does not address the needs of the product, they are probably not doing their job. The job is not to build technical architecture, but to create living systems that support the growth of whatever business you are part of. Figuring out how to work within constraints to unlock this puzzle, how to analyze the systems and find the right pieces to move at what time is a great intellectual challenge and a huge learning opportunity. Your job is to make that opportunity clear, while also making it clear that the option of the hermetic multi-year rewrite is off the table.
Learn faster. Dig deeper. See farther.
The myth of the perfect rewrite is still going strong in the tech industry. This myth tells us we can create architectures that take years to complete and that it’s possible do nothing but focus on rewriting our application while the business waits patiently for our completion. We are failing our teams by allowing this myth to persist. If you want to take an existing business application and rewrite it, you have to be able to break it into components and deliver components iteratively, while delivering features and improvements that the business needs. Even when you are building something that cannot reasonably be used until it is at least v1 feature complete, such as a new storage layer or database, the work for delivery has to be broken down into components. This is how complex work is completed.
It’s hard to get a sense of this in a world where people often switch jobs every 18 months, but architecture is both a multi-year vision and a process that requires an iterative delivery to that vision. An architect who can’t figure out what the distinct end-to-end components of the system look like and in what order they could be delivered is probably kidding himself in his ability to successfully complete a three-year project. I think that we do a disservice to our senior engineers when we let them continue with the illusion that it is OK to go off for years and build systems in a vacuum. I suspect you know this, but you are struggling to communicate this in a way that your team can hear you.
Practical advice: Use the carrot and stick method
The first piece of advice I have is that it is important to make it clear that you expect your tech leads, senior engineers, and architects to be able to break down big work into shorter deliverables. Moreover, help them understand that this challenge is a key part of becoming a good system designer. Systems must be designed within the constraints of the world they are supporting. In your world, that means that they must be designed to help deliver on needed features quickly. This is a challenge that provides the chance to learn something new!
There are things you can do to make this faster. Set up basic rules around the technologies you want to consider or the process for evaluating these technologies; this will help prevent fights over details about which language is best. Choosing widely adopted open source technology with an active community is one such policy you may want to adopt. There will be many teaching moments in this part of the process. For example, pushing the team to identify the decisions that are critical because they are hard to change later, versus places that are more plug and play. With everything that the team says must be finished before a new feature can be built, ask what the cost of choosing perfect now versus changing it later will be.
Keep in mind that what often slows people down in this process is fear. Fear of making the wrong decision and having to live with the shame and burden of that mistake. Fear of the blank canvas, of having to commit to starting on something enormous. Fear of failure. There are very few people who have successfully run a multi-year architecture project. Your team may be full of bravado externally but have underlying anxiety of this massive unknown.
There’s also the fact that breaking down projects and doing that planning process is not fun—and not something that your team may even know how to do. I have coached several senior engineers through their first big project management jobs, and it has not been fun for any of us. I remember clearly my first experience leading a big project. My beloved boss and I sat in his office for hours going through the plans that I had created. He would pick apart every area that had a lot of uncertainty and a lot of risk, and send me off to think more. I had to make a lot of guesses and grapple with a lot of details. It was absolutely dreadful, and I found myself deeply frustrated and impatient throughout the process. And yet, at the end of it all, we broke this big project down into deliverable chunks, and I went on to successfully lead a significant architectural change that ran close to on schedule, despite its complexity. The memory of the frustration of planning is burned into my brain, but so is the memory of the huge accomplishment that came out of that planning.
There is an MVP somewhere for their rewrite. A set of related features that can be pulled into a new system and then expanded. A way to experiment with new tools without migrating every single part of the business to those tools. They are going to make imperfect choices along the way—even with the freedom to do things exactly how they want to do them, they will make imperfect choices. In the ever-changing world of technology and business, what looks right from the outside can have a lot of flaws once implemented and in use for six months.
Wrap-up
Once the team breaks down the work into manageable pieces, you will still have challenges. Inevitably, even good plans fail in some way. You will almost certainly choose the wrong technology somewhere and want to migrate again in a couple of years. Worse, you will probably find yourself pushing back against the forces who don’t want to spend energy on technical improvements and just want to have more features. Even with a good iterative plan, major architectural changes are hard to sell to a business with a lot of feature development needs, and they will be looking for opportunities to reclaim the engineering cycles that you have allocated for this work. You must regularly deliver new features, or you are likely to lose the whole project, and if you’re unlucky, your job! So stay focused, prioritize delivering wins along the way, and steel yourself for a bumpy journey.