In practice, algorithm problems do not arise at the beginning of a large project. Rather, they typically arise as subproblems when it suddenly becomes clear that the programmer does not know how to proceed or that the current program is inadequate.
You live in a world of tight deadlines and complex software projects that use a multitude of tools. Your employers cannot afford the luxury of employing enough specialists to fill every role. You learn only enough about any tool to get today’s job done. You select a handful of tutorials on the language or library that you’re working with today. You make decisions without taking the time to understand the issues, and copy the toy examples provided with the tools. This works to the extent that you can turn your hand to anything. You acquire the ability to dive into a new technology and come up with a solution very quickly. You only ever learn the parts of a technology that you need to get your portion of the system working, and you depend on other members of the team to learn the other parts. For instance, you may be a server-side Java developer, and consequently have little or no knowledge of how the user interface was built.
You keep running into difficulty maintaining the code you’ve written because it turns out that the tutorials you followed cut corners and simplified complex issues. You find that your superficial knowledge of a thousand tools means you’re always floundering whenever a subtle bug arises or you have to do something that demands deep knowledge. People often accuse you of having a misleading CV because you don’t distinguish between a couple of weeks of extending an existing web service and a deep knowledge of the issues inherent in maintaining an interoperable and highly scalable enterprise system. What’s even worse is that because your knowledge is so superficial, you’re not even aware of how little you know until something or someone puts you to the test.
Learn to dig deep into tools, technologies, and techniques. Acquire the depths of knowledge to the point that you know why things are the way they are. Depth means understanding the forces that led to a design rather than just the details of a design. For instance, it means understanding type theory (or at least the simplification offered by the typing quadrant at http://c2.com/cgi/wiki?TypingQuadrant) rather than simply parroting the things you’ve heard others say.
As one of our ex-colleagues (Ravi Mohan, personal communication) found:
The areas where you have deep knowledge feed your confidence and guide you when you are deciding how to apply Sweep the Floor, because they indicate places where you can deliver value early in your time with a new team. More importantly, this depth of knowledge is something you can fall back on to give you the strength to tackle new areas. You can always say to yourself: “If I mastered EJBs then I can handle metaclasses.”
Another advantage of digging deep into a technology is that you can actually explain what’s going on beneath the surface of the systems you work on. In interviews, this understanding will distinguish you from other candidates who can’t describe the software they’ve helped build in a meaningful way because all they understand is one little portion. Once you’re part of a team, it’s the application of this pattern that separates out those who are making random piles of rubble (the Pragmatic Programmers called this “programming by coincidence” while Steve McConnell calls it “cargo cult software engineering”) from those who are building cathedrals.
How do we spot the cathedral builders? They are the ones on your team who end up doing the debugging, decompiling, and reverse-engineering, and who read the specification, RFC, or standards for the technologies you use. The people who do this have made a shift in perspective and built up a practiced understanding of the tools that support them.
This shift in perspective involves wanting to follow a problem down through the layers of a system and being willing to spend the time to acquire the knowledge that will make sense of it all. For instance, switching from a single core to a multicore laptop might alter the behavior of your Java concurrency tests. Some people will just shrug and accept that their tests will now behave unpredictably. Others will trace the problem down to the CPU level via the concurrency libraries, the Java Memory Model, and the specification of the physical hardware.
The tools you need to be familiar with include debuggers (like GDB, PDB, and RDB), which let you see into your running program; wire-level debuggers (like Wireshark), which let you see network traffic; and the willingness to read specifications. Being able to read specifications as well as code means that nothing is hidden from you. It gives you the ability to ask hard questions about the libraries you use, and if you don’t like the answers you get, you are capable of reimplementing them yourself or moving to a more standards-compliant implementation.
One of the ways to use this pattern is to get your information from primary sources. This means that the next time someone talks to you about Representation State Transfer, better known as REST, you should take that as an excuse to read Roy Fielding’s PhD thesis in which he defined the concept. Consider writing a blog post to clarify or share what you’ve learned, and to encourage others to read the original document as well.
Don’t just take the word of someone who is quoting a book that paraphrased an article that mentioned the Wikipedia page that links to the original IETF Request for Comment document. To truly understand any idea, you need to reconstruct the context in which it was first expressed. This lets you verify that the essence of the idea survived going through all those middlemen.
Find out who first came up with the ideas and understand the problems they were trying to solve. This kind of context usually gets lost in translation as the idea gets passed around. Sometimes you will find that seemingly new ideas were abandoned long ago, often for good reasons—but everybody has long since forgotten about it because the original context has been lost. Many times you will find that the original source of an idea is a much better teacher than the chain of people who have been selectively quoting each other for years. Whatever happens, tracing the lineage of ideas you have found helpful is an important exercise, and a habit that will serve you well through the years as you attempt to learn new things.
When you read a tutorial, you should not be looking for code to copy but for a mental structure in which to place your new knowledge. Your goal should be to understand the historical context of the concept and whether it is a special instance of something else. Ask yourself if there appears to be some underlying computer science concept behind what you are learning, and what trade-offs were made in the implementation that you are using. Armed with this deeper knowledge, you should be able to go beyond the initial tutorial when you run into problems.
For example, people often run into trouble with regular expressions because they only acquire a superficial understanding. You can get along just fine for years, maybe even decades, without really understanding the difference between a Deterministic Finite Automaton and a Nondeterministic Finite Automaton. Then one day your wiki stops working. It turns out that if your regular expression engine is implemented recursively, then on certain inputs that require backtracking it will run for a very long time and eventually throw a StackOverflowException. Ade discovered this the hard way, but luckily it happened on his toy wiki implementation and not in a production environment.
With all this focus on understanding technologies and tools in depth, you need to take care not to accidentally become a narrow specialist. The goal is to be able to acquire as much specialized knowledge as necessary to solve any problem, without losing your perspective about the relative importance of different aspects of software development.
One thing you will learn from trying to apply this pattern is that gaining deep knowledge is hard. This is why most people’s knowledge of the computer science that supports software development is a mile wide and an inch thick. It is easier and often more profitable to depend on other people’s knowledge of the fundamentals than to spend the extra effort to acquire it yourself. You can tell yourself that you can always learn it when you need it; however, when that day comes you’ll need to know everything by the end of the week, but it will take a month just to learn all the prerequisites.
Another possible consequence of possessing only surface knowledge is that you may never realize that the problem you are trying to solve either has a well-known solution or is in fact impossible (in this case, there may be a lot of academic papers on why it’s impossible and how to redefine the problem to make it soluble). If you’re only skimming the surface, then you won’t know the things you don’t know, and without understanding the bounds of your knowledge you can’t discover new things. Often the process of burrowing through all the layers of a problem will reveal an underlying concept from computer science. While the work of computer scientists can seem impractical, those who can apply the most advanced theories to real-world problems gain the ability to do things that can seem almost magical to others. Simple changes in your choice of algorithm or data structure can turn a batch job that takes months to run into something that finishes before the user has even let go of the mouse button. Someone who only knows about Lists, Sets, and HashMaps is unlikely to realize that he needs a Trie to solve his problem. Instead, he will just assume that a problem like longest-prefix matching is impossibly hard, and will either give up or ask if the feature can be deprioritized.
If you apply this pattern regularly, you will become one of those people who truly understand how their tools work. You will no longer just be gluing bits of code together and depending on other people’s magic to do the heavy lifting. Be aware that this understanding will separate you from the vast majority of programmers you work with, and will make you the logical choice for the most difficult assignments. Consequently, you will be the most likely to fail completely or succeed spectacularly. In addition, do not allow this knowledge to make you arrogant. Instead, continue to seek out opportunities to Be the Worst. Challenge yourself to assemble useful tools from these fundamental building blocks, rather than just basking in your ability to take things apart.
Find and read RFC 2616, which describes HTTP1.1, and RFC 707, which describes the state of the art in Remote Procedure Call technology as of January 1976. Armed with your deeper knowledge of HTTP, try to implement a client and a server for RFC 707. When you feel you have a good understanding of the trade-offs made by the editors of RFC 707, examine a modern open source implementation of the same ideas, such as the Apache Thrift framework that powers Facebook. Then, from your informed vantage point, write a blog post describing the evolution of our knowledge regarding remote procedure calls and distributed systems over the last three decades.
Now, go and read Steve Vinoski’s articles about RPC. Do you now have doubts about the depth of your understanding? Write a blog post about your doubts and your current level of understanding.