Chapter 4. Evaluating and Choosing Technologies

Without doubt, at some point you will have to evaluate and choose between technologies. You could just throw a dart at the wall or trust your gut but in the long (and short) term it is far better to follow a more objective approach. This chapter will explore what it takes to evaluate a new technology with a set of criteria you can use on your next assessment. It also includes a case study that uses this technique to compare Angular and React.

Evaluation Criteria

There are any number of things we can use as evaluation criteria. Depending on what kind of technology you are exploring, some will make more sense than others. Ultimately you will be guided by your own experience as well as the things that matter most to your organization. The following are a set of criteria I typically use when evaluating a new technology:

Documentation

Is there any? The era of “the code is the documentation” is over. Is the documentation clear and concise? Is it up to date? Is there enough to get you started as well as enough to keep you going? What you need from the documentation will change dramatically from week two to year two.

Community

What is the size of the community behind this technology? More importantly, is it thriving? A small but passionate community may be better than a large apathetic one. Does the community welcome new members or does it prefer to make them feel inferior for the first few years? What does the community value? Some languages encourage testing, others fetishize obfuscation.

Committer diversity

Many open source projects have corporate backers. While that isn’t a bad thing, it is important to ask what happens if that company decides this technology isn’t core to their business anymore. What happens if the major corporate backer walks away? If the community is large enough, it probably won’t have much of an impact. Development may slow somewhat but well-established projects will continue.

Codebase

You won’t get to read the code for most vendor projects but if you are evaluating an open source project, you should spend some time in the source files. Is it readable? Are there tests? Would your team be able to fix an issue if they had to? Could you fork the project if it became necessary?

Testability

Does the technology in question encourage testing? Does it make it nearly impossible to write automated tests?

Update history

When was the last commit? When was the last release? You should distinguish between dead projects and mature ones. Check the issue tracker. Are there heaps of open tickets? Are the forums filled with tumbleweeds? The project might be abandoned.

Maturity

What is the version number? While some months-old projects boast of higher version numbers than decade-old established technologies, be cautious with anything with leading zeros. Newer tools are likely to change frequently and many of those changes will break your code. If there are major changes on the horizon, what is the upgrade path? How quickly are new releases coming out? Can your projects consume those changes at that rate?

Stability

When a new version comes out, is it solid? Or does it require several point releases before it inspires confidence? In other words, how quickly would you put a new release into production?

Extensibility

How hard is it to add something? Is there a plugin model? Or are you simply at the mercy of what the committers think matters most?

Support

Can you purchase a support contract? I know many architects find them worth less than the effort to procure them, many organizations want someone they can call for help. Can you find help online? What do you find on Stack Overflow?

Training

Can you get training for this technology? If there aren’t commercial options available, can you build your own? Are there tutorials and examples you can follow?

Hiring

What happens if you post a job looking for experience with this technology? Scan through the last set of resumes you received. Does this technology show up anywhere? What do your developers think about this technology? Do not underestimate their feelings; developers will vote with their feet. If your developers are avoiding this technology like the plague, you should reevaluate the process that led you to this point.

Corporate fit

Does this technology align with your corporate culture? What does your company value in a technology? Some technologies will be a breaking change for your company. Will there be any political roadblocks? Does this technology compete with an existing solution in your organization? What guidance will you provide teams on using one over the other?

Security

Does this technology have any known security issues? Are security vulnerabilities fixed in a timely fashion?

Licensing

How is this product licensed? If it is open source, how will your legal department react to the specific license used? If it is a commercial product, perform due diligence on the vendor. Are they well funded? Do they have a strong track record? Do they have satisfied customers?

Usage

Be aware of the lemming effect in software, that tendency to follow the crowd. However, if no one else is using a given technology put in the effort to figure out why. Software archeology of this sort is a mix of clever web searches, sifting through forums and consulting with your professional network. It may be a niche product or an undiscovered gem. Or everyone else might know something you don’t yet. Be sure you know which bucket this technology falls into.

The Spreadsheet Approach

Spreadsheets make it easy to show how various options fulfill (or don’t) a set of criteria. In general, put your options across the top and your criteria down the side. Some criteria may be more equal than others: don’t be afraid to weight them. For example you might decide that hiring is a bigger deal than extensibility so you give hiring more emphasis. Often, this is a factor you multiply the score by to give it a higher weighting in the overall score. I strongly advise the use of Harvey Balls in place of a numeric scale. Harvey Balls are just an ideogram ranging from an empty circle to a filled circle with one quarter increment in between (e.g., ◔ or ◑).

While we can argue over the meaning of “1” an empty circle is unequivocal. If you use 1–5, part of your audience will think a 1 is good even if you repeatedly say a 5 is the top score (or vice versa). If you must use a numeric scale, be consistent. Nothing is as confusing to your readers as a scale that randomly changes.

Tipping the Scale

Resist the temptation to put your thumb on the scale of an evaluation. You can always structure an “evaluation” to prove whatever you want. But in nearly every instance, your audience will notice you stacked the deck. If you have to rig the game, what is the point of an evaluation?

The criteria you choose and the weighting (if any) you apply is completely up to you. In some circumstances the criteria might be defined for you. But in general, you will have to determine the most important aspects to evaluate.

Proof of Concept

While you can always perform a paper-based evaluation, few things beat hands-on experience with a technology. I strongly advise you to acquire first-hand knowledge of anything new. Of course, you will never have enough time to work on a proof of concept, you will have to be efficient with your work, but you will have to focus on the aspects that will teach you the most about this technology.

Hello World won’t teach you enough and you need to go beyond canned demos. Before starting a proof of concept, identify the areas you need to understand. What are the key use cases? What are your primary concerns? Tune the demo application to hit the major points that will give you confidence to make a decision one way or the other.1

A proof of concept should help you identify risks. As an architect, your job is to shed light on the risks, but others generally decide what to do about them. Keep track of the risks along with their impact and likelihood. You can document them in anyway you like: spreadsheet, wiki, whiteboard, etc. Every project does it their own way.

Embracing Risk

Risks are not inherently bad. Every time you cross the street, drive a car, or go for a bike ride, you are taking some risk. We embrace risk in our daily lives2 just by getting out of bed.

Risks should not paralyze you, but you have to manage them appropriately. You may decide you are unwilling to accept a risk or that you want to transfer that risk to a third party, but there is nothing you can do to remove all risk from a project. You can, and should, proactively identify them and work to mitigate them.

Politics

Many people get into software because it is an objective field. Your code compiles or it doesn’t, your tests pass or they don’t. But there is ample room for ambiguity in software.3 While I wish it were not so, politics will play a role in almost every technology decision you are a part of. It doesn’t matter how big or small your company is, you should prepare to deal with the nontechnical aspects of an evaluation.

Gauge the political winds at your organization. Talk with your peers, especially those with the longest tenure in your company. What technologies have had the roughest roads to adoption? Why? Consider the legacy technology you are trying to replace or augment. Who might see their role diminished in this new world order? How will they react to that? Can you identify who the most influential people are in your company? Aligning them to your goals can work miracles. Will the decision makers be receptive to the technology you want to introduce? If they aren’t, you might be able to change their minds…but it will involve retail politics. Recruit allies. Can you tie this technology choice back to a pain point in the organization, even if it is a bit of a stretch? Do everything you can to align the technology to executive-level goals to ensure success. For example, years ago a new CIO made it clear that release weekends included far too much drama and he wanted to see improvement. A few of us used that as an opportunity to reintroduce a set of code quality initiatives we’d been advocating with limited success. By connecting that drive back to the CIO’s goal of smoother releases, we made significant progress.

Every so often, the right answer is to just do it and let the chips fall where they may. Channel your inner Grace Hopper who rather famously opined: “It’s easier to ask forgiveness than it is to get permission.” Just be aware of your organization and your place within it. If you are already in a precarious situation, watch your step.

Evaluation in Name Only

At some point in your career you will fall into an “evaluation in name only,” which has many of the same trappings of an actual evaluation. Sometimes the signs are hard to read but eventually they become unmistakeable. Negative results may be buried. A risk list might be moved to a less prominent location on the wiki. Outcomes from a proof of concept might be heavily couched.

As much as you may want to perform an unbiased appraisal, there may be vested interests in the outcome. You may or may not be aware of these motivations. In many of these instances, an “evaluation” is simply cover for a decision that was already made. Your work will be spun to prove whatever narrative has already been chosen.

Many organizations have very centralized (and hierarchical) decision-making structures. In most cases, you won’t really know what went into the outcome of the decision.

If you are highly influential, you might be able to alter the outcome, but don’t be surprised if you can’t. At the end of the day there is only so much you can control. Don’t overthink it, but try to understand the context of the decision and don’t be shocked if you don’t agree with it.

What you do next is a very personal decision. You may decide you can go along with the scripted result, in which case you should consider how you can insulate your projects from the technology. Perhaps another layer of indirection is in order. Add a proxy layer that hides the details of the technology from your application. If (or when) the time comes to replace the offensive library, you will only need to rework the proxy.

Some corporate cultures expect commitment to a decision but they may encourage a “dissent memo” where you can outline your disagreements. Simply having an outlet for your thoughts can be valuable.

Depending on how strongly you feel, it may make sense to explore other opportunities. Working for a company when you disagree with the corporate direction is challenging at best. If you aren’t happy, it will show up in your work (and your personal life as well). Never forget that you are responsible for your career.

Case Study: Angular Versus React

How might this approach look applied to Angular and React? First things first, any comparison is highly subjective. There is no magic formula! Some people actually reject this comparison outright because “Angular is a framework and React is a library.” Pedantic yes, but don’t be surprised when a colleague rejects your hard work based on a type coercion error.

Documentation is a plus for both tools. Both have good tutorials and extensive guides. The communities are largely comparable with Angular being slightly larger. Though both are backed by large companies, they have strong committer diversity. React’s extensive contributing page is excellent.

React and Angular have clear codebases with a strong focus on testing and testability. Both are updated regularly although the Angular 2 conversion was disruptive.4 Angular has a regular upgrade cadence with a deprecation period. A predictable schedule helps planning but can your projects consume changes at that rate?

Both are mature and stable. React often involves stitching together multiple libraries that in turn move at their own pace. In terms of extensibility, Angular allows you to swap out just about every component with something else if you wish and React is built around projects crafting their own lightsabers as it were.

As massive open source projects, each returns a plethora of search results. For better or worse, Stack Overflow has far more Angular questions. Is that good or bad? It may indicate a larger community, but it also might mean developers struggle to understand Angular.

How Many Questions Should We Find on Stack Overflow?

Stack Overflow is a fantastic resource for developers. Search on almost any topic and you will find good answers and strong opinions. When evaluating a new technology, spend some research time on Stack Overflow.

Finding no questions (or no answers to very few questions) should give you pause. A dearth of results indicates the technology is either very new or not popular enough to warrant a community yet. An endless list of questions about a library should also cause your spidey sense to tingle. It may indicate insufficient documentation or a framework that has far too many issues to be trusted in production.

Ideally, we want a “Goldilocks” set of questions. Enough to ensure us we can find help if we need it, but not so many as to give us acid reflux about the health of the technology.

Online and live training abound for both Angular and React. Hiring for either should not be difficult though you may find more Angular experience on resumes. Both are widely used with massive properties backing them.

All in all, this is largely a toss up! While my subjective reasoning gives a slight edge to Angular, I prefer React. That said, it is hard to go wrong either way. Your project might have a very different set of needs and priorities, which is exactly why you need to perform your own evaluation before making a decision. While you should spend time exploring the choices others made, nothing replaces your own experience and intuition.

What Should You Choose?

There is no one answer to which technology you should choose.5 Are you making a decision for your project or your entire company? The larger the impact of the choice the more consensus you will need to achieve. In these situations, spend the necessary time with the interested stakeholders to earn their buy-in. Overpowering them rarely works and usually just breeds resentment. Overcommunicate why you think your choice is the right one and be prepared to talk through it.

For project-level decisions, you still need to make sure your team understands why you went a certain direction. They may not agree with your decision, but you should be very open and transparent about how you arrived at it. Do your homework. Be strategic!

Katas

Now that you understand the importance of a thorough evaluation as well as the impact politics can have on a technology decision, put that knowledge into practice in your world.

  • Compare and contrast two similar technologies.

    • What criteria did you choose? Why?

    • Which one came out on top? Why?

  • Consider a technology you would like to introduce to your organization:

    • What political roadblocks exist?

    • How could you manage the politics of that technology?

1 It should go without saying that the vendor should not perform the evaluation for you.

2 Some more than others—I’m looking at you skydivers.

3 Just go ask one of your business analysts.

4 Enough so that one friend said “Angular 2 broke me…HARD!” Your mileage may vary.

5 Beyond “it depends” at least.

Get Thinking Architecturally now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.