Arista Warrior, 2nd Edition

Chapter 4. Merchant Silicon

If you’ve shopped for data center switches with any of the major networking equipment vendors recently, you’ve likely heard the term merchant silicon thrown around. When I wrote the first edition of this book, there was a lot of back and forth between the major players about custom silicon versus merchant silicon, and which one is better. Although many of the big players have adopted the ways of merchant silicon, there are still times when vendors will try to convince a potential customer that the Arista way is the wrong way. Let’s take a look at the details and see whether one really is better than the other.

The Debate

To begin, let’s define our terms:

Custom silicon

Custom silicon is a term used to describe chips, usually Application Specific Integrated Circuits (ASICs), that are custom designed and usually built by the company selling the switches in which they are used. Another term I might use would be in house when describing such chips. As an example, Cisco Nexus 7000 switches use Cisco-designed proprietary ASICs.

Merchant silicon

Merchant silicon is a term used to describe chips, usually ASICs, that are designed and made by an entity other than the company selling the switches in which they are used. I might be tempted to say such switches use off-the-shelf ASICs, though that might imply that I could buy these chips from a retail store. I’ve looked, and Walmart doesn’t carry them. As an example, Arista’s 7280R switches use Broadcom’s Jericho ASIC.

So that seems pretty cut and dried, but which one is better? That all depends on what you mean by better. Let’s take a look at the benefits and drawbacks of each. First, these are the benefits and drawbacks of custom silicon:

Benefits of custom silicon

• Can be designed to integrate perfectly with a custom operating system

• Can be designed to support proprietary features

• Can be purpose built

• Can provide a significant competitive advantage due to the previous bullet points

Drawbacks of custom silicon

• Requires expensive on-staff expertise

• Requires expensive fabrication facilities

• Often slow to market

• Return on investment (ROI) can be slow

• Long ROI can lead to longer product life cycles

Now let’s take a look at the benefits and drawbacks of merchant silicon:

Benefits of merchant silicon

• Easy to design around with well-supported APIs

• ASIC vendors are motivated to make stable, successful, and fast products

• Fast to market

• ASIC vendor does one thing: make ASICs

• No overhead involved (no expensive ASIC designers to staff, expensive manufacturing facilities to build and maintain, etc.)

• Easy to implement

Drawbacks of merchant silicon

• No custom or proprietary hardware features are possible (the chips might support proprietary features, but anyone that uses these chips has access to them)

• No inherent competitive advantage; any vendor can use same ASIC, although the implementation might be better with one vendor over another

Arista and Merchant Silicon

Arista uses merchant silicon exclusively for all of the reasons listed, but what about the drawbacks? The two drawbacks I listed for merchant silicon seem pretty severe to me, especially the one about there being no competitive advantage. I mean, isn’t that why people buy one brand of switch over another, for the competitive advantages?

When I say there’s no competitive advantage, I mean that there is no competitive advantage to using that ASIC compared to another vendor using that ASIC. There are a couple of things to take into consideration with that statement, though. Let’s take a look at the Arista 7280SR-48C6 as an example. It uses the Broadcom Jericho ASIC to deliver a 48 front-panel interface of 10 Gbps nonblocking goodness in addition to six additional 100 Gbps ports, all in a 1–rack unit (RU) box. Many other vendors offer similar switches that use the Broadcom Jericho ASIC. Arista’s advantage in this space is that it has very efficient, modular, and portable hardware designs, and when a newer ASIC such as the Jericho2 came out, Arista quickly incorporated it into new products such as the 7280R2. Other vendors might very well have the same ability, so this advantage might be small or fleeting, but it exists nonetheless. Remember, too, that how a vendor implements an ASIC can have a tremendous advantage. This is one of the areas where Arista shines.

Another issue is the idea that no proprietary features are possible, and that’s true so far as the ASIC hardware is concerned. Arista overcomes this limitation by differentiating itself with its Extensible Operating System (EOS). Much of this book is dedicated to the features of EOS, so I won’t go into them here, but suffice it to say that EOS gives a significant competitive advantage to Arista that, so far as I’ve seen, can’t be matched by any other vendor.

Proprietary features can be a good thing, but they can limit the ability to expand a network using different vendors and, in some cases, cause designs to be so tightly integrated into a single vendor as to cause severe limitations in the future. This limitation, commonly called vendor lock, can be a real problem when it comes time to upgrade the network. It’s worth noting that Arista is generally against vendor lock on the principle that given the choice, customers will choose the best option available. Instead of trying to force customers into buying a solution, Arista instead strives to be the best choice.

Perhaps the most compelling argument for the success of merchant silicon–based switches is that some of the biggest proponents of custom silicon have released merchant silicon switches, and why wouldn’t they? If the SuperCool6000 ASIC is an advantage for Arista, and anyone can buy them from BroadTellSuperCom, every other vendor has every right to build the best switch they can, using the same hardware. It’s up to you to decide whether Cisco’s NX-OS is a better choice than Arista’s EOS. At this point, I don’t think I need to tell you what I think.

Fast-forward to 2019, and nearly all of the major networking vendors have followed Arista’s lead and are now using merchant silicon. Arista’s biggest competitor, Cisco, uses the Trident 2 chipset in its Nexus 9500s, though it does it in combination with its own custom ASICs.

Finally, there are some very cool aspects of Arista that aren’t quite obvious to the casual observer. For example, though Arista uses the same ASICs as many other vendors, there have been times where it has improved the ASIC vendor’s SDK in order to accomplish things with the chip that other vendors can’t figure out how to do. I like to say that Arista has some of the best developers in the world, and that’s what makes Arista the better choice, even when using the same off-the-shelf ASICs.

Arista Product ASICs

EOS offers the ability to show what ASIC is installed in your switch. To see the ASIC in use, use the show platform ? command. Here’s the output from a 7010T, which shows that the ASIC is a Trident:

Arista-7010T(config)#sho platform ?
  WORD           Forwarding Agent name
  pending-state  Show all agents state
  pkt            CPU packet info
  schanaccel     S-Channel Accelerator Info
  smbus          Smbus-device info
  trident     	 Trident chip

Here’s the output from a 7150S-24, which shows an FM6000 ASIC:

Arista-7150(config)#sho platform ?
  fm6000      FM6000 chip
  pkt         CPU packet info
  schanaccel  S-Channel Accelerator Info

The choices offered by each switch are different, depending on the ASIC installed. Here are the first few options for the 7010T:

Arista-7010T(config)#sho platform trident ?
  CHIP               Chip name or pattern. eg. Linecard*
  SLICE              Slice name or pattern
  agent              Agent information
  copp               Trident CoPP information
  counters           Trident debug counters
  [--output truncated--]

And here are the first few options presented on a 7150S-24:

Arista-7150(config)#sho platform fm6000 ?
  acl                         Alta ACL information
  agileport                   Agile ports let groups of 4 SFP+...
  bst                         Show internal bst registers
  copp                        Control plane policing
  counters                    FM6000 debug counters
  [--output truncated--]

If you can get your hands on an Arista switch, I encourage you to dig around in these commands, because there is some really useful information in there. Although I used to include a list of what switches use what ASICs, I stopped doing that because they can update far more quickly than the production cycle of an O’Reilly book.

Certain ASICs provide certain features. For example, the DANZ feature called Agile Ports is available only on Arista 7150 switches due to the FM6000 ASICs they contain. Because the 7280R does not use the FM6000 ASIC, the switch does not support this feature. The 7280R switches use Jericho ASICs, and thanks to Arista’s EOS and some of the world’s best developers, these switches can support internet-scale routing tables. In fact, they can support multiple copies of said tables (see Chapter 20).

In your day-to-day network operation duties, do you care what ASICs are in your switches? Probably not. Still, it pays to know what you’re talking about when the vendors come a-courtin’.

It’s also important to consider what sort of power we’re talking about here. Consider this: the Arista 7150S-52 supports 52 10 Gbps nonblocking Ethernet ports in a 1 RU switch, using one ASIC. The admittedly aging (but still widely deployed as of 2019) Cisco 6509 supports only 28 10 Gbps nonblocking (in theory) Ethernet ports, and that’s in a full 15 RUs, consuming much more power and producing much more heat. It also uses a lot more than one ASIC to do it, which is one of the reasons that these big switches consume more power and generate more heat. The 6509 is capable of many more enterprise features and is almost infinitely more expandable than the Arista 7150S-52, so it’s not a strictly apples-to-apples comparison—unless all you need is nonblocking 10 Gbps port density, in which case the Arista 1RU switch wins handily.

I should note that not all Arista switches use only one ASIC. The 7280R2 supports many 10 Gbps ports in a 1 RU chassis, but it does that using Jericho+ ASICs, which happen to be the same ASICs used in the Arista 7500R2 chassis switch.

The Cisco 6509 is a great switch, and I’m not knocking it here. It is, however, a great example of the long product cycle induced by the custom silicon mindset. Though it supports high-density 10 Gbps blades, with only 40 Gbps available in each slot (using Sup 720s), those blades are highly oversubscribed. It’s been around for a long time, but there are still a lot of them out there. Newer switches from competing vendors have better specs, but most of them are using merchant silicon these days.

There is one more potential benefit to merchant silicon, and that is the possible future of Software Defined Networks (SDNs). Think of an SDN as a cluster of switches, all controlled by a single software brain that is running outside of the physical switches. With such a design, the switches become nothing more than ASICs in a box that receive instructions from the master controller. In such an environment, the switch’s operating system would be much simpler, and the hardware would need to be commoditized so that any vendor’s switch could be added to the master controller with ease. Merchant silicon–based switches lend themselves to this type of design paradigm, whereas a custom silicon solution would likely support a master controller from only that switch’s vendor.

Note

This is a bit of an oversimplification of the idea behind SDNs, but it does seem to excite the executives who hear about it. We’re a few years away from this application in my opinion. Currently, SDN-type features are being used for things like security and monitoring.

To add to my statement that we’re “still a few years away” from such an application, while writing the second edition, I was amused as I read that some three years after the first edition was published because I still think we’re “a few years away” from the SDN dream. There’s an old joke from the telecom world that ISDN stands for I Still Don’t Know. I find it amusing to be able to recycle old telecom jokes.

Will SDN become a widespread reality? I don’t know. I think the idea has merit, but as of mid-2012, er, 2019, I wouldn’t be making any purchasing decisions based on it. That viewpoint may change in the next few years. Arista has a cool new tool called CloudVision that is a step in the right direction, and we talk about that a bit later in the book.

There’s another topic that seems to come up when merchant silicon is discussed, and that’s the idea of white-box switches. The concept of white-box switches is simply that you can buy a switch as a generic commodity product that will accept whatever network operating system you’d like to install. Although this sounds like a good idea to many, in my experience very few installations are actually following through with it. The consultant part of my brain seems to think that executives would rather have a company they can yell at when things go wrong (the one throat to choke paradigm). Additionally, if you want the best network operating system, I’m going to go ahead and recommend EOS, and if you have EOS, you might as well get some of that sweet Arista hardware to go with it, no?

Note

Here’s where I begin to wonder whether I’m in too deep and have become biased. Then I think about the hundreds of companies I’ve worked for or with and think about what I know about the industry as a whole. When I step away from my Arista goggles for a bit and survey the landscape, I realize why I’m still at Arista some six years later: Arista still has the best solution.

Conclusion

So, which is better: custom silicon or merchant silicon? As far as Arista is concerned, merchant silicon is the path they’ve chosen. I can’t tell you which is better, because I’ve seen great products from both camps. I will tell you that I really like what I’m seeing as a result of the competition caused by vendors moving to merchant silicon. Take EOS, for example. I think it’s the best networking operating system I’ve ever seen. If that came to be as a result of using merchant silicon, then I’m a fan.