High-speed Ethernet Planning Guide
Ten-gigabit Ethernet was so last year.
Standards-based 40- and 100-gigabit Ethernet switches and routers are starting to show up in enterprise networks, following ratification of the IEEE 802.3ba specification in mid-2010.
It's easy to understand the motivation: Fast downlinks require even faster uplinks. The current solution, link aggregation of multiple 10-gigabit pipes, works well but only scales up to a point.
At the same time, servers for some high-performance applications now use 10-gigabit network interface cards, again requiring a faster uplink at the switch. It won't be long before 10-gigabit interfaces will be a standard part of server motherboards, just as gigabit Ethernet comes standard today.
For network managers, migrating to "higher-speed Ethernet", as it's been dubbed by the Ethernet Alliance, will definitely require some changes. Most of these are at the physical layer (new cabling is required, for starters). Also, some monitoring and management gear may not be able to keep up with HSE rates.
On the plus side, HSE will help reduce prices for 10G Ethernet devices. "The real leverage [with HSE] is with pushing down the price point of 10-gigabit Ethernet, rather than the first-order effects of 100-gigabit deployment," says a senior architect at one of the largest ISPs in the U.S., who requested anonymity. "If bigger pipes are good, then bigger pipes that are affordable and create greater commoditization of 10-gigabit Ethernet are better."
Also, HSE is far more evolutionary than revolutionary. Network professionals who've worked with Ethernet will feel right at home with the 40G- and 100Gbps versions. Still, an understanding of what's new is essential (see sidebar: "5 steps to high-speed Ethernet").
No design changes
Migration to 40- and 100-gigabit Ethernet requires no modification to upper-layer network design, networking protocols, or applications. "It all looks the same to the upper layers," says Brandon Ross, Eastern U.S. director of network engineering at Torrey Point Group, a network design consultancy. "There aren't any changes needed."
That means, for example, that a network using the rapid spanning tree protocol (RSTP) between switches and open shortest path first (OSPF) between routers can continue to run these protocols across HSE interfaces, without configuration changes.
Applications, databases, and server farms similarly won't be affected by the addition of HSE interfaces to enterprise networks. Lower latency and improved response time should be the only noticeable effects, although Ross cautions that adoption of faster networking technologies inevitably exposes bottlenecks that weren't previously visible. If, for instance, network latency previously masked a disk I/O bottleneck and HSE is now faster than the bottleneck, application performance won't improve as much as expected.
That raises a critical question when it comes to HSE adoption: Even if the protocols are ready, is the network infrastructure ready for higher speeds?
As with previous speed bumps, capturing and monitoring traffic at higher rates will impose higher horsepower requirements on network management and security systems. The old adage "you can't manage what you can't see" still holds true. In other words, the requirement to see all traffic doesn't go away just because the network got faster.
A migration to HSE may require upgrades to network management systems and security devices such as firewalls and intrusion detection/prevention systems. Network managers are well advised to check with vendors as to the maximum frame rates their devices can support with zero frame loss.
Faster versions of Ethernet pose special challenges for security devices that use Secure Sockets Layer (SSL). Dedicated hardware already is needed to encrypt and decrypt SSL traffic at gigabit and 10-gigabit rates. Even with the hardware assist, next-generation firewall testing done by Network World suggests that security devices move traffic far more slowly when handling SSL traffic.
Performance penalties for SSL traffic at HSE rates are, if anything, likely to be even steeper. Network managers may want to consider deploying dedicated SSL decryption/encryption systems alongside existing security devices to mitigate SSL performance bottlenecks.
Another key observation point is the "span" or monitor port capability in many Ethernet switches. This feature allows traffic entering or leaving a switch port (or both) to be copied to another port, where it can be redirected to an external analyzer. Some switches support multiple monitoring instances. In an HSE context, the key question is whether the switch can copy monitored traffic at 40- or 100-gigabit Ethernet line rates with no frame loss.
Even if the switch can support such a capability, there is then the question of whether the analyzer will be able to capture frames in real time. Software-based protocol analyzers are woefully underpowered for this task.
What's needed here is a hardware-based analyzer capable of lossless traffic capture at 40G- or 100Gbps rates. The analyzer also will need much larger storage capacity. At 100Gbps rates, an analyzer capturing 1,518-byte frames at line rate will need to store 750GB of data per minute. At 40Gbps rates, the storage requirement is "only" 296GB per minute, but keep in mind both numbers are only for a single port. Hardware-based monitoring tools often are used to capture multiple simultaneous feeds, each with very significant storage requirements.
A final item in the monitoring checklist is network taps - line splitters capable of HSE rates. Not all network devices have embedded monitoring capabilities, and there are situations in which monitoring can't be enabled for administrative or technical reasons.
In these cases, an external network tap is needed. These are available in copper and fiber versions with anywhere from two to dozens of ports. The key question with these devices, as with all gear monitoring HSE traffic, is whether they can keep up with 40G- and 100Gbps rates without dropping frames.
Next Page: Standards and fiber...