Transport Technologies
Overview
This chapter covers multiple transport technologies that can be leveraged as network design options from an enterprise perspective. Not all transport technology options are covered here; some are covered in later chapters, and some are beyond the scope of the CCDE exam. The transport technologies included are the most common options seen today in production networks.
The chapter covers the following topics:
- Optical Transport (CWDM and DWDM): Wavelength-division multiplexing technologies for fiber capacity expansion
- Layer 2 Media Access: Layer 2 transport and connectivity model
- Virtual Private Wire Service Design Considerations: VPLS models, concepts, and the corresponding network design elements
- EVPN Design Model: EVPN concepts, protocols, and the corresponding network design elements
Optical Transport (CWDM and DWDM)
Wavelength-division multiplexing (WDM) is a Layer 1 technology that allows multiple optical signals to share a single fiber by assigning each signal a different wavelength (color) of light. This dramatically increases the capacity of existing fiber infrastructure without laying new cables. There are two main variants.
CWDM (Coarse Wavelength-Division Multiplexing)
CWDM uses widely spaced wavelength channels (20 nm apart), supporting up to 18 channels over a single fiber. Because the channel spacing is wide, CWDM transceivers do not require temperature stabilization (no cooled lasers), making them significantly cheaper than DWDM optics.
CWDM is typically used for shorter distances (up to about 80 km without amplification) and lower channel counts. It is a good fit for metro and campus networks where the number of required wavelengths is modest and cost is a primary concern.
Limitations of CWDM include the inability to use optical amplifiers (EDFAs) across the full spectrum because some CWDM wavelengths fall outside the EDFA amplification window. This restricts CWDM to unamplified, shorter-reach applications.
DWDM (Dense Wavelength-Division Multiplexing)
DWDM uses tightly spaced wavelength channels (typically 0.8 nm or 0.4 nm apart on the ITU-T grid), supporting 40, 80, 96, or even more channels per fiber. DWDM requires temperature-stabilized (cooled) lasers to maintain precise wavelength alignment, which increases cost per channel compared to CWDM.
The key advantages of DWDM over traditional optical networks include inherent topology flexibility with built-in service protection, and the ability to expand bandwidth over existing optical infrastructure by simply adding more wavelengths. DWDM supports long-haul distances (hundreds or thousands of kilometers) using optical amplifiers (EDFAs) and can be combined with ROADM (Reconfigurable Optical Add-Drop Multiplexer) technology for dynamic wavelength provisioning without manual fiber patching.
DWDM is the standard choice for service provider backbone networks, long-haul transport, submarine cables, and large-scale data center interconnects where maximum fiber utilization and long reach are required.
CWDM vs. DWDM Design Considerations
Table 1 summarizes the key differences between CWDM and DWDM that network designers should evaluate when selecting an optical transport technology.
| Attribute | CWDM | DWDM |
|---|---|---|
| Channel spacing | 20 nm |
0.8 nm or 0.4 nm |
| Number of channels | Up to 18 | 40, 80, 96+ |
| Typical reach | Up to ~80 km (unamplified) | Hundreds to thousands of km |
| Amplification | Not supported across full spectrum | Supported (EDFA) |
| Cost | Lower (uncooled lasers) | Higher (cooled/temperature-stabilized lasers) |
| Use case | Metro, campus, short-haul | Backbone, long-haul, DCI |
When selecting between CWDM and DWDM, network designers should consider the required number of wavelengths, the distance between sites, whether optical amplification is needed, and the budget. In many designs, CWDM serves the access/metro layer while DWDM serves the core/long-haul layer.
IPoDWDM
IP over DWDM (IPoDWDM) is an architecture where routers or switches have integrated DWDM optics, allowing them to place their traffic directly onto a DWDM wavelength without needing a separate optical transport platform. This eliminates an entire layer of equipment (the traditional transponder shelf), reducing cost, power, space, and latency. IPoDWDM is increasingly common in modern SP and large enterprise core designs where routers connect directly to the optical layer.
Review Questions
1. Which two advantages of using DWDM over traditional optical networks are true? (Choose two.)
- Inherent topology flexibility and service protection provided without penalty through intelligent oversubscription of bandwidth reservation
- Inherent topology flexibility with a service protection provided through a direct integration with an upper layer protocol
- Inherent topology flexibility with built-in service protection
- Ability to expand bandwidth over existing optical infrastructure
- Inherent topology flexibility with intelligent chromatic dispersion
c and d. DWDM provides inherent topology flexibility through features like optical-layer protection switching and reconfigurable optical add-drop multiplexers (ROADMs), with built-in service protection at the optical layer. It also allows expanding bandwidth by adding more wavelengths to existing fiber infrastructure without laying new fiber. Option A is incorrect because DWDM provides dedicated wavelengths, not oversubscribed bandwidth. Option B is incorrect because DWDM is protocol-agnostic at Layer 1. Option E is incorrect because chromatic dispersion is a challenge to compensate for, not an advantage.
Layer 2 Media Access
Operators understand the importance of an infrastructure that offers flexible and responsive network architecture, supporting current and future requirements without requiring major changes or upgrades. In today’s market, almost all service providers have either partially or fully migrated their core infrastructure to be MPLS enabled. This allows them to offer the same services to their customers (Layer 2 over legacy and modern WAN protocols) over one unified MPLS core infrastructure. This is a significant incentive for providers already offering MPLS L3VPN, because it reduces capital expenditures and operational expenses where the same core infrastructure and control plane can transport both L2 and L3 services. Modern service providers that offer Layer 2 Ethernet services (for example, Metro Ethernet) are commonly referred to as Carrier Ethernet providers.
The primary business drivers for service providers to adopt modern Layer 2 services (such as L2VPN and Metro Ethernet) are:
- Flexible offerings: Customized solutions with a flexible mix of services, data rates, and revenue, such as Ethernet Private Line (EPL), Ethernet Virtual Private Line (EVPL), and Ethernet LAN (E-LAN)
- Flexible bandwidth: Providers can provision 1G or 10G Ethernet physical links and offer fractional line access rates based on the agreed SLA. This provides a seamless upgrade roadmap without changing any link or device (compared to legacy TDM-based WANs). An access port upgrade from 1G to 10G can be as simple as an SFP replacement.
- Flexible media access: A variety of access networks leveraging Ethernet, such as WiMAX, IP DSLAMs, Ethernet over Fiber, Ethernet over Copper, and more. Also supports interworking between legacy access media (ATM, Frame Relay) that might need to be retained for remote areas where Metro Ethernet access coverage is not available.
- Cost savings: Ethernet infrastructure offers lower CAPEX compared to legacy hardware such as ATM switches. Bandwidth flexibility offers optimized CAPEX and investment protection for both providers and customers.
- Scalable services that can increase revenue: Support a large number of customers over one common infrastructure.
This list covers the technologies and designs from a service provider point of view to provide the benefits to the enterprise network design if one of these technologies is chosen from a transport perspective. An enterprise could also deploy any of these services across its own networks, such as a self-deployed Metro Ethernet implementation to interconnect multiple campus locations over a Layer 3 WAN core network.
Figure 1 illustrates the different Layer 2 media access methods and topologies that can be transported over an MPLS-enabled backbone.
L2VPN Ethernet-Based Models
This section focuses on the L2VPN Ethernet-based models of modern service provider networks, as depicted in Figure 2.
Carrier Ethernet refers to modern service provider architectures that offer a cost-effective converged network infrastructure capable of delivering new and future services by providing scalable and flexible designs and technologies that meet different bandwidth and network requirements. Carrier Ethernet offers a wide range of modern services, including:
- Business services such as Metro Ethernet enterprise connectivity services (E-Line, E-LAN)
- Consumer and residential services (VoD, IPTV)
- Business services over DOCSIS
- Broadband mobility services
- Wholesale services
Figure 3 shows a high-level view of the Carrier Ethernet architecture.
The primary elements that comprise the Carrier Ethernet architecture are:
- Access: Provides access to residential and business customers over DSL, fiber, cable, or wireless
- Carrier Ethernet aggregation: Aggregates the access network across a Carrier Ethernet network and provides interconnectivity to the IP/MPLS edge and IP/MPLS core
- IPoDWDM optical network: Enables optical aggregation services with intelligent Ethernet multiplexing using MPLS/IP over dense wavelength-division multiplexing (IPoDWDM)
- Multiservice edge: Interfaces services with the IP/MPLS core; this is the provider edge for both residential and business subscriber services
- IP/MPLS core: Provides scalable IP/MPLS routing in the core network
Metro Ethernet Service Categories
Metro Ethernet services can be classified into two general categories:
- Point to point (P2P): A P2P Ethernet circuit is provisioned between two user network interfaces (UNIs)
- Multipoint to multipoint (MP2MP): A multipoint-to-multipoint Ethernet circuit is provisioned between two or more UNIs. If there are only two UNIs in the circuit, more UNIs can be provisioned to the same Ethernet virtual connection if required, which distinguishes this from the point-to-point type.
The Metro Ethernet Forum (MEF) defines these two categories as two main types of Layer 2 Ethernet services:
- Ethernet Line Service (E-Line): Point to point, such as Virtual Private Wire Service (VPWS)
- Ethernet LAN Service (E-LAN): Multipoint to multipoint, such as Virtual Private LAN Services (VPLS)
There is also a P2P Metro Ethernet service known as E-Tree, an abstraction of E-LAN, in which the spoke “leaves” can communicate with the hub or “root” location but not with each other. The typical application for E-Tree is in franchise operations. A useful comparison for E-Tree is that of private VLANs, as they share the same conceptual model.
Figure 4 shows the three Metro Ethernet models.
Metro Ethernet Service Attributes
Metro Ethernet services can be created by assigning values to a set of attributes grouped according to the following:
- User Network Interface (UNI): Represents a physical demarcation point between the connection and the subscriber. Also known as port-based ME.
- Ethernet Virtual Connection (EVC): Represents the association of two or more UNIs, which limits the exchange of service frames to UNIs within the EVC in Ethernet. When multiple EVCs exist per single UNI and each EVC is distinguished by 802.1Q VLAN tag identification, this is known as VLAN-based (EVPL).
The following table summarizes the relationship between the transport model and the Metro Ethernet service definitions.
| Service Type | Port Based | VLAN Based |
|---|---|---|
| E-Line | Ethernet Private Line (EPL) | Ethernet Virtual Private Line (EVPL) |
| E-LAN | Ethernet Private LAN (EPLAN) | Ethernet Virtual Private LAN (EVPLAN) |
In addition to E-Line and E-LAN services, Layer 2 services are available to facilitate carrying legacy WAN transport over MPLS networks:
- Frame Relay over MPLS (FRoMPLS)
- ATM over MPLS (ATMoMPLS)
Cisco’s implementation of VPWS is known as Any Transport over MPLS (AToM) and delivers what is known as Ethernet over MPLS (EoMPLS). L2TPv3 can be used as an analogous service to AToM over any IP transport. Cisco’s AToM includes more services (PPP, HDLC, FR, and ATM) as its “any transport” than EoMPLS. EoMPLS and Metro Ethernet services cannot provide this same level of flexibility from a connectivity and mixed circuits perspective.
Review Questions
2. Which of the following options is the correct port-based Metro Ethernet transport mode for an E-Line service type?
- Ethernet private line
- Ethernet virtual private line
- Ethernet private LAN
- Ethernet virtual private LAN
a. The correct port-based Metro Ethernet transport mode for an E-Line service is Ethernet private line (EPL).
3. Which of the following options is the correct VLAN-based Metro Ethernet transport mode for an E-LAN service type?
- Ethernet private line
- Ethernet virtual private line
- Ethernet private LAN
- Ethernet virtual private LAN
d. The correct VLAN-based Metro Ethernet transport mode for an E-LAN service is Ethernet virtual private LAN (EVPLAN).
4. Which of the following options is the correct VLAN-based Metro Ethernet transport mode for an E-Line service type?
- Ethernet private line
- Ethernet virtual private line
- Ethernet private LAN
- Ethernet virtual private LAN
b. The correct VLAN-based Metro Ethernet transport mode for an E-Line service is Ethernet virtual private line (EVPL).
5. Which of the following options is the correct port-based Metro Ethernet transport mode for an E-LAN service type?
- Ethernet private line
- Ethernet virtual private line
- Ethernet private LAN
- Ethernet virtual private LAN
c. The correct port-based Metro Ethernet transport mode for an E-LAN service is Ethernet private LAN (EPLAN).
Virtual Private Wire Service Design Considerations
VPWS is based on the concept of pseudowire (PW) described in RFC 3916, which can be defined as a connection between two edge nodes connecting two attachment circuits (ACs) to emulate a direct AC connection of a packet-switched network (PSN). Typical implementations of PWs are carried either over native IP (such as L2TP) or over an MPLS network (such as EoMPLS). Within an MPLS-enabled network, the core visibility is limited to the transport MPLS labels (LSP) that each PW uses to communicate with the remote end, while the remote Provider Edge (PE) relies on the virtual circuit (VC) label to identify the intended AC per pseudowire (PW-to-AC mapping), as shown in Figure 5.
The primary goal of service providers is to offer services that satisfy customer requirements; therefore, their core public services network (PSN) is the main business enabler. Pseudowire over PSN gives the service provider flexibility to handle customer connectivity using primary transport connectivity models, each of which can serve a different set of requirements:
- VLAN-based model: A VLAN-based transport model where VLANs defined on the physical port can be switched to different remote PEs (Figure 6). This is applicable in scenarios where enterprise customers seek a service analogous to their current WAN service (traditional Frame Relay or ATM), in which no routing and IP addressing changes are required.
- Port-based model: This transport model tunnels all traffic entering the AC physical port and transports it to the targeted remote PE without any change (Figure 7). This model is applicable in scenarios where the Layer 3 customer network uses traditional WAN serial links that need to be replaced with higher-speed P2P links. Another usage is when a customer needs to interconnect two LAN islands over the service provider’s Layer 2 P2P PW (whether native IP core or MPLS enabled). This model is also commonly used to provide Layer 2 data center interconnection to support extending multiple Layer 2 VLANs over the same link.
In some situations, the user access side may not always be provisioned as Ethernet. Customers with legacy WAN media access such as ATM and Frame Relay might be provisioned with L2VPN services. With VPWS (L2VPN MPLS based), carriers can maintain customer access as-is and offer the same service capability. Service providers can even maintain connectivity between sites that have different media access methods, such as Frame Relay, ATM, and standard Ethernet.
The following table compares the primary L2 WAN media access technologies along with their transport modes over PW.
| L2 Technology | Transport Mode over PW |
|---|---|
| Frame Relay | Port based; Per Data Link Connection Identifier (DLCI) |
| ATM | AAL5 protocol data units (PDUs) over PW; Cell relay over PW |
| Ethernet | Protocol based; VLAN based |
Review Questions
6. Which of the following options is the correct transport mode over pseudowire for a Frame Relay access connection?
- AAL5 protocol data units over pseudowire cell relay over pseudowire
- Protocol-based per VLAN
- Port-based per DLCI
c. The correct transport mode over pseudowire (PW) for a Frame Relay access connection is port-based per DLCI.
7. Which of the following options is the correct transport mode over pseudowire for an ATM access connection?
- AAL5 protocol data units over pseudowire cell relay over pseudowire
- Protocol-based per VLAN
- Port-based per DLCI
a. The correct transport mode over PW for an ATM access connection is AAL5 protocol data units over PW cell relay over PW.
8. Which of the following options is the correct transport mode over pseudowire for an Ethernet access connection?
- AAL5 protocol data units over pseudowire cell relay over pseudowire
- Protocol-based per VLAN
- Port-based per DLCI
b. The correct transport mode over PW for an Ethernet access connection is Protocol-based per VLAN.
Virtual Private LAN Service Design Considerations
In today’s market there is a significantly increasing demand for interconnecting multiple sites over one common Layer 2 network. Along with that, there is an increasing demand to extend multiple modern data centers over Layer 2 transport. This supports the requirements of next-generation data centers and the mobility of applications, such as virtual machine mobility and distributed workload, which supports the overall business continuity plan of the customer’s business. This architecture is known as Virtual Private LAN Service (VPLS), or in MEF terms, E-LAN.
With the VPLS architecture, end users see their network nodes as interconnected directly over a shared Ethernet LAN segment; this shared segment is an emulated LAN created by the VPLS domain (emulated switched network). In addition, it supports legacy media access methods that support Ethernet encapsulation, such as Frame Relay.
As with L3VPN, in a typical VPLS architecture service providers normally only need to update the PE (or PEs) to which the customer connects within the same VPLS domain (equivalent to adding a new L3VPN customer to an existing VPN). In contrast, with P2P Layer 2 VPN solutions, the service provider must reconfigure each of the peering PEs every time a change is required (a typical scalability limitation of a P2P architecture). This optimized Layer 2 VPN architecture, along with its simplified operation compared to legacy Layer 2 WAN services, is a key business driver that attracts many Carrier Ethernet providers. As a result, VPLS is one of the most common, reliable, and mature Layer 2 WAN services in today’s market. Figure 8 illustrates the VPLS conceptual architecture.
Although this section always refers to the network as a service provider network, all the concepts are applicable, to a certain extent, to large-scale enterprise networks that have self-deployed Layer 2 VPN services, such as VPLS.
VPLS Architecture Building Blocks
VPLS brings the standard IEEE 802.1 MAC address learning, flooding, and forwarding concept over the MPLS-based packet-switched network extended by pseudowires (PWs), which commonly interconnect different customer LANs. In other words, VPLS is the glue that interconnects multiple LANs across one common packet-switched network over MPLS.
VPLS Functional Components
Figure 9 illustrates the common functional components that construct a typical VPLS architecture. These components may vary slightly based on the VPLS model used:
- Network-facing Provider Edge (N-PE): Provides VPLS termination and L3 services
- User-facing Provider Edge (U-PE): Provides customer UNI
- Customer Edge (CE): The customer device
Virtual Switching Instance
The virtual switching instance (VSI) performs standard LAN (Ethernet) bridging functions, including participation in the learning and forwarding process based on MAC addresses and VLAN tags. Each VSI defined at each provider edge node usually handles the forwarding decisions of a single VPLS instance.
VPLS Control Plane
The concept of the VPLS control plane is largely the same as VPWS. However, the primary difference is the need for full-mesh PWs among all participating PEs in any given VPLS instance. With VPWS, the typical scenario is P2P PWs between two sites or a few sites in a full-mesh or hub-and-spoke connectivity model. Consequently, with VPLS, the automation of PE discovery and PW setup among the relevant PEs are important factors to consider. Otherwise, the design will encounter increased operational complexity as the network grows with many customers (VPLS instances) and remote sites (PEs per VPLS instance).
VPLS can be deployed by enterprises across their backbone to form an overlaid L2VPN connectivity (self-deployed VPLS). In such an environment, the number of PEs and VPLS instances can be limited compared to large-scale networks. Therefore, using a manual mechanism to set up the PWs, such as Label Distribution Protocol (LDP), can be a viable solution. LDP is widely deployed as a control plane for L2VPN, even though it may not be the optimal scalable choice. Upgrading this control plane may not be an option for some operators due to constraints such as inability to afford service interruption to existing VPLS customers or existing PE nodes not supporting BGP as the L2VPN control plane protocol. In these scenarios, network designers are forced to accommodate these design constraints in their decisions.
Designing a typical VPLS with automated PE discovery and signaling can offer a more scalable and operationally simple solution. This can be achieved by:
- BGP as L2VPN control plane for signaling and auto-discovery (RFC 4761)
- Hybrid approach: An optimized version that retains LDP-based signaling while automating the discovery of participating PEs in each VPLS instance using BGP (RFC 6074)
VPLS Design Models
One of the main drivers in selecting a VPLS design model is the level of scalability possible. The primary factors that influence VPLS solution scalability are the supported number of VLANs (customers or end-user networks) and the number of PWs established between PE nodes. Network designers should consider the following questions during the planning phase:
- What is the goal of the solution (enterprise grade to interconnect multiple data centers, or carrier Ethernet service provider grade)?
- Number of customers?
- Number of participating PEs?
- Scale of the interconnected sites in terms of the number of MAC addresses? (Hosting cloud providers usually require support for an extremely large number of MACs.)
- PW termination between N-PEs versus between U-PEs along with the VSI?
- Any platform limitations, such as supporting VPLS VSI configuration or hardware resource capacity?
- Is traffic load-balancing/sharing for multihomed sites required (active/active versus active/standby)?
- What is the targeted resiliency level? And how can it be achieved (for example, MPLS-TE FRR, redundant PWs)?
Flat VPLS Design Model
The flat VPLS design model is the classic and simplest VPLS design model, with the fewest functional components, as shown in Figure 10.
With this design model, each PE hosts the VSIs of VPLS domains along with a full mesh of PWs among the participating PEs per VPLS domain. Each PE maintains a P2MP perspective of all other PEs per VPLS domain, and each PE is seen as the root bridge of the PW mesh to other PEs (Figure 11).
When PEs maintain a full mesh of PW connectivity, it eliminates the need for Spanning Tree Protocol (STP) across the PSN MPLS network. However, from the customer point of view, their STP (BPDUs) can be transparently carried over the emulated VPLS LAN between sites without impacting the service provider network. The split-horizon concept is used to block any potential loop of traffic received over one core/PE PW into another core/PE PW.
The characteristics of the flat VPLS design model:
- Limited scalability: The more PEs and VSIs, the more network hardware resources consumed per PE, with associated operational complexity:
- Many PWs due to the full mesh of directed LDP sessions required (\(N \times (N - 1) / 2\) PWs)
- Potential signaling and packet replication overhead when the number of PWs increases across multiple PEs using LDP as the control plane protocol
- Large amount of multicast replication, which may result in inefficient bandwidth utilization (unless mitigated by mechanisms such as IGMP snooping with VPLS)
- CPU overhead for replication
- MAC distribution across the network limitations
- Support limited number of customers/VLANs (maximum 4096)
- VLAN and port-level support (no QinQ)
- Support multihomed CEs in active-standby manner
- Suitable for simple and small deployments, such as a self-deployed enterprise VPLS solution
Hierarchical VPLS Design Model
The hierarchical VPLS (H-VPLS) design model aims to optimize the high complexity of PW meshes and the limited scalability of the flat VPLS model by introducing a hierarchical structure. This structure consists of hub-and-spoke and full-mesh networks, as shown in Figure 12.
The hub or core of fully meshed PWs in each PE node forms a multipoint-to-multipoint forwarding relationship with all other PE nodes at the same level within the VPLS domain. At the hub-and-spoke level, each PE node needs a single PW to the hub PE node and can operate in a non-split-horizon mode. This allows inter-VC connectivity between PEs at the same level connected to the same hub PE node. In other words, the hub PE nodes perform PW aggregation of the edge node PWs.
This hierarchical model offers significant reductions to the number of mesh PWs and overcomes the scalability and performance efficiency limitations of the flat VPLS model to a great extent (Figure 13).
H-VPLS introduces additional terms for its functional components:
- MTU-s: Multitenant unit switch capable of bridging (U-PE)
- PE-r: Nonbridging PE router
- PE-rs: Bridging- and routing-capable PE
Provider Backbone Bridging with H-VPLS
The H-VPLS model offers an optimized version of flat VPLS in terms of scalability and performance efficiency. However, there are still scalability and performance limitations for Carrier Ethernet providers serving a large number of customers with large-scale networks. This is especially true for large-scale modern data centers (virtualized and cloud-based) with Layer 2 interconnect, where the service provider may need to carry millions of MAC addresses across their backbone. The following limitations become constraints:
- In both flat and hierarchical VPLS models, each PE that performs MAC switching must learn customer MAC addresses, which often leads to performance and scalability deficiencies.
- With the H-VPLS Ethernet access model, each access network is limited to about 4094 customer instances. With the MPLS access model, the larger the number of customers, the larger the number of PWs and VSIs to be handled by the N-PE (limited performance and scalability).
The IEEE standard Provider Backbone Bridging (PBB) (802.1ah-2008) was developed to provide an optimized approach that can scale to a very large number of customer instances and MAC addresses (millions) without compromising network performance efficiency. When combined with VPLS/H-VPLS, this is known as PBB-VPLS (Provider Backbone Bridging with Virtual Private LAN Service).
PBB Overview
PBB defines an architecture based on a MAC tunneling mechanism that enables service providers to build a large-scale Ethernet bridged network. The primary concept is hierarchical MAC address learning and forwarding, similar to the MPLS label stacking concept. Instead of labels, PBB uses multitier MAC address learning and forwarding in which the core PEs, also known as backbone core bridges (BCBs), communicate based on backbone MAC addresses of the core components. Customer MAC frames (C-MAC) are encapsulated (tunneled) at the edge of the bridged network into the backbone MAC, enabling Carrier Ethernet to scale a large number of C-MACs without impacting core components, because communication (including BUM: broadcast, unknown destination address, multicast flooding) is based only on backbone MACs.
With PBB architecture, besides MAC tunneling, the network provides VLAN tunneling at two stages:
- At the edge of the bridged network, where customer VLANs (C-VLAN) are tunneled using the 802.1ad IEEE standard (QinQ), referred to as the service provider tag or S-VLAN
- Where the S-VLANs are mapped into the PBB backbone VLAN (B-tag or B-VLAN)
Across the PBB bridged network, end-to-end mapping and identification of each customer instance is based on the service instance ID (I-SID), which must be unique across the entire bridged network and is usually performed at the edge of the network.
The flexibility of PBB offers multiple options to network designers for mapping between customer VLANs and the service interface, which is more flexible compared to the typical L2VPN VLAN and port-based forwarding modes.
PBB with this hierarchical architecture can offer:
- Up to \(2^{24}\) service instances per bridge domain by defining a 24-bit service identification field (I-SID), also known as MAC-in-MAC
- MAC address hiding (tunneling) from the SP core, significantly increasing solution scalability by confining customer MAC address learning to the edge and mapping customer MAC addresses to backbone MAC addresses on the backbone edge bridges (BEB). All the intelligence resides on the BEBs, as they are responsible for translating frames to/from the PBB format.
EVPN Design Model
Ethernet VPN (EVPN) (RFC 7432) is a next-generation Ethernet L2VPN technology that uses the same principle as MPLS L3VPN, where the BGP control plane handles Ethernet segment and MAC address signaling and learning over the service provider MPLS core. In addition to access topology and VPN endpoint discovery, EVPN eliminates the need to use PWs, thereby avoiding their operational complexities and scalability limitations. The introduction of EVPN marks a significant milestone for the industry because it aligns the well-understood technical and operational principles of IP VPNs to Ethernet services. Operators can now leverage their experience and the scalability characteristics inherent in IP VPNs for their Ethernet offerings. EVPN supports various L2VPN Ethernet over MPLS topologies, including Ethernet multipoint (E-LAN), Ethernet P2P (E-Line), and Ethernet rooted-multipoint (E-Tree).
Business Drivers
One of the main drivers toward EVPN is the increased demand on distributed virtualized and cloud-based data centers, which commonly require scalable and reliable stretched Layer 2 connectivity among them. Data Center Interconnects (DCI) has become a leading application for Ethernet multipoint L2VPNs. Virtual machine (VM) mobility, storage clustering, and other data center services require nodes and servers in the same Layer 2 network to be extended across data centers over the WAN. These trends add new requirements for L2VPN operators:
- Flow-based load balancing (PE based and multipathing) across the PSN
- Fast convergence (avoid C-MAC flushing)
- Support of large-scale, virtualized data centers
With increased demand and expectations from enterprise customers and cloud hosting providers, operators also need a solution that offers:
- Flexible forwarding policies and topologies
- A scalable solution supporting a very large number of customers and MAC addresses
- Operational simplicity
EVPN Business Strengths
Based on market trends and new requirements of modern enterprise customers, EVPN offers the following advantages for Carrier Ethernet:
- Business continuity: Ability to meet very strict customer SLA requirements by offering all-active (per-flow) access load balancing and fast convergence (at link, node, and MAC move levels), in addition to optimizing customer ROI of their multiple links (multihomed)
- Scalable design: Avoids PW limitations and supports extremely large numbers of customers and MAC addresses
- Simplified operations: Uses the same L3VPN address learning and forwarding principle where the same control plane (MP-BGP) is used for both L2 and L3 services. Auto-discovery of PEs simplifies adding new nodes and eliminates PW complexities.
- Topology flexibility: Supports primary Ethernet services topologies (E-LAN, E-Line, E-Tree, and VLAN-aware bundling)
- Seamless interworking: Between TRILL, 802.1Qaq, and 802.1Qbp
- Investment protection: Open standard technology supported by multiple vendors (RFC 7432)
From the enterprise (customer) point of view, the greatest advantage of EVPN is a flexible and reliable L2 WAN/MAN service that meets new requirements, such as large-scale next-generation data center interconnects (virtualized and cloud based).
With a solution like EVPN and VXLAN, a network design can allow for Layer 2 mobility across disparate locations (such as multiple data centers) while limiting the inherent drawbacks of legacy Layer 2 protocols such as STP and First Hop Redundancy Protocol (FHRP). The network design can leverage the Layer 3 architecture to provide a scalable Layer 2 overlay that mitigates fate sharing for the traditional stretching of Layer 2 in a dual data center design.
Provider Backbone Bridging with Ethernet VPN
By combining the MAC tunneling and hiding principle of PBB (802.1ah) with the MP-BGP-based MAC learning and PE discovery of EVPN, network designers can achieve the optimal L2VPN architecture supporting an extremely large number of customers and MACs. At the same time, it reduces control plane overhead and scales to a larger extent than EVPN alone, while offering faster convergence time. Combining PBB with EVPN (PBB-EVPN) can achieve a superior solution for modern DCIs (virtualized and cloud based) and next-generation E-LAN offerings. The drawbacks of MAC-in-MAC tunneling are the increase in encapsulation requirements, the substantial increase in configuration needed, and the increased design complexity.
Both LDP and BGP are valid and proven choices. The selection of one over the other should be based on the target environment (VPWS versus VPLS) and the scale of the network (SP versus enterprise), considering design constraints. For instance, if the SP is offering both MPLS L2 and L3VPN services with many clients requiring full mesh or several sites connected using VPLS or VPWS, the use of BGP for L2VPN signaling with auto-discovery can be considered a scalable solution, especially if the same PEs are involved in both Layer 3 VPN and Layer 2 VPN services. The same BGP sessions can be used for both L3VPN NLRI and L2VPN NLRI, and the same BGP architecture (for example, BGP RRs) can be leveraged. This simplifies design and operations by eliminating the need to maintain multiple separate protocols (BGP and LDP) for the control plane.
EVPN Architecture Components
The architecture of EVPN has three primary foundational building blocks:
- EVPN instance (EVI)
- Ethernet segment (ES)
- EVPN BGP routes and extended communities
EVPN Instance
An EVI represents an L2VPN instance on a PE node. Similar to the VRF in MPLS L3VPN, import and export Route Targets (RTs) are allocated to each EVI. A bridge domain (BD) is associated with each EVI. Mapping traffic to the bridge domain is dependent on the multiplexing behavior of the UNI. Any given EVI can include one or more BDs based on the PE’s service interface deployment type, as summarized in Figure 14.
The four EVI models are:
- Port-based: All VLANs mapped to a single BD
- VLAN-based: VLAN to BD 1:1 mapping
- VLAN bundling: A group of VLANs can be mapped to a single BD/EVI selectively. MACs must be unique per VLAN per EVI.
- VLAN-aware bundling: Single EVI with multiple BDs to maintain end-to-end forwarding plane separation per VLAN. Supports MAC overlapping per VLAN/BD.
The VLAN bundling model is suited for environments that require multiple VLANs to be carried transparently across the EVPN cloud between two or more sites. The VLAN-aware bundling model is more feasible for multitenant data center environments where multiple VLANs must be carried over the DCI over a single EVI with multiple BDs (VLAN to BD 1:1 mapping), because overlapping of tenant MAC addresses across different VLANs is supported.
Ethernet Segment
Ethernet segment (ES) refers to a site connected to one or more PEs. An ES can be either a single CE or an entire network. Each network segment is assigned a unique Ethernet segment identifier (ESI), which can eliminate any STP-type protocol used for loop prevention and normally adds limitations to the design, especially for multihomed CE scenarios. Based on this identifier, EVPN provides access redundancy offering georedundancy and multihoming, where a site (CE or entire network with multiple CEs) can be attached to one or multiple PEs using multiple combinations of CE-PE connectivity.
Figure 15 illustrates the various EVPN-supported access connectivity models:
- Single-homed device (SHD)
- Multihomed device (MHD) with Multi-Chassis Link Aggregation (mLAG)
- Single-homed network (SHN)
- Multihomed network (MHN)
In EVPN, the PE advertises in BGP a split-horizon label (ESI MPLS label) associated with each multihomed ES to prevent flooded traffic from echoing back to a multihomed Ethernet segment.
EVPN BGP Routes and Extended Communities
Because both EVPN and PBB-EVPN PEs signal and learn MAC addresses over the core via BGP, a new MP-BGP address family and BGP extended communities were created to allow PE routers to advertise and learn prefixes that identify MAC addresses and Ethernet segments over the network. This control plane learning significantly enhances E-LAN capability over EVPN architecture to address the VPLS (data plane learning) shortcomings discussed earlier, such as supporting multihoming with per-flow load balancing.
The sequence number BGP extended community attribute offers optimized MAC mobility control plane learning and forwarding, because it facilitates and enhances the triggering of the advertising PE to withdraw its MAC advertisement in case of a MAC move (mobility) to a new network with a different ESI. This provides a reliable solution for customers of a DCI with a stretched Ethernet segment.
By using BGP as a common VPN control plane, providers can leverage their operational experience and the scalability characteristics inherent to IP VPNs for their Ethernet offerings.
EVPN Load-Balancing Models
EVPN and PBB-EVPN support two primary load-balancing models for multihomed devices (CEs):
- All-active load balancing (per-flow LB): Supports multihomed devices with per-flow load balancing. Access devices are connected over a single mLAG to multiple PEs, and traffic of the same VLAN can be sent and received from all PEs by the same Ethernet segment (Figure 16).
- Single-active load balancing (per-service LB): Supports multihomed devices with a per-VLAN load-balancing model. Access devices connect over separate Ethernet bundles to multiple PEs. PE routers automatically perform service carving to divide VLAN forwarding responsibilities across the PEs in the Ethernet segment. The access device learns, via the data plane, which Ethernet bundle to use for a given VLAN. This is unlike the traditional VPLS model, where manual administration is required to compensate for the lack of access auto-discovery and automatic service carving mechanisms (Figure 17).
RFC 6391 describes a mechanism that introduces a flow label that allows P routers to distribute flows within a PW.
VXLAN Design Model
The IETF standard VXLAN was created and commonly used as a data center fabric overlay protocol to facilitate and optimize modern data center performance and agility in response to virtualized and cloud-based data center requirements. In VXLAN, VMs can be moved anywhere within the DC because of abstraction between the location of the VM/application within the DC and the physical access/leaf switch and port (VM mobility). VXLAN adds simplicity, transparency to applications, and flexibility to modern Clos fabric architectures; in addition, it enhances the ability to scale an extremely large number of VMs/MACs, especially when an MP-BGP EVPN control plane is used for VXLAN.
BGP as a control plane, together with VXLAN headend replication, can optimize VXLAN behavior with regard to multi-destination traffic for MAC learning (broadcast and unknown unicast). The behavior becomes like Layer 3 routing (MAC learning is based on MAC address advertisement). Flooding is eliminated unless there is a host that has not advertised or learned MAC, and only in this case will typical Layer 2 flooding be used.
VXLAN is also considered a DCI solution, especially the unicast mode, because it can be more controllable at the DCI edge (can be policed more deterministically than multicast mode). This is applicable to any of the VXLAN design models (host based or network based), as shown in Figure 18.
The placement of the VXLAN controller functionality with BGP (EVPN) as a control plane can vary from vendor to vendor and solution to solution. For example, it can be placed at the spine nodes of the data center architecture, at a virtualized (software-based) controller, or used as a BGP route reflector to exchange VTEP list information between multiple VSMs.
Technically, you can use all VXLAN models (host and network based) at the same time. However, from a design point of view, this adds design and operational complexity. It is always recommended to keep it simple and start with one model that can achieve the desired goal.
One of the challenges of using VXLAN as a DCI solution is the behavior of the distributed controller across two DCs following a DCI failure event (split-brain) and after DCI link recovery. Considerations about the VXLAN gateway location (localized versus remote) can also impact the design.
The scope of each DC VXLAN can be contained within each DC locally, while the terminated VXLAN VTEPs at the DC/SP border nodes can be extended across the SP network that has EVPN enabled. Each VXLAN VNI is mapped to a unique EVPN instance (VNI-to-EVI 1:1 mapping) to maintain end-to-end path separation per VXLAN. This design offers a scalable, integrated DCI solution that takes advantage of both solutions in scenarios where EVPN is already in place as a DCI and the DC operator is deploying VXLAN within each DC (Figure 19).
Layer 2 Access Solution Comparison
The following table provides a summarized comparison of the different VPLS design models discussed in this chapter.
| VPLS | H-VPLS | PBB-VPLS | EVPN | PBB-EVPN | |
|---|---|---|---|---|---|
| Scalability | Very limited | Limited | Scalable | Highly scalable | Highly scalable |
| Control plane protocol | BGP/LDP | BGP/LDP | BGP/LDP | BGP | BGP |
| Network core control plane complexity | Very high | High | Moderate | High | Low |
| Flow-based load balancing (CE-PE) | No | No | No | Yes | Yes |
| Flow-based multipathing in the core | Yes | Yes | Yes | Yes | Yes |
| Operational complexity | High* | High* | High | Low | Moderate |
| Service interface VLAN-aware bundling | No | No | No | Yes | Yes |
| Loop prevention with multihomed CEs | STP | STP | STP | Split-horizon label (ESI MPLS label) | Split-horizon label (ESI MPLS label) |
| Fast convergence with local repair | No | No | No | Yes | Yes |
| MAC mobility | Yes | Yes | Yes | Yes | Yes |
| Optimized control plane learning of MAC mobility | No | No | No | Yes, by sequence number attribute | Yes, by sequence number attribute |
| Targeted DCI solution | Small (e.g., enterprise controlled) | Medium | Large | Very large | Very large |
* If BGP is used as the control plane for VPLS, operational complexity will be reduced.
What determines small, medium, and large DCI solutions is the number of interconnected sites per customer, scale of the VMs/MACs, and the number of customers; therefore, the suggestion here can be considered generic and not absolute.
Review Questions
9. Which Layer 2 transport options are the best options for a very large enterprise Data Center Interconnect (DCI) solution where there are many data centers with many interconnection points with only a few thousand MAC addresses in total? (Choose two.)
- VPLS
- EVPN
- H-VPLS
- PBB-EVPN
- PBB-VPLS
d and e. The Layer 2 transport options that are the best options for a very large enterprise Data Center Interconnect (DCI) solution are Provider Backbone Bridging with Ethernet VPN (PBB-EVPN) and Provider Backbone Bridging with Virtual Private LAN Service (PBB-VPLS). EVPN can support up to a large enterprise scenario, H-VPLS can support up to a medium enterprise scenario, and VPLS can only support up to a small scenario.
10. Which Layer 2 transport options are best options for an optimized MAC mobility requirement? (Choose two.)
- VPLS
- EVPN
- H-VPLS
- PBB-EVPN
- PBB-VPLS
b and d. The Layer 2 transport options that are best for MAC mobility are EVPN and PBB-EVPN because of the sequence number attribute being leveraged.
Summary
This chapter covered the various transport technologies design models, protocols, and approaches, along with the characteristics of each. All these design options and protocols are technically valid and proven solutions still in use by many operators today. However, as a network designer, you must evaluate the scenario using the top-down design approach, where business goals and requirements are at the top, followed by the application requirements that should collectively drive the functional and technical requirements.
For instance, if an enterprise needs a basic self-deployed Layer 2 DCI solution between three distributed data centers with a future plan to add a fourth data center within two years, a flat VPLS solution can be cost-effective and simple to deploy and manage, in addition to being scalable enough for this particular scenario. By contrast, if a service provider is already running flat VPLS and experiencing high operational complexity and scalability challenges and is interested in a solution that supports future expansion plans while minimizing current operational complexity, then H-VPLS with BGP signaling, H-VPLS with PBB, EVPN, or EVPN PBB are possible solutions. More detail gathering would be required to narrow down the selection. For example, is this operator offering MPLS L3VPN? Does it plan to offer multihoming to L2VPN customers with active-active forwarding? If the answer to any of these questions is yes, EVPN (with or without PBB) can be an optimal solution, because the same control plane (BGP) can be used for both MPLS VPN services, simplifying operational complexity and offering a more scalable solution.
Adding PBB to this solution will optimize scalability to a large extent, especially if the operator provides L2VPN connectivity to cloud-based data centers where a large number of virtual machine MAC addresses is expected. If multihoming with active-active forwarding is required, EVPN will be a business enabler along with optimized scalability and simplified operation compared to existing flat VPLS. However, design constraints may exist. For example, if current network nodes do not support EVPN and the business is not allocating budget for hardware or software upgrades, or if the provider has an existing interprovider L2VPN link with a global Carrier Ethernet that does not support EVPN.
In both situations, the network designer is forced to look into other suitable design alternatives such as H-VPLS. The design requirements and constraints must drive the decision for which solution is suitable or optimal by looking at the bigger picture and not focusing only on the technical characteristics of the design option or protocol.
Previous: Reference Architecture Models and Frameworks | Next: Layer 2 Technologies