Scalable Enterprise Campus Architecture Design
Overview
A campus network is the portion of the enterprise network infrastructure that provides access to network communication services and resources to end users and devices spread over a single geographic location, whether a single building or a group of buildings. Enterprises typically own the physical wires deployed in the campus and design it to be optimized for the fastest functional architecture running on high-speed physical infrastructure (1/10/40/100 Gbps). Modern converged enterprise campus networks should leverage three core architectural principles:
- Hierarchy
- Modularity
- Resiliency
This chapter covers three main topics:
- Campus Hierarchical Design Models: Campus hierarchical models and corresponding network design considerations
- Campus Layer 3 Routing Design Considerations: Campus Layer 3 routing and corresponding network design considerations
- Campus Network Virtualization Design Considerations: Campus network virtualization and corresponding network design considerations
Campus Hierarchical Design Models
The hierarchical network design model breaks a complex flat network into multiple smaller and more manageable networks, with each tier focused on a specific set of roles. This approach gives network designers flexibility to select the right hardware, software, and features for each layer.
A typical hierarchical enterprise campus network includes three layers:
- Core layer: Provides optimal transport between sites and high-performance routing. Due to its criticality, the core must provide an appropriate level of resilience to recover quickly after any failure.
- Distribution layer: Provides policy-based connectivity and boundary control between the access and core layers.
- Access layer: Provides workgroup/user access to the network.
The two primary hierarchical design architectures are the three-tier and two-tier models.
Three-Tier Model
The three-tier model is typically used in large enterprise campus networks constructed of multiple functional distribution layer blocks (Figure 1).
Two-Tier Model
The two-tier model is more suitable for small to medium-size campus networks (ideally not more than three functional distribution blocks), where the core and distribution functions are combined into one layer. This is also known as collapsed core-distribution architecture (Figure 2).
The term functional distribution block refers to any block in the campus network that has its own distribution layer, such as a user access block, WAN block, or data center block.
Campus Modularity
Applying the hierarchical design model across multiple functional blocks of the enterprise campus network achieves a scalable and modular campus architecture (commonly referred to as building blocks). This modular approach promotes fault domain isolation and more deterministic traffic patterns, allowing network changes and upgrades to be performed in a controlled and staged manner (Figure 3).
Within each functional block of the modular enterprise architecture, the same hierarchical network design principle should be applied to achieve the optimal structured design.
When Is the Core Block Required?
A separate core provides the capability to scale the campus network in a structured fashion that minimizes overall complexity as the number of distribution blocks and interconnections grows significantly. Without a core, this growth leads to physical and control plane complexities (Figure 4).
One of the primary influencing factors for selecting two-tier versus three-tier architecture is the type of site:
- A typical small to medium-size remote site rarely requires a three-tier architecture even with future growth
- A regional HQ or secondary campus has high potential to grow significantly, making a core layer a feasible option
The actual answer must always align with business goals and plans, including projected organic growth, merger and acquisition plans, and design constraints such as cost. If the business priority is to reduce CAPEX, a collapsed two-tier architecture may be the design constraint even if it is not technically optimal. However, network designers should highlight to IT leaders the long-term OPEX cost of a network not designed for projected growth, as the additional CAPEX for a three-tier architecture may be justified in the long run.
Following the principle “build today with tomorrow in mind” can lead to gold-plating. For the CCDE exam, candidates should solve the problem at hand and no more, unless the scenario states otherwise.
Access-Distribution Design Model
The main difference between access-distribution design models is where the Layer 2/Layer 3 boundary is placed and how Layer 3 gateway services are handled. The three primary models are compared in Table 1:
- Multitier STP-based: The classical model where access layer switches operate in Layer 2 only and distribution switches operate in both Layer 2 and Layer 3. Primary limitation is reliance on STP and FHRP.
- Routed access: Access layer switches act as Layer 3 routing nodes. The Layer 2/Layer 3 demarcation moves from the distribution layer to the access layer, replacing Layer 2 trunk links with Layer 3 point-to-point routed links.
- Switch clustering: Introduces switch clustering across functional modules, simplifying the design while providing higher node and path resiliency and significantly optimized convergence time. The cleanest way to extend Layer 2 without adding STP or FHRP (see Figure 5).
The routed access model offers several advantages over the classical STP-based model:
- Simpler to troubleshoot using standard routing techniques with fewer protocols to manage
- Eliminates reliance on STP and FHRP; relies on ECMP to utilize all available uplinks
- Minimizes convergence time during link or node failure
The routed access model does not support spanning Layer 2 VLANs across multiple access switches. Extending Layer 2 over routed infrastructure is achievable using overlay technologies but may add complexity or require features not supported on existing platforms.
| Multitier STP-Based | Routed Access | Switch Clustering | |
|---|---|---|---|
| Design flexibility | Limited (topology dependent) | Limited (spanning L2 across access switches requires overlay) | Flexible |
| Scalability | Scale up, limited scale out | Scale up and scale out | Scale up, limited scale out (typically two distribution switches per cluster) |
| Layer 3 gateway | Distribution layer (FHRP-based) | Access layer (Layer 3 routing) | Distribution layer (may or may not require FHRP*) |
| mLAG | Not supported | Not supported (relies on L3 ECMP) | Supported |
| Convergence time | Dependent on STP and FHRP timers (relatively slow) | IGP dependent, commonly fast | Fast |
| Operational complexity | Complex (STP, FHRP) | Moderate (advanced routing expertise may be required) | Simple |
* Some switch clustering technologies such as Cisco Nexus vPC use FHRP (HSRP). However, from a forwarding plane point of view, both upstream switches (vPC peers) do forward traffic, unlike the classical active-standby behavior.
All design models discussed in this section are valid options. The design choice must be driven by requirements and constraints such as cost. For example, an access switch with Layer 3 capabilities is more expensive than a Layer 2-only switch, which can be a valid tiebreaker if cost is a concern.
Review Questions
1. Which of the following are the benefits of a routed access design model over a classical STP-based access design model? (Choose two.)
- Easier troubleshooting
- Flexibility to span Layer 2 natively
- Faster convergence time
- Potentially lower costs based on licensed features
a and c. The routed access design model is easier to troubleshoot and has a faster convergence time than a classical STP-based access design by eliminating the reliance on STP and FHRP, and relying on ECMP for traffic load sharing.
2. Which of the following is a technical network design limitation of a routed access design model?
- More difficult troubleshooting
- Inability to span Layer 2 natively
- Slower convergence time
- Potentially higher costs based on licensed features
b. One of the technical network design limitations of a routed access design model is the inability to span Layer 2 natively.
3. Which of the following is a business network design limitation of a routed access design model?
- More difficult troubleshooting
- Inability to span Layer 2 natively
- Slower convergence time
- Potentially higher costs based on licensed features
d. A business network design limitation of the routed access design model is the higher monetary cost for routing capabilities on the switch infrastructure for the corresponding licensed features required for routing protocols.
4. Which of the following is the best access-distribution connectivity model if a network design requires the most flexible design?
- Multitier STP based
- Switch clustering
- Routed access
- Layer 2 access
b. Switch clustering is the best access-distribution connectivity model for flexibility.
5. Which of the following is the best access-distribution connectivity model if a network design requires the most scalable design, both scale up and scale out?
- Multitier STP based
- Switch clustering
- Routed access
- Layer 2 access
c. The routed access connectivity model supports both scale up and scale out.
Campus Layer 3 Routing Design Considerations
The hierarchical enterprise campus architecture facilitates a structured hierarchical Layer 3 routing design, which is the key to achieving routing scalability in large networks. This reduces the number of Layer 3 nodes and adjacencies in any given routing domain within each tier.
In a typical hierarchical campus, the distribution layer is the demarcation point between Layer 2 and Layer 3 domains. Layer 3 uplinks at the distribution layer participate in campus core routing using an IGP or BGP to interconnect multiple distribution blocks for end-to-end IP connectivity. With the routed access model, Layer 3 routing is extended to the access layer switches.
Figure 6 illustrates a typical IGP design (OSPF) aligned with the hierarchical campus architecture across multiple functional modules. Route summarization to the core with OSPF can lead to suboptimal routing when links fail. This is a common trade-off in any design with summaries at multiple ABRs.
In some designs, the data center is interconnected over a WAN or dedicated fiber links with the campus network. Also, the other blocks, such as Internet and WAN blocks, might be co-resident at the data center’s physical location as well. In this case, you can use the same IGP design concept, and you can treat the WAN interconnect as an external link. Also, the enterprise core routing (that is, the OSPF backbone area) can be extended over the WAN. In other words, all the concepts of IGP and BGP design discussed earlier in this book have to be considered to make the right design decision. (There is no single standard design that you can use in different scenarios.)
Each routing protocol has its own characteristics when applied to different network topologies:
- EIGRP offers more flexible, scalable, and easier-to-control design over hub-and-spoke topologies. However, in large-scale campus networks, if EIGRP is not designed properly with regard to query scope containment (EIGRP stubs and summarization), any topology change may flood the network with EIGRP queries and increase the risk of stuck-in-active (SIA) impacts, resulting in long convergence times following a failure event.
- OSPF is one of the most commonly implemented protocols in campus networks and has proven to be powerful, scalable, and reliable in multitier topologies, though the more tiers the network has, the less flexible the design can be without proper area design.
Network designers must evaluate each design scenario and balance technical (protocol characteristics) and non-technical (business priorities, future plans, staff knowledge) aspects when making routing design decisions. Table 2 summarizes the key differences.
| Design Consideration | EIGRP (DUAL) | Link State (Dijkstra) |
|---|---|---|
| Architecture flexibility | High (natively supports multitier architectures with route summarization) | High, with limitations (the more tiers, the less flexible without proper area design) |
| Scalability | High with proper query domain containment via EIGRP stubs and summarization | High with proper OSPF area design and area type selection |
| Convergence time | Fast (ideally with route summarization) | Fast (ideally with topology hiding, route summarization, and timer tuning) |
| MPLS-TE support | No | Yes |
Review Questions
6. Which of the following routing protocols for the campus has the highest architecture flexibility, proper route summarization, and no limitation to the number of tiers?
- EIGRP
- OSPF
- RIP
- IS-IS
a. EIGRP has the highest architecture flexibility without limitations to the number of tiers while including proper route summarization. OSPF and IS-IS have inherent limitations as the number of tiers increases. Note that BGP is technically the most flexible option but is not listed.
7. Which of the following routing protocols for the campus supports MPLS-TE?
- EIGRP
- OSPF
- RIP
- BGP
b. Of the routing protocols listed, OSPF is the only IGP that supports MPLS Traffic Engineering (MPLS-TE).
Campus Network Virtualization Design Considerations
Virtualization in IT generally refers to the concept of having two or more instances of a system component or function (such as an operating system, network services, control plane, or applications). Typically, these instances are represented in a logical virtualized manner instead of being physical.
Virtualization can generally be classified into two primary models:
- Many-to-one virtualization model: Multiple physical resources appear as a single logical unit. The classical example is the switch clustering concept discussed earlier. Other examples include firewall clustering and FHRP with a single virtual IP (VIP) that front-ends a pair of physical upstream network nodes (switches or routers).
- One-to-many virtualization model: A single physical resource can appear as many logical units, such as virtualizing an x86 server where the software (hypervisor) hosts multiple virtual machines (VMs) on the same physical server. The concept of network function virtualization (NFV) can also be considered a one-to-many system virtualization model.
Drivers to Consider Network Virtualization
To meet the current expectations of business and IT leaders, a more responsive IT infrastructure is required. Network infrastructures need to move from the classical architecture (based on providing basic interconnectivity between different siloed departments) into a more flexible, resilient, and adaptive architecture that can support and accelerate business initiatives and remove inefficiencies. The IT and network infrastructure will become like a service delivery business unit that can quickly adapt and deliver services. In other words, it will become a “business enabler.” This is why network virtualization is considered one of the primary principles that enable IT infrastructures to become more dynamic and responsive to the rapidly changing requirements of today’s enterprises.
The primary drivers motivating enterprises to adopt network virtualization:
- Cost efficiency and design flexibility: Network virtualization provides a level of abstraction from the physical network infrastructure that can offer cost-effective network designs along with a higher degree of design flexibility, where multiple logical networks can be provisioned over one common physical infrastructure. This ultimately leads to lower CAPEX because of the reduction in device complexity and the number of devices. Similarly, it lowers OPEX because the operations team will have fewer devices to manage.
- Simplified and flexible integrated security: Network virtualization promotes flexible security designs by allowing the use of separate security policies per logical or virtualized entity, where users’ groups and services can be logically separated.
- Design and operational simplicity: Network virtualization simplifies the design and provisioning of path and traffic isolation per application, group, service, and various other logical instances that require end-to-end path isolation.
It is important that network designers understand the drivers toward adopting network virtualization from a business point of view, along with the strengths and weaknesses of each design model. This ensures that when a network virtualization concept is considered in a given area within the network or across the entire network, it will deliver the promised value (and not be used only because it is easy to implement or it is an innovative approach). A design that does not address the business’s functional requirements is considered a poor design. Refer to the “no gold plating” principle discussed in Network Design.
One of the main concerns about network virtualization is the concept of fate sharing (shared failure state), because any failure in the physical network can lead to a failure of multiple virtual networks running over the same physical infrastructure. Therefore, when the network virtualization concept is used, ideally a reliable and highly available network design should be considered as well. Besides the constraints about virtual network availability, there is always a concern about network virtualization (multitenant environment) where multiple virtual networks (VNs) operate over a single physical network infrastructure and each VN probably has different traffic requirements (different applications and utilization patterns). Therefore, there is a higher potential of having traffic congestion and degraded application quality and user experience if there is no efficient planning with regard to the available bandwidth, number of VNs, traffic volume per VN, applications in use, and the characteristics of the applications. In other words, if there is no adequate bandwidth available and the quality of service (QoS) policies to optimize and control traffic behaviors, one VN may overutilize the available bandwidth of the underlying physical network infrastructure. This will usually lead to traffic congestion because other VNs are using the same underlying physical network infrastructure, resulting in fate sharing.
Network Virtualization Design Elements
As shown in Figure 7, the main elements in an end-to-end network virtualization design are:
- Edge control: Represents the network access point. This is typically a host or end-user access (wired, wireless, or VPN) to the network where identification (authentication) for physical-to-logical network mapping can occur. For example, a contracting employee might be assigned to VLAN X, whereas internal staff are assigned to VLAN Y.
- Transport virtualization: Represents the transport path that carries different virtualized networks over one common physical infrastructure, such as an overlay technology like GRE tunneling. The terms path isolation and path separation are commonly used to refer to transport virtualization.
- Services virtualization: Represents the extension of the network virtualization concept to the services edge, which can be shared services among different logically isolated groups. Examples include an Internet link or a file server located in the data center that must be accessed by only one logical group (business unit).
Enterprise Network Virtualization Deployment Models
Now that the different elements that individually or collectively form the foundational elements to create network virtualization within the enterprise network architecture have been identified, this section covers how these elements can be used with different design techniques and approaches to deploying network virtualization across the enterprise campus. Network virtualization can be categorized into the following three primary models, each of which has different techniques that can serve different requirements:
- Device virtualization
- Path isolation
- Services virtualization
These techniques can be used individually to serve certain requirements or combined to achieve one cohesive end-to-end network virtualization solution. Network designers must have a good understanding of the different techniques and approaches, along with their attributes, to select the most suitable virtualization technologies and design approaches for delivering value to the business.
Device Virtualization
Also known as device partitioning, device virtualization represents the ability to virtualize the data plane, control plane, or both, in a certain network node such as a switch or a router. Using device-level virtualization by itself helps to achieve separation at Layer 2, Layer 3, or both, on a local device level. The primary techniques are:
- VLAN: The most common Layer 2 network virtualization technique. A single switch is divided into multiple logical Layer 2 broadcast domains that are virtually separated from other VLANs. VLANs are used at the network edge to place an endpoint into a certain virtual network. Each VLAN has its own MAC forwarding table and spanning-tree instance (Per-VLAN Spanning Tree [PVST]).
- VRF (Virtual Routing and Forwarding): Conceptually similar to a VLAN but from a control plane and forwarding perspective on a Layer 3 device. A VRF can be combined with a VLAN to provide a virtualized Layer 3 gateway service. Each VLAN over an 802.1Q trunk can be mapped to a different subinterface that is assigned to a unique VRF, where each VRF maintains its own forwarding and routing instance and potentially leverages different VRF-aware routing protocols (for example, OSPF or EIGRP instance per VRF).
Path Isolation
Path isolation refers to the concept of maintaining end-to-end logical path transport separation across the network. The end-to-end path separation can be achieved using the following main design approaches:
Hop by hop: This design approach, as shown in Figure 8, is based on deploying end-to-end VLANs + 802.1Q trunk links + VRFs per device in the traffic path. This approach offers a simple and reliable path separation solution. However, for large-scale dynamic networks (a large number of virtualized networks), it will be a complicated solution to manage. This complexity is associated with design scalability limitations.
Multihop: This approach is based on using tunneling and other overlay technologies to provide end-to-end path isolation and carry the virtualized traffic across the network. The most common proven methods include:
Tunneling (GRE/mGRE/DMVPN): Eliminates the reliance on deploying end-to-end VRFs and 802.1Q trunks across the enterprise network because the virtualized traffic is carried over the tunnel. This method offers a higher level of scalability compared to hop-by-hop and with simpler operation to some extent. This design is ideally suitable for scenarios where only a part of the network needs path isolation. However, for large-scale networks with multiple logical groups or business units to be separated across the enterprise, the tunneling approach can add complexity to the design and operations. mGRE can provide the same transport and path isolation goal for larger networks with lower design and operational complexities. (See the section “WAN Virtualization” in Enterprise WAN Architecture Design for a detailed comparison between the different path separation approaches over different types of tunneling mechanisms.)
MPLS VPN: By converting the enterprise to be like a service provider type of network, where the core is MPLS-enabled and the distribution layer switches act as provider edge (PE) devices, each PE (distribution block) exchanges VPN routing over MP-BGP sessions, as shown in Figure 9. The route reflector (RR) concept can be introduced to reduce the complexity of full-mesh MP-BGP peering sessions. Furthermore, L2VPN capabilities can be introduced in this architecture, such as Ethernet over MPLS (EoMPLS), to provide extended Layer 2 communications across different distribution blocks if required. With this design approach, the end-to-end virtualization and traffic separation can be simplified to a very large extent with a high degree of scalability.
Figure 10 illustrates a summary of the different enterprise campus network virtualization design techniques.
Table 3 summarizes the key trade-offs between the different network virtualization techniques.
| VLANs + VRFs + 802.1Q (End to End) | VLANs + VRFs + GRE Tunnels | VLANs + VRFs + mGRE Tunnels | MPLS with MP-BGP | |
|---|---|---|---|---|
| Scalability | Low | Low | Moderate | High |
| Design flexibility | High | Moderate | Moderate | Moderate to high |
| Operational complexity | Low | Moderate | Moderate | High |
| Architecture | Per-hop end-to-end virtualization | P2P multihop end-to-end virtualization | P2MP multihop end-to-end virtualization | MPLS L3VPN-based virtualization |
| Operations staff routing expertise | Basic | Medium | Medium | Advanced |
Service Virtualization
One of the main goals of virtualization is to separate services access into different logical groups, such as user groups or departments. However, in some scenarios, there may be a mix of these services in terms of service access, in which some services must only be accessed by a certain group and others are to be shared among different groups. Examples include a file server in the data center or Internet access, as shown in Figure 11.
Therefore, in scenarios where service access must be separated per virtual network or group, the concept of network virtualization must be extended to the services access edge. This includes devices such as a server with multiple VMs or an Internet edge router with single or multiple Internet links.
The virtualization of a network can be extended to other network service appliances, such as firewalls. For instance, you can have a separate virtual firewall per virtual network, to facilitate access control between the virtual user network and the virtualized services and workload. The virtualization of network services can be considered a “one-to-many” network device-level virtualization.
In multitenant network environments, multiple security contexts offer a flexible and cost-effective solution for enterprises (and for service providers). This approach enables network operators to partition a single pair of redundant firewalls or a single firewall cluster into multiple virtual firewall instances per business unit or tenant. Each tenant can then deploy and manage its own security policies and service access, which are virtually separated. This approach also allows controlled inter-tenant communication. For example, in a typical multitenant enterprise campus network environment with MPLS VPN (L3VPN) enabled at the core, traffic between different tenants (VPNs) is normally routed via a firewalling service for security and control (who can access what).
The following are the common techniques that facilitate accessing shared applications and network services in multitenant environments:
- VRF-Aware NAT: One of the common requirements in today’s multitenant environments with network and service virtualization enabled is to provide each virtual (tenant) network the ability to access certain services (shared services) either hosted on premises (such as at the enterprise data center or services block) or hosted externally (in a public cloud). Also, providing Internet access to the different tenants’ (virtual) networks is a common example. To maintain traffic separation between the different tenants (virtual networks) where private IP address overlapping is a common attribute, NAT is considered one of the common and cost-effective solutions to provide NAT per tenant without compromising path separation requirements. When NAT is combined with different virtual network instances (VRFs), it is commonly referred to as VRF-Aware NAT, as shown in Figure 12. When multiple routing instances are combined into a single instance on a common device, with or without NAT, that is called fusion routing and the common device is called a fusion router.
VRF-Aware Services Infrastructure (VASI) refers to the ability of an infrastructure or a network node, such as a router, to facilitate the application of features and management services (such as encryption and NAT) between VRFs internally within the same node, using virtual interfaces. For two VRFs to communicate internally within a network node (router), a VASI virtual interface pair can be configured. Each interface in this pair must be associated with a different VRF so that those two virtual interfaces can be logically wired.
- NFV (Network Functions Virtualization): The concept of NFV is based on virtualizing network functions that typically require a dedicated physical node, appliances, or interfaces. In other words, NFV can potentially take any network function typically residing in purpose-built hardware and abstract it from that hardware. This concept offers businesses several benefits:
- Reduce the total cost of ownership (TCO) by reducing the required number and diversity of specialized appliances
- Reduce operational cost (for example, less power and space)
- Offer a cost-effective capital investment
- Reduce the level of complexity of integration and network operations
- Reduce time to market for the business by offering the ability to enable specialized network services (especially in multitenant environments where a separate network function/service per tenant can be provisioned faster)
The NFV concept helps businesses to adopt and deploy new services quickly (faster time to market) and is consequently considered a business innovation enabler. This is because purpose-built hardware functionalities have been virtualized, and it is a matter of service enablement rather than relying on new hardware (along with infrastructure integration complexities).
The concept of NFV is commonly adopted by service provider networks. Nonetheless, this concept is applicable and usable in enterprise networks and enterprise data center networks that want to gain its benefits and flexibility. In large-scale networks with a very high volume of traffic (typically carrier-grade), hardware resource utilization and limits must be considered.
Review Questions
8. Which of the following network virtualization techniques would be the proper fit if the network design required a high level of scalability?
- VLANs + 802.1Q + VRF
- VLANs + VRFs + GRE tunnels
- VLANs + VRFs + mGRE tunnels
- MPLS with MP-BGP
d. The most scalable virtualization option is MPLS with MP-BGP. This also comes with high operational complexity and requires staff to have advanced routing experience.
9. Which of the following designs would suit customers with a basic level of routing expertise?
- VLAN + 802.1Q + VRF
- VLANs + VRFs + GRE tunnels
- VLANs + VRFs + mGRE tunnels
- MPLS with MP-BGP
a. The design option that requires the least level of routing expertise is VLANs + 802.1Q + VRFs.
Summary
The enterprise campus is one of the vital parts of the modular enterprise network. It is the medium that connects end users and the different types of endpoints (such as printers, video endpoints, and wireless access points) to the enterprise network. Therefore, having the right structure and design layout that meets current and future requirements is critical, including the physical infrastructure layout, Layer 2, and Layer 3 designs. To achieve a scalable and flexible campus design, it should ideally be based on hierarchical and modular design principles that optimize the overall design architecture in terms of fault isolation, simplicity, and network convergence time. It should also offer a desirable level of flexibility to integrate other networks and new services and grow in size.
The concept of network virtualization helps enterprises to utilize the same underlying physical infrastructure while maintaining access, path, and services access isolation, to meet certain business goals or functional security requirements. As a result, enterprises can lower CAPEX and OPEX and reduce the time and effort required to provision a new service or a new logical network. However, the network designer must consider the different network virtualization design options, along with the strengths and weaknesses of each, to deploy the suitable network virtualization technique that meets current and future needs. These needs must take into account the different variables and constraints, such as staff knowledge and the hardware platform-supported features and capabilities.
Previous: Network Services and Management | Next: Enterprise Internet Edge Architecture Design