This helps ensure infrastructure is deployed consistently in a single data center or across multiple data centers, while also helping to reduce costs and the time employees spend maintaining it. This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Customer edge links (access and trunk) carry traditional VLAN tagged and untagged frames. It provides control-plane and data-plane separation and a unified control plane for both Layer 2 and Layer 3 forwarding in a VXLAN overlay network. Due to the limitations of In fact, according to Moore’s Law (named after the co-founder of Intel, Gordon Moore), computing power doubles every few years. Its control-plane protocol, FabricPath IS-IS, is designed to determine FabricPath switch ID reachability information. With the ingress replication feature, the underlay network is multicast free. Most customers use eBGP because of its scalability and stability. This architecture is the physical and logical layout of the resources and equipment within a data center facility. It also performs internal inter-VXLAN routing and external routing. Its architecture is based around the idea of a simple volumetric block enveloped by opaque, transparent, and translucent surfaces. Intel RSD is an implementation specification enabling interoperability across hardware and software vendors. A central datastructure or data store or data repository, which is responsible for providing permanent data storage. His experience also includes providing analysis of critical application support facilities. Underlay IP PIM or the ingress replication feature is used to send broadcast and unknown unicast traffic. The Layer 2 and Layer 3 function is enabled on some FabricPath leaf switches called border leaf switches. Common Layer 3 designs provide centralized routing: that is, the Layer 3 routing function is centralized on specific switches (spine switches or border leaf switches). ●      It uses the decade-old MP-BGP VPN technology to support scalable multitenant VXLAN overlay networks. This design complies with the IETF RFC 7348 and draft-ietf-bess-evpn-overlay standards. A typical FabricPath network uses a spine-and-leaf architecture. Hyperscale users and increased demand have turned data into the new utility, making quicker, leaner facilities a must. The multicast distribution tree for this group is built through the transport network based on the locations of participating VTEPs. Many different tools are available from Cisco, third parties, and the open-source community that can be used to monitor, manage, automate, and troubleshoot the data center fabric. A legacy mindset in data center architecture revolves around the notion of “design now, deploy later.” The approach to creating a versatile, digital-ready data center must involve the deployment of infrastructure during the design session. Data Center Architects are responsible for adequately securing the Data Center and should examine factors such as facility design and architecture. As shown in the design for internal and external routing at the border leaf in Figure 7, the spine switch functions as the Layer 2 FabricPath switch and performs intra-VLAN FabricPath frame switching only. With this design, tenant traffic needs to take only one underlay hop (VTEP to spine) to reach the external network. To learn end-host reachability information, FabricPath switches rely on initial data-plane traffic flooding. The multi-tier model uses software that runs as separate processes on the same machine using interprocess communication (IPC), or on different machines with communication… Internal and external routing on the border leaf. If one of the top tier switches were to fail, it would only slightly degrade performance throughout the data center. Each VTEP performs local learning to obtain MAC address (though traditional MAC address learning) and IP address information (based on Address Resolution Protocol [ARP] snooping) from its locally attached hosts. Although the concept of a network overlay is not new, interest in network overlays has increased in the past few years because of their potential to address some of these requirements. The Azure Architecture Center provides best practices for running your workloads on Azure. Similarly, there is no single way to manage the data center fabric. Table 3. The three-tier is the common network architecture used in data centers. To overcome the limitations of flood-and-learn VXLAN, Cisco VXLAN MP-BGP EVPN spine-and-leaf architecture uses Multiprotocol Border Gateway Protocol Ethernet Virtual Private Network, or MP-BGP EVPN, as the control plane for VXLAN. Cisco VXLAN MP-BGP EVPN spine-and-leaf network multitenancy, Cisco VXLAN MP BGP-EVPN spine-and-leaf network summary. The path is randomly chosen so that the traffic load is evenly distributed among the top-tier switches. It is clear from past history that code minimum is not the best practice. The target of maximum efficiency is achieved by considering these below-mentioned factors. The Cisco VXLAN MP-BGP EVPN spine-and-leaf architecture uses MP-BGP EVPN for the control plane for VXLAN. The original Layer 2 frame is encapsulated in a VXLAN header and then placed in a UDP-IP packet and transported across the IP network. In most cases, the spine switch is not used to directly connect to the outside world or to other MSDC networks, but it will forward such traffic to specialized leaf switches acting as border leaf switches. The VN-segment feature provides a new way to tag packets on the wire, replacing the traditional IEEE 802.1Q VLAN tag. The impact of broadcast and unknown unicast traffic flooding needs to be carefully considered in the FabricPath network design. It is designed to simplify, optimize, and automate the modern multitenancy data center fabric environment. Explore HED’s integrated architectural and engineering practice. Enterprise and High Performance Computing users recognize the value of critical facilities — connecting to a brand is as important as connecting to the campus. The origins of the Uptime Institute as a data center users group established it as the first group to measure and compare a data center’s reliability. The entire purpose of designing a data center revolves around maximum utilization of IT resources for the sake of boosted efficiency, improved sales, and operational costs and fewer environmental effects. This feature uses a 24-bit increased name space. VLANs are extended within each pod that servers can move freely within the pod without the need to change IP address and default gateway configurations. DCP_2047.JPG 1/6 Internal and external routed traffic needs to travel one underlay hop from the leaf VTEP to the spine switch to be routed. AWS pioneered cloud computing in 2006, creating cloud infrastructure that allows you to securely build and innovate faster. VXLAN MP-BGP EVPN uses distributed anycast gateways for internal routed traffic. For example, fabrics need to support scaling of forwarding tables, scaling of network segments, Layer 2 segment extension, virtual device mobility, forwarding path optimization, and virtualized networks for multitenant support on shared physical infrastructure. As shown in the design for internal and external routing on the spine layer in Figure 12, the leaf ToR VTEP switch is a Layer 2 VXLAN gateway to transport the Layer 2 segment over the underlay Layer 3 IP network. The three major data center design and infrastructure standards developed for the industry include: This standard develops a performance-based methodology for the data center during the design, construction, and commissioning phases to determine the resiliency of the facility with respect to four Tiers or levels of redundancy/reliability. It enables the logical Cisco MSDC Layer 3 spine-and-leaf network. Traditional three-tier data center design The architecture consists of core routers, aggregation routers (sometimes called distribution routers), and access switches. VXLAN MP-BGP EVPN supports overlay tenant Layer 2 multicast traffic using underlay IP multicast or the ingress replication feature. Gensler, Corgan, and HDR top Building Design+Construction’s annual ranking of the nation’s largest data center sector architecture and A/E firms, as reported in the 2016 Giants 300 Report. The multi-tier approach includes web, application, and database tiers of servers. They must also play an active role in manageability and operations of the data center. It delivers tenant Layer 3 multicast traffic in an efficient and resilient way. The spine switch learns external routes and advertises them to the EVPN domain as EVPN routes so that other VTEP leaf nodes can also learn about the external routes for sending outbound traffic. You need to consider MAC address scale to avoid exceeding the scalability limit on the border leaf switch. Layer 2 multitenancy example with FabricPath VN-Segment feature. The VXLAN MP-BGP EVPN spine-and-leaf architecture uses VXLAN encapsulation. To support multitenancy, same VLANs can be reused on different FabricPath leaf switches, and IEEE 802.1Q tagged frames are mapped to specific VN-segments. After traffic is routed to the destination VLAN, then it is forwarded using the multidestination tree in the destination VLAN. Modern virtualized data center fabrics must meet certain requirements to accelerate application deployment and support DevOps needs. We will discuss best practices with respect to facility conceptual design, space planning, building construction, and physical security, as well as mechanical, electrical, plumbing, and fire protection. Up to four FabricPath anycast gateways can be enabled in the design with routing at the border leaf. The data center architecture specifies where and how the server, storage networking, racks and other data center resources will be physically placed. Typically, data center architecture … You need to design multicast group scaling carefully, as described earlier in the section discussing Cisco VXLAN flood-and-learn multicast traffic. The spine switch is just part of the underlay Layer 3 IP network to transport the VXLAN encapsulated packets. This approach reduces network flooding for end-host learning and provides better control over end-host reachability information distribution. Data Center Design, Inc. provides customers with projects ranging from new Data Center design and construction to Data Center renovation and expansion with follow-up service. However, it is still a flood-and-learn-based Layer 2 technology. Data Center Design and Implementation Best Practices: This standard covers the major aspects of planning, design, construction, and commissioning of the MEP building trades, as well as fire protection, IT, and maintenance. Cisco spine-and-leaf layer 2 and layer 3 fabric comparison, Cisco Spine-and-Leaf Layer 2 and Layer 3 Fabric, Forwarded by underlay PIM or ingress replication, (Note: Ingress-replication is supported only on Cisco Nexus 9000 Series Switches. Common Layer 3 designs use centralized routing: that is, the Layer 3 routing function is centralized on specific switches (spine switches or border leaf switches). The key is to choose a standard and follow it. Features exist, such as the FabricPath Multitopology feature, to help limit traffic flooding in a subsection of the FabricPath network. In MP-BGP EVPN, multiple tenants can co-exist and share a common IP transport network while having their own separate VPNs in the VXLAN overlay network (Figure 19). For feature support and more information about TRM, please refer to the configuration guides, release notes, and reference documents listed at the end of this document. Moreover, scalability is another major issue in three-tier DCN. Cisco spine-and-leaf layer 2 and layer 3 fabric comparison. Each tenant has its own VRF routing instance. IP subnets of the VNIs for a given tenant are in the same Layer 3 VRF instance that separates the Layer 3 routing domain from the other tenants. Each FabricPath switch is identified by a FabricPath switch ID. Mecanoo has unveiled their design for the Qianhai Data Center in Shenzhen, China, from which they received second prize in an international design … A data center floor plan includes the layout of the boundaries of the room (or rooms) and the layout of IT equipment within the room. Another challenge in a three-tier architecture is that server-to-server latency varies depending on the traffic path used. Massively scalable data centers (MSDCs) are large data centers, with thousands of physical servers (sometimes hundreds of thousands), that have been designed to scale in size and computing capacity with little impact on the existing infrastructure. An edge or leaf device can optimize its functions and all its relevant protocols based on end-state information and scale, and a core or spine device can optimize its functions and protocols based on link-state updates, optimizing with fast convergence. The VXLAN MP-BGP EVPN spine-and-leaf architecture uses MP-BGP EVPN for the control plane. The VXLAN flood-and-learn spine-and-leaf network also supports Layer 3 multitenancy using VRF-lite (Figure 15). In 2010, Cisco introduced virtual-port-channel (vPC) technology to overcome the limitations of Spanning Tree Protocol. Between the aggregation routers and access switches, Spanning Tree Protocol is used to build a loop-free topology for the Layer 2 part of network. The Tiers are compared in the table below and can be found in greater definition in UI’s white paper TUI3026E. The FabricPath IS-IS control plane builds reachability information about how to reach other FabricPath switches. In a VXLAN flood-and-learn spine-and-leaf network, overlay tenant Layer 2 multicast traffic is supported using underlay IP PIM or the ingress replication feature. We are continuously innovating the design and systems of our data centers to protect them from man-made and natural risks. As the number of hosts in a broadcast domain increases, the negative effects of flooding packets are more pronounced. About the author: Steven Shapiro has been in the mission critical industry since 1988 and has a diverse background in the study, reporting, design, commissioning, development and management of reliable electrical distribution, emergency power, lighting, and fire protection systems for high tech environments. Underlay IP multicast is used to reduce the flooding scope of the set of hosts that are participating in the VXLAN segment. Benefits of a network virtualization overlay include the following: ●      Optimized device functions: Overlay networks allow the separation (and specialization) of device functions based on where a device is being used in the network. Data center design is the process of modeling an,.l designing (Jochim 2017) a data center's IT resources, architectural layout and entire ilfrastructure. Table 2 summarizes the characteristics of a VXLAN flood-and-learn spine-and-leaf network. It provides workflow automation, flow policy management, and third-party studio equipment integration, etc. Most users do not understand how critical the floor layout is to the performance of a data center, or they only understand its importance after a It has modules on all the major sub-systems of a mission critical facility and their interdependencies, including power, cooling, compute and network. Servers are virtualized into sets of virtual machines that can move freely from server to server without the need to change their operating parameters. This section describes VXLAN MP-BGP EVPN on Cisco Nexus hardware switches such as the Cisco Nexus 5600 platform switches and Cisco Nexus 7000 and 9000 Series Switches. Many MSDC customers write scripts to make network changes, using Python, Puppet and Chef, and other DevOps tools and Cisco technologies such as Power-On Auto Provisioning (POAP). Data Centre World Singapore speaker and mission critical architect Will Ringer attests to the importance of an architect’s eye to data centre design. Distributed anycast gateway for internal routing. Cisco VXLAN MP-BGP EVPN spine-and-leaf network. A data center is going to probably be the most expensive facility your company ever builds or operates. But the FabricPath network is flood-and-learn-based Layer 2 technology. It provides control-plane and data-plane separation and a unified control plane for both Layer 2 and Layer 3 forwarding in a VXLAN overlay network. The higher layers of the three-tier DCN are highly oversubscribed. Multicast group scaling needs to be designed carefully. That is definitely not best practice. Data center architecture and engineering firm Integrated Design Group is merging with national firm HED in a deal that illustrates the rising profile for the data center industry. With the anycast gateway function in EVPN, end hosts in a VNI always can use their local VTEPs for this VNI as their default gateway to send traffic out of their IP subnet. Data center architecture is usually created in the data center design and constructing phase. Cisco VXLAN MP-BGP EVPN network characteristics, Localized flood and learn with ARP suppression, Forwarded by underlay multicast (PIM) or ingress replication, (Note: Ingress replication is supported only on Cisco Nexus 9000 Series Switches. At the same time, it runs the normal IPv4 or IPv6 unicast routing in the tenant VRF instances with the external routing device on the outside. The FabricPath spine-and-leaf network uses Layer 2 FabricPath MAC-in-MAC frame encapsulation, and it uses FabricPath IS-IS for the control-plane in the underlay network. It is a for-profit entity that will certify a facility to its standard, for which the standard is often criticized. The Layer 3 routing function is laid on top of the Layer 2 network. ●      It provides optimal forwarding for east-west and north-south traffic and supports workload mobility with the distributed anycast function on each ToR switch. There are also many operational standards to choose from. The standard breaks down as follows: Government regulations for data centers will depend on the nature of the business and can include HIPPA (Health Insurance Portability and Accountability Act), SOX (Sarbanes Oxley) 2002, SAS 70 Type I or II, GLBA (Gramm-Leach Bliley Act), as well as new regulations that may be implemented depending on the nature of your business and the present security situation. In the VXLAN flood-and-learn mode defined in RFC 7348, end-host information learning and VTEP discovery are both data-plane based, with no control protocol to distribute end-host reachability information among the VTEPs. For more details regarding MSDC designs with Cisco Nexus 9000 and 3000 switches, please refer “Cisco’s Massively Scalable Data Center Network Fabric White Paper”. Because the fabric network is so large, MSDC customers typically use software-based approaches to introduce more automation and more modularity into the network. Note that the maximum number of inter-VXLAN active-active gateways is two with a Hot-Standby Router Protocol (HSRP) and vPC configuration. If deviations are necessary because of site limitations, financial limitations, or availability limitations, they should be documented and accepted by all stakeholders of the facility. Figure 20 shows an example of a Layer 3 MSDC spine-and-leaf network with an eBGP control plane (AS = autonomous system). The layered methodology is the elementary foundation of the data center design that improves scalability, flexibility, performance, maintenance, and resiliency. The ease of expansion optimizes the IT department’s process of scaling the network. With overlays used at the fabric edge, the spine and core devices are freed from the need to add end-host information to their forwarding tables. Lines and paragraphs break automatically. Note that the ingress replication feature is supported only on Cisco Nexus 9000 Series Switches. The spine layer is the backbone of the network and is responsible for interconnecting all leaf switches. Ideally, you should map one VXLAN segment to one IP multicast group to provide optimal multicast forwarding. (This mode is not relevant to this white paper.). The most efficient and effective data center designs use relatively new design fundamentals to create the required high energy density, high reliability environment. The investment giant is one of the biggest advocates outside Silicon Valley for open source hardware, and the new building itself is a modular, just-in-time construction design. ●      Its underlay and overlay management tools provide many network management capabilities, simplifying workload visibility, optimizing troubleshooting, automating fabric component provisioning, automating overlay tenant network provisioning, etc. Since 2003, with the introduction of virtual technology, the computing, networking, and storage resources that were segregated in pods in Layer 2 in the three-tier data center design can be pooled. January 15, 2020. For a FabricPath network, the FabricPath IS-IS control plane by default creates two multidestination trees that carry broadcast traffic, unknown unicast traffic, and multicast traffic through the FabricPath network. 5. Hosts attached to remote VTEPs are learned remotely through the MP-BGP control plane. The VTEP then distributes this information through the MP-BGP EVPN control plane. If oversubscription of a link occurs (that is, if more traffic is generated than can be aggregated on the active link at one time), the process for expanding capacity is straightforward. It doesn’t learn host MAC addresses. A data accessoror a collection of independent components that operate on the central data store, perform computations, and might put back the results. This design complies with IETF VXLAN standards RFC 7348 and draft-ietf-bess-evpn-overlay. With this design, tenant traffic needs to take two underlay hops (VTEP to spine to border leaf) to reach the external network. Depending on the number of servers that need to be supported, there are different flavors of MSDC designs: two-tiered spine-leaf topology, three-tiered spine-leaf topology, hyperscale fabric plane Clos design. The routing protocol can be regular eBGP or any IGP of choice. Regarding routing design, the Cisco MSDC control plane uses dynamic Layer 3 protocols such as eBGP to build the routing table that most efficiently routes a packet from a source to a spine node. To support multitenancy, the same VLAN can be reused on different VTEP switches, and IEEE 802.1Q tagged frames received on VTEPs are mapped to specific VNIs. Each host is associated with a host subnet and talks with other hosts through Layer 3 routing. The multi-tier data center model is dominated by HTTP-based applications in a multi-tier approach. Also, the spine Layer 3 VXLAN gateway learns the host MAC address, so you need to consider the MAC address scale to avoid exceeding the scalability limits of your hardware. With a spine-and-leaf architecture, no matter which leaf switch to which a server is connected, its traffic always has to cross the same number of devices to get to another server (unless the other server is located on the same leaf). For Layer 2 multicast traffic, traffic entering the FabricPath switch is hashed to a multidestination tree to be forwarded. ●      Cisco Network Insights – Resources (NIR): provides a way to gather information through data collection to get an overview of available resources and their active processes and configurations across the entire Data Center Network Manager (DCNM). FabricPath enables new capabilities and design options that allow network operators to create Ethernet fabrics that increase bandwidth availability, provide design flexibility, and simplify and reduce the costs of network and application deployment and operation. Table 3 summarizes the characteristics of the VXLAN MP-BGP EVPN spine-and-leaf network. Codes must be followed when designing, building, and operating your data center, but “code” is the minimum performance requirement to ensure life safety and energy efficiency in most cases. The three major data center design and infrastructure standards developed for the industry include:Uptime Institute's Tier StandardThis standard develops a performance-based methodology for the data center during the design, construction, and commissioning phases to determine the resiliency of the facility with respect to four Tiers or levels of redundancy/reliability. You can also have multiple VXLAN segments share a single IP multicast group in the core network; however, the overloading of multicast groups leads to suboptimal multicast forwarding. Best practices ensure that you are doing everything possible to keep it that way. The leaf layer consists of access switches that connect to devices such as servers. As shown in the design for internal and external routing on the border leaf in Figure 13, the leaf ToR VTEP switch is a Layer 2 VXLAN gateway to transport the Layer 2 segment over the underlay Layer 3 IP network. As the number of hosts in a broadcast domain increases, it suffers the same flooding challenges as a FabricPath spine-and-leaf network. For Layer 3 IP multicast traffic, traffic needs to be forwarded by Layer 3 multicast using Protocol-Independent Multicast (PIM). It provides real-time health summaries, alarms, visibility information, etc. These are the VN-segment edge ports. It provides a simple, flexible, and stable network, with good scalability and fast convergence characteristics, and it can use multiple parallel paths at Layer 2. With vPC technology, Spanning Tree Protocol is still used as a fail-safe mechanism. Table 2. The overlay network uses flood-and-learn semantics (Figure 11). Table 1. The border leaf switch can also be configured to send EVPN routes learned in the Layer 2 VPN EVPN address family to the IPv4 or IPv6 unicast address family and advertise them to the external routing device. The architecture consists of core routers, aggregation routers (sometimes called distribution routers), and access switches. ●      Overlapping addressing: Most overlay technologies used in the data center allow virtual network IDs to uniquely scope and identify individual private networks. These formats include Virtual Extensible LAN (VXLAN), Network Virtualization Using Generic Routing Encapsulation (NVGRE), Transparent Interconnection of Lots of Links (TRILL), and Location/Identifier Separation Protocol (LISP). The border leaf router is enabled with the Layer 3 VXLAN gateway and performs internal inter-VXLAN routing and external routing. The border leaf switch learns external routes and advertises them to the EVPN domain as EVPN routes so that other VTEP leaf nodes can also learn about the external routes for sending outbound traffic. (This mode is not relevant to this white paper. This document presented several spine-and-leaf architecture designs from Cisco, including the most important technology components and design considerations for each architecture at the time of the writing of this document. The Cisco Nexus 9000 Series introduced an ingress replication feature, so the underlay network is multicast free. It also addresses how these resources/devices will be interconnected and how physical and logical security workflows are arranged. This capability enables optimal forwarding for northbound traffic from end hosts in the VXLAN overlay network. Fidelity is opening a new data center in Nebraska this fall. It extends Layer 2 segments over a Layer 3 infrastructure to build Layer 2 overlay logical networks. However, the spine switch needs to run the BGP-EVPN control plane and IP routing and the VXLAN VTEP function. ), Cisco’s Massively Scalable Data Center Network Fabric White Paper, https://www.cisco.com/c/en/us/products/cloud-systems-management/prime-data-center-network-manager/index.html, https://www.cisco.com/c/en/us/support/data-center-analytics/network-insights-data-center/products-installation-and-configuration-guides-list.html, https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-730116.html, https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/guide-c07-734107.html, https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-743245.html, https://blogs.cisco.com/datacenter/vxlan-innovations-on-the-nexus-os-part-1-of-2, Cisco MDS 9000 10-Gbps 8-Port FCoE Module Extends Fibre Channel over Ethernet to the Data Center Core. The MP-BGP EVPN control plane provides integrated routing and bridging by distributing both Layer 2 and Layer 3 reachability information for the end host residing in the VXLAN overlay network. On each FabricPath leaf switch, the network keeps the 4096 VLAN spaces, but across the whole FabricPath network, it can support up to 16 million VN-segments, at least in theory. Common Layer 3 designs use centralized routing: that is, the Layer 3 routing function is centralized on specific switches (spine switches or border leaf switches). Intel RSD defines key aspects of a logical architecture to implement CDI. Layer 2 multitenancy example using the VNI. The SVIs on the border leaf switches perform inter-VLAN routing for east-west internal traffic and exchange routing adjacency with Layer 3 routed uplinks to route north-south external traffic. Green certifications, such as LEED, Green Globes, and Energy Star are also considered optional. If you have multiple facilities across the US, then the US standards may apply. Internal and external routing at the border spine. With Layer 2 segments extended across all the pods, the data center administrator can create a central, more flexible resource pool that can be reallocated based on needs. The data center is a dedicated space were your firm houses its most important information and relies on it being safe and accessible. End-host information in the overlay network is learned through the flood-and-learn mechanism with conversational learning. The VXLAN flood-and-learn spine-and-leaf network supports up to two active-active gateways with vPC for internal VXLAN routing. Cisco VXLAN flood-and-learn network characteristics, (Note: Ingress replication is supported only on Cisco Nexus 9000 Series Switches), (static, Open Shortest Path First [OSPF], IS-IS, External BGP [eBGP], etc.). Environments of this scale have a unique set of network requirements, with an emphasis on application performance, network simplicity and stability, visibility, easy troubleshooting and easy life cycle management, etc. Both designs provide centralized routing: that is, the Layer 3 internal and external routing functions are centralized on specific switches. The architect must demonstrate the capacity to develop a robust server and storage architecture. ●      Fabric scalability and flexibility: Overlay technologies allow the network to scale by focusing scaling on the network overlay edge devices. Internal and external routing on the spine layer. Data Centered Architecture is also known as Database Centric Architecture. Border leaf switches can inject default routes to attract traffic intended for external destinations. Join millions of people using Oodle to find unique job listings, employment offers, part time jobs, and employment news. The VXLAN flood-and-learn spine-and-leaf network supports up to two active-active gateways with vPC for internal VXLAN routing. ●      The EVPN address family carries both Layer 2 and Layer 3 reachability information, thus providing integrated bridging and routing in VXLAN overlay networks. Web page addresses and e-mail addresses turn into links automatically. Interactions or communication between the data accessors is only through the data stor… For a FabricPath network, the FabricPath IS-IS control plane by default creates two multidestination trees that carry broadcast traffic, unknown unicast traffic, and multicast traffic through the FabricPath network. Linkedin Twitter Facebook Subscribe. ), Supports both Layer 2 multitenancy and Layer 3 multitenancy, RFC 7348 and RFC8365 (previously draft-ietf-bess-evpn-overlay). The VXLAN flood-and-learn network is a Layer 2 overlay network, and Layer 3 SVIs are laid on top of the Layer 2 overlay network. Data center design, construction, and operational standards should be chosen based on definition of that mission. It doesn’t learn the overlay host MAC address. Many aspects of this standard reflect the UI, TIA, and BCSI standards. ), ●      Storage Area Network (SAN) controller mode: manages Cisco MDS Series switches for storage network deployment with graphical control for all SAN administration functions. Because the gateway IP address and virtual MAC address are identically provisioned on all VTEPs in a VNI, when an end host moves from one VTEP to another VTEP, it doesn’t need to send another ARP request to relearn the gateway MAC address. Our client-first culture and multi-disciplinary architecture and engineering experts recognize the power of design in transforming the human experience. Will has experience with large US hyperscale clients, serving as project architect for three years on a hyperscale project in Holland, and with some of the largest engineering firms. Cisco Data Center Network Manager (DCNM) is a management system for the Cisco® Unified Fabric. Also, the border leaf Layer 3 VXLAN gateway learns the host MAC address, so you need to consider the MAC address scale to avoid exceeding the scalability limits of your hardware. Traditional three-tier data center design. But most networks are not pure Layer 2 networks. Learn more about our thought leaders and innovative projects for a variety of market sectors ranging from Corporate Commercial to Housing, Pre-K – 12 to Higher Education, Healthcare to Science & Technology (including automotive, data centers and crime laboratories). Example of MSDC Layer 3 spine-and-leaf network with BGP control plane. at the time of this writing. https://www.datacenterknowledge.com/sites/datacenterknowledge.com/files/logos/DCK_footer.png, The choice of standards should be driven by the organization’s business mission, Top500: Japan’s Fugaku Still the World’s Fastest Supercomputer, Intel’s Ice Lake Chips to Enable Confidential Computing on Data Center-Grade Servers. Note that the ingress-replication feature is supported only on Cisco Nexus 9000 Series Switches. Ratings/Reliability is defined by Class 0 to 4 and certified by BICSI-trained and certified professionals. It represents the current state. Both designs provide centralized routing: that is, the Layer 3 routing functions are centralized on specific switches. Underlay IP PIM or the ingress replication feature is used to send broadcast and unknown unicast traffic. With this design, the spine switch needs to support VXLAN routing. Data Centered Architecture serves as a blueprint for designing and deploying a data center facility. However, vPC can provide only two active parallel uplinks, and so bandwidth becomes a bottleneck in a three-tier data center architecture. Best practices mean different things to different people and organizations. It is an industry-standard protocol and uses underlay IP networks. Software management tools such as DCIM (Data Center Infrastructure Management), CMMS (Computerized Maintenance Management System), EPMS (Electrical Power Monitoring System), and DMS (Document Management System) for operations and maintenance can provide a “single pane of glass” to view all required procedures, infrastructure assets, maintenance activities, and operational issues. Examples of MSDCs are large cloud service providers that host thousands of tenants, and web portal and e-commerce providers that host large distributed applications. Cisco VXLAN flood-and-learn technology complies with the IETF VXLAN standards (RFC 7348), which defined a multicast-based flood-and-learn VXLAN without a control plane. A Layer 3 function is laid on top of the Layer 2 network. Similarly, Layer 3 segmentation among VXLAN tenants is achieved by applying Layer 3 VRF technology and enforcing routing isolation among tenants by using a separate Layer 3 VNI mapped to each VRF instance. Layer 3 multitenancy example using VRF-lite, Cisco VXLAN flood-and-learn spine-and-leaf network summary. Design for external routing at the border leaf. Data centers often have multiple fiber connections to the internet provided by multiple … Each VTEP device is independently configured with this multicast group and participates in PIM routing. These are standards that guide your day-to-day processes and procedures once the data center is built: These standards will also vary based on the nature of the business and include guidelines associated with detailed operations and maintenance procedures for all of the equipment in the data center. It retains the easy-configuration, plug-and-play deployment model of a Layer 2 environment. But routed traffic needs to traverse two hops: leaf to spine and then to the default gateway on the border leaf to be routed. It is simple, flexible, and stable; it has good scalability and fast convergence characteristics; and it supports multiple parallel paths at Layer 2. Cisco FabricPath network characteristics, FabricPath (MAC-in-MAC frame encapsulation), Flood and learn plus conversational learning, Flood by FabricPath IS-IS multidestination tree. In MP-BGP EVPN, any VTEP in a VNI can be the distributed anycast gateway for end hosts in its IP subnet by supporting the same virtual gateway IP address and the virtual gateway MAC address (shown in Figure 16). Table 5. Two major design options are available: internal and external routing at a border spine, and internal and external routing at a border leaf. Please note that TRM is only supported on newer generation of Nexus 9000 switches such as Cloud Scale ASIC–based switches. VN-segments are used to provide isolation at Layer 2 for each tenant. The control-plane learns end-host Layer 2 and Layer 3 reachability information (MAC and IP addresses) and distributes this information through the EVPN address family, thus providing integrated bridging and routing in VXLAN overlay networks. It is part of the underlay Layer 3 IP network and transports the VXLAN encapsulated packets. The VXLAN flood-and-learn spine-and-leaf network doesn’t have a control plane for the overlay network. From Cisco DCNM Release 11.2, Cisco Network Insights applications are supported; these applications consist of monitoring utilities that can be added to the Data Center Network Manager (DCNM). The Layer 3 internal routed traffic is routed directly by the distributed anycast gateway on each ToR switch in a scale-out fashion. Connectivity. The IT industry and the world in general are changing at an exponential pace. A distributed anycast gateway also offers the benefit of transparent host mobility in the VXLAN overlay network. NIA constantly scans the customer’s network and provides proactive advice with a focus on maintaining availability and alerting customers about potential issues that can impact uptime. The VXLAN MP-BGP EVPN spine-and-leaf network needs to provide Layer 3 internal VXLAN routing as well as maintain connectivity with the networks that are external to the VXLAN fabric, including the campus network, WAN, and Internet. These IP addresses are exchanged between VTEPs through the static ingress replication configuration (Figure 10). Mr. Shapiro is the author of numerous technical articles and is also a speaker at many technical industry seminars. The spine switch can also be configured to send EVPN routes learned in the Layer 2 VPN EVPN address family to the IPv4 or IPv6 unicast address family and advertise them to the external routing device. Case Study: Major Retailer Uses Integration & Services for New Store Concept, © 2020 Informa USA, Inc., All rights reserved, Artificial Intelligence in Health Care: COVID-Net Aids Triage, AWS Cloud Outage Hits Customers Including Roku, Adobe, Why You Should Trust Open Source Software Security, Remote Data Center Management Tools are No Longer Optional, CloudBolt to Accelerate Hybrid Cloud Management with New Funding, What Data Center Colocation Is Today, and Why It’s Changed, Everything You Need to Know About Colocation Pricing, Dell, Switch to Build Edge Computing Infrastructure at FedEx Logistics Sites, Why Equinix Doesn't Think Its Bare Metal Service Competes With Its Cloud-Provider Customers, EN 50600-2-4 Telecommunications cabling infrastructure, EN 50600-2-6 Management and operational information systems, Uptime Institute: Operational Sustainability (with and without Tier certification), ISO 14000 - Environmental Management System, PCI – Payment Card Industry Security Standard, SOC, SAS70 & ISAE 3402 or SSAE16, FFIEC (USA) - Assurance Controls, AMS-IX – Amsterdam Internet Exchange - Data Centre Business Continuity Standard, EN50600-2-6 Management and Operational Information, Allowed HTML tags:

. However, the spine switch only needs to run the BGP-EVPN control plane and IP routing; it doesn’t need to support the VXLAN VTEP function. Cisco VXLAN MP-BGP EVPN spine-and-leaf architecture is one of the latest innovations from Cisco. Servers may talk with other servers in different subnets or talk with clients in remote branch offices over the WAN or Internet. The FabricPath spine-and-leaf network supports Layer 2 multitenancy with the VXLAN network (VN)-segment feature (Figure 8). The Tiers are compared in the table below and can b… From client-inclusive idea generation to collaborative community engagement, Shive-Hattery is grounded in the belief that design-thinking is a … For additional information, see the following references: ●      Data center overlay technologies: https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-730116.html, ●      VXLAN network with MP-BGP EVPN control plane: https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/guide-c07-734107.html, ●      Cisco Massively Scalable Data Center white paper: https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-743245.html, ●      XLAN EVPN TRM blog: https://blogs.cisco.com/datacenter/vxlan-innovations-on-the-nexus-os-part-1-of-2, View with Adobe Reader on a variety of devices, Ingress-replication is supported only on Cisco Nexus 9000 Series Switches. The FabricPath spine-and-leaf network is proprietary to Cisco, but it is mature technology and has been widely deployed. Modern Data Center Design and Architecture. Interest in overlay networks has also increased with the introduction of new encapsulation frame formats specifically built for the data center. Facility operations, maintenance, and procedures will be the final topics for the series. That traffic needs to be routed by a Layer 3 function enabled on FabricPath switches (default gateways and border switches). Telecommunication Infrastructure Standard for Data Centers: This standard is more IT cable and network oriented and has various infrastructure redundancy and reliability concepts based on the Uptime Institute’s Tier Standard. This course encompasses the basic principles of data center design, tracking its history from the early days of the mainframe to the modern enterprise data center in its many forms and the future. At the same time, it runs the normal IPv4 or IPv6 unicast routing in the tenant VRF instances with the external routing device on the outside. The VLAN has local significance on the FabricPath leaf switch, and VN-segments have global significance across the FabricPath network. After MAC-to-VTEP mapping is complete, the VTEPs forward VXLAN traffic in a unicast stream. If no oversubscription occurs between the lower-tier switches and their uplinks, then a nonblocking architecture can be achieved. Layer 3 multitenancy example with VRF-lite, Cisco FabricPath Spine-and-Leaf network summary. ), (Note: The spine switch only needs to run BGP-EVPN control plane and IP routing. Application and Virtualization Infrastructure Are Directly Linked to Data Center Design. Spine switches are performing intra-VLAN FabricPath frame switching. Layer 3 IP multicast traffic is forwarded by Layer 3 PIM-based multicast routing. ●      It enables control-plane learning of end-host Layer 2 and Layer 3 reachability information, enabling organizations to build more robust and scalable VXLAN overlay networks. In a typical VXLAN flood-and-learn spine-and-leaf network design, the leaf Top-of-Rack (ToR) switches are enabled as VTEP devices to extend the Layer 2 segments between racks. ), ●      Border spine switch for external routing, (Note: The spine switch needs to support VXLAN routing on hardware. As the number of hosts in a broadcast domain increases, the negative effects of flooding packets become more pronounced. Architecture & Design Jobs in Davenport, IA posted on Oodle. Between the aggregation routers and access switches, Spanning Tree Protocol is used to build a loop-free topology for the Layer 2 part of network. This document reviews several spine-and-leaf architecture designs that Cisco has offered in the recent past as well as current designs and those the Cisco expects to offer in the near future to address fabric requirements in the modern virtualized data center: ●      Cisco® FabricPath spine-and-leaf network, ●      Cisco VXLAN flood-and-learn spine-and-leaf network, ●      Cisco VXLAN Multiprotocol Border Gateway Protocol (MP-BGP) Ethernet Virtual Private Network (EVPN) spine-and-leaf network, ●      Cisco Massively Scalable Data Center (MSDC) Layer 3 spine-and-leaf network. It uses FabricPath MAC-in-MAC frame encapsulation. VerifythateachendsystemresolvesthevirtualgatewayMACaddressforasubnet usingthegatewayIRBaddressonthecentralgateways(spinedevices). It enables you to provision, monitor, and troubleshoot the data center network infrastructure. The FabricPath network supports up to four anycast gateways for internal VLAN routing. It is arranged as a guide for data center design, construction, and operation. To learn end-host reachability information, FabricPath switches rely on initial data-plane traffic flooding. Critical facilities are becoming more diverse as technology advances create market shifts. A good data center design should plan to automate as many of the operational functions that employees perform as possible. This section describes Cisco VXLAN flood-and-learn characteristic on these Cisco hardware switches. Two Cisco Network Insights applications are supported: ●      Cisco Network Insights - Advisor (NIA): monitors the data center network and pinpoints issues that can be addressed to maintain availability and reduce surprise outages. These IP addresses are exchanged between VTEPs through the BGP EVPN control plane or static configuration. As the number of hosts in a broadcast domain increases, the negative effects of flooding packets become more pronounced. Number 8860726. The external routing function is centralized on specific switches. Figure 17 shows a typical design using a pair of border leaf switches connected to outside routing devices. The VXLAN flood-and-learn spine-and-leaf network supports Layer 2 multitenancy (Figure 14). The Layer 3 routing function is laid on top of the Layer 2 network. settling within the mountainous site of sejong city, BEHIVE presents the ‘cloud ring’ data center for naver, the largest internet enterprise in korea. Overlay tenant Layer 3 multicast traffic is supported by two ways: (1) Layer 3 PIM-based multicast routing on an external router for Cisco Nexus 7000 Series Switches including the Cisco Nexus 7700 platform switches and Cisco Nexus 9000 Series Switches. As the number of hosts in a broadcast domain increases, it suffers the same flooding challenges as the FabricPath spine-and-leaf network. It provides rich-insights telemetry information and other advanced analytics information, etc. VXLAN uses a 24-bit segment ID, or VNID, which enables up to 16 million VXLAN segments to coexist in the same administrative domain. Don't miss what's happening in your neighborhood. The Cisco FabricPath spine-and-leaf network is proprietary to Cisco. The FabricPath network is a Layer 2 network, and Layer 3 SVIs are laid on top of the Layer 2 FabricPath switch. If the spine-and-leaf network has more than four spine switches, the Layer 2 and Layer 3 boundary needs to be distributed across the spine switches. The switch virtual interfaces (SVIs) on the spine switch are performing inter-VLAN routing for east-west internal traffic and exchange routing adjacency information with Layer 3 routed uplinks to route north-south external traffic. The FabricPath network supports up to four anycast gateways for internal VLAN routing. Regardless of the standard followed, documentation and record keeping of your operation and maintenance activities is one of the most important parts of the process. Its control plane protocol is FabricPath IS-IS, which is designed to determine FabricPath switch ID reachability information. Broadcast and unknown unicast traffic in FabricPath is flooded to all FabricPath edge ports in the VLAN or broadcast domain. With VRF-lite, the number of VLANs supported across the FabricPath network is 4096. As shown in the design for internal and external routing at the border spine in Figure 6, the spine switch functions as the Layer 2 and Layer 3 boundary and server subnet gateway. Registered in England and Wales. FabricPath is a Layer 2 network fabric technology, which allows you to easily scale the network capacity simply by adding more spine nodes and leaf nodes at Layer 2. This traffic needs to be handled efficiently, with low and predictable latency. For feature support and more information about VXLAN MP-BGP EVPN, please refer to the configuration guides, release notes, and reference documents listed at the end of this document. The data center is at the foundation of modern software technology, serving a critical role in expanding capabilities for enterprises. ), Note: Ingress replication is supported only on Cisco Nexus 9000 Series Switches. The original Layer 2 frame is encapsulated with a VXLAN header and then placed in a UDP-IP packet and transported across an IP network. The VXLAN MP-BGP EVPN spine-and-leaf architecture uses Layer 3 IP for the underlay network. As in a traditional VLAN environment, routing between VXLAN segments or from a VXLAN segment to a VLAN segment is required in many situations. Data center design and infrastructure standards can range from national codes (required), like those of the NFPA, local codes (required), like the New York State Energy Conservation Construction Code, and performance standards like the Uptime Institute’s Tier Standard (optional). The VXLAN MP-BGP EVPN spine-and-leaf architecture uses MP-BGP EVPN for the control plane for VXLAN. ●      Media controller mode: manages Cisco IP Fabric network for Media solution and helps transition from an SDI router to an IP-based infrastructure. Facility ratings are based on Availability Classes, from 1 to 4. Data center design is a relatively new field that houses a dynamic and evolving technology. Data center design with extended Layer 3 domain. Note that ingress replication is supported only on Cisco Nexus 9000 Series Switches. This series of articles will focus on the major best practices applicable across all types of data centers, including enterprise, colocation, and internet facilities. MSDCs are highly automated to deploy configurations on the devices and discover any new devices’ roles in the fabric, to monitor and troubleshoot the fabric, etc. The common designs used are internal and external routing on the spine layer, and internal and external routing on the leaf layer. Each VTEP device is independently configured with this multicast group and participates in PIM routing. The Layer 3 spine-and-leaf design intentionally does not support Layer 2 VLANs across ToR switches because it is a Layer 3 fabric. The maximum number of inter-VXLAN active-active gateways is two with an HSRP and vPC configuration. In 2013, UI requested that TIA stop using the Tier system to describe reliability levels, and TIA switched to using the word “Rated” in lieu of “Tiers,” defined as Rated 1-4. That’s the goal of Intel Rack Scale Design (Intel RSD), a blueprint for unleashing industry innovation around a common CDI-based data center architecture. The placement of a Layer 3 function in a FabricPath network needs to be carefully designed. Network overlays are virtual networks of interconnected nodes that share an underlying physical network, allowing deployment of applications that require specific network topologies without the need to modify the underlying network (Figure 5). Internal and external routed traffic needs to travel two underlay hops from the leaf VTEP to the spine switch and then to the border leaf switch to reach the external network. The multicast distribution tree for this group is built through the transport network based on the locations of participating VTEPs. The requirement to enable multicast capabilities in the underlay network presents a challenge to some organizations because they do not want to enable multicast in their data centers or WANs. Note that ingress replication is supported only on Cisco Nexus 9000 Series Switches. VXLAN, one of many available network virtualization overlay technologies, offers several advantages. TIA uses tables within the standard to easily identify the ratings for telecommunications, architectural, electrical, and mechanical systems. ●      LAN Fabric mode: provides Fabric Builder for automated VXLAN EVPN fabric underlay deployment, overlay deployment, end-to-end flow trace, alarm and troubleshooting, configuration compliance and device lifecycle management, etc. VLAN has local significance on the leaf VTEP switch, and the VNI has global significance across the VXLAN network.