Hi everyone, today I just would like to share a lab configurations on one of the validated design from Cumulus Linux, which is the Data Center Layer 2 High Availability. Cumulus Linux is a specialized network-focused Linux distribution based on Debian that run on a bare-metal network switch.
The specific configuration will include:
- Layer 2 interface (access port and trunk port) and VLAN-aware bridge
- Bonding – Link Aggregation (EtherChannel)
- Multi-Chassis Link Aggregation (MLAG)
- Layer 3 interface (routed port) and Switch Virtual Interface (SVI)
- Virtual Router Redundancy Protocol (VRRP)
- Routing Protocol OSPF (Quagga)
Our topology in this case as Cumulus Linux called Traditional Layer 2 Hierarchical Enterprise Data Center Network Pod. On the actual deployment, the number of links between the leaf and spine switches may vary depending on your data centre requirements. The example in the topology on the right where the uplink ports in the leaf switch have a dual connection to each spine switch to increase resiliency and performance in the network or if required you may add more.
Building the lab
Create an account in Cumulus Linux so you can able to download the Cumulus VX. At the download page just select a qcow2 format so that we can use it in GNS3. After that, download the Cumulus VX appliance template from GNS3 Marketplace and load it in GNS3.
1 – Configure MLAG (spine switches)
First, we’ll configure MLAG to form our spine-01 and 02 into one logical switch. From the leaf-01 and leaf-02 point of view, they connected to just one switch. One of the benefits of using MLAG is to eliminate blocking ports because of the default behaviour of Spanning Tree Protocol (STP is a Layer 2 loop prevention mechanism). By implementing MLAG, all the uplink ports to spine switches are all in forwarding mode. With the same scenario, we can implement MLAG on leaf switches to form one logical switch, and from the server and spine switch perspective, it connects to just one leaf switch (see our topology ).
#/etc/network/interfaces # MLAG peerlink auto peerlink #logical name of the interface iface peerlink bond-slaves swp1 swp2 #the ports where the to peering switch connect # MLAG sub interface peerlink auto peerlink.4094 #VLAN subinterface of peerlink iface peerlink.4094 address 169.254.1.1/24 #unique address of the link. Cumulus recommend #unrouteable link-local address which start from 169.254/16, except for #169.254.0/24 that BGP IP unnumbered used. This is address used by clagd #services to communicate between peering switch. clagd-peer-ip 169.254.1.2 #the unique link-local address of peering switch. clagd-sys-mac 44:38:39:ff:90:01 #this is a virtual mac address of the peer #switch and advertised it as the bridge ID. This must be the same with #peering switch.
# MLAG peerlink auto peerlink #logical name of the interface iface peerlink bond-slaves swp1 swp2 #the ports where the to peering switch connect #MLAG subinterface peerlink auto peerlink.4094 #VLAN subinterface of the peerlink iface peerlink.4094 address 169.254.1.2/24 clagd-peer-ip 169.254.1.1 clagd-sys-mac 44:38:39:ff:90:01
Other attributes under peerlink.4096 such as:
- clagd-backup-ip – This will be used by clagd service if ever the peerlink will fail, where it use to communicate with peering switch to check its status. Cumulus recommends using the management IP address of the switch which is reachable by the route. Also, recommend it in a production network.
- clagd-priority – This determines which switch will become the primary or secondary, by default the switch with the lowest mac address will become the primary.
Note: The peerlink subinterface which is peerlink.4094 also known as VLAN subinterface. This VLAN will not be included in VLAN-aware bridge and STP. Cumulus recommends a higher VLAN like VLAN 4094 in this example.
Below is the sample output from spine-01 using a clagctl command. Notice the 32768 that is the default priority followed by the mac address of the bond interface, in this case, the peerlink interface. The system mac is the one we specify in the configuration file, and if you include the back-up IP it will show also in clagctl command.
2 – Configure Trunk port
Next, we will configure the downlink ports connected to leaf switches.
#/etc/network/interfaces #downlink to leaf-1 and leaf-02 auto downlink01 #logical name of the interface. iface downlink01 bond-slaves swp5 swp6 #the ports where the leaf switches connect. clag-id 1 #a number that identify which interfaces/bond the leaf switches #connected on the peering switch. This must be same with peering switch. #downlink to leaf-03 and leaf-04 auto downlink02 iface downlink02 bond-slaves swp3 swp4 clag-id 2
#/etc/network/interfaces #downlink to leaf-01 and leaf-02 auto downlink01 #logical name of the interface. iface downlink01 bond-slaves swp3 swp4 #the ports where the leaf switches connect. clag-id 1 #a number that identify which interfaces/bond the leaf switches #connected on the peering switch. This must be same with peering switch. #downlink to leaf-03 and leaf-04 auto downlink02 iface downlink02 bond-slaves swp5 swp6 clag-id 2
Below is the sample output from spine-01. Notice the CLAG interfaces, that is the ports where the leaf and spine connected each other.
3 – Configure Bridge
Next, we will configure a VLAN bridge. If you came from Cisco background you can refer to this site here, it is a comparison of common layer 2 command between Cumulus and Cisco.
#/etc/network/interfaces auto brvlan #< logical name of the bridge iface brvlan bridge-vlan-aware yes #< a vlan-aware bridge mode bridge-ports peerlink downlink01 downlink02 #< list of logical or physical #ports where VLAN will run. It may a trunk or an access port. bridge-vids 10 20 30 #< list of vlan that will run on this bridge. bridge-pvid 1 #< also known as native VLAN and by default it is VLAN 1. #If you want to use the different native VLAN, you to configure your own using #this attribute. bridge-stp on #< turn on STP. In vlan-aware mode, it only operate in RSTP mode. mstpctl-treeprio 4096 #< this determine which bridge will become the root #bridge. The lowest priority will become the root bridge.
#/etc/network/interfaces auto brvlan iface brvlan bridge-vlan-aware yes bridge-ports peerlink downlink01 downlink02 bridge-stp on bridge-vids 10 20 30 bridge-pvid 1 mstpctl-treeprio 4096
Below is the sample output that shows the bridge information, in this case, the brvlan and the trunk port, which is the peerlink, downlink01, and downlink02.
4 – Configure MLAG (leaf switches)
Next, we’ll configure our leaf switches: The configuration mostly identical in terms of MLAG, bridge VLAN and bond uplink ports to spine switches.
#/etc/network/interfaces # MLAG peerlink auto peerlink iface peerlink bond-slaves swp1 swp2 # MLAG subinterface peerlink auto peerlink.4094 iface peerlink.4094 address 169.254.1.3/24 clagd-peer-ip 169.254.1.4 clagd-sys-mac 44:38:39:ff:90:02 #uplink to spine-01 and spine-02 auto uplink01 iface uplink01 bond-slaves swp8 swp9 clag-id 12 auto brvlan iface brvlan bridge-vlan-aware yes bridge-ports peerlink uplink01 bridge-stp on bridge-vids 10 20 30 bridge-pvid 1 mstpctl-treeprio 8192
#/etc/network/interfaces # MLAG peerlink auto peerlink iface peerlink bond-slaves swp1 swp2 # MLAG subinterface peerlink auto peerlink.4094 iface peerlink.4094 address 169.254.1.4/24 clagd-peer-ip 169.254.1.3 clagd-sys-mac 44:38:39:ff:90:02 #uplink to spine-01 and spine-02 auto uplink01 iface uplink01 bond-slaves swp8 swp9 clag-id 12 auto brvlan iface brvlan bridge-vlan-aware yes bridge-ports peerlink uplink01 bridge-stp on bridge-vids 10 20 30 bridge-pvid 1 mstpctl-treeprio 8192
Below is the sample output from leaf-01 to verify the MLAG, bond interface, trunk, and bridge.
Note: In leaf03 and 04 they basically have the same configuration as leaf01 and 02.
In the validated design, there is two option to connect to the core layer. One is layer 2, and the other is layer 3. But in this lab, we will configure a layer 3 option to connect to the core. In this case, we need our spine switches to provide routing and gateway within the cluster.
In the next part, we will configure our spine switches with the SVI and VRRP to provide routing between our VLAN.