Configuring Cumulus Linux High Availability Layer 2 Network (Part 1)

Hi everyone, today I just would like to share a lab configurations on one of the validated design from Cumulus Linux,  which is the Data Center Layer 2 High Availability. Cumulus Linux is a specialized network-focused Linux distribution based on Debian that run on a bare-metal network switch.

The specific configuration will include:

  • Layer 2 interface (access port and trunk port) and VLAN-aware bridge
  • Bonding – Link Aggregation (EtherChannel)
  • Multi-Chassis Link Aggregation (MLAG)
  • Layer  3 interface (routed port)  and Switch Virtual Interface (SVI)
  • Virtual Router Redundancy Protocol (VRRP)
  • Routing Protocol OSPF (Quagga)

Network Topology

Our topology in this case as Cumulus Linux called Traditional Layer 2 Hierarchical Enterprise Data Center Network Pod. On the actual deployment, the number of links between the leaf and spine switches may vary depending on your data center requirements. The example in the topology on the right where the uplink ports in the leaf switch have a dual connection to each spine switch to increase resiliency and performance in the network or if required you may add more.

Building the lab

For this lab, we’re going to use GNS3 and Cumulux VX (a virtual appliance version of Cumulus Linux) to build a proof of concept can be downloaded at a free cost.

Create an account in Cumulus Linux so you can able to download the Cumulus VX. At the download page just select a qcow2 format in order we can use it in GNS3. After that, download the Cumulus VX appliance template from GNS3 Marketplace and load it in GNS3.

1 – Configure MLAG (spine swicthes)

First, we’ll configure MLAG to form our spine-01 and 02 into one logical switch. From the leaf-01 and leaf-02 point of view, they connected to just one switch. One of the benefits of using MLAG is to eliminate blocking ports because of the default behaviour of Spanning Tree Protocol (STP is layer 2 loop prevention mechanism). By implementing MLAG, all the uplink ports to spine switches are all in forwarding mode. With the same scenario, we can implement MLAG on leaf switches to form one logical switch, and from server and spine switch perspective, it connects to just one leaf switch (see our topology ).

spine-mlag-1
MLAG

spine-01:

#/etc/network/interfaces

# MLAG peerlink
auto peerlink #logical name of the interface
iface peerlink 
 bond-slaves swp1 swp2 #the ports where the to peering switch connect

# MLAG sub interface peerlink
auto peerlink.4094 #VLAN subinterface of peerlink
iface peerlink.4094
 address 169.254.1.1/24 #unique address of the link. Cumulus recommend 
 #unrouteable link-local address which start from 169.254/16, except for 
 #169.254.0/24 that BGP IP unnumbered used. This is address used by clagd 
 #services to communicate between peering switch.
 clagd-peer-ip 169.254.1.2 #the unique link-local address of peering switch.
 clagd-sys-mac 44:38:39:ff:90:01 #this is a virtual mac address of the peer 
 #switch and advertised it as the bridge ID. This must be the same with 
 #peering switch.

spine-02:

# MLAG peerlink
auto peerlink #logical name of the interface
iface peerlink
 bond-slaves swp1 swp2 #the ports where the to peering switch connect

#MLAG subinterface peerlink
auto peerlink.4094 #VLAN subinterface of the peerlink
iface peerlink.4094
 address 169.254.1.2/24 
 clagd-peer-ip 169.254.1.1
 clagd-sys-mac 44:38:39:ff:90:01

Other attributes under peerlink.4096 such as:

  • clagd-backup-ip –  This will be used by clagd service if ever the peerlink will fail, where it use to communicate with peering switch to check its status. Cumulus recommends using the management IP address of the switch which is reachable by route. Also, recommend it in a production network.
  • clagd-priority –  This determines which switch will become the primary or secondary, by default the switch with the lowest mac address will become the primary.

Note: The peerlink subinterface which is peerlink.4094 also known as VLAN subinterface. This VLAN will not be included in VLAN-aware bridge and STP. Cumulus recommends a higher VLAN like VLAN 4094 in this example.

Verification:

Below is the sample output from spine-01 using a clagctl command. Notice the 32768 that is the default priority followed by the mac address of the bond interface, in this case, the peerlink interface. The system mac is the one we specify in the configuration file, and if you include the back-up IP it will show also in clagctl command.

spine-01-clagctl.PNG

2 – Configure Trunk port

Next, we will configure the downlink ports connected to leaf switches.

downlink01
Downlink01
downlink02.PNG
Downlink02

spine-01:

#/etc/network/interfaces

#downlink to leaf-1 and leaf-02
auto downlink01 #logical name of the interface.
iface downlink01
 bond-slaves swp5 swp6 #the ports where the leaf switches connect.
 clag-id 1 #a number that identify which interfaces/bond the leaf switches 
 #connected on the peering switch. This must be same with peering switch.

#downlink to leaf-03 and leaf-04
auto downlink02
iface downlink02
 bond-slaves swp3 swp4
 clag-id 2

spine-02:

#/etc/network/interfaces

#downlink to leaf-01 and leaf-02
auto downlink01 #logical name of the interface.
iface downlink01
 bond-slaves swp3 swp4 #the ports where the leaf switches connect.
 clag-id 1 #a number that identify which interfaces/bond the leaf switches 
 #connected on the peering switch. This must be same with peering switch.

#downlink to leaf-03 and leaf-04
auto downlink02
iface downlink02
 bond-slaves swp5 swp6
 clag-id 2

Verification:

Below is the sample output from spine-01. Notice the CLAG interfaces, that is the ports where the leaf and spine connected each other.

spine-01-clagctl-02.PNG

3 – Configure Bridge

Next, we will configure a VLAN bridge. If you came from Cisco background you can refer to this site here, it is a comparison of common layer 2 command between Cumulus and Cisco.

spine-01:

#/etc/network/interfaces
auto brvlan #< logical name of the bridge
iface brvlan
 bridge-vlan-aware yes #< a vlan-aware bridge mode
 bridge-ports peerlink downlink01 downlink02 #< list of logical or physical
 #ports where VLAN will run. It may a trunk or an access port.
 bridge-vids 10 20 30 #< list of vlan that will run on this bridge.
 bridge-pvid 1 #< also known as native VLAN and by default it is VLAN 1. 
 #If you want to use the different native VLAN, you to configure your own using 
 #this attribute.
 bridge-stp on #< turn on STP. In vlan-aware mode, it only operate in RSTP mode.
 mstpctl-treeprio 4096 #< this determine which bridge will become the root 
 #bridge. The lowest priority will become the root bridge.

spine-02:

#/etc/network/interfaces
auto brvlan
iface brvlan
 bridge-vlan-aware yes
 bridge-ports peerlink downlink01 downlink02 
 bridge-stp on
 bridge-vids 10 20 30 
 bridge-pvid 1
 mstpctl-treeprio 4096

Verification:

Below is the sample output that shows the bridge information, in this case, the brvlan and the trunk port, which is the peerlink, downlink01, and downlink02.

spine-01-netshow-01

4 – Configure MLAG (leaf switches)

Next, we’ll configure our leaf switches: The configuration mostly identical in terms of MLAG, bridge VLAN and bond uplink ports to spine switches.

leaf-01:

#/etc/network/interfaces
# MLAG peerlink
auto peerlink
iface peerlink
 bond-slaves swp1 swp2

# MLAG subinterface peerlink
auto peerlink.4094
iface peerlink.4094
 address 169.254.1.3/24
 clagd-peer-ip 169.254.1.4
 clagd-sys-mac 44:38:39:ff:90:02

#uplink to spine-01 and spine-02
auto uplink01 
iface uplink01 
 bond-slaves swp8 swp9
 clag-id 12

auto brvlan
iface brvlan
 bridge-vlan-aware yes
 bridge-ports peerlink uplink01
 bridge-stp on
 bridge-vids 10 20 30
 bridge-pvid 1
 mstpctl-treeprio 8192

leaf-02:

#/etc/network/interfaces

# MLAG peerlink
auto peerlink
iface peerlink
 bond-slaves swp1 swp2

# MLAG subinterface peerlink
auto peerlink.4094
iface peerlink.4094
 address 169.254.1.4/24
 clagd-peer-ip 169.254.1.3
 clagd-sys-mac 44:38:39:ff:90:02

#uplink to spine-01 and spine-02
auto uplink01 
iface uplink01 
 bond-slaves swp8 swp9
 clag-id 12

auto brvlan
iface brvlan
 bridge-vlan-aware yes
 bridge-ports peerlink uplink01
 bridge-stp on
 bridge-vids 10 20 30
 bridge-pvid 1
 mstpctl-treeprio 8192

Verification:

Below is the sample output from leaf-01 to verify the MLAG, bond interface, trunk, and bridge.

leaf-01-config

Note: In leaf03 and 04 they basically have the same configuration as leaf01 and 02.

In the validated design, there is two option to connect to the core layer. One is the layer 2, and other is layer 3. But in this lab, we will configure a layer 3 option to connect to the core. In this case, we need our spine switches to provide a routing and gateway within the cluster.

In the next part, we will configure our spine switches with the SVI and VRRP to provide routing between our VLAN.

Reference: Multi-Chassis Link Aggregation, Data Center Layer 2 HA Validated Design

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s