Configure Dell S4048 switches for Storage Spaces Direct

When we deploy Storage Spaces Direct (S2D), either hyperconverged or disaggregated, we have to configure the networking part. Usually we work with Dell hardware to deploy Storage Spaces Direct and the one of the switches supported by the Dell reference architectures is the Dell S4048 (Force 10). In this topic, we will see how to configure this switch from scratch.

Stack or not

Usually, customers know the stack feature which is common to all network vendors such as Cisco, Dell, HP and so on. This feature enables to add several identical switches in a single configuration managed by a master switch. Because all switches share the same configuration, for the network administrators, all these switches are seen like a single one. So, the administrators connect to the master switch and then edit the configuration on all switches member of the stack.

If the stacking is sexy on the paper, there is a main issue especially with storage solution such as S2D. With S4048 stack, when you run an update, all switches reload at the same time. Because S2D highly relies on the network, your storage solution will crash. This is why the Dell reference architecture for S2D recommends to deploy a VLT (Virtual Link Trunking).

With Stacking you have a single control plane (you configure all switches from a single switch) and a single data plane in a loop free topology. In a VLT configuration, you have also a single data plane in a loop free topology but several control planes, which allow you to reboot switches one by one.

For this reason, the VLT (or MLAG) technology is the preferred way for Storage Spaces Direct.

S4048 overview

A S4048 switch has 48x 10GB/s SFP+ ports, 6x 40GB/s QSFP+ ports, a management port (1GB/s) and a serial port. The management and the serial ports are located on the back. In the below diagram, there is three kinds of connection:

• Connection for S2D (in this example from port 1 to 16, but you can connect until port 48)
• VLTi connection
• Core connection: the uplink to connect to core switches

In the below architecture schema, you can find both S4048 interconnected by using VLTi ports and several S2D nodes (hyperconverged or disaggregated, that doesn’t matter) connected to port 1 to 16. In this topic, we will configure these switches regarding this configuration.

Switches initial configuration

When you start the switch for the first time you have to configure the initial settings such as switch name, IP address and so on. Plug a serial cable from the switch to your computer and connect through Telnet with the following settings:

• Baud Rate: 115200
• No Parity
• 8 data bits
• 1 stop bit
• No flow control

Then you can run the following configuration:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34

Enable Configure   # Configure the hostname hostname SwitchName-01   # Set the IP address to the management ports, to connect to switch through IP interface ManagementEthernet 1/1 ip address 192.168.1.1/24 no shutdown exit   # Set the default gateway ip route 0.0.0.0/0 192.168.1.254/24   # Enable SSH ip ssh server enable   # Create a user and a password to connect to the switch username admin password 7 MyPassword privilege 15   # Disable Telnet through IP no ip telnet server enable Exit   # We leave enabled Rapid Spanning Tree Protocol. protocol spanning-tree rstp no disable Exit   Exit   # Write the configuration in memory Copy running-configuration startup-configuration

After this configuration is applied, you can connect to the switch through SSH. Apply the same configuration to the other switch (excepted the name and IP address).

Configure switches for RDMA (RoCEv2)

N.B: For this part we assume that you know how RoCE v2 is working, especially DCB, PFC and ETS.

Because we implement the switches for S2D, we have to configure the switches for RDMA (RDMA over Converged Ethernet v2 implementation). Don’t forget that with RoCE v2, you have to configure DCB and PFC end to end (on servers and on switches side). In this configuration, we assume that you use the Priority ID 3 for SMB traffic.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

# By default the queue value is 0 for all dot1p (QoS) traffic. We enable <a href="http://www.dell.com/support/manuals/fr/fr/frbsdt1/force10-s4810/s4810_9.7.0.0_cli_pub-v1/service-class-dynamic-dot1p?guid=guid-6bbc7b99-4dde-433c-baf2-98a614eb665e&amp;lang=en-us">this command</a> globally to change this behavivor. service-class dynamic dot1p   # Data-Center-Bridging enabled. This enable to configure Lossless and latency sensitive traffic in a Priority Flow Control (PFC) queue. dcb enable   # Provide a name to the DCB buffer threshold dcb-buffer-threshold RDMA priority 3 buffer-size 100 pause-threshold 50 resume-offset 35 exit   # Create a dcb map to configure the PFC and ETS rule (Enhanced Transmission Control) dcb-map RDMA   # For priority group 0, we allocate 50% of the bandwidth and PFC is disabled priority-group 0 bandwidth 50 pfc off   # For priority group 3, we allocate 50% of the bandwidth and PFC is enabled priority-group 3 bandwidth 50 pfc on   # Priority group 3 contains traffic with dot1p priority 3. priority-pgid 0 0 0 3 0 0 0 0   Exit   Exit Copy running-configuration startup-configuration

Repeat this configuration on the other switch.

VLT domain implementation

First of all, we have to create Port Channel with two QSFP+ ports (port 1/49 and 1/50):

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Enable Configure   # Configure the port-channel 100 (make sure it is not used) interface Port-channel 100   # Provide a description description VLTi   # Do not apply an IP address to this port channel no ip address   #Set the maximum MTU to 9216 mtu 9216   # Add port 1/49 and 1/50 channel-member fortyGigE 1/49,1/50   # Enable the port channel no shutdown   Exit   Exit Copy Running-Config Startup-Config

Repeat this configuration on the second switch Then we have to create the VLT domain and use this port-channel. Below the configuration on the first switch:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

# Configure the VLT domain 1 vlt domain 1   # Specify the port-channel number which will be used by this VLT domain peer-link port-channel 100   # Specify the IP address of the other switch back-up destination 192.168.1.2   # Specify the priority of each switch primary-priority 1   # Give an used MAC address for the VLT system-mac mac-address 00:01:02:01:02:05   # Give an ID for each switch unit-id 0   # Wait 10s before the configuration saved is applied after the switch reload or the peer link restore delay-restore 10   Exit   Exit Copy Running-Configuration Startup-Configuration

On the second switch, the configuration looks like this:

1 2 3 4 5 6 7 8 9 10 11 12

vlt domain 1 peer-link port-channel 100 back-up destination 192.168.1.1 primary-priority 2 system-mac mac-address 00:01:02:01:02:05 unit-id 1 delay-restore 10   Exit   Exit Copy Running-Configuration Startup-Configuration

No the VLT is working. You don’t have to specify VLAN ID on this link. The VLT manage itself tagged and untagged traffic.

S2D port configuration

To finish the switch configuration, we have to configure ports and VLAN for S2D nodes:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

Enable Configure Interface range Ten 1/1-1/16   # No IP address assigned to these ports no ip address   # Enable the maximum MTU to 9216 mtu 9216   # Enable the management of untagged and tagged traffic portmode hybrid   # Enable switchport Level 2 and this port is added to default VLAN to send untagged traffic. Switchport   # Configure the port to Edge-Port spanning-tree 0 portfast   # Enable BPDU guard on these port spanning-tree rstp edge-port bpduguard   #Apply the DCB policy to these port dcb-policy buffer-threshold RDMA   # Apply the DCB map to this port dcb-map RDMA   # Enable port no shutdown   Exit   Exit Copy Running-Configuration Startup-Configuration

You can copy this configuration to the other switch. Now just VLAN are missing. To create VLAN and assign to port you can run the following configuration:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Interface VLAN 10 Description "Management" Name "VLAN-10" Untagged TenGigabitEthernet 1/1-1/16 Exit   Interface VLAN 20 Description "SMB" Name "VLAN-20" tagged TenGigabitEthernet 1/1-1/16 Exit   [etc.] Exit Copy Running-Config Startup-Config

Once you have finished, copy this configuration on the second switch.