Ruckus : Using Multiple RADIUS Servers

I have recently been configuring Network Access Control with 802.1x, and I had been having issues with using multiple RADIUS servers on Ruckus ICX Switches. The main issue being:

RADIUS Authenticate over SSH to switch using Microsoft NPS RADIUS Server
RADIUS Authenticate using 802.1x or MAC-Auth using DOT1x RADIUS Server

In Ruckus ICX switches there isn’t any concept of AAA groups like in Cisco, where you can designate specific RADIUS traffic to go to various different RADIUS Servers.

I have found 2 work arounds, I did however also call support and spend 1 hour troubleshooting with them and they didn’t have an answer for me.

Some Basic Setup Information

Microsoft NPS RADIUS Server : 1.1.1.1
DOT1x RADIUS Server : 1.1.1.2

Here are my AAA Authentication Commands:

SSH@tpw-sw1# sh run | inc authentication
aaa authentication web-server default radius local
aaa authentication enable default radius local
aaa authentication dot1x default radius
aaa authentication login default radius local

Here are my 2 work arounds:

WORKAROUND 1

SSH@tpw-sw1(config)# radius-server host 1.1.1.1 auth-port 1812 acct-port 1813 default key RADIUS1SECRET
SSH@tpw-sw1(config)# radius-server host 1.1.1.2 auth-port 1812 acct-port 1813 default key RADIUS2SECRET dot1x mac-auth

If you use the 1.1.1.2 RADIUS server first in the list you cannot authenticate to the switch at all, even over super-user-password. So the only way I have it working is to have the DOT1x Radius Server listed 2nd but calling out DOT1x and MAC-AUTH.

WORKAROUND 2

The other method I found is to apply a command at the interface level:

SSH@tpw-sw1# conf t
SSH@tpw-sw1(config)# int ethernet 1/1/1
SSH@tandy-lab-sw1(config-if-e1000-1/1/1)#use-radius-server 1.1.1.2

I hope that this helps, I spent a day trying to figure it out 🙂

 

 

The Packet Wizard : Migrating from Cisco 6500 to Ruckus ICX

Just a quick post this week, I have been busy migrating from Cisco 6500 to Ruckus ICX. Here are some before and after photos and a video of the all important turn off. The main thing I learned in this migration is to chose your ports that are different, do Trunk Ports, Wireless, Printers, anything that is unique or requires a slightly different configuration do them first, then the regular desktop/user ports are just easy swaps.

The before picture we had already started to move the patch panels.

Listen to that power noise drop when it turns off. Turning off Cisco 6500 after Migration

Cisco Switch : Private VLAN’s

Private VLAN’s are a very interesting and mostly used for Network segmentation and fun concept but it can take a little to get your head around, so here goes.

Private VLAN’s split a VLAN into Sub-VLANs, called Primary and Secondary.  Secondary VLAN’s have 2 different types :  Isolated and Community.

In this example the Primary VLAN is 100 and the Secondary VLAN’s are Isolated VLAN 200, Community VLAN 300 and Community VLAN 400.

An important port to know about before beginning is called the Promiscuous Port. It acts like a Gateway that routes Primary and Secondary-VLAN traffic, and all Secondary-VLAN traffic must pass through the Promiscuous Port.

Isolated ports can only talk to the Primary VLAN through a Promiscuous Port (Uplink/Gateway Port)

Community ports can talk to each other, if they are in the same Community Secondary-VLAN.

VTP must be set to transparent mode for Private VLAN’s to work.

Here is how to configure Private VLAN’s

First we need to configure the Primary VLAN

TPW-SW1(config)#vlan 100
TPW-SW1(config-vlan)#private-vlan primary 
TPW-SW1(config-vlan)#exit

Configure the Isolated VLAN

TPW-SW1(config)#vlan 200
TPW-SW1(config-vlan)#private-vlan isolated 
TPW-SW1(config-vlan)#exit

Configure the Community VLAN’s

TPW-SW1(config)#vlan 300
TPW-SW1(config-vlan)#private-vlan community 
TPW-SW1(config-vlan)#exit

TPW-SW1(config)#vlan 400
TPW-SW1(config-vlan)#private-vlan community 
TPW-SW1(config-vlan)#exit

Now we have to associate the Primary VLAN to the Isolated and Community VLAN’s

TPW-SW1(config)#vlan 100 
TPW-SW1(config-vlan)#private-vlan association 200 
TPW-SW1(config-vlan)#private-vlan association 300
TPW-SW1(config-vlan)#private-vlan association 400

This is where we configure for fa0/1 as the Promiscuous Port

TPW-SW1(config-if)# int fa0/1 
TPW-SW1(config-if)#switchport mode private-vlan promiscuous

We have to tell the Promiscuous Port that it is associated with the  (Isolated and Community VLAN’s) that it can also see and talk to them appropriately.

TPW-SW1(config-if)#switchport private-vlan host-association 100 200,300,400
TPW-SW1(config-if)#exit

Configure fa0/2 and fa0/7 as the Isolated port, but also about its Primary VLAN 100

TPW-SW1(config-if)# int fa0/2 
TPW-SW1(config-if)#switchport mode private-vlan host
TPW-SW1(config-if)#switchport private-vlan host-association 100 200
TPW-SW1(config-if)#exit

TPW-SW1(config-if)# int fa0/7 
TPW-SW1(config-if)#switchport mode private-vlan host 
TPW-SW1(config-if)#switchport private-vlan host-association 100 200 
TPW-SW1(config-if)#exit

Configure fa0/3 and 4 as community ports, but also about its Primary VLAN 100

TPW-SW1(config)#int range fa0/3 - 4 
TPW-SW1(config-if-range)# 
TPW-SW1(config-if-range)# switchport mode private-vlan host
TPW-SW1(config-if-range)# switchport private-vlan host-association 100 300
TPW-SW1(config-if-range)# exit

Configure fa0/5 and 6 as community ports, but also about its Primary VLAN 100

TPW-SW1(config)#int range fa0/5 - 6 
TPW-SW1(config-if-range)# 
TPW-SW1(config-if-range)# switchport mode private-vlan host
TPW-SW1(config-if-range)# switchport private-vlan host-association 100 400
TPW-SW1(config-if-range)# exit

You can confirm the Private VLAN’s are setup correctly with the following show command

TPW-SW1#show vlan private-vlan

Primary      Secondary    Type                Ports
-------      ---------    -----------------   ----------------------------------
100          200          isolated            fa0/2, fa0/7
100          300          community           fa0/3, fa0/4
100          400          community           fa0/5, fa0/6

Here is the topology of what was just built.

Here is a table of what can talk to each other

PC

Computer PC1 – Isolated – VLAN 200 PC2 – Isolated – VLAN 200 PC3 – Community VLAN 300 PC4 – Community VLAN 300 PC5 – Community VLAN 400 PC6 – Community VLAN 400
PC1 – Isolated – VLAN 200 YES NO NO NO NO NO
PC2 – Isolated – VLAN 200 NO YES NO NO NO NO
PC3 – Community VLAN 300 NO NO YES YES NO NO
PC4 – Community VLAN 300 NO NO YES YES NO NO
PC5 – Community VLAN 400 NO NO NO NO YES YES
PC6 – Community VLAN 300 NO NO NO NO YES YES

 

Cisco Switch : Dynamic ARP Inspection

Dynamic ARP inspection, starts with a man in the middle attack, with the rogue user and PC sending a gratuitous ARP to say the MAC address for your default gateway is this, then the user accessing anything using the default gateway is sending all its traffic to the rogue users pc, which is then copying all the packets before sending it on to its destination.

 

DHCP snooping must be enabled, for DAI to be enabled.

 

Here is how to configure DAI:

tpw-sw1(conf)# ip arp inspection vlan 10
tpw-sw1(conf)# int gi0/1
tpw-sw1(config-int)#ip arp inspection trust


How to Check and Verify DAI:

tpw-sw1# show ip arp inspection

Cisco Switch : Storm Control

Storm Control can shut down an interface that is causing a Unicast, Multicast or Broadcast Storm. It works with Rising and Falling Thresholds. If a specific type of traffic hits the rising Threshold it will block that type of traffic until the number of packets it see’s is below the set falling threshold. This can be in Percentage, Bits Per Second or Packets Per Second

Here is how to enable its various options:

tpw-sw1(conf)# int gi0/1
tpw-sw1(config-int)# storm-control broadcast level 10 5 - this command means if I see more than 10% broadcast traffic I will block until I see it fall under 5%
tpw-sw1(conf)# int gi0/1
tpw-sw1(config-int)# storm-control multicast level bps 20m 10m - this command means if I see more than 20Mbps multicast traffic I will block until I see it fall under 10Mbps
tpw-sw1(conf)# int gi0/1
tpw-sw1(config-int)# storm-control unicast level pps 50k  - this command means if I see more than 50,000 unicast packets per second, I will drop traffic until I see it fall below 50,000 packers per second, that is what happens if you do not put a falling threshold. Both rising and falling threshold is 50,000
tpw-sw1(conf)# int gi0/1
tpw-sw1(config-int)# storm-control action shutdown

Check and Verify

tpw-sw1(conf)# show storm-control broadcast

tpw-sw1(conf)# show storm-control multicast

tpw-sw1(conf)# show storm-control unicast

Cisco Switch : DHCP Snooping

DHCP seems like a seemingly innocent, but common protocol, that can be used against our network. Since we know the DHCP discovery packet is a broadcast packet, just looking for a DHCP server and the host doesn’t care what DHCP server sends a DHCP OFFER back, it will accept the first offer, the DHCP offer includes information such as IP address, Subnet Mask, Default Gateway, DNS information. What if the first offer that is returned is a Rouge or Malicious DHCP server? Does that mean all traffic from that host using the Rogue DHCP servers gateway could be looking at all of the traffic passing through it? Yes! We can prevent this from happening with a feature called DHCP Snooping.

DHCP Snooping is going to snoop or listen into DHCP traffic to make sure that DHCP conversations go to the correct interface and allow that traffic to pass, otherwise it will be dropped. The interfaces to a known good DHCP server will be ‘trusted’ and all other interfaces will be untrusted, therefor the switch will know if DHCP conversations are happening on an untrusted interface then the traffic will be dropped and the interface will be put into err-disabled mode.

By default the switch considers all ports untrusted. We have to enable DHCP snooping globally, then trust at the interface level. IP ARP inspection and IP source-guard are dependent on DHCP snooping being enabled.

Enabled DHCP Snooping

tpw-sw1(config)#ip dhcp snooping

 

Enable DHCP Snooping on a VLAN

tpw-sw1(config)#ip dhcp snooping vlan 10

 

Trust Interface with DHCP server on it

tpw-sw1(config)#int gigabitEthernet 1/1

tpw-sw1(config-if)#ip dhcp snooping ?

  information  DHCP Snooping information

  limit        DHCP Snooping limit

  trust        DHCP Snooping trust config



tpw-sw1(config-if)#ip dhcp snooping trust

 

DHCP option 82

When packets come in on an untrusted port with option 82 set, those packets are not dropped. The switch will insert its own DHCP option 82 information (the switches MAC address), and when the packet is returned it will make sure its own DHCP option 82 information is in the reply if it is, it will remove its option 82 information and forward the packet normally, if not it will drop it.  This check is enabled by default.

Turn off the validity check

tpw-sw1(config)#no ip dhcp relay information check

 

Turn on Option 82

tpw-sw1(config)#ip dhcp snooping information option

 

Show Commands

tpw-sw1#sh ip dhcp snooping

Switch DHCP snooping is enabled

DHCP snooping is configured on following VLANs:

10

DHCP snooping is operational on following VLANs:

10

DHCP snooping is configured on the following L3 Interfaces:

Insertion of option 82 is enabled

Option 82 on untrusted port is not allowed

Verification of hwaddr field is enabled

Verification of giaddr field is enabled

DHCP snooping trust/rate is configured on the following Interfaces:



Interface                    Trusted    Relay Info policy     Rate limit (pps)

------------------------     -------    -----------------     ----------------

GigabitEthernet1/1        yes                               unlimited

 

Cisco : Port Security

Is this frame from a trusted Source? That is the question!

As I continue my CCNP studies, I near the end of the CCNP Switch Course, this is a topic that was covered in the CCNA, and I haven’t really touched it since, so here is a refresher.

A port enabled with port security will expect to see frames sourced from a particular MAC address or group of MAC addresses, if the switch receives a frame from a MAC address it does not have listed the port takes action according to the violation mode that is set. The violation mode by default is to shutdown the port.

Enable Port Security

*You can only enable port security on a edge port, so that means that port can never become a trunk port*

tpw-sw1(conf)# int gi0/1
tpw-sw1(config-int)#switchport mode access
tpw-sw1(config-int)#switchport access vlan 10
tpw-sw1(config-int)#switchport port-security

Confirm it is working and information gathering

tpw-sw1# show port-security
tpw-sw1# show port-security address
tpw-sw1# show port-security interface gi0/1

More Port Security options

tpw-sw1(conf)# int gi0/1
tpw-sw1(config-int) switchport port-security ?
          aging - Port-security aging commands
          mac-address  - Secure mac address
          maximum - Max secure addresses
          violation - Security violation mode
tpw-sw1(config-int) switchport port-security maximum <1-6144>
tpw-sw1(config-int) switchport port-security violation ?
          protect - Security violation protect mode (drops the offending frames )
          restrict - Security violation restrict mode (drops, creates log messages, but port remains active )
          shutdown - Security violation shutdown mode (default will put the port into err-disabled (fix problem first and shut, no shut))
tpw-sw1(config-int) switchport port-security mac-address ?
         H.H.H - 48 bit mac address
         sticky - Configure dynamic secure addresses as sticky (dynamic addresses but kept on switch reload )
tpw-sw1(config-int) switchport port-security aging ?
         time Port-security aging time
         type Port-security aging type

This is just the very basics of Port Security, there is similar implementations on all switches I have worked with.

Arista : VARP Configuration

Virtual-ARP or VARP is a routing technique that allows multiple switches or routers to simultaneously route packets from a common Virtual IP (VIP) address in an active/active switch/router configuration. Each switch or router is configured with the same VIP address on the corresponding VLAN interfaces (SVI) and a common virtual MAC address. In MLAG topologies, VARP is preferred over VRRP because VARP does not require traffic to traverse the peer-link to the master router as VRRP would.

A maximum of 500 VIP addresses can be assigned to a single VLAN interface. All virtual addresses on all VLAN interfaces resolve to the same virtual MAC address. However you cannot have a secondary VIP on the same VLAN interface, you can however implement VRRP on the same VLAN interface as VARP.

VARP functions by having each switch respond to ARP and GARP requests for the configured router IP address with the virtual MAC address. The virtual MAC address is only for inbound packets and never used in the source field of outbound packets.

The following commands configures 10.10.10.1 as the virtual IP address for VLAN 10. The Virtual-Router MAC address is entirely invented by you, I had a real issue finding clarification that it was just a made up MAC address, so here is my invented made up Virtual-Router MAC 1010.1010.1010 as the virtual MAC address on both switches. I also ran into an issue where #ip routing had to be enabled.

Here is what the Topology would look like:

Configuration that implements VARP on the first switch

TPW-SW1(config)#ip virtual-router mac-address 1010.1010.1010

TPW-SW1(config)#interface vlan 10

TPW-SW1(config-if-vl10)#ip address 10.10.10.2/24

TPW-SW1(config-if-vl10)#ip virtual-router address 10.10.10.1

Configuration that implements VARP on the second switch

TPW-SW2(config)#ip virtual-router mac-address 1010.1010.1010

TPW-SW2(config)#interface vlan 10

TPW-SW2(config-if-vl10)#ip address 10.10.10.3/24

TPW-SW2(config-if-vl10)#ip virtual-router address 10.10.10.1

 

The Packet Wizard : Spanning Tree Explained

Spanning Tree Protocol also known as STP

There are many different types of STP but here are a couple of the main ones

STP/802.1D – Original STP
PVST+ – Cisco Improved STP adding per VLAN feature
RSTP/802.1w – Improved STP with a much faster convergence time (Rapid Spanning Tree)
Rapid PVST+ – Cisco improved RSTP adding per VLAN feature

Why Per VLAN STP?
If you have a large network with lots of switches and VLAN’s you can use Per VLAN STP to plan for a more efficient network

Even although there are many versions of STP they all use a very similar set of rules.

What is STP?

STP is a feature used to prevent loops when you are using redundant switches and without STP a loop could form and cause a number of problems on the network.

During a unicast broadcast message (which happen all the time) the switch will forward the frame out of every port except the one it came in on. Therefore if SW1 sends a frame out and SW2 and SW3 receive it then SW2 and SW3 will forward out all ports except the one it came in on.  SW2 sends to SW3 and SW1. SW3 send to SW2 and SW1 and you can see how the loop is now beginning to form. This is known as a broadcast storm, this can kill a switches CPU and Memory usage very quickly.

The second problem is the MAC address being changed all the time as it receives frames. For example SW1 sends a broadcast message, SW2 and SW3 receive it, then forward it out all other ports like in the scenario above. However each switch learns the MAC address of the next switch and assigns that in the MAC address table, but if you consider SW1 sending to SW2 and SW3 and then SW2 and SW3 forwarding those frames and they eventually get back to SW1 but on different ports, then the MAC Address table will change constantly from I know about SW2 on this port,  I now know about SW2 via SW3 on this port, and that can cause unstable MAC address tables.

Another issues is explained below

HOST1  sends data to HOST2, however since SW2 doesn’t know how to get to SW2 it sends frames out all ports, thus sending to SW1 and SW3 so HOST2 receives frames from HOST1 via SW3 and then again via SW1>SW3. This is known as Duplicate Frames.

So how do we fix the issues mentioned above? Thats right Spanning Tree Protocol by blocking one of the redundant paths.

The question now becomes how do the switches decide on that Port to block? STP follow’s strict rules, when deciding what ports to block. 

1) Elect a Root Bridge (ROOT)
2) Place root interfaces into forwarding (FWD)
3) Select Root Port on non-Root Bridge Switches (RP) – this is the best root to the Root Bridge.
4) Non Root Switches decide on a Designated Port (DP)
5) All other ports put into Blocking State (BLK)

On per VLAN STP You could have this on VLAN 10

and this on VLAN 20

I will now cover the port roles and the port states so you know what each is:

ROLES
Root Ports : The best port to get to the Root Bridge

Designated Ports : The Lowest cost alternate best root to the Root Bridge.
Non Designated Ports : All other ports that are in blocking mode.

STATES
Disabled : A Port is shutdown
Blocking : A Port that is blocking traffic
Listening : A Port that is not forwarding and not learning MAC addresses
Learning: A Port that is learning MAC addresses but is not forwarding traffic
Forwarding : A Port that is sending and receiving traffic as normal

When ports change from one Role to another it will go through the Port States. Note also that the Listening and Learning states are transitional and it wont stay on either.

Root Bridge Election

Each switch has and sends messages to each other called Bridge Protocol Data Units (BPDU’s) These BPDU’s contain specific information pertaining to each switch, such as Root Cost, Bridge ID (BID) for Itself and for the Root.  A BID is made up of STP Priority and MAC address, the default value of The BID on SW1 would be 327691111:1111:1111 since 32769 is the default STP priority and the MAC address. The switch with the lowest BID will become the Root Bridge. This is what is looks like before the Root Bridge Election and the exchange of the BPDU’s

This is what it looks like after, when the lowest BID wins.

The ports on each switch now transition into their respective states following the STP Rules as mentioned above.

The ports can change based on the Cost of each link. The port costs are listed below, however in this example we will just be using Gig Ports, but for clarity a FastEthernet Port will be slower than a GigEthernetPort, the faster the port the lower the cost. The Root Port (RP) is the lowest port cost.

Data rate STP cost RSTP cost
(Link Bandwidth) (802.1D-1998) (802.1W-2004, default value)
4 Mbit/s 250 5,000,000
10 Mbit/s 100 2,000,000
16 Mbit/s 62 1,250,000
100 Mbit/s 19 200,000
1 Gbit/s 4 20,000
2 Gbit/s 3 10,000
10 Gbit/s 2 2,000
100 Gbit/s N/A 200
1 Tbit/s N/A 20

This is a quick diagram of how the port costs are worked out to get back to the Root Bridge. SW2 to get to SW1 is 0+4=4 and SW2 via SW3 to SW1 is 4+4=8

Of course there can be ties between multiple connections and STP can be tuned.

Designated Ports are selected by Root Cost the by Lowest BID and then by lowest numbered Interface. Therefor in the diagram above the Designated port would be GigEth1 on SW3 since it is a lower numbered interface than SW2 GigEth2.

All ports that are not Root Ports or Designated Ports are Blocking Ports.

STP Convergence Times

STP:
BPDU/Hello time = 2 secs – Hello messages to each switch to see its still there
Max Age = 20 secs – How long a switch will wait for a response to the Hello message
Listening = 15 secs
Learning = 15 secs

= 52 secs to convergence

From the time a link goes down to convergence it takes a total of 52 Seconds. When STP was designed that was fine but now, this is much too slow which is where Rapid Spanning Tree Comes in.

RSTP:
3 missed BDPU/Hello at 2 sec each = 6 secs
Learning (no listening) = 15 secs

= 21 secs to convergence.

I hope this have given you a good explanation of STP. 

 

The Packet Wizard : DHCP Troubleshooting

In todays scenario, I am going to walk through some changes I made and troubleshooting steps for when I recently added a moved a old SSID/Subnet off an old legacy wireless network onto a new network same IP space and SSID that requires RADIUS authentication.

These steps can be applied to many different scenarios for troubleshooting DHCP, I just made these ones specific since it was something I recently had to troubleshoot.

Here is a basic diagram of the setup, showing all the moving parts would be overkill for the diagram. The steps on what to do and troubleshooting are below the diagram.

What you will need:

Authentication Server IP

Authentication Secret Key

DHCP Server IP

Subnet and Mask that is being moved

SSID/Subnet being moved

Work and or Troubleshooting that needs to be done:

  1. Add the VLAN to the switches required
  2. Add the virtual interface on the firewall (gateway)
  3. Trunk the new vlan to the switch and configure the ports
  4. Setup DHCP helper to point to the DHCP server
  5. Allow DHCP traffic from the new subnet to the DHCP server
  6. Configure Radius on new Network
  7. Configure new SSID and network settings on Wireless LAN Controller