I’ve just passed a year of my job working at a smallish non-profit, and one part that I really am enjoying is passing on knowledge to the front-line staff. This week, there was an interesting case, and I had to explain to my colleagues what was happening and why. So, I did a little demo, and thought I would share it here.
While the situation may be familiar to me (and probably to many here), it was totally new to people who have never really done anything with a switch apart from plug in the blue cable.
Prior to my coming onboard, the organization was using unmanaged (and unmanageable) switches. This was less than ideal, as some traffic separation is required between staff and clients, and there is a requirement for some Layer 2 security. So I implemented Cisco 2960 LAN Lite switches. They are missing some features when compared to even the IP Base 2960s, but for the organization they are a nice balance between cost and functionality. One of the simple things I did was implement port security, and this didn’t cause any issues; in a year, I could could the number of psecure-violations on one hand. However, this week, port security threw up an interesting issue.
At a remote site, the staff moved a PC from one building to another. The layout was similar to Figure 1.
You might notice the little hub icon connected to the top switch. That was what caused the issue, and some of you might be nodding already. I didn’t know that hub was there, but I quickly sussed out what was going on and fixed it while my colleague was still getting information over the phone.
After moving the PC, it no longer worked. Of course, my colleagues wanted to know why. So this is how I explained the situation to the guys, by replicating the topology. First we took a quick look at how the switches keep track of a MAC address. No port security, no weirdness; just two switches and a PC connected through a hub.
The hub and PC is plugged into port Fa0/4 and in Figure 3, we can see the MAC address in the MAC address table.
On Switch 2, we see the MAC address on the inter-switch link, port Fa0/24 (Figure 4).
When I disconnect the PC from its hub, we wait five minutes (the default) and note that the MAC address has timed out of the MAC address tables on both switches (Figures 5 and 6).
So far, so good.
Now, I turn on port-security (Figure 7). Note that unlike the hub in the real-life situation, my test device has a MAC of its own, so I added in the maximum keyword.
The important thing to note is that the MAC address table on Switch 1 is now showing the MAC address of my PC as a STATIC entry, not DYNAMIC. Interesting.
On the remote switch, Switch 2, nothing has changed; the MAC is there as a DYNAMIC entry on the inter-switch link (Figure 8).
Now, let’s disconnect the PC from its hub and wait five minutes, replicating our remote staff unplugging the PC. On Switch 2, as before, the switch has timed out the MAC address (Figure 9).
Okay, lets have a look at Switch 1 (Figure 10). Hmm. The MAC address is still there. Check the aging time. Yep, should be gone. But no – STATIC MAC entries don’t time out.
And this is why that hub is a problem. Without bouncing the port, the port security causes the MAC address to be permanently in the MAC address table; although they are learned dynamically, port security treats them just like a static MAC you punch in yourself.
Let’s now plug in our PC to Switch 2 – replicating the remote staff moving the PC up to the other building. We see it there on port Fa0/1 on Switch 2, no problem (Figure 11).
But things are busted, because of that static MAC. Switch 1, which is where the DHCP and DNS servers are connected still thinks our PC is connected out the local port Fa0/4 (Figure 12).
Even more fun, let’s say we don’t move the PC to another building, just to another office – same switch with port-security enabled by default, different port. That’s right, we get a port-security violation (Figure 13).
So how do we fix this? Simple answer – unplug that hub and chuck it out the nearest window. Or failing that, bounce the port? Well, that will fix the problem, sure. But what if it’s not a case of an old hub lying under a desk forgotten? For example a cheap IP phone with a PC passthrough, but which is not smart enough to withdraw the MAC. Bouncing a phone which may have an active call may cause a problem.
I know – let’s clear the entry from the MAC address table (Figure 14).
We run into an immediate problem. Clearing the MAC address table only gives you the option of clearing dynamic entries. Try as you might, it isn’t going to go away. Perhaps because port-security put it there, port security can clear it?
Bingo. Clear our MAC address, plug it into the new port and away you go (Figure 15).
My colleagues now knew what had happened. But why did it happen? Surely port security should stop the hub from being there? Yes, if that hub is managed and has its own address correctly configured, meaning that it shows a MAC address. Then we’d have seen a port security alert. But this device was sitting there, lurking with only one PC attached, presenting only one MAC address. Were it a smarter switch or if someone had found it and plugged in another device, it would have shown up earlier. And in the case of a phone that may have a PC passthrough where you actually allow multiple MACs on the secure port, then this issue may legitimately crop up.
As I explained this, I wonder if anyone was thinking quietly that things were much easier with unmanaged devices. Probably. But where’s the fun in that?