Cisco Nexus switches are running Linux under the hood for a while, but until Nexus 9K and updated code for Nexus 3K it was not easily accessible. It has changed in recent versions though and now you can have all the power of Linux, if you wanted it of course.
How do I access Linux on Cisco Nexus?
There are 2 main ways you can access Linux on Cisco Nexus 9K and 3K: 1) guest shell and 2) run bash. The documentation about Nexus 7K is a bit confusing so I can’t tell for sure whether bash or guest shell are available for it. And I unfortunately don’t have one to test it out.
Guest shell is a Linux container with CentOS 7 running on top of underlying OS. It acts pretty much just like a regular container, the only main difference I ran into was a chvrf command, which allows you to change your current VRF. You can find more details about it here: https://developer.cisco.com/docs/nx-os/#!guest-shell/-transaction-summary
You can also get into the bash shell of the underlying OS on Nexus switch, however it is NOT a regular Linux distribution and generally speaking unless you are troubleshooting some issue specifically with NX-OS, you should not be there. More details about the environment could be found here: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/programmability/guide/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Configuration_Guide_chapter_01000.html#concept_29EEE45363D14F70BAEFFC3779F6F110
Why would I want to do anything in Linux container on a switch?
That is a very valid question. Generally speaking, there should be no need to run any software on top of switches, as you generally want to let your switches do what they are best at – forward packets and let other systems do the network management. However I can see some use cases for troubleshooting and network monitoring – for example, running Thousand Eyes probe from a switch may be very useful at times. There’s another operation where having access to Linux is very useful – initial build of environment.
At my current work, our product is deployed in a very small footprint – 3-6 racks of gear. But we deploy lots of those small environments, a couple of hundred a year. Different teams are responsible for building network and servers and generally servers need internet access to be build fully, so it’s a requirement to have network up first. Being able to run everything on the switches without having a dependency on other teams is a great way to save time and simplify process.
So what can you run on Cisco Nexus?
Essential part of initial provisioning. There’s already a great guide available, not much else to add to it: https://github.com/ndelecro/nx-os-programmability/tree/master/Guest_Shell/PXE_Server
While Python in the guestshell is a bit old (2.7.5), you still can do pretty much everything you need with it. You could also install python3, if you are willing to give guest shell sufficient space. There’s also python package manager – pip – which you can use to install anything you want.
In my case, there is a small issue though – in order for me to get internet access from the switches, I have to configure Firewalls. And to configure firewalls, I need to install some python modules. Luckily, pip in guestshell is modern enough that you can install everything you need from files. Here’s how you do it:
First step: download the files of packages that you need
pip download f5-sdk -d packages/
This will download all the necessary files (including dependencies) for the f5-sdk package and put them in the packages subfolder. Note that packages and their dependencies may be platform-specific. You can try and specify it with switches “–platform=manylinux1_x86_64 –only-binary=:all:” for pip command, but I have found that I am very limited in which versions of packages that I can download in this fashion from my Mac. I have opted to use a Nexus 9000V with internet access in order to download packages, but I would imagine that since container runs regular CentOS, any CentOS host should be able to do the same just fine.
Second step: get the files to the switch you need to install them on. SCP or USB are your friends here.
Third step: install the packages from the directory
pip install f5-sdk --no-index --find-links file:packages/
This will install the package f5-sdk and look for all the necessary packages in the current subfolder packages/
Now you can use them with any of your regular python scripts and they will work just fine.
But wait, there’s more
Pip can install many different packages, including one of the more powerful tools – Ansible. Yes, you can install it exactly like any other package, just make sure to increase size of rootfs in order to fit it. There are some deprecation warnings that may require you to be more careful with some package versions in the future or may require some python upgrade, but at this moment it works just fine. And yes, you can run all of your playbooks and they will work just fine, albeit a little slower than from a real Linux host.
With the combination of DHCP server and Ansible, I am able to build pretty much full network stack without relying on any other systems, which does make life a lot easier.
Currently I am running NX-OS code 7, which only supports guest shell. In NX-OS code 9, there is a Docker support. With proper container support, I should be able to further simplify my provisioning process, as lots of manual steps would be taken care of.
My biggest concern with Docker (as well as current guest shell) is the effect of this container on performance of the rest of the system. I use it just for build process and shut it down once I am done with it, as I can’t say I fully trust it to run on my production infrastructure for an extended period of life. I hope over time more people will be running extra features on their switches and we can all get more confident in that process.