As one of the least discussed frontiers of network automation, automated testing has always been a gray area in my mind. For the tiny fraction of enterprises that are doing it today, I vaguely pictured a collection of one-off scripts and playbooks, but never quite saw the big picture of how this would all come together in an automated fashion. Let’s face it, it’s hard enough getting the masses to buy into pushing configurations in an automated way, let alone tackling unit and integration testing. However, as any meticulous network engineer that has done complex changes will tell you, you’re only as good as your post change check-out plan. In an effort to demystify this area and address some of my knowledge gaps, I stumbled upon Robot Framework and decided to kick the tires.
Data Driven Workflow
Even though “keyword driven” and “abstractions” seemed like the main selling points in the documentation, I wanted to start off as simple and explicit as possible in order to grasp the syntax and workflow. For this purpose, I settled on using the Robot Framework SSHLibrary to interact with a Cumulus VX switch image. The idea is to create a simple ping test case, that will succeed assuming the ping itself did. We’ll create a file named “network_checkout.robot“:
Starting from the top, we have 4 main sections. Our “Settings” section indicates the library we’re looking to use (this can be an existing library or your own custom one via .py script). The “Suite Setup” and “Suite Teardown” actions are what will take place when this workflow is executed. The “Suite Setup” calls the “Open Connection And Log In” keyword that we define below, while the “Close All Connections” is a known keyword that is self explanatory in this case. The “Variables” section defines 3 variables that we will use to connect to our network device. The “Keywords” section defines the aforementioned “Open Connection And Log In” keyword, which utilizes the variables we defined above it and is called in the “Settings” section, whenever our workflow is executed. Finally, we have our actual “Test Cases” section where we define a command we want to run on a remote host and gather whether it succeeded or not.
At a quick glance after running our workflow, we see that our test passed, so we’re now ready for additional test cases.
As you can see, we’re sticking to the same logic of command output parsing for the simplicity of this example, but adding various layer 2 and layer 3 tests.
Executing our latest group of tests, you can see that the first three passed but the last one failed. We were looking to match on an Established BGP state, but instead encountered an Idle one.
Now that we have a base understanding of how these workflows function, you might be thinking to yourself that “I already have this functionality available in my favorite language or tool, what is Robot Framework actually bringing to the table?” With that question in mind, let’s dive a little deeper into the abstractions that the tool offers.
Keyword Driven Workflow
Let’s go ahead and take our previous workflow and separate it into two files. Our variables and keywords will go into a file named “resources.robot“, while a second file will contain our test cases that will abstract all the technical details simply by calling the “resources.robot” file.
One main difference you’ll notice, is that the “Test Cases” from our previous data driven workflow, have now become “Keywords” that we can reference in any “Test Cases” outside of this file. If another workflow separate from ours wanted to call any of these keywords, it would have the ability to do so without worrying about details like credentials and generally what’s under the hood.
We simply end up referencing our “resource.robot” file in our new “network_checkout.robot” file and now have abstracted access into our tests.
After correcting the BGP issue that previously caused our test to fail, we can now run our new workflow and see that our Layer 2/3 Tests succeeded.
Cross Team Testing
To be honest with you, even after implementing the keyword driven workflows, I was left questioning the significance of adding another tool to accomplish similar tasks to what the existing popular automation options already offer. The “aha” moment came to me after I realized what this tool and its workflows would look like in an environment where multiple teams and silos are all using it for automated testing. Network specific pre and post change testing has always suffered from not having visibility into the applications those changes affected. Even with recent efforts into simulating end to end application testing to address this exact need, the ability to do so in a 1-for-1 manner is not quite there. However, if those same teams are already using and relying on the framework, I can just reference their existing test cases in my pre and post change testing with the comfort that they’re using these exact same tests (without the complexities of what those tests entail):
As network teams slowly become warmer to the idea of CI/CD, tools like Robot Framework (or ones similar to it) will become integral in those pipelines. So to end off with a DevOps reference, walk over to your application team and ask them what they’re using for automated testing? You’d be surprised by what comes out of that conversation.