In Part 1 of our look into the feasibility of infrastructure as code for the network stack, we laid the foundation for device provisioning. Now comes the challenge of operating our newly built or converted infrastructure and incorporating it into our change control processes.
Day 1 & Beyond: Daily Operations & Change Management
Version Control : Now that we have a basic idea of how new devices will be brought onto the network, lets commit all of our Ansible files to a git repository. It’s important to note that this repository will become our central source of truth when it comes to device state, further helping us integrate peer review and change control into the process. To put it plainly, our intention is to have engineers push their changes solely through git, in our goal to move away from the cli and ad-hoc changes.
Once our files are in a repository under version control, we’d like to introduce a GIT front end (Bitbucket) to help us better manage our repository. A front end is technically not required, but is of great help in terms of visualizing config changes, pull request approvals, etc.
Notice that we’re looking at the master branch in this view. Additionally, we’re going to create a development branch, with the intention of directly pushing changes to only that branch. Once a change is pushed to the development branch, a user will create a pull request to merge that change into the master branch. That pull request will be subject to peer approval. The main advantage to this multi branch approach is to give engineers freedom to make multiple iterations of a given change before submitting it to the master branch for approval.
To ensure that no one can push a change directly to the master branch and bypass peer review, we’ll prevent that ability. We’ll also make sure that every pull request will need to be approved.
Deployment : Now that our version control and approval process is accounted for, the last piece of the puzzle will be Jenkins, the system that will schedule and push our actual changes. The idea here is to have a tool that will automatically push the changes we’ve committed to git. By not running one-off scripts/playbooks and specifying new target devices for every single change, we’re utilizing a consistent and reliable process of introducing changes into our environment.
In Jenkins, we’ll create a new project/pipeline that will have visibility into our git branch as well as our Ansible playbook.
Once Jenkins has the ability to pull your intended changes from git and push them via Ansible, the only thing left is to decide how and when those changes should be pushed. We have the ability to get rather creative about how “continuous” we really want our CI/CD pipeline to be, based on our comfort level with the process. For the sake of this exercise, we’ll set Jenkins to poll git for changes once a minute.
Now that all of our components are in place, let’s bring it all together with an example change. We’re going to add a tertiary NTP address of 220.127.116.11. This can either be done from the git shell or via the Bitbucket web interface.
Now we’ll commit that change to git.
Once it is committed, we’ll submit a pull request to merge this change into our master branch.
At this point a peer will be notified of this pull request, go to review exactly what is changing and approve it.
Jenkins will see that the master branch of git has changed and automatically run the Ansible playbook to execute said change.
To verify, let’s check the Console Output section and see exactly what took place behind the scenes.
To recap and drive home an important point, instead of relying on separate scripts per change, we instead run the same Ansible playbook regardless of what is changing, which ends up only making the necessary changes needed to achieve our desired state config. As already mentioned, the uniformity and consistency of this process goes a long way in building confidence in our CI/CD model. In Part 3, we’ll conclude our exercise and discuss the many takeaways and concerns of operating your network infrastructure in this manner.