written by Nathan Bennet
Code updates can be very frustrating. Having spent time on the operations side of a code deploy, it is always difficult for an engineer to perform updates without the developer handy. Then, the question of who performs the code updates arises. The operations team would protest that the developers needed to do the code updates because operations did not understand the code. The developers would always respond with “You’re right, so give us access!” or worse, “We control the code, you get it where it needs to go and maintain it.” Both answers present challenges, and the operations team gets it in the end, “Well if nothing else, give it to the ops guys, they’ll handle it.”
DevOps is a melding of development and operations. These two groups were warring factions in the IT community, fighting over who did what, and who had access to what. One process that came from DevOps is known as CI/CD. The “C” in CI/CD stands for Continuous.
It starts with the “CI” meaning “Continuous Integration.” This is a development practice where developers are continually pushing changes to code to their Version Control System (VCS). The VCS is sometimes referred to as the “Single Source of Truth,” which refers to the main codebase by which all other code will be judged against. This development process allows users to “push” their changes to the same VCS without stepping on each other’s toes. DevOps uses an added tool, or a built-in feature, that controls the “build” and “push” of the code to the production boxes. This is the “CD” part of the process referred to as, “Continuous Delivery.” Code delivery is applied to legacy applications, where the code for the solution goes through a gated process followed by the interruption. “CD” can also stand for “Code Deployment” that does not require a gated process to push the code. Instead, a commit to the codebase equals a commit to production. This part of CI/CD can build a virtual “Test” of the code base, compile it, verify its good, and send it to all production servers. This CI/CD process (Figure 1.0) must supply a solution to the challenges discussed in the first paragraph. Let us revisit these challenges.
Challenge one: Who controls the Code?
The challenge is simple. The Operations team works on multiple solutions to support the up-time, management, and delivery of the environment for the day-to-day. Adding code management and control to their responsibilities is a big jump for most operations teams as illustrated in figure 2.0
Another challenge is having your code testing environment mirror production. Developers will code the script and code for the test environment, only to find out once it is pushed, something is different in the production environment leading to code failure. A simple solution to the question of control comes from an earlier employer who said, “If you build the code you control the code.”
Developers are always working in their IDE or “Integrated Development Environment” to test and compile their functions. After the mechanics of those functions check out, the code is pushed to the repository. Then, it is tested against the development/test environments to ensure the functionality is good and has the means to be verified for production via a request and Change-Advisory Board (CAB), Some companies refer to this as a Pull Request (PR). Once approved, the code is then ready for delivery. The question of who controls the code does not end here. Differences in the test/development/production environment can create another obstacle; the code that is already in production. This presents an exceedingly complicated issue as developers are now out of the loop (they are not part of the push to production). Operations must figure out what has changed in the current production environment and report back. As bad as that sounds, it is even worse when your codebase is not up to date at this point, as the code in production does not reflect the code in the repository. Hopefully, by now you see the challenge that invariably can lead to a huge mess.
Challenge Two: Access
There are two ways to control access in an operational mindset, whether to give it to developers or not? What happens if you give access to the developers? Having always been an advocate for helping developers with their job, as servers with no apps rarely make money. For this reason, granting specific rights to the developers makes complete sense. However, this does lead to security and management teams, and even developer managers to get involved with managing stability and phrases like, “time to release”. Once developers move into the server plane, they start asking the same questions that have already been asked and answered by the ops team. This duplication of efforts causes inefficiency and loss of time. This is especially true in the environment surrounding their production servers. See figure 3.0 shows the breakdown in communication.
On the other hand, when Operations says, “we will handle it,” the control of code is then broken, and the Operations team moves into the Developers world. This is less invasive than the other as Operations only controls one function (See figure 2). To a Developer, it is the same. The cycle of controlling the code to production, changing of code, communication trials, etc., it all leads back to the same mess. The code in the repository will not perfectly match what is running in the production environment. The Operations team takes on too much and is blamed for bad code rolls. Thus, the divide between Development and Operations grows as each side points fingers, creating more disconnect between the Development and Operations teams.
Moving to Continuous Integration
Finally, our hero enters; Continuous Integration (CI) is an amazing tool designed to solve these challenges. First, starting on the Operations side, the CI tool is granted access to both the Development and the Operations side of the environment. This ensures the proper delivery of the code to the environment it is intended for. CI can even run checks on canary runs to verify if the code is good before deploying. This can save a ton in rollback and interruption time. From its roots in mining, the term “Canary runs” refers to test runs. Miners would take canaries into mines. If there was harmful gas or anything that could harm the miners, the canary would be a warning to the miners to get out. This same principle applies to the code. When a canary test is run, a user (or virtual test) can test the code and report whether the code is genuinely good for production to the developer.
If your development and production environments are different, the process of using the CI tool will shine a light on that because you can code errors. That will dictate what Operations will need to change to continue the code roll. This also helps Operations, as they live in a mindset that dictates, “If it’s not broke, don’t fix it.” Operations will not touch anything unless a specific notation shows an error to support a stringent CAB policy that would take weeks to adjust to production.
Moving past the subject of finding solutions to the challenges that plague the Development and Operations teams, going to a Continuous Integration/Continuous Delivery tool-set grants even more functionality to the Development team. The acronym uses the word “continuous” or “constant” to show how your repository for your code is always attached to the environment it has targeted. This means that, as soon as the code is approved for your “Master” or “Production” branch, it is at once pushed to the production environment. This can insight fear into those who worry about the interruption. Again, that goes to the CI/CD tool you use. You can set the tool to run against the environment at the exact time you approve it, but you can also schedule the deployment for a maintenance window that would not interfere with daily operations or cause a stoppage. All of this would be hashed out during the PR (Pull Request) discussion.
By finding and recognizing the challenges that face and divide Development and Operations teams, using a CI/CD toolset, with a single source of truth (Code Repository) can resolve these challenges easily. In a world of division, it is nice to have resources and solutions that will help create harmony in the workplace for a better outcome.