When I started writing Cloud-based Software, whenever I had some issue in making an application to run on a cloud platform, I used to login to the Cloud Management Console and configure the things the way it was needed for the software and it just worked!
This is a pretty straightforward process and doesn’t seem to be a problem. But what if I needed to deploy the same software code base into a different environment – may it be production or alternatively spin a new parallel cloud stack for bench-marking, high-availability or geo-localization reasons?
There are a few problems with that.
- Treat infrastructure like cattle not like pets, destroying servers should be common practice
- Infrastructure states should be programmable and error-prone-task free
- If embracing DevOps or Agile then reduce human intervention and the limit towards zero should account only to triggers or 4-eye principles for security
- Start easy, get familiar, a tool is not a process, and the process starts by taking action
Problem 1: Do I need to remember what I changed?
Not really, of course I can document it, but will that be reliable?
Is there a standard method to document my infrastructure states and changes which can be understood by other stakeholders like developers or operations teams?
Problem 2: Do I perform this process several times?
Even If I document it (though I don’t think it’s the coolest way to do it!) what if I need to apply the same set of changes to hundreds of environment or instances?
Problem 3: How do I version control my infrastructure?
All the above points may look simple at first glance but if we think in terms of Agile and DevOps practices we can start relating to many areas like:
- Efforts and estimation for operational activities
- We will end up estimating efforts for such repeated and manual activities
- Fitting into Agile practice
- Can this approach really help us to deliver faster?
- Can this approach assure reliability?
- Can we version control our infrastructure?
- Especially in Cloud, we pay for resource usage (# of minutes, or # of instances). Simple misconfiguration can cost us for unwanted additional resources.
- Cost of additional human resources for such activities
The problems that we saw above can be summarized in below four major areas:
- Auditable Infrastructure – Changes in Infrastructure should be traceable in role and time, like git logs
- Repeatability – Repeated changes and deployment of infrastructure should minimize error-prone tasks, like a program function
- Reliability – Methods should reduce manual-entry or human intervention, a program perhaps?
- Version Controlled – infrastructure should be documented, and version controlled.
Doesn’t it sound like, Software Development? Yes, it does and that’s the exact philosophy behind doing something “as code”.
Thinking of infrastructure as code is applying software engineering approaches to operational functions, which primarily are functions of operation engineers. But it doesn’t mean it is not a function of the development engineering team.
Once we start looking into this operational function we start getting benefits what software development had for years, like source code control, version control, test, deployment and many more and that’s exactly what we talk about – IaC [Infrastructure As Code]
What is IaC – Infrastructure as Code?
In the simplest terms, “Infrastructure as Code” is a concept which let you manage your infrastructure environment same way you maintain your software code for releases.
Instead of making manual changes to your Infrastructure configuration, you follow the same rules as software development code.
This means, now the changes in my environment go through the same process as the software code changes. I will make changes in infrastructure code, commit to a git repository, review and merge. This commit may trigger some CI/CD pipeline and run tests and can also deploy infrastructure changes into my cloud environment.
What IaC is not?
IaC is not just infrastructure automation, but IaC is a concept that extends beyond simple infrastructure automation which requires applying DevOps practices to automation scripts to ensure they’re error free, are able to be redeployed on multiple servers, can be rolled back in case of problems, and can be engaged by both operations and development teams.
There are many tools available to achieve IaC in complex cloud infrastructure architectures. We can classify them into two categories.
- Configuration Orchestration
- Configuration Management
Configuration orchestration tools, which include Terraform, AWS CloudFormation, Google Cloud deployment manager and Azure Resource Manager are designed to automate the deployment of servers and other infrastructure.
Configuration management tools like Chef, Puppet, and the others help configure the software and systems on this infrastructure that has already been provisioned.
Configuration orchestration tools do some level of configuration management, and configuration management tools do some level of orchestration. It’s not uncommon to use a mix of these tools to achieve your IaC plan to excel the DevOps practices.
Let’s look at some of the tools useful for IaC
Terraform is an Infrastructure provisioning tool from HashiCorp, which let you write infrastructure specifications in its own domain-specific language (DSL) called Hashicorp Configuration Language (HCL). Terraform also allows writing.Terraform Plugins which helps to build your reusable modules.
Terraform is cloud agnostic and can be used irrespective of your cloud provider.
- AWS Cloud Formation
Like Terraform, AWS Cloud Formation is configuration orchestration from AWS. This works only with AWS Cloud. It allows you to write AWS Cloud Formation templates using YAML or JSON format.Similarly, Microsoft provides a similar solution called Azure Resource Manager which is tightly integrated with their Azure Cloud Platform, and Google provides Google Cloud Deployment Manager on their own Google Cloud Platform.
Any cloud provider who offers API, can be powered by Terraform, as long as there is developer community to create “provider” for Terraform.
Chef is one of the most popular configuration management tools which allows you to create “recipes” and “cookbooks” using its Ruby-based DSL. These recipes and cookbooks specify the exact steps needed to achieve the desired configuration of your applications and utilities on existing servers. This is called a “procedural” approach to configuration management, as you describe the procedure necessary to get your desired state.
Similar to Chef, Puppet is another popular configuration management tool that helps teams to continuously deliver software.
Using Puppet’s Ruby-based DSL, you can define the desired end state of your infrastructure and exactly what you want it to do. Then Puppet automatically enforces the desired state and fixes any incorrect changes.
This “declarative” approach – where you declare what you want your configuration to look like, and then Puppet figures out how to get there – is the primary difference between Puppet and Chef.
Also, Puppet is mainly directed toward system administrators, while Chef primarily targets developers.
Puppet integrates with the leading cloud providers like AWS, Azure, Google Cloud, and VMware, allowing you to automate across multiple clouds.
Ansible is an infrastructure automation tool by Red Hat.Ansible models the infrastructure by describing how the components and systems relate to one another, as opposed to managing systems independently.
Ansible is agent-less unlike Puppet and Chef and its code is written in YAML in the form of Ansible Playbooks. These playbooks can make it really easy to understand and deploy.
Ansible’s functionality can be extended by writing custom Ansible modules and plugins.
Docker is not really a tool specifically for IaC but it is a critical part of IaC approach nowadays. It helps to create containers that package your code and dependencies together. This makes your software containerized and ready to ship to any environment, from your local workstation to any cloud platform.
Docker makes it easy to build reliably and automate some of the tasks which otherwise would have been tedious.
Getting your hands dirty
Comparing all these tools and discussing best-fit use cases for each is beyond the scope of this article. However if you are planning to automate your infrastructure and looking for the best tool, it’s generally a best approach to look at each tool’s capabilities and consider if they offer you what you are looking for.
I have observed some teams debating about one tool versus the another for weeks and never getting started. I would argue that you can’t be sure about the tool unless you start using it.
It is most likely along the way you would want to mix one tool with another to achieve what you want. In short choose the tool which offers the simplest way to get moving.
It is exciting to work with Cloud Infrastructure using such tools and its even more fun when it comes to setting it up in your CI/CD pipelines! I am planning for my next article more on the similar line, but If you have any suggestion, please do post your comments below!
Meanwhile, it is worth to take a look at IaC best practices here.