How to Estimate Cloud Costs with Terraform (Azure, AWS, GCP, etc.) via Azure DevOps Pipelines

In the tech world we’re living in, it feels like everyone’s racing to push their systems and apps into the cloud, aiming to slash costs on stuff like maintenance, operational expenses, staffing, and beefing up security. But, it’s kinda funny (and a bit ironic) how often you bump into folks who end up shocked by their cloud bills, to the point where some even beat a retreat back to the cozy confines of their on-prem setups.

Digging into these stories, you usually find a common thread: a lot of these companies jumped into the cloud without really mapping out their journey based on what the cloud gurus or the big cloud providers recommend. It’s like setting sail without a compass; they end up lost at sea, clueless about where their virtual machines or databases have drifted off to. This chaos mostly boils down to shooting from the hip with their cloud setup and not really planning or working together properly.

Now, here’s a bit of a plot twist: when you bring Terraform into the picture to manage your cloud infrastructure the smart way, a lot of these headaches start to fade. But (and it’s a big ‘but’), even with Terraform in your toolkit, those sneaky cost issues can still pop up. Nobody’s perfect, right? But in the DevOps spirit, we’re all about tackling problems head-on and finding ways through the maze.

So, imagine you’re in the thick of managing your cloud setup with Terraform, and out of the blue, your boss swings by with a “Hey, we need to trim the fat on our cloud spending and keep a closer eye on it moving forward.” That’s where this neat little tool called InfraCost comes into play.

Let’s take a look at it and see how to integrate this tool to automate this FinOps process in Azure DevOps Pipelines.

Implementation of InfraCost in Azure DevOps Pipelines:

1. Acquiring an API Key from InfraCost:

• Installation is straightforward across various operating systems. For macOS users, for instance, the tool can be installed using Homebrew:
• For other OS, please see the Install Infracost

brew install infracost

infracost --version # Should show 0.10.33

To upgrade Infracost, run brew update then brew upgrade infracost.

• Register for a free API key to utilize the Cloud Pricing API for price retrieval, ensuring no cloud credentials are exposed.
• Authenticate your CLI with the API key via:

infracost auth login

The key can be retrieved with infracost configure get api_key or from the UI.

- Go to ORG SETTINGS

Once we retrieve our API key, it’s time to implement this tool to our Terraform pipeline.

2. Integrating InfraCost into Azure DevOps:

You might request to get this tool in your organization, once the Admin approves you are good to use it!

Infracost Azure DevOps Task Marketplace

3. Configuring an InfraCost Stage in Your Pipeline:

Customize your pipeline with InfraCost tasks to automate cost estimation and monitoring. This involves setup tasks, generating baseline cost estimates, and diff reports, followed by updating pull requests with cost insights.

- stage: Infra_Cost
    displayName: Infra Cost Prediction
    jobs:
      - job: Infra_Cost
        displayName: Infra Cost Prediction
        steps
        
# Install the Infracost CLI
        - task: InfracostSetup@2
          inputs:
            apiKey: $(infracostApiKey) #You can use your API key from pipeline variables or KV in variable group.
            version: '0.10.x'
            currency: 'CAD'
      
# Clone the base branch of the pull request (e.g. main/master) into a temp directory.
        - bash: |
            git clone https://$(PAT)@dev.azure.com/<organization>/<project>/_git/<repository> --branch=$(System.PullRequest.TargetBranchName) --single-branch /tmp/base
          displayName: Checkout base branch

# Generate an Infracost cost estimate baseline
        - bash: |
            cd /tmp/base #I go into the directory where my PR branch cloned in
            infracost breakdown --path . \
              --format=json \
              --out-file=/tmp/infracost-base.json
          displayName: Generate Infracost cost estimate baseline
        
# Generate an Infracost diff and save it to a JSON file.
        - bash: |
            infracost diff --path=. \
              --format=json \
              --compare-to=/tmp/infracost-base.json \
              --out-file=/tmp/infracost.json
          displayName: Generate Infracost diff
# Add a cost estimate comment to a Azure Repos pull request.
        - bash: |
            infracost comment azure-repos \
              --path=/tmp/infracost.json \
              --azure-access-token=$(PAT) \ #You can use your PAT token from variables or KV in variable group.
              --pull-request=$(System.PullRequest.PullRequestId) \
              --repo-url=$(Build.Repository.Uri) \
              --behavior=update
          displayName: Post PR comment
        
# The following job is needed when using Infracost Cloud
        - bash: |
            PATTERN="Merged PR ([0-9]+):"
            if [[ "$(Build.SourceVersionMessage)" =~ $PATTERN ]]; then
              PR_ID=${BASH_REMATCH[1]}
              echo "Updating status of $PR_ID"
              curl \
                --request POST \
                --header "Content-Type: application/json" \
                --header "X-API-Key: $(infracostApiKey)" \
                --data "{ \"query\": \"mutation {updatePullRequestStatus( url: \\\"$(Build.Repository.Uri)/pullrequest/${PR_ID}\\\", status: MERGED )}\" }" \
                "https://dashboard.api.infracost.io/graphql";
            else
              echo "Nothing to do as the commit message did not contain a merged PR ID."
            fi
          displayName: 'Update PR status in Infracost Cloud'
        
        - bash: |
            infracost breakdown \
              --path=. \
              --format=json \
              --out-file=/tmp/infracost.json
            infracost upload --path=/tmp/infracost.json || echo "Always pass main branch runs even if there are policy failures"
          displayName: 'Run Infracost on default branch and update Infracost Cloud'

Notes:

• In the Infracost CLI Installation task:

apiKey: $(infracostApiKey) #You can use your API key from pipeline variables or KV in variable group. Make sure this is private and not exposed in your pipeline

• In the Generate Infracost cost estimate baseline task:

cd /tmp/base : I go into the directory where my PR branch cloned in

• In the Post PR comment task:

--azure-access-token=$(PAT) \ #You can use your PAT token from variables or KV in variable group. Make sure this is private and not exposed in your pipeline

• In the Checkout Base Branch task, make sure you have your variable values secretly:

git clone https://$(PAT)@dev.azure.com/<organization>/<project>/_git/<repository> --branch=$(System.PullRequest.TargetBranchName) --single-branch /tmp/base

or

• You can use this:

git clone $(Build.Repository.Uri) --branch=$(System.PullRequest.TargetBranchName) --single-branch /tmp/base

4. Enforcing Branch Policy for Pull Request Comments:

• Ensure the Azure Pipelines service has permission to contribute to pull requests, enabling automated cost feedback on code changes.

• Enable pull request build triggers. Without this, Azure Pipelines do not trigger builds with the pull request ID, thus comments cannot be posted by the integration.
• Go to your repository from the left panel where your pipeline resides in
• Click on Branches and click on the 3 dot next to main branch and select “Branch policies”

• Scroll to Build Validation and click + sign to add one if you don’t have one already

Select your Terraform Pipeline and specify path filters if applicable to do so

5. Testing the Workflow

Add your new resource into your Terraform code on your own branch. Mine is a tiny AKS cluster LOL. Then create a pull request against main branch

• As the branch policy is set, pull request will be tied up to successful completion of our Terraform pipeline.

When take a look at my pipeline, I see that after my plan stage I wanna see my cost prediction before deployment

RESULT

• I see that Infra Cost Prediction job is done and should be updated my PR with the information about cost.

When I check my PR, I see that lovely automatically generated cost table shows me my estimated cost :)

PS: This basically compares your own PR branch with your main branch and predicts costs.

OK, I’ll pay $281 = 20 SHAWARMA ($14 each) = for the new resources LOL

• You may wonder, what would cost this much? Youcan see that by clicking Cost details:

If everything satisfies, good to deploy :)

6. Reviewing Infracost Cloud for Detailed Insights

For a comprehensive analysis, the Infracost Cloud platform offers an extensive dashboard detailing your infrastructure costs.

Conclusion

Diving into cloud cost management with Terraform and InfraCost has been enlightening. InfraCost, in particular, has been a standout, not just for its pre-deployment cost estimates but also for its ability to align with FinOps policies and compliance checks. It’s like having a financial compass for navigating cloud expenses, ensuring we’re always on the right path.

I’m curious about your experiences with InfraCost or any other strategies you’ve employed to keep cloud costs under control. How have FinOps policies and compliance checks impacted your projects? Your insights and stories are invaluable, and I’m looking forward to exchanging ideas and learning from each other.

Let’s connect and share our journey through the cloud’s financial landscape. Your feedback and experiences are highly welcome!

Thanks for reading!