The Story Thus Far...
For the past 3 years, after landing my first "DevOps Engineer" position, just year after college. I have been asking the following question "How do I become more developer minded?" or "What exactly am I supposed to develop?" I was fortunate or rather unlucky depending on how you see it, to have landed a devops role after doing a few months of Linux Support and another few months as an internal Sysadmin. Needless to say, with the guidance of my team from Reliant Solutions (Shoutout to the senior engineers that really pushed and guided me..) I had a workable background with Systems Administration tasks, but the Dev in "DevOps" kept whispering in one ear: "Imposter."
Alright, so what does this story of imposter syndrome have to do with Terratest and automated testing? You see I have been with 3 startups before my time at Square-Enix. With the exception of Enigma Technologies (which we will touch upon later..) Our infrastructure did not have any automated tests or checks. Despite working with some brilliant people, testing wasn't really a thing people automated for the infrastructure (and if I'm honest it's not something I thought was possible.) When I joined Engima, and saw how they performed integration tests, I was thrilled at how mant of the terraform issues we were facing at Quovo (and Plaid by acquisition.) were resolved by treating the infrastructure as application code, and running a series of linting and integration tests before merging. The only problem was... our AWS environment got real messy if devs forgot to run the "Delete Pipeline" job in gitlab; and trust me.. they forgot. Often.
Fast-forward to Square-Enix, I was missing the saftey net of deploying infrastructure code and have been yearning to start building tools for Kubernetes. What better way to kill two birds with one stone, than writing infrastructure tests? After hours of putting my sysadmin google skills to use, I stumbled upon Terratest.
Terratest is a Go library that provides patterns and functions for testing infrastructure code. So after watching the introduction video and going through the examples, I decided to dust off "The Go Programming Language" book and purchase a udemy course to dive in.
Although met with initial resistance due to GoLang being foreign to the team, people have started to see the benefits of some proof of concepts.
With story time out of the way let's jump into a basic Docker test, that you can hopefully use to add to your organization and help guide your growth.
Testing Your Containerized Apps
Let's say you want to reduce the number of bad images being pushed out to your cloud repository. Testing your container will probably look something like this...
docker build -t nshipman-io/devwhoops .docker run -p 8080:8080 -d nshipman-io/devwhoops:latestdocker pscurl http://localhost:8080/healthcheck
This process may take you around 5+ minutes to verify your changes have not broken the newly built docker image. Instead, we can automate this process with the following code:
package test
import (
"crypto/tls"
"github.com/gruntwork-io/terratest/modules/docker"
http_helper "github.com/gruntwork-io/terratest/modules/http-helper"
"github.com/stretchr/testify/assert"
"testing"
"time"
)
func TestDockerContainer(t *testing.T) {
// Configure the tag to use on the Docker image.
tag := "nshipman-io/devwhoops"
buildOptions := &docker.BuildOptions{
Tags: []string{tag},
}
stopOptions := &docker.StopOptions{
Time: 10,
}
// Build the Docker image.
docker.Build(t, "../", buildOptions)
// Run the Docker image, inspect the container and make sure it contains the status is running.
options := &docker.RunOptions{
Detach: true,
OtherOptions: []string{"-p", "8080:8080"},
}
// Run the docker container and store the container ID
container := docker.Run(t, tag, options)
// Defer in GoLang, will run even if the code fails. You will want this for cleanup.
defer docker.Stop(t, []string{container}, stopOptions)
// Verify our container is in a running state
output := docker.Inspect(t, container)
running := output.Running
assert.Equal(t, running, true)
// Verify the webpage returns a 200 status code.
maxRetries := 5
timeBetweenRetries := 2 * time.Second
url := "http://localhost:8080/healthcheck"
tlsConfig := tls.Config{}
expectedServerText := "OK"
http_helper.HttpGetWithRetry(t, url, &tlsConfig, 200, expectedServerText, maxRetries, timeBetweenRetries)
}
Now before we run our code, let's setup our workspace for our Go tests.
# 1. Although this should be enabled by default, ensure you set `GOMOD111MODULE=on` or `GODMOD111MODULE=auto` to run the tests outside of the GO workspace.
# 2. Initialize go mod with the package name. It can be anything but for this example I will be using `github.com/nshipman-io/devwhoops-io`
go mod init github.com/nshipman-io/devwhoops-io
With our workspace configured, let's run our tests and check out output
go test -v docker_build_test.go
=== RUN TestDockerContainer
TestDockerContainer 2020-11-16T22:23:52-05:00 logger.go:66: Running 'docker build' in ../
TestDockerContainer 2020-11-16T22:23:52-05:00 logger.go:66: Running command docker with args [build --tag nshipman-io/devwhoops ../]
TestDockerContainer 2020-11-16T22:23:56-05:00 logger.go:66: Sending build context to Docker daemon 6.746MB
TestDockerContainer 2020-11-16T22:23:56-05:00 logger.go:66: Step 1/10 : FROM python:latest
TestDockerContainer 2020-11-16T22:23:56-05:00 logger.go:66: ---> f112835c8a07
TestDockerContainer 2020-11-16T22:23:56-05:00 logger.go:66: Step 2/10 : COPY . /devwhoops
TestDockerContainer 2020-11-16T22:23:57-05:00 logger.go:66: ---> 9aeac3369865
TestDockerContainer 2020-11-16T22:23:57-05:00 logger.go:66: Step 3/10 : WORKDIR /devwhoops
TestDockerContainer 2020-11-16T22:23:57-05:00 logger.go:66: ---> Running in 08d94ddf96b2
TestDockerContainer 2020-11-16T22:23:57-05:00 logger.go:66: Removing intermediate container 08d94ddf96b2
TestDockerContainer 2020-11-16T22:23:57-05:00 logger.go:66: ---> 9a1f13a21d1b
TestDockerContainer 2020-11-16T22:23:57-05:00 logger.go:66: Step 4/10 : RUN adduser webapp && chown -R webapp:webapp /devwhoops
TestDockerContainer 2020-11-16T22:23:57-05:00 logger.go:66: ---> Running in 09e563f0a17d
TestDockerContainer 2020-11-16T22:23:57-05:00 logger.go:66: Adding user `webapp' ...
TestDockerContainer 2020-11-16T22:23:57-05:00 logger.go:66: Adding new group `webapp' (1000) ...
TestDockerContainer 2020-11-16T22:23:57-05:00 logger.go:66: Adding new user `webapp' (1000) with group `webapp' ...
TestDockerContainer 2020-11-16T22:23:57-05:00 logger.go:66: Creating home directory `/home/webapp' ...
TestDockerContainer 2020-11-16T22:23:57-05:00 logger.go:66: Copying files from `/etc/skel' ...
... More Docker Build Stuff...
TestDockerFrontend 2020-11-16T22:55:34-05:00 logger.go:66: Successfully built 256da19fbc39
TestDockerContainer 2020-11-16T22:55:34-05:00 logger.go:66: Successfully tagged nshipman-io/devwhoops:latest
TestDockerContainer 2020-11-16T22:55:34-05:00 logger.go:66: Running 'docker run' on image 'nshipman-io/devwhoops'
TestDockerContainer 2020-11-16T22:55:34-05:00 logger.go:66: Running command docker with args [run --detach -p 8080:8080 nshipman-io/devwhoops]
TestDockerContainer 2020-11-16T22:55:34-05:00 logger.go:66: 07b24be22faa8a4e6c6cbf2cfbfc3dfcdf602c47b3c6b8da5363e38a53e610ec
TestDockerContainer 2020-11-16T22:55:34-05:00 retry.go:91: HTTP GET to URL http://localhost:8080/healthcheck
TestDockerContainer 2020-11-16T22:55:34-05:00 http_helper.go:32: Making an HTTP GET call to URL http://localhost:8080/healthcheck
TestDockerContainer 2020-11-16T22:55:35-05:00 retry.go:103: HTTP GET to URL http://localhost:8080/healthcheck returned an error: Get "http://localhost:8080/healthcheck": EOF. Sleeping for 2s and will try again.
TestDockerContainer 2020-11-16T22:55:37-05:00 retry.go:91: HTTP GET to URL http://localhost:8080/healthcheck
TestDockerContainer 2020-11-16T22:55:37-05:00 http_helper.go:32: Making an HTTP GET call to URL http://localhost:8080/healthcheck
TestDockerContainer 2020-11-16T22:55:37-05:00 logger.go:66: Running 'docker stop' on containers '[07b24be22faa8a4e6c6cbf2cfbfc3dfcdf602c47b3c6b8da5363e38a53e610ec]'
TestDockerContainer 2020-11-16T22:55:37-05:00 logger.go:66: Running command docker with args [stop --time 10 07b24be22faa8a4e6c6cbf2cfbfc3dfcdf602c47b3c6b8da5363e38a53e610ec]
TestDockerContainer 2020-11-16T22:55:37-05:00 logger.go:66: 07b24be22faa8a4e6c6cbf2cfbfc3dfcdf602c47b3c6b8da5363e38a53e610ec
--- PASS: TestDockerContainer (7.98s)
PASS
ok command-line-arguments 8.134s
Voila! We can see that our tests have passed and our Dockerfile is ready to be pushed. Granted this is a very basic test and you may need to tweak your tests to fit your organizational needs. I hope to have provided you with some direction in ways to grow your skillset in the DevOps / SRE role.
In the future, I'll look into providing more test cases for Kubernetes, Helm charts and how we can integrate these tests into our CI/CD workflow.
Cheers!