Automate workflows with Vault GitHub actions

GitHub Actions enable you automate workflows for your GitHub hosted repositories. The workflows that build, test, and deploy your code may require secrets to achieve their goal. The Vault GitHub action gives you the ability to pull secrets from Vault.

Challenge

A GitHub repository maintains a web application that requires a Docker image. The image requires the injection of secrets that you have stored in Vault. The process is typically handled either manually, or through a continuous integration (CI) service. The process requires human intervention and does not offer rich feedback alongside the GitHub repository.

Solution

Define a GitHub workflow within your repository and request the required secrets with Vault GitHub actions.

Prerequisites

This tutorial requires Vault, git, Docker, a GitHub account, and the sample web application.

Retrieve the web application and additional configuration by cloning the hashicorp-education/learn-github-actions repository from GitHub.

$ git clone https://github.com/hashicorp-education/learn-vault-github-actions

This repository holds supporting content for Vault learn tutorials. You can find the content for this hands on lab in a subdirectory.

Change into the learn-vault-github-actions/vault-github-action directory.

$ cd learn-vault-github-actions/vault-github-action

Create Docker image

Create a Docker image for the application and label it vault-action-exampleapp.

$ docker build . --file Dockerfile -t vault-action-exampleapp

Docker builds the image and pushes it to your local Docker repository. During image creation a default application secret gets stored in the application root, in a file named app_secret.

View the contents of the app_secret file in the Docker image.

$ docker run vault-action-exampleapp /bin/bash -c "cat ./app_secret"UNSET_SECRET_PLEASE_OVERRIDE

The contents of the file show that the file used the default value during creation. Now, you are ready to automate the image build process while overriding the secret.

Start Vault

Vault can manage the secrets required for this application. A Vault server run in development mode automatically initializes, unseals, and enables the key-value secrets engine.

1. In another terminal, start a Vault dev server with root as the root token.

   $ vault server -dev -dev-root-token-id root

   The Vault dev server defaults to running at 127.0.0.1:8200. A dev mode server automatically initializes and unseals itself at startup.

   Insecure operation

   Do not run a Vault dev server in production. You use a dev mode server in this tutorial just to simplify the unsealing process for this hands on lab.

2. Export an environment variable for the vault CLI to address the Vault server.

   $ export VAULT_ADDR=http://127.0.0.1:8200

3. Export an environment variable for the vault CLI to authenticate with the Vault server.

   $ export VAULT_TOKEN=root

The Vault server is ready to have a secret added.

Create a secret

The GitHub workflow deployed later reads a secret defined at secret/data/ci. You need to create this secret, a policy defined to access the secret, and a token generated to retrieve the secret.

1. Create the secret at the path secret/ci with an app_secret key.

   $ vault kv put secret/ci app_secret=SecretProvidedByVault

2. Verify that the secret exists at the path secret/ci.

   $ vault kv get secret/ci

   You created the secret.

3. Write a policy that grants the read capability for the secret path.

   $ vault policy write ci-secret-reader - <<EOFpath "secret/data/ci" {    capabilities = ["read"]}EOF

   You wrote the policy, and can now specify its attachment to the token you create next.

4. Export an environment variableGITHUB_REPO_TOKEN to capture the token value created with the ci-secret-reader policy attached.

   $ GITHUB_REPO_TOKEN=$(vault token create -policy=ci-secret-reader -format json | jq -r ".auth.client_token")

5. Retrieve the secret at the path using the GITHUB_REPO_TOKEN.

   $ VAULT_TOKEN=$GITHUB_REPO_TOKEN vault kv get secret/ci====== Metadata ======Key              Value---              -----created_time     2020-11-06T20:53:00.845238Zdeletion_time    n/adestroyed        falseversion          1 ======== Data ========Key              Value---              -----app_secret    SecretProvidedByVault

   You retrieved the secret using the token.

This token gets assigned to the GitHub repository in the next section.

Setup the GitHub self-hosted runner

You can define GitHub actions for a repository. Create a new repository associated with your user account or a GitHub organization.

Create a GitHub repository

1. Initialize the current directory as a git repository

   $ git init

2. Stage all files to commit.

   $ git add .

3. Commit all staged files.

   $ git commit -m "Initial Commit"

4. On GitHub, in the upper-right corner of any page, use the + (plus) drop-down menu, and select New repository.

   

5. Name your repository "vault-action-exampleapp"

6. Click Create repository.

   

   The view changes to show you the main page of the repository.

7. Follow the instructions from the …or push an existing repository from the command line section. Return to the terminal and paste those commands.

   $ git remote add origin https://github.com/<YOUR_ACCOUNT_OR_ORG>/vault-action-exampleapp$ git branch -M main$ git push -u origin main

Setup Vault auth credentials with the repository

The GitHub self-hosted runner requires a token for it to authenticate with the Vault server. This token value gets defined as a repository secret.

1. On GitHub, navigate to the main page of the repository.

2. Under your repository name, click Settings.

   

3. In the left sidebar, click Secrets.

   

4. Click New secret.

   

5. Enter the name VAULT_TOKEN for your secret in the Name input box.

6. From the terminal, copy the token stored in the variable GITHUB_REPO_TOKEN

   $ echo $GITHUB_REPO_TOKEN | pbcopy

7. Paste the token as the value for the secret.

8. Click Add secret.

   

   The view returns to the secrets index and displays the new secret in the list of secrets.

You configured the GitHub repository with a token that is valid, and capable of reading the secret from the Vault server.

Setup the GitHub self-hosted runner

The GitHub self-hosted runner enables you to start a runner instance on an instance that you manage. You can use your workstation if it's supported.

1. On GitHub, navigate to the main page of the repository.

2. Under your repository name, click Settings.

3. In the left sidebar, click Actions.

   

4. Under "Self-hosted runners," click Add runner.

   

5. Select the operating system and architecture of your self-hosted runner machine.

6. Follow the instructions in the Download section.

   This prepares a directory for the GitHub runner and then downloads the runner.

   Warning

   You can download and extract the runner within this tutorial directory, but you should not check this code into your repository source as it can contain sensitive information.

7. Follow the instructions in the Configure section.

   This configures the runner to connect to GitHub with a token it generates for the runner.

Define a workflow for the GitHub action

GitHub actions express the operations that they carry out through workflows. These workflows can trigger based on different events that take place during the lifecycle of the source code in the repository. GitHub actions automatically create workflows when it detects a configuration file within the repository.

1. In a terminal, within the repository, create the directory .github/workflows.

   $ mkdir -p .github/workflows

2. Create a workflow file named image-builder.yml within that directory that defines the name of the workflow and the trigger frequency.

   $ tee .github/workflows/image-builder.yml <<EOFname: ImageBuilder# Run this workflow every time a new commit pushed to your repositoryon: pushEOF

   The name of the Workflow determines how it will appear in the repository's actions interface. The on specifies that this workflow takes action when any time that the repository receives a new commit on any branch.

   This image builder workflow needs to:

   • Check out the source code

   • Import the secret from Vault

   • Build the image with the secret

3. A workflow defines one or more jobs. A job sets up an environment, and describes a list of steps necessary to complete an operation for the workflow.

   Add a self-hosted runner job to the workflow named build.

   $ tee -a .github/workflows/image-builder.yml <<EOFjobs:  build:    runs-on: self-hosted    steps:EOF

4. The build job may define one or more steps to complete the work of the build job. The first operation for the job is to checkout the source code of the repository.

   Add the checkout step to the build job.

   $ tee -a .github/workflows/image-builder.yml <<EOF      - uses: actions/checkout@v3EOF

5. You define the steps in a YAML array. This step uses the core GitHub checkout action. After the source code check out, the job fetches secrets from the Vault server for the build step.

   Add a step, named Import Secrets, that uses the Vault GitHub action.

   $ tee -a .github/workflows/image-builder.yml <<EOF      - name: Import Secrets        uses: hashicorp/vault-action@v2        with:          url: http://127.0.0.1:8200          tlsSkipVerify: true          token: \${{ secrets.VAULT_TOKEN }}          secrets: |            secret/data/ci app_secretEOF

   This step defines its name as "Import Secrets" overriding the default name provided by the hashicorp/vault-action@v2 step. The configuration for this step communicates with the local Vault server running in dev mode. The token used to authenticate is the VAULT_TOKEN secret value you defined in the GitHub repository.

   The secrets section of this step defines the path of the secret, secret/data/ci, and the key to extract that secret app_secret. By default, this secret value gets exported to the environment variable, APP_SECRET, that is useful to the steps that follow this one.

6. Add the build step to perform the Docker image build operation.

   $ tee -a .github/workflows/image-builder.yml <<EOF      - name: Build Docker Image        run: docker build . --file Dockerfile --build-arg app_secret="\${{ env.APP_SECRET }}" -t    vault-action-exampleappEOF

This step builds a Docker image from the Dockerfile in the repository. During the build, the build_arg app_secret gets populated with the secret imported from Vault, and the Docker image stored as vault-action-exampleapp in the local Docker registry.

Trigger the GitHub runner

The workflow triggers on every push to any branch of this repository.

1. Add the unstaged files.

   $ git add .

2. Commit the staged changes.

   $ git commit -m "adds workflow to repo"

3. Push these changes to the remote repository.

   $ git push origin main

The GitHub self-hosted runner polls GitHub for changes, and executes the runner upon detecting changes.

1. On GitHub, navigate to the main page of the repository.

2. Under your repository name, click Actions.

   

3. Under "All workflows" click the result adds workflow to repo.

   

   This view displays the execution of the workflow for this commit.

4. Under the name of the workflow, click the build job.

   

   This view displays the steps that took place for this job. Each step can be expanded to see progress, results, and errors.

   

   Docker builds the image and stores it in the local Docker registry. The Docker image created a file named app_secret within the working directory and populated it with the Vault secret.

5. View the contents of the app_secret file in the Docker image.

   $ docker run vault-action-exampleapp /bin/bash -c "cat ./app_secret"SecretProvidedByVault

   The results display the Vault secret defined at the path secret/data/ci. The application, and other services within this Docker image can use the secret.

Next steps

You started Vault, created a secret, and a token with a policy to retrieve that secret. Learn more about creating versioned key/value secrets and policies.

You set up a GitHub action with a self-hosted runner that triggered when you pushed a commit. Learn more about defining GitHub actions and the Vault GitHub action.

The workflow accomplished a small task of creating a Docker image. Watch Vault engineers deploy an example app into a Kubernetes cluster running on AWS.