How do I securely give my CI server access to private Git repos?

[brikis98]

A problem we run into constantly is the following:

You want to run some sort of automation on your code (e.g., run automated tests or terraform apply) on a CI server (e.g., GitHub Actions, CircleCI, etc) against repo foo, but that repo includes references to another private Git repo bar.

Example 1: you have Terraform code in repo foo that uses a module from repo bar.

# foo/example/main.tf
module "example" {
  # A reference to private repo bar. Note this is a Git/SSH URL.
  source = "git::[email protected]:acme/bar.git//example-module"
}

Example 2: You have a Bash script in repo foo that calls git clone on repo bar.

# foo/example/build.sh
# A reference to private repo bar. Note this is an HTTPS URL.
git clone https://github.com/acme/bar.git

Although most CI servers automatically give you access to the code in the repo where the build is running (e.g., foo), they do not provide an automatic way to access code in other private repos (e.g., bar).

So the question is: how do you securely give your CI server access to private Git repos?

• Requirement 1: the solution must work completely non-interactively. On your own computer, you can manually provide credentials when prompted, but on a CI server, you need an automated solution.

• Requirement 2: the solution must work with Git/SSH and HTTPS URLs. Best practices for accessing private Git repos have changed over the years; in the past, Git/SSH URLs were typically recommended, so lots of code uses those. More recently, the tooling around HTTPS URLs has improved, and more and more code is using those.

• Requirement 3: credentials must be stored securely. I deliberately use the term "securely" to indicate that this solution should not involve storing credentials in an insecure manner: e.g., although you could put the credentials directly into your code (into the Git URLs used in git clone), storing credentials, in plain text, is NOT secure:

  # foo/example/build.sh
  # DO NOT DO THIS.
  # Putting credentials directly in the code is NOT secure and is NOT a valid solution to this problem!
  git clone https://my-username:[email protected]/acme/bar.git
  # DO NOT DO THIS.

So, how do you solve this?

I'll share Gruntwork's recommendation below. Feedback, suggestions, and other ideas are very welcome!

---

Tracked in ticket #110600

[brikis98]

Here's the approach we typically recommend. In the examples below, I'll use GitHub as the version control system and GitHub Actions as the CI server, but the same approach works with most other version control systems (e.g., GitLab, BitBucket, etc.) and CI servers (e.g., GitLab, CircleCI, etc.).

1. Create a machine user in GitHub. A machine user is a user account that isn't used by any individual person (i.e., it is not your personal GitHub account); instead, it's an account owned by your company that you use specifically for automation. Using a machine user ensures that (a) if any individual person leaves your company, all your automation doesn't suddenly break when that person loses access and (b) you can share the credentials for that account with multiple people on your team, using a secret manager tool such as 1Password, so you can make changes to your automations even if one particular person happens to be OOO.

2. Grant the machine user access to the repos they need in automation. In the example in the original question, you'll need to invite this machine user to repos foo (where the CI build will run) and bar (another private repo that contains code used in foo), and then, logged into GitHub as that machine user, accept those invites.

3. Create a personal access token (PAT) for the machine user. Logged in as the machine user, create a personal access token (PAT), and ensure that token has at least read access to the repos you need in automation: in our example, that'll be repos foo and bar. You should store this PAT in a secret manager tool such as 1Password.

4. Make the machine user username and PAT available to the CI server. Next, you want to make the username and PAT of your machine user available to the CI build in a secure manner. Most CI servers support some sort of secure way to store secrets (i.e., they are encrypted and have access controls). For example, with GitHub Actions, you can use GitHub Actions Secrets, which allow you to securely store secrets, and make them available to the build as environment variables.

5. Update your Git configuration to use your machine user credentials. As one of the first steps in your CI / CD build, you now need to take the machine user credentials from your CI server (as environment variables) and configure Git to use use them for all git clone operations. We struggled for a while to get just the right "incantation" to support all possible types of URLs until we stumbled across this StackOverflow post and so here, we'll show you the configuration that finally does the right thing.

   What you want is for your Git configuration (~/.gitconfig) to end up looking like this:

   [url "https://<MACHINE_USER_NAME>:<MACHINE_USER_PAT>@github.com"]
     insteadOf = ssh://[email protected]
     insteadOf = [email protected]
     insteadOf = https://github.com

   Where MACHINE_USER_NAME is the username of your machine user and MACHINE_USER_PAT is the PAT for that machine user. This configuration uses insteadOf to tell Git that whenever it sees Git/SSH URLs (e.g., ssh://[email protected] and [email protected]) and HTTPS URLs (e.g., https://github.com), it should instead use an HTTPS URL with your machine user credentials in it.

   The best way to write such a configuration into your ~/.gitconfig is to run the git config command a few times. Assuming that in GitHub Actions Secrets, you used the environment variables MACHINE_USER_NAME and MACHINE_USER_PAT for the machine user name and PAT, respectively, you can run the following commands at the start of your CI build to configure Git:

   git config --global --replace-all "url.https://$MACHINE_USER_NAME:$MACHINE_USER_PAT@github.com.insteadOf" ssh://[email protected]
   git config --global --add "url.https://$MACHINE_USER_NAME:$MACHINE_USER_PAT@github.com.insteadOf" [email protected]
   git config --global --add "url.https://$MACHINE_USER_NAME:$MACHINE_USER_PAT@github.com.insteadOf" https://github.com

6. Run the rest of your build! Once you have this Git configuration in place, you can now run the rest of the build, and you'll be able to git clone private repos no matter how the URLs for those repos are formatted in your actual code!

Try it out and let us know how it works for you!

[yorinasub17]

For GitHub specifically, you can also use a GitHub App!

Using GitHub Apps has a few advantages, namely:

• Does not take up a user slot in your org (so you can save on the cost).
• The token that gets stored on disk (either through .gitconfig or .git origin) is temporary/timed.
• You have access to fine grained permissions, like restricting access to specific repos and specific content (although, you can get the same effect with fine-grained PAT).

The steps for using a GitHub App are:

1. Create a GitHub App. Minimum permission necessary is read access to content.
2. Install GitHub App. Install the GitHub App into your org and restrict the permissions to just the repos that you want to grant access to.
3. Provision a private key for the GitHub App and make available to CI server. You can use the same approach as the PAT above for storing these secrets.
4. In CI, run steps to generate a temporary access token. For GitHub Actions, you can get a token from a GitHub App through the tibdex/github-app-token action. For other platforms, you can use this CLI tool I built: https://github.com/fensak-io/github-app-token. Here is an example CircleCI config (NOTE: this depends on the secrets, GITHUB_APP_ID and GITHUB_APP_PRIVATE_KEY_B64 [base64 encoded private key of the github app]):

jobs:
  example:
    steps:
      - run:
          name: download github-app-token CLI
          command: |
            curl -sLO https://github.com/fensak-io/github-app-token/releases/download/v0.0.1/github-app-token_linux_amd64.tar.gz
            tar -xvf github-app-token_linux_amd64.tar.gz
          working_directory: /tmp
      - run:
          name: semantic-release
          command: |
            export GITHUB_APP_PRIVATE_KEY="$(echo -n "$GITHUB_APP_PRIVATE_KEY_B64" | base64 -d)"
            export GITHUB_TOKEN="$(/tmp/github-app-token --repo fensak-io/github-app-token)"

5. This step is the same as step 5 for machine users, only use the generated token in place of the PAT. NOTE: I believe the username can be set to anything, but I am not sure - I usually just use the github CLI to clone, which obv doesn't work for something like tofu or terraform. I'll try it out at a later point when I have time and update.