GitLab- Pulling Images From ECR

In this blog post, we'll explore the seamless process of pulling Docker images from Amazon ECR using GitLab, streamlining your CI/CD workflows and enhancing your development pipeline.

Understanding GitLab and Amazon ECR Integration: GitLab simplifies the management of containerized applications by providing native integration with container registries like Amazon ECR. Leveraging GitLab's CI/CD capabilities along with ECR allows developers to automate the deployment of Docker images stored in ECR repositories. This integration streamlines the process of building, testing, and deploying containerized applications, fostering a more efficient and agile development environment.

Step-by-Step Guide to Pulling Images from Amazon ECR with GitLab:

ECR example:

  <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/<NAMESPACE>:<TAG>

Full job:

  test:api:dev:
    stage: test
    image: <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/<NAMESPACE>:<TAG>
    services:
      - postgres:latest
      - redis:latest
    variables:
      POSTGRES_DB: data_api
      POSTGRES_USER: runner
      POSTGRES_PASSWORD: runner
      DATABASE_URL: postgres://runner:runner@postgres:5432/data_api
    script:
      - cd api
      - export DEBUG=1
      - export ENVIRONMENT=dev
      - export CELERY_BROKER_URL=redis://redis
      - export CELERY_RESULT_BACKEND=redis://redis
      - python -m pytest -p no:warnings .
      - flake8 .
      - black --exclude="migrations|env" --check .
      - isort --skip=migrations --skip=env --check-only
      - export DEBUG=0
      - export ENVIRONMENT=prod
      - python manage.py check --deploy --fail-level=WARNING

Assuming the image exists on the registry, you can set the DOCKER_AUTH_CONFIG variable within your project’s Settings > CI/CD page:

    {
    "auths": {
    "registry.example.com:5000": {
      "auth": "TBD"
       }
     }
  }

The value of auth is a base64-encoded version of your username and password that you use to authenticate into the registry:

  $ echo -n "my_username:my_password" | base64

Continuing with the ECR example, you can generate a password using the following command:

  $ docker run --rm \
      -e AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY_ID> \
      -e AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY> \
      amazon/aws-cli ecr get-login-password \
          --region <AWS_REGION>

To test, run:

  $ docker login -u AWS -p <GENERATED_PASSWORD> <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com

  Login Succeeded

Now, add the DOCKER_AUTH_CONFIG variable to your project’s Settings > CI/CD page:

  {
   "auths": {
    "registry.example.com:5000": {
      "auth": "<GENERATED_PASSWORD>" // base 64 encoded one
      }
    }
  }

Test out your build. You should see something similar to the following in your logs, indicating that the login was successful:

  Authenticating with credentials from $DOCKER_AUTH_CONFIG
  Pulling docker image [MASKED].dkr.ecr.us-east-1.amazonaws.com/api:latest ...

Unfortunately, we’re not done yet since the generated password/token from the get-login-password command is only valid for 12 hours. So, we need to dynamically update the DOCKER_AUTH_CONFIG variable with a new password. We can set up a new job for this:

  build:aws_auth:
    stage: build
    services:
      - docker:dind
    image: docker:stable
    variables:
      DOCKER_DRIVER: overlay2
      DOCKER_BUILDKIT: 1
    script:
      - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
      - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
      - export AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION
      - export TOKEN=$TOKEN
      - export PROJECT_ID=$PROJECT_ID
      - apk add --no-cache curl jq bash
      - chmod +x ./aws_auth.sh
      - bash ./aws_auth.sh

Here, after exporting the appropriate environment variables (so we can access them in the aws_auth.sh script), we installed the appropriate dependencies, and then ran the aws_auth.sh script.

aws_auth.sh:

  #!/bin/sh

  set -e

  AWS_PASSWORD=$(docker run --rm \
      -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
      -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
      amazon/aws-cli ecr get-login-password \
      --region $AWS_DEFAULT_REGION)
  ENCODED=$(echo -n "AWS:$AWS_PASSWORD" | base64)
  PAYLOAD=$( jq -n --arg userpass "$ENCODED" '{"auths": {"263993132376.dkr.ecr.us-east-1.amazonaws.com": {"auth": $userpass}}}' )
  curl --request PUT --header "PRIVATE-TOKEN:$TOKEN" "https://gitlab.com/api/v4/projects/$PROJECT_ID/variables/DOCKER_AUTH_CONFIG" --form "value=$PAYLOAD"

What’s happening?

We generated a new password from the get-login-password command and assigned it to AWS_PASSWORD
We then base64 encoded the username and password and assigned it to ENCODED
We used jq to create the necessary JSON for the value of the DOCKER_AUTH_CONFIG variable
Finally, using a GitLab Personal access token we updated the DOCKER_AUTH_CONFIG variable

Make sure to add all variables you project’s Settings > CI/CD page.

Now, the DOCKER_AUTH_CONFIG variable should be updated with a new password for each build.

That’s it!

–

Helpful Resources:

GitLab Runner Issue Thread - Pull images from aws ecr or private registry
GitLab Docs - Define an image from a private Container Registry

DevOpsBuddies

DevOpsBuddies

GitLab- Pulling Images From ECR