DevOpsBuddies

GitLab Runner Autoscaling

GitLab Runner Autoscaling

DevOpsBuddies's photo
DevOpsBuddies
·

2 min read

You can use GitLab Runner autoscaling to automatically scale the runner on public cloud instances. When you configure a runner to use an autoscaler, you can manage increases in CI/CD job load by leveraging your cloud infrastructure to run multiple jobs simultaneously.

GitLab Runner Autoscaler is the successor to the autoscaling technology based on Docker Machine. The components of the GitLab Runner Autoscaler are:

  • Taskscaler: Manages the autoscaling logic, bookkeeping, and creates fleets for runner instances that use cloud provider autoscaling groups of instances.

  • Fleeting: An abstraction for cloud provider virtual machines.

  • Cloud provider plugin: Handles the API calls to the target cloud platform and is

  • implemented using a plugin development framework.

I am using a AWS CloudFormation template that may be used to deploy a GitLab runner with Docker executor and auto-scaling based on number of builds on AWS.

This creates everything, so all you need to do is provide an SSH key and a gitlab token. Assumes us-east-1.

Resources created

  • VPC

  • Public Subnet for Internet Gateway and Manager Instance

  • Private Subnet for runners.

  • Internet Gateway to talk to the interwebs.

  • Elastic IP for NAT Gateway.

  • NAT Gateway to allow communication from runners.

  • Public and Private Routing Tables so everything can talk.

  • S3 bucket to store runners' cache.

  • EC2 instance that is the runners' manager: it invokes AWS APIs to spawn and terminate other EC2 instances (via docker-machine) and runs Docker containers on them to process GitLab CI builds.

Obtaining a GitLab Runner token

When you launch the stack you are required to pass a GitLab Runner token. This is not to be confused with a GitLab Runner registration token!

You can obtain a registration token by navigating to the "Settings › CI / CD" page of any project for which you have administrative rights. It'll be available under "Runners settings".

You can then obtain a GitLab Runner token by using the (undocumented) endpoint POST /runners:

# Assuming the GitLab instance is available at https://gitlab.example.org
# and the GitLab Runner registration token is "abcdef1234567890":

curl -XPOST -H 'Content-Type: application/json' -H 'Accept: application/json' \
  -d '{"token":"abcdef1234567890","run_untagged":true,"locked":false}' \
  https://code.osu.edu/api/v4/runners

If everything goes fine, the response will be a JSON that has a token key: this is the GitLab Runner token you were looking for.

Once we are Ready with Token , Jump over to aws dashboard and create a Keypair.

And Use below cft template for stack creation.

ref: https://docs.gitlab.com/runner/runner_autoscale/

GitLab