Automating Integration and Deployment to Remote Server

GitLab CI/CD

Abdelrahman Aly

Published in

Level Up Coding

7 min readAug 14, 2020

Prerequisites:
- What’s Version Control?
- What’s Docker?
- What’s CI/CD?

Objective

The goal is to give a friendly look into GitLab CI/CD with an example application that helps to get started without having to read all of GitLab documentation.

Overview

Continuous Integration works by pushing small code chunks -commits- to your application’s codebase hosted in a Git repository, and, to every push, run a pipeline of scripts to build, test, and validate the code changes before merging them into the main branch.

Continuous Delivery and Deployment consist of a step further CI, deploying your application to production at every push to the default branch of the repository.

These methodologies allow you to catch bugs and errors early in the development cycle, ensuring that all the code deployed to production complies with the code standards you established for your app.

One of the major benefits of using Gitlab CI/CD is that you are freed from the cumbersome process of creating workflows using a lot of third-party plugins and tools.

If your codebase hosted on a version control system other than GitLab, and you want to use GitLab CI/CD you can read about Repository Mirroring.
If you’re coming over from Jenkins, you can also check out handy reference for converting your pipelines.

Getting started

GitLab CI/CD is configured by a file called basic .gitlab-ci.yml placed at the repository's root. The scripts set in this file are executed by the GitLab Runner.

To add scripts to that file, you’ll need to organize them in a sequence that suits your application and are by the tests you wish to perform. To visualize the process, imagine that all the scripts you add to the configuration file are the same as the commands you run on a terminal on your computer.

The scripts are grouped into jobs, and together they compose a pipeline.

Pipelines

We can struct pipelines as we want because YAML is a serialization human-readable language

The assumption to build 3 pipelines:

Project Pipeline installs dependencies, runs linters, and any scripts that deal with the code.
Continuous Integration Pipeline runs automated tests and builds a distributed version of the code.
Deploy Pipeline deploys code to a designated cloud provider and environment.

Steps that pipelines execute are called jobs. When you group a series of jobs by those characteristics it is called stages. Jobs are the basic building block for pipelines. They can be grouped in stages and stages can be grouped into pipelines.

Basic Pipelines

Example basic /.gitlab-ci.yml pipeline configuration matching the diagram:

In this hierarchy, all three components are considered three different pipelines [{build_a, build_b}, {test_a, test_b}, {deploy_a, deploy_b}].

The main bullets -build, -test, and -deploy are stages and each bullet under those sections is a job

Jobs will be executed based on the order listed in the stages directive.

You can make deploy_a deploys to a staging server and deploy_b deploys to the production server using the only directive which will trigger jobs when pushing commits to the branch under the only directive

deploy-production:stage: deployscript:
     - ./deploy_prod.shonly:
     - master

Note: pipelines' names are custom. you can rename deploy-production to what makes sense for you.

You can use CI Lint to write and validate your YAML file before adding it to the root of the repository (GitLab repository UI > left sidebar CI/CD > Pipelines CL Lint button)

You can also get started by using one of the templates available through the UI. You can use them by creating a new file, choosing a template that suits your application, and adjusting it to your needs:

After saving your file to the root of the repo, it will be detected by GitLab as a CI/CD configuration and start executing

If you go to left sidebar CI/CD > Pipelines you will find your jobs stuck and if you click one of them you will see the following issue:

GitLab Runner

GitLab Runner is the open-source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with GitLab CI/CD, the open-source continuous integration service included with GitLab that coordinates the jobs.

Install GitLab Runner

GitLab Runner can be installed and used on GNU/Linux, macOS, FreeBSD, and Windows. You can install it using Docker, download the binary manually, or use the repository for rpm/deb packages that GitLab offers. In this blog, I’ll install it as a docker service

Install the Docker image and start the container

Before you begin, ensure Docker is installed.

To run gitlab-runner inside a Docker container, you need to make sure that the configuration is not lost when the container is restarted. To do this, there are two options, which are described here. when asked for the image while installation I have typed alpine:3.7 which is light and enough for the purpose.

Note: If you are using a session_server, you will also need to expose port 8093 by adding -p 8093:8093 to your docker run command.

Register the Runner

The final step is to register a new Runner. The GitLab Runner Container won’t pick up any jobs until it’s registered.

After you finish registration, the resulting configuration will be written to your chosen configuration volume (e.g. /srv/gitlab-runner/config), and will be automatically loaded by the Runner using that configuration volume.

To register a Runner using a Docker container:

Run the register command:

For local system volume mounts:

docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

If you used a configuration volume other than /srv/gitlab-runner/config during install, then you should update the command with the correct volume.

For Docker volume mounts:

docker run --rm -it -v gitlab-runner-config:/etc/gitlab-runner gitlab/gitlab-runner:latest register

2. Run the register command:

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com) 
https://gitlab.com

3. Enter the token you obtained to register the Runner:

Please enter the gitlab-ci token for this runner
xxx

You can get URL and token from Settings>CI/CD>Runners>Expand>Set up a specific Runner manually

Now time to restart the pipeline that stuck and then you can find it executed successfully.

Deployment

I’ll mention here accessing a remote server using ssh through YAML script

To deploy for a cloud provider easily, you can use dpl

Add SSH key

When your CI/CD jobs run inside Docker containers (meaning the environment is contained) and you want to deploy your code in a private server, you need a way to access it. This is where an SSH key pair comes in handy.

Make sure you can connect to the server before jumping to GitLab CI

You will first need to create an SSH key pair. For more information, follow the instructions to generate an SSH key. Do not add a passphrase to the SSH key, or the before_script will prompt for it in the YAML file.

Here, I generate SSH RSA key

ssh-keygen -t rsa -b 4096 -C "example"

The -C flag adds a comment in the key in case you have multiple of them and want to tell which is which. It is optional.

After that, we need to copy the private key, which will be used to connect to our server, to be able to automate our deployment process:

# Copy the content of public key to authorized_keys
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Use the following command to show the content of id_rsa and copy it:

cd .ssh && cat id_rsa

Go to GitLab UI sidebar>settings>CI/CD>Variables>Expand
Add a variable called SSH_PRIVATE_KEY and in the Value field, paste the private key you just copied from the server (if it’s AWE EC2 it will be the content of the /.pem file)

Add Deploy Key

Deploy keys allow read-only or read-write (if enabled) access to your repository that’s cloned on a server.

Got to GitLab UI sidebar>settings>Repository>Deploy Keys>Expand
Create title and in the Key field paste the content existing in id_rsa.pub

cd .ssh && cat id_rsa.pub

Time to write the YAML file:

By default, alpine doesn’t come with an SSH client. That’s why I’m using the alpine package manager to add SSH client. You have to change commands depending on your flavored Linux system if you don’t use alpine in the runner registration or YAML configuration.