Jenkins

Jenkins operation is driven by Jenkinsfile in the repository. In order to create (or re-create) the project on the Jenkins server, the following initial steps must be performed:

1. Login to your GitHub account and then use it to login to the Jenkins server.

2. Select New Pipeline in the Blue Ocean interface and answer the following:

   1. Select "GitHub" for "Where do you store your code?"

   2. Select "cmu-db" for "Which organization does the repository belong to?"

   3. Select the "terrier" repository and "Create Pipeline"

3. Once the pipeline is created, navigate to the project page, and select "Configure" from the left sidebar and adjust the following:

• Branch Sources
  • GitHub
    • Behaviors -> Add
      • Discover pull requests from forks
        • Strategy: Merging the pull request with the current target branch revision
        • Trust: Contributors
      • Discover pull requests from origin
        • Strategy: Merging the pull request with the current target branch revision

4. Click "Save" at the bottom of the page.

Preparing new nodes to host Jenkins workers

Our Jenkins environment uses the Self-Organizing Swarm Modules plugin which makes it very easy to bootstrap new Jenkins worker nodes, whether physical or virtual machines. The process for the various environments is as follows:

1. a script for configuring Linux systems that will host Docker containers can be found in script/jenkins/jenkins-docker-setup.sh in the repository. Note that it contains some assumptions about the environment, including the location of the prepared Docker image described below.

2. a script for configuring Linux systems running builds on bare metal for benchmarking purposes can be found in script/jenkins/benchmarker-setup.sh in the repository. Note that it also contains some assumptions about the environment that it will run in.

3. preparing macOS hosts to run Jenkins agent virtual machines is a bit more of a manually intensive process and involves:

   1. installation of VMware Fusion on the host
   2. creation of a template image that will be used to instantiate actual worker VMs, including installation of the swarm.sh and halt.sh scripts (script/jenkins/swarm.sh and script/jenkins/halt.sh in the repository)
   3. installation of VM supervisor script (script/jenkins/vmware_startup.sh in the repository) on the host(s) that will run the worker VMs

Preparing Docker images to run as Jenkins workers

The following steps (required, unless specified otherwise) describe the preparation of images from the Docker hub as build workers since the default images from are so minimal:

1. pull and run an image of the type you want to configure, e.g.,

   docker run -it --name jenkins-prep ubuntu:xenial
   

2. install any prerequisite packages that are needed for script/installation/packages.sh to run, e.g.

   apt-get -qq update && apt-get -qq -y --no-install-recommends install python-dev lsb-release sudo
   

   Some trial and error might be required to determine this list of prerequisites for a particular platform.
3. add a user and group jenkins and grant all sudo privileges

   groupadd -g 115 jenkins && useradd -u 106 -g 115 -c jenkins jenkins && echo 'jenkins ALL=(ALL)       NOPASSWD: ALL' > /etc/sudoers.d/jenkins
   

   NOTE: the numbers specified above must exactly match the actual uid and gid of the jenkins user on the server system in order for the Docker containers to work when invoked from Jenkins
4. (optional) prepopulate the image with prerequisite software packages from packages.sh do avoid delays fetching software or errors from unavailability of mirrors. This can be accomplished by either cut and pasting the appropriate sections from packages.sh or simply creating a copy of that script in the container and running it there. The latter option has the benefit of revealing problems before Jenkins attempts to run the script in a real build.
5. (optional) clean up any residual package files to save a little on disk space, e.g.,

   apt-get clean
   

6. exit the container and save your work, e.g.,

   docker commit jenkins-prep ubuntu:xenial