Jenkins in Docker

DinD and DooD for jenkins

Docker Client - Server Arhitecture

  • Client (e.g. docker CLI) requests to the Docker Daemon (dockerd)
  • Using belows for communication
    • RESTAPI
    • Unix Socket (in a local host)
    • TCP Socket (between remote machines)

Docker in Docker

  • Two docker daemons exist.
    • One on the host
    • The other on a container (e.g. dind) on a host
  • Requires “–privieged” option
  • System-level program does not run inside a regular Docker container because
    • Container does not expose sufficient kernel resources and appropriate permissions
    • Issues related to
      • Docker’s use of overlay filesystems,
      • security profiles,
      • etc.
Sample Source Codes
Dockerfiles
FROM jenkins/jenkins:2.303.2-jdk11

# if we want to install via apt
USER root

# install packages
RUN apt-get update && apt-get install -q -y --no-install-recommends \
	apt-transport-https \
	ca-certificates curl gnupg2 \
	software-properties-common \
	tar \
	xz-utils \
	bash-completion \
	vim \
	&& apt-get clean && rm -rf /var/lib/apt/lists/*

RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
RUN apt-key fingerprint 0EBFCD88
RUN add-apt-repository \
       "deb [arch=amd64] https://download.docker.com/linux/debian \
       $(lsb_release -cs) stable"
RUN apt-get update && apt-get install -y docker-ce-cli

# drop back to the regular jenkins user - good practice
USER jenkins
RUN jenkins-plugin-cli --plugins \
	"blueocean:1.25.1" \
	"docker-workflow:1.26" \
	"extended-choice-parameter" \
	"role-strategy" \
	"pipeline-model-definition"

COPY --chown=jenkins:jenkins executors.groovy /usr/share/jenkins/ref/init.groovy.d/executors.groovy

FROM nvidia/opengl:1.2-glvnd-runtime-ubuntu20.04

RUN apt-get update && apt-get install -y \
        apt-utils \
        ca-certificates \
        openssh-client \
        curl \
        iptables \
        git \
        gnupg \
        supervisor && \
    rm -rf /var/lib/apt/list/*

# NVIDIA Container Toolkit & Docker
RUN distribution=$(. /etc/os-release;echo $ID$VERSION_ID) && \
    curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | apt-key add - && \
    curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | tee /etc/apt/sources.list.d/nvidia-docker.list && \
    apt-get update && apt-get install -y nvidia-docker2 docker.io docker-compose && \
    rm -rf /var/lib/apt/list/*

COPY modprobe startup.sh /usr/local/bin/
COPY supervisor/ /etc/supervisor/conf.d/
COPY logger.sh /opt/bash-utils/logger.sh

RUN chmod +x /usr/local/bin/startup.sh /usr/local/bin/modprobe
VOLUME /var/lib/docker

ENTRYPOINT ["startup.sh"]
CMD ["sh"]
Run Jenkins Docker (for Docker in Docker)
JENKINS_DATA=${PWD}/../../workspace/jenkins_home/data

docker network create jenkins

cd ./nvidia-dind && \
docker build -t nvidia-dind-image . && \
docker run \
	--name nvidia-dind-container \
	--rm \
	--detach \
	--privileged \
	--gpus 1 \
	--network jenkins \
	--network-alias docker \	# Makes the Docker in Docker container available as the hostname docker within the jenkins network.
	--env DOCKER_TLS_CERTDIR=/certs \	# Enables the use of TLS in the Docker server. Due to the use of a privileged container, this is recommended, though it requires the use of the shared volume described below. This environment variable controls the root directory where Docker TLS certificates are managed.
	--volume jenkins-dind-certs:/certs/client \
	--volume ${JENKINS_DATA}:/var/jenkins_home \
	--volume /mnt/motional_database:/mnt/motional_database \
	--publish 2376:2376 \	# ( Optional ) Exposes the Docker daemon port on the host machine. This is useful for executing docker commands on the host machine to control this inner Docker daemon.
	nvidia-dind-image:latest \
	--storage-driver overlay2	# The storage driver for the Docker volume. See "Docker storage drivers" for supported options.
Stop Jenkins Docker (for Docker in Docker)
docker stop nvidia-dind-container

docker rmi nvidia-dind-image

docker network rm jenkins

docker volume rm jenkins-docker-certs
Run Jenkins (installed) Container
JENKINS_DATA=${PWD}/../../workspace/jenkins_home/data

sudo chown -R 1000.1000 ${JENKINS_DATA}
sudo chmod 777 ${JENKINS_DATA}/..

cd ./jenkins-dind && \
docker build -t jenkins-image . && \
docker run \
	--name jenkins-container \
	--rm \
	--detach \
	--network jenkins \
	--env DOCKER_HOST=tcp://docker:2376 \
	--env DOCKER_CERT_PATH=/certs/client \
	--env DOCKER_TLS_VERIFY=1 \
	--publish 8081:8080 \
	--publish 50000:50000 \
	--volume ${JENKINS_DATA}:/var/jenkins_home \
	--volume ${JENKINS_DATA}/..:/var/jenkins_out \
	--volume jenkins-docker-certs:/certs/client \
	--volume /mnt/motional_database:/mnt/motional_database \
	--workdir=/var/jenkins_home \
	jenkins-image
Stop Jenkins Container
docker stop jenkins-container

docker rmi jenkins-image

Docker out of Docker

  • Only docker daemon runs on the host.
    • Client (e.g. docker CLI) is on a container on a host
  • The connection is done by mounting the host’s Docker’s socket into the container that runs the Docker CLI. For example:
  $ docker run -it -v /var/run/docker.sock:/var/run/docker.sock docker
  • Benefits
    • One key benefit: it bypasses the complexities of running the Docker daemon inside a container and does not require an unsecure privileged container.
    • Avoids having multiple Docker image caches in the system
      • since there is only one Docker daemon on the host
      • if your system is constrained on storage space
  • Drawbacks.
    • Main drawback: it results in poor context isolation because the Docker CLI runs within a different context than the Docker daemon.
      • While DinD runs within the container’s context; DooD runs within host’s context. This leads to problems such as:
        • Container naming collisions
        • Mount paths confusion:
          • Container running the Docker CLI creates a container with a bind mount, the mount path must be relative to the host (as otherwise the host Docker daemon on the host won’t be able to perform the mount correctly).
        • Port mappings:
          • Container running the Docker CLI creates a container with a port mapping, the port mapping occurs at the host level, potentially colliding with other port mappings.
    • Not good on Kubernetes
      • Any containers created by the containerized Docker CLI will not be encapsulated within the associated Kubernetes pod, and will thus be outside of Kubernetes’ visibility and control.
    • Finally, there are security concerns too:
      • Container running the Docker CLI can manipulate any containers running on the host.
        • It can remove containers created by other entities on the host, or
        • even create unsecure privileged containers putting the host at risk.
Sample Source Codes
Dockerfile
ARG tag=${DOCKER_TAG:-lts}
FROM jenkins/jenkins:${tag}

# if we want to install via apt
USER root

# Install packages
RUN apt-get update && apt-get install -q -y --no-install-recommends \
	libltdl7 \
	apt-transport-https \
	ca-certificates curl gnupg2 \
	software-properties-common \
	tar \
	xz-utils \
	bash-completion \
	vim \
	python3 \
	&& apt-get clean && rm -rf /var/lib/apt/lists/*

# Add jenkins user to docker group to make sure jenkins user has docker privileges
# getent group docker | awk -F: '{printf "%d\n", $3}'
ARG dockerGid=1013
RUN echo "${dockerGid}"
RUN echo "docker:x:${dockerGid}:jenkins" >> /etc/group

# drop back to the regular jenkins user - good practice
USER jenkins

COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli --plugins < /usr/share/jenkins/ref/plugins.txt
Run Jenkins Docker
JENKINS_DATA=/var/jenkins_home

cd jenkins-nvidia-dood

sudo mkdir -p ${JENKINS_DATA}
sudo mkdir -p ${JENKINS_DATA}/out
sudo chown -R 1000.1000 ${JENKINS_DATA}

docker build -t jenkins-dood-image . && \
docker run \
	--name jenkins-dood-container \
	--rm \
	--detach \
	--publish 8080:8080 \
	--publish 50000:50000 \
	--volume $(which docker):/usr/bin/docker \
	--volume /var/run/docker.sock:/var/run/docker.sock \
	--volume ${JENKINS_DATA}:/var/jenkins_home \
	--volume ${JENKINS_DATA}/out:/var/jenkins_home/out \
	--volume /mnt/motional_database:/mnt/motional_database \
	--workdir=/var/jenkins_home \
	jenkins-dood-image
Stop Jenkins Docker
docker stop jenkins-dood-container && \
docker rmi jenkins-dood-image