Tag Archives: Docker

Continuous Security with Jenkins and Docker Bench

Docker Bench
Image by Docker Inc. ©

Docker Bench is an open source tool for automatically validating the configuration of a host running Docker containers. The tool has been written among others by Diogo Mónica, security lead at Docker, and performs security checks at the container level following Docker’s CIS Benchmark recommendations.

As you would expect, the easiest way to run Docker Bench is via a Docker container. Just make sure you have Docker 1.10 or better, download the Docker image:

docker pull docker/docker-bench-security

and run the Docker container as follows:

docker run -it --net host --pid host --cap-add audit_control \
    -v /var/lib:/var/lib \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /usr/lib/systemd:/usr/lib/systemd \
    -v /etc:/etc --label docker_bench_security \
    docker/docker-bench-security

This will automatically generate some output as in the animated gif above with an assessment of possible Docker security issues.

I recently combined Docker Bench with Jenkins in order to integrate security testing into a typical DevOps workflow on the cloud – call it DevSecOps if you like buzzwords… This requires a little bit of Jenkins configuration but it’s not too difficult to follow.

The first thing to do is to make sure that openssh is installed on the instance where Jenkins is running and on the host you want to check. For example on Ubuntu you can install openssh with:

sudo apt-get install openssh-server

Then install the SSH Agent plugin in Jenkins. This will provide Jenkins with SSH credentials to automatically login into a machine on the cloud. Add the credentials in Jenkins -> Credentials -> System -> Global credentials (unrestricted) -> Add credentials -> SSH Username with private key.  This is an example of my credentials for user jenkins (private key details are obfuscated):

set up mock SSH credentialsThen create a Jenkins job and select the SSH agent credentials for user jenkins in Build Environment:

SSH agent credentialsThis will allow Jenkins to SSH into the machine with the private key stored securely (make sure you only grant permission to configure Jenkins to administrators otherwise your private keys are not safe).

I like to parameterize my builds so that I can run Docker Bench on any host reachable with the private key:

parameterize build with hostnameFinally, select Execute shell in the build and paste this shell script (you may want to put it under version control and retrieve it from there via Jenkins):

ssh -T -o StrictHostKeyChecking=no ec2-user@$HOSTNAME << 'EOF'
sudo docker pull docker/docker-bench-security && \
sudo docker run --net host --pid host --cap-add audit_control -v /var/lib:/var/lib -v /var/run/docker.sock:/var/run/docker.sock -v /usr/lib/systemd:/usr/lib/systemd -v /etc:/etc --label docker_bench_security docker/docker-bench-security && \
sudo docker rm $(sudo docker ps -aq -f status=exited) && \
sudo docker rmi docker/docker-bench-security
EOF

It works likes this:

  • the first command allows Jenkins to ssh into a host (I’m using AWS EC2 as you can guess by the username ec2-user, replace it with your default username but do not user root). Note that the environment variable $HOSTNAME is passed from the parameter we set up earlier. The EOF allows to run a sequence of commands directly on the host so that you don’t have to disconnect every time. The single quotes are important, don’t skip them!
  • the second command pulls the Docker image for Docker Bench directly on the host
  • the third command runs Docker Bench on the host
  • the forth command removes all exited containers from the host, including Docker Bench once it has finished its job
  • the fifth command remove the Docker image for Docker Bench so that you don’t leave any trace on the host
  • the final EOF disconnect Jenkins from host

The Jenkins console output shows the result of running Docker Bench on a specific host. Now you have to assess the results as you may see several warnings and they may just be false positives. For example, this warning may be acceptable for you:

[1;31m[WARN][0m 1.5  - Keep Docker up to date
[1;31m[WARN][0m       * Using 1.11.1, when 1.12.0 is current as of 2016-07-28

This means you are not running the latest version of Docker. This may not be an issue (unless Docker released a security release) especially if your Linux distribution hasn’t got the latest version of Docker available in its repositories.

In my case this warning was a false positive:

[1;31m[WARN][0m 2.1  - Restrict network traffic between containers

In fact, I need several containers to communicate between them so that restriction does not apply to my use case.

This warning should be taken much more seriously:

[1;31m[WARN][0m 4.1  - Create a user for the container
[1;31m[WARN][0m      * Running as root: container-name-bbf386c0b301

This means you are running a container as root. This is unsecure as if a nasty intruder manages to get inside the containers s/he can run any command in it. Basically, it’s like running a Linux system as root which is a bad security practice.

Once you have assessed your warnings, you may want to filter out the false positives. For example, you can use the Post build task plugin to make the build fail if the build log output contains a warning that you assessed as a security risk. You can use a regular expression to match the pattern identified above.

It would be good to get the Docker Bench output in JUnit format so that Jenkins can understand it natively but this option is currently not implemented in Docker Bench.

Happy security testing with Jenkins and Docker Bench!

API Days Nordic 2016

API Days Nordic 2016
Image by API Days Nordic ©

This week I’m attending API Days Nordic at the University of Tampere (Finland), a technical conference on heavy industry and APIs, government driven API platforms, and APIOps (APIs, DevOps, containers, testing, micro services, and monitoring). This is part of the API Days series of conferences and it is the first time that it is organised in Finland.

My presentation focuses on how to deploy an API gateway with Docker. The slides are available here and the code is on GitHub. This is the summary of my presentation:

An API gateway is a single entry point for APIs in a microservices infrastructure. It provides authentication and authorization layers, routes and load balances requests to API services, and caches previous requests. Being the first entry point of the API, it is crucial to manage and provision it through code rather than using a manual process. Furthermore, replicating its configuration on development and staging environments allows to load test the API gateway and to anticipate issues before it is deployed to production.

I demonstrate the deployment of an API gateway using Docker. Technologies used include:

  • Docker
  • openresty/nginx
  • 3scale API management
  • AWS EC2

I discuss the benefits of using Docker and how it simplifies changes of configuration and deployment to multiple environments. Sample code and brief documentation are available on GitHub.

DockerCon Europe 2015

DockerCon Europe 2015
Image DockerCon Europe 2015 ©

This week I am attending DockerCon Europe 2015 in Barcelona where I am going to talk about Continuous Integration with Jenkins, Docker and Compose. This is a reviewed version of a previous blog post on Jenkins and Docker. Here are the slides in PDF.

Update: DockerCon put online the full video of the presentation. Sample code is available on my GitHub account.

Continuous Integration with Jenkins, Docker and Compose

Oxford University Press (OUP) recently started the Oxford Global Languages (OGL) initiative which aims at providing language resources for digitally under represented languages. In August 2015 OUP launched two African languages websites for Zulu and Northern Sotho. The backend of these websites is based on an API retrieving data in RDF from a triple store and delivering data to the frontend in JSON-LD.

The entire micro-service infrastructure for development, staging, and production runs on Docker containers in Amazon EC2 instances. In particular, we use Jenkins to rebuild the Docker image for the API based on a Python Flask application and Docker Compose to orchestrate the containers. A typical CI workflow is as follows:

– a developer commits code to the codebase
– Jenkins triggers a job to run unit tests
– if the unit tests are successful, the Docker image of the Python Flask application is rebuilt and the container is restarted via Docker Compose
– if the unit tests or the Docker build failed, the monitor view shows the Jenkins jobs in red and displays the name of the possible culprit who broke the build.

Here is a live demo of the Continuous Integration workflow, sample code is available on GitHub.