Continuous Security with Jenkins and Docker Bench

Docker Bench
Image by Docker Inc. ©

Docker Bench is an open source tool for automatically validating the configuration of a host running Docker containers. The tool has been written among others by Diogo Mónica, security lead at Docker, and performs security checks at the container level following Docker’s CIS Benchmark recommendations.

As you would expect, the easiest way to run Docker Bench is via a Docker container. Just make sure you have Docker 1.10 or better, download the Docker image:

docker pull docker/docker-bench-security

and run the Docker container as follows:

docker run -it --net host --pid host --cap-add audit_control \
    -v /var/lib:/var/lib \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /usr/lib/systemd:/usr/lib/systemd \
    -v /etc:/etc --label docker_bench_security \
    docker/docker-bench-security

This will automatically generate some output as in the animated gif above with an assessment of possible Docker security issues.

I recently combined Docker Bench with Jenkins in order to integrate security testing into a typical DevOps workflow on the cloud – call it DevSecOps if you like buzzwords… This requires a little bit of Jenkins configuration but it’s not too difficult to follow.

The first thing to do is to make sure that openssh is installed on the instance where Jenkins is running and on the host you want to check. For example on Ubuntu you can install openssh with:

sudo apt-get install openssh-server

Then install the SSH Agent plugin in Jenkins. This will provide Jenkins with SSH credentials to automatically login into a machine on the cloud. Add the credentials in Jenkins -> Credentials -> System -> Global credentials (unrestricted) -> Add credentials -> SSH Username with private key.  This is an example of my credentials for user jenkins (private key details are obfuscated):

set up mock SSH credentialsThen create a Jenkins job and select the SSH agent credentials for user jenkins in Build Environment:

SSH agent credentialsThis will allow Jenkins to SSH into the machine with the private key stored securely (make sure you only grant permission to configure Jenkins to administrators otherwise your private keys are not safe).

I like to parameterize my builds so that I can run Docker Bench on any host reachable with the private key:

parameterize build with hostnameFinally, select Execute shell in the build and paste this shell script (you may want to put it under version control and retrieve it from there via Jenkins):

ssh -T -o StrictHostKeyChecking=no ec2-user@$HOSTNAME << 'EOF'
sudo docker pull docker/docker-bench-security && \
sudo docker run --net host --pid host --cap-add audit_control -v /var/lib:/var/lib -v /var/run/docker.sock:/var/run/docker.sock -v /usr/lib/systemd:/usr/lib/systemd -v /etc:/etc --label docker_bench_security docker/docker-bench-security && \
sudo docker rm $(sudo docker ps -aq -f status=exited) && \
sudo docker rmi docker/docker-bench-security
EOF

It works likes this:

  • the first command allows Jenkins to ssh into a host (I’m using AWS EC2 as you can guess by the username ec2-user, replace it with your default username but do not user root). Note that the environment variable $HOSTNAME is passed from the parameter we set up earlier. The EOF allows to run a sequence of commands directly on the host so that you don’t have to disconnect every time. The single quotes are important, don’t skip them!
  • the second command pulls the Docker image for Docker Bench directly on the host
  • the third command runs Docker Bench on the host
  • the forth command removes all exited containers from the host, including Docker Bench once it has finished its job
  • the fifth command remove the Docker image for Docker Bench so that you don’t leave any trace on the host
  • the final EOF disconnect Jenkins from host

The Jenkins console output shows the result of running Docker Bench on a specific host. Now you have to assess the results as you may see several warnings and they may just be false positives. For example, this warning may be acceptable for you:

[1;31m[WARN][0m 1.5  - Keep Docker up to date
[1;31m[WARN][0m       * Using 1.11.1, when 1.12.0 is current as of 2016-07-28

This means you are not running the latest version of Docker. This may not be an issue (unless Docker released a security release) especially if your Linux distribution hasn’t got the latest version of Docker available in its repositories.

In my case this warning was a false positive:

[1;31m[WARN][0m 2.1  - Restrict network traffic between containers

In fact, I need several containers to communicate between them so that restriction does not apply to my use case.

This warning should be taken much more seriously:

[1;31m[WARN][0m 4.1  - Create a user for the container
[1;31m[WARN][0m      * Running as root: container-name-bbf386c0b301

This means you are running a container as root. This is unsecure as if a nasty intruder manages to get inside the containers s/he can run any command in it. Basically, it’s like running a Linux system as root which is a bad security practice.

Once you have assessed your warnings, you may want to filter out the false positives. For example, you can use the Post build task plugin to make the build fail if the build log output contains a warning that you assessed as a security risk. You can use a regular expression to match the pattern identified above.

It would be good to get the Docker Bench output in JUnit format so that Jenkins can understand it natively but this option is currently not implemented in Docker Bench.

Happy security testing with Jenkins and Docker Bench!

API Days Nordic 2016

API Days Nordic 2016
Image by API Days Nordic ©

This week I’m attending API Days Nordic at the University of Tampere (Finland), a technical conference on heavy industry and APIs, government driven API platforms, and APIOps (APIs, DevOps, containers, testing, micro services, and monitoring). This is part of the API Days series of conferences and it is the first time that it is organised in Finland.

My presentation focuses on how to deploy an API gateway with Docker. The slides are available here and the code is on GitHub. This is the summary of my presentation:

An API gateway is a single entry point for APIs in a microservices infrastructure. It provides authentication and authorization layers, routes and load balances requests to API services, and caches previous requests. Being the first entry point of the API, it is crucial to manage and provision it through code rather than using a manual process. Furthermore, replicating its configuration on development and staging environments allows to load test the API gateway and to anticipate issues before it is deployed to production.

I demonstrate the deployment of an API gateway using Docker. Technologies used include:

  • Docker
  • openresty/nginx
  • 3scale API management
  • AWS EC2

I discuss the benefits of using Docker and how it simplifies changes of configuration and deployment to multiple environments. Sample code and brief documentation are available on GitHub.

Why we need more women in technology

female speech
Image: R.L.Trask and B. Mayblin (2000), Introducing Linguistics, Cambridge: Icon Books ©

Few months ago I sat in an interview panel for recruiting a system administrator at work. We interviewed two male candidates who applied for the job. Although they seem to have excellent technical skills they completely lacked soft skills. My colleagues and I tried to gear the interview towards a normal conversation but instead we were constantly faced with monologues featuring obscure tech acronyms and fancy technologies that were not even relevant for the job. I left the room in despair and at the end we decided not to hire any of them and to start the recruitment process from scratch. I remember asking my colleague: “Next time can we please shortlist at least one women?” Unfortunately we haven’t seen any female candidate so far…

I experienced first hand what the British linguist Jennifer Coates pinpoints as the difference between male and female speech. According to her studies, all-male conversations are like this:

male speech
Image: R.L.Trask and B. Mayblin (2000), Introducing Linguistics, Cambridge: Icon Books ©

Men tend to slip into monologues when talking to each other. Women tend to talk differently. All-female conversations look more like this:

female speech
Image: R.L.Trask and B. Mayblin (2000), Introducing Linguistics, Cambridge: Icon Books ©

Now think of software development. Think for example of an open source project on GitHub. Or building a complex software tool that needs to communicate with both legacy and new systems. Does it sound more like a male or female conversation?

Building software is by definition a co-operative enterprise. Software systems are so complex nowadays that need teams of software engineers with difference skills in order to be built. More importantly, these highly skilled engineers need to talk to each other. If tech teams are completely made of men, discussions will tend to slip into monologues: a single software engineer will take the lead and other voices may get ignored or unheard. Conversely, if tech teams are gender mixed, communication will improve and as a result the software will be better.

References:
Jennifer Coates (1986). Women, Men and Language: A Sociolinguistic Account of Gender Differences in Language. London: Longman.

I got inspired by a great blog post written by the Docker developer Jérôme Petazzoni.

My deny list of IP addresses

security
Image by GotCredit licensed under Creative Commons Attribution 2.0 Generic Licence

I monitor few websites for friends and had to configure security modules in order to prevent brute force attacks and breaches in the admin interface of the website. For CMS like WordPress and Drupal there are some good security modules like iThemesSecurity, Security Review and Login Security. These modules improve the security of a website by limiting the number of failed login attempts and blocking the suspicious IP address.

Over time I collected a list of  denied IP addresses which  I share here. One thing I learnt is that attacks can come from anywhere. I believe that some of these IP addresses belong to a botnet of compromised computers as I could trace their coordinates to some very well-known locations in the world and their origin from some major cloud providers.

Here below is an example of denied IP addresses with geolocation data. The full list is in this CSV file. The data is courtesy of MaxMind and ip-api. The list was last updated on 17th May 2020.

IP Address Country Code Location Postal Code Coordinates ISP Domain
46.20.12.30 TR Bursa,
Bursa,
Turkey,
Asia
16245 40.2719,
29.0983
Dgn Teknoloji Bilisim Yayincilik Sanayi Ve Limited ttnetdc.com
203.130.196.150 ID Jakarta,
Daerah Khusus Ibukota Jakarta,
Indonesia,
Asia
  -6.1744,
106.8294
Pt Telekomunikasi Indonesia  
109.195.228.208 RU Rostov-on-Don,
Rostov,
Russia,
Europe
344000 47.2364,
39.7139
JSC ER-Telecom Holding ertelecom.ru
66.91.141.42 US Honolulu,
Hawaii,
United States,
North America
96805 21.3069,
-157.8583
Time Warner Cable rr.com
185.53.78.144 UA Ukraine,
Europe
  50.45,
30.5233
LLC Gigatel rv.ua
91.200.12.11 UA Ukraine,
Europe
  50.45,
30.5233
PP SKS-Lugan  
37.59.56.6 FR France,
Europe
  48.86,
2.35
OVH SAS ovh.net
37.252.102.136 ES Basque Country,
Spain,
Europe
  43.2609,
-2.9388
Xtudio Networks S.l.u  
207.244.68.152 US Manassas,
Virginia,
United States,
North America
20109 38.7932,
-77.5366
Leaseweb USA  
67.43.10.129 US Lansing,
Michigan,
United States,
North America
48917 42.7257,
-84.636
Liquid Web  
193.201.224.40 UA Serhiyi,
Chernivtsi,
Ukraine,
Europe
  47.9752,
25.1286
PE Tetyana Mysyk  
86.123.247.64 RO Lugoj,
Judetul Timis,
Romania,
Europe
305500 45.6886,
21.9031
RCS & RDS  
174.136.57.228 US St Louis,
Missouri,
United States,
North America
63131 38.6143,
-90.4444
Colo4, LLC  
95.140.36.2 HU Budapest,
Budapest fovaros,
Hungary,
Europe
1012 47.5,
19.0833
Szervernet szervernet.hu
217.9.126.231 DE Germany,
Europe
  51,
9
Versatel Deutschland  
162.144.66.10 US Provo,
Utah,
United States,
North America
84606 40.2181,
-111.6133
Unified Layer unifiedlayer.com
52.90.51.76 US Ashburn,
Virginia,
United States,
North America
20147 39.0335,
-77.4838
Amazon Technologies  
88.120.254.31 RU Russia,
Europe
  55.75,
37.6166
JSC ISPsystem  
166.63.124.122 US Columbus,
Ohio,
United States,
North America
43228 39.9649,
-83.1383
Ecommerce Corporation cloudix.com
71.43.100.242 US Orlando,
Florida,
United States,
North America
32810 28.6185,
-81.4336
Time Warner Cable rr.com
213.163.66.101 NL Netherlands,
Europe
  52.3667,
4.9
i3d B.V. i3d.net
82.76.87.154 RO Lugoj,
Judetul Timis,
Romania,
Europe
305500 45.6886,
21.9031
RCS & RDS rdsnet.ro
80.13.94.242 FR France,
Europe
  48.86,
2.35
Orange wanadoo.fr
149.255.37.187 NL Netherlands,
Europe
  52.3667,
4.9
Swiftway Sp. z o.o. swiftway.net
173.254.61.236 US Provo,
Utah,
United States,
North America
84606 40.2181,
-111.6133
Unified Layer unifiedlayer.com
72.55.186.26 CA Montreal,
Quebec,
Canada,
North America
H3G 45.4987,
-73.5793
iWeb Technologies panelboxmanager.com
192.185.82.205 US Houston,
Texas,
United States,
North America
77092 29.8301,
-95.4739
Websitewelcome.com websitewelcome.com
194.28.172.219 UA Ukraine,
Europe
  50.45,
30.5233
ON-LINE besthosting.ua
62.219.197.147 IL Nazareth,
Northern District,
Israel,
Asia
  32.7036,
35.2956
Bezeq International bezeqint.net
6.20.160.53 FR Gueret,
Creuse,
Limousin,
France,
Europe
23000 46.2667,
1.8667
Adista SAS  
198.57.247.139 US Provo,
Utah,
United States,
North America
84606 40.2181,
-111.6133
Unified Layer unifiedlayer.com
74.220.207.109 US Provo,
Utah,
United States,
North America
84606 40.2181,
-111.6133
Unified Layer hostmonster.com
46.216.4.146 BY Minsk,
Minsk,
Belarus,
Europe
  53.9,
27.5667
Mobile TeleSystems JLLC  
184.68.101.154 CA Calgary,
Alberta,
Canada,
North America
T3R 51.2021,
-114.2453
Shaw Communications  
103.30.12.10 IN India,
Asia
  20,
77
Surendra Informatics  
79.116.28.250 RO Lugoj,
Judetul Timis,
Romania,
Europe
305500 45.6886,
21.9031
RCS & RDS rdsnet.ro
188.65.115.90 GB United Kingdom,
Europe
  51.5,
-0.13
UK Webhosting  
92.85.171.200 RO Timişoara,
Judetul Timis,
Romania,
Europe
  45.7494,
21.2272
Telekom Romania Communication S.A  
148.251.0.166 DE Germany,
Europe
  51,
9
Hetzner Online GmbH r-99.com
23.101.213.136 AU Sydney,
New South Wales,
Australia,
Oceania
1001 -33.8678,
151.2073
Microsoft Corporation, Microsoft Azure  
5.101.138.91 GB Rugby,
Warwickshire,
England,
United Kingdom,
Europe
CV21 52.3913,
-1.2866
UK Dedicated Servers Limited ukservers.com
61.25.200.204 JP Japan,
Asia
  35.69,
139.69
@Home Network Japan home.ne.jp
71.165.35.215 US Baldwin Park,
California,
United States,
North America
91706 34.0964,
-117.9668
Verizon Internet Services verizon.net
87.106.129.221 DE Germany,
Europe
  51,
9
1&1 Internet AG  
193.111.140.181 DE Dusseldorf,
North Rhine-Westphalia,
Germany,
Europe
40472 51.2686,
6.81
myLoc managed IT AG fastwebserver.de
193.111.140.181 DE Dusseldorf,
North Rhine-Westphalia,
Germany,
Europe
40472 51.2686,
6.81
myLoc managed IT AG fastwebserver.de
24.38.19.220 US Denville,
New Jersey,
United States,
North America
7834 40.8841,
-74.4863
Optimum Online, Denville Township School lightpath.net
64.235.39.67 US Las Vegas,
Nevada,
United States,
North America
89141 35.9781,
-115.2117
Las Vegas NV Datacenter lasvegas-nv-datacenter.com
175.137.252.141 MY Kuala Lumpur,
Kuala Lumpur,
Malaysia,
Asia
52200 3.1553,
101.6668
Telekom Malaysia TM Net

FOSDEM 2016

FOSDEM 2016
Image by FOSDEM licensed under Creative Commons Attribution 2.0 Belgium Licence

This weekend I’m going to FOSDEM 2016, a two-day event organised by volunteers at the Université Libre de Bruxelles to promote the widespread use of free and open source software. In the past I have been at this conference as a participant and this year I am really happy to contribute with a talk .

I’m presenting the work I’m currently doing at Oxford University Press on API-powered dictionaries for digitally under-represented languages. My talk takes place in the Coding for Language Communities devroom on Sunday 31st January. Here are the slides in PDF.

How to map IP addresses in bash prompt

ipv4_address
Image by Indeterminate licensed under Public Domain via Commons

I regularly log into different Linux machines on the cloud and find complicated to identify on which machine I am by looking at the bash prompt. In fact, in cloud environments the bash prompt tends to  show an ugly internal IP address which I can never memorize. Plus, I always fear to log into the production environment by mistake and run experimental commands intended to be executed in the development environment.

In order to overcome my fears, I put together few lines of bash code to be added at the beginning of ~/.bashrc:

# get public IP address
alias myip="curl -s http://whatismyip.akamai.com"

# display environment in bash prompt
case $(myip) in
    1.2.34.567 ) PS1="[\u@\h DEV \W]\$ "
        ;;
    1.2.34.568 ) PS1="[\u@\h STAGE \W]\$ "
        ;;
    1.2.34.569 ) PS1="[\u@\h LIVE \W]\$ "
        ;;
    * ) PS1="[\u@\h OTHER \W]\$ "
        ;;
esac

Line 2 retrieves the IP address from an external web service and map it to myip. Lines 5-14 modify the bash prompt (PS1) based on the value of myip and each IP address (e.g. 1.2.34.567) is mapped to a string appearing in the prompt (e.g. DEV).

Reference: http://unix.stackexchange.com/questions/22615/how-can-i-get-my-external-ip-address-in-bash

How to transfer files to a remote server

rsync logo
Image by rsync – Fair use licence

While using cloud services like AWS it sometimes occurs to transfer files from a local machine to a remote server which is only accessible via SSH keys. Linux utilities like rsync and scp come to rescue in this case. Here is an example of how to transfer files via rsync.

First, make sure that the destination folder on the remote server is writeable. If not, log into the remote server and change the permissions, e.g.:

sudo chmod 777 /home/ec2-user/destination_folder

Then go back to the local machine and transfer the file to the remote machine:

rsync -a --progress -e "ssh -i your_ssh_key.pem" file.zip ec2-user@52.1.234.567:/home/ec2-user/destination_folder

where your_ssh_key.pem is the SSH private key to access the remote server, file.zip is the file to transfer, ec2-user is the user on the remote server, 52.1.234.567 is the IP address of the remote server, and /home/ec2-user/destination_folder is the folder on the remote server where you want to transfer the file to.

rsync is a truly file synchronization tool and comes with lots of options, look at its man page for more examples.

DockerCon Europe 2015

DockerCon Europe 2015
Image DockerCon Europe 2015 ©

This week I am attending DockerCon Europe 2015 in Barcelona where I am going to talk about Continuous Integration with Jenkins, Docker and Compose. This is a reviewed version of a previous blog post on Jenkins and Docker. Here are the slides in PDF.

Update: DockerCon put online the full video of the presentation. Sample code is available on my GitHub account.

How to plot functions with LaTeX

Saddle surface

Following on a previous blog post on drawing fractals with LaTeX, today I am going to plot functions using LaTeX. As in the previous post, let’s start by preparing the LaTeX environment from scratch:

sudo apt-get install latex209-base latex209-bin latexmk texlive-full texlive-math-extra texlive-extra-utils texlive-generic-extra texlive-latex-extra

The pgfplots package for plotting functions and the standalone package for displaying the graph in a single document need to be installed manually as they are not currently present in the Ubuntu repositories:

# navigate to your local texmf folder (could also be ~/texmf or ~/.texmf)
cd ~/.texmf-var
# download the pgfplots package
wget http://sourceforge.net/projects/pgfplots/files/pgfplots/1.7/pgfplots_1.7.tds.zip/download
# unzip it in the texmf folder
unzip pgfplots_1.7.tds.zip -d .
# download the standalone package
wget  http://mirrors.ctan.org/install/macros/latex/contrib/standalone.tds.zip
# unzip it in the texmf folder
unzip standalone.tds.zip -d .
# update the texmf folder
texhash ~/.texmf-var
# remove the zip files
rm standalone.tds.zip pgfplots_1.7.tds.zip

To get started I am going to plot two simple functions which can be mathematically expressed as follows:

y = x * x
y = x / 2

The first function is a linear function whose graph is a straight line. The second one is a quadratic function whose graph is a parabola. Here is the code to plot them with LaTeX:

\documentclass[border=2pt]{standalone}
\usepackage{pgfplots}
\pgfplotsset{compat=newest}
\begin{document}
\begin{tikzpicture}
    \begin{axis}[xmax=9,ymax=9,samples=50,grid=major,xlabel={x axis},ylabel={y axis},title={Linear and Quadratic Functions}]
    \addplot[blue, ultra thick](x,x*x);
    \addplot[red, ultra thick](x,x/2);
\end{axis}
\end{tikzpicture}
\end{document}

To compile the tex file into pdf from the command line you can use the latexmk utility previously installed and open the generated file with your preferred PDF editor (I use evince here).

latexmk -pdf plot_functions.tex
evince plot_functions.pdf &

The plotted functions look like this (and like this in PDF):

Plotting linear and quadratic functionsThe next graph is obtained by joining a sequence of points whose coordinates on the x and y axis are clearly visible in the code (this example is adapted from the beautiful PGFPlots Gallery):

\documentclass[border=2pt]{standalone}
\usepackage{pgfplots}
\pgfplotsset{compat=newest}
\begin{document}

\begin{tikzpicture} 
    \begin{axis}[ xlabel=Cost, ylabel=Error] 
        \addplot[color=red,mark=x] coordinates {
        (2,-2.8559703) 
        (3,-3.5301677) 
        (4,-4.3050655) 
        (5,-5.1413136) 
        (6,-6.0322865) 
        (7,-6.9675052) 
        (8,-7.9377747)
    }; 
    \end{axis} 
\end{tikzpicture}

\end{document}

Again, I use latexmk to compile the code and evince to open the PDF.

latexmk -pdf plot_points.tex
evince plot_points.pdf &

The plotted function looks like this (and like this in PDF):

plot pointsFinally, I’m going to draw a saddle surface. This is an example of hyperbolic geometry, a non-Eucledian geometry where the postulate of parallel lines doesn’t hold. Popular examples of hyperbolic geometry are the Pringles crisps and print works by the Dutch artist M. C. Escher. So here is the LaTeX code to generate a saddle surface:

\documentclass[border=2pt]{standalone}
\usepackage{pgfplots}
\pgfplotsset{compat=newest}

\pagestyle{empty}

\usepgfplotslibrary{patchplots}

\begin{document}
\begin{tikzpicture}
    \begin{axis}
    \addplot3[patch,patch refines=3,
		shader=faceted interp,
		patch type=biquadratic] 
    table[z expr=x^2-y^2]
    {
        x  y
        -2 -2
        2  -2
        2  2
        -2 2
        0  -2
        2  0
        0  2
        -2 0
        0  0
    };
    \end{axis}
\end{tikzpicture}
\end{document}

As usual, I compile it with latexmk and open it with evince.

latexmk -pdf saddle.tex
evince saddle.pdf &

And here is the beautiful saddle surface (and here is the associated PDF):

Saddle surface
Saddle surface

Sources:

Mihalis Tsoukalos, LaTeX: Make text beautiful, Linux Format 201, Summer 2015

http://pgfplots.sourceforge.net/gallery.html