CD Summit and Jenkins Days 2016

CD Summit and Jenkins Days Berlin 2016
Image by DevOps Connect ©

This week I’m giving a talk about Continuous Security with Jenkins, Docker Bench, and Amazon Inspector at CD Summit & Jenkins Days in Amsterdam and in Berlin. CD Summit & Jenkins Days are a series of conferences in the US and in Europe focusing on Continuous Integration (CI) and Continuous Delivery (CD).

This is the abstract of my talk:

Security testing is often left out from CI/CD pipelines and perceived as an ad hoc and one-off audit performed by external security experts. However, the integration of security testing into a DevOps workflow (aka DevSecOps) allows to achieve security by design and to continuously assess software vulnerabilities within a CI/CD pipeline. But how does security fit in the world of cloud and microservices?

In this talk I show how to leverage tools like Jenkins, Docker Bench , and Amazon Inspector to perform security testing at the operating system and container levels in a cloud environment and how to integrate them into a typical CI/CD workflow. I discuss how these tools can help assessing the risk of security vulnerabilities during development, improving security and compliance, and lower support costs in the long term.

I also present two demos showing how to integrate Docker Bench with Jenkins and how to run Amazon Inspector from Jenkins.

The slides of my talk are available here.

Continuous Security with Jenkins and Amazon Inspector

Amazon Inspector
Image by Amazon Web Services, Inc. ©

Amazon Inspector is an automated security assessment service on Amazon Web Services (AWS). It allows to identify security vulnerabilities at operating system and network levels by scanning the host against a knowledge base of security best practices and rules.

I recently integrated Amazon Inspector to run in a Jenkins job so that security testing can be automated and performed prior to deployment to production.

AWS Configuration

The first thing to do is to set up the assessment target and assessment template in Amazon Inspector. An assessment target allows to select the EC2 instances via their tags in order to include them in the security scan. Here is an example of my assessment target for the EC2 instances tagged as gadictionaries-leap-ogl-stage-v360 :

Assessment TargetThe assessment template allows to specify the type of scan and its duration and is linked to the assessment target set up above. Here is an example of my assessment template (ARN is masked for security reasons). I selected the Common Vulnerabilities and Exposures (CVE) rule package scanning for 15 minutes (one hour is the recommended duration time to reliable results).

Assessment Template

Jenkins Configuration

We now move to the Jenkins configuration in order to run the security scan via a Jenkins job instead of using the AWS console.

The first thing to do is to make sure that openssh is installed on the instance where Jenkins is running and on the host you want to check. For example, on Ubuntu you can install openssh with:

Then install the SSH Agent plugin in Jenkins. This will provide Jenkins with SSH credentials to automatically login into a machine on the cloud. Add the credentials in Jenkins -> Credentials -> System -> Global credentials (unrestricted) -> Add credentials -> SSH Username with private key. This is an example of my credentials for user jenkins (private key details are obfuscated):

set up mock SSH credentialsThen create a Jenkins job and select the SSH agent credentials for user jenkins in Build Environment:

SSH agent credentialsThis will allow Jenkins to ssh into the machine with the private key stored securely (make sure you only grant permission to configure Jenkins to administrators otherwise your private keys are not safe).

I like to parameterize my builds so that I can run Amazon Inspector on a specific EC2 instance within a given Elastic Beanstalk stack:

Parameters

Then we set up build and post-build actions. The build executes a shell script invoke_aws_inspector.sh pulled from the version control system. The post-build action provides the location of the JUnit file.

The shell script invoke_aws_inspector.sh looks like this:

The shell script works as follows:

  • line 4 allows Jenkins to ssh into a host (I’m using AWS EC2 as you can guess by the username ec2-user, replace it with your default username but do not user root). Note that the environment variable $HOSTNAME is passed from the parameter we set up earlier. The EOF allows to run a sequence of commands directly on the host so that you don’t have to disconnect every time. The single quotes are important, don’t skip them!
  • lines 5-8 install and start the Amazon Inspector agent on the host
  • lines 12-13 configure and set up a Python script  execute_aws_inspector.py  for running Amazon Inspector (we’ll see it in a minute)
  • lines 16-18 remove the Amazon Inspector agent so that no trace is left on the host
  • the final EOF disconnect Jenkins from host

The Python script execute_aws_inspector.py uses the Boto3 library for interacting with AWS services. The script looks like this:

The Python script works as follows:

  • lines 10-17 read the environment variable set in the parameterized build and select the correct template (I set up two different template for two different stacks, the ARNs are obfuscated for security reasons)
  • lines 20-26 run the assessment template and waits a bit longer than 15 minutes so that the scan can finish
  • lines 29-39 filters findings with severities High and Medium
  • lines 42-58 serialize the findings into a JUnit report so that they can be automatically read by Jenkins

Finally, here is an example of Test Result Trend and JUnit test results showing security vulnerabilities on an EC2 instance running unpatched packages:

JUnit Report

Happy security testing with Jenkins and Amazon Inspector!

Euralex 2016

Euralex 2016
Image by Euralex 2016 ©

This week I am attending XVII Euralex International Congress, a lexicographic conference held between 6th and 10th September 2016 in Tbilisi, Georgia. The theme of this edition of Euralex is Lexicography and Linguistic Diversity.

On Wednesday 7th September I am giving a software presentation on API-powered dictionaries covering the work my colleagues and I have recently been doing on the Oxford Global Languages (OGL) initiative and the Oxford Dictionaries API. Here are the slides of my talk and the paper included in the conference proceedings.

Continuous Security with Jenkins and Docker Bench

Docker Bench
Image by Docker Inc. ©

Docker Bench is an open source tool for automatically validating the configuration of a host running Docker containers. The tool has been written among others by Diogo Mónica, security lead at Docker, and performs security checks at the container level following Docker’s CIS Benchmark recommendations.

As you would expect, the easiest way to run Docker Bench is via a Docker container. Just make sure you have Docker 1.10 or better, download the Docker image:

and run the Docker container as follows:

This will automatically generate some output as in the animated gif above with an assessment of possible Docker security issues.

I recently combined Docker Bench with Jenkins in order to integrate security testing into a typical DevOps workflow on the cloud – call it DevSecOps if you like buzzwords… This requires a little bit of Jenkins configuration but it’s not too difficult to follow.

The first thing to do is to make sure that openssh is installed on the instance where Jenkins is running and on the host you want to check. For example on Ubuntu you can install openssh with:

Then install the SSH Agent plugin in Jenkins. This will provide Jenkins with SSH credentials to automatically login into a machine on the cloud. Add the credentials in Jenkins -> Credentials -> System -> Global credentials (unrestricted) -> Add credentials -> SSH Username with private key.  This is an example of my credentials for user jenkins (private key details are obfuscated):

set up mock SSH credentialsThen create a Jenkins job and select the SSH agent credentials for user jenkins in Build Environment:

SSH agent credentialsThis will allow Jenkins to SSH into the machine with the private key stored securely (make sure you only grant permission to configure Jenkins to administrators otherwise your private keys are not safe).

I like to parameterize my builds so that I can run Docker Bench on any host reachable with the private key:

parameterize build with hostnameFinally, select Execute shell in the build and paste this shell script (you may want to put it under version control and retrieve it from there via Jenkins):

It works likes this:

  • the first command allows Jenkins to ssh into a host (I’m using AWS EC2 as you can guess by the username ec2-user, replace it with your default username but do not user root). Note that the environment variable $HOSTNAME is passed from the parameter we set up earlier. The EOF allows to run a sequence of commands directly on the host so that you don’t have to disconnect every time. The single quotes are important, don’t skip them!
  • the second command pulls the Docker image for Docker Bench directly on the host
  • the third command runs Docker Bench on the host
  • the forth command removes all exited containers from the host, including Docker Bench once it has finished its job
  • the fifth command remove the Docker image for Docker Bench so that you don’t leave any trace on the host
  • the final EOF disconnect Jenkins from host

The Jenkins console output shows the result of running Docker Bench on a specific host. Now you have to assess the results as you may see several warnings and they may just be false positives. For example, this warning may be acceptable for you:

This means you are not running the latest version of Docker. This may not be an issue (unless Docker released a security release) especially if your Linux distribution hasn’t got the latest version of Docker available in its repositories.

In my case this warning was a false positive:

In fact, I need several containers to communicate between them so that restriction does not apply to my use case.

This warning should be taken much more seriously:

This means you are running a container as root. This is unsecure as if a nasty intruder manages to get inside the containers s/he can run any command in it. Basically, it’s like running a Linux system as root which is a bad security practice.

Once you have assessed your warnings, you may want to filter out the false positives. For example, you can use the Post build task plugin to make the build fail if the build log output contains a warning that you assessed as a security risk. You can use a regular expression to match the pattern identified above.

It would be good to get the Docker Bench output in JUnit format so that Jenkins can understand it natively but this option is currently not implemented in Docker Bench.

Happy security testing with Jenkins and Docker Bench!

API Days Nordic 2016

API Days Nordic 2016
Image by API Days Nordic ©

This week I’m attending API Days Nordic at the University of Tampere (Finland), a technical conference on heavy industry and APIs, government driven API platforms, and APIOps (APIs, DevOps, containers, testing, micro services, and monitoring). This is part of the API Days series of conferences and it is the first time that it is organised in Finland.

My presentation focuses on how to deploy an API gateway with Docker. The slides are available here and the code is on GitHub. This is the summary of my presentation:

An API gateway is a single entry point for APIs in a microservices infrastructure. It provides authentication and authorization layers, routes and load balances requests to API services, and caches previous requests. Being the first entry point of the API, it is crucial to manage and provision it through code rather than using a manual process. Furthermore, replicating its configuration on development and staging environments allows to load test the API gateway and to anticipate issues before it is deployed to production.

I demonstrate the deployment of an API gateway using Docker. Technologies used include:

  • Docker
  • openresty/nginx
  • 3scale API management
  • AWS EC2

I discuss the benefits of using Docker and how it simplifies changes of configuration and deployment to multiple environments. Sample code and brief documentation are available on GitHub.

Why we need more women in technology

female speech
Image: R.L.Trask and B. Mayblin (2000), Introducing Linguistics, Cambridge: Icon Books ©

Few months ago I sat in an interview panel for recruiting a system administrator at work. We interviewed two male candidates who applied for the job. Although they seem to have excellent technical skills they completely lacked soft skills. My colleagues and I tried to gear the interview towards a normal conversation but instead we were constantly faced with monologues featuring obscure tech acronyms and fancy technologies that were not even relevant for the job. I left the room in despair and at the end we decided not to hire any of them and to start the recruitment process from scratch. I remember asking my colleague: “Next time can we please shortlist at least one women?” Unfortunately we haven’t seen any female candidate so far…

I experienced first hand what the British linguist Jennifer Coates pinpoints as the difference between male and female speech. According to her studies, all-male conversations are like this:

male speech
Image: R.L.Trask and B. Mayblin (2000), Introducing Linguistics, Cambridge: Icon Books ©

Men tend to slip into monologues when talking to each other. Women tend to talk differently. All-female conversations look more like this:

female speech
Image: R.L.Trask and B. Mayblin (2000), Introducing Linguistics, Cambridge: Icon Books ©

Now think of software development. Think for example of an open source project on GitHub. Or building a complex software tool that needs to communicate with both legacy and new systems. Does it sound more like a male or female conversation?

Building software is by definition a co-operative enterprise. Software systems are so complex nowadays that need teams of software engineers with difference skills in order to be built. More importantly, these highly skilled engineers need to talk to each other. If tech teams are completely made of men, discussions will tend to slip into monologues: a single software engineer will take the lead and other voices may get ignored or unheard. Conversely, if tech teams are gender mixed, communication will improve and as a result the software will be better.

References:
Jennifer Coates (1986). Women, Men and Language: A Sociolinguistic Account of Gender Differences in Language. London: Longman.

I got inspired by a great blog post written by the Docker developer Jérôme Petazzoni.

My blacklist of IP addresses

blacklist
Image by LaNotizia licensed under Creative Commons Attribution 2.0 Generic Licence

I monitor few websites for friends and had to configure security modules in order to prevent brute force attacks and breaches in the admin interface of the website. For CMS like WordPress and Drupal there are some good security modules like iThemesSecurity, Security Review and Login Security. These modules improve the security of a website by limiting the number of failed login attempts and blocking the suspicious IP address.

Over time I collected a list of  backlisted IP addresses which  I share here. One thing I learnt is that attacks can come from anywhere. I believe that some of these IP addresses belong to a botnet of compromised computers as I could trace their coordinates to some very well-known locations in the world.

Here below is the table of blacklisted IP addresses. The same list is in this CSV file. The data is courtesy of MaxMind and the list was last updated on 16th April 2016.

IP Address Country Code Location Postal Code Coordinates ISP Domain
46.20.12.30 TR Bursa,
Bursa,
Turkey,
Asia
16245 40.2719,
29.0983
Dgn Teknoloji Bilisim Yayincilik Sanayi Ve Limited ttnetdc.com
203.130.196.150 ID Jakarta,
Daerah Khusus Ibukota Jakarta,
Indonesia,
Asia
-6.1744,
106.8294
Pt Telekomunikasi Indonesia
109.195.228.208 RU Rostov-on-Don,
Rostov,
Russia,
Europe
344000 47.2364,
39.7139
JSC ER-Telecom Holding ertelecom.ru
66.91.141.42 US Honolulu,
Hawaii,
United States,
North America
96805 21.3069,
-157.8583
Time Warner Cable rr.com
185.53.78.144 UA Ukraine,
Europe
50.45,
30.5233
LLC Gigatel rv.ua
91.200.12.11 UA Ukraine,
Europe
50.45,
30.5233
PP SKS-Lugan
37.59.56.6 FR France,
Europe
48.86,
2.35
OVH SAS ovh.net
37.252.102.136 ES Basque Country,
Spain,
Europe
43.2609,
-2.9388
Xtudio Networks S.l.u
207.244.68.152 US Manassas,
Virginia,
United States,
North America
20109 38.7932,
-77.5366
Leaseweb USA
67.43.10.129 US Lansing,
Michigan,
United States,
North America
48917 42.7257,
-84.636
Liquid Web
193.201.224.40 UA Serhiyi,
Chernivtsi,
Ukraine,
Europe
47.9752,
25.1286
PE Tetyana Mysyk
86.123.247.64 RO Lugoj,
Judetul Timis,
Romania,
Europe
305500 45.6886,
21.9031
RCS & RDS
174.136.57.228 US St Louis,
Missouri,
United States,
North America
63131 38.6143,
-90.4444
Colo4, LLC
95.140.36.2 HU Budapest,
Budapest fovaros,
Hungary,
Europe
1012 47.5,
19.0833
Szervernet szervernet.hu
217.9.126.231 DE Germany,
Europe
51,
9
Versatel Deutschland
162.144.66.10 US Provo,
Utah,
United States,
North America
84606 40.2181,
-111.6133
Unified Layer unifiedlayer.com
52.90.51.76 US Ashburn,
Virginia,
United States,
North America
20147 39.0335,
-77.4838
Amazon Technologies
88.120.254.31 RU Russia,
Europe
55.75,
37.6166
JSC ISPsystem
166.63.124.122 US Columbus,
Ohio,
United States,
North America
43228 39.9649,
-83.1383
Ecommerce Corporation cloudix.com
71.43.100.242 US Orlando,
Florida,
United States,
North America
32810 28.6185,
-81.4336
Time Warner Cable rr.com
213.163.66.101 NL Netherlands,
Europe
52.3667,
4.9
i3d B.V. i3d.net
82.76.87.154 RO Lugoj,
Judetul Timis,
Romania,
Europe
305500 45.6886,
21.9031
RCS & RDS rdsnet.ro
80.13.94.242 FR France,
Europe
48.86,
2.35
Orange wanadoo.fr
149.255.37.187 NL Netherlands,
Europe
52.3667,
4.9
Swiftway Sp. z o.o. swiftway.net
173.254.61.236 US Provo,
Utah,
United States,
North America
84606 40.2181,
-111.6133
Unified Layer unifiedlayer.com
72.55.186.26 CA Montreal,
Quebec,
Canada,
North America
H3G 45.4987,
-73.5793
iWeb Technologies panelboxmanager.com
192.185.82.205 US Houston,
Texas,
United States,
North America
77092 29.8301,
-95.4739
Websitewelcome.com websitewelcome.com
194.28.172.219 UA Ukraine,
Europe
50.45,
30.5233
ON-LINE besthosting.ua
62.219.197.147 IL Nazareth,
Northern District,
Israel,
Asia
32.7036,
35.2956
Bezeq International bezeqint.net
6.20.160.53 FR Gueret,
Creuse,
Limousin,
France,
Europe
23000 46.2667,
1.8667
Adista SAS
198.57.247.139 US Provo,
Utah,
United States,
North America
84606 40.2181,
-111.6133
Unified Layer unifiedlayer.com
74.220.207.109 US Provo,
Utah,
United States,
North America
84606 40.2181,
-111.6133
Unified Layer hostmonster.com
46.216.4.146 BY Minsk,
Minsk,
Belarus,
Europe
53.9,
27.5667
Mobile TeleSystems JLLC
184.68.101.154 CA Calgary,
Alberta,
Canada,
North America
T3R 51.2021,
-114.2453
Shaw Communications
103.30.12.10 IN India,
Asia
20,
77
Surendra Informatics
79.116.28.250 RO Lugoj,
Judetul Timis,
Romania,
Europe
305500 45.6886,
21.9031
RCS & RDS rdsnet.ro
188.65.115.90 GB United Kingdom,
Europe
51.5,
-0.13
UK Webhosting
92.85.171.200 RO Timişoara,
Judetul Timis,
Romania,
Europe
45.7494,
21.2272
Telekom Romania Communication S.A
148.251.0.166 DE Germany,
Europe
51,
9
Hetzner Online GmbH r-99.com
23.101.213.136 AU Sydney,
New South Wales,
Australia,
Oceania
1001 -33.8678,
151.2073
Microsoft Corporation, Microsoft Azure
5.101.138.91 GB Rugby,
Warwickshire,
England,
United Kingdom,
Europe
CV21 52.3913,
-1.2866
UK Dedicated Servers Limited ukservers.com
61.25.200.204 JP Japan,
Asia
35.69,
139.69
@Home Network Japan home.ne.jp
71.165.35.215 US Baldwin Park,
California,
United States,
North America
91706 34.0964,
-117.9668
Verizon Internet Services verizon.net
87.106.129.221 DE Germany,
Europe
51,
9
1&1 Internet AG
193.111.140.181 DE Dusseldorf,
North Rhine-Westphalia,
Germany,
Europe
40472 51.2686,
6.81
myLoc managed IT AG fastwebserver.de
193.111.140.181 DE Dusseldorf,
North Rhine-Westphalia,
Germany,
Europe
40472 51.2686,
6.81
myLoc managed IT AG fastwebserver.de
24.38.19.220 US Denville,
New Jersey,
United States,
North America
7834 40.8841,
-74.4863
Optimum Online, Denville Township School lightpath.net
64.235.39.67 US Las Vegas,
Nevada,
United States,
North America
89141 35.9781,
-115.2117
Las Vegas NV Datacenter lasvegas-nv-datacenter.com
175.137.252.141 MY Kuala Lumpur,
Kuala Lumpur,
Malaysia,
Asia
52200 3.1553,
101.6668
Telekom Malaysia TM Net

FOSDEM 2016

FOSDEM 2016
Image by FOSDEM licensed under Creative Commons Attribution 2.0 Belgium Licence

This weekend I’m going to FOSDEM 2016, a two-day event organised by volunteers at the Université Libre de Bruxelles to promote the widespread use of free and open source software. In the past I have been at this conference as a participant and this year I am really happy to contribute with a talk .

I’m presenting the work I’m currently doing at Oxford University Press on API-powered dictionaries for digitally under-represented languages. My talk takes place in the Coding for Language Communities devroom on Sunday 31st January. Here are the slides in PDF.

How to map IP addresses in bash prompt

ipv4_address
Image by Indeterminate licensed under Public Domain via Commons

I regularly log into different Linux machines on the cloud and find complicated to identify on which machine I am by looking at the bash prompt. In fact, in cloud environments the bash prompt tends to  show an ugly internal IP address which I can never memorize. Plus, I always fear to log into the production environment by mistake and run experimental commands intended to be executed in the development environment.

In order to overcome my fears, I put together few lines of bash code to be added at the beginning of ~/.bashrc:

Line 2 retrieves the IP address from an external web service and map it to myip. Lines 5-14 modify the bash prompt (PS1) based on the value of myip and each IP address (e.g. 1.2.34.567) is mapped to a string appearing in the prompt (e.g. DEV).

Reference: http://unix.stackexchange.com/questions/22615/how-can-i-get-my-external-ip-address-in-bash