This week I’m giving a talk about Continuous Security with Jenkins, Docker Bench, and Amazon Inspector at CD Summit & Jenkins Days in Amsterdam and in Berlin. CD Summit & Jenkins Days are a series of conferences in the US and in Europe focusing on Continuous Integration (CI) and Continuous Delivery (CD).
This is the abstract of my talk:
Security testing is often left out from CI/CD pipelines and perceived as an ad hoc and one-off audit performed by external security experts. However, the integration of security testing into a DevOps workflow (aka DevSecOps) allows to achieve security by design and to continuously assess software vulnerabilities within a CI/CD pipeline. But how does security fit in the world of cloud and microservices?
In this talk I show how to leverage tools like Jenkins, Docker Bench , and Amazon Inspector to perform security testing at the operating system and container levels in a cloud environment and how to integrate them into a typical CI/CD workflow. I discuss how these tools can help assessing the risk of security vulnerabilities during development, improving security and compliance, and lower support costs in the long term.
Amazon Inspector is an automated security assessment service on Amazon Web Services (AWS). It allows to identify security vulnerabilities at operating system and network levels by scanning the host against a knowledge base of security best practices and rules.
I recently integrated Amazon Inspector to run in a Jenkins job so that security testing can be automated and performed prior to deployment to production.
The first thing to do is to set up the assessment target and assessment template in Amazon Inspector. An assessment target allows to select the EC2 instances via their tags in order to include them in the security scan. Here is an example of my assessment target for the EC2 instances tagged as
The assessment template allows to specify the type of scan and its duration and is linked to the assessment target set up above. Here is an example of my assessment template (ARN is masked for security reasons). I selected the Common Vulnerabilities and Exposures (CVE) rule package scanning for 15 minutes (one hour is the recommended duration time to reliable results).
We now move to the Jenkins configuration in order to run the security scan via a Jenkins job instead of using the AWS console.
The first thing to do is to make sure that openssh is installed on the instance where Jenkins is running and on the host you want to check. For example, on Ubuntu you can install openssh with:
sudo apt-get install openssh-server
Then install the SSH Agent plugin in Jenkins. This will provide Jenkins with SSH credentials to automatically login into a machine on the cloud. Add the credentials in Jenkins -> Credentials -> System -> Global credentials (unrestricted) -> Add credentials -> SSH Username with private key. This is an example of my credentials for user jenkins (private key details are obfuscated):
Then create a Jenkins job and select the SSH agent credentials for user jenkins in Build Environment:
This will allow Jenkins to ssh into the machine with the private key stored securely (make sure you only grant permission to configure Jenkins to administrators otherwise your private keys are not safe).
I like to parameterize my builds so that I can run Amazon Inspector on a specific EC2 instance within a given Elastic Beanstalk stack:
Then we set up build and post-build actions. The build executes a shell script
invoke_aws_inspector.sh pulled from the version control system. The post-build action provides the location of the JUnit file.
The shell script
invoke_aws_inspector.sh looks like this:
echo"ERROR! Parameters HOSTNAME and STACK required from Jenkins job security_checks_aws_inspector"
The shell script works as follows:
line 4 allows Jenkins to ssh into a host (I’m using AWS EC2 as you can guess by the username ec2-user, replace it with your default username but do not user root). Note that the environment variable $HOSTNAME is passed from the parameter we set up earlier. The EOF allows to run a sequence of commands directly on the host so that you don’t have to disconnect every time. The single quotes are important, don’t skip them!
lines 5-8 install and start the Amazon Inspector agent on the host
lines 12-13 configure and set up a Python script
execute_aws_inspector.py for running Amazon Inspector (we’ll see it in a minute)
lines 16-18 remove the Amazon Inspector agent so that no trace is left on the host
the final EOF disconnect Jenkins from host
The Python script
execute_aws_inspector.py uses the Boto3 library for interacting with AWS services. The script looks like this:
lines 10-17 read the environment variable set in the parameterized build and select the correct template (I set up two different template for two different stacks, the ARNs are obfuscated for security reasons)
lines 20-26 run the assessment template and waits a bit longer than 15 minutes so that the scan can finish
lines 29-39 filters findings with severities High and Medium
lines 42-58 serialize the findings into a JUnit report so that they can be automatically read by Jenkins
Finally, here is an example of Test Result Trend and JUnit test results showing security vulnerabilities on an EC2 instance running unpatched packages:
Happy security testing with Jenkins and Amazon Inspector!