Amazon Inspector is an automated security assessment service on Amazon Web Services (AWS). It allows to identify security vulnerabilities at operating system and network levels by scanning the host against a knowledge base of security best practices and rules.

I recently integrated Amazon Inspector to run in a Jenkins job so that security testing can be automated and performed prior to deployment to production.

AWS Configuration

The first thing to do is to set up the assessment target and assessment template in Amazon Inspector. An assessment target allows to select the EC2 instances via their tags in order to include them in the security scan. Here is an example of my assessment target for the EC2 instances tagged as gadictionaries-leap-ogl-stage-v360 :

The assessment template allows to specify the type of scan and its duration and is linked to the assessment target set up above. Here is an example of my assessment template (ARN is masked for security reasons). I selected the Common Vulnerabilities and Exposures (CVE) rule package scanning for 15 minutes (one hour is the recommended duration time to reliable results).

Jenkins Configuration

We now move to the Jenkins configuration in order to run the security scan via a Jenkins job instead of using the AWS console.

The first thing to do is to make sure that openssh is installed on the instance where Jenkins is running and on the host you want to check. For example, on Ubuntu you can install openssh with:

sudo apt-get install openssh-server

Then install the SSH Agent plugin in Jenkins. This will provide Jenkins with SSH credentials to automatically login into a machine on the cloud. Add the credentials in Jenkins -> Credentials -> System -> Global credentials (unrestricted) -> Add credentials -> SSH Username with private key. This is an example of my credentials for user jenkins (private key details are obfuscated):

Then create a Jenkins job and select the SSH agent credentials for user jenkins in Build Environment:

This will allow Jenkins to ssh into the machine with the private key stored securely (make sure you only grant permission to configure Jenkins to administrators otherwise your private keys are not safe).

I like to parameterize my builds so that I can run Amazon Inspector on a specific EC2 instance within a given Elastic Beanstalk stack:

Then we set up build and post-build actions. The build executes a shell script invoke_aws_inspector.sh pulled from the version control system. The post-build action provides the location of the JUnit file.

The shell script invoke_aws_inspector.sh looks like this:

# check parameters expected from Jenkins job
if [[ -n "$HOSTNAME" ]] && [[ -n "$STACK" ]]; then
	# install and start AWS Inspector agent on host
	ssh -T -o StrictHostKeyChecking=no ec2-user@$HOSTNAME << 'EOF'
	curl -O https://d1wk0tztpsntt1.cloudfront.net/linux/latest/install
	sudo bash install
	sudo /etc/init.d/awsagent start
	sudo /opt/aws/awsagent/bin/awsagent status
	exit
EOF
	# run AWS Inspector from localhost
	export AWS_DEFAULT_REGION=us-east-1
	python execute_aws_inspector.py
	
	# stop and uninstall AWS Inspector agent on host
	ssh -T -o StrictHostKeyChecking=no ec2-user@$HOSTNAME << 'EOF'
	sudo /etc/init.d/awsagent stop &&
	sudo yum remove -y AwsAgent
EOF
else
    echo "ERROR! Parameters HOSTNAME and STACK required from Jenkins job security_checks_aws_inspector"
    exit 1
fi

The shell script works as follows:

• line 4 allows Jenkins to ssh into a host (I’m using AWS EC2 as you can guess by the username ec2-user, replace it with your default username but do not user root). Note that the environment variable $HOSTNAME is passed from the parameter we set up earlier. The EOF allows to run a sequence of commands directly on the host so that you don’t have to disconnect every time. The single quotes are important, don’t skip them!
• lines 5-8 install and start the Amazon Inspector agent on the host
• lines 12-13 configure and set up a Python script execute_aws_inspector.py  for running Amazon Inspector (we’ll see it in a minute)
• lines 16-18 remove the Amazon Inspector agent so that no trace is left on the host
• the final EOF disconnect Jenkins from host

The Python script execute_aws_inspector.py uses the Boto3 library for interacting with AWS services. The script looks like this:

import boto3
import os, sys
import datetime, time
import xml.etree.cElementTree as etree

# initialize boto library for AWS Inspector
client = boto3.client('inspector')

# set assessment template for stack
stack = os.environ['STACK']

if stack == 'gadictionaries-leap-ogl-stage-v360':
	assessmentTemplate  = 'arn:aws:inspector:us-east-1:XXXXXXXXXXXXX:target/XXXXXXXXXXXXX/template/XXXXXXXXXXXXX'
elif stack == 'gadictionaries-leap-odenoad-stage-v350':
	assessmentTemplate  = 'arn:aws:inspector:us-east-1:XXXXXXXXXXXXX:target/XXXXXXXXXXXXX/template/XXXXXXXXXXXXX'
else:
	sys.exit('You must provide a supported stack name (either gadictionaries-leap-ogl-stage-v360 or gadictionaries-leap-odenoad-stage-v350')

# start assessment run
assessment = client.start_assessment_run(
	assessmentTemplateArn = assessmentTemplate,
	assessmentRunName = datetime.datetime.now().isoformat()
)

# wait for the assessment to finish
time.sleep(1020)

# list findings
findings = client.list_findings(
	assessmentRunArns = [
		assessment['assessmentRunArn'],
	],
	filter={
		'severities': [
			'High','Medium',
		],
	},
	maxResults=100
)

# describe findings and output to JUnit 
testsuites = etree.Element("testsuites")
testsuite = etree.SubElement(testsuites, "testsuite", name="Common Vulnerabilities and Exposures-1.1")

for item in findings['findingArns']:
	description = client.describe_findings(
		findingArns=[
			item,
		],
		locale='EN_US'
	)

	for item in description['findings']:
		testcase = etree.SubElement(testsuite, "testcase", name=item['severity'] + ' - ' + item['title'])
		etree.SubElement(testcase, "error", message=item['description']).text = item['recommendation']

tree = etree.ElementTree(testsuites)
tree.write("inspector-junit-report.xml")

The Python script works as follows:

• lines 10-17 read the environment variable set in the parameterized build and select the correct template (I set up two different template for two different stacks, the ARNs are obfuscated for security reasons)
• lines 20-26 run the assessment template and waits a bit longer than 15 minutes so that the scan can finish
• lines 29-39 filters findings with severities High and Medium
• lines 42-58 serialize the findings into a JUnit report so that they can be automatically read by Jenkins

Finally, here is an example of Test Result Trend and JUnit test results showing security vulnerabilities on an EC2 instance running unpatched packages:

Happy security testing with Jenkins and Amazon Inspector!