How to download an entire S3 bucket recursively

Amazon S3 bucket
Image:  Amazon Web Services, Inc. ©

Sometimes I have the need to keep a local copy of an S3 bucket. Using the AWS console is ok if you just have few objects in the S3 bucket. But what do you do if you have hundreds of objects in your S3 bucket? The aws cli comes to rescue with this simple command:

aws s3 cp --recursive s3://my_s3_bucket .

The recursive flag downloads the entire S3 bucket recursively into the local directory (that’s what the dot at the end is for). The operation may take some time depending on the number of objects stored in the S3 bucket so be patient!

How to Validate a Jenkinsfile

Jenkins Pipeline
Image ©

As I’m using more and more often Jenkins Pipelines, I found the need to validate a Jenkinsfile in order to fix syntax errors before committing the file into version control and running the pipeline job in Jenkins. This can saves some time during development and allows to follow best practices when writing a Jenkinsfile as validation returns both errors and warnings.

There are several ways to validate a Jenkinsfile and some editors like VS Code even have a built-in linter. Personally, the easiest way I found is to validate a Jenkinsfile is to run the following command from the command line (provided you have Jenkins running somewhere):

curl --user username:password -X POST -F "jenkinsfile=<Jenkinsfile" http://jenkins-url:8080/pipeline-model-converter/validate

Note the following:

  1. If your Jenkins is authenticating users, you need to pass the username and password otherwise you can omit that part.
  2. By default, this command expects your Jenkinsfile to be called Jenkinsfile. If not, change the name in the command.
  3. Replace jenkins_url and possibly port 8080 based on the URL and port where you are running Jenkins. You can also use localhost as URL if you are running Jenkins on your machine.

If the Jenkinsfile validates, it will show a message like this one:

Jenkinsfile successfully validated.

Or, if you forgot to use steps within stage in your Jenkinsfile, the validation will flag an error like this:

Errors encountered validating Jenkinsfile:
WorkflowScript: 10: Unknown stage section "git". Starting with version 0.5, steps in a stage must be in a ‘steps’ block. @ line 10, column 9.
           stage('Checkout Code') {

WorkflowScript: 10: Expected one of "steps", "stages", or "parallel" for stage "Checkout Code" @ line 10, column 9.
           stage('Checkout Code') {

Happy validation!

How to encrypt and decrypt emails and files

Privacy Encryption
Image  Richard Patterson CC BY 2.0

Somewhere I read that sending unencrypted email is like sending postcards: anyone can potentially read them. This is not nice for privacy but becomes very dangerous when the content of the email or attached files contains secrets like passwords, access keys, etc. Anyone who can get hold of your email can also potentially access your systems.

For sending encrypted email I generally use Enigmail which is data encryption and decryption extension for the Thunderbird email client. I also used Mailvelope which is an add-on for Firefox and Chrome allowing to integrate encryption in webmail providers such as Gmail, Outlook, etc. These tools simplify the encryption/decryption process, especially if you are not familiar with it.

However, it has occurred to me to have to encrypt large files containing data dumps. The challenge with email extensions is that they don’t allow you to send email with such huge attachments. Plus, Mailvelope doesn’t allow to encrypt files larger than 25 MB. This is when knowing how to encrypt and decrypt a file via the command line comes in handy. You can easily upload a large encrypted file on an FTP server or cloud hosting service without worrying that the file will end in the wrong hands. As a bonus, an encrypted file is generally smaller than a non-encrypted file so the upload is also quicker.

The encryption process requires to first get the GPG public key from the person you want to send the encrypted file or email to. Once you have the recipient’s public key, you can encrypt a file with that key. You send the email or upload the file and then ask the recipient to decrypt it at their end using their GPG private key. I’m going to cover both processes. Note that this is also useful in order to encrypt the content of an email that you want to keep secret and send it as attachment in a non-encrypted email.

Generate GPG public and private keys

  1. Install gpg or gpg2 on Linux or MacOS. This is generally part of the standard packages, for example on Ubuntu:
    sudo apt install gnupg2

    If you are on Windows, you can use Cygwin and install gpg or use the GnuPG utility which should work similarly (although I have not tried it).

  2. Generate a GPG key and follow the instructions. I recommend selecting RSA and RSA (default) as kind of key and 4096 as keysize of the key:
    gpg2 --gen-key
  3. You should now have two files in .gnupg within your home directory (e.g. /home/sandro/.gnupg):
    -- pubring.gpg: this is your public key
    -- secring.gpg: this is your private key

    Verify your public key with:

    gpg2 --list-keys

    Verify your private key with:

    gpg2 --list-secret-keys

Encrypt and decrypt files

You have received a public key from someone and you want to encrypt a file with their public key in order to transmit it securely. The file containing the public key will typically have an extension .gpg or .asc.

  1. Import the public key (e.g. someonekey.asc is the filename of the key):

    gpg2 --import someonekey.asc
  2. Trust the public key ( is the email associated with the key and should be shown as output of the import command):
    gpg2 --edit-key

    You’ll get a prompt command>, type trust and select 5 = I trust ultimately. Type quit to exit.

  3. Encrypt the file with the public key of the user (replace the email address with the email address of the user associated to the public key):
    gpg2 -e -r mysecretdocument.txt
  4. This will generate an encrypted file mysecretdocument.txt.gpg which is smaller than the original file. Transmit the encrypted file and tell the user to decrypt it at their end with the following command:
    gpg2 -o mysecretdocument.txt -d mysecretdocument.txt.gpg

Stay safe and encrypt important emails and files!

Reference: How to easily encrypt a file with GPG on Linux