How to download an entire S3 bucket recursively

Amazon S3 bucket
Image:  Amazon Web Services, Inc. ©

Sometimes I have the need to keep a local copy of an S3 bucket. Using the AWS console is ok if you just have few objects in the S3 bucket. But what do you do if you have hundreds of objects in your S3 bucket? The aws cli comes to rescue with this simple command:

The recursive flag downloads the entire S3 bucket recursively into the local directory (that’s what the dot at the end is for). The operation may take some time depending on the number of objects stored in the S3 bucket so be patient!

How to Validate a Jenkinsfile

Jenkins Pipeline
Image  jenkins.io ©

As I’m using more and more often Jenkins Pipelines, I found the need to validate a Jenkinsfile in order to fix syntax errors before committing the file into version control and running the pipeline job in Jenkins. This can saves some time during development and allows to follow best practices when writing a Jenkinsfile as validation returns both errors and warnings.

There are several ways to validate a Jenkinsfile and some editors like VS Code even have a built-in linter. Personally, the easiest way I found is to validate a Jenkinsfile is to run the following command from the command line (provided you have Jenkins running somewhere):

Note the following:

  1. If your Jenkins is authenticating users, you need to pass the username and password otherwise you can omit that part.
  2. By default, this command expects your Jenkinsfile to be called Jenkinsfile. If not, change the name in the command.
  3. Replace jenkins_url and possibly port 8080 based on the URL and port where you are running Jenkins. You can also use localhost as URL if you are running Jenkins on your machine.

If the Jenkinsfile validates, it will show a message like this one:

Or, if you forgot to use steps within stage in your Jenkinsfile, the validation will flag an error like this:

Happy validation!

How to encrypt and decrypt emails and files

Privacy Encryption
Image  Richard Patterson CC BY 2.0

Somewhere I read that sending unencrypted email is like sending postcards: anyone can potentially read them. This is not nice for privacy but becomes very dangerous when the content of the email or attached files contains secrets like passwords, access keys, etc. Anyone who can get hold of your email can also potentially access your systems.

For sending encrypted email I generally use Enigmail which is data encryption and decryption extension for the Thunderbird email client. I also used Mailvelope which is an add-on for Firefox and Chrome allowing to integrate encryption in webmail providers such as Gmail, Outlook, etc. These tools simplify the encryption/decryption process, especially if you are not familiar with it.

However, it has occurred to me to have to encrypt large files containing data dumps. The challenge with email extensions is that they don’t allow you to send email with such huge attachments. Plus, Mailvelope doesn’t allow to encrypt files larger than 25 MB. This is when knowing how to encrypt and decrypt a file via the command line comes in handy. You can easily upload a large encrypted file on an FTP server or cloud hosting service without worrying that the file will end in the wrong hands. As a bonus, an encrypted file is generally smaller than a non-encrypted file so the upload is also quicker.

The encryption process requires to first get the GPG public key from the person you want to send the encrypted file or email to. Once you have the recipient’s public key, you can encrypt a file with that key. You send the email or upload the file and then ask the recipient to decrypt it at their end using their GPG private key. I’m going to cover both processes. Note that this is also useful in order to encrypt the content of an email that you want to keep secret and send it as attachment in a non-encrypted email.

Generate GPG public and private keys

  1. Install gpg or gpg2 on Linux or MacOS. This is generally part of the standard packages, for example on Ubuntu:

    If you are on Windows, you can use Cygwin and install gpg or use the GnuPG utility which should work similarly (although I have not tried it).

  2. Generate a GPG key and follow the instructions. I recommend selecting RSA and RSA (default) as kind of key and 4096 as keysize of the key:

  3. You should now have two files in .gnupg within your home directory (e.g. /home/sandro/.gnupg):

    Verify your public key with:

    Verify your private key with:

Encrypt and decrypt files

You have received a public key from someone and you want to encrypt a file with their public key in order to transmit it securely. The file containing the public key will typically have an extension .gpg or .asc.

  1. Import the public key (e.g. someonekey.asc is the filename of the key):

  2. Trust the public key ( user@example.com is the email associated with the key and should be shown as output of the import command):

    You’ll get a prompt command>, type trust and select 5 = I trust ultimately. Type quit to exit.

  3. Encrypt the file with the public key of the user (replace the email address with the email address of the user associated to the public key):

  4. This will generate an encrypted file mysecretdocument.txt.gpg which is smaller than the original file. Transmit the encrypted file and tell the user to decrypt it at their end with the following command:

Stay safe and encrypt important emails and files!

Reference: How to easily encrypt a file with GPG on Linux

Oxford Dictionaries API at Google Developer Group Meetup

Oxford Dictionaries API
Image  Oxford University Press ©

This week together with my colleagues Phil and Amos I gave a presentation on the Oxford Dictionaries API at the Google Developer Group Reading & Thames Valley Meetup.

We were kindly hosted by Perusha and Ming from the local Google Developer Group Meetup and got lots of interesting questions from the audience. This also gave us few ideas on how to develop further the Oxford Dictionaries API and what developers may want to do with it in the fields of Natural Language Processing and Machine Learning.

XML Prague 2018

XML Prague
Image  XML Prague ©

This week I am attending XML Prague at the University of Economics College campus in Prague, a conference on markup languages and data on the web. Together with other XSpec developers I am organising the XSpec Users Meetup. I’m also giving a lightning talk in the Schematron Users Meetup on how to test Schematron with XSpec.

Slides of the XSpec Users Meetup are available here whereas my lightning talk on testing Schematron with XSpec is available here.