I wrote a post on the Medium blog space of my new employer The Scale Factory and described how to update RDS SSL certificates in AWS. The blog post is available on Medium.
Update your RDS SSL certificates
How to download an entire S3 bucket recursively
Sometimes I have the need to keep a local copy of an S3 bucket. Using the AWS console is ok if you just have few objects in the S3 bucket. But what do you do if you have hundreds of objects in your S3 bucket? The aws cli comes to rescue with this simple command:
|
1 |
aws s3 cp --recursive s3://my_s3_bucket . |
The recursive flag downloads the entire S3 bucket recursively into the local directory (that’s what the dot at the end is for). The operation may take some time depending on the number of objects stored in the S3 bucket so be patient!
AWS blog post
I recently wrote a blog post for the AWS blog. The blog post is available on the AWS Public Sector blog and describes how we are using AWS in the Dictionaries department of Oxford University Press to make high-quality language data available to licensees, software developers, and the wider public.
How to Validate a Jenkinsfile
As I’m using more and more often Jenkins Pipelines, I found the need to validate a Jenkinsfile in order to fix syntax errors before committing the file into version control and running the pipeline job in Jenkins. This can saves some time during development and allows to follow best practices when writing a Jenkinsfile as validation returns both errors and warnings.
There are several ways to validate a Jenkinsfile and some editors like VS Code even have a built-in linter. Personally, the easiest way I found is to validate a Jenkinsfile is to run the following command from the command line (provided you have Jenkins running somewhere):
|
1 |
curl --user username:password -X POST -F "jenkinsfile=<Jenkinsfile" http://jenkins-url:8080/pipeline-model-converter/validate |
Note the following:
- If your Jenkins is authenticating users, you need to pass the username and password otherwise you can omit that part.
- By default, this command expects your Jenkinsfile to be called Jenkinsfile. If not, change the name in the command.
- Replace jenkins_url and possibly port 8080 based on the URL and port where you are running Jenkins. You can also use localhost as URL if you are running Jenkins on your machine.
If the Jenkinsfile validates, it will show a message like this one:
|
1 |
Jenkinsfile successfully validated. |
Or, if you forgot to use steps within stage in your Jenkinsfile, the validation will flag an error like this:
|
1 2 3 4 5 6 7 8 |
Errors encountered validating Jenkinsfile: WorkflowScript: 10: Unknown stage section "git". Starting with version 0.5, steps in a stage must be in a ‘steps’ block. @ line 10, column 9. stage('Checkout Code') { ^ WorkflowScript: 10: Expected one of "steps", "stages", or "parallel" for stage "Checkout Code" @ line 10, column 9. stage('Checkout Code') { ^ |
Happy validation!
Markup UK 2019
This weekend I am attending Markup UK at King’s College London , a 2 day conference on XML and other markup technologies. I am presenting a paper on running XSpec tests in a serverless environment with AWS Lambda (which I blatantly titled XSpec in the Cloud with Diamonds). The paper is available here and the slides of my presentation are available here.
How to encrypt and decrypt emails and files
Somewhere I read that sending unencrypted email is like sending postcards: anyone can potentially read them. This is not nice for privacy but becomes very dangerous when the content of the email or attached files contains secrets like passwords, access keys, etc. Anyone who can get hold of your email can also potentially access your systems.
For sending encrypted email I generally use Enigmail which is data encryption and decryption extension for the Thunderbird email client. I also used Mailvelope which is an add-on for Firefox and Chrome allowing to integrate encryption in webmail providers such as Gmail, Outlook, etc. These tools simplify the encryption/decryption process, especially if you are not familiar with it.
However, it has occurred to me to have to encrypt large files containing data dumps. The challenge with email extensions is that they don’t allow you to send email with such huge attachments. Plus, Mailvelope doesn’t allow to encrypt files larger than 25 MB. This is when knowing how to encrypt and decrypt a file via the command line comes in handy. You can easily upload a large encrypted file on an FTP server or cloud hosting service without worrying that the file will end in the wrong hands. As a bonus, an encrypted file is generally smaller than a non-encrypted file so the upload is also quicker.
The encryption process requires to first get the GPG public key from the person you want to send the encrypted file or email to. Once you have the recipient’s public key, you can encrypt a file with that key. You send the email or upload the file and then ask the recipient to decrypt it at their end using their GPG private key. I’m going to cover both processes. Note that this is also useful in order to encrypt the content of an email that you want to keep secret and send it as attachment in a non-encrypted email.
Generate GPG public and private keys
- Install gpg or gpg2 on Linux or MacOS. This is generally part of the standard packages, for example on Ubuntu:
1sudo apt install gnupg2
If you are on Windows, you can use Cygwin and install gpg or use the GnuPG utility which should work similarly (although I have not tried it).
- Generate a GPG key and follow the instructions. I recommend selecting RSA and RSA (default) as kind of key and 4096 as keysize of the key:
1gpg2 --gen-key
- You should now have two files in
.gnupg within your home directory (e.g.
/home/sandro/.gnupg):
12-- pubring.gpg: this is your public key-- secring.gpg: this is your private key
Verify your public key with:
1gpg2 --list-keysVerify your private key with:
1gpg2 --list-secret-keys
Encrypt and decrypt files
You have received a public key from someone and you want to encrypt a file with their public key in order to transmit it securely. The file containing the public key will typically have an extension .gpg or .asc.
-
Import the public key (e.g.
someonekey.asc is the filename of the key):
1gpg2 --import someonekey.asc
- Trust the public key (
user@example.com is the email associated with the key and should be shown as output of the import command):
1gpg2 --edit-key user@example.com
You’ll get a prompt command>, type trust and select 5 = I trust ultimately. Type quit to exit.
- Encrypt the file with the public key of the user (replace the email address with the email address of the user associated to the public key):
1gpg2 -e -r user@example.com mysecretdocument.txt
- This will generate an encrypted file
mysecretdocument.txt.gpg which is smaller than the original file. Transmit the encrypted file and tell the user to decrypt it at their end with the following command:
1gpg2 -o mysecretdocument.txt -d mysecretdocument.txt.gpg
Stay safe and encrypt important emails and files!
Oxford Dictionaries API at Google Developer Group Meetup
This week together with my colleagues Phil and Amos I gave a presentation on the Oxford Dictionaries API at the Google Developer Group Reading & Thames Valley Meetup.
We were kindly hosted by Perusha and Ming from the local Google Developer Group Meetup and got lots of interesting questions from the audience. This also gave us few ideas on how to develop further the Oxford Dictionaries API and what developers may want to do with it in the fields of Natural Language Processing and Machine Learning.
XML Prague 2018
This week I am attending XML Prague at the University of Economics College campus in Prague, a conference on markup languages and data on the web. Together with other XSpec developers I am organising the XSpec Users Meetup. I’m also giving a lightning talk in the Schematron Users Meetup on how to test Schematron with XSpec.
Slides of the XSpec Users Meetup are available here whereas my lightning talk on testing Schematron with XSpec is available here.
How to exclude a package from being updated on Linux
Sometimes you prefer not to update a specific package in Linux. This may be because you don’t want to upgrade to a new version with new features but no security updates. Or maybe because upgrading requires a service restart that you want to avoid just yet. This was the case for me recently when a new version of Docker came up and upgrading would have restarted the docker daemon and stopped the running containers.
It is possible to exclude a package from being updated. On Linux RPM systems (RedHat, CentOS, Fedora, etc.) this is the command to install all updates but exclude a specific package (say docker):
|
1 |
sudo yum update --exclude=docker |
On Debian-like systems (Debian, Ubuntu, Mint, etc.) it is slightly more convoluted because you need to hold a package first and then upgrade the system
|
1 |
sudo apt-mark hold docker && sudo apt-get upgrade |
and remember to remove the hold when you’re ready to upgrade that package too
|
1 |
sudo apt-mark unhold docker |
References:
XML London 2017
This weekend I am attending XML London at University College London, a 2 day conference on XML, Open Data, Digital Publishing, and Data Management. I am presenting a paper on XSpec. The paper is available here and the slides of my presentation are available here.