Arjan Schaaf bio photo

Arjan Schaaf

Linux, AWS, Azure, Docker & Java rock my world. But far more important: I'm a husband and father of two. Love to BBQ and I'm famous in my world for my burgers & ribs :-)

Twitter LinkedIn Github

CloudInit, AWS and Azure

I realy like CloudInit. Not the syntax or the lack of documentation… But I love what you can achieve using CloudInit. I’ve been using it for years to bootstrap new servers on the Amazon Cloud (AWS). Installing new packages, updating existing packages in the AMI, running scripts to bootstrap my application environment, etc. It can all be achieved in a reproducable way using CloudInit. If you are not familiar with CloudInit, have a look at:

CloudInit runcmd & scripts

So CloudInit is great, especially the part were you can define a list of commands that should be run at the initial boot of your VM. For example to fix the locale settings of a plain Amazon Linux or Ubuntu image.

But another requirement I had is a bit more complex: running custom (bash) scripts. Creating scripts and other files in CloudInit is often done by using echo to print the contents to a file, like:

#cloud-config
runcmd:
- echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php

Works fine for your single line magic, but not so good for scripts that are dozens or even hunderds of lines long…

CloudInit scripts & S3

On my AWS deployments S3 came to the rescue. I started storing these scripts not in my CloudInit configurations but in a S3 bucket. Off course a S3 bucket that was secured and only accessible from our VPC network. But the usage with CloudInit was very simple, because S3 offers an HTTP based interface I could simply wget the script from my CloudInit configuration and start using it. Something like:

#cloud-config
runcmd:
- wget https://s3-eu-west-1.amazonaws.com/<my-bucket>/myscript-1.1.sh -O /tmp/myscript.sh
- chmod +x /tmp/myscript.sh
- sh /tmp/myscript.sh

CloudInit scripts & Azure Blob Storage

So when I started working on another project that needed to deploy on Azure, I immediatly thought: do the same thing, but this time around use Azure Blob Storage instead of Amazon S3. We’re still talking blobs, right? Right…. well the interface provided by Azure is a bit different then what I was used to with S3. Azure doesn’t provide an easy way to just use plain http to download a private blob. This MSDN article explains how to download a blob, but getting the authentication right is complicated with the basic tools available in CloudInit.

I considered two approaches:

  • use a Azure Web Role to serve these files
  • use another interface to access the blob data

I went for the second option, having no experience yet with the Azure Web Roles. And I needed to use the Blob storage for other purposes anyways, so it felt like the best fit. Azure offers API’s to access the Blob data, one of those API’s is used by the Azure cross platform CLI, which I already used to manage my VM’s! Greate project, check it out at https://github.com/Azure/azure-xplat-cli

CloudInit, Azure ClI and Azure Blob Storage

So I could use CloudInit to install the Azure CLI, configure it with the relevant configuration data and use the Azure CLI commands to donwload my scripts from the Azure Blob Storage. Sounds easy enough? It is:

#cloud-config
packages:
- nodejs-legacy
- npm
runcmd:
- locale-gen UTF-8
- npm install -g azure-cli
- HOME=/tmp/ azure storage blob download --account-name <accountName> --account-key <accountKey> -b myscript.sh --container <blob-container> --destination /tmp/myscript.sh
- chmod +x /tmp/myscript.sh
- sh /tmp/myscript.sh

So there are a couple of tricks to this setup:

  • You need to install the nodejs-legacy & npm packages (based on Ubuntu 14.04 image, your milage may vary on other distributions)
  • you need to install the azure-cli using npm https://github.com/Azure/azure-xplat-cli
  • azure cli requires a folder to execute from, so start that command with HOME=/tmp/ for example