Monday, November 10, 2014

Tag your AWS resources in StarCluster

StarCluster logo
StarCluster is a convenient way to manage HPC clusters on Amazon Web Services (AWS). If you use Billing docs.
the same AWS account for multiple tasks such as hosting your web servers and its associated databases, running StarCluster how can you tell how much of your bill is from StarCluster? AWS lets you tag your detailed billing information, this can be used to get a feel* for how much you spent on a specific tag. You need to enable detailed billing and pick your allocation tags, start by reading the

There is also another reason to tag your resources in StarCluster - to utilize IAM resource-level permissions to restrict access, e.g. a IAM StarCluster user can only start/stop StarCluster, not terminate every EC2 instance in your account. This is a topic for another blog post, but requires tagging to help restrict permissions.

There are three steps to enable tagging within StarCluster:
  1. Save a custom plugin
  2. Configure the plugin in your config
  3. Add the plugin to your cluster stanza
First, save this custom plugin, which we call tagger, to $HOME/.starcluster/plugins/

Second, at the bottom of your StarCluster configuration (typically $HOME/.starcluster/config) define the tags you want to attach. Take care in picking your tags, you want everyone using the same tag key/values.

Finally, get your cluster to call the new plugin, by editing your cluster stanza:

[cluster smallcluster]
# Various other settings
# Enable the following plugins
PLUGINS = tagger

Now when you launch your next cluster you will see a few extra lines in the output confirming tagger has run and you can check on the AWS Management console that EC2 instances and ELB volumes are now tagged.

*It is not possible to tag every resource or the resource is shared so how do you split the cost? The actual cost will likely be higher than just the resources you tag.

Sunday, September 14, 2014

DjangoCon 2014: Top tips for developing and deploying on AWS

I was at my third DjangoCon earlier this month. As with the previous DjangoCon's it was a lot of fun. For the first time I was speaking, which was a first for me, at a tech conference, traditionally I've spoken at science conferences. My talk was entitled "Top tips for developing and deploying on AWS", granted more to do with AWS than Django. However, deploying your shiny new site is just as important as writing it, it's a rather poor website if no one can view it! With the rise of DevOps there has been a push for developers to deploy as well. You can read more on Full Stack Python, an excellent resource maintained by Matt Makai.

Unlike my previous talks this was the first one to be recorded. I'll be honest I was dreading seeing the recording (as I don't like seeing/hearing myself on video). However, I was pleasantly surprised and actually the talk sounded much better that I thought it went! Most importantly for me it will be useful to improve my presentation technique and with video evidence I can track my progress (hopefully anyway).

You can review the slides and videos below - I hope you find it useful.