DjangoCon 2014: Top tips for developing and deploying on AWS

I was at my third DjangoCon earlier this month. As with the previous DjangoCon's it was a lot of fun. For the first time I was speaking, which was a first for me, at a tech conference, traditionally I've spoken at science conferences. My talk was entitled "Top tips for developing and deploying on AWS", granted more to do with AWS than Django. However, deploying your shiny new site is just as important as writing it, it's a rather poor website if no one can view it! With the rise of DevOps there has been a push for developers to deploy as well. You can read more on Full Stack Python, an excellent resource maintained by Matt Makai.

Unlike my previous talks this was the first one to be recorded. I'll be honest I was dreading seeing the recording (as I don't like seeing/hearing myself on video). However, I was pleasantly surprised and actually the talk sounded much better that I thought it went! Most importantly for me it will be useful to improve my presentation technique and with video evidence I can track my progress (hopefully anyway).

You can review the slides and videos below - I hope you find it useful.

Install NumPy and SciPy without Fortran

NumPy and SciPy are two great Python packages for scientists, as is the popular Matplotlib. However, installing NumPy and SciPy is not for the faint hearted if you install your Python packages via pip. Assuming you have fortran, blas, lapack and atlas already installed it is actually quite a slow installation, especially SciPy. NumPy took 46 seconds to install, whereas SciPy took 6 mins and 50 seconds on my MacBook Pro. So what if you install once and forget? Two problems with that. First I use mktmpenv when debugging issues. Second I also use tox to test against multiple version of Python and/or Django. All of a sudden 6 build configurations is 42 minutes of SciPy compilation!

Let's not forget Windows users, Fortran - I don't think so and they should be able to enjoy pip and virtualenv as much as any Python developer.

The obvious solution is for SciPy to be packaged with wheel, the new Python binary distribution format. However, I appreciate that would be very hard for the authors, but hopefully one day.

In the meantime Anaconda might be of interest. It is like apt-get/yum for scientific Python, but a new feature has just been announced, you can pip install anaconda itself then take advantage of the binary distributions it provides for you.

Want easy installation of NumPy stack in *your* Python? pip install conda now works! Then conda init, then conda install scikit-learn
— Travis Oliphant (@teoliphant) November 17, 2013

So try this (assuming you have pip, virtualenv and virtualenvwrapper installed)

$ mktmpenv
$ pip install conda
$ conda init
$ conda install scipy

SciPy plus NumPy and numerous dependencies are installed in under a minute! Obviously, you can not convert this to a requirements.txt per se, but using Fabric you can make a task to install conda and then the conda packages all with a one liner.

Book chapter on the use of open source software in the pharmaceutical industry

During my final year at AstraZeneca I was asked to contribute a chapter to "Open source software in life science research: Practical solutions to common challenges in the pharmaceutical industry and beyond". Given this work would be very hard to publish in JCIM, JMC etc and I have never written a book chapter before it was the perfect opportunity. The chapter was entitled: Design Tracker: an easy to use and flexible hypothesis tracking system to aid project team working. It was coauthored by Martin Harrison, who wrote Design Tracker. The abstract sums up best what it covers:

Design Tracker is a hypothesis tracking system used across all sites and research areas in AstraZeneca by the global chemistry community. It is built on the LAMP (Linux, Apache, MySQL, PHP/ Python) software stack, which started as a single server and has now progressed to a six-server cluster running cutting-edge high availability software and hardware. This chapter describes how a local tool was developed into a global production system.

Design Tracker has been mentioned in a few external presentations before but I believe this is the first firm details about it. We talk about its use and how it came to be a global chemistry tool from a prototype at one site. As the book topic suggests we also cover the open source technologies we used to power it. While LAMP is not new, it is not exactly mainstream in the corporate environment for many pharmaceutical companies. We had to harden our setup to make it suitable for 24/7 use, so in addition to the regulars on LAMP we added Red Hat Cluster Suite, Continuent Tungsten and NGINX. We also took the opportunity to move away from apache/mod_python to apache/mod_wsgi. The end result was a service which is available 24/7 and future proofed compared to our previous solution.

The worlds most dynamic and frequently visited websites are powered by similar technologies so they have clearly proven themselves to be suited for the relatively modest needs of a single pharmaceutical company.

The book is available on Amazon UK & US and probably your favourite book reseller as well (ISBN: 978-1907568978). I hope you enjoy our chapter and the many others interesting topics covered.

Getting ready for PyCon 2013

PyCon US 2013 is nearly upon us. This is my first PyCon and completes my attendance at both major US Python conferences (after attending DjangoCon last September). So thanks to my employer for letting me attend.

I haven't left for Santa Clara yet but I already feel part of the conference because of several factors:

• The conference schedule has been made available via Guidebook and Lanyrd. Guidebook is an excellent medium for keeping track of your schedule across tutorials, talks, poster sessions and more. Plus the organizers can (and already have) update the information making it the goto source of information.  The only gripe I have is I can not find a way to sync my schedule between multiple iOS devices.
• I'll check back with Lanyrd later to collect the slides from SlideShare for talks I attended and importantly those I couldn't.
• Various events outside the regular talks have been posted on Eventbrite (which also features Passport integration on iOS now).
• Of course you can follow everything on Twitter via the #pycon hashtag and @pycon feed.

Attending a conference with thousands of attendees feels much more personal when these tools are available. I may not be able to catch speakers after their talks (or even attend the talk at all) but I can tweet them to seek answers and potentially arrange to meet or I can grab their slides from SlideShare. I would encourage all speakers to share a contact medium of their preference and post their slides online.

All that is left for me to do now is try and pick my schedule from the plethora of interesting talks and other sessions running!

DjangoCon US 2012

I was lucky enough to attend DjangoCon earlier this month. It was a great experience, I'm still following up on the new tips and tricks I picked up. 

This was my first purely software focused conference, they are normally scientific. There are some stark differences. Notably I'm not the only person tweeting! Thanks to the wonders of lanyrd.com and slideshare.net I was able to leave DC with all the presentations I attended. Although I took copious notes during the talks it is handy to double check details. Plus it was dual track so I missed half the talks, some of which I would of liked to of seen. 

So scientists, why don't you tweet and share your research more? You have gone to the effort of writing (thus getting approval to disclose your work) and attending a conference - that is the hard part. Uploading to slideshare.net and sharing on twitter and/or LinkedIn is the easy step! By the same token I will follow my own advice too.

Using Gunicorn with PyCharm

The Green Unicorn

When you develop projects with Django it comes with a handy runserver which is a lightweight webserver bundled with Django, see more here. It goes without saying you would never want to deploy any production code with this as it doesn't scale nor is it particularly secure. It does not get much attention from the Django developers, lets face it there are better things to work on. So a few alternatives have popped up. The one which caught my eye is gunicorn. Now gunicorn can be used on preprod or production sites as well, but I stick to development here (what I use for production will feature in another post).

Gunicorn is actually a fork of Unicorn which is designed for use with Ruby. It is a WSGI webserver, WSGI is the way to run webservers. It also features integration with Django so can be a drop in replacement for runserver. Installation is a breeze, pip install gunicorn and add gunicorn to your INSTALLED_APPS in your settings.py - Easy.


From the command-line you might do ./manage.py runserver 192.168.1.1:9000 with the Django runserver. To use Gunicorn simply run ./manage.py run_gunicorn 192.168.1.1:9000. Again straight forward. Within PyCharm you need edit your django run configuration and tick the "run custom command" box. Add run_gunicorn and run that. run_gunicorn is a helper function that gunicorn adds to manage.py. Unfortunately, I found not all gunicorn options can be specified via this, which can be annoying.  To access all gunicorn options you need to run gunicorn direct and feed it a wsgi application, handily django is wsgi ready and from 1.4 includes a wsgi file when you create a new project (look for wsgi.py in your project directory). When using gunicorn direct you do not need to list gunicorn in your INSTALLED_APPS. From your virtualenv run gunicorn -b 192.168.1.1:9000 myproject.wsgi:application, where myproject is your Django project name. You can drop in any gunicorn options you like as well. As for running in PyCharm you need to create a Python run configuration, not Django, and provide the following information:

1. Script - /Users/craig/.virtualenvs/blogenv/bin/gunicorn (your virtualenv of choice)
2. Script parameters: -b 192.168.1.1:9000 myproject.wsgi:application
3. Working directory: /Users/craig/PycharmProjects/MyDjangoApp

There is another reason to use gunicorn directly - you can use it with supervisor.

You may come across the gunicorn_django application as well. That is fine to use if you are not on Django 1.4, otherwise use the gunicorn application direct, as shown above.

There is also devserver from David Cramer. I've not used it personally but the other projects that David works on are top rate, so I imagine this is too.

PyCharm for Django development

I have been dabbling with Django for a few years now. Previously, I always used Eclipse for this. Eclipse by itself doesn't support Python or Django but add PyDev or Aptana Studio and it merrily understands what is going on. This was great when I worked on multiple projects in multiple languages. I now find myself working solely in a Python/Django environment. PyCharm was getting rave reviews from my former work colleagues, but I did so little Django it didn't seem worth making the transition. However, with a new role I thought now was the time and I haven't looked back. If you have ever used IntelliJ IDEA for Java you'll be familiar with PyCharm as it is built upon the same framework. Being written in Java also means you are able to code on your Mac, Linux and Windows devices.

For once PyCharm is fully aware of Django, not just syntax highlighting (which was also hit and miss especially in templates mixing HTML/CSS/Django template syntax in Eclipse).  Truth be told I have barely touched the surface of features that PyCharm packs in. See http://www.jetbrains.com/pycharm/features/index.html for a full run down. Each release, of which updates are frequent, either refine or add more features. In the 2.5 release virtualenv support was added, it worked with manual intervention before, but now PyCharm automatically detects them. In addition the support at JetBrains is superb. Admittedly this is a commercial product, although they license favorably for open source project - free.

Just like IntelliJ various plugins are available. A whole host ship by default, Git, CoffeeScript and JavaScript to name a few. More are available online at the plugin repository. The nginx support and NodeJS plugins caught my eye.

One of my favorite features is suport for running mange.py tasks from within the IDE, no need to setup the environment and run from there, as shown (image courtesy of JetBrains).

In fairness I did look at other Python IDEs with Django support, but feel they fall into the same camp as Eclipse. The Django support is just tacked on and is limited. With PyCharm you can see Django support was core to the design and features implemented. I see the PyCharm team will be at DjangoCon US. This is great for two reasons a) I can congratulate them on a great IDE and b) they will be bring a new update (http://blog.jetbrains.com/pycharm/2012/06/pycharm-at-djangocons-around-the-world/). 

Python Virtual Environments

Anyone who uses Python regularly will soon encounter the need to install an extra package be it MySQLdb, numpy or Django. Thanks to tools like pip and easy_install this is very trivial to do (as long as you have internet access from your command-line. Those of you behind corporate firewalls might want to try cntlm to help). Typically they will install the package into your home directory for only you to use. If you are root they will install into the system installation for all users to use. In both scenarios it is hard to mix and match multiple versions of packages. As a developer this becomes very important, I might want to try the lastest version of a package, but for my production code I want to remain on the stable release. The easiest way to manage this is using virtual environments. Virtualenv is a python package that allows you to manage multiple python instances with different combinations of packages. It is easy to mix python and package versions for development and production code. Virtualenv is great and has been extended with virtualenvwrappers, which adds even more functionality.

To get started run the following:

easy_install virtualenv
easy_install virtualenvwrappers

Next you need to alter your environment slightly in ~/.bashrc or equivalent add:


export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/projects
export VIRTUALENVWRAPPER_VIRTUALENV=/usr/local/bin/virtualenv
source /usr/local/bin/virtualenvwrapper.sh

Fire up a new terminal or run source ~/.bashrc and the new commands are now available to you. Lets create our new virtual environment with mkvirtualenv test. This creates and loads the new virtual environment called test. Your command prompt will indicate you are using test as well. Now run easy_install to install your packages. You can see what packages are available with lssitepackages. To leave this environment run deactivate. Create another environment and list your packages, there will be none. Switch back to test with workon test.

You can tie this virtual environment to Eclipse by pointing at the python for the test environment, which can be found at ~/.virtualenvs/test/bin/python.

See all the commands available for virtualenvwrappers at http://www.doughellmann.com/projects/virtualenvwrapper/. Enjoy your new virtual python environments!

Image courtesy of http://www.flickr.com/photos/ptshello/2312382347/

Eclipse with Django

Django is popular web framework written in Python. For Python users it is an obvious choice to deploy web applications. Once again I return to my favourite IDE to develop with Django - Eclipse.

Two plugins are needed to bring Django support to Eclipse. The second handles the Django template language. Grab the plugins:

1. PyDev (Python)
2. Aptana Studio (Python + Django)

Both plugins are from Aptana, PyDev is a plugin to their IDE. Thankfully, both work through Eclipse as well.

Eclipse with Python/OEChem

Eclipse is an excellent IDE. Alas it doesn't have support for Python - fear not the PyDev plugin comes to the rescue. (Installation information here).

While that now works like a charm, if you have an OpenEye license to hand (academics see this post) if would be great to use the Python toolkit through Eclipse.

Follow these steps to make your Python project OEChem ready:

Window > Preferences > Pydev > Interpreter - Python.
To Libraries add your $OE_DIR/python folder.
To Environment add OE_LICENSE and OE_DIR, with their respective locations.
Create a new PyDev project and it should have OE capability by default.

Java users don't worry, I'll cover OEChem/Java in my next post.

Python training

Forgot to mention in my last post. If you are after Python/OEChem training and are not coming to EuroCUP, there is an alternative. Andrew Dalke is offering Python training for Cheminformatics. OEChem is just one part of this course, Django, SQL, R are also covered.

I've spoken to a previous attendee and he recommended it.

See Andrew's website for more details.