Book chapter on the use of open source software in the pharmaceutical industry

During my final year at AstraZeneca I was asked to contribute a chapter to "Open source software in life science research: Practical solutions to common challenges in the pharmaceutical industry and beyond". Given this work would be very hard to publish in JCIM, JMC etc and I have never written a book chapter before it was the perfect opportunity. The chapter was entitled: Design Tracker: an easy to use and flexible hypothesis tracking system to aid project team working. It was coauthored by Martin Harrison, who wrote Design Tracker. The abstract sums up best what it covers:

Design Tracker is a hypothesis tracking system used across all sites and research areas in AstraZeneca by the global chemistry community. It is built on the LAMP (Linux, Apache, MySQL, PHP/ Python) software stack, which started as a single server and has now progressed to a six-server cluster running cutting-edge high availability software and hardware. This chapter describes how a local tool was developed into a global production system.

Design Tracker has been mentioned in a few external presentations before but I believe this is the first firm details about it. We talk about its use and how it came to be a global chemistry tool from a prototype at one site. As the book topic suggests we also cover the open source technologies we used to power it. While LAMP is not new, it is not exactly mainstream in the corporate environment for many pharmaceutical companies. We had to harden our setup to make it suitable for 24/7 use, so in addition to the regulars on LAMP we added Red Hat Cluster Suite, Continuent Tungsten and NGINX. We also took the opportunity to move away from apache/mod_python to apache/mod_wsgi. The end result was a service which is available 24/7 and future proofed compared to our previous solution.

The worlds most dynamic and frequently visited websites are powered by similar technologies so they have clearly proven themselves to be suited for the relatively modest needs of a single pharmaceutical company.

The book is available on Amazon UK & US and probably your favourite book reseller as well (ISBN: 978-1907568978). I hope you enjoy our chapter and the many others interesting topics covered.

Getting ready for PyCon 2013

PyCon US 2013 is nearly upon us. This is my first PyCon and completes my attendance at both major US Python conferences (after attending DjangoCon last September). So thanks to my employer for letting me attend.

I haven't left for Santa Clara yet but I already feel part of the conference because of several factors:

• The conference schedule has been made available via Guidebook and Lanyrd. Guidebook is an excellent medium for keeping track of your schedule across tutorials, talks, poster sessions and more. Plus the organizers can (and already have) update the information making it the goto source of information.  The only gripe I have is I can not find a way to sync my schedule between multiple iOS devices.
• I'll check back with Lanyrd later to collect the slides from SlideShare for talks I attended and importantly those I couldn't.
• Various events outside the regular talks have been posted on Eventbrite (which also features Passport integration on iOS now).
• Of course you can follow everything on Twitter via the #pycon hashtag and @pycon feed.

Attending a conference with thousands of attendees feels much more personal when these tools are available. I may not be able to catch speakers after their talks (or even attend the talk at all) but I can tweet them to seek answers and potentially arrange to meet or I can grab their slides from SlideShare. I would encourage all speakers to share a contact medium of their preference and post their slides online.

All that is left for me to do now is try and pick my schedule from the plethora of interesting talks and other sessions running!

DjangoCon US 2012

I was lucky enough to attend DjangoCon earlier this month. It was a great experience, I'm still following up on the new tips and tricks I picked up. 

This was my first purely software focused conference, they are normally scientific. There are some stark differences. Notably I'm not the only person tweeting! Thanks to the wonders of lanyrd.com and slideshare.net I was able to leave DC with all the presentations I attended. Although I took copious notes during the talks it is handy to double check details. Plus it was dual track so I missed half the talks, some of which I would of liked to of seen. 

So scientists, why don't you tweet and share your research more? You have gone to the effort of writing (thus getting approval to disclose your work) and attending a conference - that is the hard part. Uploading to slideshare.net and sharing on twitter and/or LinkedIn is the easy step! By the same token I will follow my own advice too.

Using Gunicorn with PyCharm

The Green Unicorn

When you develop projects with Django it comes with a handy runserver which is a lightweight webserver bundled with Django, see more here. It goes without saying you would never want to deploy any production code with this as it doesn't scale nor is it particularly secure. It does not get much attention from the Django developers, lets face it there are better things to work on. So a few alternatives have popped up. The one which caught my eye is gunicorn. Now gunicorn can be used on preprod or production sites as well, but I stick to development here (what I use for production will feature in another post).

Gunicorn is actually a fork of Unicorn which is designed for use with Ruby. It is a WSGI webserver, WSGI is the way to run webservers. It also features integration with Django so can be a drop in replacement for runserver. Installation is a breeze, pip install gunicorn and add gunicorn to your INSTALLED_APPS in your settings.py - Easy.


From the command-line you might do ./manage.py runserver 192.168.1.1:9000 with the Django runserver. To use Gunicorn simply run ./manage.py run_gunicorn 192.168.1.1:9000. Again straight forward. Within PyCharm you need edit your django run configuration and tick the "run custom command" box. Add run_gunicorn and run that. run_gunicorn is a helper function that gunicorn adds to manage.py. Unfortunately, I found not all gunicorn options can be specified via this, which can be annoying.  To access all gunicorn options you need to run gunicorn direct and feed it a wsgi application, handily django is wsgi ready and from 1.4 includes a wsgi file when you create a new project (look for wsgi.py in your project directory). When using gunicorn direct you do not need to list gunicorn in your INSTALLED_APPS. From your virtualenv run gunicorn -b 192.168.1.1:9000 myproject.wsgi:application, where myproject is your Django project name. You can drop in any gunicorn options you like as well. As for running in PyCharm you need to create a Python run configuration, not Django, and provide the following information:

1. Script - /Users/craig/.virtualenvs/blogenv/bin/gunicorn (your virtualenv of choice)
2. Script parameters: -b 192.168.1.1:9000 myproject.wsgi:application
3. Working directory: /Users/craig/PycharmProjects/MyDjangoApp

There is another reason to use gunicorn directly - you can use it with supervisor.

You may come across the gunicorn_django application as well. That is fine to use if you are not on Django 1.4, otherwise use the gunicorn application direct, as shown above.

There is also devserver from David Cramer. I've not used it personally but the other projects that David works on are top rate, so I imagine this is too.

PyCharm for Django development

I have been dabbling with Django for a few years now. Previously, I always used Eclipse for this. Eclipse by itself doesn't support Python or Django but add PyDev or Aptana Studio and it merrily understands what is going on. This was great when I worked on multiple projects in multiple languages. I now find myself working solely in a Python/Django environment. PyCharm was getting rave reviews from my former work colleagues, but I did so little Django it didn't seem worth making the transition. However, with a new role I thought now was the time and I haven't looked back. If you have ever used IntelliJ IDEA for Java you'll be familiar with PyCharm as it is built upon the same framework. Being written in Java also means you are able to code on your Mac, Linux and Windows devices.

For once PyCharm is fully aware of Django, not just syntax highlighting (which was also hit and miss especially in templates mixing HTML/CSS/Django template syntax in Eclipse).  Truth be told I have barely touched the surface of features that PyCharm packs in. See http://www.jetbrains.com/pycharm/features/index.html for a full run down. Each release, of which updates are frequent, either refine or add more features. In the 2.5 release virtualenv support was added, it worked with manual intervention before, but now PyCharm automatically detects them. In addition the support at JetBrains is superb. Admittedly this is a commercial product, although they license favorably for open source project - free.

Just like IntelliJ various plugins are available. A whole host ship by default, Git, CoffeeScript and JavaScript to name a few. More are available online at the plugin repository. The nginx support and NodeJS plugins caught my eye.

One of my favorite features is suport for running mange.py tasks from within the IDE, no need to setup the environment and run from there, as shown (image courtesy of JetBrains).

In fairness I did look at other Python IDEs with Django support, but feel they fall into the same camp as Eclipse. The Django support is just tacked on and is limited. With PyCharm you can see Django support was core to the design and features implemented. I see the PyCharm team will be at DjangoCon US. This is great for two reasons a) I can congratulate them on a great IDE and b) they will be bring a new update (http://blog.jetbrains.com/pycharm/2012/06/pycharm-at-djangocons-around-the-world/). 

Homebrew

Courtesy of https://twitter.com/MacHomebrew/

Homebrew, the OS X package manager, not the beverage, is the third package manager I've used on OS X. I started in 2005 with Fink and MacPorts. MacPorts seemed to be more active, especially with the flurry of Leopard releases. Indeed it also has some advanced features including dependencies to let you install MAMP stacks. However, I find I prefer to have control of my MAMP stack especially in my volatile development environments, upgrades can be particularly painful with MacPorts.

Homebrew has quite a different approach. There is no need for sudo, all binaries get symlinks in /usr/local, but point to /usr/local/Cellar which controls the formula (aka package) version. The formula scripts are ruby based, so adding your own is very easy.

If you don't want to search forumlae via the command-line then head to http://braumeister.org/ (another GitHub project).

Installation is pretty simple. If you are a developer, e.g. you have Xcode installed it is probably a one-liner. Installation alongside MacPorts is not recommended, but to check you can get everything you need from Homebrew it will probably not cause too many problems.

Now you have brew in your path lets explore some use cases:

• brew list - list installed formulae
• brew search wget - search for formula called wget
• brew info wget - get information about the formula wget
• brew install wget - install wget
• brew uninstall wget -uninstall wget
• brew update - update homebrew (effectively a git pull)
• brew outdated - list formulae which are out of date
• brew upgrade - upgrade all outdated formulae

For more features see the FAQ and follow @MacHomebrew.

Now I wonder if I can convince puppet to use Homebrew over MacPorts...

Python Virtual Environments

Anyone who uses Python regularly will soon encounter the need to install an extra package be it MySQLdb, numpy or Django. Thanks to tools like pip and easy_install this is very trivial to do (as long as you have internet access from your command-line. Those of you behind corporate firewalls might want to try cntlm to help). Typically they will install the package into your home directory for only you to use. If you are root they will install into the system installation for all users to use. In both scenarios it is hard to mix and match multiple versions of packages. As a developer this becomes very important, I might want to try the lastest version of a package, but for my production code I want to remain on the stable release. The easiest way to manage this is using virtual environments. Virtualenv is a python package that allows you to manage multiple python instances with different combinations of packages. It is easy to mix python and package versions for development and production code. Virtualenv is great and has been extended with virtualenvwrappers, which adds even more functionality.

To get started run the following:

easy_install virtualenv
easy_install virtualenvwrappers

Next you need to alter your environment slightly in ~/.bashrc or equivalent add:


export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/projects
export VIRTUALENVWRAPPER_VIRTUALENV=/usr/local/bin/virtualenv
source /usr/local/bin/virtualenvwrapper.sh

Fire up a new terminal or run source ~/.bashrc and the new commands are now available to you. Lets create our new virtual environment with mkvirtualenv test. This creates and loads the new virtual environment called test. Your command prompt will indicate you are using test as well. Now run easy_install to install your packages. You can see what packages are available with lssitepackages. To leave this environment run deactivate. Create another environment and list your packages, there will be none. Switch back to test with workon test.

You can tie this virtual environment to Eclipse by pointing at the python for the test environment, which can be found at ~/.virtualenvs/test/bin/python.

See all the commands available for virtualenvwrappers at http://www.doughellmann.com/projects/virtualenvwrapper/. Enjoy your new virtual python environments!

Image courtesy of http://www.flickr.com/photos/ptshello/2312382347/