TODO: Break out parts not covered by other docs.

The lifecycle of a Grok application

Author:reinoud

This how-to tries to give an overview of what has to be done in which phase of the lifetime of a typical Grok application.

Step by step

An application usually has different phases in its lifetime.

Starting development of a new application

Usually, an application starts with a realy bright idea. Let’s suppose you have that idea and have made the decision to implement it in Grok. So you want to start the development. Now there is a decision to make: do you want to use buildout or virtualenv (or both?).

Virtualenv is a tool to build an isolated local Python environment that is not inflenced by the global site packages. It allows you to install (versions of) packages that are local to the aplication you are developing. It is really easy to use and great if you want to start coding right now and not bother with buildout configurations (yet). It provides a sandbox where you can safely plan without causing problems outside it. Buildout is a advanced tool to create a virtual environment, install all dependencies of your application in a reliable and reproducable way. It is usable both for development and deployment. It is more difficult to use than virtualenv, but provides lots more flexibility. It is most likely the tool to use for serious applications. It allows for formalized upgrade procedures, version pinning and is extendable with external recipes. It can maintain different configurations for development and production environments. The combination of the two tools is preferable. If you use builduit within a virtualenv environment, you are sure to explicitely include all the needed packages that might be installed in your operating system, but are not on the production server where your application with be rolled out. You will start with an empty list of site packages so you will have to include the needed ones.

And it gets even better: grokproject is a tool around buildout that sets up a Grok environment for you. It saves you a lot of typing and you use the normal builduit after creation of your project.

So in short (assuming you already have easy_installed virtualenv and grokproject):

$ mkdir myproject
$ virtualenv --no-site-packages myproject
New python executable in myproject/bin/python2.4
Installing setuptools.............done.
$ . myproject/bin/activate
(myproject)$ grokproject --no-buildout myproject
Enter module (Name of a demo Python module placed into the package) ['app.py']:
Enter user (Name of an initial administrator user): admin
Enter passwd (Password for the initial administrator user): mysecretpassword
Enter eggs_dir (Location where zc.buildout will look for and place packages) ['/home/username/buildout-eggs']:
(myproject)$ cp myproject/buildout.cfg myproject/buildout.org
(myproject)@iss$ sed 's/^eggs-directory = .*//' myproject/buildout.org  > myproject/buildout.cfg

(the last two lines remove a line from buildout.cfg that you do not want)

Developing with a team and version control

Even when you’re working alone, it is a very good idea to use version control. We use the svn tool here, for users of other systems the commands might be slightly different. What is important is what you do and don’t want to put in version control.

Since you’ve not run buildout at this moment, no auto generated content or downloaded dependencies are present. This might be a good time to run your first checkin. Checkin these files:

svn add buildout.cfg setup.py src/myproject Other developers start basically with the same steps by creating a virtualenv and a grokproject on their own environment. Then they do a checkout of your code and start coding.

Running buildout

Before you can actually run buildout you first have to install it:

easy_install zc.buildout

And after that you can run it:

bin/buildout

(this can take a while; a lot of software is downloaded)

Make a release

After some coding, you and your team probably want to release something. Of course you have the latest version of all the softeware of your team and all test pass. Do not forget to fill in all the required information in setup.py, and write release notes. Usually at this moment you create a tag in the revision control system to be able to easily identity the current state of all the files (something like !.0-RELEASE)

In this example we will make both a source and a binary release:

(myproject)$ python setup.py sdist
running sdist
running egg_info
[...]
Writing myproject-0.0/setup.cfg
creating dist
tar -cf dist/myproject-0.0.tar myproject-0.0
gzip -f9 dist/myproject-0.0.tar
removing 'myproject-0.0' (and everything under it)

Now you have a tgz file in your dist directory. To change the version number edit your setup.py (before creating the dist). Since this is a python script you can off course automate this to get the revision number or tag name from a version control substitution like $Revision:$. This left as an excersize to the reader.

To make a binary (egg) distribution:

(myproject)reinoud@iss$ python setup.py bdist_egg
running bdist_egg
running egg_info
[...]
copying src/myproject.egg-info/requires.txt -> build/bdist.freebsd-6.2-RELEASE-i386/egg/EGG-INFO
copying src/myproject.egg-info/top_level.txt -> build/bdist.freebsd-6.2-RELEASE-i386/egg/EGG-INFO
creating 'dist/myproject-0.0-py2.4.egg' and adding 'build/bdist.freebsd-6.2-RELEASE-i386/egg' to it
removing 'build/bdist.freebsd-6.2-RELEASE-i386/egg' (and everything under it)

Now you have an egg in the dist directory.

Creating a local package index

(todo: make a HTML file with links)

Specifying search paths for development eggs

In the [buildout] section: devel = ../foo

Namespace packages

Add this to your setup.py:

namespace_packages = [‘foobar’], And add this to your src/foobar/__init__.py:

__import__ (‘pkg_resources’).declare_namespace(__name__)

Extending buildout with new recipies

(todo)

Adding scripts to the bin directory using entry points

Add this to your setup.py:

entry_points = """
[console_scripts]
foo=myproject.scripts.foo:main"""

When you run buildout again you will find a new script bin/foo that will call the main function in src/myproject/scripts/foo.py

Upgrading your buildout.cfg

Setting up a production environment and installing the first release

Note that there is a limit to what you can rollout within a Python application. You might need shared libraries to connect to a relation database or shared libraries that are needed for XML parsing. Since every operating system uses its own way for the installation of packages, you cannot easily automate this in a cross-platform way. However: somtimes a different version of a shared library is needed than the one that might be installed on the system (and might be needed by other applications). You can include a recipe to build shared libraries in you buildout configuration. But not all operating systems include a C compiler by default...

In this example we’ll try to avoid needing root rights as much as possbile. This makes it easier to install multiple applications under different user accounts, or enables you to upgrade your application without the need to wake-up the system administrator... Having root rights only makes it easier.

Note that the people that maintain the production server might have other ideas than developers about what should be installed where. While a deveper might be used to run everything from (a subdirectory of) his home directory on a unix system, a system administrator might prefer a path somewhere near /usr/local/application. Another point is whether to use centrally installed python packages for all applications on a server, or let every application use its own. The virtualenv makes is possible to use your own, and run the application from a central /usr/local or from a home directory of a dedicated account on the server. The developer does not have to bother with such decisions. If the system administrator wants to use centrally administered packages, he should not use virtualenv at installation time. That might save diskspace, but can also cause problems with some package managers on some Unix systems.

First we start with a clean virtual environment to make sure the application does not bother existing applications and vice versa:

easy_install virtualenv
virtualenv --no-site-packages myproject
. myproject/bin/activate

We can now either use an SVN checkout, which contains both the buildout configuration and and the application packages, or we can use the source distribution. In case you want to use SVN, the procedure is the same as in a development environment.

We will assume that you want to deploy an official distribution. In this case you want to have received the buildout.cfg from the developers.

To start a Grok application, you use paster in the bin directory. To start it as a daemon run

bin/paster serve parts/etc/deploy.ini –daemon

Starting the application from a @reboot crontab entry

If the application developer has root rights on the server, he will typically start the daemon from something like /usr/local/etc/rc.d, according to what is usual on the Unix flavor.

But when you have installed the application in the home directory of a user, and do not have root rights you can also use another trick: create a crontab entry (for that user) that starts the application at boot time. Most modern crontab implementations allow something like this:

@reboot /home/myapplicaionuser/myapplication/bin/paster serve parts/etc/deploy.ini --daemon

Apache config

It is very common to run a Grok or Zope application behind a Apache server. Usually the Grok process listens to a not-reserved high portnumber (like 8080) that any user can open. It is a good idea to only let the localhost connect to that port. In that way the Apache server (that is started with root privileges to open ports like 80 or 443) can redirect the requests with mod_rewrite.

This usualy means that at installation time the apache configfile has to be edited once.

Install an upgrade

The nice thing about buildout is that most of the things are installed automatically. But that also includes your configuration files. So if you have edited them, be sure to back them up before upgrading.

Rollback an upgrade

Administration of a Grok application

Once the application is running, the administrator still has some work to do:

What to backup

Grok applications usually store at least something in the Zope database. That is a file called Data.fs in the var/filestorage subdirectory of your application.

Log files go (by default) to the var/log directory.

Configuration files are in the same place: parts/etc