Re: buildout development vs. deployment

Hi Tom,

great answer, thanks. A lot of other sites mentioned Fabric. I'll definitely check it out.

/Reik

On Wed, Apr 25, 2012 at 1:22 PM, Tom Evans <tevans.uk@googlemail.com> wrote:

On Wed, Apr 25, 2012 at 11:21 AM, Reikje <reik.schatz@gmail.com> wrote:
> Hi, I am looking into buildout to deploy a django webapp and all it's
> dependencies. I find it quite useful even during development because it
> forces you to keep track of your dependencies. A question regarding some
> best practices. Lets say I have buildout.cfg and setup.py in my project root
> and checked in into SCM. My webapp is listed under develop in buildout.cfg.
> While this is great, during deployment this is probably not what you want
> because you wanna lock the version of your source code. I want to do a git
> revision checkout to archive this. So i guess I need to maintain two
> different buildout.cfg files, one for development and one for deployment.
> How can this be organized to avoid DRY?
>
> On a side note, what are the alternatives to buildout. Maybe there is
> something even better :)
>

I have no experience of buildout, but I have spent a lot of time
working on SCM for my django sites, so I'll describe what I do.

Everything here is built around subversion, we don't use git. If you
haven't used subversion, 'svn:externals' check out another repository
in the specified location. It's a way of composing disparate bits of
versioned repositories.

The first thing to discuss is the project source code. This lives in
svn/projectname/trunk, with release branches in
svn/projectname/releases. Minor work happens on trunk, and is merged
to release branch(es). Major work happens on feature branches, which
is then merged to trunk and then to release branches.

Of course, project source code in itself is not enough to run a site.
The project must be deployed and configured for a specific host. We do
this with 'deployments'. Each deployment corresponds to an instance of
a project running on a specific server, and lives in
svn/projectname/deployments/servername, and consists of a skeleton
directory structure (derived from a base version), and lots of
'svn:externals' links.

So the deployment contains the structure of the project, and links to
check out the project code and other required files, like scripts for
starting/stopping the project, a bootstrap script for creating the
virtualenv environment, the settings files, etc, which are also all
stored in subversion.

The next stage is libraries. For this, we use virtualenv and pip. We
have a bootstrap script which creates the virtualenv if it is not
present, and a requirements.pip file, which lists the libraries and
specific versions to install.
All of our libraries are installed via pip, and we have a process
which checks out both our own libraries and patched vendor libraries
(eg, like django-debug-toolbar), builds packages from them, and pushes
them to our own pypi site¹. All our packages are then installed from
this pypi site.
The requirements.pip is another 'svn:externals' file, checked out from
svn/projectname/configurations/. All the production sites share one
requirements.pip.

The project has a default_settings.py file, and each deployment also
checks out a deployment specific settings.py, which imports from
default_settings.py and adds the host-specific settings for that site.

Every check-in on the project results in a buildbot updating a
buildbot specific deployment and running tests. There is a buildbot
for trunk and for each active release branch.

All schema migrations are taken care of by south.

Finally, all the production deployments are managed by a Fabric
fabfile. This contains programmatic instructions for doing various
tasks, eg update app servers to latest version, which:
 takes one set of backend servers out of the rotation
 stops DB replication from that deployment
 updates that deployment
 installs/uninstalls libraries as indicated in (updated) requirements.pip
 runs south sschema migrations
 marks that backend as the only active backend
 restarts DB replication
 updates all other servers.

If you're not using fabric, or some other automated rollout tool, to
manage your services, you should be!

For our 'bulletproof' sites, we have an exact replica of a production
stack in test. We never update the live site until we've updated this
test replica successfully.

Creating a new backend deployment is as simple as copying the stock
deployment, creating the site specific settings.py, checking it out on
a host and running the bootstrap script. In fact, it's slightly easier
than that, its 'fab new_prod_site hostname'.

Since everything is in subversion, we can easily re-constitute a site
as it was, or reproduce the exact same configuration in development as
we have in production. This aids debugging and increases confidence in
the system.

Cheers

Tom

Tools:

virtualenv: http://www.virtualenv.org/en/latest/index.html
pip: http://www.pip-installer.org/en/latest/index.html
south: http://south.aeracode.org/
fabric: http://fabfile.org/
buildbot: http://trac.buildbot.net/


¹ It's not as good as the real pypi site, its really just a big folder
served over http with all the packages in it. Works well enough.

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django-users@googlegroups.com.
To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.