Re: Request for comments on a new Open Source Paas platform for Django

Hi Patrick,

Great to hear you're interested in writing a Django charm for juju! I have toyed around with the idea, but never got around to implementing something good.

I started looking at the current Django charm a little while ago, and while it works to some extend I think we could make really great things happen with a little work.

As far as feedback for your point goes, here are a few points and suggestion I'd like to add to the discussion:

- Most of the Django websites will likely live in private git/bzr/whatever repositories, and so in the workflow you outlined, you need to somehow push the *private identifier* to the running juju instance. In the "standard" scenario that means pushing your private ssh key to the instance, so it can git clone from a private repository on github... I think it's safe to say that most people will at least frown at the idea :)

Maybe we should instead make this a "push" process?

- It seems a little strange to me to run gunicorn on another machine. Most of the Django project I have encountered run Django with gunicorn on the webservers themselves (add gunicorn to INSTALLED_APPS and then "manage.py run_gunicorn"). Perhaps we should be a little more opinionated about things and for the sake of scaling simplicity deploy nginx or apache locally too (wither with a charm subordinate or at install), so that we can load-balance to all of the servers easily with any frontend (that means all webservers would serve static files, which might not be optimal, but we can refine that later).

- We should absolutely define a cache relation (redis or memcached).

Theses points would make the whole workflow look like the following (the juju syntax might be a little wrong, but please bear with me :) )

  juju bootstrap

  juju deploy --config my_django_conf.yaml cs:django_server my_django_site

  juju deploy cs:postgresql # or mysql,mongodb, etc

  juju deploy cs:memcached # or redis if that's still popular

  juju deploy cs:haproxy

  juju add-relation my_django_site postgresql

  juju add-relation my_django_site memcached

  juju add-relation my_django_site haproxy # strictly speaking that's optional if you have only one django machine

  juju expose haproxy

  # when needed (I hope we all need it someday!)

  juju add-unit my_django_site

  juju add-unit memcached

  juju add-unit postgresql

So now we would have a running django server with no code.

But if it's a push process, we can implement many of the config changes as git hooks, which makes the workflow continue with:

  cd my-django-site

  git init . # If that's not done already of course

  git add .

  git commit -m "produciton push yay!"

  git remote add production git+ssh://my_django_site/some_configurable_url.git

  git push production master # or of course whatever branch you put in the config.yaml

Of course, that requires a non-trivial amount of git triggers to be written, and we should put some requirements.pip.txt file and requirements.apt.txt or whatever in the project tree, but I think that's acceptable.

The whole thing basically follows what many PaaS providers already do, so I guess most Django developers with some sites in production probably are familiar with the workflow. 

This would just add the juju coolness to it :)

Hope this fuels the discussion,

- Chris

On Friday, March 1, 2013 8:13:36 PM UTC+1, Patrick wrote:

Hi,

I'm building a Juju based Open Source Paas platform for Django and

I need your help because it is a hard task to make a PAAS system

that is flexible enough to deploy any projects and at the same time

simple to use.

For the ones that don't know Juju, it's a service orchestration software

compatible with LXC (local), EC2, HPCloud, OpenStack and Baremetal/Maas

developed by Canonical (the company that makes Ubuntu).

Check out the web site for more details: https://juju.ubuntu.com/

So quickly, here's how it would works:

After installing Juju and configuring it with for your favourite cloud provider you

will need to create a configuration file in the YAML format named my_django_conf.yaml

in this example::

  my_django_site:

      vcs: git

      repos_url: https://github.com/my_username/my_site.git 

      site_secret_key: abcdefgh123456789

      use_virtualenv: True

Then you will need these commands to bootstrap and launch all the servers::

  juju bootstrap

  juju deploy --config my_django_conf.yaml my_django_site

  juju deploy postgresql # or mysql,mongodb, etc

  juju deploy gunicorn # Or mod_wsgi, etc

  juju add-relation my_django_site postgresql

  juju add-relation my_django_site gunicorn

  juju expose gunicorn # Open the tcp port in the firewall

You will end up with 3 servers running. One will be the controller

and one for each service (django and the database). 

Gunicorn will be a special charm that will be installed on your Django server.

After that, adding a new Django node would be as simple as::

  juju add-unit my_django_site

As I said, where it gets tricky is how do I make the configuration flexible enough

and at the same time simple.

After looking at what was existing in Django's Paas world, I came with this:

1 - We need a configurable requirements files for both pip and apt-get. By default

it would be looking for package in there files at install time::

  requirements_pip_files: requirements.txt,requirements.pip

  requirements_apt_files: requirements.apt

and we could also configure extra packages by adding variables like this in the YAML file::

  additional_distro_packages: vim,emacs,etc

  additional_pip_packages: virtualenvwrapper,celery,South,etc

2 - I'm suggesting to use separate configurations files in a settings/ directory

so by default it will be injecting configuration in those files::

    settings_database_path: settings/20-engine.py

    settings_static_path: settings/20-static.py

    settings_uploads_path: settings/20-media.py

    settings_cache_path: settings/30-cache.py

    settings_secret_key_path: settings/20-secret.py

I'm suggesting splitting settings because when the configuration is modified,

for some reason, it would be difficult and risky to parse settings.py and 

change only the right thing.

So instead, I would be using topic files rendered with templates.

So if you would need to do more advanced stuff you could just fork the charm

and modify the templates for your needs.

3 - Finally, I was thinking adding some options to execute custom scripts that

would run a various time during the deployment. Like after packages installation

,database configuration and static file configuration::

    post_database_script:

        type: string

        default: |

          #!/bin/sh

          python manage.py syncdb --noinput

          python manage.py migrate --noinput

    post_static_script:

        type: string

        default: |

          #!/bin/sh

          python manage.py collectstatic -v 0 --noinput

Note that this is not making unanimity so far.

There is several reasons that makes the scripts approach tricky:

* You don't want to execute these scripts every time a little detail change.

* You might need the database configuration to be ready for some script.

* You could be not using south

* You might want to import some initial data and maybe only once at install time.

* You could want to compress static files after running collectstatic

* etc

An other idea could be to use a Fabric plug-in that use Juju's database to connect

to the machines and run commands like this for example::

  fab -R my_django_site python manage.py pull

would pull the latest version of the site and reload the application on every

deployed Django machines.

The bottom line here is that it's not simple to find out what a standard

Django deployment (and is maintenance) looks like.

That being said, I'm really looking forwards for you comments and suggestions.

Patrick

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.