Re: Running pending migrations in Middleware

The reason I want to do this by the way, is because I'm using Google AppEngine and I have no access to the server. The alternative is to have the exact same code checked out on my local machine. Connect to the remote database with a special proxy tool that google provides (so the db becomes available on a local port) and then run the migrations. Problems with this are: 1) Hitting the remote database via the proxy is extremely slow, like ridiculously slow and 2) there is a chance I have the wrong local code. ie. this could mean I run some extra migrations or not run enough migrations, thus causing production to be broken.

Thanks for bringing up those points.

All middleware are run for every request 

Yep, this is an issue if "migrate" is called on every request. I solved this by recording whether or not migrations have been run or not on the current version of the app, and also whether it was successful or not. So the only hit I take on every request one query, which will be cached in the db and could also be cached in memory if I really wanted to. Currently, with just the one query it's like 10ms when I do it locally. Not the bottleneck.

What you're doing can slow down your request response time to minutes, depending on how many migrations you have.

What's the alternative? Put the app in a maintenance mode? Either way the app will be unusable for a bit. I guess there is an issue where the requests could exceed the maximum request time (in my case in AppEngine it is 60 seconds) and so if the migration takes longer that, that would be a problem. I could kick off migration asynchonously?

Depending on your database, migrations might not be atomic thus causing conflicts 

I'm using Postgresql so my understanding is my migrations can all run atomically.

On Sunday, July 28, 2019 at 3:05:47 AM UTC-7, Markus Holtermann wrote:

Yes, I do see several problems. Some of them:

- All middleware are run for every request
- What you're doing can slow down your request response time to minutes, depending on home many migrations you have
- Depending on your database, migrations might not be atomic thus causing conflicts
- You have no way to intervene when something goes south
- ...

I can elaborate in more detail when you want. But please, don't run migrations as part of the request-response cycle.

/Markus

On Sun, Jul 28, 2019, at 3:48 PM, David Grant wrote:
> Anyone see any problems with running migrations in Middleware?
>
> import logging
> import os
> import time
>
> from django.core.management import call_command
> from django.http import HttpResponse
>
> from products.models.models import AppVersion
>
> LOG = logging.getLogger(__name__)
>
>
> # Used for testing locally when the GAE_VERSION environment is not set
> FAKE_VERSION_TESTING = 8
>
>
> class MigrateMiddleware:
>
>     def __init__(self, get_response):
>         self.get_response = get_response
>
>     def __call__(self, request):
>         start_time = time.time()
>         try:
>             LOG.info("Running migrate middleware")
>             # Figure out what version of our app is now deployed        
>                 app_version, created =
> AppVersion.objects.get_or_create(version=os.getenv('GAE_VERSION',
> FAKE_VERSION_TESTING))
>             if not app_version.migrations_succeeded or created:        
>        # Previous attempt at this deployed version failed, or this was
> a new deployment.
>                 try:
>                     call_command('migrate')
>                 except:
>                     return HttpResponse("Running a Django migration
> failed! Investigate.", status=500)
>                 app_version.migrations_succeeded = True
>                 app_version.save()
>         finally:
>             LOG.info("Took: %ss", time.time() - start_time)
>         return self.get_response(request)
>
>
> It seems to work just fine in a quick test.
>
>  --
>  You received this message because you are subscribed to the Google
> Groups "Django users" group.
>  To unsubscribe from this group and stop receiving emails from it, send
> an email to django...@googlegroups.com.
>  To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/bdfd6393-6d68-4285-b5a6-eb1be98ef467%40googlegroups.com <https://groups.google.com/d/msgid/django-users/bdfd6393-6d68-4285-b5a6-eb1be98ef467%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/756ca689-59d8-4e0f-8806-107f779e8c12%40googlegroups.com.