Re: Django app crashes with out of memory

Hello. 

Thanks for advices. 

Crash occures, when I run Django command through python manage.py my_command_name.

It doesn't relate to nginx and even gunicorn, because, again, it's Django command, which I call from shell.

I switched off redis caching (I use cacheops), but it' didn't help too.

пятница, 7 октября 2016 г., 13:21:48 UTC+5 пользователь Горобец Дмитрий написал:

Hello.

I have VPS with 2Gb RAM with django, gunicorn, mysql, nginx and redis. 

My app crashes with out of memory error, when I run django command on model, which has approximately 7 million records. It takes each premise object and updates one field by checking value with regular expression. Please, help optimize that command.

class Command(BaseCommand):

    help = 'Updates premise.number_order'

    def handle(self, *args, **options):

        for premise in Premise.objects.iterator():

            premise.number_order = premise.set_number_order()

            premise.save()

        self.stdout.write('Finished')

# Method of Premise model

def set_number_order(self):

    tr = {

        'А': '.10',

        'A': '.10',

        'Б': '.20',

        'В': '.30',

        'Г': '.40',

        'Д': '.50',

        'Е': '.60',

        'Ж': '.70',

        'З': '.80',

        'И': '.90',

    }

    only_digit = re.compile(r'^(?P<number>[0-9]{1,9})$')

    digit_with_separator = re.compile(r'^(?P<number>[0-9]{1,9})(?P<separator>[-|/])(?P<rest>\w+)$')

    digit_with_letter = re.compile(r'^(?P<number>[0-9]{1,9})(?P<letter>[А-Яа-я]+)')

    result = 0

    title = self.title.strip().upper()

    if only_digit.match(title):

        number = only_digit.match(title).group('number')

        result = number + '.00'

    elif digit_with_separator.match(title):

        number = digit_with_separator.match(title).group('number')

        rest = digit_with_separator.match(title).group('rest')

        if rest[0].isalpha():

            floating = tr.get(rest[0], '.90')

            result = number + floating

        elif rest[0].isdigit():

            try:

                if rest[1].isdigit():

                    result = number + '.{}'.format(rest[:2])

                else:

                    result = number + '.0{}'.format(rest[0])

            except IndexError:

                result = number + '.0{}'.format(rest[0])

    elif digit_with_letter.match(title):

        number = digit_with_letter.match(title).group('number')

        letter = digit_with_letter.match(title).group('letter')[0]

        floating = tr.get(letter, '.90')

        result = number + floating

    return Decimal(result)

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/2e87323c-294c-4b82-89dd-783e36e23291%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.