Re: Django app crashes with out of memory

These are all memory hungry applications. Please provide some detail that how many worker processes initialized by the gunicorn and how do you bind it with Nginx. Nginx configurations also required to see details. Plus use "free -m" to see how much memory you've got allocated to all services. Use "df -h" to see hard drive space available. Finally if you have a swap file then what's the size. 

You can create swapfile with "sudo fallocate -l 4G /swapfile" but after getting this message it might not be good idea to do it now. Or if you have feeling that swapping might solve the is issue  you can backup the code and re-create your droplet and manage swapping from the start. 4GB swapzie in total will be enough I guess.

If I were you I would comment out redis and see if that solves problem. Not sure what other applications you may have but remember that Python consumes a lot of memory and using redis for cache management is really a good tool. But in some cases depending on your settings it needs some good processing power with memory to manage cache. This is why it is suitable for large scale projects and for middle size projects cache is managed within Django. 

I cannot pin point the solution for you right away because according to the scenario there could be multiple reasons. You can also configure Nginx for browser based cache at user end to minimize load. 

For piece of code you've provided looks like no issue there as per my limited understanding. I had issues with PHP based CMS but never had issue with Django/Bottle/Pyramid. 

Hope something might point you to the right direction but for now that's all I can say with provided information. But if solution is different than one of the above reasons please share with group as well. It is difficult to recreate the error using your exact scenario so hard to come up with exact solution.

Good Luck!

Regards,

Mudassar

On Fri, Oct 7, 2016 at 1:21 AM, Горобец Дмитрий <dmitra90@gmail.com> wrote:

Hello.

I have VPS with 2Gb RAM with django, gunicorn, mysql, nginx and redis. 

My app crashes with out of memory error, when I run django command on model, which has approximately 7 million records. It takes each premise object and updates one field by checking value with regular expression. Please, help optimize that command.

class Command(BaseCommand):

    help = 'Updates premise.number_order'

    def handle(self, *args, **options):

        for premise in Premise.objects.iterator():

            premise.number_order = premise.set_number_order()

            premise.save()

        self.stdout.write('Finished')

# Method of Premise model

def set_number_order(self):

    tr = {

        'А': '.10',

        'A': '.10',

        'Б': '.20',

        'В': '.30',

        'Г': '.40',

        'Д': '.50',

        'Е': '.60',

        'Ж': '.70',

        'З': '.80',

        'И': '.90',

    }

    only_digit = re.compile(r'^(?P<number>[0-9]{1,9})$')

    digit_with_separator = re.compile(r'^(?P<number>[0-9]{1,9})(?P<separator>[-|/])(?P<rest>\w+)$')

    digit_with_letter = re.compile(r'^(?P<number>[0-9]{1,9})(?P<letter>[А-Яа-я]+)')

    result = 0

    title = self.title.strip().upper()

    if only_digit.match(title):

        number = only_digit.match(title).group('number')

        result = number + '.00'

    elif digit_with_separator.match(title):

        number = digit_with_separator.match(title).group('number')

        rest = digit_with_separator.match(title).group('rest')

        if rest[0].isalpha():

            floating = tr.get(rest[0], '.90')

            result = number + floating

        elif rest[0].isdigit():

            try:

                if rest[1].isdigit():

                    result = number + '.{}'.format(rest[:2])

                else:

                    result = number + '.0{}'.format(rest[0])

            except IndexError:

                result = number + '.0{}'.format(rest[0])

    elif digit_with_letter.match(title):

        number = digit_with_letter.match(title).group('number')

        letter = digit_with_letter.match(title).group('letter')[0]

        floating = tr.get(letter, '.90')

        result = number + floating

    return Decimal(result)

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/6ccbf103-c24b-4e3a-982d-4e5db0f01972%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CANoUts56PV0MVE7inGKkXOSoTLw4_riiZNndKFqxXpe_1abOtQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.