Wednesday, September 2, 2020

getattr() issue with copied model instance (2.2.16)

Hello,

- python version: 3.7
- django version: 2.2.16

After upgrading django from 2.2.15 to 2.2.16, I am experiencing errors with the code shown below (see ----code----). Since 2.2.16 the "getattr(self.original, field_name)" line results in the following stack trace, which did not happen in 2.2.15:

--------stack trace---------
 File "/app/src/contrib/changed_model_fields/mixins.py", line 20, in save
    self.has_changed_fields(related_model)
  File "/app/src/contrib/changed_model_fields/mixins.py", line 44, in has_changed_fields
    print(getattr(self.original, field_name))
  File "/usr/local/lib/python3.7/site-packages/django/db/models/fields/related_descriptors.py", line 189, in __get__
    "%s has no %s." % (self.field.model.__name__, self.field.name)
src.core.models.Variant.product.RelatedObjectDoesNotExist: Variant has no product.
--------stack trace---------


The purpose of the "ChangedFieldMixin" is to determine if it contains any modified (changed) attributes. The patch notes of 2.2.16 mentions something, which I believe to be related to my issue: https://code.djangoproject.com/ticket/31863


----------code------------

from copy import copy


class ChangedFieldMixin(object):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.original = copy(self)


    def save(self, related_model: ModelType = None, *args, **kwargs):
        force_changed_field = kwargs.pop('force_changed_field', False)

        if self.has_changed_fields(related_model):
            ...

        super().save(*args, **kwargs)

    def has_changed_fields(self, related_model: ModelType):
        for field in self._meta.fields:
            field_name = str(field.name)

            if not getattr(self, field_name) == getattr(self.original, field_name):
                return True

        return False


class Variant(ChangedFieldMixin, models.Model):
    product = models.ForeignKey('Product', on_delete=models.CASCADE)
    code = models.CharField(max_length=255, null=True, blank=True)
    name = models.CharField(max_length=255, null=True, blank=True)
    key = models.CharField(max_length=255, unique=True, null=False, blank=False)
    quantity = models.PositiveIntegerField(null=True, blank=True)
    default_price = models.PositiveIntegerField(default=None, null=True, blank=True)

----------code------------

Is this expected behaviour now, or did the patch introduce a bug? Let me know if I can provide additional information for the subject.


Thanks in advance

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAC%2BLcyiD9WQK9GX76d1seC%2Bq%2B0Oh%3DhO%3DCZx46-tUR2B-WCAALg%40mail.gmail.com.

Tuesday, September 1, 2020

Re: DB migration: Loading a massive fixture dump

Hi Mike,

Thanks for the answer.

Yes, I was aware yet hopeful regarding the size.

Regarding the converter, I would be interested to take a look at your py3 work, if you could forward it along.

Thanks.


On Wed, Sep 2, 2020, 00:21 Fred Blaise <fred.blaise@gmail.com> wrote:
Hi,

I have to migrate a database from mysql to postgresql. Unfortunately, there is quite a bit of data in there.

As a strategy, I thought of using Django's ORM feature, dump the data from my setup using mysql, and load the fixtures on the other setup using postgresql, such as:

# on mysql
$ docker exec -ti madchap_ddojo_uwsgi_1 bash -c 'python manage.py dumpdata --indent=4 -e sessions -e admin --natural-foreign --natural-primary -e contenttypes -e auth.permission -v2' > ~/Downloads/datadump_orm_dump_1.json

This yields a 4GB json file.

Reload on the other side, such as:

# on postgresql
$  docker exec -ti madchap_ddojo_uwsgi_1 bash -c 'python manage.py loaddata datadump_orm_dump_small.json'

I let the loaddata run for 4hours, and it eventually returns with no error -- but no success either. I actually now think that the message indicating "Installed x object(s) from 1 fixture(s)" does not show, I will see again.

Looking at the postgresql side, querying metatables such as:

SELECT nspname || '.' || relname AS "relation",
    pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
  FROM pg_class C
  LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
  WHERE nspname NOT IN ('pg_catalog', 'information_schema')
    AND C.relkind <> 'i'
    AND nspname !~ '^pg_toast'
  ORDER BY pg_total_relation_size(C.oid) DESC
  LIMIT 5;

tells me loads of data gets in there. a "\d+" does not agree, however (weird, but I don't know enough to get the difference between the two). And yet, I cannot see anything through any SELECT statement (hinting to the transaction isolation).

All things equal, this strategy works fine when working with small json files. I just successfully did it with a file less than 1MB.

I am guessing I have a few questions, I am stuck here.

1. Are there any hidden args/switches to manage.py loaddata or else that I could look at?
2. Are there specific django configuration that could help me load the data?
3. Does a loaddata actually result in a massive single transaction? (looks like so, sounds crazy to me :))
4. Is there a better way to migrate my data?

Thanks a lot for any pointers!
Cheers.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d6f5e874-e96b-4984-b69d-d7857e95a0a4n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAEhd%3DqfEQfxu_DxF6cJwUSfnY9fJ-y%2BKF8hSO0CfnXtRqsx%3DNw%40mail.gmail.com.

Re: DB migration: Loading a massive fixture dump

-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEE/NCg7Xf1UydoVFgpGvW31BqCOLMFAl9PHh4ACgkQGvW31BqC
OLNbvgf+P1OQ/vR5hq0SqGiZjc4Wqjnprnl9YG9GSZ/fmK/8SfritRI2CzI13F19
DIGDXGuvxOKXo0KblQDihPVzBbndNtFKD0HTXRbDNyuDGOeTS2uDAk+I2G+zoJYS
jXaPD5SUJazfS04j1q75kZ8kYW6L4qITzU/PYh0CTixv0SY9r20JGIfE2jX7v9zs
fezuUzgxU2TZvJradugKoqw87UWeMxwq5YR3yBs+fU2h6xq8ZjPMIgtY815tPn/V
uyPAL2msWr7zAymFSWFsjb4R9rJmLlXb0hvuLhNMsptus5yocfu/GL6iXp2/DSpe
xvhqk5U9SLv9cTTA6aqOM9s082LJmA==
=aARv
-----END PGP SIGNATURE-----
On 2/09/2020 3:51 am, Fred Blaise wrote:
> Hi,
>
> I have to migrate a database from mysql to postgresql. Unfortunately,
> there is quite a bit of data in there.
>
> As a strategy, I thought of using Django's ORM feature, dump the data
> from my setup using mysql, and load the fixtures on the other setup
> using postgresql, such as:
>
> # on mysql
> $ docker exec -ti madchap_ddojo_uwsgi_1 bash -c 'python manage.py
> dumpdata --indent=4 -e sessions -e admin --natural-foreign
> --natural-primary -e contenttypes -e auth.permission -v2' >
> ~/Downloads/datadump_orm_dump_1.json
>
> This yields a 4GB json file.
>
> Reload on the other side, such as:
>
> # on postgresql
> $  docker exec -ti madchap_ddojo_uwsgi_1 bash -c 'python manage.py
> loaddata datadump_orm_dump_small.json'
>
> I let the loaddata run for 4hours, and it eventually returns with no
> error -- but no success either. I actually now think that the message
> indicating "Installed x object(s) from 1 fixture(s)" does not show, I
> will see again.
>
> Looking at the postgresql side, querying metatables such as:
>
> SELECT nspname || '.' || relname AS "relation",
>     pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
>   FROM pg_class C
>   LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
>   WHERE nspname NOT IN ('pg_catalog', 'information_schema')
>     AND C.relkind <> 'i'
>     AND nspname !~ '^pg_toast'
>   ORDER BY pg_total_relation_size(C.oid) DESC
>   LIMIT 5;
>
> tells me loads of data gets in there. a "\d+" does not agree, however
> (weird, but I don't know enough to get the difference between the
> two). And yet, I cannot see anything through any SELECT statement
> (hinting to the transaction isolation).
>
> All things equal, this strategy works fine when working with small
> json files. I just successfully did it with a file less than 1MB.
>
> I am guessing I have a few questions, I am stuck here.
>
> 1. Are there any hidden args/switches to manage.py loaddata or else
> that I could look at?

loaddata is intended for small imports such as skeleton data for unit
testing


> 2. Are there specific django configuration that could help me load the
> data?
> 3. Does a loaddata actually result in a massive single transaction?
> (looks like so, sounds crazy to me :))

Probably does because that would be what you most likely want. But as
you have discovered it probably runs out of resources if the load is too
big.

> 4. Is there a better way to migrate my data?

Yes.

https://github.com/lanyrd/mysql-postgresql-converter/blob/master/db_converter.py

I used this successfully a few months ago. I had to convert it from
py2.7 to 3.x.

Briefly, you will have to dump MySQL again using a similar incantation
to this  ...

mysqldump --compatible=postgresql --default-character-set=utf8 -r
databasename.mysql -u root databasename

If you (or anyone) would like a copy of my adjusted db_converter.py,
drop me a line off-list.

Cheers

Mike

> Thanks a lot for any pointers!
> Cheers.
> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users+unsubscribe@googlegroups.com
> <mailto:django-users+unsubscribe@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/d6f5e874-e96b-4984-b69d-d7857e95a0a4n%40googlegroups.com
> <https://groups.google.com/d/msgid/django-users/d6f5e874-e96b-4984-b69d-d7857e95a0a4n%40googlegroups.com?utm_medium=email&utm_source=footer>.


--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/83fa51d5-7e52-a1fd-694b-d20ab295bb51%40dewhirst.com.au.

Re: DJANGO ADMINISTRATION ERROR & TemplateDoesNotExist

Hi King Niko,

On 01/09/2020 07.43, King Niko wrote:
> The console does not seem to provide much information. It is simply
> stating that there is a 500 Server Error. I tried sifting for more
> details but there were none.
>

The console in your browser is not providing any information because it
doesn't make much sense to try and debug a *Server Error* on the client
side.

Ignore the advice from RANGA BHARATH JINKA on looking in your browsers
console.


> However, in the:
>
> *> heroku logs --tail *
> *
> *
> It simply mentions that none of the paths exist to /the cart / and /the
> images /that derive from the DJANGO ADMINISTRATION page.

That sounds a lot more useful, but are you really sure you're seeing the
logs from Django and not just the web server forwarding requests to your
Django application?

You need to get the logs from your actual Django application to see the
actual cause of the error. I remember there has been other discussions
on getting those logs from Heroku, but I have no experience with Heroku
so I don't know how to achieve that, but you need access to those logs
sooner or later so start by figuring out how to do that.

Kind regards,

Kasper Laudrup

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/299d7a71-faf2-71aa-fdfb-d5b53bc6127b%40stacktrace.dk.

DB migration: Loading a massive fixture dump

Hi,

I have to migrate a database from mysql to postgresql. Unfortunately, there is quite a bit of data in there.

As a strategy, I thought of using Django's ORM feature, dump the data from my setup using mysql, and load the fixtures on the other setup using postgresql, such as:

# on mysql
$ docker exec -ti madchap_ddojo_uwsgi_1 bash -c 'python manage.py dumpdata --indent=4 -e sessions -e admin --natural-foreign --natural-primary -e contenttypes -e auth.permission -v2' > ~/Downloads/datadump_orm_dump_1.json

This yields a 4GB json file.

Reload on the other side, such as:

# on postgresql
$  docker exec -ti madchap_ddojo_uwsgi_1 bash -c 'python manage.py loaddata datadump_orm_dump_small.json'

I let the loaddata run for 4hours, and it eventually returns with no error -- but no success either. I actually now think that the message indicating "Installed x object(s) from 1 fixture(s)" does not show, I will see again.

Looking at the postgresql side, querying metatables such as:

SELECT nspname || '.' || relname AS "relation",
    pg_size_pretty(pg_total_relation_size(C.oid)) AS "total_size"
  FROM pg_class C
  LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
  WHERE nspname NOT IN ('pg_catalog', 'information_schema')
    AND C.relkind <> 'i'
    AND nspname !~ '^pg_toast'
  ORDER BY pg_total_relation_size(C.oid) DESC
  LIMIT 5;

tells me loads of data gets in there. a "\d+" does not agree, however (weird, but I don't know enough to get the difference between the two). And yet, I cannot see anything through any SELECT statement (hinting to the transaction isolation).

All things equal, this strategy works fine when working with small json files. I just successfully did it with a file less than 1MB.

I am guessing I have a few questions, I am stuck here.

1. Are there any hidden args/switches to manage.py loaddata or else that I could look at?
2. Are there specific django configuration that could help me load the data?
3. Does a loaddata actually result in a massive single transaction? (looks like so, sounds crazy to me :))
4. Is there a better way to migrate my data?

Thanks a lot for any pointers!
Cheers.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d6f5e874-e96b-4984-b69d-d7857e95a0a4n%40googlegroups.com.

Adding to User new variable / money system in game

New to Django,

I want create a money system in my game that each user have a different coins

I tried add to User in views a variable 'balance'

My views.py:

  1. from monety.models import Gamer
  2. from django.contrib.auth.models import User 
  3. from .forms import RegisterForm

  4. def registration(response):
  5.     if response.method == "POST":
  6.         form = RegisterForm(response.POST)
  7.         if form.is_valid():
  8.             user = User.objects.all().filter(username=username)[0] 
  9.             gamer = Gamer(myuser=user)
  10.             form.save()
  11.             gamer.save()
  12.             return redirect('/garden')
  13.     else:
  14.         form = RegisterForm()

  15.     return render(response, "registration.html", {"form":form})
My models.py

  1. from django.db import models
  2. from django.contrib.auth.models import User 
  3.  
  4. class Gamer(models.Model):
  5.     myuser = models.OneToOneField(
  6.         User, 
  7.         on_delete=models.CASCADE
  8.     )
  9.     balance = models.IntegerField(default=100)
My forms.py

  1. from django.contrib.auth import login, authenticate
  2. from django.contrib.auth.forms import UserCreationForm
  3. from django import forms
  4. from django.contrib.auth.models import User

  5. class RegisterForm(UserCreationForm):
  6.     email = forms.EmailField()

  7.     class Meta:
  8.         model = User
  9.         fields = ["username", "email", "password1", "password2"]
I get "NameError: name 'username' is not defined" and I have no idea how to fix it to make it work

(sorry for my bad english)

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/7e414e72-0665-475b-bcdd-1eb66d905b83n%40googlegroups.com.

Django security releases issued: 3.1.1, 3.0.10 and 2.2.16

Today the Django team issued 3.1.1, 3.0.10 and 2.2.16 as part of our security process. These releases address two security issues, and we encourage all users to upgrade as soon as possible:

https://www.djangoproject.com/weblog/2020/sep/01/security-releases/

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/10D39764-E9D0-4A59-8053-E2960F777BE4%40gmail.com.