[Solved]: Unit testing migrations [Was: Related model 'company.user' cannot be resolved]

On 7/07/2021 9:56 am, Mike Dewhirst wrote:
> It seems I have to meddle with history and persuade the testing
> harness that my custom user model has been in the company app from the
> beginning of time. See story below.
>
> Upside is that testing will (might) start working again. Downside is
> I'll go mad.

All upside!

So in summary ...

0. The mission was to move all models (User and UserProfile) from the
common app to the company app so users and companies would both appear
in the Admin in the same app. This simplifies the site adminstration
main menu for new users.

1. In class Meta: in both models User/UserProfile, add db_table =
common_user/userprofile[1]. This will have no effect and should be
committed, migrated and deployed to production.

2. Branch the code and switch to create an escape route.

3. Relocate User and UserProfile models from app common to app company[2]

4. Refactor entire codebase using company_ instead of common_ [3]

5. makemigrations[4] but do not migrate to avoid deleting tables.

Now the tricky bit. What I did was probably wrong and any expert is
welcome to suggest an improvement but because of steps 1 and 2 there was
always a retreat. The mission here is to tweak the migrations so the
test harness can create a test database without error.

6. Delete the migration which deletes User and UserProfile tables from
common

7. Edit company/migrations/0001_initial.py to include the contents of
the latest migration which created User and UserProfile in company so
that the test harness thinks they have been there since the beginning.
Then delete that latest company migration too.

8. Delete all migrations in common/migrations including __pycache__ [5]

9. Cross fingers and run your tests. Resolve any errors by adjusting
dependencies which includes positioning User and UserProfile higher or
lower in 0001_initial.py in company/migrations[6].

10. Consider editing your django_migrations table to remove rows
relating to deleted migrations. This is for future-proofing in case you
ever decide to put any tables back in the common app. If 0001_initial is
mentioned in that table it will be a confusing preventer.

11. Merge your successful changes back to your main branch and deploy.


[1] One day I will rename the tables so everything lines up but for now
that is a step too far

[2] I did try leaving the models in place and importing them into the
company app but things quickly got messy. Much easier to bite the bullet
and just physically move the files.

[3] At this point the software should work but unit testing will fail
because the test database is always generated from the sum total of all
migrations

[4] This will make table deletions from common and table creations in
company BUT the creations will be for the current state. This becomes
significant later in the process.

[5] All migrations in common/migrations were for the only two models in
the common app. Because the creation migration was for the current
state, none of the other migrations were needed. However, your mileage
may vary perhaps if your migrations did other things than just changing
structure etc. If you also did some data tweaking in some migrations you
will want to think a bit harder before deleting them.

[6] In my case I needed to put User at the beginning and UserProfile at
the end of that migration. Here is the relevant piece of my
company/migrations/0001_initial.py  ... note the db_table in options ...



class Migration(migrations.Migration):

initial = True

dependencies = [
('auth', '0012_alter_user_first_name_max_length'),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]

operations = [
migrations.CreateModel(
name='User',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('password', models.CharField(max_length=128, verbose_name='password')),
('last_login', models.DateTimeField(blank=True, null=True, verbose_name='last login')),
('is_superuser', models.BooleanField(default=False, help_text='Designates that this user has all permissions without explicitly assigning them.', verbose_name='superuser status')),
('username', models.CharField(error_messages={'unique': 'A user with that username already exists.'}, help_text='Required. 150 characters or fewer. Letters, digits and @/./+/-/_ only.', max_length=150, unique=True, validators=[django.contrib.auth.validators.UnicodeUsernameValidator()], verbose_name='username')),
('date_joined', models.DateTimeField(default=django.utils.timezone.now, verbose_name='date joined')),
('first_name', models.CharField(blank=True, max_length=30, verbose_name='short name')),
('last_name', models.CharField(blank=True, max_length=150, verbose_name='full name')),
('email', models.EmailField(blank=True, help_text='This is the address used for a forgotten password reset token', max_length=254, verbose_name='email address')),
('is_staff', models.BooleanField(default=True, help_text='Designates whether the user can log into this admin site', verbose_name='staff status')),
('is_active', models.BooleanField(default=True, help_text='Designates if this account is usable. Uncheck for users away on leave.', verbose_name='active')),
('groups', models.ManyToManyField(blank=True, help_text='The groups this user belongs to. A user will get all permissions granted to each of their groups.', related_name='user_set', related_query_name='user', to='auth.Group', verbose_name='groups')),
('user_permissions', models.ManyToManyField(blank=True, help_text='Specific permissions for this user.', related_name='user_set', related_query_name='user', to='auth.Permission', verbose_name='user permissions')),
],
options={
'verbose_name': 'user',
'verbose_name_plural': 'users',
'db_table': 'common_user',
'abstract': False,
'swappable': 'AUTH_USER_MODEL',
},
managers=[
('objects', company.models.user.CaseInsensitiveUserManager()),
],
),

... inital company migrations which are untouched ...

migrations.CreateModel(
name='UserProfile',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('job_title', models.CharField(blank=True, default='', help_text="Only required for users in the 'manager' role. Used for AICIS annual declarations.", max_length=128, null=True, verbose_name='Job title')),
('cellphone', models.CharField(blank=True, default='', help_text='In the form +61 (0)411 704 143. A valid phone number is required here if this user needs admin permissions.', max_length=32, null=True, verbose_name='Mobile or cellphone')),
('region', models.CharField(blank=True, choices=[('au', 'Australia'), ('ca', 'Canada'), ('cn', 'China'), ('de', 'Germany'), ('es', 'Spain'), ('fr', 'France'), ('ie', 'Ireland'), ('in', 'India'), ('it', 'Italy'), ('jp', 'Japan'), ('nl', 'The Netherlands'), ('nz', 'New Zealand'), ('uk', 'United Kingdom'), ('us', 'United States')], default='au', help_text='Used for selecting currency and GST/VAT rules if necessary.', max_length=2)),
('created', models.DateTimeField(auto_now_add=True)),
('modified', models.DateTimeField(auto_now=True)),
('company', models.ForeignKey(blank=True, help_text='Each user may only be associated with <strong>one</strong> company', null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='usercompany', to='company.company')),
('user', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='userprofile', to=settings.AUTH_USER_MODEL)),
],
options={
'verbose_name': 'user profile',
'verbose_name_plural': 'user profile',
'db_table': 'common_userprofile',
},
),
]

Cheers

Mike


>
> Any ideas?
>
> Thanks
>
> Mike
>
> On 6/07/2021 11:40 am, Mike Dewhirst wrote:
>> I moved my custom user and userprofile models from my 'common' app
>> into my 'company' app by way of locking the model tables first
>> (db_table = "common_user" and so on) and just plain refactoring.
>>
>> That has all worked sort of as expected in that Company and User
>> models now appear in the company app space in the Admin which was my
>> original motivation.
>>
>> makemigrations wanted to destroy data with delete/create models in
>> order to move them and of course I couldn't let that happen. migrate
>> --fake and migrate --fake-initial wouldn't work due to lots of other
>> models looking for settings.AUTH_USER_MODEL which Django couldn't
>> find in the company app so I finessed my own fake migrations. That
>> also seemed to work and the job seemed done.
>>
>> BUT manage.py test now fails with "Related model 'company.user'
>> cannot be resolved"
>>
>> StackOverflow seems to indicate my finessed migrations are at fault.
>> I have tried to tweak the dependencies with no luck as follows ...
>>
>> # company.migrations.0005_user_userprofile.py
>> ...
>>     dependencies = [
>>         ('auth', '0012_alter_user_first_name_max_length'),
>>         ('company', '0004_auto_20210510_1431'),       # older pre
>> move migration
>>         ('common', '0008_auto_20210705_1740'),        # see below
>>     ]
>>
>>     operations = [
>>         migrations.CreateModel(
>>             name='User',
>>             fields=[
>>                 ('id', models.BigAutoField(auto_created=True,
>> primary_key=True, serialize=False, verbose_name='ID')),
>>                 ('password', models.CharField(max_length=128,
>> verbose_name='password')),
>> ... etc for creating both User and UserProfile models
>>
>> # common.migrations.0008_auto_20210705_1740.py
>> ...
>>     dependencies = [
>>         ('common', '0007_userprofile_job_title'),    # older pre-move
>> migration
>>     ]
>>
>>     operations = [
>>         migrations.RemoveField(
>>             model_name='userprofile',
>>             name='company',
>>         ),
>>         migrations.RemoveField(
>>             model_name='userprofile',
>>             name='user',
>>         ),
>>         migrations.DeleteModel(
>>             name='User',
>>         ),
>>         migrations.DeleteModel(
>>             name='UserProfile',
>>         ),
>>     ]
>>
>>
>> Django 3.2.5, Python 3.8
>>
>> I feel I should have zigged when I should've zagged. Thanks for any
>> suggestions
>>
>> Mike
>>
>> Traceback (most recent call last):
>>   File "manage.py", line 21, in <module>
>>     main()
>>   File "manage.py", line 17, in main
>>     execute_from_command_line(sys.argv)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\core\management\__init__.py",
>> line 419, in execute_from_command_line
>>     utility.execute()
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\core\management\__init__.py",
>> line 413, in execute
>>     self.fetch_command(subcommand).run_from_argv(self.argv)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\core\management\commands\test.py",
>> line 23, in run_from_argv
>>     super().run_from_argv(argv)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\core\management\base.py",
>> line 354, in run_from_argv
>>     self.execute(*args, **cmd_options)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\core\management\base.py",
>> line 398, in execute
>>     output = self.handle(*args, **options)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\core\management\commands\test.py",
>> line 55, in handle
>>     failures = test_runner.run_tests(test_labels)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\test\runner.py",
>> line 725, in run_tests
>>     old_config = self.setup_databases(aliases=databases)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\test\runner.py",
>> line 643, in setup_databases
>>     return _setup_databases(
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\test\utils.py",
>> line 179, in setup_databases
>>     connection.creation.create_test_db(
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\backends\base\creation.py",
>> line 74, in create_test_db
>>     call_command(
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\core\management\__init__.py",
>> line 181, in call_command
>>     return command.execute(*args, **defaults)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\core\management\base.py",
>> line 398, in execute
>>     output = self.handle(*args, **options)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\core\management\base.py",
>> line 89, in wrapped
>>     res = handle_func(*args, **kwargs)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\core\management\commands\migrate.py",
>> line 244, in handle
>>     post_migrate_state = executor.migrate(
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\migrations\executor.py",
>> line 117, in migrate
>>     state = self._migrate_all_forwards(state, plan, full_plan,
>> fake=fake, fake_initial=fake_initial)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\migrations\executor.py",
>> line 147, in _migrate_all_forwards
>>     state = self.apply_migration(state, migration, fake=fake,
>> fake_initial=fake_initial)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\migrations\executor.py",
>> line 227, in apply_migration
>>     state = migration.apply(state, schema_editor)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\migrations\migration.py",
>> line 126, in apply
>>     operation.database_forwards(self.app_label, schema_editor,
>> old_state, project_state)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\migrations\operations\models.py",
>> line 92, in database_forwards
>>     schema_editor.create_model(model)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\backends\base\schema.py",
>> line 343, in create_model
>>     sql, params = self.table_sql(model)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\backends\base\schema.py",
>> line 162, in table_sql
>>     definition, extra_params = self.column_sql(model, field)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\backends\base\schema.py",
>> line 215, in column_sql
>>     db_params = field.db_parameters(connection=self.connection)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\models\fields\related.py",
>> line 1004, in db_parameters
>>     return {"type": self.db_type(connection), "check":
>> self.db_check(connection)}
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\models\fields\related.py",
>> line 1001, in db_type
>>     return self.target_field.rel_db_type(connection=connection)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\models\fields\related.py",
>> line 897, in target_field
>>     return self.foreign_related_fields[0]
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\utils\functional.py",
>> line 48, in __get__
>>     res = instance.__dict__[self.name] = self.func(instance)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\models\fields\related.py",
>> line 644, in foreign_related_fields
>>     return tuple(rhs_field for lhs_field, rhs_field in
>> self.related_fields if rhs_field)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\utils\functional.py",
>> line 48, in __get__
>>     res = instance.__dict__[self.name] = self.func(instance)
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\models\fields\related.py",
>> line 632, in related_fields
>>     return self.resolve_related_fields()
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\models\fields\related.py",
>> line 936, in resolve_related_fields
>>     related_fields = super().resolve_related_fields()
>>   File
>> "D:\Users\mike\envs\xxai\lib\site-packages\django\db\models\fields\related.py",
>> line 615, in resolve_related_fields
>>     raise ValueError('Related model %r cannot be resolved' %
>> self.remote_field.model)
>> ValueError: Related model 'company.user' cannot be resolved
>>
>>
>
>


--
Signed email is an absolute defence against phishing. This email has
been signed with my private key. If you import my public key you can
automatically decrypt my signature and be sure it came from me. Just
ask and I'll send it to you. Your email software can handle signing.


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/a8ae6974-3fbc-0103-06db-fd79518136b6%40dewhirst.com.au.