Django models and multiprocessing: "no results to fetch"

Hi all,

in my project, I'm traversing through a huge directory with millions of files, parse them and write the results into a Postgres database.
Unfortunately, I'm facing erros when I'm using multiprocessing.Pool with more than one processes.

This is how I'm inserting new values into the database:

obj, created = MyModel.objects.get_or_create(foo=foo, defaults={'foo': foo,}) # foo is unique


And this is the error message I'm getting: 


[ERROR] no results to fetch
Traceback (most recent call last):
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py", line 97, in inner
    return func(*args, **kwargs)
psycopg2.ProgrammingError: no results to fetch

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/develop/myApp/web/service/ais.py", line 94, in save
    db_foo = utils.create_foo(obj.get("foo"), obj.get("source"))
  File "~/develop/myApp/web/utils.py", line 85, in create_foo
    obj = MyModel.objects.get(foo=foo)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 411, in get
    num = len(clone)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 258, in __len__
    self._fetch_all()
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 1261, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 57, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1177, in execute_sql
    return list(result)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1576, in cursor_iter
    for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1576, in <lambda>
    for rows in iter((lambda: cursor.fetchmany(itersize)), sentinel):
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py", line 97, in inner
    return func(*args, **kwargs)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py", line 97, in inner
    return func(*args, **kwargs)
django.db.utils.ProgrammingError: no results to fetch
[ERROR] error with status PGRES_TUPLES_OK and no message from the libpq
Traceback (most recent call last):
  File "~/develop/myApp/web/utils.py", line 85, in create_foo
    obj = MyModel.objects.get(foo=foo)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 415, in get
    raise self.model.DoesNotExist(
web.models.MyModel.DoesNotExist: MyModel matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
psycopg2.DatabaseError: error with status PGRES_TUPLES_OK and no message from the libpq

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/develop/myApp/web/service/ais.py", line 94, in save
    db_foo = utils.create_foo(obj.get("foo"), obj.get("source"))
  File "~/develop/myApp/web/utils.py", line 89, in create_foo
    obj = MyModel.objects.create(foo=foo)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 433, in create
    obj.save(force_insert=True, using=self.db)
  File "~/develop/myApp/web/models.py", line 34, in save
    return super().save(*args, **kwargs)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/base.py", line 745, in save
    self.save_base(using=using, force_insert=force_insert,
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/base.py", line 782, in save_base
    updated = self._save_table(
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/base.py", line 887, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/base.py", line 924, in _do_insert
    return manager._insert(
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/query.py", line 1204, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1384, in execute_sql
    cursor.execute(sql, params)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "~/.virtualenvs/myApp/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.DatabaseError: error with status PGRES_TUPLES_OK and no message from the libpq

---

Some additional info:

• This does not happen with sqlite
• I'm pretty sure it has something to do with multithreading, since I don't get this error with Pool(processes=1) and the higher the processes-parameter is, the more often does this error occur.
• Versions:
• Django==3.0.3
• psycopg2-binary==2.8.4

Is there any chance I can get this working with ~8 threads? Is there something wrong with how I insert the data?
Unfortunately, I have several models that depend on each other (although I get this error with just inserting the first - didn't even get to the other models), so bulk-insert is not an option.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d32a2dfd-4d27-43a0-9f2d-446e1d6302aa%40googlegroups.com.