Re: Bulk-creating two ForeignKey linked models - id field for relationship isn't set properly?

Victor Hooi, did you ever find a work-around?

I'm having the same problem of trying to efficiently create models that have a Many to One relationship (so that they are linked by a Foreign Key).

The ticket referenced above is still open and I manually checked myself that using bulk_create does not update the pk_id field of a model.

One work-around I thought of was to have a dictionary from a Product to it's ProductImages.
In the main loop, create the ProductImage without specifying a Product and just map the ProductImage to it's Product using the dictionary.

Then call bulk_create on the Products.

Then loop through the dictionary, update each ProductImage's product to the newly created Product.

Then bulk_create the ProductImages.

On Tuesday, August 20, 2013 at 5:27:36 PM UTC-4, Victor Hooi wrote:

Hi,

Cool, thanks for the link to the ticket. Very interesting reading, and I learnt something =).

Apparently the ticket says the patch still needs docs/tests - guess it'll be a while before this gets integrated then...hmm.

Cheers,

Victor

On Wed, Aug 21, 2013 at 2:11 AM, Simon Charette <chare...@gmail.com> wrote:

This is a known limitation of `bulk_create`: objects used for bulk creation are not assigned a primary key.

Le mardi 20 août 2013 05:57:52 UTC-4, Victor Hooi a écrit :

Hi,

1. Bulk Creating Products/ProductImages

I have a Django custom management command that bulk-loads a list of products and product images from a JSON file.

I have a Product model, along with an ProductImage model - each Product may have many ProductImages.

class Product(models.Model):
    ...
    name = models.CharField(max_length=200, help_text='Name of this product.')
    description = models.TextField(help_text='Short description of this product.')
    external_id = models.CharField(max_length=100, unique=True, help_text='Unique identifier for this product provided by the supplier.')
    brand = models.ForeignKey(Brand)
    supplier = models.ForeignKey(Supplier)
    selling_price = models.DecimalField(max_digits=10, decimal_places=2, help_text='The price which we\'re selling this product at.')
    original_price = models.DecimalField(max_digits=10, decimal_places=2, help_text='The original retail price of this product, before any discounts.')
    current_stock_level = models.PositiveIntegerField(default=0)
    current_sold_count = models.PositiveIntegerField(default=0)
    ...

class ProductImage(models.Model):
    product = models.ForeignKey(Product, related_name='images')
    image = models.ImageField(max_length=200, upload_to='product_images')
    # TODO - Do we actually want to make retina images a distinct column in the database? May need to revisit this.
    retina_image = models.ImageField(max_length=200, upload_to='product_images')
    size = models.ForeignKey(ImageSize)
    ordinal = models.PositiveIntegerField(default=0)
    class Meta:
        ordering = ['ordinal']
        unique_together = ('product', 'size', 'ordinal')
    def __unicode__(self):
        return u'%s image %i for %s' % (self.size, self.ordinal, self.product.name)

I have a single look that iterates through the JSON file, creating a list of Products, as well as a list of ProductImages that link back to those products. (The below is an extract)

        products = []
        product_images = []
        ...
        for product in json_data:
            brand, created_new_brand = Brand.objects.get_or_create(name=product['brand'])
            if created_new_brand:
                new_brands_created += 1
            ...
                current_product = Product(name = product['name'],
                                        brand = brand,
                                        supplier = supplier,
                                        description = product['description'],
                                        ...
                                )
                ...
                for image_set in product['images']:
                    for size in image_sizes.keys():
                        product_images.append(ProductImage(product = current_product,
                                                           image = image_set[size],
                                                           retina_image = image_set[size],
                                                           size = image_sizes[size],
                                                           ordinal = image_set_counter,
                                                           ))
                    image_set_counter += 1
        ...
        Product.objects.bulk_create(products)
        ...
        for product_image in product_images:
            product_image.save()

I then call bulk_create() on the products list, which succeeds.

However, when I try to call .save() or bulk_create() on the ProductImage, they fail, saying that product_id is NULL.

IntegrityError: null value in column "product_id" violates not-null constraint
DETAIL:  Failing row contains (8, null, https://cdn.foobar.com.au/site_media/uploads/product_im..., https://cdn.foobar.com.au/site_media/uploads/product_im..., 4, 0).

If I actually inspect one one of the items in the ProductImages list - product_image.product seems to point to a valid product, however, product_image.product_id seems to be None:

ipdb> p product_image.product
<Product: Widget 101 #4383>
ipdb> p product_image.product_id
None

This is strange, because if I manually create the same ProductImage from the shell, the product_image.product_id field is populated.

I'm guessing this product_id field should resolve to the pk id of the product in product_image.product, right?

However, in my case, when I instantiate each ProductImage and set it to a Product, that actual Product hasn't been saved to the database - I'm guessing that Product.id hasn't been generated yet, and the ProductImage object I have is somehow broken?

Is there another way I can achieve this then? (Loop through file, creating a list of Products and ProductImages, then bulk_creating them?)

2. Image Set Counter

Also, second question - in the last code segment above, I have a image_set_counter that I use to set the ordinal - basically, product['images'] is actually a list of several images. I'm using the counter to set the ordinal. However, am I better off doing some kind of:

for image_set_counter in xrange(len(product['images'])):
    ...

       product['images'][image_set_counter] 

and using product['images'][image_set_counter] to access it?

Cheers,

Victor

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/3167ad29-7301-49dc-a778-a987ccd6f238%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.