Yes it seems that I can influence to the 'jumps' with my management command. First my bot went through tens of items per loop, however when I changed the bot to check only 2 items per loop, the 'jumps' got smaller. So even though the table rows do not accrue (nothing is added) when no new items are found in a loop, the ID numbers still accrue behind the scenes in every loop.
So I just gotta figure out if there are any other way than INSERT OR IGNORE command, or just decrease the sleep time of the bot and keep the checked item numbers low per a loop.
I actually tried ROLLBACK and some other commands as well, but they killed the loop.
Thanks for the help ^^
-Mikko
ma 27. elok. 2018 klo 15.01 Andréas Kühne (andreas.kuhne@hypercode.se) kirjoitti:
I have looked at this several times myself. :-)Regards,Andréas--Den mån 27 aug. 2018 kl 13:33 skrev Jason <jjohns98684@gmail.com>:oh, good catch. I didn't think of that, but you're right.--https://stackoverflow.com/questions/449346/mysql-auto-increment-does-not-rollback
On Monday, August 27, 2018 at 3:50:28 AM UTC-4, Andréas Kühne wrote:The management command could be wrapped within a transaction, this would then in the database create a lot of models, but if the transaction fails - the rollback will not reset the database id.Regards,AndréasDen sön 26 aug. 2018 kl 12:29 skrev Mikko Meronen <mikkovil...@gmail.com>:Hi,Thanks for advice ^^I have one new concern. Is there a limit for for Django model IDs?Now I have automated the data insert with python code loop, but the ID-number is not consecutive. It jumps for example from ...56 to 571... and from ...587 to 3763... and so forth.Do you know why it is doing that?I'm not using django custom command (if that makes the difference or can be used for my case, I will get familiar with it a bit later), I'm just inserting the data to django's database with separate python code.-Mikko--ma 20. elok. 2018 klo 20.41 Jani Tiainen (red...@gmail.com) kirjoitti:Hi,Instead of doing everything by hand I would recommend doing a custom management command in Django.Because management commands live inside Django ecosystem you get full benefits of Django. Migrations to setup database tables from model definitions. Models and ORM to interact with database. (Model)forms to validate your incoming data etc.Also I really suggest that you do the official tutorial from the docs to get hang of basic django concepts.--ma 20. elok. 2018 klo 19.59 Mikko Meronen <mikkovil...@gmail.com> kirjoitti:Hi and thanks for your help.I have tried postgress and now I have a problem using it. Here's my code following an error. I appreciate any help.def do():x = 1while x == 1:create_table()print('finding data')find_data()time.sleep(120)def create_table():c.execute('CREATE TABLE IF NOT EXISTS testi(url TEXT, title TEXT, published TEXT, UNIQUE(url))')def find_data():r = requests.get(homepage)soup = BeautifulSoup(r.text, 'html.parser')results = soup.find_all('data')for result in results:title = result.find('h1').texturl = result.find('a')['href']published = result.find('time')['datetime']c.execute("INSERT OR IGNORE INTO testi (url, title, published) VALUES (%s, %s, %s)",('url', 'title', 'published'))connection.commit()do()ERROR:Traceback (most recent call last):File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 43, in <module>do()File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 21, in dofind_data()File "C:\Users\Mikko\django-project\top\datas\dataTESTI.py", line 39, in find_data('url', 'title', 'published'))psycopg2.ProgrammingError: syntax error at or near "OR"LINE 1: INSERT OR IGNORE INTO testi (url, title, published) VALUES (...-Mikko--2018-08-17 19:09 GMT+03:00 Mikhailo Keda <mri...@gmail.com>:You could generate models from sqlite database - https://docs.djangoproject.com/en/2.1/howto/legacy-databases/--
Than switch to PostgresQL
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/52f5c754-7310-4baa-8008-edbcea15eafb%40googlegroups.com.
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAGD0jjJof7v9nMPF593DJiQHRv6z_%3DmQ_ZtQWob47PBT1gvL9Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAHn91oex8cAE1yFshSABt9QZpdiiRVtALtLx0AWDNU84w0%2Bqvw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAGD0jjKcf%2BMDDjoB83fnBsGiRD16vvHe-uLAcEW6XZhU31UDug%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/81f734c8-3140-4d1e-ae41-3e20bf59d9e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAK4qSCfpMr2EL5GQY83TyFb7x8z9BwDAEAhLE_rLX3uTs02oag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAGD0jjJv%2Bg%3DbTrXFXgDVjymz_MzeEknJhaDqnPVcu0kAQqZ%3DWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
No comments:
Post a Comment