[Feature] update existing parts #5

si-hb · 2024-01-08T19:51:56Z

First off, hats off to the brilliant effort here - thank you!

I am new to Inventree, needed to inventory years of old stock which has now been added (by way of some python btw, I couldn't get the Inventree csv import tools to work for the life of me). Alas, it is all void of datasheet links, parameters, images et al.

Maybe this already exists and I'm not aware of it, but it sure would be handy to have a tool I could run against my existing Inventree DB of bare naked parts, and reach out to the clouds for all the dressings...

for example: $ inventree-part-import update Electronics/Passives would update that subcategory.

I tried to export my parts to a csv file, which I then tried $ inventree-part-import parts.csv. But it only got past a coupla parts before it would trigger #4 and/or #3.

Thanks again.

The text was updated successfully, but these errors were encountered:

30350n · 2024-01-09T01:51:47Z

This seems useful. Quickly implemented it, you can update categories via --update CATEGORY_PATH and --update-recursive CATEGORY_PATH now!

Edit:

First off, hats off to the brilliant effort here - thank you!

Thanks btw!

si-hb · 2024-01-09T21:43:46Z

Brilliant!!!!!!!!!!!!!!!!!! Works like a charm.

It does kick out on a certain part however. Is this an issue with my DB? If so if you can catch/skip the error and let it continue, would be wicked, then I can deal with that specific part on my own.

requests.exceptions.HTTPError: 

{
  "detail": "Error occurred during API request",
  "url": "<my api...>/company/part/1004/",
  "method": "PATCH",
  "status_code": 400,
  "body": "{\"manufacturer_part\":[\"Linked manufacturer part must reference the same base part\"]}",
  "headers": {
    "AUTHORIZATION": "Token <token>"
  },
  "params": {
    "format": "json"
  },
  "data": {
    "manufacturer_part": 27,
    "description": "TRIAC 600V 4A TO220F",
    "link": "https://www.digikey.ca/en/products/detail/ween-semiconductors/BT136X-600-127/1155132",
    "packaging": "Tube",
    "available": 1126
  }
}

For anyone else reading this, $ pipx run --spec git+https://github.com/30350n/inventree_part_import.git inventree-part-import --update-recursive <category> allowed me to run this patch.

30350n · 2024-01-09T21:45:30Z

It does kick out on a certain part however. Is this an issue with my DB? If so if you can catch/skip the error and let it continue, would be wicked, then I can deal with that specific part on my own.

Do you have a full traceback?

si-hb · 2024-01-09T21:46:40Z

Yes...

Traceback (most recent call last):
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/bin/inventree-part-import", line 8, in <module>
    sys.exit(inventree_part_import())
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/cli.py", line 38, in wrapper
    raise e
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/cli.py", line 22, in wrapper
    func(*args, **kwargs)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/cli.py", line 199, in inventree_part_import
    last_import_result = importer.import_part(part, supplier, only_supplier)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/part_importer.py", line 77, in import_part
    import_result |= self.import_supplier_part(supplier, api_part)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/part_importer.py", line 172, in import_supplier_part
    update_object_data(
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/inventree_helpers.py", line 77, in update_object_data
    obj.save(data)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree/base.py", line 242, in save
    response = self._api.patch(self._url, data, files=files)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree/api.py", line 457, in patch
    response = self.request(
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree_part_import/retries.py", line 68, in request
    return super().request(api_url, **kwargs)
  File "/home/hb/.local/pipx/.cache/a03185e0786c024/lib/python3.10/site-packages/inventree/api.py", line 371, in request
    raise requests.exceptions.HTTPError(detail)
    ```

30350n · 2024-01-09T21:46:58Z

For anyone else reading this, $ pipx run --spec git+https://github.com/30350n/inventree_part_import.git inventree-part-import --update-recursive <category> allowed me to run this patch.

You can also just install via pipx install git+https://github.com/30350n/inventree_part_import.git, although you might need to uninstall the pypi version first.

30350n · 2024-01-09T22:23:42Z

Try again (with the master branch). I'm handling all HTTPErrors now, so this doesn't fix the actual error, but it shouldn't crash anymore.

30350n · 2024-01-09T22:28:27Z

Actually I think I also know what the actual problem is. The updating I implemented is somewhat stupid. It just dumps all the part names from the category and imports them. This doesn't ensure the parts actually get updated though. In some cases it might keep the old parts and create new duplicate ones. So maybe wait a bit more until updating everything, I should be able to fix that fairly easily.

si-hb · 2024-01-09T22:50:48Z

OK, some more insight, an error from my DB:

2024-01-09 20:19:51.192 UTC [84594] ERROR: insert or update on table "part_supplierpart" violates foreign key constraint "part_supplierpart_part_id_d20a9b93_fk_part_part_id"

ChatGPT's take on the error:

Non-Existent Reference: You are trying to insert or update a row in part_supplierpart with a part_id that does not exist in the part table. The foreign key constraint ensures that every part_id in part_supplierpart must correspond to an existing part_id in the part table.

Deleting or Altering a Referenced Row: If you're performing an update, it's possible that you're trying to alter or delete a part_id in the part table that is still being referenced by a row in part_supplierpart.

To resolve this error, you will need to:

Ensure that any part_id you're trying to insert into or update in the part_supplierpart table exists in the part table.
If you're updating the part table, make sure you're not altering or deleting any part_id values that are still being referenced in the part_supplierpart table.

30350n · 2024-01-09T22:53:14Z

ChatGPT's take on the error: <bunch of gibberish>

Sorry, but that's really not helpful.

si-hb · 2024-01-09T23:01:57Z

ChatGPT's take on the error:

Sorry, but that's really not helpful.

No worries, fortunately it handles critique rather unflinchingly lol.

30350n · 2024-01-09T23:25:12Z

No worries, fortunately it handles critique rather unflinchingly lol.

I mean 'ChatGPT'/"""AI""" in general 🙂

Okay, with 5e89920 now, it should actually update the existing parts and that specific error you got should also not happen anymore 🤞

si-hb · 2024-01-10T00:01:13Z

Closer! It cut out at this part with:

searching for 1803277 ...
successfully added Mouser part 651-1803277 (https://<link>/supplier-part/1028/)
found existing DigiKey part 277-1206-ND ...
error: failed to import part with: 
    link: ['Ensure this field has no more than 200 characters.']


the following parts failed to import:
100-44-316-41-001000
100-93-308-41-001000
1803277

the following parts are incomplete:
103
1803280
1803578
etc...

30350n · 2024-01-10T00:07:22Z

Can you run that part again with -v to get the full traceback? (I'm mainly curious why that digikey link is >200 chars long).

30350n · 2024-01-10T00:09:07Z

Btw, it's generally a good idea to use DigiKey as the first supplier to check. You can reorder the suppliers in the suppliers.yaml in your config directory.

si-hb · 2024-01-10T00:15:35Z

Can you run that part again with -v to get the full traceback? (I'm mainly curious why that digikey link is >200 chars long).

209 characters long to be precise...

https://www.phoenixcontact.com/product/pdf/MTgwMzI3Nw?_realm=us&_locale=en-US&blocks=commercial-data%2Ctechnical-data%2Cdrawings%2Capprovals%2Cclassifications%2Cenvironmental-compliance-data%2Call-accessories

si-hb · 2024-01-10T00:16:28Z

Btw, it's generally a good idea to use DigiKey as the first supplier to check. You can reorder the suppliers in the suppliers.yaml in your config directory.

Good to know! Thnx

30350n · 2024-01-10T00:17:06Z

209 characters long to be precise...

and dead LMAO.

si-hb · 2024-01-10T00:19:58Z

209 characters long to be precise...

and dead LMAO.

LOL!, hmm... possible to skip adding link if it's dead?

si-hb · 2024-01-10T00:20:16Z

Can you run that part again with -v to get the full traceback? (I'm mainly curious why that digikey link is >200 chars long).

searching for 1803277 ...
searching at Mouser ...
found existing Mouser part 651-1803277 ...
successfully updated Mouser part 651-1803277 (https://<url>/supplier-part/1028/)
searching at DigiKey ...
found existing DigiKey part 277-1206-ND ...
error: failed to import part with: 
    link: ['Ensure this field has no more than 200 characters.']

FULL TRACEBACK:
Traceback (most recent call last):
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/part_importer.py", line 87, in import_part
    import_result |= self.import_supplier_part(supplier, api_part, existing_part)
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/part_importer.py", line 183, in import_supplier_part
    part.addLinkAttachment(datasheet_url_safe, comment="datasheet")
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/base.py", line 501, in addLinkAttachment
    return AttachmentSubClass.add_link(
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/base.py", line 401, in add_link
    if response := api.post(cls.URL, data):
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/api.py", line 419, in post
    response = self.request(
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/retries.py", line 68, in request
    return super().request(api_url, **kwargs)
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/api.py", line 371, in request
    raise requests.exceptions.HTTPError(detail)
requests.exceptions.HTTPError: {'detail': 'Error occurred during API request', 'url': 'https://<url>/api/part/attachment/', 'method': 'POST', 'status_code': 400, 'body': '{"link":["Ensure this field has no more than 200 characters."]}', 'headers': {'AUTHORIZATION': '<token>'}, 'params': {'format': 'json'}, 'data': {'part': 1123, 'comment': 'datasheet', 'link': 'https://www.phoenixcontact.com/product/pdf/MTgwMzI3Nw%3F_realm%3Dus%26_locale%3Den-US%26blocks%3Dcommercial-data%252Ctechnical-data%252Cdrawings%252Capprovals%252Cclassifications%252Cenvironmental-compliance-data%252Call-accessories'}}


the following parts failed to import:
1803277

30350n · 2024-01-10T00:20:49Z

It's not dead dead, the page loads. Just limited the link to 200 chars now, should be fixed.

si-hb · 2024-01-10T00:27:52Z

It's not dead dead, the page loads. Just limited the link to 200 chars now, should be fixed.

still getting same error for part# 1803277

$ inventree-part-import -v 1803277

30350n · 2024-01-10T00:29:23Z

Ahhhh, it's a datasheet link. Gimme a sec.

30350n · 2024-01-10T00:31:23Z

Okay, now it should be fixed.

si-hb · 2024-01-10T00:36:49Z

Okay, now it should be fixed.

Running --update, now past 1803277 and onwards... stay tuned.

side note, when it says warning: found 3 parts at Mouser, skipping import, what's the reasoning behind say not selecting just the first part? Perhaps could be a config setting..?

30350n · 2024-01-10T00:38:59Z

side note, when it says warning: found 3 parts at Mouser, skipping import, what's the reasoning behind say not selecting just the first part? Perhaps could be a config setting..?

If you are running with --interactive twice (which is default) it'll ask you what part you want to import after it's done going through all parts once.

If you run with --interactive true it'll ask you right away.

Just picking the first part is not really a good idea in many cases.

si-hb · 2024-01-10T00:53:54Z

made it through about 100 parts, and stopped on this part:

$ inventree-part-import -v PBC36DFAN

searching for PBC36DFAN ...
searching at Mouser ...
(no results at Mouser)
searching at DigiKey ...
found existing DigiKey part S2211E-36-ND ...
updating DigiKey part ...
error: failed to import part with: 
    manufacturer_part: ['Linked manufacturer part must reference the same base part']

FULL TRACEBACK:
Traceback (most recent call last):
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/part_importer.py", line 87, in import_part
    import_result |= self.import_supplier_part(supplier, api_part, existing_part)
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/part_importer.py", line 198, in import_supplier_part
    update_object_data(
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/inventree_helpers.py", line 77, in update_object_data
    obj.save(data)
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/base.py", line 242, in save
    response = self._api.patch(self._url, data, files=files)
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/api.py", line 457, in patch
    response = self.request(
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree_part_import/retries.py", line 68, in request
    return super().request(api_url, **kwargs)
  File "/home/hb/.local/pipx/venvs/inventree-part-import/lib/python3.10/site-packages/inventree/api.py", line 371, in request
    raise requests.exceptions.HTTPError(detail)
requests.exceptions.HTTPError: {'detail': 'Error occurred during API request', 'url': 'https://<url>/api/company/part/902/', 'method': 'PATCH', 'status_code': 400, 'body': '{"manufacturer_part":["Linked manufacturer part must reference the same base part"]}', 'headers': {'AUTHORIZATION': '<token>'}, 'params': {'format': 'json'}, 'data': {'manufacturer_part': 110, 'description': 'CONN HEADER VERT 72POS 2.54MM', 'link': 'https://www.digikey.ca/en/products/detail/sullins-connector-solutions/PBC36DFAN/861346', 'packaging': 'Bulk', 'available': 1510.0}}


the following parts failed to import:
PBC36DFAN

30350n · 2024-01-10T01:02:53Z

Thought I'd have already fixed that one. Guess I missed another edge case, should actually be fixed now 🤞

si-hb · 2024-01-10T01:10:01Z

Thought I'd have already fixed that one. Guess I missed another edge case, should actually be fixed now 🤞

Same error.

30350n · 2024-01-10T01:20:42Z

Okay, going to need some more information here probably:

What's the PK of the PBC36DFAN Part? (should be part of the URL)
Is that exact same part set as the base part on the S2211E-36-ND SupplierPart?
There should be a PBC36DFAN ManufacturerPart aswell. Is the same part also set as it's base part?

30350n · 2024-01-10T01:23:49Z

Actually nvm, I might have got it, gimme a sec again ^^

si-hb · 2024-01-10T01:27:14Z

Actually nvm, I might have got it, gimme a sec again ^^

ok, there's actually three, 1095,1096,1097. Same manufacturer part, different part variations though and different Digikey parts because it's a pin header.

si-hb · 2024-01-10T01:29:26Z

I can give you a token and url if you want.

30350n · 2024-01-10T01:30:35Z

Ah I see, do you actually need all 3? 🤔

30350n · 2024-01-10T01:32:51Z

I can give you a token and url if you want.

Thanks for the offer, let's try it one more time now first, if it doesn't work now, something very weird is going on 😅

si-hb · 2024-01-10T01:33:58Z

Ah I see, do you actually need all 3? 🤔

Hmmm. I suppose not, now that you mention it. However, the different variations are ideal.

si-hb · 2024-01-10T01:37:06Z

I can give you a token and url if you want.

Thanks for the offer, let's try it one more time now first, if it doesn't work now, something very weird is going on 😅

Worked!! I'll run it against the whole DB again...

30350n · 2024-01-10T01:39:35Z

Perfect! Thanks for helping to work through all of this btw, your database seems to be a great "integration testsuite" ^^

si-hb · 2024-01-10T01:44:33Z

Perfect! Thanks for helping to work through all of this btw, your database seems to be a great "integration testsuite" ^^

You are most welcome. I can't tell you how much I appreciate your work and effort and responsiveness! Happy to test anything else you might need.

Yes my great "integration testsuite" is what you get when you give a noob a youtube video and access to a computer...

si-hb · 2024-01-10T01:56:37Z

Successfully made it through all parts in my DB 🥇

30350n · 2024-01-10T01:59:35Z

Wohoo! Great, I guess I can close this then ^^

Thanks again and if you run into any more issues, be sure to let me know!

30350n added the enhancement New feature or request label Jan 9, 2024

30350n self-assigned this Jan 9, 2024

30350n added a commit that referenced this issue Jan 9, 2024

Add --update, --update-recursive arguments (#5)

3ee5cbb

30350n added the bug Something isn't working label Jan 10, 2024

30350n closed this as completed Jan 10, 2024

[Feature] update existing parts #5

[Feature] update existing parts #5

Comments

si-hb commented Jan 8, 2024

30350n commented Jan 9, 2024 • edited Loading

si-hb commented Jan 9, 2024

30350n commented Jan 9, 2024

si-hb commented Jan 9, 2024

30350n commented Jan 9, 2024

30350n commented Jan 9, 2024 • edited Loading

30350n commented Jan 9, 2024 • edited Loading

si-hb commented Jan 9, 2024

30350n commented Jan 9, 2024

si-hb commented Jan 9, 2024

30350n commented Jan 9, 2024

si-hb commented Jan 10, 2024 • edited Loading

30350n commented Jan 10, 2024

30350n commented Jan 10, 2024

si-hb commented Jan 10, 2024

si-hb commented Jan 10, 2024

30350n commented Jan 10, 2024

si-hb commented Jan 10, 2024

si-hb commented Jan 10, 2024 • edited Loading

30350n commented Jan 10, 2024 • edited Loading

si-hb commented Jan 10, 2024 • edited Loading

30350n commented Jan 10, 2024

30350n commented Jan 10, 2024

si-hb commented Jan 10, 2024

30350n commented Jan 10, 2024 • edited Loading

si-hb commented Jan 10, 2024 • edited Loading

30350n commented Jan 10, 2024

si-hb commented Jan 10, 2024

30350n commented Jan 10, 2024

30350n commented Jan 10, 2024

si-hb commented Jan 10, 2024

si-hb commented Jan 10, 2024

30350n commented Jan 10, 2024

30350n commented Jan 10, 2024

si-hb commented Jan 10, 2024

si-hb commented Jan 10, 2024

30350n commented Jan 10, 2024

si-hb commented Jan 10, 2024

si-hb commented Jan 10, 2024

30350n commented Jan 10, 2024 • edited Loading

30350n commented Jan 9, 2024 •

edited

Loading

30350n commented Jan 9, 2024 •

edited

Loading

30350n commented Jan 9, 2024 •

edited

Loading

si-hb commented Jan 10, 2024 •

edited

Loading

si-hb commented Jan 10, 2024 •

edited

Loading

30350n commented Jan 10, 2024 •

edited

Loading

si-hb commented Jan 10, 2024 •

edited

Loading

30350n commented Jan 10, 2024 •

edited

Loading

si-hb commented Jan 10, 2024 •

edited

Loading

30350n commented Jan 10, 2024 •

edited

Loading