As a user, I want deep archive to support LBLX label extensions

[tbarnes4]

Checked for duplicates

No - I haven't checked

🐛 Describe the bug

When I run the software on a bundle that has only lblx files, the software runs without errors, but produces empty SIP, checksum, and manifest *.tab files.

🕵️ Expected behavior

I expect the software to create the tables as if the bundle contained xml labels.

📜 To Reproduce

1. Run pds-deep-archive
2. Ex: pds-deep-archive -s PDS_SBN -b https://pdssbn.astro.umd.edu/holdings/ --include-latest-collection-only pds4-nh_documents-v1.0/bundle.lblx

...

🖥 Environment Info

• Version of this software 1.1.4
• Operating System: MacOSX 12.6.8 and Red Hat Enterprise Linux 9.3
  ...

📚 Version of Software Used

pds-deep-archive 1.1.4

🩺 Test Data / Additional context

https://pdssbn.astro.umd.edu/holdings/pds4-nh_documents-v1.0

🦄 Related requirements

🦄 #xyz

⚙️ Engineering Details

No response

[tbarnes4]

@jordanpadams Since I've seen no further comment on this, just pinging again.

[jordanpadams]

@tbarnes4 thanks for the ping. I updated this ticket to be a new requirement since this is an overall new requirement on the system that was not implemented yet.

That being said, we added this to the backlog, but it is not on the top of our radar since there is a workaround.

If you load this data into the Registry/API (which it looks like it is), then you can run pds-deep-registry-archive to generate the SIPs much easier (and faster), not to mention it is a good forcing factor to load the data into the registry.

$ pds-deep-registry-archive --site PDS_SBN urn:nasa:pds:nh_documents::1.0

[tbarnes4]

@jordanpadams Thanks! I was thinking of trying that work around as well. It would be the first we've done from SBN-UMD thru the registry, so that will be fun.

[jordanpadams]

@tbarnes4 actually... standby on running deep-archive. I forgot we have a bug right now. A PR is up and we will hopefully have a new version tagged by end of day (or first thing Monday).

[jordanpadams]

@tbarnes4 you should be able to upgrade your PDS Deep Archive software and run the utility as described above. I just tested it and it appears to work great.

[tbarnes4]

@jordanpadams Upgraded to v1.1.15 on two different systems (same as above) and I am still getting the same result with the same bundle (as above) where the 3 *tab files are all empty.

% source ~/.virtualenvs/pds-deep-archive/bin/activate.csh
% pds-deep-archive -s PDS_SBN -b https://pdssbn.astro.umd.edu/holdings/ pds4-nh_documents-v1.0/bundle.lblx
INFO 👟 PDS Deep Archive, version 1.1.5
INFO 🏃‍♀️ Starting AIP generation for pds4-nh_documents-v1.0/bundle.lblx
INFO 🎉 Success! AIP done, files generated:
INFO 📄 Checksum manifest: nh_documents_v1.0_20240227_checksum_manifest_v1.0.tab
INFO 📄 Transfer manifest: nh_documents_v1.0_20240227_transfer_manifest_v1.0.tab
INFO 📄 XML label for them both: nh_documents_v1.0_20240227_aip_v1.0.xml
INFO 🏃‍♀️ Starting SIP generation for pds4-nh_documents-v1.0/bundle.lblx
INFO 🎉 Success! From /Volumes/echeclus02/TEMP/nssdc_test/pds4-nh_documents-v1.0/bundle.lblx, generated these output files:
INFO 📄 SIP Manifest: nh_documents_v1.0_20240227_sip_v1.0.tab
INFO 📄 XML label for the SIP: nh_documents_v1.0_20240227_sip_v1.0.xml
INFO 👋 That's it for now. Bye.

[jordanpadams]

@tbarnes4 sorry for the confusion, I meant you are good to use the pds-deep-registry-archive utility to generate the SIPs using the next-gen Registry. This requirement has still not been implemented, and will unfortunately not be implemented for a bit since we have a known workaround.

[tbarnes4]

@jordanpadams That makes much more sense. Tracking down other issues and I felt I was going crazy. I expect to try the registry-archive later.

[jordanpadams]

@tbarnes4 copy that. it should take ~30 seconds to try it :-) . and you can run it from anywhere (your own laptop works).

[tbarnes4]

@jordanpadams That it does, even less. For kicks and giggles tried a collection LIDVID and it errored out with unexpected errors. But can't expect it to do things it wasn't designed to do. ;-)

[smclaughlin7]

Subscribed to monitor progress. @tbarnes4 Do you expect to submit this bundle shortly after deep archive is enhanced?

[smclaughlin7]

@jordanpadams Should NSSDCA expect Deep Archive tool continue to use '.xml' as the extension for SIP products, at least until IM 2.0? I assume '.lblx' is not presently an option for that tool, correct? Thanks!

[jordanpadams]

@smclaughlin7 correct. And I am not even sure we will be upgrading with IM 2.0 since all of these SIPs belong to the same collection. We don't plan on going back and reprocessing those SIPs to use lblx.

[smclaughlin7]

@jordanpadams Thanks! We/NSSDCA will continue to expect only '.xml' for SIPs.

[tbarnes4]

Subscribed to monitor progress. @tbarnes4 Do you expect to submit this bundle shortly after deep archive is enhanced?

@smclaughlin7 Sorry for the delay. I have just submitted the two bundles. Hopefully they go thru without problem. Thanks!

[tbarnes4]

@jordanpadams I think this needs to be addressed sooner. I have been sitting on data sets in hopes to register them and then using the pds-deep-registry-archive, but it has been months since we have been able to register new data. I have submitted my non-lblx data, but I am unable to submit my data sets that utilize lblx label file extensions. Is there a quick fix? It is not good for us to sit on ready to deep archive data. Thanks.

[jordanpadams]

@tbarnes4 apologies for the delay, but we have no available staff to work on this at this time, as we have been focused on getting the Registry up and running, which was completed last week. Pat should be receiving info to again begin loading data into the registry. Sorry for the inconvenience.