Minimal working example of safetensors support for hezar

[Adversarian]

Pull Request

Description

The save and load methods for the hezar.models.Model class have been altered to accommodate for safetensors support.

Changes

• Loading safetensors exposed through the keyword argument load_safetensors under Model.load().
• Saving safetensors exposed through the keyword argument safe_serialization under Model.save().
• SAFETENSORS added to the enumlist of Backends in constants.py.
• A minimal working example unittest is available at test_safetensors.py. This serves just to demonstrate the changes and is by no means a comprehensive test ready for production.

Related Issues

Resolves #153

Additional Comments

Please keep in mind that this is meant to serve as a draft PR and is by no means production-ready. The changes made to the code base are very crude and the aim was only to show how it may be possible to incorporate safetensors support in hezar. After the architectural details and design decisions regarding how and where this change should be introduced within the codebase are approved, I can happily edit this PR (or submit a new one) with a cleaner code that adheres to this library's standards.

Notes

• Model.push_to_hub() also requires changes but for the purposes of this quick prototype it was left unchanged for the time being.

[arxyzan]

Great job @Adversarian, this is so cool! I just took a quick look at it and it seems pretty solid. I'll try to test it out very soon in a day or two and let you know.

[arxyzan]

@Adversarian Can you please rename or re-apply your changes in another branch instead of main ? (To avoid conflicts and easier checkout in dev tools)

[Adversarian]

@arxyzan Of course, it's no problem. Should I close this PR and re-open another one from the new branch? Because I'm not sure how to edit this one so that the changes are pulled from the new branch.

[arxyzan]

@Adversarian Actually the correct way to do it is to create a branch from the main branch of your fork and create a PR to merge that branch to the main branch of Hezar.

[Adversarian]

@arxyzan I know, that's what I meant by what I said. The reason I didn't do that was because I was just prototyping so I didn't really bother with git etiquette. I'm going to close this and re-open another PR from a new branch after syncing with hezar (since it's 1 commit ahead of mine since yesterday).

[Adversarian]

@arxyzan Reopened this at PR #157 with a different branch name. Let's continue the discussion from there.