Encryption client compatibility

[jonashaag]

I'm interested in contributing compatibility with AWS's Encryption Client encryption.

https://docs.aws.amazon.com/amazon-s3-encryption-client/latest/developerguide/what-is-s3-encryption-client.html

I wonder if that would be accepted in obstore and whether parts of it should be factored in a third-party extension/library.

I let Codex built a fully working and compatible implementation that's tested against the Java reference implementation; Git stats are below to give a rough idea of the amount of code needed.

xref apache/arrow-rs-object-store#647

 Cargo.toml                                                                   |    2 +-
 obstore/Cargo.toml                                                           |    3 +
 obstore/python/obstore/__init__.py                                           |   20 +
 obstore/python/obstore/_obstore.pyi                                          |   38 ++
 obstore/python/obstore/s3_encryption.py                                      |  366 ++++++++++++++++++
 obstore/src/lib.rs                                                           |    2 +
 obstore/src/s3_encryption.rs                                                 |  258 +++++++++++++
 s3-encryption-client-rs/Cargo.toml                                           |   34 ++
 s3-encryption-client-rs/src/algorithm.rs                                     |   73 ++++
 s3-encryption-client-rs/src/client.rs                                        |  470 +++++++++++++++++++++++
 s3-encryption-client-rs/src/error.rs                                         |   46 +++
 s3-encryption-client-rs/src/key_derivation.rs                                |  136 +++++++
 s3-encryption-client-rs/src/keyring.rs                                       |  470 +++++++++++++++++++++++
 s3-encryption-client-rs/src/lib.rs                                           |   22 ++
 s3-encryption-client-rs/src/metadata.rs                                      |  326 ++++++++++++++++
 s3-encryption-client-rs/src/object_store_wrapper.rs                          |  269 +++++++++++++
 s3-encryption-client-rs/tests/live_aws.rs                                    |  189 +++++++++
 tests/helpers/go_s3ec_bridge/main.go                                         |  136 +++++++
 tests/helpers/java_s3ec_bridge/cp.txt                                        |    1 +
 tests/helpers/java_s3ec_bridge/pom.xml                                       |   28 ++
 tests/helpers/java_s3ec_bridge/src/main/java/com/example/JavaS3EcBridge.java |  182 +++++++++
 tests/test_s3_encryption.py                                                  |   73 ++++
 tests/test_s3_encryption_e2e.py                                              |  183 +++++++++
 tests/test_s3_encryption_e2e_java.py                                         |  249 ++++++++++++
 tests/test_s3_encryption_ported_java.py                                      |  234 ++++++++++++
 26 files changed, 4886 insertions(+), 41 deletions(-)

[kylebarron]

Hmm that seems like a decent amount of code. What about a third party rust module? Then both obstore and any other Rust users of object_store could use your module? I know there had been discussion of a repository with object store utilities

[kylebarron]

I think if you wanted to implement this, the ideal way to handle it would be a Python callback that you could pass in to customize encryption, so we don't add code into obstore

[jonashaag]

If you'd like to keep this in Python, we could also just build a wrapper library on Python level and need no changes in obstore at all, I think?

[kylebarron]

I think it would be best as a separately-published package that reduces the maintenance overhead here. We can link to it from the documentation