feat[python]: support full read/write from object storage #6022
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds the ability to write in PyVortex directly to object storage, along with more ergonomic configuration of object store credentials for the major PyVortex read/write paths:
vx.open,vx.io.read, andvx.io.write.This is using the https://docs.rs/pyo3-object_store/latest/pyo3_object_store/ crate, which is what runs the obstore Python library.
Notably, we can't reuse
obstoreand all of its types directly. This is b/c of the way it works, you're required to actually link in pyo3-object_store and then expose it as a new module within your Python bundle.There are some other examples of this pattern:
Sadly there's no nicely distributed types package, so I need to copy all of the pyi files here with their original MIT license.
I've added a Python unit test to demonstrate using the new object store builders with simple local file system object store
Resolves #5673