-
Notifications
You must be signed in to change notification settings - Fork 616
Allow s3Location as Document, Image, and Video #1572
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
2e47315 to
e17e8bb
Compare
9c2aca3 to
a4874cb
Compare
a4874cb to
b4a8f0a
Compare
b4a8f0a to
97f435d
Compare
97f435d to
b34f588
Compare
b34f588 to
27d9463
Compare
| """ | ||
|
|
||
| bytes: bytes | ||
| s3Location: S3Location |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where did we land on the location vs s3Location discussion @mkmeral ?
Do we think something like the following is a poor abstraction?
bytes: bytes
location: Location
class Location(TypedDict):
uri: Required[str]
class S3Location(Location):
bucketOwner: str
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I made a note in the description of this pr:
One decision I made in this pr is the choice between a generic Location source, vs the more specific S3Location source. Some model providers support url for a document location (ref), so it could make sense to make this generic for s3 as well. But since S3 also has an optional bucketOwner, I thought it would make more sense to have a separate S3Location, and then later add something like a URLLocation to cover the openai case.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahh, I see what you mean. Im not too strongly opinionated about that. I think a separate key is a bit more explicit, but I dont really mind either way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we think something like the following is a poor abstraction?
Yeah I was also thinking of that abstraction. My question right now is, will we get non-S3 URIs today?
If so, we should implement that. Otherwise, I'd be in favor of moving to that structure once we have other Locations (so the abstraction actually means sth, and not over-engineered)
| # Handle source - supports bytes or s3Location | ||
| if "source" in document: | ||
| result["source"] = {"bytes": document["source"]["bytes"]} | ||
| source = document["source"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We seem to have so much code duplication here. We write this logic 3 times. Can we improve it?
27d9463 to
1b60ecc
Compare
Description
Adds native S3 location support for media content types in Bedrock, allowing users to reference images, documents, and videos stored in S3 directly rather than requiring base64 encoding.
One decision I made in this pr is the choice between a generic
Locationsource, vs the more specificS3Locationsource. Some model providers support url for a document location (ref), so it could make sense to make this generic for s3 as well. But since S3 also has an optionalbucketOwner, I thought it would make more sense to have a separateS3Location, and then later add something like aURLLocationto cover the openai case.What Changed
Type System Updates
ImageSource,DocumentSource, andVideoSourceintypes/media.pyto accepts3Locationas an alternative tobytesS3LocationTypedDict with requiredurifield and optionalbucketOwnerfor cross-account accessProvider Behavior
Updated Integ Tests
I added integ tests which create an s3 bucket (post-fixed with the accountId) where the local file resources are uploaded for testing. Then they are referenced in the tests to ensure the model can recognize and identify the content in them.
Ive also added a short 5 second video resource to the integ tests to cover video modalities
Usage Example
Notes on backwards compatibility:
This change updates
ImageSource,DocumentSource, andVideoSourceto usetotal=Falsein their typed dict definition, ultimately making thebytesparameter optional now when it was required before. This is considered safe since all existing code should not be impacted by this. However, going forward, since these fields are essentially union types now, they need to be treated as such by customers going forward.Related Issues
#1482
Documentation PR
TODO
Type of Change
New feature
Testing
How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli
hatch run prepareChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.