Change migration 19 to reset metadata_sha256 to null #1073
+7
−186
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jobselko
approved these changes
Jan 21, 2026
jobselko
reviewed
Jan 21, 2026
pulp_python/app/migrations/0019_create_missing_metadata_artifacts.py
Outdated
Show resolved
Hide resolved
jobselko
reviewed
Jan 21, 2026
pulp_python/app/migrations/0019_create_missing_metadata_artifacts.py
Outdated
Show resolved
Hide resolved
Closed
gerrod3
force-pushed
the
19-fix
branch
3 times, most recently
from
59be7f9 to
dc4a082
Compare
jobselko
reviewed
Jan 21, 2026
| .exclude(metadata_sha256="") | ||
| .prefetch_related("_artifacts") | ||
| .only("filename", "metadata_sha256") | ||
| .only("sha256", "filename", "metadata_sha256", "pulp_domain_id", "pulp_type") |
Contributor
Author
There was a problem hiding this comment.
Doing some testing of the generated SQL I found it is since instantiating the package model requires pulp_type else we would get another query: https://github.com/pulp/pulpcore/blob/main/pulpcore/app/models/base.py#L163
gerrod3
changed the title
gerrod3
changed the title
Contributor
Author
|
Ok, looks like the data migration plan isn't going to work. New plan: have the migration just set the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ok, the last fix was correct for duplicate artifacts across domains, but it didn't solve for duplicate metadata artifacts within a domain. At first this seems impossible, but there is a common scenario where this can occurs. A user uploads package a.1.whl with metadata xyz. They realize the package is missing some files and rebuild with the new files, exact same name and crucially the exact same metadata. They reupload and pulp creates a new package since the entire package has a new sha256 even though the metadata is the same as the old one. Then in our migration we will encounter two "different" packages with the same metadata artifact inside them.
My changes try to fix this by keeping track of the metadata artifacts shad256s and avoiding making duplicates. Since we do the saves in batches I have to do a check first within the batch to make sure there are no dups and then do a second check to make sure there are no dups from previous batches. Also, I'm grouping the packages by domain, so all the batches should be inside the same domain.
Hopefully I didn't screw up the logic anywhere.
fixes: #1071