snakemake pipeline #34
Merged
+1,284
−63
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 task
Closed
6 tasks
test to ensure sensitive things do not get uploaded
because cron gives us a clean environment. Create FTPS sentinel file.
Fix docformatter errors in place, and stop ruff and docformatter from fighting with each other.
jeremyestein
force-pushed
the
jeremy/pipeline
branch
from
4aef42c to
e8d2f24
Compare
jeremyestein
force-pushed
the
jeremy/pipeline
branch
from
c387fef to
4fe3e95
Compare
jeremyestein
force-pushed
the
jeremy/pipeline
branch
from
48c945a to
5424a3e
Compare
1 task
skeating
approved these changes
Jan 27, 2026
Collaborator
skeating
left a comment
skeating
left a comment
There was a problem hiding this comment.
I couldn't get it running locally and I certainly couldn't find a 'tools' directory as given in it develop.md file
I think this could be filed as an issue to revisit if we have time; so not a blocker
Collaborator
Author
Interesting, @thompson318 is that your script and did you intend to commit it to this repo? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements #29 and fixes #26.
Snakemake is run according to a cron specification (default is once a day in the early hours of the morning).
It causes the CSVs output by the waveform-controller to be converted to both kinds of parquet, and then the de-id parquet to be uploaded to the DSH, leaving behind marker files with upload stats so that snakemake knows they are done, and so do the humans!
Toy hasher needed a fix because Python's
hashmethod is not stable from run to run. (Switch to using real hasher is in #35)Also add a pipeline debugging guide.