Target refactor in anchoredMC; Improvements for early file removal#2140
Conversation
Changes the way we construct targets in anchorMC: Instead of calling different workflows in order (AOD, TPC-time-series, QC), we now execute one single workflow with all necessary targets in one go. This has the advantage that we can apply "early-file-removal" more easily and should also lead to CPU efficiency improvements. Some other improvements in anchoredMC: - allow to create just the workflow.json file - allow to inject additional options for the workflow runner The changes in o2dpg_workflow_runner.py concern mostly improvements for the early-artefact removal. E.g.: - do not remove artefacts when they belong to a final target task - add global disc space monitoring to measure benefit of early-file removal
|
REQUEST FOR PRODUCTION RELEASES: This will add The following labels are available |
|
REQUEST FOR PRODUCTION RELEASES: This will add The following labels are available |
1 participant
Changes the way we construct targets in anchorMC: Instead of calling different workflows in order (AOD, TPC-time-series, QC), we now execute one single workflow with all necessary targets in one go. This has the advantage that we can apply "early-file-removal" more easily and should also lead to CPU efficiency improvements.
Some other improvements in anchoredMC:
The changes in o2dpg_workflow_runner.py concern mostly improvements for the early-artefact removal. E.g.: