Fix up L1C setting of background rates and bg_sys_err

[subagonsouth]

Change Summary

Overview

• For now assume that backgrounds ancillary product must not change mid-pointing when identifying bg rows to apply.
• Modify csv definition so that bin_end is and inclusive index rather than exclusive.

[Copilot]

Pull request overview

This PR fixes the handling of background rates and systematic errors in L1C processing by standardizing the bin_end indexing convention and adjusting the background data filtering logic.

Changes:

• Modified CSV background data format to use inclusive bin_end indexing (bin 0 is now represented as bin_start=0, bin_end=0 instead of bin_start=0, bin_end=1)
• Updated background filtering to require that background data fully encompasses the pointing period (more conservative approach)
• Adjusted test data timing to align with the new filtering logic

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
imap_processing/lo/l1c/lo_l1c.py	Updated background filtering logic to use <= and >= operators; modified bin_end calculation to add 1 before scaling to handle inclusive indexing; added TODO comment about mid-pointing assumption; contains debug print statements that should be removed
imap_processing/tests/lo/test_lo_l1c.py	Adjusted test pointing times to match the background data time range (start +100s, end -100s)
imap_processing/tests/lo/test_anc/imap_lo_oxygen-background-small_20250101_20270101_v001.csv	Changed bin_end from exclusive to inclusive indexing for all 240 data rows
imap_processing/tests/lo/test_anc/imap_lo_hydrogen-background-small_20250101_20270101_v001.csv	Changed bin_end from exclusive to inclusive indexing for all 240 data rows

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[ahotasu]

I approve (can't figure out how to self-assign as a reviewer). However, I will note that the met end times in the ancillary background file aren't large enough to cover the testing we're doing for the data on 2026/01/27. I have tested this using a modified backgrounds file with an end met one (second?) higher than what the algorithm is looking for for 2026/01/27. This successfully loads the backgrounds and applies them in the pset and L2 CG corrected fluxes (per a PR I'll put in momentarily).

[greglucas]

Do you want to change this to GoodTime start here? My thought is that right now you're requiring the full pointing period to be covered, so if you choose just the leading edge the we can just say we will use anything straddling that time period. Alternatively swap them and say (goodtimestart < pointing end AND goodtimeend > pointing start) which should cover the joint cases a little more robustly I think?

[subagonsouth]

I'm not sure I am following this. I think that my hesitation with the suggestion to do:

(background_df["GoodTime_start"] < pointing_end_met)
& (background_df["GoodTime_end"] > pointing_start_met)

would be that it would make it likely that multiple sets of backgrounds would get selected. For example if pointing start/end times are every 10: [10, 20, 30, 40] and you want different backgrounds for each. If you aren't super precise with your start/end times, the goodtimes start/ends might look like:

[[9, 21], [19, 31], [29, 41]]

But then the logic would match multiple goodtime entries to each pointing.

[greglucas]

Yeah, I agree with you. This whole way of doing it is brittle IMO. We should be doing it per pointing number or something like that if it has to be constant per pset and not dealing with SHCOARSE at all.

[subagonsouth]

@ahotasu

However, I will note that the met end times in the ancillary background file aren't large enough to cover the testing we're doing for the data on 2026/01/27. I have tested this using a modified backgrounds file with an end met one (second?) higher than what the algorithm is looking for for 2026/01/27. This successfully loads the backgrounds and applies them in the pset and L2 CG corrected fluxes (per a PR I'll put in momentarily).

The files in this PR are just for testing. In production, the ancillary files in the SDC are used.