Fix pandas 3.0 compatibility #561
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Update deprecated frequency strings to lowercase (H→h, S→s) - Add explicit compat and join parameters to xarray merge to avoid FutureWarnings - Apply ruff formatting - Add Git workflow guidelines to CLAUDE.md Changes: - src/modelskill/comparison/_comparison.py: Add explicit compat="no_conflicts" and join="outer" to merge() - src/modelskill/model/dummy.py: Change freq="H" to freq="h" in docstring example - tests/test_simple_compare.py: Change freq="4S" to freq="4s" - CLAUDE.md: Add Git Workflow section with reminder to never commit directly to main - Format code with ruff All tests pass (589 passed, 5 skipped) with only expected UserWarnings remaining.
Pandas 3.0 changes default datetime precision from datetime64[ns] to
datetime64[us], which causes xarray.interp() to fail when source and
target datasets have mismatched precision. Additionally, pd.cut() NaN
handling and StringDtype compatibility changed.
Changes:
- Normalize datetime coordinates to datetime64[ns] for backward
compatibility with pandas 2.x in _convert_to_dataset() and
_parse_dataset()
- Fix pd.cut() NaN handling: check pd.notna(x) before accessing .mid
- Replace np.issubdtype() with pd.api.types.is_numeric_dtype() for
pandas 3.0 StringDtype compatibility
- Fix peak_ratio metric: use .total_seconds() instead of hardcoded
nanosecond division to work with any datetime precision
- Replace peak_ratio test with synthetic data test that has verifiable
peaks (ratio=1.1)
- Improve code clarity: iterate over self.data instead of matched_data,
use .get("kind") to handle auxiliary variables safely
- Remove unused imports in test files (ruff auto-fix)
ecomodeller
changed the title
Test both pandas versions on every PR to ensure compatibility with both versions until pandas 2.x support is dropped.
ecomodeller
commented
Feb 2, 2026
| self.raw_mod_data = { | ||
| str(key): PointModelResult(self.data[[str(key)]], name=str(key)) | ||
| for key, value in matched_data.data_vars.items() | ||
| for key, value in self.data.data_vars.items() |
Member
Author
There was a problem hiding this comment.
This is the key change, using self.data already validated and fixed, instead of matched_data.
Reverted changes that were not related to Pandas 3 compatibility: - CLAUDE.md git workflow docs - _comparer_plotter.py formatting - _comparison.py raw_mod_data refactoring and merge params - Test file formatting and unused import cleanup These changes are tracked in issues #568 and #569 for separate PRs.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix pandas 3.0 compatibility issues
Critical fixes:
datetime64[ns]at data loading to prevent xarray interpolation failures caused by pandas 3.0's default datetime resolution changenp.issubdtype()withpd.api.types.is_numeric_dtype()for pandas 3.0 StringDtype compatibilitypeak_ratiometric to use.total_seconds()instead of hardcoded nanosecond divisionpd.notna()checks before accessing.midin pd.cut() operations.get("kind")instead of["kind"]to handle auxiliary variables safelyAdditional improvements:
Maintains backward compatibility with pandas 2.x while supporting pandas 3.0.