Skip to content

Fix numpy and pandas compatibility#475

Merged
martin-springer merged 60 commits intodevelopmentfrom
fix-numpy-and-pandas-compatibility
Feb 11, 2026
Merged

Fix numpy and pandas compatibility#475
martin-springer merged 60 commits intodevelopmentfrom
fix-numpy-and-pandas-compatibility

Conversation

@martin-springer
Copy link
Collaborator

@martin-springer martin-springer commented Feb 4, 2026

This pull request focuses on improving compatibility with recent versions of pandas (3.0+), numpy (2.x), and xgboost (3.x), while also removing deprecated normalization functions and addressing several minor bugs and deprecations. The main changes involve updating internal calculations to be resolution-agnostic, cleaning up deprecated code, and ensuring the codebase works with the latest dependencies.

Dependency and compatibility updates:

  • Removed upper version restrictions for pandas and numpy in setup.py to support pandas 3.0 and numpy 2.x; updated related requirements and changelog documentation.
  • Fixed pandas 3.0 compatibility throughout normalization.py by replacing .view('int64') with .total_seconds() for datetime calculations, making code resolution-agnostic and robust to changes in pandas datetime resolution. [1] [2] [3] [4] [5] [6]
  • Fixed numpy 2.x compatibility in soiling.py by using .item() and explicit indexing for scalar extraction, and replaced deprecated implicit conversions. [1] [2]

Deprecations and removals:

  • Removed deprecated normalization.delta_index and normalization.check_series_frequency functions (deprecated since v2.0.0); their private helpers remain for internal use. [1] [2] [3] [4]

Bug fixes and minor enhancements:

  • Ensured datetime resolution preservation in normalization.interpolate() so output matches input resolution (e.g., microseconds vs nanoseconds).
  • Fixed xgboost 3.x compatibility in filtering.xgboost_clip_filter() by using xgb.DMatrix with explicit feature names for model prediction. [1] [2]
  • Fixed pandas 4.0 deprecation warnings by switching Timedelta strings from lowercase 'd' to uppercase 'D' and using axis= keyword for DataFrame aggregation methods. [1] [2] [3] [4] [5]

Issues
Closes #472
Closes #445

These changes collectively ensure that the codebase remains compatible with the latest versions of key dependencies and removes legacy code, while also addressing minor bugs and improving code clarity.- [ ] Code changes are covered by tests

  • [ ] Code changes have been evaluated for compatibility/integration with TrendAnalysis
  • [ ] New functions added to __init__.py
  • API.rst is up to date, along with other sphinx docs pages
  • [ ] Example notebooks are rerun and differences in results scrutinized
  • Updated changelog

@martin-springer martin-springer changed the base branch from master to allow-clip-filter-for-energy February 4, 2026 15:36
@codecov-commenter
Copy link

codecov-commenter commented Feb 4, 2026

Codecov Report

❌ Patch coverage is 95.45455% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.14%. Comparing base (5c674a9) to head (fc9fb11).

Files with missing lines Patch % Lines
rdtools/normalization.py 93.33% 2 Missing ⚠️
Additional details and impacted files
@@               Coverage Diff               @@
##           development     #475      +/-   ##
===============================================
- Coverage        96.22%   96.14%   -0.09%     
===============================================
  Files               12       12              
  Lines             2279     2280       +1     
===============================================
- Hits              2193     2192       -1     
- Misses              86       88       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@martin-springer martin-springer changed the title [WIP] Fix numpy and pandas compatibility Fix numpy and pandas compatibility Feb 4, 2026
@martin-springer
Copy link
Collaborator Author

martin-springer commented Feb 4, 2026

@mdeceglie - Commit 87d2b63 simplified the tz handling. Tests are already extensive with tz-naive and tz-aware unit tests. All are passing. The simplification gets us closer to the original implementation.

Base automatically changed from allow-clip-filter-for-energy to development February 4, 2026 22:00
Copy link
Collaborator

@mdeceglie mdeceglie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, but please double check the pending.rst which still has some merge markup in it

@martin-springer martin-springer merged commit 674e384 into development Feb 11, 2026
19 checks passed
@martin-springer martin-springer deleted the fix-numpy-and-pandas-compatibility branch February 11, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants