Skip to content

Conversation

@edmundmiller
Copy link
Contributor

Summary

Enables CO2 footprint tracking during nf-test runs and uploads reports to S3 for analysis.

  • Adds [email protected] plugin to test configuration
  • Uploads all CO2 trace files to S3 after each test shard completes
  • Organizes reports by date/branch/profile/shard for easy querying with DuckDB

Changes

Configuration

  • tests/config/nf-test.config: Added nf-co2footprint plugin

CI/CD

  • .github/workflows/nf-test.yml: Added S3 upload step for CO2 reports
    • Finds all co2footprint_trace*.txt files per shard
    • Uploads to s3://nf-core-resource-optimization/co2reports/modules/YYYY-MM-DD/branch-name/profile/shard/
    • Runs even if tests fail (uses if: always())

S3 Report Structure

Reports are organized for analysis:

s3://nf-core-resource-optimization/co2reports/modules/
  └── 2025-10-29/
      └── co2-footprint-tracking/
          └── docker/
              └── 1/
                  ├── co2footprint_trace.txt
                  └── co2footprint_trace_*.txt

Report Format

Each trace file contains TSV data with:

  • task_id, status, name
  • energy_consumption, CO2e, CO2e_market, carbon_intensity
  • %cpu, memory, realtime, cpus
  • powerdraw_cpu, cpu_model

Testing

  • Verify plugin loads successfully
  • Confirm CO2 reports are generated during test runs
  • Check S3 uploads complete successfully
  • Validate report structure for DuckDB compatibility

Refs: #9291

🤖 Generated with Claude Code

Co-Authored-By: Claude [email protected]

@edmundmiller edmundmiller requested review from a team as code owners October 29, 2025 10:39
@edmundmiller edmundmiller self-assigned this Oct 29, 2025
@edmundmiller edmundmiller linked an issue Oct 29, 2025 that may be closed by this pull request
Track CO2 emissions during nf-test runs and upload reports to S3
for analysis with DuckDB.

Refs: #9291

test: Trigger CI to test CO2 footprint tracking
@edmundmiller edmundmiller force-pushed the co2-footprint-tracking branch from b7f3447 to 42c0b16 Compare October 29, 2025 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Track GitHub Actions CO2 usage

2 participants