Skip to content

Conversation

@mhairifin
Copy link
Contributor

@mhairifin mhairifin commented Dec 17, 2025

Change Summary

Overview

Some of the MAG variable types are unnecessarily large for the precision of the underlying data. This forces the types of those variables to be more appropriate to the underlying data. Specifically, the magnetic field vectors and magnitude variables will now be CDF_FLOAT rather than CDF_DOUBLE, and the range (which can only be 0,1,2,3) is CDF_UINT1.

New Dependencies

N/A

New Files

N/A

Deleted Files

N/A

Updated Files

  • imap_processing/cdf/config/imap_mag_l2_variable_attrs.yaml
    • Set CDF_DATA_TYPE for range, magnitude, and all vectors to CDF_UINT1, and CDF_FLOAT respectively
  • imap_processing/mag/l2/mag_l2_data.py
    • Don't check schema for attributes so the CDF_DATA_TYPE attribute is carried through

Testing

Updated tests to test the CDF_DATA_TYPE is set for the vectors

@greglucas
Copy link
Collaborator

Interesting, I didn't realize you could put CDF_DATA_TYPE in and it would write these differently to disk. Are you OK with the data originally being in-memory with the larger data type and only at the end writing to the lower precision or do you want this to be in-place initially when creating the arrays?

Is this ready for review, it still says draft but looks good to me.

@mhairifin
Copy link
Contributor Author

@greglucas Thanks for taking a look! We are fine with them being larger in memory - we are mostly concerned with generating unnecessarily large data files and our calibration/science data is not precise to the level of a CDF_DOUBLE. We were discussing whether other levels or data types would benefit from similar treatment, which is why this is a draft PR, but I will mark this as ready to review and can make those changes later if needed.

@mhairifin mhairifin marked this pull request as ready for review December 18, 2025 15:41
@greglucas greglucas merged commit 75e9083 into IMAP-Science-Operations-Center:dev Dec 18, 2025
14 checks passed
@greglucas
Copy link
Collaborator

Awesome, I like doing things incrementally to keep everything moving, so that works well for me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants