Skip to content

Conversation

@matthew-frank
Copy link
Contributor

There are a number of fields in the results.txt files that were made redundant (and therefor error prone and inconsistent) when we introduced the systems/*.json files in v1.0.

This PR removes the compliance checker checks for the submission_org and submission_platform fields from the training v5.0.0 compliance checker common.yaml file. (Since a training v5.0.0 compliance checker common.yaml didn't exist yet, I copied the one from training_v4.1.0).

These two fields in the result*.txt files are redundant, and (other than their presence) are not used for anything or checked by any of the compliance or package checkers. They are difficult for many submitters to get correct in their submissions, and are often a source of confusion and/or extra work during the submission review period.

Historically, there were originally 4 fields:

  1. submission_platform which is redundant with some combination of the sys.json system_name, and number_of_nodes fields. This PR removes the compliance check for this field from training v5.0.0.
  2. submission_org which is now redundant with the sys.json submitter field. This PR removes the compliance check for this field from training v5.0.0.
  3. submission_status which was replaced by the sys.json status field, and removed from the compliance checker before the v4.1.0 round (since it used values inconsistent with the values used in the sys.json).
  4. submission_division which is now redundant with the sys.json division field. This PR does not remove the compliance check for this field from training v5.0.0 because the compliance checker uses this field to determine whether the maximal allowed initialization time is 30 minutes (closed division) or 240 minutes (open division).

@matthew-frank matthew-frank requested review from a team as code owners November 4, 2024 20:24
@github-actions
Copy link

github-actions bot commented Nov 4, 2024

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@hiwotadese
Copy link
Contributor

@pgmpablo157321 can you take a look at this?

@hiwotadese
Copy link
Contributor

@pgmpablo157321 to first create v5.0 PR and add these changed on top of that.

@pgmpablo157321
Copy link
Contributor

@matthew-frank Is there any change in this PR that is not in the master branch?

@matthew-frank
Copy link
Contributor Author

The requested change was to remove https://github.com/mlcommons/logging/blob/master/mlperf_logging/compliance_checker/training_5.0.0/common.yaml#L31-L39, and it looks like that didn't happen. Let me rebase the PR to make that clear.

@matthew-frank matthew-frank force-pushed the nvidia/remove-extraneous-submission-fields branch from 7c7f5d2 to 8b8e048 Compare January 30, 2025 16:48
@github-actions github-actions bot locked and limited conversation to collaborators Jan 30, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants