Llama3.1-8B converges faster than the RCPs

[psyhtest]

As discussed in the Training WG meeting on 2/Oct, Llama3.1-8B converges faster than the RCPs, at least with GBS=32.

Here's one example:

INFO - ------------------------------
INFO -  Running RCP Checker, pass: pruned_rcps
INFO - ------------------------------
INFO -  RCP Record: {'Benchmark': 'llama31_8b', 'BS': 32, 'Hyperparams': {'opt_base_learning_rate': 0.001, 'opt_learning_rate_warmup_samples': 16348, 'gradient_accumulation_steps': 2},
'Epochs to converge': [196608, 196608, 196608, 208896, 208896, 208896, 208896, 208896, 208896, 208896, 208896, 221184, 221184, 221184, 221184, 221184, 233472, 233472, 233472, 233472],
'RCP Mean': np.float64(215040.0), 'RCP Stdev': np.float64(11976.860890901255), 'Max Speedup': np.float64(1.042198772353707), 'Min Epochs': np.float64(206333.00067543983)}
INFO -  Submission mean epochs: 180576.0000
ERROR - RCP Test Failed: RCP found
INFO - ------------------------------

The mean epochs is ~206.3k, while the submission mean epochs is 180.6k.

Proposed workarounds:

• Slowing down convergence by increasing the warm up samples (up to 16k as in the RCPs), as well as adjusting the learning rate.
• Submitting new RCPs by early next week (NVIDIA).

[ShriyaRishab]

NVIDIA new RCPs PR - mlcommons/logging#440
@psyhtest - please review this

[psyhtest]

@ShriyaRishab LGTM!

[psyhtest]

So we have new results aligned to the new RCPs, but probably cannot upload them until mlcommons/logging#440 is merged. Or is there a workaround to disable the RCP check on the whole or part of the repository?

[ShriyaRishab]

@pgmpablo157321 - can you help provide instructions on using RCP bypass for a submission?

[psyhtest]

My question was: everyone was referring to RCP bypass as common knowledge, but I could not find clear guidance how to apply it. (There is no checkbox in the Submission UI!)

It took me first taking a leap of faith from clicking on the User Documentation link in the right top corner of the Submission UI to discovering a FAQ there:

How to make the submission checker run with the rcp_bypass flag in the UI?

The script that is running in the UI can be found here. Notice that it provides a way to pass flags to the submission checker by putting them into a file in your submission. Create a file called package_checker_params in the root directory of your submission and put the flags you want to use.

Then having to deal with another rejection from the Submission UI because the package checker flag is actually called --rcp-bypass with a dash, not --rcp_bypass with an underscore.

Hope this is helpful for other submitters in this round too, but we ought to improve the docs.

[pgmpablo157321]

@psyhtest How would you like to document how to pass the rcp-bypass flag?

We currently have that documented in the User Documentation and also cover that topic in the submitter orientation. But how should we make this more visible to submitters? Maybe in the policies repository?

[psyhtest]

Yes, please, I was initially looking in the main Training policies document.