Move docs to separate folder

Author: fsschneider

README.md

@@ -7,11 +7,11 @@
 
 <p align="center">
   <a href="https://arxiv.org/abs/2306.07179" target="_blank">Paper (arXiv)</a> •
-  <a href="/CALL_FOR_SUBMISSIONS.md">Call for Submissions</a> •
-  <a href="/GETTING_STARTED.md">Getting Started</a> •
-  <a href="/COMPETITION_RULES.md">Competition Rules</a> •
-  <a href="/DOCUMENTATION.md">Documentation</a> •
-  <a href="/CONTRIBUTING.md">Contributing</a>
+  <a href="/docs/CALL_FOR_SUBMISSIONS.md">Call for Submissions</a> •
+  <a href="/docs/GETTING_STARTED.md">Getting Started</a> •
+  <a href="/docs/COMPETITION_RULES.md">Competition Rules</a> •
+  <a href="/docs/DOCUMENTATION.md">Documentation</a> •
+  <a href="/docs/CONTRIBUTING.md">Contributing</a>
 </p>
 
 [![CI](https://github.com/mlcommons/algorithmic-efficiency/actions/workflows/CI.yml/badge.svg)](https://github.com/mlcommons/algorithmic-efficiency/actions/workflows/CI.yml)
@@ -22,7 +22,7 @@
 
 ---
 
-> *AlgoPerf* is a suite of benchmarks and competitions to measure neural network training speedups due to algorithmic improvements in both training algorithms and models. This is the repository for the *AlgoPerf: Training Algorithms benchmark* and its associated competition. It is developed by the [MLCommons Algorithms Working Group](https://mlcommons.org/en/groups/research-algorithms/). This repository holds the [**competition rules**](/COMPETITION_RULES.md), the [**technical documentation**](/DOCUMENTATION.md) of the benchmark, [**getting started guides**](/GETTING_STARTED.md), and the benchmark code. For a detailed description of the benchmark design, see our [**paper**](https://arxiv.org/abs/2306.07179).
+> *AlgoPerf* is a suite of benchmarks and competitions to measure neural network training speedups due to algorithmic improvements in both training algorithms and models. This is the repository for the *AlgoPerf: Training Algorithms benchmark* and its associated competition. It is developed by the [MLCommons Algorithms Working Group](https://mlcommons.org/en/groups/research-algorithms/). This repository holds the [**competition rules**](/docs/COMPETITION_RULES.md), the [**technical documentation**](/docs/DOCUMENTATION.md) of the benchmark, [**getting started guides**](/docs/GETTING_STARTED.md), and the benchmark code. For a detailed description of the benchmark design, see our [**paper**](https://arxiv.org/abs/2306.07179).
 
 ---
 
@@ -45,9 +45,9 @@
 > [!TIP]
 > **If you have any questions about the benchmark competition or you run into any issues, please feel free to contact us.** Either [file an issue](https://github.com/mlcommons/algorithmic-efficiency/issues), ask a question on [our Discord](https://discord.gg/5FPXK7SMt6) or [join our weekly meetings](https://mlcommons.org/en/groups/research-algorithms/).
 
-You can install this package and dependencies in a [Python virtual environment](/GETTING_STARTED.md#python-virtual-environment) or use a [Docker/Singularity/Apptainer container](/GETTING_STARTED.md#docker) (recommended).
+You can install this package and dependencies in a [Python virtual environment](/docs/GETTING_STARTED.md#python-virtual-environment) or use a [Docker/Singularity/Apptainer container](/docs/GETTING_STARTED.md#docker) (recommended).
 We recommend using a Docker container (or alternatively, a Singularity/Apptainer container) to ensure a similar environment to our scoring and testing environments.
-Both options are described in detail in the [**Getting Started**](/GETTING_STARTED.md) document.
+Both options are described in detail in the [**Getting Started**](/docs/GETTING_STARTED.md) document.
 
 *TL;DR to install the Jax version for GPU run:*
 
@@ -67,7 +67,7 @@ pip3 install -e '.[full]'
 
 ## Getting Started
 
-For detailed instructions on developing and scoring your own algorithm in the benchmark see the [Getting Started](/GETTING_STARTED.md) document.
+For detailed instructions on developing and scoring your own algorithm in the benchmark see the [Getting Started](/docs/GETTING_STARTED.md) document.
 
 *TL;DR running a JAX workload:*
 
@@ -95,21 +95,21 @@ python3 submission_runner.py \
 
 ## Call for Submissions
 
-The [Call for Submissions](/CALL_FOR_SUBMISSIONS.md) announces the first iteration of the AlgoPerf: Training Algorithms competition based on the benchmark by the same name. This document also contains the schedule and key dates for the competition.
+The [Call for Submissions](/docs/CALL_FOR_SUBMISSIONS.md) announces the first iteration of the AlgoPerf: Training Algorithms competition based on the benchmark by the same name. This document also contains the schedule and key dates for the competition.
 
 ### Competition Rules
 
-The competition rules for the *AlgoPerf: Training Algorithms* competition can be found in the separate [**Competition Rules**](/COMPETITION_RULES.md) document.
+The competition rules for the *AlgoPerf: Training Algorithms* competition can be found in the separate [**Competition Rules**](/docs/COMPETITION_RULES.md) document.
 
 ### Technical Documentation of the Benchmark & FAQs
 
-We provide additional technical documentation of the benchmark and answer frequently asked questions in a separate [**Documentation**](/DOCUMENTATION.md) page. Suggestions, clarifications and questions can be raised via pull requests, creating an issue, or by sending an email to the [working group](mailto:[email protected]).
+We provide additional technical documentation of the benchmark and answer frequently asked questions in a separate [**Documentation**](/docs/DOCUMENTATION.md) page. Suggestions, clarifications and questions can be raised via pull requests, creating an issue, or by sending an email to the [working group](mailto:[email protected]).
 
 ## Contributing
 
 We invite everyone to look through our rules, documentation, and codebase and submit issues and pull requests, e.g. for rules changes, clarifications, or any bugs you might encounter. If you are interested in contributing to the work of the working group and influence the benchmark's design decisions, please [join the weekly meetings](https://mlcommons.org/en/groups/research-algorithms/) and consider becoming a member of the working group.
 
-Our [**Contributing**](/CONTRIBUTING.md) document provides further MLCommons contributing guidelines and additional setup and workflow instructions.
+Our [**Contributing**](/docs/CONTRIBUTING.md) document provides further MLCommons contributing guidelines and additional setup and workflow instructions.
 
 ## License
 

datasets/README.md

@@ -45,7 +45,7 @@ Some dataset setups will require you to sign a third-party agreement with the da
 
 If you are running the `dataset_setup.py` script from a Docker container, please
 make sure the data directory is mounted to a directory on your host with
-`-v` flag. If you are following instructions from the [Getting Started guide](/GETTING_STARTED.md) you will have used
+`-v` flag. If you are following instructions from the [Getting Started guide](/docs/GETTING_STARTED.md) you will have used
 the `-v $HOME/data:/data` flag in the `docker run` command. This will mount
 the `$HOME/data` directory to the `/data` directory in the container.
 In this case set, `--data_dir` to  `/data`.

CALL_FOR_SUBMISSIONS.md renamed to docs/CALL_FOR_SUBMISSIONS.md

@@ -19,11 +19,11 @@ Please fill out the (mandatory but non-binding) [**registration form**](https://
 - **Submission deadline: April 04th, 2024** *(moved by a week from the initial March 28th, 2024)*
 - [Announcement of all results](https://mlcommons.org/2024/08/mlc-algoperf-benchmark-competition/): August 1st, 2024
 
-For a detailed and up-to-date timeline see the [Competition Rules](/COMPETITION_RULES.md).
+For a detailed and up-to-date timeline see the [Competition Rules](/docs/COMPETITION_RULES.md).
 
 ## Participation
 
-For details on how to participate in the competition, please refer to our [Competition Rules](/COMPETITION_RULES.md). To learn more about the benchmark, see our [technical documentation](/DOCUMENTATION.md). The benchmark is further motivated, explained, and justified in the accompanying [paper](https://arxiv.org/abs/2306.07179). We require all submissions to be provided under the open-source [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0).
+For details on how to participate in the competition, please refer to our [Competition Rules](/docs/COMPETITION_RULES.md). To learn more about the benchmark, see our [technical documentation](/docs/DOCUMENTATION.md). The benchmark is further motivated, explained, and justified in the accompanying [paper](https://arxiv.org/abs/2306.07179). We require all submissions to be provided under the open-source [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0).
 
 ## Prize Money & Funding
 

COMPETITION_RULES.md renamed to docs/COMPETITION_RULES.md

@@ -71,8 +71,8 @@ We recommend developing in our Docker image to ensure a consistent environment b
 
 To get started see also:
 
-- [Installation with Docker](/GETTING_STARTED.md#docker)
-- [Running a submission inside a Docker Container](/GETTING_STARTED.md#run-your-submission-in-a-docker-container)
+- [Installation with Docker](/docs/GETTING_STARTED.md#docker)
+- [Running a submission inside a Docker Container](/docs/GETTING_STARTED.md#run-your-submission-in-a-docker-container)
 
 ### Pre-built Images on Google Cloud Container Registry
 

CONTRIBUTING.md renamed to docs/CONTRIBUTING.md

@@ -65,7 +65,7 @@ The intention is that a training algorithm submission will be broadly applicable
 
 ### Competition Rules
 
-For a description of the competition rules and how to submit a training algorithm to the AlgoPerf: Training Algorithms Benchmark, see the [Competition Rules](/COMPETITION_RULES.md), which details the entire competition process.
+For a description of the competition rules and how to submit a training algorithm to the AlgoPerf: Training Algorithms Benchmark, see the [Competition Rules](/docs/COMPETITION_RULES.md), which details the entire competition process.
 
 ### Submissions
 
@@ -222,7 +222,6 @@ def update_params(
 - Cannot replace the model parameters with pre-trained ones.
 - Batch norm should work here because the `model_fn` will return updated batch norm moving averages when it is told to with `update_batch_norm`.
 
-
 ###### Prepare for evaluation function
 
 ```python
@@ -278,7 +277,7 @@ def data_selection(
 
 In general, with noisy, non-deterministic training, evaluation frequency can affect training time measurements as more "bites of the apple" potentially allows the training code to exploit instability. We also want to discourage submissions from complicated and unrealistic logic that attempts to guess when training is close to complete and increases the evaluation rate, while not producing a well-sampled training curve at the start of training. Simply allowing submissions complete freedom over evaluation frequency encourages competitors to work to minimize the number of evaluations, which distracts from the primary goal of finding better training algorithms.
 
-Submissions are eligible for an untimed eval every `eval_period` seconds. Before proceeding to evaluation, the submission can prepare the model through a call to `prepare_for_eval`, effectively modifying the model parameters and state as well as the the optimizer state. Any additional evaluations performed by the submission code count against the runtime for scoring. 
+Submissions are eligible for an untimed eval every `eval_period` seconds. Before proceeding to evaluation, the submission can prepare the model through a call to `prepare_for_eval`, effectively modifying the model parameters and state as well as the the optimizer state. Any additional evaluations performed by the submission code count against the runtime for scoring.
 The harness that runs the submission code will attempt to eval every `eval_period` seconds by checking between each submission step (call of `update_params`) whether it has been at least `eval_period` seconds since that last eval, if so, the submission is given the possibility to prepare for evaluation (through a timed call to `prepare_for_eval`). If the accumulated runtime does not exceed the maximum allowed runtime after the preparation step, the clock is paused, and the submission is evaluated. This means that if calls to `update_params` typically take a lot more than `eval_period` seconds, such submissions will not receive as many untimed evals as a submission that had an `update_params` function that took less time. However, for appropriate settings of `eval_period`, we expect this to be quite rare. Submissions are always free to restructure their `update_params` code to split work into two subsequent steps to regain the potential benefits of these untimed model evaluations. For each workload, the `eval_period` will be set such that the total evaluation time is roughly between 10% and 20% of the total training time for the target-setting runs.
 
 #### Valid submissions
@@ -475,7 +474,7 @@ Our scoring procedure uses the held-out workloads only to penalize submissions t
 
 NOTE: Submitters are no longer required to self-report results for AlgoPerf competition v0.5.
 
-The qualification set is designed for submitters that may not have the compute resources to self-report on the full set of [fixed](#fixed-workloads) and [held-out workloads](#randomized-workloads). They may instead self-report numbers on this smaller qualification set. The best-performing submissions may then qualify for compute sponsorship offering a free evaluation on the full benchmark set and therefore the possibility to win [awards and prizes](/COMPETITION_RULES.md#prizes).
+The qualification set is designed for submitters that may not have the compute resources to self-report on the full set of [fixed](#fixed-workloads) and [held-out workloads](#randomized-workloads). They may instead self-report numbers on this smaller qualification set. The best-performing submissions may then qualify for compute sponsorship offering a free evaluation on the full benchmark set and therefore the possibility to win [awards and prizes](/docs/COMPETITION_RULES.md#prizes).
 
 The qualification set consists of the same [fixed workloads](#fixed-workloads) as mentioned above, except for both workloads on *ImageNet*, both workloads on *LibriSpeech*, and the *fastMRI* workload. The remaining three workloads (*WMT*, *Criteo 1TB*, and *OGBG*) form the qualification set. There are no [randomized workloads](#randomized-workloads) in the qualification set. The qualification set of workloads aims to have a combined runtime of roughly 24 hours on the [benchmarking hardware](#benchmarking-hardware).
 
@@ -585,7 +584,7 @@ GPUs with higher per GPU memory, please monitor your memory usage to make sure i
 
 #### How do I run this on my SLURM cluster?
 
-You may run into issues with `sudo` and `docker` on a SLURM cluster. To run the workloads in a SLURM cluster you can use Apptainer (previously Singularity), see this [section](/GETTING_STARTED.md#using-singularityapptainer-instead-of-docker).
+You may run into issues with `sudo` and `docker` on a SLURM cluster. To run the workloads in a SLURM cluster you can use Apptainer (previously Singularity), see this [section](/docs/GETTING_STARTED.md#using-singularityapptainer-instead-of-docker).
 
 #### How can I run this on my AWS/GCP/Azure cloud project?
 
@@ -627,13 +626,13 @@ You only have to use the benchmarking hardware for runs that are directly involv
 
 NOTE: Submitters are no longer required to self-report results for AlgoPerf competition v0.5.
 
-Submitters unable to self-fund scoring costs can instead self-report only on the [qualification set of workloads](/COMPETITION_RULES.md#qualification-set) that excludes some of the most expensive workloads. Based on this performance on the qualification set, the working group will provide - as funding allows - compute to evaluate and score the most promising submissions. Additionally, we encourage researchers to reach out to the [working group](mailto:[email protected]) to find potential collaborators with the resources to run larger, more comprehensive experiments for both developing and scoring submissions.
+Submitters unable to self-fund scoring costs can instead self-report only on the [qualification set of workloads](/docs/COMPETITION_RULES.md#qualification-set) that excludes some of the most expensive workloads. Based on this performance on the qualification set, the working group will provide - as funding allows - compute to evaluate and score the most promising submissions. Additionally, we encourage researchers to reach out to the [working group](mailto:[email protected]) to find potential collaborators with the resources to run larger, more comprehensive experiments for both developing and scoring submissions.
 
 #### Can I submit previously published training algorithms as submissions?
 
 Yes, you may, as long as it isn't an exact copy of an existing submission.
 For example, you may submit the Adam optimizer with your particularly effective hyperparameter search space and hyperparameter configuration, as different choices for hyperparameter values and/or search spaces constitute different training algorithms and are potential sources of innovation.
-That said, while submitting Adam with some novel heuristic to set various hyperparameters, some especially effective hyperparameter search space, or your single best hyperparameter configuration is fine, avoid making multiple submissions that only differ by their hyperparameter configuration without a convincing justification they are substantially different (see ["Can I submit multiple times to the benchmark competition?"](/COMPETITION_RULES.md#can-i-submit-multiple-times-to-the-benchmark-competition), above).
+That said, while submitting Adam with some novel heuristic to set various hyperparameters, some especially effective hyperparameter search space, or your single best hyperparameter configuration is fine, avoid making multiple submissions that only differ by their hyperparameter configuration without a convincing justification they are substantially different (see ["Can I submit multiple times to the benchmark competition?"](/docs/COMPETITION_RULES.md#can-i-submit-multiple-times-to-the-benchmark-competition), above).
 
 ## Disclaimers