Results from self hosted Github actions - NVIDIARTX4090

Author: arjunsuresh

open/MLCommons/measurements/gh_action-reference-gpu-pytorch_v2.5.1-cu124/README.md

@@ -1,3 +1,3 @@
-| Model               | Scenario   | Accuracy   |   Throughput | Latency (in ms)   |
-|---------------------|------------|------------|--------------|-------------------|
-| stable-diffusion-xl | offline    | ()         |        0.352 | -                 |
+| Model   | Scenario   |   Accuracy |   Throughput | Latency (in ms)   |
+|---------|------------|------------|--------------|-------------------|
+| gptj-99 | offline    |        264 |       49.207 | -                 |

open/MLCommons/measurements/gh_action-reference-gpu-pytorch_v2.5.1-cu124/gptj-99/offline/README.md

@@ -16,7 +16,7 @@ pip install -U cmind
 
 cm rm cache -f
 
-cm pull repo gateoverflow@mlperf-automations --checkout=37a8fb04148356c3b7c35014ca75477580652179
+cm pull repo gateoverflow@mlperf-automations --checkout=ee7be856e5def9e46bc4535b6128ec342f6931db
 
 cm run script \
 	--tags=app,mlperf,inference,generic,_reference,_gptj-99,_pytorch,_cuda,_test,_r5.0-dev_default,_float16,_offline \
@@ -105,4 +105,4 @@ Model Precision: fp32
 `GEN_LEN`: `264.0`, Required accuracy for closed division `>= 42.55663`
 
 ### Performance Results 
-`Samples per second`: `48.2391`
+`Samples per second`: `49.2068`

open/MLCommons/measurements/gh_action-reference-gpu-pytorch_v2.5.1-cu124/gptj-99/offline/accuracy_console.out

@@ -2,17 +2,17 @@ Constructing QSL
 Encoding Samples
 Finished constructing QSL.
 Loading PyTorch model...
-Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]Loading checkpoint shards:  33%|███▎      | 1/3 [00:00<00:01,  1.41it/s]Loading checkpoint shards:  67%|██████▋   | 2/3 [00:01<00:00,  1.44it/s]Loading checkpoint shards: 100%|██████████| 3/3 [00:01<00:00,  1.97it/s]Loading checkpoint shards: 100%|██████████| 3/3 [00:01<00:00,  1.79it/s]
+Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]Loading checkpoint shards:  33%|███▎      | 1/3 [00:00<00:01,  1.71it/s]Loading checkpoint shards:  67%|██████▋   | 2/3 [00:01<00:00,  1.76it/s]Loading checkpoint shards: 100%|██████████| 3/3 [00:01<00:00,  2.30it/s]Loading checkpoint shards: 100%|██████████| 3/3 [00:01<00:00,  2.11it/s]
 Some weights of the model checkpoint at /home/cmuser/CM/repos/local/cache/31767c21a8f149e5/checkpoint/checkpoint-final were not used when initializing GPTJForCausalLM: ['transformer.h.0.attn.bias', 'transformer.h.0.attn.masked_bias', 'transformer.h.1.attn.bias', 'transformer.h.1.attn.masked_bias', 'transformer.h.10.attn.bias', 'transformer.h.10.attn.masked_bias', 'transformer.h.11.attn.bias', 'transformer.h.11.attn.masked_bias', 'transformer.h.12.attn.bias', 'transformer.h.12.attn.masked_bias', 'transformer.h.13.attn.bias', 'transformer.h.13.attn.masked_bias', 'transformer.h.14.attn.bias', 'transformer.h.14.attn.masked_bias', 'transformer.h.15.attn.bias', 'transformer.h.15.attn.masked_bias', 'transformer.h.16.attn.bias', 'transformer.h.16.attn.masked_bias', 'transformer.h.17.attn.bias', 'transformer.h.17.attn.masked_bias', 'transformer.h.18.attn.bias', 'transformer.h.18.attn.masked_bias', 'transformer.h.19.attn.bias', 'transformer.h.19.attn.masked_bias', 'transformer.h.2.attn.bias', 'transformer.h.2.attn.masked_bias', 'transformer.h.20.attn.bias', 'transformer.h.20.attn.masked_bias', 'transformer.h.21.attn.bias', 'transformer.h.21.attn.masked_bias', 'transformer.h.22.attn.bias', 'transformer.h.22.attn.masked_bias', 'transformer.h.23.attn.bias', 'transformer.h.23.attn.masked_bias', 'transformer.h.24.attn.bias', 'transformer.h.24.attn.masked_bias', 'transformer.h.25.attn.bias', 'transformer.h.25.attn.masked_bias', 'transformer.h.26.attn.bias', 'transformer.h.26.attn.masked_bias', 'transformer.h.27.attn.bias', 'transformer.h.27.attn.masked_bias', 'transformer.h.3.attn.bias', 'transformer.h.3.attn.masked_bias', 'transformer.h.4.attn.bias', 'transformer.h.4.attn.masked_bias', 'transformer.h.5.attn.bias', 'transformer.h.5.attn.masked_bias', 'transformer.h.6.attn.bias', 'transformer.h.6.attn.masked_bias', 'transformer.h.7.attn.bias', 'transformer.h.7.attn.masked_bias', 'transformer.h.8.attn.bias', 'transformer.h.8.attn.masked_bias', 'transformer.h.9.attn.bias', 'transformer.h.9.attn.masked_bias']
 - This IS expected if you are initializing GPTJForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
 - This IS NOT expected if you are initializing GPTJForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
 Casting models to GPU...
-  0%|          | 0/285 [00:00<?, ?it/s]100%|██████████| 285/285 [00:00<00:00, 1934266.41it/s]
+  0%|          | 0/285 [00:00<?, ?it/s]100%|██████████| 285/285 [00:00<00:00, 1962851.63it/s]
 Running LoadGen test...
 Number of Samples in query_samples :  1
   0%|          | 0/1 [00:00<?, ?it/s]/home/cmuser/venv/cm/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:676: UserWarning: `num_beams` is set to 1. However, `early_stopping` is set to `True` -- this flag is only used in beam-based generation modes. You should set `num_beams>1` or unset `early_stopping`.
   warnings.warn(
-100%|██████████| 1/1 [00:01<00:00,  1.22s/it]100%|██████████| 1/1 [00:01<00:00,  1.22s/it]
+100%|██████████| 1/1 [00:01<00:00,  1.21s/it]100%|██████████| 1/1 [00:01<00:00,  1.21s/it]
 
 No warnings encountered during test.
 

open/MLCommons/measurements/gh_action-reference-gpu-pytorch_v2.5.1-cu124/gptj-99/offline/cm-deps.mmd

@@ -31,22 +31,22 @@ graph TD
     get-generic-python-lib,94b62a682bc44791_(_package.numpy_) --> get,generic-python-lib,_pip
     get-cuda-devices,7a3ede4d3558427a_(_with-pycuda_) --> get,generic-python-lib,_package.numpy
     app-mlperf-inference,d775cac873ee4231_(_reference,_gptj-99,_pytorch,_cuda,_test,_r5.0-dev_default,_float16,_offline_) --> get,cuda-devices,_with-pycuda
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> detect,os
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> detect,os
     detect-cpu,586c8a43320142f7 --> detect,os
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> detect,cpu
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,sys-utils-cm
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,python
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,cuda,_cudnn
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> detect,cpu
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,sys-utils-cm
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,python
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,cuda,_cudnn
     get-generic-python-lib,94b62a682bc44791_(_torch_cuda_) --> get,python3
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_torch_cuda
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_torch_cuda
     get-generic-python-lib,94b62a682bc44791_(_torchvision_cuda_) --> get,python3
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_torchvision_cuda
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_torchvision_cuda
     get-generic-python-lib,94b62a682bc44791_(_transformers_) --> get,python3
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_transformers
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_transformers
     download-and-extract,c67e81a4ce2649f5_(_rclone,_url.mlc-inference:mlcommons-inference-wg-public/gpt-j_) --> download,file,_rclone,_url.mlc-inference:mlcommons-inference-wg-public/gpt-j
     get-ml-model-gptj,a41166210f294fbf_(_pytorch_) --> download-and-extract,_rclone,_url.mlc-inference:mlcommons-inference-wg-public/gpt-j
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,ml-model,large-language-model,gptj,raw,_pytorch
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,dataset,cnndm,_validation
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,ml-model,large-language-model,gptj,raw,_pytorch
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,dataset,cnndm,_validation
     generate-mlperf-inference-user-conf,3af4475745964b93 --> detect,os
     detect-cpu,586c8a43320142f7 --> detect,os
     generate-mlperf-inference-user-conf,3af4475745964b93 --> detect,cpu
@@ -57,22 +57,22 @@ graph TD
     generate-mlperf-inference-user-conf,3af4475745964b93 --> get,mlcommons,inference,src
     get-mlperf-inference-sut-configs,c2fbf72009e2445b --> get,cache,dir,_name.mlperf-inference-sut-configs
     generate-mlperf-inference-user-conf,3af4475745964b93 --> get,sut,configs
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> generate,user-conf,mlperf,inference
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,loadgen
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> generate,user-conf,mlperf,inference
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,loadgen
     get-mlperf-inference-src,4b57186581024797 --> detect,os
     get-mlperf-inference-src,4b57186581024797 --> get,python3
     get-mlperf-inference-src,4b57186581024797 --> get,git,repo,_branch.master,_repo.https://github.com/mlcommons/inference
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,mlcommons,inference,src
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,mlcommons,inference,src
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,mlcommons,inference,src
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,mlcommons,inference,src
     get-generic-python-lib,94b62a682bc44791_(_package.psutil_) --> get,python3
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_package.psutil
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_package.psutil
     get-generic-python-lib,94b62a682bc44791_(_package.datasets_) --> get,python3
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_package.datasets
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_package.datasets
     get-generic-python-lib,94b62a682bc44791_(_package.attrs_) --> get,python3
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_package.attrs
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_package.attrs
     get-generic-python-lib,94b62a682bc44791_(_package.accelerate_) --> get,python3
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_package.accelerate
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> get,generic-python-lib,_package.accelerate
     detect-cpu,586c8a43320142f7 --> detect,os
     benchmark-program,19f369ef47084895 --> detect,cpu
     benchmark-program-mlperf,cfff0132a8aa4018 --> benchmark-program,program
-    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_pytorch,_cuda,_offline,_gptj-99,_float16_) --> benchmark-mlperf
+    app-mlperf-inference-mlcommons-python,ff149e9781fc4b65_(_cuda,_pytorch,_offline,_gptj-99,_float16_) --> benchmark-mlperf