Changes for SCC24 (#18)

* Added doc files * Added doc files * Added publish file * Added mkdocs.yml file * Support generic result repositories * Fixes for SCC24 results * Fixes for SCC24 results * Added submission checker gh action * Use v4.1 version for scc24 * Results on system test * Restrict CLA check to MLC * Fix submission checker version * Added results updater GH action * Updated results summary * Updated results summary * Update update-results.yml * Updated results summary * Added results updater GH action * Updated results summary * Added results updater GH action * Updated results summary * Use incremental dbversion * Use incremental dbversion * Updated results summary * Delete summary_results.json * Updated results summary * Update update-results.yml * Updated results summary * Update publish.yml * Updated results summary * Update update-results.yml * Updated results summary * Results from self hosted Github actions - NVIDIARTX4090 * Updated results summary * Results from self hosted Github actions - NVIDIARTX4090 * Updated results summary * Results from self hosted Github actions - NVIDIARTX4090 * Updated results summary --------- Co-authored-by: admin <admin@mlcommons.org> Co-authored-by: GitHub Action <action@github.com>
mlcommons · Oct 1, 2024 · c4270cb · c4270cb
1 parent 6678c20
commit c4270cb
Show file tree

Hide file tree

Showing 105 changed files with 5,751 additions and 2 deletions.
diff --git a/.github/workflows/cla.yml b/.github/workflows/cla.yml
@@ -8,6 +8,7 @@ on:
 
 jobs:
   cla-check:
+    if: github.repository_owner == 'mlcommons'
     runs-on: ubuntu-latest
     steps:
       - name: "MLCommons CLA bot check"

diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml
@@ -0,0 +1,31 @@
+name: Publish visualization site
+
+
+on:
+  release:
+    types: [published]
+  push:
+    branches:
+      - mlperf-inference-results-scc24
+      - docs
+
+jobs:
+
+  publish:
+    name: Publish the site
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository normally
+        uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.11"
+
+      - name: Init docs site
+        run: bash run.sh
+
+      - name: Run Mkdocs deploy
+        run: mkdocs gh-deploy --force
diff --git a/.github/workflows/run-mlperf-inference-submission-checker.yml b/.github/workflows/run-mlperf-inference-submission-checker.yml
@@ -0,0 +1,31 @@
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: MLPerf inference submission checker
+
+on:
+  pull_request:
+    branches: [ "main", "mlperf-inference-results-scc24" ]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    env:
+      CM_INDEX: "on"
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: [ "3.10" ]
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v3
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python3 -m pip install cm4mlops
+    - name: Run MLPerf Inference Submission Checker
+      run: |
+        cm run script --tags=run,mlperf,inference,submission,checker,_short-run --adr.submission-checker-src.tags=_repo.https://github.com/gateoverflow/inference --src_version=v4.1  --quiet --extra_args=" --skip-extra-files-in-root-check" --submission_dir=./
diff --git a/.github/workflows/update-results.yml b/.github/workflows/update-results.yml
@@ -0,0 +1,51 @@
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: MLPerf inference results updater
+
+
+on:
+  push:
+    branches: [ "main", "mlperf-inference-results-scc24" ]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    env:
+      CM_INDEX: "on"
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: [ "3.10" ]
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v3
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Install dependencies
+      run: |
+        python3 -m pip install cm4mlops
+
+    - name: Run MLPerf Inference Submission Checker and generate results summary
+      run: |
+        cm run script --tags=run,mlperf,inference,submission,checker,_short-run --src_version=v4.1 --adr.submission-checker-src.tags=_repo.https://github.com/gateoverflow/inference,_branch.improve_result_generation --quiet --extra_args=" --skip-extra-files-in-root-check"  --submission_dir=./ > >(tee -a out.txt) 2> >(tee -a checker_log.txt >&2)
+        cmr "convert from-csv to-md" --csv_file=summary.csv --md_file=README.md
+
+        USER="GitHub Action"
+        EMAIL=action@github.com
+
+        git config --global user.name "$USER"
+        git config --global user.email "$EMAIL"
+        git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN_TOKEN }}@github.com/${{ github.repository }}
+        git add summary*
+        echo -e 'Please download [summary.xlsx](summary.xlsx) to view the most recent results. \n ```' > temp
+        tail -n 16 checker_log.txt >> temp
+        echo -e '\n```\n' >> temp
+        cat temp | cat - README.md > temp1
+        head -n 100 temp1 > README.md
+        git add README.md
+        
+        git diff-index --quiet HEAD || (git commit -am "Updated results summary" && git push origin)
diff --git a/README.md b/README.md
@@ -0,0 +1,29 @@
+Please download [summary.xlsx](summary.xlsx) to view the most recent results. 
+ ```
+[2024-10-01 12:37:20,009 submission_checker1.py:2936 INFO] Results=6, NoResults=0, Power Results=0
+[2024-10-01 12:37:20,009 submission_checker1.py:2943 INFO] ---
+[2024-10-01 12:37:20,009 submission_checker1.py:2944 INFO] Closed Results=0, Closed Power Results=0
+
+[2024-10-01 12:37:20,009 submission_checker1.py:2949 INFO] Open Results=6, Open Power Results=0
+
+[2024-10-01 12:37:20,009 submission_checker1.py:2954 INFO] Network Results=0, Network Power Results=0
+
+[2024-10-01 12:37:20,009 submission_checker1.py:2959 INFO] ---
+[2024-10-01 12:37:20,009 submission_checker1.py:2961 INFO] Systems=5, Power Systems=0
+[2024-10-01 12:37:20,009 submission_checker1.py:2962 INFO] Closed Systems=0, Closed Power Systems=0
+[2024-10-01 12:37:20,009 submission_checker1.py:2967 INFO] Open Systems=5, Open Power Systems=0
+[2024-10-01 12:37:20,009 submission_checker1.py:2972 INFO] Network Systems=0, Network Power Systems=0
+[2024-10-01 12:37:20,009 submission_checker1.py:2977 INFO] ---
+[2024-10-01 12:37:20,009 submission_checker1.py:2982 INFO] SUMMARY: submission looks OK
+INFO:root:       ! call "postprocess" from /home/runner/CM/repos/mlcommons@cm4mlops/script/run-mlperf-inference-submission-checker/customize.py
+
+```
+
+|    | Organization   | Availability   | Division   | SystemType   | SystemName   | Platform                                               | Model               | MlperfModel         | Scenario   |    Result | Accuracy                                                      |   number_of_nodes | host_processor_model_name   |   host_processors_per_node |   host_processor_core_count | accelerator_model_name   |   accelerators_per_node | Location                                                                                                  | framework      | operating_system                                | notes                             |   compliance |   errors | version   |   inferred | has_power   | Units     | weight_data_types   |
+|---:|:---------------|:---------------|:-----------|:-------------|:-------------|:-------------------------------------------------------|:--------------------|:--------------------|:-----------|----------:|:--------------------------------------------------------------|------------------:|:----------------------------|---------------------------:|----------------------------:|:-------------------------|------------------------:|:----------------------------------------------------------------------------------------------------------|:---------------|:------------------------------------------------|:----------------------------------|-------------:|---------:|:----------|-----------:|:------------|:----------|:--------------------|
+|  0 | MLCommons      | available      | open       | datacenter   | 48ed6105bd85 | 48ed6105bd85-nvidia-gpu-TensorRT-scc24-main            | stable-diffusion-xl | stable-diffusion-xl | Offline    |  1.13292  | CLIP_SCORE: 15.586050063371658  FID_SCORE: 236.8087101317688  |                 1 | Intel(R) Xeon(R) w7-2495X   |                          1 |                          24 | NVIDIA GeForce RTX 4090  |                       1 | open/MLCommons/results/48ed6105bd85-nvidia-gpu-TensorRT-scc24-main/stable-diffusion-xl/offline            | TensorRT       | Ubuntu 20.04 (linux-6.2.0-39-generic-glibc2.31) | Automated by MLCommons CM v2.3.6. |            1 |        0 | v4.1      |          0 | False       | Samples/s | int8                |
+|  1 | MLCommons      | available      | open       | datacenter   | 48ed6105bd85 | 48ed6105bd85-nvidia-gpu-TensorRT-scc24-base            | stable-diffusion-xl | stable-diffusion-xl | Offline    |  1.13598  | CLIP_SCORE: 15.586050063371658  FID_SCORE: 236.8087101317688  |                 1 | Intel(R) Xeon(R) w7-2495X   |                          1 |                          24 | NVIDIA GeForce RTX 4090  |                       1 | open/MLCommons/results/48ed6105bd85-nvidia-gpu-TensorRT-scc24-base/stable-diffusion-xl/offline            | TensorRT       | Ubuntu 20.04 (linux-6.2.0-39-generic-glibc2.31) | Automated by MLCommons CM v2.3.6. |            1 |        0 | v4.1      |          0 | False       | Samples/s | int8                |
+|  2 | MLCommons      | available      | open       | edge         | gh_action    | gh_action-reference-gpu-pytorch_v2.4.1-default_config  | gptj-99             | gptj-99             | Offline    | 52.9478   | nan                                                           |                 1 | Intel(R) Xeon(R) w7-2495X   |                          1 |                          24 | NVIDIA GeForce RTX 4090  |                       1 | open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.4.1-default_config/gptj-99/offline              | pytorch v2.4.1 | Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35) | Automated by MLCommons CM v2.3.4. |            1 |        0 | v4.1      |          0 | False       | Tokens/s  | fp32                |
+|  3 | MLCommons      | available      | open       | edge         | gh_action    | gh_action-reference-gpu-pytorch_v2.4.1-default_config  | stable-diffusion-xl | stable-diffusion-xl | Offline    |  0.345721 | CLIP_SCORE: 15.18544016778469  FID_SCORE: 235.69504308101006  |                 1 | Intel(R) Xeon(R) w7-2495X   |                          1 |                          24 | NVIDIA GeForce RTX 4090  |                       1 | open/MLCommons/results/gh_action-reference-gpu-pytorch_v2.4.1-default_config/stable-diffusion-xl/offline  | pytorch v2.4.1 | Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35) | Automated by MLCommons CM v2.3.4. |            1 |        0 | v4.1      |          0 | False       | Samples/s | fp32                |
+|  4 | MLCommons      | available      | open       | datacenter   | 48ed6105bd85 | 48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base | stable-diffusion-xl | stable-diffusion-xl | Offline    |  0.373636 | CLIP_SCORE: 15.236237794160843  FID_SCORE: 238.78369342212613 |                 1 | Intel(R) Xeon(R) w7-2495X   |                          1 |                          24 | NVIDIA GeForce RTX 4090  |                       1 | open/MLCommons/results/48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base/stable-diffusion-xl/offline | TensorRT       | Ubuntu 20.04 (linux-6.2.0-39-generic-glibc2.31) | Automated by MLCommons CM v2.3.6. |            1 |        0 | v4.1      |          0 | False       | Samples/s | fp32                |
+|  5 | MLCommons      | available      | open       | datacenter   | f9ac88850adc | f9ac88850adc-reference-gpu-pytorch_v2.4.1-scc24-base   | stable-diffusion-xl | stable-diffusion-xl | Offline    |  0.376944 | CLIP_SCORE: 15.18544016778469  FID_SCORE: 235.69504308101006  |                 1 | Intel(R) Xeon(R) w7-2495X   |                          1 |                          24 | NVIDIA GeForce RTX 4090  |                       1 | open/MLCommons/results/f9ac88850adc-reference-gpu-pytorch_v2.4.1-scc24-base/stable-diffusion-xl/offline   | pytorch v2.4.1 | Ubuntu 22.04 (linux-6.2.0-39-generic-glibc2.35) | Automated by MLCommons CM v2.3.9. |            1 |        0 | v4.1      |          0 | False       | Samples/s | fp32                |
diff --git a/dbversion b/dbversion
@@ -0,0 +1 @@
+100
diff --git a/docinit.sh b/docinit.sh
@@ -0,0 +1,43 @@
+#!/bin/bash
+
+if [ ! -e docs ]; then
+    git clone https://github.com/GATEOverflow/inference_results_visualization_template.git docs
+    test $? -eq 0 || exit $?
+fi
+
+python3 -m pip install -r docs/requirements.txt
+
+if [ ! -e overrides ]; then
+    cp -r docs/overrides overrides
+    test $? -eq 0 || exit $?
+fi
+
+repo_owner=${INFERENCE_RESULTS_REPO_OWNER:-mlcommons}
+repo_branch=${INFERENCE_RESULTS_REPO_BRANCH:-main}
+repo_name=${INFERENCE_RESULTS_REPO_NAME:-inference_results_${INFERENCE_RESULTS_VERSION}}
+ver_num=$(cat dbversion)
+let ver_num++
+echo "ver_num=$ver_num" > dbversion
+if [ ! -e docs/javascripts/config.js ]; then
+    if [ -n "${INFERENCE_RESULTS_VERSION}" ]; then
+         echo "const results_version=\"${INFERENCE_RESULTS_VERSION}\";" > docs/javascripts/config.js;
+         echo "var repo_owner=\"${repo_owner}\";" >> docs/javascripts/config.js;
+         echo "var repo_branch=\"${repo_branch}\";" >> docs/javascripts/config.js;
+         echo "var repo_name=\"${repo_name}\";" >> docs/javascripts/config.js;
+         #ver_num=`echo ${INFERENCE_RESULTS_VERSION} | tr -cd '0-9'`
+         echo "const dbVersion =\"${ver_num}\";" >> docs/javascripts/config.js;
+    else
+       echo "Please export INFERENCE_RESULTS_VERSION=v4.1 or the corresponding version";
+       exit 1
+    fi
+fi
+
+if [ ! -e docs/thirdparty/tablesorter ]; then
+    cd docs/thirdparty && git clone https://github.com/Mottie/tablesorter.git && cd -
+    test $? -eq 0 || exit $?
+fi
+
+python3 process.py
+test $? -eq 0 || exit $?
+python3 process_results_table.py
+test $? -eq 0 || exit $?
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -0,0 +1,43 @@
+site_name: MLPerf Inference Results Comparison
+repo_url: https://github.com/mlcommons/inference_results_v4.0
+theme:
+  name: material
+  logo: img/logo_v2.svg
+  favicon: img/logo_v2.svg
+  palette:
+    primary: deep purple
+    accent: yellow
+  custom_dir: overrides
+  features:
+    - content.tabs.link
+    - content.code.copy
+    - navigation.expand
+    - navigation.sections
+    - navigation.indexes
+    - navigation.instant
+    - navigation.tabs
+    - navigation.tabs.sticky
+    - navigation.top
+    - toc.follow
+nav:
+  - Results: index.md
+  - Compare: compare/index.md
+  - Top Results: top_results/index.md
+
+markdown_extensions:
+  - pymdownx.tasklist:
+      custom_checkbox: true
+  - pymdownx.details
+  - admonition
+  - attr_list
+  - def_list
+  - footnotes
+  - pymdownx.superfences:
+      custom_fences:
+        - name: mermaid
+          class: mermaid
+          format: !!python/name:pymdownx.superfences.fence_code_format
+  - pymdownx.tabbed:
+      alternate_style: true
+plugins:
+  - macros
diff --git a/open/MLCommons/code/gptj-99/README.md b/open/MLCommons/code/gptj-99/README.md
@@ -0,0 +1 @@
+TBD
diff --git a/...s/measurements/48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base/README.md b/...s/measurements/48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base/README.md
@@ -0,0 +1,3 @@
+| Model               | Scenario   | Accuracy              |   Throughput | Latency (in ms)   |
+|---------------------|------------|-----------------------|--------------|-------------------|
+| stable-diffusion-xl | offline    | (15.23624, 238.78369) |        0.374 | -                 |
diff --git a/...e/stable-diffusion-xl/offline/48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base.json b/...e/stable-diffusion-xl/offline/48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base.json
@@ -0,0 +1,7 @@
+{
+  "starting_weights_filename": "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0",
+  "retraining": "no",
+  "input_data_types": "fp32",
+  "weight_data_types": "fp32",
+  "weight_transformations": "no"
+}
diff --git a/...reference-gpu-pytorch_v2.1.0a0-scc24-base/stable-diffusion-xl/offline/README.md b/...reference-gpu-pytorch_v2.1.0a0-scc24-base/stable-diffusion-xl/offline/README.md
@@ -0,0 +1,57 @@
+This experiment is generated using the [MLCommons Collective Mind automation framework (CM)](https://github.com/mlcommons/cm4mlops).
+
+*Check [CM MLPerf docs](https://docs.mlcommons.org/inference) for more details.*
+
+## Host platform
+
+* OS version: Linux-6.2.0-39-generic-x86_64-with-glibc2.29
+* CPU version: x86_64
+* Python version: 3.8.10 (default, Sep 11 2024, 16:02:53) 
+[GCC 9.4.0]
+* MLCommons CM version: 2.3.6
+
+## CM Run Command
+
+See [CM installation guide](https://docs.mlcommons.org/inference/install/).
+
+```bash
+pip install -U cmind
+
+cm rm cache -f
+
+cm pull repo mlcommons@cm4mlops --checkout=735b581a345d5a04f8cd7195f6fbff3a37c538d8
+
+cm run script \
+	--tags=run-mlperf,inference,_r4.1-dev,_short,_scc24-base \
+	--model=sdxl \
+	--implementation=reference \
+	--framework=pytorch \
+	--category=datacenter \
+	--scenario=Offline \
+	--execution_mode=test \
+	--device=cuda \
+	--quiet \
+	--precision=float16
+```
+*Note that if you want to use the [latest automation recipes](https://docs.mlcommons.org/inference) for MLPerf (CM scripts),
+ you should simply reload mlcommons@cm4mlops without checkout and clean CM cache as follows:*
+
+```bash
+cm rm repo mlcommons@cm4mlops
+cm pull repo mlcommons@cm4mlops
+cm rm cache -f
+
+```
+
+## Results
+
+Platform: 48ed6105bd85-reference-gpu-pytorch_v2.1.0a0-scc24-base
+
+Model Precision: fp32
+
+### Accuracy Results 
+`CLIP_SCORE`: `15.23624`, Required accuracy for closed division `>= 31.68632` and `<= 31.81332`
+`FID_SCORE`: `238.78369`, Required accuracy for closed division `>= 23.01086` and `<= 23.95008`
+
+### Performance Results 
+`Samples per second`: `0.373636`
diff --git a/...eference-gpu-pytorch_v2.1.0a0-scc24-base/stable-diffusion-xl/offline/accuracy_console.out b/...eference-gpu-pytorch_v2.1.0a0-scc24-base/stable-diffusion-xl/offline/accuracy_console.out