mlcommons · gfursin · Aug 8, 2023 · Aug 4, 2023 · Aug 4, 2023 · Aug 4, 2023
@@ -14,27 +14,34 @@
 We deeply believe in the power of open science and open source to solve the world's most challenging problems.
 
 Following our [tedious experience reproducing 150 research papers and validating them in the real world](https://learning.acm.org/techtalks/reproducibility),
-we started developing this open-source Collective Knowledge technology to provide a common interface to access and reuse
-all shared knowledge (research projects, experiments, AI/ML models, code and data), facilitate reproducible research, 
-and simplify transfer to production across rapidly evolving models, software, hardware and data 
-as described in our [ACM REP'23 keynote](https://doi.org/10.5281/zenodo.8105339).
-
-Collective Knowledge project consists of the following sub-projects:
-* Non-intrusive and technology-agnostic [Collective Mind automation language (CM)](cm) to bridge the growing gap between research and production. It is extended by the community via
+we started developing Collective Knowledge technology to provide a common interface to access and reuse
+any shared knowledge (research projects, experiments, AI/ML models, code, data and automation scripts), 
+facilitate reproducible research, and simplify transfer to production across rapidly evolving models, software, hardware and data.
+
+Collective Knowledge technology consists of the following sub-projects:
+* Non-intrusive and technology-agnostic [Collective Mind automation language (CM)](https://doi.org/10.5281/zenodo.8105339) 
+  helps to convert any project, code or data
+  into a collection of portable and reusable components 
+  with a unified interaface. 
+  CM language allows users to run various automation workflows and applications in the same way either inside automatically generated container snapshots
+  or the latest software/hardware stacks (that may fail and then collaboratively improved by the community).
+  CM is intended to be always backward compatible and continously extended by the community via Python automation recipes and JSON/YAML meta descriptions:
   * [Common CM automations](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation) 
   * [Portable and reusable CM scripts](https://github.com/mlcommons/ck/tree/master/cm-mlops/script)
-* [Collective Knowledge Platform (CK Playground)](https://access.cKnowledge.org) to provide a user-friendly GUI 
-  to help the community explore, reproduce and reuse the state-of-the-art AI, ML and Systems research.
+  CM provides the same interface 
+* [Collective Knowledge Platform (CK Playground)](https://access.cKnowledge.org) provides a user-friendly GUI 
+  to help the community explore, reproduce, understand, optimize and reuse the state-of-the-art AI/ML Systems.
+
 
-The first practical use case for CM language and CK platform is to let everyone from an expert to a child participate in collaborative benchmarking,
+The first practical use case for CM language and CK platform is to let everyone participate in collaborative benchmarking,
 optimization and validation of the state-of-the-art AI/ML applications across rapidly evolving models, data, software and hardware 
 from different vendors - see our [reproducibility and optimization challenges](https://access.cknowledge.org/playground/?action=challenges), 
 [shared benchmarking and optimization results for ML Systems (performance, accuracy, power consumption, costs)](https://access.cknowledge.org/playground/?action=experiments) 
 and the [leaderboard](https://access.cknowledge.org/playground/?action=contributors).
 
 Read [our documentation](docs/README.md) to learn about how our open-source technology can help you.
 
-Join our [Discord channel](https://discord.gg/JjWNWXKxwT) to ask questions, provide feedback and participate in collaborative developments.
+Join our [Discord server](https://discord.gg/JjWNWXKxwT) to ask questions, provide feedback and participate in collaborative developments.
 
 ### Copyright
 

@@ -1244,6 +1244,10 @@ def docker(i):
 
     ii['out'] = None
 
+    image_repo = i.get('image_repo','')
+    if image_repo == '':
+        image_repo = 'cknowledge'
+
     # Search for automations in repos
     lst = []
     for repo in list_of_repos:
@@ -1370,28 +1374,29 @@ def docker(i):
         all_gpus = i.get('docker_all_gpus', docker_settings.get('all_gpus'))
 
         cm_docker_input = {'action': 'run',
-                            'automation': 'script',
-                            'tags': 'run,docker,container',
-                            'recreate': 'yes',
-                            'docker_os': _os,
-                            'cm_repo': cm_repo,
-                            'env': env,
-                            'image_repo': 'cm',
-                            'mounts': mounts,
-                            'image_tag': script_alias,
-                            'docker_os_version': version,
-                            'detached': 'no',
-                            'script_tags': f'{tag_string}',
-                            'run_cmd': run_cmd if docker_skip_run_cmd not in [ 'yes', True, 'True' ] else 'echo "cm version"',
-                            'v': i.get('v', False),
-                            'quiet': True,
-                            'pre_run_cmds': docker_pre_run_cmds,
-                            'real_run': True,
-                            'add_deps_recursive': {
-                                'build-docker-image': {
-                                    'dockerfile': dockerfile_path
-                                }
-                            }
+                           'automation': 'script',
+                           'tags': 'run,docker,container',
+                           'recreate': 'yes',
+                           'docker_os': _os,
+                           'cm_repo': cm_repo,
+                           'env': env,
+                           'image_repo': image_repo,
+                           'mounts': mounts,
+                           'image_name': 'cm-script-'+script_alias,
+#                            'image_tag': script_alias,
+                           'docker_os_version': version,
+                           'detached': 'no',
+                           'script_tags': f'{tag_string}',
+                           'run_cmd': run_cmd if docker_skip_run_cmd not in [ 'yes', True, 'True' ] else 'echo "cm version"',
+                           'v': i.get('v', False),
+                           'quiet': True,
+                           'pre_run_cmds': docker_pre_run_cmds,
+                           'real_run': True,
+                           'add_deps_recursive': {
+                               'build-docker-image': {
+                                   'dockerfile': dockerfile_path
+                               }
+                           }
                         }
 
         if all_gpus:

@@ -23,8 +23,10 @@ export CK_BATCH_COUNT=${CM_BATCH_COUNT}
 
 if [[ "${CM_INPUT}" != "" ]]; then export CM_IMAGE=${CM_INPUT}; fi
 
+PIP_EXTRA=`${CM_PYTHON_BIN} -c "import pkg_resources; print(' --break-system-packages ' if int(pkg_resources.get_distribution('pip').version.split('.')[0]) >= 23 else '')"`
+
 echo ""
-${CM_PYTHON_BIN} -m pip install -r ${CM_TMP_CURRENT_SCRIPT_PATH}/requirements.txt
+${CM_PYTHON_BIN} -m pip install -r ${CM_TMP_CURRENT_SCRIPT_PATH}/requirements.txt ${PIP_EXTRA}
 test $? -eq 0 || exit 1
 
 echo ""

@@ -145,9 +145,10 @@ class Backend {
     void UnloadSampleFromRam(mlperf::QuerySampleIndex sample_index) {
         for (size_t i = 0; i < num_inputs; i++)
             sample_memory_size[i] -= GetSampleSize(sample_index, i);
-            if (sample_map.find(sample_index) != sample_map.end()) {
+           /* if (sample_map.find(sample_index) != sample_map.end()) {
 		sample_map.erase(sample_index);
-	    }
+	    }*/
+	//Temporarily commenting out above until segfault issue is fixed
         batches.children.erase(sample_index);
         num_samples_in_memory--;
     }

@@ -25,6 +25,7 @@ class InputSettings {
     InputSettings() {
         mlperf_conf_path = getenv("CM_MLPERF_CONF", "../inference/mlperf.conf");
         user_conf_path = getenv("CM_MLPERF_USER_CONF", "../inference/vision/classification_and_detection/user.conf");
+        audit_conf_path = getenv("CM_MLPERF_INFERENCE_AUDIT_PATH", "");
         output_dir = getenv("CM_MLPERF_OUTPUT_DIR", ".");
         backend_name = getenv("CM_MLPERF_BACKEND", "onnxruntime");
         device_name = getenv("CM_MLPERF_DEVICE", "cpu");
@@ -54,6 +55,7 @@ class InputSettings {
 
     std::string mlperf_conf_path;
     std::string user_conf_path;
+    std::string audit_conf_path;
     std::string output_dir;
     std::string backend_name;
     std::string device_name;
@@ -202,5 +204,5 @@ int main(int argc, const char *argv[]) {
 
     // start benchmark
     std::cerr << "starting benchmark" << std::endl;
-    mlperf::StartTest(sut.get(), qsl.get(), test_settings, log_settings);
+    mlperf::StartTest(sut.get(), qsl.get(), test_settings, log_settings, input_settings.audit_conf_path);
 }
@@ -297,6 +297,7 @@ void TestSingleStream(Program *prg) {
 
   const std::string mlperf_conf_path = getenv_s("CM_MLPERF_CONF");
   const std::string user_conf_path = getenv_s("CM_MLPERF_USER_CONF");
+  const std::string audit_conf_path = getenv_opt_s("CM_MLPERF_INFERENCE_AUDIT_PATH","");
 
   std::string model_name = getenv_opt_s("CM_MODEL", "unknown_model");
   std::string logs_dir = getenv_opt_s("CM_MLPERF_LOADGEN_LOGS_DIR", "");
@@ -344,7 +345,7 @@ void TestSingleStream(Program *prg) {
     prg->ColdRun();
   }
 
-  mlperf::StartTest(&sut, &qsl, ts, log_settings);
+  mlperf::StartTest(&sut, &qsl, ts, log_settings, audit_conf_path);
 }
 
 int main(int argc, char* argv[]) {

@@ -9,10 +9,6 @@ def preprocess(i):
 
     env = i['env']
 
-    if env.get('CM_MLPERF_SUBMISSION_GENERATION_STYLE', '') == "short":
-        if env.get('CM_MODEL', '') == "resnet50":
-            env['CM_TEST_QUERY_COUNT'] = "500" #so that accuracy script doesn't complain
-
     if env.get('CM_MLPERF_IMPLEMENTATION', '') == 'nvidia-original':
         if env.get('CM_NVIDIA_GPU_NAME', '') in [ "rtx_4090", "a100", "t4", "l4", "orin", "custom" ]:
             env['CM_NVIDIA_HARNESS_GPU_VARIATION'] = "_" + env['CM_NVIDIA_GPU_NAME']
@@ -89,7 +85,8 @@ def postprocess(i):
 
     scenario = env['CM_MLPERF_LOADGEN_SCENARIO']
 
-    if env.get("CM_MLPERF_FIND_PERFORMANCE_MODE", '') == "yes" and mode == "performance" and scenario != "Server":
+    #if env.get("CM_MLPERF_FIND_PERFORMANCE_MODE", '') == "yes" and mode == "performance" and scenario != "Server":
+    if mode == "performance" and scenario != "Server":
         os.chdir(output_dir)
         if not os.path.exists("mlperf_log_summary.txt"):
             return {'return': 0}
@@ -131,7 +128,7 @@ def postprocess(i):
             yaml.dump(sut_config, f)
 
 
-    elif mode in [ "performance", "accuracy" ]:
+    if mode in [ "performance", "accuracy" ]:
         measurements = {}
         measurements['starting_weights_filename'] = env.get('CM_ML_MODEL_STARTING_WEIGHTS_FILENAME', env.get('CM_ML_MODEL_FILE', ''))
         measurements['retraining'] = env.get('CM_ML_MODEL_RETRAINING','no')

@@ -31,10 +31,15 @@ def preprocess(i):
     if "CM_DOCKER_IMAGE_REPO" not in env:
         env['CM_DOCKER_IMAGE_REPO'] = "local"
 
-    if "CM_DOCKER_IMAGE_NAME" not in env:
-        env['CM_DOCKER_IMAGE_NAME'] = "cm"
+    docker_image_name = env.get('CM_DOCKER_IMAGE_NAME', '')
+    if docker_image_name == '':
+        docker_image_name = env.get('CM_DOCKER_RUN_SCRIPT_TAGS','').replace(',', '-').replace('_','')
+    if docker_image_name == '':
+        docker_image_name = 'cm'
 
-    if "CM_DOCKER_IMAGE_TAG" not in env:
+    env['CM_DOCKER_IMAGE_NAME'] = docker_image_name
+
+    if env.get("CM_DOCKER_IMAGE_TAG", "") == '':
         env['CM_DOCKER_IMAGE_TAG'] = "latest"
 
     if env.get("CM_DOCKER_CACHE", "yes") == "no":

@@ -0,0 +1,21 @@
+set DOCKER_IMAGE_REPO=cknowledge
+
+set DOCKER_OS=ubuntu
+
+rem set DOCKER_OS_VER=22.04
+set DOCKER_OS_VER=23.04
+set DOCKER_PIP_EXTRA_FLAGS=--break-system-packages
+
+set DOCKER_IMAGE_NAME=cm-base
+rem set DOCKER_IMAGE_NAME=cm-script-app-image-classification-onnx-py
+rem set DOCKER_IMAGE_POST_FILE=%CD%\extra-cmd.cm-script-app-image-classification-onnx-py
+
+rem set DOCKER_IMAGE_TAG=%DOCKER_OS%-%DOCKER_OS_VER%-20230804
+
+set DOCKER_IMAGE_TAG=%DOCKER_OS%-%DOCKER_OS_VER%-latest
+set DOCKERFILE_EXT=%DOCKER_IMAGE_NAME%-%DOCKER_IMAGE_TAG%
+
+set DOCKER_PACKAGE_MANAGER_UPDATE_CMD="apt-get update -y && apt-get upgrade -y"
+
+set DOCKER_CM_MLOPS_REPO="ctuning@mlcommons-ck"
+rem set DOCKER_CM_MLOPS_REPO="mlcommons@ck"
@@ -0,0 +1,9 @@
+call 0-common.bat
+
+cmr "build dockerfile" --file_path=%CD%\Dockerfile.%DOCKERFILE_EXT% ^
+                       --docker_os=%DOCKER_OS% ^
+                       --docker_os_version=%DOCKER_OS_VER% ^
+                       --package_manager_update_cmd=%DOCKER_PACKAGE_MANAGER_UPDATE_CMD% ^
+                       --pip_extra_flags=%DOCKER_PIP_EXTRA_FLAGS% ^
+                       --post_file=%DOCKER_IMAGE_POST_FILE% ^
+                       --cm_repo=%DOCKER_CM_MLOPS_REPO%
@@ -0,0 +1,8 @@
+call 0-common.bat
+
+cmr "build docker image" --dockerfile=%CD%\Dockerfile.%DOCKERFILE_EXT% ^
+                         --docker_os=%DOCKER_OS% ^
+                         --docker_os_version=%DOCKER_OS_VER% ^
+                         --image_repo=%DOCKER_IMAGE_REPO% ^
+                         --image_name=%DOCKER_IMAGE_NAME% ^
+                         --image_tag=%DOCKER_IMAGE_TAG%
@@ -0,0 +1,3 @@
+call 0-common.bat
+
+docker run -it %DOCKER_IMAGE_REPO%/%DOCKER_IMAGE_NAME%:%DOCKER_IMAGE_TAG% "cmr 'detect os' -j"
@@ -0,0 +1,3 @@
+call 0-common.bat
+
+cmr "run docker container" --image_repo=%DOCKER_IMAGE_REPO% --image_name=%DOCKER_IMAGE_NAME% --image_tag=%DOCKER_IMAGE_TAG%  --run_cmd="cmr 'detect os' -j"
@@ -0,0 +1,3 @@
+call 0-common.bat
+
+cmr "run docker container" --image_repo=%DOCKER_IMAGE_REPO% --image_tag=%DOCKER_IMAGE_TAG% --script_tags=detect,os
@@ -0,0 +1,3 @@
+call 0-common.bat
+
+cmr "run docker container" --image_repo=%DOCKER_IMAGE_REPO% --image_tag=%DOCKER_IMAGE_TAG% --script_tags=detect,os --it
@@ -0,0 +1,3 @@
+call 0-common.bat
+
+cm docker script --tags=detect,os -j
@@ -0,0 +1,3 @@
+call 0-common.bat
+
+docker run -it %DOCKER_IMAGE_REPO%/%DOCKER_IMAGE_NAME%:%DOCKER_IMAGE_TAG% -c bash
@@ -0,0 +1,3 @@
+call 0-common.bat
+
+cmr "run docker container" --image_repo=%DOCKER_IMAGE_REPO% --image_name=%DOCKER_IMAGE_NAME% --image_tag=%DOCKER_IMAGE_TAG% --it
@@ -0,0 +1,3 @@
+call 0-common.bat
+
+docker push %DOCKER_IMAGE_REPO%/%DOCKER_IMAGE_NAME%:%DOCKER_IMAGE_TAG%
@@ -1,21 +1,24 @@
 FROM ubuntu:22.04
 
-SHELL ["/bin/bash", "-c"]
-ENTRYPOINT ["/bin/bash", "-c"]
+# Maintained by the MLCommons taskforce on automation and reproducibility
+LABEL github="https://github.com/mlcommons/ck"
+LABEL maintainer="https://cKnowledge.org/mlcommons-taskforce"
 
+SHELL ["/bin/bash", "-c"]
 ARG CM_GH_TOKEN
 
 # Notes: https://runnable.com/blog/9-common-dockerfile-mistakes
 # Install system dependencies
-RUN apt-get update -y
+RUN apt-get update -y && apt-get upgrade -y
 RUN apt-get install -y python3 python3-pip git sudo wget
 
 # Install python packages
-RUN python3 -m pip install cmind requests
+RUN python3 -m pip install cmind requests  
 
 # Setup docker environment
 ENTRYPOINT ["/bin/bash", "-c"]
-ENV TZ=US/Pacific
+ENV TZ="US/Pacific"
+ENV PATH="${PATH}:/home/cmuser/.local/bin"
 RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ >/etc/timezone
 
 # Setup docker user
@@ -26,10 +29,10 @@ USER cmuser:cm
 WORKDIR /home/cmuser
 
 # Download CM repo for scripts
-RUN cm pull repo mlcommons@ck
+RUN cm pull repo mlcommons@ck --dummy
 
 # Install all system dependencies
 RUN cm run script --quiet --tags=get,sys-utils-cm
 
-# Get CM version
-CMD cm version
+# Run commands
+RUN cm version
@@ -1,21 +1,24 @@
-FROM ubuntu:22.04
+FROM ubuntu:23.04
 
-SHELL ["/bin/bash", "-c"]
-ENTRYPOINT ["/bin/bash", "-c"]
+# Maintained by the MLCommons taskforce on automation and reproducibility
+LABEL github="https://github.com/mlcommons/ck"
+LABEL maintainer="https://cKnowledge.org/mlcommons-taskforce"
 
+SHELL ["/bin/bash", "-c"]
 ARG CM_GH_TOKEN
 
 # Notes: https://runnable.com/blog/9-common-dockerfile-mistakes
 # Install system dependencies
-RUN apt-get update -y
+RUN apt-get update -y && apt-get upgrade -y
 RUN apt-get install -y python3 python3-pip git sudo wget
 
 # Install python packages
-RUN python3 -m pip install cmind requests
+RUN python3 -m pip install cmind requests --break-system-packages 
 
 # Setup docker environment
 ENTRYPOINT ["/bin/bash", "-c"]
-ENV TZ=US/Pacific
+ENV TZ="US/Pacific"
+ENV PATH="${PATH}:/home/cmuser/.local/bin"
 RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ >/etc/timezone
 
 # Setup docker user
@@ -26,13 +29,10 @@ USER cmuser:cm
 WORKDIR /home/cmuser
 
 # Download CM repo for scripts
-RUN cm pull repo mlcommons@ck
+RUN cm pull repo mlcommons@ck --dummy
 
 # Install all system dependencies
 RUN cm run script --quiet --tags=get,sys-utils-cm
 
-# Run image classification and install all related CM components automatically
-RUN cm run script --tags=python,app,image-classification,onnx --quiet
-
-# Start workflow
-CMD ["cm run script --tags=python,app,image-classification,onnx"]
+# Run commands
+RUN cm version