diff --git a/README.md b/README.md index 5b9f764ba7..7cbf3b65e2 100755 --- a/README.md +++ b/README.md @@ -23,7 +23,10 @@ from [MLCommons projects](https://mlcommons.org) and [research papers](https://c in a unified way on any operating system with any software and hardware either natively or inside containers. -Here are some most commonly used examples from the community: +Here are a few most commonly used examples from the CM users +that should run in the same way on Linux, MacOS, Windows and other platforms +(see [Gettings Started Guide](docs/getting-started.md) to understand +how they work and how to reuse them in your projects): ```bash pip install cmind @@ -31,32 +34,36 @@ pip install cmind cm pull repo mlcommons@ck cm run script "python app image-classification onnx" +cmr "python app image-classification onnx" -cm run script "download file _wget" --url=https://cKnowledge.org/ai/data/computer_mouse.jpg --verify=no --env.CM_DOWNLOAD_CHECKSUM=45ae5c940233892c2f860efdf0b66e7e - -cm run script "python app image-classification onnx" --input=computer_mouse.jpg +cmr "download file _wget" --url=https://cKnowledge.org/ai/data/computer_mouse.jpg --verify=no --env.CM_DOWNLOAD_CHECKSUM=45ae5c940233892c2f860efdf0b66e7e cmr "python app image-classification onnx" --input=computer_mouse.jpg -cm show cache -cm rm cache -f - -cmr "python app image-classification onnx _cuda" --input=computer_mouse.jpg - -cmr "cm gui" --script="python app image-classification onnx" +cm find script "python app image-classification onnx" -cm docker script "python app image-classification onnx" --input=computer_mouse.jpg -cm docker script "python app image-classification onnx" --input=computer_mouse.jpg -j -docker_it -cm docker script "python app image-classification onnx" --input=computer_mouse.jpg -j -docker_it +cmr "get python" --version_min=3.8.0 --name=mlperf-experiments +cmr "install python-venv" --version_max=3.10.11 --name=mlperf -cmr "get generic-python-lib _package.onnxruntime" -cmr "get coco dataset _val _2014" cmr "get ml-model stable-diffusion" cmr "get ml-model huggingface zoo _model-stub.alpindale/Llama-2-13b-ONNX" --model_filename=FP32/LlamaV2_13B_float32.onnx --skip_cache +cmr "get coco dataset _val _2014" cm show cache cm show cache "get ml-model stable-diffusion" +cmr "get generic-python-lib _package.onnxruntime" --version_min=1.16.0 +cmr "python app image-classification onnx" --input=computer_mouse.jpg + cm rm cache -f +cmr "python app image-classification onnx" --input=computer_mouse.jpg --adr.onnxruntime.version_max=1.16.0 + + +cmr "python app image-classification onnx _cuda" --input=computer_mouse.jpg + +cmr "cm gui" --script="python app image-classification onnx" + +cm docker script "python app image-classification onnx" --input=computer_mouse.jpg +cm docker script "python app image-classification onnx" --input=computer_mouse.jpg -j -docker_it cmr "run common mlperf inference" --implementation=nvidia --model=bert-99 --category=datacenter --division=closed cm find script "run common mlperf inference" @@ -65,8 +72,6 @@ cm pull repo ctuning@cm-reproduce-research-projects cmr "reproduce paper micro-2023 victima _install_deps" cmr "reproduce paper micro-2023 victima _run" -... - ``` ```python @@ -78,8 +83,9 @@ if output['return']==0: print (output) ``` -Collective Mind is a community project being developed by the [MLCommons Task Force on Automation and Reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md) -with great help from [MLCommons (70+ AI organizations)](https://mlcommons.org/), +Collective Mind is a community project being developed by the +[MLCommons Task Force on Automation and Reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md) +with great help from [MLCommons (70+ AI organizations)](https://mlcommons.org/, [research community]( https://www.youtube.com/watch?v=7zpeIVwICa4 ) and [individual contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) - we want to have a simple, non-intrusive, technology-agnostic, portable and easily-extensible interface @@ -89,7 +95,7 @@ running experiments, processing logs, and reproducing results without thinking where and how they run. That is why we implemented CM as a [small Python library](https://github.com/mlcommons/ck/tree/master/cm) -with minimal dependencies (Python 3.7+, git, wget), simple Python API, StreamLit GUI and human-friendly command line. +with minimal dependencies (Python 3.7+, git, wget), simple Python API, GUI and human-friendly command line. CM simply searches for CM scripts by tags or Unique IDs in all pulled Git repositories, automatically generates command lines for a given script or tool on a given platform, updates all paths and environment variables, runs a given automation either natively or inside automatically-generated containers @@ -115,7 +121,7 @@ or have questions and suggestions. ### Documentation -* [Getting Started tutorial](docs/getting-started.md) +* [Getting Started Guide](docs/getting-started.md) * [CM interface for MLPerf benchmarks](docs/mlperf) * [CM interface for ML and Systems conferences](docs/tutorials/common-interface-to-reproduce-research-projects.md) * [CM automation recipes for MLOps and DevOps](cm-mlops/script) diff --git a/cm-mlops/automation/script/module.py b/cm-mlops/automation/script/module.py index d67eb0cdec..4f81ece1ad 100644 --- a/cm-mlops/automation/script/module.py +++ b/cm-mlops/automation/script/module.py @@ -1704,6 +1704,14 @@ def _update_variation_tags_from_variations(self, variation_tags, variations, var v_static = self._get_name_for_dynamic_variation_tag(v) tmp_variation_tags_static[v_i] = v_static + combined_variations = [ t for t in variations if ',' in t ] + # We support default_variations in the meta of cmbined_variations + combined_variations.sort(key=lambda x: x.count(',')) + ''' By sorting based on the number of variations users can safely override + env and state in a larger combined variation + ''' + tmp_combined_variations = {k: False for k in combined_variations} + # Recursively add any base variations specified if len(variation_tags) > 0: tmp_variations = {k: False for k in variation_tags} @@ -1748,28 +1756,34 @@ def _update_variation_tags_from_variations(self, variation_tags, variations, var tag_to_append = None # default_variations dictionary specifies the default_variation for each variation group. A default variation in a group is turned on if no other variation from that group is turned on and it is not excluded using the '-' prefix - if "default_variations" in variations[variation_name]: - default_base_variations = variations[variation_name]["default_variations"] - for default_base_variation in default_base_variations: - tag_to_append = None + r = self._get_variation_tags_from_default_variations(variations[variation_name], variations, variation_groups, tmp_variation_tags_static, excluded_variation_tags) + if r['return'] > 0: + return r - if default_base_variation not in variation_groups: - return {'return': 1, 'error': 'Default variation "{}" is not a valid group. Valid groups are "{}" '.format(default_base_variation, variation_groups)} + variations_to_add = r['variations_to_add'] + for t in variations_to_add: + tmp_variations[t] = False + variation_tags.append(t) - unique_allowed_variations = variation_groups[default_base_variation]['variations'] - # add the default only if none of the variations from the current group is selected and it is not being excluded with - prefix - if len(set(unique_allowed_variations) & set(tmp_variation_tags_static)) == 0 and default_base_variations[default_base_variation] not in excluded_variation_tags and default_base_variations[default_base_variation] not in tmp_variation_tags_static: - tag_to_append = default_base_variations[default_base_variation] + tmp_variations[variation_name] = True - if tag_to_append: - if tag_to_append not in variations: - variation_tag_static = self._get_name_for_dynamic_variation_tag(tag_to_append) - if not variation_tag_static or variation_tag_static not in variations: - return {'return': 1, 'error': 'Invalid variation "{}" specified in default variations for the variation "{}" '.format(tag_to_append, variation_name)} - variation_tags.append(tag_to_append) - tmp_variations[tag_to_append] = False + for combined_variation in combined_variations: + if tmp_combined_variations[combined_variation]: + continue + v = combined_variation.split(",") + all_present = set(v).issubset(set(variation_tags)) + if all_present: + combined_variation_meta = variations[combined_variation] + tmp_combined_variations[combined_variation] = True - tmp_variations[variation_name] = True + r = self._get_variation_tags_from_default_variations(combined_variation_meta, variations, variation_groups, tmp_variation_tags_static, excluded_variation_tags) + if r['return'] > 0: + return r + + variations_to_add = r['variations_to_add'] + for t in variations_to_add: + tmp_variations[t] = False + variation_tags.append(t) all_base_processed = True for variation_name in variation_tags: @@ -1785,6 +1799,31 @@ def _update_variation_tags_from_variations(self, variation_tags, variations, var return {'return': 0} + def _get_variation_tags_from_default_variations(self, variation_meta, variations, variation_groups, tmp_variation_tags_static, excluded_variation_tags): + # default_variations dictionary specifies the default_variation for each variation group. A default variation in a group is turned on if no other variation from that group is turned on and it is not excluded using the '-' prefix + + tmp_variation_tags = [] + if "default_variations" in variation_meta: + default_base_variations = variation_meta["default_variations"] + for default_base_variation in default_base_variations: + tag_to_append = None + + if default_base_variation not in variation_groups: + return {'return': 1, 'error': 'Default variation "{}" is not a valid group. Valid groups are "{}" '.format(default_base_variation, variation_groups)} + + unique_allowed_variations = variation_groups[default_base_variation]['variations'] + # add the default only if none of the variations from the current group is selected and it is not being excluded with - prefix + if len(set(unique_allowed_variations) & set(tmp_variation_tags_static)) == 0 and default_base_variations[default_base_variation] not in excluded_variation_tags and default_base_variations[default_base_variation] not in tmp_variation_tags_static: + tag_to_append = default_base_variations[default_base_variation] + + if tag_to_append: + if tag_to_append not in variations: + variation_tag_static = self._get_name_for_dynamic_variation_tag(tag_to_append) + if not variation_tag_static or variation_tag_static not in variations: + return {'return': 1, 'error': 'Invalid variation "{}" specified in default variations for the variation "{}" '.format(tag_to_append, variation_name)} + tmp_variation_tags.append(tag_to_append) + + return {'return': 0, 'variations_to_add': tmp_variation_tags} ############################################################ def version(self, i): @@ -1824,6 +1863,15 @@ def search(self, i): console = i.get('out') == 'con' + # Check simplified CMD: cm run script "get compiler" + # If artifact has spaces, treat them as tags! + artifact = i.get('artifact','') + if ' ' in artifact: # or ',' in artifact: + del(i['artifact']) + if 'parsed_artifact' in i: del(i['parsed_artifact']) + # Force substitute tags + i['tags']=artifact.replace(' ',',') + ############################################################################################################ # Process tags to find script(s) and separate variations # (not needed to find scripts) @@ -2125,6 +2173,16 @@ def add(self, i): tags_list = utils.convert_tags_to_list(i) if 'tags' in i: del(i['tags']) + if len(tags_list)==0: + if console: + x=input('Please specify a combination of unique tags separated by comma for this script: ') + x = x.strip() + if x!='': + tags_list = x.split(',') + + if len(tags_list)==0: + return {'return':1, 'error':'you must specify a combination of unique tags separate by comman using "--new_tags"'} + # Add placeholder (use common action) ii['out']='con' ii['common']=True # Avoid recursion - use internal CM add function to add the script artifact @@ -2150,18 +2208,19 @@ def add(self, i): # Check if preloaded meta exists meta = { - 'cache':False, - 'new_env_keys':[], - 'new_state_keys':[], - 'input_mapping':{}, - 'docker_input_mapping':{}, - 'deps':[], - 'prehook_deps':[], - 'posthook_deps':[], - 'post_deps':[], - 'versions':{}, - 'variations':{}, - 'input_description':{} + 'cache':False +# 20240127: Grigori commented that because newly created script meta looks ugly +# 'new_env_keys':[], +# 'new_state_keys':[], +# 'input_mapping':{}, +# 'docker_input_mapping':{}, +# 'deps':[], +# 'prehook_deps':[], +# 'posthook_deps':[], +# 'post_deps':[], +# 'versions':{}, +# 'variations':{}, +# 'input_description':{} } fmeta = os.path.join(template_path, self.cmind.cfg['file_cmeta']) @@ -2200,6 +2259,7 @@ def add(self, i): ii['yaml']=True ii['automation']='script,5b4e0237da074764' + r_obj=self.cmind.access(ii) if r_obj['return']>0: return r_obj diff --git a/cm-mlops/automation/script/module_misc.py b/cm-mlops/automation/script/module_misc.py index a23502ea6c..a0e55b8408 100644 --- a/cm-mlops/automation/script/module_misc.py +++ b/cm-mlops/automation/script/module_misc.py @@ -1384,6 +1384,8 @@ def dockerfile(i): fake_run_deps = i.get('fake_run_deps', docker_settings.get('fake_run_deps', False)) docker_run_final_cmds = docker_settings.get('docker_run_final_cmds', []) + gh_token = i.get('docker_gh_token') + if i.get('docker_real_run', False): fake_run_option = " " fake_run_deps = False @@ -1436,6 +1438,9 @@ def dockerfile(i): 'real_run': True } + if gh_token: + cm_docker_input['gh_token'] = gh_token + r = self_module.cmind.access(cm_docker_input) if r['return'] > 0: return r @@ -1646,8 +1651,12 @@ def docker(i): all_gpus = i.get('docker_all_gpus', docker_settings.get('all_gpus')) + device = i.get('docker_device', docker_settings.get('device')) + + gh_token = i.get('docker_gh_token') + + port_maps = i.get('docker_port_maps', docker_settings.get('port_maps', [])) - # # Regenerate run_cmd # if i.get('cmd'): @@ -1678,7 +1687,6 @@ def docker(i): print (run_cmd) print ('') - cm_docker_input = {'action': 'run', 'automation': 'script', 'tags': 'run,docker,container', @@ -1709,6 +1717,15 @@ def docker(i): if all_gpus: cm_docker_input['all_gpus'] = True + if device: + cm_docker_input['device'] = device + + if gh_token: + cm_docker_input['gh_token'] = gh_token + + if port_maps: + cm_docker_input['port_maps'] = port_maps + print ('') diff --git a/cm-mlops/script/app-image-classification-onnx-py/README-extra.md b/cm-mlops/script/app-image-classification-onnx-py/README-extra.md index 053a4894fa..e379e2544e 100644 --- a/cm-mlops/script/app-image-classification-onnx-py/README-extra.md +++ b/cm-mlops/script/app-image-classification-onnx-py/README-extra.md @@ -4,13 +4,14 @@ See [this tutorial](https://github.com/mlcommons/ck/blob/master/docs/tutorials/m # Collaborative testing +## Windows 11 + +* CUDA 11.8; cuDNN 8.7.0; ONNX GPU 1.16.1 + ## Windows 10 -* ONNX Runtime 1.13.1 with CPU and CUDA -* CUDA 11.6 -* cuDNN 8.5.0.96 +* CUDA 11.6; cuDNN 8.6.0.96; ONNX GPU 1.13.1 ## Ubuntu 22.04 -* ONNX Runtime 1.12.0 with CPU and CUDA -* CUDA 11.3 +* CUDA 11.3; ONNX 1.12.0 diff --git a/cm-mlops/script/app-image-classification-onnx-py/_cm.yaml b/cm-mlops/script/app-image-classification-onnx-py/_cm.yaml index 6c07135d7d..c3f869081e 100644 --- a/cm-mlops/script/app-image-classification-onnx-py/_cm.yaml +++ b/cm-mlops/script/app-image-classification-onnx-py/_cm.yaml @@ -25,12 +25,20 @@ deps: - python - python3 tags: get,python3 + - tags: get,cuda names: - cuda enable_if_env: USE_CUDA: - yes +- tags: get,cudnn + names: + - cudnn + enable_if_env: + USE_CUDA: + - yes + - tags: get,dataset,imagenet,image-classification,original - tags: get,dataset-aux,imagenet-aux,image-classification - tags: get,ml-model,resnet50,_onnx,image-classification @@ -41,16 +49,20 @@ deps: - tags: get,generic-python-lib,_package.numpy - tags: get,generic-python-lib,_package.opencv-python + - tags: get,generic-python-lib,_onnxruntime + names: + - onnxruntime skip_if_env: USE_CUDA: - yes - tags: get,generic-python-lib,_onnxruntime_gpu + names: + - onnxruntime enable_if_env: USE_CUDA: - yes - variations: cuda: env: @@ -61,6 +73,7 @@ input_mapping: input: CM_IMAGE output: CM_APP_IMAGE_CLASSIFICATION_ONNX_PY_OUTPUT + new_env_keys: - CM_APP_IMAGE_CLASSIFICATION_ONNX_PY* diff --git a/cm-mlops/script/app-mlperf-inference-cpp/_cm.yaml b/cm-mlops/script/app-mlperf-inference-cpp/_cm.yaml index 83562e086f..80f28a0ff3 100644 --- a/cm-mlops/script/app-mlperf-inference-cpp/_cm.yaml +++ b/cm-mlops/script/app-mlperf-inference-cpp/_cm.yaml @@ -221,6 +221,7 @@ variations: CM_MLPERF_LOADGEN_MAX_BATCHSIZE: 8 batch-size.#: + group: batch-size env: CM_MLPERF_LOADGEN_MAX_BATCHSIZE: "#" @@ -242,3 +243,15 @@ variations: group: loadgen-scenario env: CM_MLPERF_LOADGEN_SCENARIO: Server + + multistream,resnet50: + default_variations: + batch-size: batch-size.8 + + offline,resnet50: + default_variations: + batch-size: batch-size.8 + + multistream,retinanet: + default_variations: + batch-size: batch-size.1 diff --git a/cm-mlops/script/app-mlperf-inference-cpp/src/main.cpp b/cm-mlops/script/app-mlperf-inference-cpp/src/main.cpp index beb0b72f20..c5a3c809ea 100644 --- a/cm-mlops/script/app-mlperf-inference-cpp/src/main.cpp +++ b/cm-mlops/script/app-mlperf-inference-cpp/src/main.cpp @@ -42,7 +42,8 @@ class InputSettings { if (mode_name == "performance") mode_name = "PerformanceOnly"; query_count_override = std::stol(getenv("CM_MLPERF_LOADGEN_QUERY_COUNT", "0")); - performance_sample_count = std::stol(getenv("CM_MLPERF_LOADGEN_PERFORMANCE_SAMPLE_COUNT", "1024")); + query_count_override = 0; + performance_sample_count = std::stol(getenv("CM_MLPERF_LOADGEN_PERFORMANCE_SAMPLE_COUNT", "0")); batch_size = std::stol(getenv("CM_MLPERF_LOADGEN_MAX_BATCHSIZE", "32")); std::cout << "MLPerf Conf path: " << mlperf_conf_path << std::endl; std::cout << "User Conf path: " << user_conf_path << std::endl; @@ -52,6 +53,7 @@ class InputSettings { std::cout << "Mode: " << mode_name << std::endl; std::cout << "Batch size: " << batch_size << std::endl; std::cout << "Query count override: " << query_count_override << std::endl; + std::cout << "Performance sample count override in application: " << performance_sample_count << std::endl; } std::string mlperf_conf_path; diff --git a/cm-mlops/script/app-mlperf-inference-reference/_cm.yaml b/cm-mlops/script/app-mlperf-inference-reference/_cm.yaml index a0f0690ae5..650bdffaff 100644 --- a/cm-mlops/script/app-mlperf-inference-reference/_cm.yaml +++ b/cm-mlops/script/app-mlperf-inference-reference/_cm.yaml @@ -840,6 +840,7 @@ variations: group: models env: CM_MODEL: stable-diffusion-xl + CM_NUM_THREADS: "1" deps: - tags: get,generic-python-lib,_package.diffusers names: diff --git a/cm-mlops/script/app-mlperf-inference/_cm.yaml b/cm-mlops/script/app-mlperf-inference/_cm.yaml index 118b521b1a..e517c90d21 100644 --- a/cm-mlops/script/app-mlperf-inference/_cm.yaml +++ b/cm-mlops/script/app-mlperf-inference/_cm.yaml @@ -131,6 +131,8 @@ variations: CM_SKIP_RUN: - yes + mil: + alias: cpp tflite-cpp: default_variations: @@ -383,6 +385,10 @@ variations: env: CM_MODEL: stable-diffusion-xl + default_variations: + precision: float16 + backend: pytorch + device: cuda add_deps_recursive: mlperf-inference-implementation: tags: _sdxl diff --git a/cm-mlops/script/app-mlperf-inference/customize.py b/cm-mlops/script/app-mlperf-inference/customize.py index bd5447cac9..21b95b67d2 100644 --- a/cm-mlops/script/app-mlperf-inference/customize.py +++ b/cm-mlops/script/app-mlperf-inference/customize.py @@ -90,8 +90,15 @@ def postprocess(i): accuracy_log_file_option_name = " --log_file " datatype_option = " --output_dtype "+env['CM_SQUAD_ACCURACY_DTYPE'] + elif 'stable-diffusion-xl' in model: + pass #No compliance check for now + elif 'gpt' in model: + pass #No compliance check for now + elif 'llama2-70b' in model: + pass #No compliance check for now else: - return {'return': 1, 'error': f'Accuracy paths not done for model {model}'} + pass # Not giving an error now. But accuracy paths need to be done for other benchmarks which may need the non-determinism test + #return {'return': 1, 'error': f'Accuracy paths not done for model {model}'} scenario = env['CM_MLPERF_LOADGEN_SCENARIO'] #if env.get("CM_MLPERF_FIND_PERFORMANCE_MODE", '') == "yes" and mode == "performance" and scenario != "Server": diff --git a/cm-mlops/script/benchmark-any-mlperf-inference-implementation/_cm.yaml b/cm-mlops/script/benchmark-any-mlperf-inference-implementation/_cm.yaml index f42b4067c6..020eadfec4 100644 --- a/cm-mlops/script/benchmark-any-mlperf-inference-implementation/_cm.yaml +++ b/cm-mlops/script/benchmark-any-mlperf-inference-implementation/_cm.yaml @@ -5,10 +5,19 @@ cache: false category: MLPerf benchmark support deps: - tags: detect,cpu - +default_env: + DIVISION: open + CATEGORY: edge docker_input_mapping: {} input_description: {} -input_mapping: {} +input_mapping: + models: MODELS + backends: BACKENDS + devices: DEVICES + division: DIVISION + category: CATEGORY + power_server: POWER_SERVER + power_server_port: POWER_SERVER_PORT new_env_keys: [] new_state_keys: [] post_deps: [] @@ -105,7 +114,7 @@ variations: env: POWER: yes default_env: - POWER_SERVER_IP: 192.168.0.15 + POWER_SERVER: 192.168.0.15 POWER_SERVER_PORT: 4950 @@ -141,5 +150,3 @@ variations: env: CATEGORY: edge DIVISION: closed - -versions: {} diff --git a/cm-mlops/script/benchmark-any-mlperf-inference-implementation/customize.py b/cm-mlops/script/benchmark-any-mlperf-inference-implementation/customize.py index 2902feffb4..b62daab258 100644 --- a/cm-mlops/script/benchmark-any-mlperf-inference-implementation/customize.py +++ b/cm-mlops/script/benchmark-any-mlperf-inference-implementation/customize.py @@ -29,7 +29,7 @@ def preprocess(i): power = env.get('POWER', '') if str(power).lower() in [ "yes", "true" ]: - POWER_STRING = " --power yes --adr.mlperf-power-client.power_server=" + env.get('POWER_SERVER', '192.168.0.15') + " --adr.mlperf-power-client.port=" + env.get('POWER_SERVER_PORT', '4950') + " " + POWER_STRING = " --power=yes --adr.mlperf-power-client.power_server=" + env.get('POWER_SERVER', '192.168.0.15') + " --adr.mlperf-power-client.port=" + str(env.get('POWER_SERVER_PORT', '4950')) + " " else: POWER_STRING = "" @@ -37,7 +37,12 @@ def preprocess(i): return {'return': 1, 'error': 'No device specified. Please set one or more (comma separated) of {cpu, qaic, cuda, rocm} for --env.DEVICES=<>'} cmds = [] - run_script_content = '#!/bin/bash\nsource '+ os.path.join(script_path, "run-template.sh") + run_script_content = '#!/bin/bash\n\n' + run_script_content += "POWER_STRING=\"" +POWER_STRING +"\"\n" + run_script_content += "DIVISION=\"" + env['DIVISION'] +"\"\n" + run_script_content += "CATEGORY=\"" + env['CATEGORY'] +"\"\n" + run_script_content += 'source '+ os.path.join(script_path, "run-template.sh") + "\nPOWER_STRING=\"" +POWER_STRING +"\"\n\n" + run_file_name = 'tmp-'+implementation+'-run' for model in models: @@ -107,7 +112,7 @@ def preprocess(i): test_query_count = 10000 else: test_query_count = 1000 - cmd = f'run_test "{model} "{backend}" "{test_query_count}" "{implementation}" "{device}" "$find_performance_cmd"' + cmd = f'run_test "{model}" "{backend}" "{test_query_count}" "{implementation}" "{device}" "$find_performance_cmd"' cmds.append(cmd) #second argument is unused for submission_cmd cmd = f'run_test "{model}" "{backend}" "100" "{implementation}" "{device}" "$submission_cmd"' @@ -118,6 +123,10 @@ def preprocess(i): f.write(run_script_content) print(run_script_content) + run_script_input = i['run_script_input'] + r = automation.run_native_script({'run_script_input':run_script_input, 'env':env, 'script_name':run_file_name}) + + if r['return']>0: return r return {'return':0} diff --git a/cm-mlops/script/benchmark-any-mlperf-inference-implementation/run-template.sh b/cm-mlops/script/benchmark-any-mlperf-inference-implementation/run-template.sh index 0f75941f18..c396c42e4d 100644 --- a/cm-mlops/script/benchmark-any-mlperf-inference-implementation/run-template.sh +++ b/cm-mlops/script/benchmark-any-mlperf-inference-implementation/run-template.sh @@ -29,16 +29,15 @@ category=$CATEGORY rerun=$RERUN function run_test() { - backend=$1 - test_query_count=$2 - implementation=$3 - device=$4 - model=$5 + model=$1 + backend=$2 + test_query_count=$3 + implementation=$4 + device=$5 run "$6" } #power=' --power=yes --adr.mlperf-power-client.power_server=192.168.0.15 --adr.mlperf-power-client.port=4950 ' -power=${POWER_STRING} results_dir=$HOME/results_dir #Add your run commands here... @@ -49,22 +48,22 @@ find_performance_cmd='cm run script --tags=generate-run-cmds,inference,_find-per submission_cmd='cm run script --tags=generate-run-cmds,inference,_submission,_all-scenarios \ --model=$model --implementation=$implementation --device=$device --backend=$backend \ --category=$category --division=$division --quiet --results_dir=$results_dir \ ---skip_submission_generation=yes --execution-mode=valid $power' +--skip_submission_generation=yes --execution-mode=valid ${POWER_STRING}' submission_cmd_scenario='cm run script --tags=generate-run-cmds,inference,_submission --scenario=$scenario \ --model=$model --implementation=$implementation --device=$device --backend=$backend \ --category=$category --division=$division --quiet --results_dir=$results_dir \ ---skip_submission_generation=yes --execution-mode=valid $power' +--skip_submission_generation=yes --execution-mode=valid ${POWER_STRING}' readme_cmd_single='cm run script --tags=generate-run-cmds,inference,_populate-readme --scenario=$scenario \ --model=$model --implementation=$implementation --device=$device --backend=$backend \ --category=$category --division=$division --quiet --results_dir=$results_dir \ ---skip_submission_generation=yes --execution-mode=valid $power' +--skip_submission_generation=yes --execution-mode=valid ${POWER_STRING}' readme_cmd='cm run script --tags=generate-run-cmds,inference,_populate-readme,_all-scenarios \ --model=$model --implementation=$implementation --device=$device --backend=$backend \ --category=$category --division=$division --quiet --results_dir=$results_dir \ ---skip_submission_generation=yes --execution-mode=valid $power' +--skip_submission_generation=yes --execution-mode=valid ${POWER_STRING}' tflite_accuracy_cmd='cm run script --tags=run,mobilenet-models,_tflite,_accuracy-only$extra_tags \ @@ -73,13 +72,13 @@ ${extra_option} \ --results_dir=$results_dir' tflite_performance_cmd='cm run script --tags=run,mobilenet-models,_tflite,_performance-only$extra_tags \ -${POWER} \ +${POWER_STRING} \ --adr.compiler.tags=gcc \ ${extra_option} \ --results_dir=$results_dir' tflite_readme_cmd='cm run script --tags=run,mobilenet-models,_tflite,_populate-readme$extra_tags \ -${POWER} \ +${POWER_STRING} \ --adr.compiler.tags=gcc \ ${extra_option} \ --results_dir=$results_dir' diff --git a/cm-mlops/script/build-dockerfile/customize.py b/cm-mlops/script/build-dockerfile/customize.py index c5c473d8de..15d52fe1c1 100644 --- a/cm-mlops/script/build-dockerfile/customize.py +++ b/cm-mlops/script/build-dockerfile/customize.py @@ -174,7 +174,6 @@ def preprocess(i): if gh_token: run_cmd_extra = " --env.CM_GH_TOKEN=$CM_GH_TOKEN" - f.write(EOL+'# Run commands' + EOL) for comment in env.get('CM_DOCKER_RUN_COMMENTS', []): f.write(comment + EOL) diff --git a/cm-mlops/script/build-dockerfile/dockerinfo.json b/cm-mlops/script/build-dockerfile/dockerinfo.json index 12a654d617..5fba91b03f 100644 --- a/cm-mlops/script/build-dockerfile/dockerinfo.json +++ b/cm-mlops/script/build-dockerfile/dockerinfo.json @@ -1,6 +1,6 @@ { "python-packages": [ - "cmind", "requests" + "cmind", "requests", "giturlparse" ], "ARGS": [ "CM_GH_TOKEN" diff --git a/cm-mlops/script/create-patch/_cm.yaml b/cm-mlops/script/create-patch/_cm.yaml new file mode 100644 index 0000000000..fa6a3bd6c1 --- /dev/null +++ b/cm-mlops/script/create-patch/_cm.yaml @@ -0,0 +1,16 @@ +uid: 0659dc1f75664c65 +alias: create-patch + +automation_alias: script +automation_uid: 5b4e0237da074764 + +cache: false + +input_mapping: + new: CM_CREATE_PATCH_NEW + old: CM_CREATE_PATCH_OLD + +tags: +- create +- patch + diff --git a/cm-mlops/script/create-patch/customize.py b/cm-mlops/script/create-patch/customize.py new file mode 100644 index 0000000000..f716d8303d --- /dev/null +++ b/cm-mlops/script/create-patch/customize.py @@ -0,0 +1,43 @@ +from cmind import utils +import os + +def preprocess(i): + + os_info = i['os_info'] + + env = i['env'] + + meta = i['meta'] + + automation = i['automation'] + + quiet = (env.get('CM_QUIET', False) == 'yes') + + + new_dir = env.get('CM_CREATE_PATCH_NEW', '') + if new_dir == '': + return {'return':1, 'error':'specify NEW directory using --new'} + if not os.path.isdir(new_dir): + return {'return':1, 'error':'NEW directory doesn\'t exist {}'.format(new_dir)} + + old_dir = env.get('CM_CREATE_PATCH_OLD', '') + if old_dir == '': + return {'return':1, 'error':'specify OLD directory using --new'} + if not os.path.isdir(old_dir): + return {'return':1, 'error':'OLD directory doesn\'t exist {}'.format(old_dir)} + + cmd = 'diff -Naur {} {} > patch.patch'.format(old_dir, new_dir) + + print ('Running command:') + print (cmd) + + os.system(cmd) + + + return {'return':0} + +def postprocess(i): + + env = i['env'] + + return {'return':0} diff --git a/cm-mlops/script/detect-sudo/_cm.yaml b/cm-mlops/script/detect-sudo/_cm.yaml index 5f8da3f890..56c83b8304 100644 --- a/cm-mlops/script/detect-sudo/_cm.yaml +++ b/cm-mlops/script/detect-sudo/_cm.yaml @@ -1,22 +1,17 @@ +uid: 1d47ffc556e248dc alias: detect-sudo + automation_alias: script automation_uid: 5b4e0237da074764 + cache: false + category: DevOps automation -deps: [] -docker_input_mapping: {} -input_description: {} -input_mapping: {} + new_env_keys: - CM_SUDO_* -new_state_keys: [] -post_deps: [] -posthook_deps: [] -prehook_deps: [] + tags: - detect - sudo - access -uid: 1d47ffc556e248dc -variations: {} -versions: {} diff --git a/cm-mlops/script/fail/README-extra.md b/cm-mlops/script/fail/README-extra.md new file mode 100644 index 0000000000..582991f6d2 --- /dev/null +++ b/cm-mlops/script/fail/README-extra.md @@ -0,0 +1 @@ +# CM script diff --git a/cm-mlops/script/fail/_cm.yaml b/cm-mlops/script/fail/_cm.yaml new file mode 100644 index 0000000000..4038d83b5b --- /dev/null +++ b/cm-mlops/script/fail/_cm.yaml @@ -0,0 +1,16 @@ +uid: 3aaee82e19d243cd +alias: fail + +automation_alias: script +automation_uid: 5b4e0237da074764 + +cache: false + +tags: +- fail +- filter + +variations: + windows: + env: + CM_FAIL_WINDOWS: true diff --git a/cm-mlops/script/fail/customize.py b/cm-mlops/script/fail/customize.py new file mode 100644 index 0000000000..855c39b5bb --- /dev/null +++ b/cm-mlops/script/fail/customize.py @@ -0,0 +1,28 @@ +from cmind import utils +import os + +def preprocess(i): + + os_info = i['os_info'] + + env = i['env'] + + meta = i['meta'] + + automation = i['automation'] + + quiet = (env.get('CM_QUIET', False) == 'yes') + + # Checking conditions + if env.get('CM_FAIL_WINDOWS','').lower()=='true': + if os_info['platform'] == 'windows': + return {'return':1, 'error': 'CM detected fail condition: running on Windows'} + + + return {'return':0} + +def postprocess(i): + + env = i['env'] + + return {'return':0} diff --git a/cm-mlops/script/get-compiler-rust/_cm.yaml b/cm-mlops/script/get-compiler-rust/_cm.yaml index c0a7d010b0..ca1a372d9c 100644 --- a/cm-mlops/script/get-compiler-rust/_cm.yaml +++ b/cm-mlops/script/get-compiler-rust/_cm.yaml @@ -1,23 +1,19 @@ +uid: 97ffbd9e537b4b59 alias: get-compiler-rust + automation_alias: script automation_uid: 5b4e0237da074764 + cache: true + category: Compiler automation + deps: - tags: get,python3 -docker_input_mapping: {} -input_description: {} -input_mapping: {} new_env_keys: - "+PATH" -new_state_keys: [] -post_deps: [] -posthook_deps: [] -prehook_deps: [] + tags: - get - rust-compiler -uid: 97ffbd9e537b4b59 -variations: {} -versions: {} diff --git a/cm-mlops/script/get-cuda-devices/_cm.json b/cm-mlops/script/get-cuda-devices/_cm.json index ffa5ede93d..2a01c191df 100644 --- a/cm-mlops/script/get-cuda-devices/_cm.json +++ b/cm-mlops/script/get-cuda-devices/_cm.json @@ -2,6 +2,8 @@ "alias": "get-cuda-devices", "automation_alias": "script", "automation_uid": "5b4e0237da074764", + "can_force_cache": true, + "cache": false, "category": "CUDA automation", "clean_files": [ "tmp-run.out" diff --git a/cm-mlops/script/get-cuda-devices/run.sh b/cm-mlops/script/get-cuda-devices/run.sh index 87c51fbb77..9c774758fc 100644 --- a/cm-mlops/script/get-cuda-devices/run.sh +++ b/cm-mlops/script/get-cuda-devices/run.sh @@ -25,7 +25,8 @@ echo "" echo "Running program ..." echo "" -cd ${CM_TMP_CURRENT_PATH} +cd - +#${CM_TMP_CURRENT_PATH} ${CM_TMP_CURRENT_SCRIPT_PATH}/a.out > tmp-run.out test $? -eq 0 || exit 1 diff --git a/cm-mlops/script/get-cuda/_cm.json b/cm-mlops/script/get-cuda/_cm.json index 65e2956ecb..d397624bd9 100644 --- a/cm-mlops/script/get-cuda/_cm.json +++ b/cm-mlops/script/get-cuda/_cm.json @@ -13,7 +13,8 @@ "run": false }, "input_mapping": { - "cudnn_tar_path": "CM_CUDNN_TAR_FILE_PATH" + "cudnn_tar_path": "CM_CUDNN_TAR_FILE_PATH", + "cudnn_tar_file": "CM_CUDNN_TAR_FILE_PATH" }, "new_env_keys": [ "CUDA_HOME", diff --git a/cm-mlops/script/get-cudnn/README-extra.md b/cm-mlops/script/get-cudnn/README-extra.md new file mode 100644 index 0000000000..3746808138 --- /dev/null +++ b/cm-mlops/script/get-cudnn/README-extra.md @@ -0,0 +1,3 @@ +# TBD + +We need to add detection of cuDNN version on Windows, Linux and MacOS diff --git a/cm-mlops/script/get-cudnn/customize.py b/cm-mlops/script/get-cudnn/customize.py index fbc4e67077..6d57c5ac32 100644 --- a/cm-mlops/script/get-cudnn/customize.py +++ b/cm-mlops/script/get-cudnn/customize.py @@ -28,6 +28,7 @@ def preprocess(i): libfilename = extra_pre + 'cudnn.' +extra_ext env['CM_CUDNN_VERSION'] = 'vdetected' + if os.path.exists(os.path.join(cuda_path_lib, libfilename)): env['CM_CUDA_PATH_LIB_CUDNN'] = env['CM_CUDA_PATH_LIB'] return {'return': 0} diff --git a/cm-mlops/script/get-dataset-imagenet-calibration/_cm.yaml b/cm-mlops/script/get-dataset-imagenet-calibration/_cm.yaml index 7d973c68a3..212c506a1f 100644 --- a/cm-mlops/script/get-dataset-imagenet-calibration/_cm.yaml +++ b/cm-mlops/script/get-dataset-imagenet-calibration/_cm.yaml @@ -1,7 +1,11 @@ +uid: 30361fad3dff49ff alias: get-dataset-imagenet-calibration + automation_alias: script automation_uid: 5b4e0237da074764 + cache: true + deps: - tags: download,file force_cache: true @@ -11,21 +15,15 @@ deps: env: CM_DOWNLOAD_FINAL_ENV_NAME: CM_MLPERF_IMAGENET_CALIBRATION_LIST_FILE_WITH_PATH -docker_input_mapping: {} -input_description: {} -input_mapping: {} new_env_keys: - CM_MLPERF_IMAGENET_CALIBRATION_LIST_FILE_WITH_PATH -new_state_keys: [] -post_deps: [] -posthook_deps: [] -prehook_deps: [] + tags: - get - dataset - imagenet - calibration -uid: 30361fad3dff49ff + variations: mlperf.option1: group: calibration-option diff --git a/cm-mlops/script/get-dataset-openimages-calibration/_cm.yaml b/cm-mlops/script/get-dataset-openimages-calibration/_cm.yaml index 2ae594f52b..c2ffadffcd 100644 --- a/cm-mlops/script/get-dataset-openimages-calibration/_cm.yaml +++ b/cm-mlops/script/get-dataset-openimages-calibration/_cm.yaml @@ -1,7 +1,11 @@ +uid: 27228976bb084dd0 alias: get-dataset-openimages-calibration + automation_alias: script automation_uid: 5b4e0237da074764 + cache: true + deps: - tags: download,file force_cache: true @@ -11,21 +15,15 @@ deps: env: CM_DOWNLOAD_FINAL_ENV_NAME: CM_MLPERF_OPENIMAGES_CALIBRATION_LIST_FILE_WITH_PATH -docker_input_mapping: {} -input_description: {} -input_mapping: {} new_env_keys: - CM_MLPERF_OPENIMAGES_CALIBRATION_LIST_FILE_WITH_PATH -new_state_keys: [] -post_deps: [] -posthook_deps: [] -prehook_deps: [] + tags: - get - dataset - openimages - calibration -uid: 27228976bb084dd0 + variations: mlperf.option1: group: calibration-option diff --git a/cm-mlops/script/get-generic-python-lib/customize.py b/cm-mlops/script/get-generic-python-lib/customize.py index 1014fe2e05..5027d82376 100644 --- a/cm-mlops/script/get-generic-python-lib/customize.py +++ b/cm-mlops/script/get-generic-python-lib/customize.py @@ -15,6 +15,7 @@ def preprocess(i): return automation._available_variations({'meta':meta}) if package_name == "onnxruntime_gpu": + # https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements cuda_version_split = env['CM_CUDA_VERSION'].split(".") if int(cuda_version_split[0]) >= 12: # env['CM_INSTALL_ONNXRUNTIME_GPU_FROM_SRC'] = "yes" diff --git a/cm-mlops/script/get-ipol-src/README.extra.md b/cm-mlops/script/get-ipol-src/README.extra.md new file mode 100644 index 0000000000..1618d0ed0b --- /dev/null +++ b/cm-mlops/script/get-ipol-src/README.extra.md @@ -0,0 +1 @@ +20240127: Grigori added patch to support latest PIL diff --git a/cm-mlops/script/get-ipol-src/customize.py b/cm-mlops/script/get-ipol-src/customize.py index b4da451adf..f2b5dd1fca 100644 --- a/cm-mlops/script/get-ipol-src/customize.py +++ b/cm-mlops/script/get-ipol-src/customize.py @@ -7,6 +7,8 @@ def preprocess(i): env = i['env'] + script_path = i['artifact'].path + automation = i['automation'] cm = automation.cmind @@ -47,4 +49,10 @@ def preprocess(i): env['CM_IPOL_PATH']=os.path.join(path, subdir) + # Applying patch + cmd = 'patch -p0 < {}'.format(os.path.join(script_path, 'patch', '20240127.patch')) + + print ('Patching code: {}'.format(cmd)) + os.system(cmd) + return {'return':0} diff --git a/cm-mlops/script/get-ipol-src/patch/20240127.patch b/cm-mlops/script/get-ipol-src/patch/20240127.patch new file mode 100644 index 0000000000..6610d0ceea --- /dev/null +++ b/cm-mlops/script/get-ipol-src/patch/20240127.patch @@ -0,0 +1,10 @@ +diff -Naur 439-master/main.py 439-master.new/main.py +--- 439-master/main.py Sat Jan 27 22:11:55 2024 ++++ 439-master.new/main.py Sat Jan 27 22:06:51 2024 +@@ -135,5 +135,5 @@ + args = parser.parse_args() + #print('before plume detection', os.path.dirname(os.path.realpath('__file__')), file=sys.stderr) + p = compute_map(args.input_0,args.input_1) +- imageio.imsave("cm.png", ((255*p[0,:,:])).numpy()) ++ imageio.imsave("cm.png", np.array((255*p[0,:,:]).numpy(), np.uint8)) + diff --git a/cm-mlops/script/get-mlperf-inference-src/_cm.json b/cm-mlops/script/get-mlperf-inference-src/_cm.json index 8db3b73b47..6b0dbb6bfd 100644 --- a/cm-mlops/script/get-mlperf-inference-src/_cm.json +++ b/cm-mlops/script/get-mlperf-inference-src/_cm.json @@ -165,7 +165,7 @@ "versions": { "master": { "env": { - "CM_MLPERF_LAST_RELEASE": "v3.1", + "CM_MLPERF_LAST_RELEASE": "v4.0", "CM_GIT_CHECKOUT": "master" } }, diff --git a/cm-mlops/script/get-preprocessed-dataset-squad/_cm.yaml b/cm-mlops/script/get-preprocessed-dataset-squad/_cm.yaml index a428b6f081..cff348c266 100644 --- a/cm-mlops/script/get-preprocessed-dataset-squad/_cm.yaml +++ b/cm-mlops/script/get-preprocessed-dataset-squad/_cm.yaml @@ -1,8 +1,13 @@ +uid: 7cd1d9b7e8af4788 alias: get-preprocessed-dataset-squad + automation_alias: script automation_uid: 5b4e0237da074764 + cache: true + category: AI/ML datasets + deps: - tags: get,python3 names: @@ -21,25 +26,19 @@ deps: - tags: get,generic-python-lib,_package.transformers - tags: get,generic-python-lib,_package.tensorflow -docker_input_mapping: {} -input_description: {} env: CM_DATASET_MAX_QUERY_LENGTH: 64 -input_mapping: {} new_env_keys: - CM_DATASET_SQUAD_TOKENIZED_* -new_state_keys: [] -post_deps: [] -posthook_deps: [] -prehook_deps: [] + tags: - get - dataset - preprocessed - tokenized - squad -uid: 7cd1d9b7e8af4788 + variations: calib1: group: calibration-set @@ -92,5 +91,3 @@ variations: inherit_variation_tags: true skip_inherit_variation_groups: - packing - -versions: {} diff --git a/cm-mlops/script/get-spec-ptd/_cm.json b/cm-mlops/script/get-spec-ptd/_cm.json index ccbffa3838..2e7a2c1cb8 100644 --- a/cm-mlops/script/get-spec-ptd/_cm.json +++ b/cm-mlops/script/get-spec-ptd/_cm.json @@ -8,7 +8,6 @@ "CM_GIT_CHECKOUT": "main", "CM_GIT_DEPTH": "--depth 1", "CM_GIT_PATCH": "no", - "CM_GIT_AUTH": "yes", "CM_GIT_RECURSE_SUBMODULES": " ", "CM_GIT_URL": "https://github.com/mlcommons/power.git" }, @@ -33,7 +32,11 @@ { "tags": "get,git,repo,_repo.https://github.com/mlcommons/power", "extra_cache_tags": "mlperf,power,spec,ptdaemon,ptd", + "force_env_keys": [ + "CM_GIT_*" + ], "env": { + "CM_GIT_AUTH": "yes", "CM_GIT_CHECKOUT_PATH_ENV_NAME": "CM_MLPERF_POWER_SOURCE" } } diff --git a/cm-mlops/script/get-spec-ptd/customize.py b/cm-mlops/script/get-spec-ptd/customize.py index 7cab98136a..250ddd887b 100644 --- a/cm-mlops/script/get-spec-ptd/customize.py +++ b/cm-mlops/script/get-spec-ptd/customize.py @@ -14,8 +14,12 @@ def postprocess(i): env = i['env'] state = i['state'] + if env['CM_HOST_OS_TYPE'].lower() == "windows": + binary_name = "ptd-windows-x86.exe" + else: + binary_name = "ptd-linux-x86" if 'CM_MLPERF_PTD_PATH' not in env: - env['CM_MLPERF_PTD_PATH'] = os.path.join(env['CM_MLPERF_POWER_SOURCE'], 'inference_v1.0', 'ptd-linux-x86') + env['CM_MLPERF_PTD_PATH'] = os.path.join(env['CM_MLPERF_POWER_SOURCE'], 'inference_v1.0', binary_name) env['CM_SPEC_PTD_PATH'] = env['CM_MLPERF_PTD_PATH'] return {'return':0} diff --git a/cm-mlops/script/install-onnxruntime-from-src/_cm.json b/cm-mlops/script/install-onnxruntime-from-src/_cm.json index 9b16611d33..a06b735852 100644 --- a/cm-mlops/script/install-onnxruntime-from-src/_cm.json +++ b/cm-mlops/script/install-onnxruntime-from-src/_cm.json @@ -11,6 +11,9 @@ { "tags": "detect,cpu" }, + { + "tags": "fail,filter,_windows" + }, { "names": [ "python", diff --git a/cm-mlops/script/reproduce-ipol-paper-2022-439/README-extra.md b/cm-mlops/script/reproduce-ipol-paper-2022-439/README-extra.md index 2c5d45c526..fa37618881 100644 --- a/cm-mlops/script/reproduce-ipol-paper-2022-439/README-extra.md +++ b/cm-mlops/script/reproduce-ipol-paper-2022-439/README-extra.md @@ -49,12 +49,15 @@ cm run script "get ipol src" --year=2022 --number=439 cm show cache --tags=ipol,src ``` -4. Run demo (CM will detect or install missing dependencies) +4. Download sample images and run demo (CM will detect or install missing dependencies) ```bash +cm run script "download file _wget" --url=https://cKnowledge.org/ai/data/ipol-paper-2024-439-sample-image-1.png --verify=no --env.CM_DOWNLOAD_CHECKSUM=850639287ad23194576582680c2ecfc3 +cm run script "download file _wget" --url=https://cKnowledge.org/ai/data/ipol-paper-2024-439-sample-image-2.png --verify=no --env.CM_DOWNLOAD_CHECKSUM=31364c03d91873ed2d244cce6d664dd0 cm run script "reproduce ipol 2022-439" +cm run script "reproduce ipol 2022-439" --adr.torch.version=1.13.1 --adr.torchvision.version=0.14.1 ``` -This script will use these [sample images](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/reproduce-ipol-paper-2022-439/sample-images) +This script will use 2 sample images from this paper and should produce *diff.png* in the current directory. ## Usage with different images diff --git a/cm-mlops/script/reproduce-ipol-paper-2022-439/_cm.yaml b/cm-mlops/script/reproduce-ipol-paper-2022-439/_cm.yaml index cf08b1f4dc..bd7c9e1401 100644 --- a/cm-mlops/script/reproduce-ipol-paper-2022-439/_cm.yaml +++ b/cm-mlops/script/reproduce-ipol-paper-2022-439/_cm.yaml @@ -22,6 +22,9 @@ deps: - tags: get,generic-python-lib,_torch names: - torch +- tags: get,generic-python-lib,_torchvision + names: + - torchvision tags: - app diff --git a/cm-mlops/script/reproduce-ipol-paper-2022-439/customize.py b/cm-mlops/script/reproduce-ipol-paper-2022-439/customize.py index 1409115a1f..6b57ab932f 100644 --- a/cm-mlops/script/reproduce-ipol-paper-2022-439/customize.py +++ b/cm-mlops/script/reproduce-ipol-paper-2022-439/customize.py @@ -6,8 +6,29 @@ def preprocess(i): env = i['env'] + # Check if input files are empty and add files + input_file_1 = env.get('CM_INPUT_1','') + if input_file_1 == '': input_file_1 = 'ipol-paper-2024-439-sample-image-1.png' + + if not os.path.isfile(input_file_1): + return {'return':1, 'error':'input file 1 "{}" not found'.format(input_file_1)} + + env['CM_INPUT_1']=os.path.abspath(input_file_1) + + input_file_2 = env.get('CM_INPUT_2','') + if input_file_2 == '': input_file_2 = 'ipol-paper-2024-439-sample-image-2.png' + + if not os.path.isfile(input_file_2): + return {'return':1, 'error':'input file 2 "{}" not found'.format(input_file_2)} + + env['CM_INPUT_2']=os.path.abspath(input_file_2) + return {'return':0} def postprocess(i): + print ('') + print ('Please check "diff.png"') + print ('') + return {'return':0} diff --git a/cm-mlops/script/reproduce-ipol-paper-2022-439/requirements.txt b/cm-mlops/script/reproduce-ipol-paper-2022-439/requirements.txt index 0778518560..82a4d6034b 100644 --- a/cm-mlops/script/reproduce-ipol-paper-2022-439/requirements.txt +++ b/cm-mlops/script/reproduce-ipol-paper-2022-439/requirements.txt @@ -1,7 +1,5 @@ jupyter numpy imageio -torch -torchvision IPython scikit-image diff --git a/cm-mlops/script/reproduce-ipol-paper-2022-439/run.bat b/cm-mlops/script/reproduce-ipol-paper-2022-439/run.bat index bd1506709e..7aafa4a34a 100644 --- a/cm-mlops/script/reproduce-ipol-paper-2022-439/run.bat +++ b/cm-mlops/script/reproduce-ipol-paper-2022-439/run.bat @@ -17,15 +17,6 @@ cd %CM_IPOL_PATH% echo Current path in CM cache: %cd% -rem Check default images -if "%CM_INPUT_1%" == "" ( - set CM_INPUT_1=%CM_TMP_CURRENT_SCRIPT_PATH%\sample-images\1.png -) - -if "%CM_INPUT_2%" == "" ( - set CM_INPUT_2=%CM_TMP_CURRENT_SCRIPT_PATH%\sample-images\2.png -) - echo Running author's code ... del /F /Q cm.png diff --git a/cm-mlops/script/reproduce-ipol-paper-2022-439/sample-images/1.png b/cm-mlops/script/reproduce-ipol-paper-2022-439/sample-images/1.png deleted file mode 100644 index 722f0944ee..0000000000 Binary files a/cm-mlops/script/reproduce-ipol-paper-2022-439/sample-images/1.png and /dev/null differ diff --git a/cm-mlops/script/reproduce-ipol-paper-2022-439/sample-images/2.png b/cm-mlops/script/reproduce-ipol-paper-2022-439/sample-images/2.png deleted file mode 100644 index a681f89a5a..0000000000 Binary files a/cm-mlops/script/reproduce-ipol-paper-2022-439/sample-images/2.png and /dev/null differ diff --git a/cm-mlops/script/reproduce-mlperf-inference-intel/customize.py b/cm-mlops/script/reproduce-mlperf-inference-intel/customize.py index 7b1cf69a5c..05279e631e 100644 --- a/cm-mlops/script/reproduce-mlperf-inference-intel/customize.py +++ b/cm-mlops/script/reproduce-mlperf-inference-intel/customize.py @@ -36,7 +36,8 @@ def preprocess(i): pass elif "retinanet" in env.get('CM_MODEL'): pass - + elif "gptj" in env.get('CM_MODEL'): + env['CHECKPOINT_DIR'] = env['GPTJ_CHECKPOINT_PATH'] script_path = i['run_script_input']['path'] if env['CM_MODEL'] == "retinanet": @@ -63,18 +64,29 @@ def preprocess(i): env['CM_MLPERF_INFERENCE_INTEL_HARNESS_PATH'] = os.path.join(os.getcwd(), "harness", "build", "gptj_inference") env['DATA_PATH'] = os.path.join(os.getcwd(), "harness", "gptj") env['MLPERF_INFERENCE_ROOT'] = env['CM_MLPERF_INFERENCE_SOURCE'] - env['CHECKPOINT_DIR'] = env['GPTJ_CHECKPOINT_PATH'] + final_model_path = os.path.join(harness_root, "data", "gpt-j-int8-model") + env['INT8_MODEL_DIR'] = os.path.dirname(final_model_path) + env['CM_ML_MODEL_PATH'] = final_model_path elif env['CM_LOCAL_MLPERF_INFERENCE_INTEL_RUN_MODE'] == "run_harness": - env['MODEL_PATH'] = os.path.dirname(os.path.dirname(env['CM_MLPERF_INFERENCE_INTEL_HARNESS_PATH'])) - env['DATASET_PATH'] = os.path.dirname(os.path.dirname(env['CM_MLPERF_INFERENCE_INTEL_HARNESS_PATH'])) - env['CM_RUN_DIR'] = i['run_script_input']['path'] - env['CM_RUN_CMD'] = "bash run_harness.sh " + ("--accuracy" if env['CM_MLPERF_LOADGEN_MODE'] == "accuracy" else "") - if env.get('CM_MLPERF_LOADGEN_MODE', '') == "compliance": audit_path = env['CM_MLPERF_INFERENCE_AUDIT_PATH'] shutil.copy(audit_path, env['CM_RUN_DIR']) + if 'bert' in env['CM_MODEL']: + env['MODEL_PATH'] = os.path.dirname(os.path.dirname(env['CM_MLPERF_INFERENCE_INTEL_HARNESS_PATH'])) + env['DATASET_PATH'] = os.path.dirname(os.path.dirname(env['CM_MLPERF_INFERENCE_INTEL_HARNESS_PATH'])) + env['CM_RUN_DIR'] = i['run_script_input']['path'] + env['CM_RUN_CMD'] = "bash run_bert_harness.sh " + ("--accuracy" if env['CM_MLPERF_LOADGEN_MODE'] == "accuracy" else "") + elif "gptj" in env['CM_MODEL']: + if env['CM_MLPERF_LOADGEN_MODE'] == "accuracy": + env['LOADGEN_MODE'] = 'Accuracy' + else: + env['LOADGEN_MODE'] = 'Performance' + env['INT8_MODEL_DIR'] = env['CM_ML_MODEL_PATH'] + env['CM_RUN_DIR'] = i['run_script_input']['path'] + env['CM_RUN_CMD'] = "bash run_gptj_harness.sh " + return {'return':0} def postprocess(i): diff --git a/cm-mlops/script/reproduce-mlperf-inference-intel/run_harness.sh b/cm-mlops/script/reproduce-mlperf-inference-intel/run_bert_harness.sh similarity index 100% rename from cm-mlops/script/reproduce-mlperf-inference-intel/run_harness.sh rename to cm-mlops/script/reproduce-mlperf-inference-intel/run_bert_harness.sh diff --git a/cm-mlops/script/reproduce-mlperf-inference-intel/run_gptj_harness.sh b/cm-mlops/script/reproduce-mlperf-inference-intel/run_gptj_harness.sh new file mode 100755 index 0000000000..2b2c733476 --- /dev/null +++ b/cm-mlops/script/reproduce-mlperf-inference-intel/run_gptj_harness.sh @@ -0,0 +1,51 @@ +#!/bin/bash +export PATH=${CM_CONDA_BIN_PATH}:$PATH + +export KMP_BLOCKTIME=4 +export KMP_AFFINITY=granularity=fine,compact,1,0 +export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libiomp5.so +export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so + +export num_physical_cores=`lscpu -b -p=Core,Socket | grep -v '^#' | sort -u | wc -l` +num_numa=$(numactl --hardware|grep available|awk -F' ' '{ print $2 }') + +NUM_PROC=$num_numa +CPUS_PER_PROC=$((num_physical_cores/num_numa)) +WORKERS_PER_PROC=1 +TOTAL_SAMPLE_COUNT=13368 +BATCH_SIZE=14 +TIMESTAMP=$(date +%m-%d-%H-%M) +HOSTNAME=$(hostname) +#OUTPUT_DIR=offline-output-${HOSTNAME}-batch-${BATCH_SIZE}-procs-${NUM_PROC}-ins-per-proc-${WORKERS_PER_PROC}-${TIMESTAMP} + +export WORKLOAD_DATA=${CM_HARNESS_CODE_ROOT}/data +export VALIDATION_DATA_JSON=${WORKLOAD_DATA}/validation-data/cnn_dailymail_validation.json + +cd ${CM_HARNESS_CODE_ROOT} +OUTPUT_DIR="${CM_MLPERF_OUTPUT_DIR}" + +USER_CONF="${CM_MLPERF_USER_CONF}" + + + #--mode Performance \ +cmd="python runner.py --workload-name gptj \ + --scenario Offline \ + --mode ${LOADGEN_MODE} \ + --num-proc ${NUM_PROC} \ + --cpus-per-proc ${CPUS_PER_PROC} \ + --model-checkpoint-path ${CHECKPOINT_DIR} \ + --warmup \ + --dataset-path ${VALIDATION_DATA_JSON} \ + --batch-size ${BATCH_SIZE} \ + --mlperf-conf ${CM_MLPERF_CONF} \ + --user-conf ${CM_MLPERF_USER_CONF} \ + --precision int8 \ + --pad-inputs \ + --quantized-model ${INT8_MODEL_DIR}/best_model.pt \ + --workers-per-proc ${WORKERS_PER_PROC} \ + --total-sample-count ${TOTAL_SAMPLE_COUNT} \ + --output-dir ${OUTPUT_DIR} \ + 2>&1 | tee ${OUTPUT_DIR}.log" + +echo "$cmd" +eval "$cmd" diff --git a/cm-mlops/script/run-all-mlperf-models/_cm.yaml b/cm-mlops/script/run-all-mlperf-models/_cm.yaml index 6404e53543..a53b73a193 100644 --- a/cm-mlops/script/run-all-mlperf-models/_cm.yaml +++ b/cm-mlops/script/run-all-mlperf-models/_cm.yaml @@ -1,24 +1,18 @@ +uid: 8d3cd46f54464810 alias: run-all-mlperf-models + automation_alias: script automation_uid: 5b4e0237da074764 + cache: false + category: MLPerf benchmark support -deps: [] -docker_input_mapping: {} -input_description: {} -input_mapping: {} -new_env_keys: [] -new_state_keys: [] -post_deps: [] -posthook_deps: [] -prehook_deps: [] -tags: [ - "run", - "natively", - "all", - "mlperf-models" -] -uid: 8d3cd46f54464810 + +tags: +- run +- natively +- all +- mlperf-models variations: @@ -134,5 +128,3 @@ variations: env: CATEGORY: edge DIVISION: closed - -versions: {} diff --git a/cm-mlops/script/run-all-mlperf-models/run-cpp-implementation.sh b/cm-mlops/script/run-all-mlperf-models/run-cpp-implementation.sh index 694edf6d69..704abff2d7 100644 --- a/cm-mlops/script/run-all-mlperf-models/run-cpp-implementation.sh +++ b/cm-mlops/script/run-all-mlperf-models/run-cpp-implementation.sh @@ -88,13 +88,13 @@ ${POWER} \ run "cm run script --tags=generate-run-cmds,inference,_find-performance \ --model=resnet50 --implementation=cpp --device=cuda --backend=onnxruntime \ --adr.compiler.tags=gcc \ ---execution-mode=valid \ ---category=edge --division=open --scenario=Offline --quiet --test_query_count=100" +--test_query_count=20000 \ +--category=edge --division=open --scenario=Offline --quiet" run "cm run script --tags=generate-run-cmds,inference,_find-performance \ --model=retinanet --implementation=cpp --device=cuda --backend=onnxruntime \ --adr.compiler.tags=gcc \ ---execution-mode=valid \ +--test_query_count=2000 \ --category=edge --division=open --scenario=Offline --quiet" @@ -139,3 +139,25 @@ run "cm run script --tags=generate-run-cmds,inference,_submission \ --skip_submission_generation=yes \ ${POWER} \ --results_dir=$HOME/results_dir" + +#multistream +run "cm run script --tags=generate-run-cmds,inference,_submission \ +--scenario=Offline \ +--model=resnet50 --implementation=cpp --device=cuda --backend=onnxruntime \ +--scenario=MultiStream \ +--category=edge --division=$division --quiet \ +--adr.compiler.tags=gcc \ +--execution-mode=valid \ +--skip_submission_generation=yes \ +${POWER} \ +--results_dir=$HOME/results_dir" + +run "cm run script --tags=generate-run-cmds,inference,_submission \ +--model=retinanet --implementation=cpp --device=cuda --backend=onnxruntime \ +--scenario=MultiStream \ +--category=edge --division=$division --quiet \ +--adr.compiler.tags=gcc \ +--execution-mode=valid \ +--skip_submission_generation=yes \ +${POWER} \ +--results_dir=$HOME/results_dir" diff --git a/cm-mlops/script/run-docker-container/customize.py b/cm-mlops/script/run-docker-container/customize.py index beceaff111..f01f9d1eb9 100644 --- a/cm-mlops/script/run-docker-container/customize.py +++ b/cm-mlops/script/run-docker-container/customize.py @@ -197,4 +197,4 @@ def update_docker_info(env): docker_image_tag = env.get('CM_DOCKER_IMAGE_TAG', docker_image_base.replace(':','-').replace('_','') + "-latest") env['CM_DOCKER_IMAGE_TAG'] = docker_image_tag - return \ No newline at end of file + return diff --git a/cm-mlops/script/run-mlperf-inference-app/customize.py b/cm-mlops/script/run-mlperf-inference-app/customize.py index fe26967e47..eb96904b14 100644 --- a/cm-mlops/script/run-mlperf-inference-app/customize.py +++ b/cm-mlops/script/run-mlperf-inference-app/customize.py @@ -210,9 +210,12 @@ def get_valid_scenarios(model, category, mlperf_version, mlperf_path): import submission_checker as checker if "dlrm-99" in model: - model= model.replace("dlrm-99", "dlrm-v2-99") + model = model.replace("dlrm-99", "dlrm-v2-99") + if "sdxl" in model: + model = "stable-diffusion-xl" config = checker.MODEL_CONFIG + internal_model_name = config[mlperf_version]["model_mapping"].get(model, model) valid_scenarios = config[mlperf_version]["required-scenarios-"+category][internal_model_name] diff --git a/cm-mlops/script/run-mlperf-power-server/_cm.json b/cm-mlops/script/run-mlperf-power-server/_cm.json index 23b47e8ae2..5082ca3f05 100644 --- a/cm-mlops/script/run-mlperf-power-server/_cm.json +++ b/cm-mlops/script/run-mlperf-power-server/_cm.json @@ -52,6 +52,15 @@ "skip_if_env": { "CM_HOST_OS_TYPE": "windows" } + }, + { + "names": [ + "win32" + ], + "tags": "get,generic-python-lib,_package.pypiwin32", + "enable_if_env": { + "CM_HOST_OS_TYPE": "windows" + } } ], "tags": [ @@ -63,5 +72,11 @@ "server", "power-server" ], - "uid": "5bc68aaf389a40bd" + "uid": "5bc68aaf389a40bd", + "docker": { + "device": "/dev/usbtmc0", + "port_maps": [ + "4950:4950" + ] + } } diff --git a/cm-mlops/script/run-mlperf-power-server/customize.py b/cm-mlops/script/run-mlperf-power-server/customize.py index 87638b5e58..65c7830420 100644 --- a/cm-mlops/script/run-mlperf-power-server/customize.py +++ b/cm-mlops/script/run-mlperf-power-server/customize.py @@ -20,11 +20,16 @@ def preprocess(i): config.write(configfile) print({section: dict(config[section]) for section in config.sections()}) - cmd = 'python3 ' + env['CM_MLPERF_POWER_SOURCE'] + '/ptd_client_server/server.py -c power-server.conf' + if env['CM_HOST_OS_TYPE'] == "windows": + cmd_prefix = "" + else: + cmd_prefix = "sudo " + + cmd = env['CM_PYTHON_BIN_WITH_PATH'] + ' ' + os.path.join(env['CM_MLPERF_POWER_SOURCE'], 'ptd_client_server', 'server.py') +' -c power-server.conf' if env.get('CM_MLPERF_POWER_SERVER_USE_SCREEN', 'no') == 'yes': - cmd = 'sudo screen -d -m ' + cmd + ' ' + cmd = cmd_prefix + ' screen -d -m ' + cmd + ' ' else: - cmd = 'sudo ' + cmd + cmd = cmd_prefix + cmd env['RUN_CMD'] = cmd diff --git a/cm-mlops/script/run-mlperf-power-server/run.bat b/cm-mlops/script/run-mlperf-power-server/run.bat new file mode 100644 index 0000000000..d23f0addf5 --- /dev/null +++ b/cm-mlops/script/run-mlperf-power-server/run.bat @@ -0,0 +1,7 @@ +@echo off + +echo %RUN_CMD% + +%RUN_CMD% + +IF %ERRORLEVEL% NEQ 0 EXIT %ERRORLEVEL% diff --git a/cm/README.md b/cm/README.md index f9c82f5cf0..99e6e2ccf3 100644 --- a/cm/README.md +++ b/cm/README.md @@ -15,8 +15,10 @@ from [MLCommons projects](https://mlcommons.org) and [research papers](https://c in a unified way on any operating system with any software and hardware either natively or inside containers. -Here are some most commonly used examples from the community: - +Here are a few most commonly used examples from the CM users +that should run in the same way on Linux, MacOS, Windows and other platforms +(see [Gettings Started Guide](docs/getting-started.md) to understand +how they work and how to reuse them in your projects): ```bash pip install cmind @@ -24,31 +26,44 @@ pip install cmind cm pull repo mlcommons@ck cm run script "python app image-classification onnx" +cmr "python app image-classification onnx" -cm run script "download file _wget" --url=https://cKnowledge.org/ai/data/computer_mouse.jpg --verify=no --env.CM_DOWNLOAD_CHECKSUM=45ae5c940233892c2f860efdf0b66e7e +cmr "download file _wget" --url=https://cKnowledge.org/ai/data/computer_mouse.jpg --verify=no --env.CM_DOWNLOAD_CHECKSUM=45ae5c940233892c2f860efdf0b66e7e +cmr "python app image-classification onnx" --input=computer_mouse.jpg -cm run script "python app image-classification onnx" --input=computer_mouse.jpg +cm find script "python app image-classification onnx" -cm docker script "python app image-classification onnx" --input=computer_mouse.jpg -cm docker script "python app image-classification onnx" --input=computer_mouse.jpg -j -docker_it +cmr "get python" --version_min=3.8.0 --name=mlperf-experiments +cmr "install python-venv" --version_max=3.10.11 --name=mlperf -cm run script "get generic-python-lib _package.onnxruntime" -cm run script "get coco dataset _val _2014" -cm run script "get ml-model stable-diffusion" -cm run script "get ml-model huggingface zoo _model-stub.alpindale/Llama-2-13b-ONNX" --model_filename=FP32/LlamaV2_13B_float32.onnx --skip_cache +cmr "get ml-model stable-diffusion" +cmr "get ml-model huggingface zoo _model-stub.alpindale/Llama-2-13b-ONNX" --model_filename=FP32/LlamaV2_13B_float32.onnx --skip_cache +cmr "get coco dataset _val _2014" cm show cache cm show cache "get ml-model stable-diffusion" -cm run script "run common mlperf inference" --implementation=nvidia --model=bert-99 --category=datacenter --division=closed +cmr "get generic-python-lib _package.onnxruntime" --version_min=1.16.0 +cmr "python app image-classification onnx" --input=computer_mouse.jpg + +cm rm cache -f +cmr "python app image-classification onnx" --input=computer_mouse.jpg --adr.onnxruntime.version_max=1.16.0 + + +cmr "python app image-classification onnx _cuda" --input=computer_mouse.jpg + +cmr "cm gui" --script="python app image-classification onnx" + +cm docker script "python app image-classification onnx" --input=computer_mouse.jpg +cm docker script "python app image-classification onnx" --input=computer_mouse.jpg -j -docker_it + +cmr "run common mlperf inference" --implementation=nvidia --model=bert-99 --category=datacenter --division=closed cm find script "run common mlperf inference" cm pull repo ctuning@cm-reproduce-research-projects cmr "reproduce paper micro-2023 victima _install_deps" cmr "reproduce paper micro-2023 victima _run" -... - ``` ```python diff --git a/cm/cmind/__init__.py b/cm/cmind/__init__.py index 55d7315691..315daab681 100644 --- a/cm/cmind/__init__.py +++ b/cm/cmind/__init__.py @@ -1,4 +1,4 @@ -__version__ = "1.6.0" +__version__ = "1.6.0.1" from cmind.core import access from cmind.core import error diff --git a/cmr.yaml b/cmr.yaml index 125a72fca5..b94693a602 100644 --- a/cmr.yaml +++ b/cmr.yaml @@ -2,4 +2,4 @@ alias: mlcommons@ck git: true prefix: cm-mlops uid: a4705959af8e447a -version: 1.5.4 +version: 1.6.0 diff --git a/docs/README.md b/docs/README.md index 91ffa1a468..31f3bda991 100644 --- a/docs/README.md +++ b/docs/README.md @@ -9,7 +9,7 @@ The goal is to provide a common, simple and human-readable interface to manage, and customize diverse AI/ML apps, benchmarks and research projects across continuously changing models, datasets, software and hardware from different vendors in a unified and automated way. -* [Getting Started tutorial](getting-started.md) +* [Getting Started Guide](getting-started.md) * [Introduction](introduction-cm.md) * [CM installation and customization](installation.md) * [Unified CLI and Python API](interface.md) diff --git a/docs/installation.md b/docs/installation.md index 7a6444bb69..c9504757cc 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -223,6 +223,13 @@ cm rm repo mlcommons@ck --all cm pull repo --url={URL of the fork of github.com/mlcommons/ck} ``` +If you want to use stable CM snapshots of reusable automation recipes (CM scripts), +you can download a stable repository from Zenodo: +```bash +cm rm repo mlcommons@ck --all +cm pull repo --url=https://zenodo.org/records/10576423/files/cm-mlops-repo.zip +``` + If you use CM scripts with Python outside containers, we suggest you to set up CM Python virtual environment as described [here](../cm-mlops/automation/script/README-extra.md#using-python-virtual-environments). diff --git a/docs/tutorials/mlperf-inference-power-measurement.md b/docs/tutorials/mlperf-inference-power-measurement.md index c45e19c1a8..661c179b08 100644 --- a/docs/tutorials/mlperf-inference-power-measurement.md +++ b/docs/tutorials/mlperf-inference-power-measurement.md @@ -6,8 +6,8 @@ ## Requirements -1. Power analyzer (any one [certified by SPEC PTDaemon](https://www.spec.org/power/docs/SPECpower-Device_List.html)). - Yokogawa is the one which most submitters have submitted with and a new single-channel model like 310E can cost around 3000$. +1. Power analyzer (anyone [certified by SPEC PTDaemon](https://www.spec.org/power/docs/SPECpower-Device_List.html)). + Yokogawa is the one that most submitters have submitted with and a new single-channel model like 310E can cost around 3000$. The [MLCommons taskforce on automation and reproducibility ](../taksforce.md) is also using the [Yokogawa 310E](https://tmi.yokogawa.com/solutions/products/power-analyzers/digital-power-meter-wt300e/) to automate and simplify MLPerf submissions. @@ -29,7 +29,7 @@ An adapter like [this](https://amzn.to/3Cl2TV5) can help avoid cutting the elect ![pages (14)](https://user-images.githubusercontent.com/4791823/210117283-82375460-5b3a-4e8a-bd85-9d33675a5843.png). -The director machine runs the `server.py` script and loads a server process which communicates with the SPEC PTDaemon. +The director machine runs the `server.py` script and loads a server process that communicates with the SPEC PTDaemon. When a client connects to it (using `client.py`), it in turn connects to the PTDaemon and initiates a measurement run. Once the measurement ends, the power log files are transferred to the client. @@ -52,12 +52,14 @@ to install the MLCommons CM automation tool. If you are having GitHub access to [MLCommons power](https://github.com/mlcommons/power) repository, PTDaemon should be automatically installed using the following CM command: + +PS: The below command will ask for `sudo` permission on Linux and should be run with administrator privilege on Windows (to do NTP time sync). ```bash cm run script --tags=mlperf,power,server --device_type=49 --device_port=/dev/usbtmc0 ``` * ``--interface_flag="-U" and `--device_port=1` (can change as per the USB slot used for connecting) can be used on Windows for USB connection -* `--device_type=49` corresponds to Yokogawa 310E and `ptd -h` should list the device_type for all supported devices. Location of `ptd` can be found using the below command -* `--device_port=20` and `--interface_flag="-g" can be used to connect to GPIB interface (currently supported only on Windows) with serial address set to 20 +* `--device_type=49` corresponds to Yokogawa 310E and `ptd -h` should list the device_type for all supported devices. The location of `ptd` can be found using the below command +* `--device_port=20` and `--interface_flag="-g" can be used to connect to GPIB interface (currently supported only on Windows) with the serial address set to 20 ```bash cat `cm find cache --tags=get,spec,ptdaemon`/cm-cached-state.json ```