Skip to content

Using rocTX with MIGraphX

Jump to bottom
Cagri edited this page Nov 3, 2021 · 2 revisions

Introduction

MIGraphX provides perf knob via migraphx-driver, which provides wall clock time for each operation of a given model. For detailed performance debugging and optimization, it is preferred to have the exact GPU timing. One of the functionalities of rocTracer library is the code annotation API, named rocTX. By using rocTX, it is possible to wrap around GPU execution per instruction and get more accurate performance numbers.
Take a look below for the difference between the actual execution of "conv_bias_relu" on GPU and the information host provides. (Light orange ones)

Motivation

MIGraphX can be run with rocTX markers, where each op is marked at the beginning and end of execution. This allows finer grain timing information.

Methodology

The general steps are given below. The steps are provided to explain how it works. We also provide a script that provides this functionality with one command line. Please see the rocTX Helper Script section.

  1. Run rocprof binary with migraphx-driver with trace knob. Example CMD: rocprof --hip-trace --roctx-trace --flush-rate 10ms --timestamp on -d rocout /opt/rocm/bin/migraphx-driver trace $ONNX_PATH --onnx --gpu
  2. After running rocprof, it will provide an output at the folder given with -d knob. (/rocout/ at example above). We will need to read the output files (HCC, HIP, ROCTX outputs) and create a JSON file from this. To create a JSON file, we utilize the rocmProfileData repository:
    1. Clone rocmProfileData: git clone https://github.com/ROCmSoftwarePlatform/rocmProfileData.git
    2. cd into cd /rocpd_python/
    3. python setup.py install
  3. Process the output:
    1. python -m rocpd.rocprofiler_import --ops_input_file hcc_ops_trace.txt --api_input_file hip_api_trace.txt --roctx_input_file roctx_trace.txt trace.rpd
    2. python /tmp/roctx_temp/rocmProfileData/rpd2tracing.py trace.rpd trace.json
  4. cd /AMDMIGraphX/tools
  5. python roctx.py --parse --json_path ../trace.json

rocTX Helper Script

Simply provide --onnx_file and if needed --migraphx_args to roctx.py file in AMDMIGraphX\tools. Example command line is given below:
python roctx.py --run '--onnx --gpu ../../../AMDMIGraphX/fcn-resnet50-11.onnx' --out outfolder

After this, an output similar to the following will be printed on your terminal.

Clone this wiki locally