openvino/workbench 2022.1 Accuracy configuration failed with Custom Dataset in ImageNet format

[josejacomeb]

Hi everyone, (sorry for not posting this issue in the Intel Community Forums, I can't log to the side due to a platform error) I'm working with a classification task for 3 classes based in MobileNetv2 which I saved as a Saved Model, I'd like to measure the performance drop of the quantization with the OpenVINO Workbench.
The conversion and inference of the model in the OV Workbench is fine, however I've uploaded a custom dataset in ImageNet format as explained here Dataset Types/ImageNet, basically my last test to confirm that it's possible an OV error was with three images and a single val.txt file as shown below:

THREE_IMAGES_DATASET
│   0000.png
│   0001.png
│   0002.png
│   val.txt

Where val.txt contains:

0000.png 0
0001.png 1
0002.png 2 

When I'm trying to create the Accuracy Report, the following error message appear

[setupvars.sh] OpenVINO environment initialized
[RUN COMMAND] + python3 /opt/intel/openvino_2022/tools/workbench/wb/main/scripts/accuracy_tool/check_accuracy.py --yml-config /opt/intel/openvino_2022/tools/workbench/wb/data/accuracy_artifacts/6/scripts/accuracy.config.yaml --log-dir /opt/intel/openvino_2022/tools/workbench/wb/data/accuracy_artifacts/6/job_artifacts --profile
2023-01-05 17:03:11.671895: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/opt/intel/openvino_2022.1.0.643/extras/opencv/python/cv2/../../bin:/opt/intel/openvino/extras/opencv/lib:/opt/intel/openvino/tools/compile_tool:/opt/intel/openvino/runtime/3rdparty/tbb/lib::/opt/intel/openvino/runtime/3rdparty/hddl/lib:/opt/intel/openvino/runtime/lib/intel64::/opt/intel/openvino_2022.1.0.643/extras/opencv/python/cv2/../../bin:/opt/intel/openvino/extras/opencv/lib:/opt/intel/openvino/runtime/lib/intel64:/opt/intel/openvino/tools/compile_tool:/opt/intel/openvino/runtime/3rdparty/tbb/lib:/opt/intel/openvino/runtime/3rdparty/hddl/lib
2023-01-05 17:03:11.671913: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/usr/local/lib/python3.8/dist-packages/defusedxml/__init__.py:30: DeprecationWarning: defusedxml.cElementTree is deprecated, import from defusedxml.ElementTree instead.
from . import cElementTree
Traceback (most recent call last):
File "/opt/intel/openvino_2022/tools/workbench/wb/main/scripts/accuracy_tool/check_accuracy.py", line 129, in <module>
main()
File "/opt/intel/openvino_2022/tools/workbench/wb/main/scripts/accuracy_tool/check_accuracy.py", line 110, in main
model_evaluator.process_dataset(
File "/usr/local/lib/python3.8/dist-packages/openvino/tools/accuracy_checker/evaluators/model_evaluator.py", line 418, in process_dataset_sync
self._process_batch_results(
File "/usr/local/lib/python3.8/dist-packages/openvino/tools/accuracy_checker/evaluators/model_evaluator.py", line 452, in _process_batch_results
_, profile_result = self.metric_executor.update_metrics_on_batch(
File "/usr/local/lib/python3.8/dist-packages/openvino/tools/accuracy_checker/metrics/metric_executor.py", line 104, in update_metrics_on_batch
results[input_id] = self.update_metrics_on_object(single_annotation, single_prediction)
File "/usr/local/lib/python3.8/dist-packages/openvino/tools/accuracy_checker/metrics/metric_executor.py", line 82, in update_metrics_on_object
metric_results.append(metric.metric_fn.submit(annotation, prediction))
File "/usr/local/lib/python3.8/dist-packages/openvino/tools/accuracy_checker/metrics/metric.py", line 248, in submit
metric_result = self.update(annotation_, prediction_)
File "/usr/local/lib/python3.8/dist-packages/openvino/tools/accuracy_checker/metrics/classification.py", line 89, in update
accuracy = self.accuracy.update(annotation.label, prediction.top_k(self.top_k))
File "/usr/local/lib/python3.8/dist-packages/openvino/tools/accuracy_checker/representation/classification_representation.py", line 60, in top_k
return np.argpartition(self.scores, -k)[-k:]
File "<__array_function__ internals>", line 5, in argpartition
File "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py", line 832, in argpartition
return _wrapfunc(a, 'argpartition', kth, axis=axis, kind=kind, order=order)
File "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py", line 58, in _wrapfunc
return bound(*args, **kwds)
ValueError: kth(=-2) out of bounds (3)

Via Model / Visualize Model Output I can confirm that an image of the dataset is correctly classified

I have some hundreds of images in my testing dataset, which I wouldn't like to label by hand in the Workbench, so What can I do?

My testing specs:

• Ubuntu 22.04
• GPU: Yes
• OV Workbench: 2022.1
• Running command via docker: docker run -p 0.0.0.0:5665:5665 --name workbench --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -1) -it openvino/workbench:2022.1

Best Regards,