You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
PR #532 introduced the option to measure inference time for many frameworks, but it introduced a lot of code duplication along the way. The PR should be "finished" by:
introducing a common interface that abstracts away a lot of the duplicated boilerplate code
extend inference time measurement support to frameworks that don't have it yet
add documentation
Another open issue (that may be tackled independently altogether), is that the inference time measurement can take a considerable amount of time. This may lead to the benchmark aborting a job because of the overhead, while it would otherwise would have been successful. There should be the option to do "best effort" inference time measurement within the time limit, or perhaps configure it dynamically. I don't have a concrete idea on how to do this robustly.
The text was updated successfully, but these errors were encountered:
PR #532 introduced the option to measure inference time for many frameworks, but it introduced a lot of code duplication along the way. The PR should be "finished" by:
Another open issue (that may be tackled independently altogether), is that the inference time measurement can take a considerable amount of time. This may lead to the benchmark aborting a job because of the overhead, while it would otherwise would have been successful. There should be the option to do "best effort" inference time measurement within the time limit, or perhaps configure it dynamically. I don't have a concrete idea on how to do this robustly.
The text was updated successfully, but these errors were encountered: