feat: Add support for `TorchTensorRTModule` in Dynamo [1 / x] #2003

gs-olive · 2023-06-08T19:58:27Z

Description

Rename TRTModuleNext to TorchTensorRTModule across the repository, and move the source directory to dynamo
Update imports across the repository
Refactor convert_module code to support conversion to a TorchTensorRTModule
Add tests for TorchTensorRTModule functionality in Dynamo

Fixes #2019

Type of change

New feature (non-breaking change which adds functionality)

Checklist:

[ x ] My code follows the style guidelines of this project (You can use the linters)
[ x ] I have performed a self-review of my own code
[ x ] I have commented my code, particularly in hard-to-understand areas and hacks
[ x ] I have made corresponding changes to the documentation
[ x ] I have added tests to verify my fix or my feature
[ x ] New and existing unit tests pass locally with my changes
[ x ] I have added the relevant labels to my PR in so that relevant reviewers are notified

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

py/torch_tensorrt/__init__.py

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

py/torch_tensorrt/dynamo/backend/__init__.py

py/torch_tensorrt/dynamo/__init__.py

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

gs-olive · 2023-07-05T22:00:59Z

py/torch_tensorrt/dynamo/fx_ts_compat/lower.py

+        use_python_runtime: Whether to strictly use Python runtime or C++ runtime. To auto-select a runtime
+            based on C++ dependency presence (preferentially choosing C++ runtime if available), leave the
+            argument as None


Added a new compilation argument called use_python_runtime to replace use_experimental_rt and use_experimental_fx_rt. This new compilation argument has three potential settings/implications:

True: Strictly use Python runtime

False: Strictly use C++ runtime

None [DEFAULT]: Automatically detect presence of C++ dependency and select C++ runtime if available, else default to Python runtime

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

gs-olive · 2023-07-07T00:23:09Z

py/torch_tensorrt/dynamo/test/test_dynamo_backend.py

+        "optimization_level": 1,
+        "min_block_size": 8,


Decreased optimization level (from 3 to 1) and increased block size (from 5 to 8) to improve compilation times and reduce segmentation in models, pending upcoming converter implementations.

- Rename `TRTModuleNext` to `TorchTensorRTModule` across the repository, and move the source directory to `dynamo` - Update imports across the repository - Refactor `convert_module` code to support conversion to a `TorchTensorRTModule` - Add logging information about which runtime is being used in Dynamo compile - Add tests for `TorchTensorRTModule` functionality in Dynamo

- Default automatically to test for presence of C++ dependency and use appropriate runtime if not specified by the user - Improve test timing for TRT Backend testing, which now utilizes C++ runtime

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

gs-olive added the component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths label Jun 8, 2023

gs-olive requested a review from narendasan June 8, 2023 19:58

gs-olive self-assigned this Jun 8, 2023

facebook-github-bot added cla signed fx labels Jun 8, 2023

github-actions bot added component: api [Python] Issues re: Python API component: fx component: tests Issues re: Tests labels Jun 8, 2023

github-actions bot requested a review from yinghai June 8, 2023 19:58

gs-olive removed the request for review from yinghai June 8, 2023 19:58

github-actions bot approved these changes Jun 8, 2023

View reviewed changes

gs-olive commented Jun 8, 2023

View reviewed changes

py/torch_tensorrt/__init__.py Outdated Show resolved Hide resolved

gs-olive force-pushed the dynamo_compile_trt_module_next branch from 7df445e to 8ec3f01 Compare June 8, 2023 21:07

github-actions bot approved these changes Jun 8, 2023

View reviewed changes

gs-olive force-pushed the dynamo_compile_trt_module_next branch from 8ec3f01 to 76a85d9 Compare June 8, 2023 21:53

github-actions bot approved these changes Jun 8, 2023

View reviewed changes

gs-olive force-pushed the dynamo_compile_trt_module_next branch from 76a85d9 to 08e9d5a Compare June 8, 2023 22:05

github-actions bot approved these changes Jun 8, 2023

View reviewed changes

narendasan reviewed Jun 13, 2023

View reviewed changes

py/torch_tensorrt/dynamo/backend/__init__.py Outdated Show resolved Hide resolved

py/torch_tensorrt/dynamo/backend/__init__.py Outdated Show resolved Hide resolved

gs-olive commented Jun 13, 2023

View reviewed changes

py/torch_tensorrt/dynamo/__init__.py Outdated Show resolved Hide resolved

gs-olive requested a review from narendasan June 13, 2023 21:09

github-actions bot approved these changes Jun 13, 2023

View reviewed changes

narendasan changed the base branch from dynamo_compile_interpreter_update to main June 13, 2023 21:13

gs-olive force-pushed the dynamo_compile_trt_module_next branch from e9ec251 to 487ba93 Compare June 28, 2023 02:46

github-actions bot approved these changes Jun 28, 2023

View reviewed changes

gs-olive force-pushed the dynamo_compile_trt_module_next branch from 487ba93 to 49cd3a6 Compare June 28, 2023 21:45

github-actions bot approved these changes Jun 28, 2023

View reviewed changes

gs-olive force-pushed the dynamo_compile_trt_module_next branch from 49cd3a6 to e2594b6 Compare June 29, 2023 00:47

github-actions bot approved these changes Jun 29, 2023

View reviewed changes

github-actions bot approved these changes Jul 5, 2023

View reviewed changes

gs-olive commented Jul 5, 2023

View reviewed changes

gs-olive force-pushed the dynamo_compile_trt_module_next branch from 7247ce7 to f31e528 Compare July 5, 2023 22:48

github-actions bot approved these changes Jul 5, 2023

View reviewed changes

gs-olive mentioned this pull request Jul 6, 2023

🐛 [Bug] Encountered TRT Error in Dynamo compile path #2072

Closed

gs-olive force-pushed the dynamo_compile_trt_module_next branch from f31e528 to de669eb Compare July 6, 2023 22:47

github-actions bot approved these changes Jul 6, 2023

View reviewed changes

gs-olive force-pushed the dynamo_compile_trt_module_next branch from de669eb to c0f0a22 Compare July 6, 2023 22:49

github-actions bot approved these changes Jul 6, 2023

View reviewed changes

gs-olive commented Jul 7, 2023

View reviewed changes

gs-olive added 3 commits July 6, 2023 17:29

fix: Add new TRT 8.6 features to Dynamo compile [3 / x] (#1973)

1c8d93a

feat: Automatically detect C++ dependency presence

1756b12

- Default automatically to test for presence of C++ dependency and use appropriate runtime if not specified by the user - Improve test timing for TRT Backend testing, which now utilizes C++ runtime

gs-olive force-pushed the dynamo_compile_trt_module_next branch from c0f0a22 to 1756b12 Compare July 7, 2023 00:29

github-actions bot approved these changes Jul 7, 2023

View reviewed changes

gs-olive merged commit b5dbc11 into main Jul 7, 2023
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add support for `TorchTensorRTModule` in Dynamo [1 / x] #2003

feat: Add support for `TorchTensorRTModule` in Dynamo [1 / x] #2003

gs-olive commented Jun 8, 2023 •

edited

Loading

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

gs-olive Jul 5, 2023

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

gs-olive Jul 7, 2023

github-actions bot left a comment

github-actions bot left a comment

feat: Add support for TorchTensorRTModule in Dynamo [1 / x] #2003

feat: Add support for TorchTensorRTModule in Dynamo [1 / x] #2003

Conversation

gs-olive commented Jun 8, 2023 • edited Loading

Description

Type of change

Checklist:

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

gs-olive Jul 5, 2023

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

gs-olive Jul 7, 2023

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

feat: Add support for `TorchTensorRTModule` in Dynamo [1 / x] #2003

feat: Add support for `TorchTensorRTModule` in Dynamo [1 / x] #2003

gs-olive commented Jun 8, 2023 •

edited

Loading