Parser changes to handle MatMulIntegerToFloat #3445

TedThemistokleous · 2024-09-16T02:48:25Z

Changes to MatMul parser to handle the Microsoft Contrib operator MatMulintegarToFloat

Since we have the scale and zero points in our operands we can just perform a multiplied after int8 biases are added and then insert a regular dot on the scaled input values which should give the same output as the input data types.

Able to leverage the existing set of tests for matmul

Needs #3526 as there's a bug with dequantizelinear this has uncovered

TedThemistokleous · 2024-09-16T02:49:32Z

TODO:

Add Parser tests for err cases
Add parser tests for base case
Add parser test for bias and zero point cases
Add verify tests for all of the above

codecov · 2024-09-16T20:56:55Z

Codecov Report

Attention: Patch coverage is 90.54054% with 7 lines in your changes missing coverage. Please review.

Project coverage is 92.17%. Comparing base (2803cb3) to head (c0c8120).
Report is 4 commits behind head on develop.

Files with missing lines	Patch %	Lines
src/onnx/parse_matmul.cpp	90.54%	7 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3445      +/-   ##
===========================================
- Coverage    92.17%   92.17%   -0.01%     
===========================================
  Files          512      512              
  Lines        21387    21459      +72     
===========================================
+ Hits         19714    19779      +65     
- Misses        1673     1680       +7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Updated parser to handle bias case as well as bad scale conditions Initial float/half tests bad scale tests bad bias tests

avoid tidy screaming about complexity

Use dequantizelinear which elminates the need to add in shifts due to int8/uint8 mismatches still needs parser tests

Needed to update initial tests and gen onnx

lakhinderwalia · 2024-10-17T23:04:57Z

src/onnx/parse_matmul.cpp

+
+        if(not(contains(supported_dq_types, scale_arg->get_shape().type())))
+        {
+            MIGRAPHX_THROW("PARSE_QUANT_DOT_SCALDED: Scales must be float or half_type");


Also, would this message be proper for MatMul operator?

It won't reach there as its gated by whether the operator contains the scaled inputs. This variant of MatMul also includes the dequantize to convert the quantized input types to float

lakhinderwalia · 2024-10-17T23:10:56Z

src/onnx/parse_matmul.cpp

+    {
+        has_valid_scale_bias = false;
+
+        if(args.size() > index)


I don't see an index defined in MatMulIntegerToFloat. Is this for some other operator. Thanks.

Its argument index. We're doing the check here so its done for every arg

lakhinderwalia · 2024-10-17T23:16:59Z

src/onnx/parse_matmul.cpp

+                MIGRAPHX_THROW("PARSE_QUANT_DOT_SCALED: Bias have same dim as matrix B column");
+            }
+
+            has_valid_scale_bias = true;


As against invalid? ;-)

If scale bias doesn't exist there isn't a bias at the end of the matmulintergertofloat added then.

lakhinderwalia · 2024-10-17T23:19:18Z

src/onnx/parse_matmul.cpp

+                                         instruction_ref& zp_a0,
+                                         bool no_zp)
+    {
+        if(no_zp)


It seems only zp_a0 needs to be in the if clause..?

No the second input as its bound by column of the input vector (which is 1-d always).

I broke this out instead of adding it inline to encapsulate logic.

lakhinderwalia · 2024-10-17T23:21:51Z

src/onnx/parse_matmul.cpp

+        return dequantized_op;
+    }
+
+    static instruction_ref handle_scaled_output(const onnx_parser::node_info& info,


Too many parameters. Ideally they should be handled by a struct parameter.

They're the same amount of a parameters gathered by the operator. These are all needed for dequantize steps and adding the proper unsqueeze->transpose paths. Order matters here with respect to matrix input A or B

lakhinderwalia · 2024-10-17T23:34:57Z

src/onnx/parse_matmul.cpp

@@ -173,12 +333,20 @@ struct parse_matmul : op_parser<parse_matmul>
        }

        auto is_quant_dot = opd.op_name == "quant_dot";
+        auto has_scales   = opd.op_name == "quant_dot_scaled";


A little confusing naming convention. Between quant_dots and Matmul**. And then there is has_scales: which is presumably also a quant_dot.

What do you suggest I name it? quant_dot_dequant? This operator essentially takes in quantized input and dequantizes the output.

lakhinderwalia · 2024-10-17T23:36:57Z

src/onnx/parse_matmul.cpp

@@ -200,23 +368,50 @@ struct parse_matmul : op_parser<parse_matmul>
            auto s0_lens        = a0->get_shape().lens();
            auto s1_lens        = a1->get_shape().lens();

-            if(not is_quant_dot and args.size() > 2)
+            if(not is_quant_dot and args.size() > 2 and not has_scales)


Would it be simpler to just check if it is just a dot, instead of looking for quant_dot and quant_dot_scaled, as this clause seems to be doing? Thanks.

Sure that can easily be swapped

lakhinderwalia · 2024-10-17T23:38:28Z

src/onnx/parse_matmul.cpp


            // Only INT8 or UINT8 type currently supported
            std::set<migraphx::shape::type_t> supported_types = {migraphx::shape::uint8_type,
                                                                 migraphx::shape::int8_type};
            const auto a0_type                                = a0->get_shape().type();
            const auto a1_type                                = a1->get_shape().type();

-            if(is_quant_dot and
+            if((is_quant_dot or has_scales) and


If it has_scales, then it is perhaps not a MATLMULINTEGER: as shown in the exception message.

Sure, simple to just add op.name() here as part of the string. Both MatMulInteger and MatMulIntegerToFloat have the same error on this

migraphx-bot · 2024-10-18T03:32:27Z

Test	Batch	Rate new c0c812	Rate old 275f85	Diff	Compare
torchvision-resnet50	64	3,261.05	3,260.33	0.02%	✅
torchvision-resnet50_fp16	64	6,995.46	6,996.93	-0.02%	✅
torchvision-densenet121	32	2,436.48	2,436.13	0.01%	✅
torchvision-densenet121_fp16	32	4,085.19	4,080.65	0.11%	✅
torchvision-inceptionv3	32	1,639.24	1,639.01	0.01%	✅
torchvision-inceptionv3_fp16	32	2,762.50	2,759.60	0.11%	✅
cadene-inceptionv4	16	776.44	775.88	0.07%	✅
cadene-resnext64x4	16	811.17	811.11	0.01%	✅
slim-mobilenet	64	7,533.85	7,539.27	-0.07%	✅
slim-nasnetalarge	64	211.54	211.54	0.00%	✅
slim-resnet50v2	64	3,505.27	3,505.47	-0.01%	✅
bert-mrpc-onnx	8	1,147.46	1,151.85	-0.38%	✅
bert-mrpc-tf	1	462.50	463.50	-0.22%	✅
pytorch-examples-wlang-gru	1	423.10	411.56	2.80%	✅
pytorch-examples-wlang-lstm	1	408.41	388.27	5.19%	🔆
torchvision-resnet50_1	1	780.36	764.51	2.07%	✅
cadene-dpn92_1	1	398.48	402.09	-0.90%	✅
cadene-resnext101_1	1	384.08	381.69	0.62%	✅
onnx-taau-downsample	1	342.65	342.54	0.03%	✅
dlrm-criteoterabyte	1	33.35	33.35	0.01%	✅
dlrm-criteoterabyte_fp16	1	52.79	52.74	0.08%	✅
agentmodel	1	8,480.76	8,186.92	3.59%	🔆
unet_fp16	2	58.90	58.82	0.14%	✅
resnet50v1_fp16	1	969.99	953.22	1.76%	✅
resnet50v1_int8	1	1,016.65	1,003.11	1.35%	✅
bert_base_cased_fp16	64	1,172.15	1,170.57	0.13%	✅
bert_large_uncased_fp16	32	363.63	363.63	-0.00%	✅
bert_large_fp16	1	199.98	201.70	-0.85%	✅
distilgpt2_fp16	16	2,200.56	2,202.28	-0.08%	✅
yolov5s	1	534.60	537.82	-0.60%	✅
tinyllama	1	43.46	43.44	0.06%	✅
vicuna-fastchat	1	170.74	173.47	-1.57%	✅
whisper-tiny-encoder	1	418.63	418.26	0.09%	✅
whisper-tiny-decoder	1	428.98	436.61	-1.75%	✅

Check results before merge 🔆

migraphx-bot · 2024-10-18T03:32:29Z

✅ bert-mrpc-onnx: PASSED: MIGraphX meets tolerance

✅ bert-mrpc-tf: PASSED: MIGraphX meets tolerance

✅ pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

✅ pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

✅ torchvision-resnet50_1: PASSED: MIGraphX meets tolerance

✅ cadene-dpn92_1: PASSED: MIGraphX meets tolerance

✅ cadene-resnext101_1: PASSED: MIGraphX meets tolerance

✅ dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

✅ agentmodel: PASSED: MIGraphX meets tolerance

✅ unet: PASSED: MIGraphX meets tolerance

✅ resnet50v1: PASSED: MIGraphX meets tolerance

✅ bert_base_cased_fp16: PASSED: MIGraphX meets tolerance

🔴bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output

✅ bert_large: PASSED: MIGraphX meets tolerance

✅ yolov5s: PASSED: MIGraphX meets tolerance

✅ tinyllama: PASSED: MIGraphX meets tolerance

✅ vicuna-fastchat: PASSED: MIGraphX meets tolerance

✅ whisper-tiny-encoder: PASSED: MIGraphX meets tolerance

✅ whisper-tiny-decoder: PASSED: MIGraphX meets tolerance

✅ distilgpt2_fp16: PASSED: MIGraphX meets tolerance

TedThemistokleous added 2 commits September 16, 2024 02:31

Initial commit of parser changes to handle MatMulIntegerToFloat

b19ce16

Update output to handle dot and broadcasted instead of mul

ae9f722

TedThemistokleous self-assigned this Sep 16, 2024

TedThemistokleous added onnxruntime PR changes interaction between MIGraphX and Onnxruntime Onnx Operators Adding or modifying an Onnx Operator in the MIGraphX codebase UAI labels Sep 16, 2024

Fix typo

7f62a33

TedThemistokleous added 2 commits September 17, 2024 03:30

Add parser tests and updated gen onnx

92d8ea4

Updated parser to handle bias case as well as bad scale conditions Initial float/half tests bad scale tests bad bias tests

Handle scaled output result better

cdb307d

avoid tidy screaming about complexity

TedThemistokleous force-pushed the add_matmulintegertofloat_contrib_op branch from 74f8ae0 to cdb307d Compare September 17, 2024 15:48

Merge branch 'develop' into add_matmulintegertofloat_contrib_op

f912e61

TedThemistokleous mentioned this pull request Oct 11, 2024

Fix dequantizelinaer simple case with no zero point present #3526

Closed

TedThemistokleous and others added 2 commits October 11, 2024 17:45

Merge branch 'develop' into add_matmulintegertofloat_contrib_op

02c3918

Update fixes for parser tests

3ca3e6a

Use dequantizelinear which elminates the need to add in shifts due to int8/uint8 mismatches still needs parser tests

TedThemistokleous requested review from lakhinderwalia and shivadbhavsar October 11, 2024 23:26

TedThemistokleous marked this pull request as ready for review October 11, 2024 23:26

TedThemistokleous requested a review from causten as a code owner October 11, 2024 23:26

TedThemistokleous added 4 commits October 16, 2024 02:46

Update parser to use dequantizelinear

547826d

Add test case for zero point in matmulintegertofloat

c6d8679

Needed to update initial tests and gen onnx

Add parser tests for scalar inputs in matmulintegertofloat

6d89fdd

Add parser for bias with zero points

c0c8120

lakhinderwalia reviewed Oct 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parser changes to handle MatMulIntegerToFloat #3445

Parser changes to handle MatMulIntegerToFloat #3445

TedThemistokleous commented Sep 16, 2024 •

edited

Loading

TedThemistokleous commented Sep 16, 2024 •

edited

Loading

codecov bot commented Sep 16, 2024 •

edited

Loading

lakhinderwalia Oct 17, 2024

lakhinderwalia Oct 17, 2024

TedThemistokleous Oct 18, 2024

lakhinderwalia Oct 17, 2024

TedThemistokleous Oct 18, 2024

lakhinderwalia Oct 17, 2024

TedThemistokleous Oct 18, 2024

lakhinderwalia Oct 17, 2024

TedThemistokleous Oct 18, 2024

lakhinderwalia Oct 17, 2024

TedThemistokleous Oct 18, 2024

lakhinderwalia Oct 17, 2024

TedThemistokleous Oct 18, 2024

lakhinderwalia Oct 17, 2024

TedThemistokleous Oct 18, 2024

lakhinderwalia Oct 17, 2024

TedThemistokleous Oct 18, 2024

migraphx-bot commented Oct 18, 2024

migraphx-bot commented Oct 18, 2024

Parser changes to handle MatMulIntegerToFloat #3445

Are you sure you want to change the base?

Parser changes to handle MatMulIntegerToFloat #3445

Conversation

TedThemistokleous commented Sep 16, 2024 • edited Loading

TedThemistokleous commented Sep 16, 2024 • edited Loading

codecov bot commented Sep 16, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

migraphx-bot commented Oct 18, 2024

migraphx-bot commented Oct 18, 2024

TedThemistokleous commented Sep 16, 2024 •

edited

Loading

TedThemistokleous commented Sep 16, 2024 •

edited

Loading

codecov bot commented Sep 16, 2024 •

edited

Loading