Skip to content

[feature](dictionary) Add FLAT layout for dictionary - #67116

Open
zwy991114 wants to merge 1 commit into
apache:masterfrom
zwy991114:feature/flat_layout_dict
Open

[feature](dictionary) Add FLAT layout for dictionary#67116
zwy991114 wants to merge 1 commit into
apache:masterfrom
zwy991114:feature/flat_layout_dict

Conversation

@zwy991114

@zwy991114 zwy991114 commented Aug 25, 2026

Copy link
Copy Markdown

What problem does this PR solve?

Problem Summary:
Doris dictionary (CREATE DICTIONARY) previously supported only the HASH_MAP and IP_TRIE layouts. This PR adds a ClickHouse-style FLAT layout for integer-keyed dictionaries.

In a FLAT dictionary, a single non-negative integer key is used directly as an array index into column-wise value storage, with a presence bitmap distinguishing absent keys from present-with-default keys. This avoids the per-lookup hashing that HASH_MAP performs, making it a good fit for dense integer surrogate keys (e.g. dimension-table enrichment).

Release note

Support FLAT layout in CREATE DICTIONARY for integer-keyed dictionaries.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zwy991114

Copy link
Copy Markdown
Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16831 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bb9073e19f5630a9069d6348d188680c67b27e57, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17387	3008	3005	3005
q2	1907	237	173	173
q3	10224	903	527	527
q4	4666	256	206	206
q5	7707	571	395	395
q6	141	119	93	93
q7	536	508	391	391
q8	9278	973	895	895
q9	3509	2431	2436	2431
q10	6513	903	725	725
q11	406	199	176	176
q12	675	262	208	208
q13	18063	1555	1179	1179
q14	163	148	132	132
q15	q16	446	402	374	374
q17	824	766	708	708
q18	3168	2345	2280	2280
q19	2511	956	788	788
q20	394	284	200	200
q21	5613	1714	1800	1714
q22	348	266	231	231
Total cold run time: 94479 ms
Total hot run time: 16831 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3467	3339	3339	3339
q2	206	214	152	152
q3	2282	2408	2210	2210
q4	1206	1187	893	893
q5	2234	2142	2157	2142
q6	180	128	91	91
q7	1013	917	863	863
q8	1591	1405	1398	1398
q9	3126	3114	3113	3113
q10	1910	1802	1668	1668
q11	360	271	252	252
q12	460	432	342	342
q13	1501	1553	1170	1170
q14	194	170	163	163
q15	q16	407	403	362	362
q17	1048	1023	1015	1015
q18	5058	4513	4858	4513
q19	866	835	844	835
q20	982	934	821	821
q21	3789	3098	3280	3098
q22	406	359	325	325
Total cold run time: 32286 ms
Total hot run time: 28765 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 83172 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bb9073e19f5630a9069d6348d188680c67b27e57, data reload: false

query5	4283	411	333	333
query6	375	137	127	127
query7	4958	395	239	239
query8	291	126	116	116
query9	8690	2911	2893	2893
query10	390	218	183	183
query11	5402	1046	923	923
query12	115	69	75	69
query13	1213	447	316	316
query14	6016	2200	2089	2089
query14_1	1984	1964	1953	1953
query15	175	118	112	112
query16	913	366	360	360
query17	792	477	343	343
query18	2352	330	232	232
query19	160	138	107	107
query20	80	67	69	67
query21	201	100	84	84
query22	5490	5461	5316	5316
query23	6809	6155	6210	6155
query23_1	6037	6305	6313	6305
query24	7190	1083	780	780
query24_1	765	789	770	770
query25	443	304	268	268
query26	1234	250	131	131
query27	2783	419	256	256
query28	4640	1509	1517	1509
query29	950	453	371	371
query30	249	162	134	134
query31	871	404	332	332
query32	100	49	47	47
query33	457	211	185	185
query34	1016	801	488	488
query35	410	435	346	346
query36	583	550	537	537
query37	116	76	67	67
query38	1022	856	814	814
query39	518	497	490	490
query39_1	460	479	476	476
query40	201	90	76	76
query41	53	50	51	50
query42	75	72	71	71
query43	240	240	207	207
query44	1024	552	560	552
query45	115	104	105	104
query46	802	832	550	550
query47	771	771	732	732
query48	314	302	233	233
query49	576	243	193	193
query50	775	266	194	194
query51	7916	7952	8144	7952
query52	66	66	61	61
query53	188	204	149	149
query54	215	210	173	173
query55	75	59	55	55
query56	208	160	185	160
query57	656	649	669	649
query58	189	183	157	157
query59	1191	1225	1078	1078
query60	240	179	197	179
query61	117	120	115	115
query62	349	199	193	193
query63	174	146	149	146
query64	2851	726	583	583
query65	1691	1664	1618	1618
query66	1880	263	237	237
query67	10293	9764	9990	9764
query68	2790	1210	756	756
query69	333	233	199	199
query70	667	636	611	611
query71	243	180	170	170
query72	2319	1805	1739	1739
query73	632	617	357	357
query74	1577	1257	1157	1157
query75	1206	1112	993	993
query76	2236	742	562	562
query77	258	284	228	228
query78	4266	3940	3296	3296
query79	1466	831	598	598
query80	1229	337	319	319
query81	495	160	136	136
query82	635	126	97	97
query83	326	221	199	199
query84	293	117	94	94
query85	901	429	348	348
query86	405	172	176	172
query87	1032	976	909	909
query88	2784	2107	2090	2090
query89	294	196	174	174
query90	1892	132	128	128
query91	130	123	101	101
query92	55	49	46	46
query93	1425	1108	727	727
query94	669	241	217	217
query95	517	326	236	236
query96	761	557	267	267
query97	1094	1051	1029	1029
query98	149	135	134	134
query99	410	343	314	314
Total cold run time: 176701 ms
Total hot run time: 83172 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 16.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bb9073e19f5630a9069d6348d188680c67b27e57, data reload: false

query1	0.00	0.01	0.01
query2	0.11	0.06	0.06
query3	0.34	0.20	0.19
query4	1.60	0.17	0.19
query5	0.25	0.23	0.22
query6	1.15	0.39	0.39
query7	0.03	0.01	0.00
query8	0.07	0.06	0.05
query9	0.41	0.27	0.27
query10	0.39	0.39	0.38
query11	0.26	0.15	0.15
query12	0.28	0.15	0.15
query13	0.36	0.36	0.36
query14	0.46	0.45	0.45
query15	0.48	0.40	0.41
query16	0.28	0.28	0.28
query17	0.66	0.67	0.68
query18	0.25	0.23	0.23
query19	1.20	1.10	1.15
query20	0.02	0.01	0.01
query21	15.39	0.28	0.25
query22	4.87	0.11	0.10
query23	15.91	0.39	0.24
query24	2.54	0.46	0.35
query25	0.13	0.08	0.08
query26	0.73	0.22	0.17
query27	0.07	0.07	0.07
query28	3.55	0.65	0.37
query29	12.42	3.30	2.67
query30	0.36	0.21	0.17
query31	2.75	0.40	0.21
query32	3.42	0.38	0.28
query33	1.52	1.39	1.51
query34	15.38	2.41	1.94
query35	1.88	1.85	1.86
query36	0.50	0.37	0.34
query37	0.09	0.06	0.06
query38	0.07	0.05	0.05
query39	0.05	0.05	0.04
query40	0.13	0.11	0.10
query41	0.11	0.06	0.05
query42	0.06	0.05	0.04
query43	0.05	0.05	0.04
Total cold run time: 90.58 s
Total hot run time: 16.4 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 76.00% (19/25) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.09% (34417/45233)
Line Coverage 61.05% (387932/635421)
Region Coverage 57.25% (325696/568883)
Branch Coverage 58.04% (148510/255864)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 2.69% (9/335) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants