Skip to content

[fix](fe) Derive table stream schema from base table - #67130

Open
luwei16 wants to merge 1 commit into
apache:masterfrom
luwei16:fix/doris-28093-dynamic-stream-schema
Open

[fix](fe) Derive table stream schema from base table#67130
luwei16 wants to merge 1 commit into
apache:masterfrom
luwei16:fix/doris-28093-dynamic-stream-schema

Conversation

@luwei16

@luwei16 luwei16 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #67109

Related PR: N/A

Problem Summary: Table Streams persisted a copy of their base table schema at creation time. Light schema changes updated the base table but left Stream metadata stale. Keep the persisted Stream schema empty, derive every Stream schema API from the current base table under its read lock, and fail explicitly when the base table is unavailable.

Release note

Table Stream metadata now follows the current base table schema after supported schema changes.

Check List (For Author)

  • Test: Unit Test
    • CreateTableStreamTest
    • ExplainTableStreamPlanTest
    • TableStreamBaseTableInfoTest
  • Behavior changed: Yes (Table Stream schema metadata is derived dynamically from the base table.)
  • Does this need documentation: No

### What problem does this PR solve?

Issue Number: close apache#67109

Related PR: N/A

Problem Summary: Table Streams persisted a copy of their base table schema at creation time. Light schema changes updated the base table but left Stream metadata stale. Keep the persisted Stream schema empty, derive every Stream schema API from the current base table under its read lock, and fail explicitly when the base table is unavailable.

### Release note

Table Stream metadata now follows the current base table schema after supported schema changes.

### Check List (For Author)

- Test: Unit Test
    - `CreateTableStreamTest`
    - `ExplainTableStreamPlanTest`
    - `TableStreamBaseTableInfoTest`
- Behavior changed: Yes (Table Stream schema metadata is derived dynamically from the base table.)
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@luwei16

luwei16 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 17082 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit de20fba4353163f092b939d5d440ebf3db72e977, data reload: false

------ Round 1 ----------------------------------
============================================
q1	18071	3111	3058	3058
q2	2449	273	227	227
q3	10354	904	517	517
q4	4738	245	202	202
q5	7688	554	382	382
q6	141	118	94	94
q7	545	504	383	383
q8	9375	853	929	853
q9	3849	2439	2411	2411
q10	6524	859	731	731
q11	620	194	184	184
q12	656	260	197	197
q13	18179	1579	1158	1158
q14	159	153	143	143
q15	q16	431	402	375	375
q17	1317	880	878	878
q18	3251	2254	2271	2254
q19	1205	894	762	762
q20	362	279	205	205
q21	5316	1837	1876	1837
q22	332	267	231	231
Total cold run time: 95562 ms
Total hot run time: 17082 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3435	3353	3369	3353
q2	517	400	362	362
q3	2257	2398	2219	2219
q4	1199	1170	898	898
q5	2205	2157	2107	2107
q6	169	130	87	87
q7	1081	912	873	873
q8	1619	1416	1426	1416
q9	3184	3136	3150	3136
q10	1893	1813	1657	1657
q11	367	280	253	253
q12	462	437	352	352
q13	1509	1539	1159	1159
q14	178	184	174	174
q15	q16	398	404	358	358
q17	3673	3376	3294	3294
q18	4994	4657	4878	4657
q19	926	830	840	830
q20	1002	938	853	853
q21	3670	3003	3151	3003
q22	403	359	321	321
Total cold run time: 35141 ms
Total hot run time: 31362 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82782 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit de20fba4353163f092b939d5d440ebf3db72e977, data reload: false

query5	4328	417	326	326
query6	422	139	129	129
query7	5026	434	231	231
query8	698	131	112	112
query9	8649	2903	2911	2903
query10	448	226	191	191
query11	5431	1044	929	929
query12	130	73	71	71
query13	1196	445	331	331
query14	6094	2225	2096	2096
query14_1	1995	1983	1970	1970
query15	175	114	113	113
query16	3267	387	378	378
query17	828	470	373	373
query18	2173	335	253	253
query19	245	144	109	109
query20	73	74	73	73
query21	742	107	88	88
query22	5838	5627	5482	5482
query23	7189	6312	6058	6058
query23_1	6091	5970	6267	5970
query24	7433	1099	800	800
query24_1	769	772	761	761
query25	437	301	268	268
query26	1267	228	135	135
query27	2746	444	261	261
query28	4633	1514	1538	1514
query29	932	600	396	396
query30	409	165	137	137
query31	836	417	354	354
query32	132	79	70	70
query33	502	214	192	192
query34	1040	846	495	495
query35	408	403	344	344
query36	583	563	518	518
query37	131	75	75	75
query38	1036	855	844	844
query39	515	501	482	482
query39_1	510	483	475	475
query40	229	84	75	75
query41	87	48	49	48
query42	69	71	73	71
query43	258	247	219	219
query44	1067	543	560	543
query45	120	104	98	98
query46	796	861	559	559
query47	836	761	724	724
query48	302	275	226	226
query49	626	236	192	192
query50	771	256	190	190
query51	8324	8314	8173	8173
query52	65	66	60	60
query53	194	197	145	145
query54	220	174	152	152
query55	86	58	54	54
query56	214	161	159	159
query57	1044	667	644	644
query58	207	161	146	146
query59	1225	1226	1101	1101
query60	273	193	166	166
query61	118	111	114	111
query62	375	199	196	196
query63	166	138	139	138
query64	2332	662	559	559
query65	1677	1623	1663	1623
query66	1803	257	211	211
query67	9821	10297	9758	9758
query68	2794	1204	722	722
query69	564	226	198	198
query70	666	619	614	614
query71	263	174	172	172
query72	2634	1718	1584	1584
query73	676	609	338	338
query74	1561	1248	1156	1156
query75	1193	1121	981	981
query76	1871	735	536	536
query77	331	266	219	219
query78	3875	3730	3226	3226
query79	2183	815	565	565
query80	1325	331	293	293
query81	587	159	136	136
query82	668	131	96	96
query83	287	210	193	193
query84	348	109	88	88
query85	869	359	290	290
query86	414	180	176	176
query87	1037	993	889	889
query88	2846	2120	2135	2120
query89	319	204	181	181
query90	1977	131	128	128
query91	134	121	99	99
query92	80	70	64	64
query93	1429	1088	657	657
query94	645	278	224	224
query95	523	273	227	227
query96	805	556	280	280
query97	1042	1090	1090	1090
query98	146	141	142	141
query99	547	352	312	312
Total cold run time: 187696 ms
Total hot run time: 82782 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.78 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit de20fba4353163f092b939d5d440ebf3db72e977, data reload: false

query1	0.02	0.00	0.00
query2	0.36	0.03	0.03
query3	0.76	0.11	0.11
query4	2.18	0.11	0.10
query5	0.18	0.16	0.16
query6	1.88	0.68	0.70
query7	0.04	0.01	0.00
query8	0.05	0.03	0.02
query9	0.83	0.21	0.21
query10	0.36	0.35	0.34
query11	0.32	0.11	0.11
query12	0.22	0.12	0.12
query13	0.32	0.31	0.32
query14	0.46	0.47	0.46
query15	0.40	0.35	0.36
query16	0.22	0.23	0.25
query17	0.68	0.69	0.66
query18	0.19	0.17	0.17
query19	1.60	1.18	1.19
query20	0.02	0.01	0.01
query21	16.69	0.15	0.12
query22	4.77	0.05	0.04
query23	16.91	0.26	0.10
query24	18.12	0.34	0.31
query25	0.12	0.05	0.03
query26	0.72	0.17	0.13
query27	0.04	0.03	0.03
query28	3.69	0.58	0.30
query29	12.66	3.17	2.62
query30	0.25	0.11	0.12
query31	2.89	0.36	0.16
query32	3.58	0.33	0.23
query33	1.38	1.43	1.39
query34	15.40	2.14	1.75
query35	1.72	1.76	1.70
query36	0.47	0.29	0.29
query37	0.27	0.04	0.04
query38	0.04	0.03	0.02
query39	0.30	0.02	0.02
query40	0.22	0.08	0.07
query41	0.35	0.02	0.03
query42	0.26	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 111.98 s
Total hot run time: 14.78 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 58.97% (23/39) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Table Stream] Stream metadata retains a dropped column after light DROP COLUMN

2 participants