[DSIP-76] Support ssl in netty #16673

xdu-chenrj · 2024-09-27T12:39:33Z

close #16688

SbloodyS

Need to discuss this first in #16672

wangxj3 · 2024-09-28T13:55:53Z

I think you miss the code of master-server .The task of other system sholud call other system in master-server .

SbloodyS

This requires adding at least one of the following tests to ensure stability.

cluster-test
api-test
e2e-test

xdu-chenrj · 2024-10-12T07:36:24Z

This requires adding at least one of the following tests to ensure stability.

cluster-test

api-test

e2e-test

I will complete it later

xdu-chenrj · 2024-10-12T08:45:28Z

This requires adding at least one of the following tests to ensure stability.

cluster-test

api-test

e2e-test

Is it adding UT, testing functions in the class?

SbloodyS · 2024-10-12T08:51:47Z

Is it adding UT, testing functions in the class?

UT and these tests are different types of tests. You can find the corresponding examples in the following links.

xdu-chenrj · 2024-10-12T08:53:36Z

Is it adding UT, testing functions in the class?

UT and these tests are different types of tests. You can find the corresponding examples in the following links.

cluster-test

api-test

e2e

Thank you

SbloodyS · 2024-10-17T01:30:33Z

I found that it has not reached a coverage rate of 60%. How can I increase coverage? Is it possible to increase arbitrarily?

Yes. You should add more UT.

SbloodyS

Missing docs, unit-test.

...er-e2e-case/src/test/java/org/apache/dolphinscheduler/e2e/cases/ssl/SslShellTaskE2ETest.java

SbloodyS · 2024-10-17T01:34:06Z

.github/workflows/e2e.yml

+          - name: SslShellTaskE2ETest
+            class: org.apache.dolphinscheduler.e2e.cases.ssl.SslShellTaskE2ETest


This test case is not running normally, the cluster is not started. You should check it.
https://github.com/apache/dolphinscheduler/actions/runs/11365198466?pr=16673

This test case is not running normally, the cluster is not started. You should check it. https://github.com/apache/dolphinscheduler/actions/runs/11365198466?pr=16673

can you tell me where the log of cluster startup exception is? how to determine if a cluster is starting up properly? thank you

You can take a look at this. https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-e2e/README.md

This is still unaddressed. Don't resolve it.

I found that all the checks have been passed, why hasn't this problem been resolved yet

SslShellTaskE2ETest is not executed at all in e2e test cases.

...ct-base/src/main/java/org/apache/dolphinscheduler/extract/base/config/NettyServerConfig.java

...er-e2e-case/src/test/java/org/apache/dolphinscheduler/e2e/cases/ssl/SslShellTaskE2ETest.java

xdu-chenrj · 2024-10-18T05:35:59Z

Missing docs, unit-test.

ok

SbloodyS · 2024-10-18T05:37:45Z

.github/workflows/e2e-k8s.yml

+    strategy:
+      matrix:
+        case:
+          - name: SslShellTaskE2ETest
+            class: org.apache.dolphinscheduler.e2e.cases.ssl.SslShellTaskE2ETest
+


Why adding this?

add an entrance for running E2E in consolidation, k8s.yml, where the running image is a standalone one

I don't think this is meaningful for the current e2e-k8s tests. Please remove this modification.

.github/workflows/e2e.yml

docs/docs/en/guide/installation/cluster.md

ruanwenjun

It's better to add a new class NettyRpcConfig, and NettySslConfig as a field in NettyRpcConfig, we will add more configuration in the latest.

ruanwenjun · 2024-10-18T06:58:40Z

...lphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertServer.java

+        RegistryConfiguration.class,
+        NettySslConfig.class})


Suggested change

RegistryConfiguration.class,

NettySslConfig.class})

RegistryConfiguration.class})

ruanwenjun · 2024-10-18T07:00:19Z

...inscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/MasterServer.java

+        RegistryConfiguration.class,
+        NettySslConfig.class})


Suggested change

RegistryConfiguration.class,

NettySslConfig.class})

RegistryConfiguration.class})

ruanwenjun · 2024-10-18T07:00:34Z

...inscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/MasterServer.java

+    @Autowired
+    NettySslConfig nettySslConfig;
+


Suggested change

@Autowired

NettySslConfig nettySslConfig;

ruanwenjun · 2024-10-18T07:01:01Z

...inscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/WorkerServer.java

+        RegistryConfiguration.class,
+        NettySslConfig.class})


Suggested change

RegistryConfiguration.class,

NettySslConfig.class})

RegistryConfiguration.class})

ruanwenjun · 2024-10-18T07:01:10Z

...inscheduler-worker/src/main/java/org/apache/dolphinscheduler/server/worker/WorkerServer.java

+    @Autowired
+    NettySslConfig nettySslConfig;


Suggested change

@Autowired

NettySslConfig nettySslConfig;

Gallardot

If this DSIP is to be merged, I believe there is more work to be done.

How will certificates be managed? How will certificate expiration be handled? Is there a certificate rotation and renewal mechanism?
If deploy on K8S through Helm charts, how can SSL be enabled? Are there relevant configuration options in the Helm charts? Can E2E-K8S add corresponding test cases for this?

Gallardot · 2024-10-18T07:07:05Z

.github/workflows/e2e-k8s.yml

+    strategy:
+      matrix:
+        case:
+          - name: SslShellTaskE2ETest
+            class: org.apache.dolphinscheduler.e2e.cases.ssl.SslShellTaskE2ETest
+


I don't think this is meaningful for the current e2e-k8s tests. Please remove this modification.

sonarcloud · 2024-10-19T04:36:00Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

xdu-chenrj requested review from songjianet, SbloodyS, ruanwenjun and caishunfeng as code owners September 27, 2024 12:39

github-actions bot added UI ui and front end related backend labels Sep 27, 2024

github-actions bot assigned xdu-chenrj Sep 27, 2024

SbloodyS added the don't merge label Sep 27, 2024

SbloodyS reviewed Sep 27, 2024

View reviewed changes

SbloodyS changed the title ~~support other business systems to run in DS~~ [DSIP] Support other business systems to run in DS Sep 27, 2024

xdu-chenrj closed this Oct 8, 2024

xdu-chenrj force-pushed the dev branch from 505bf13 to 9a58eca Compare October 8, 2024 11:53

tmp store

4ab065a

xdu-chenrj reopened this Oct 11, 2024

github-actions bot removed the UI ui and front end related label Oct 11, 2024

wangxj3 changed the title ~~[DSIP] Support other business systems to run in DS~~ [DSIP] Support ssl in netty Oct 12, 2024

wangxj3 changed the title ~~[DSIP] Support ssl in netty~~ [DSIP-76] Support ssl in netty Oct 12, 2024

xdu-chenrj and others added 4 commits October 12, 2024 09:20

format the code

9d0515b

Merge branch 'dev' into dev

7c5b8c2

fix ut

71a167a

Merge branch 'dev' into dev

60fc3ee

SbloodyS reviewed Oct 12, 2024

View reviewed changes

SbloodyS removed the don't merge label Oct 12, 2024

SbloodyS added this to the 3.3.0 milestone Oct 12, 2024

SbloodyS added the DSIP label Oct 12, 2024

add SslShellTaskE2ETest

006581a

github-actions bot added the CI&CD label Oct 16, 2024

SbloodyS requested changes Oct 17, 2024

View reviewed changes

SbloodyS added miss:docs missing documents in PR miss:tests missing unit tests in PR labels Oct 17, 2024

add doc and fix package

46e9db3

xdu-chenrj requested a review from EricGao888 as a code owner October 17, 2024 03:07

github-actions bot added the document label Oct 17, 2024

xdu-chenrj added 7 commits October 17, 2024 11:13

format doc

32d0edb

change docker-compose name

0333f83

remove unnecessary content

332fdfd

fix e2e-test

289dd80

change e2e-test workflow name ad task name

a23a59c

delete unnecessary functions

be8df43

change SslPythonTaskE2ETest

a4ed1d0

wangxj3 previously approved these changes Oct 18, 2024

View reviewed changes

...er-e2e-case/src/test/java/org/apache/dolphinscheduler/e2e/cases/ssl/SslShellTaskE2ETest.java Outdated Show resolved Hide resolved

change SslShellTaskE2ETest

a9638b4

xdu-chenrj dismissed wangxj3’s stale review via a9638b4 October 18, 2024 01:38

xdu-chenrj added 3 commits October 18, 2024 09:43

change ssl-test docker-compose

60ed520

chanage e2e-k8s.yml

7a0e835

chanage

3bc797e

SbloodyS reviewed Oct 18, 2024

View reviewed changes

ruanwenjun reviewed Oct 18, 2024

View reviewed changes

Gallardot reviewed Oct 18, 2024

View reviewed changes

xdu-chenrj and others added 3 commits October 18, 2024 16:44

fix doc and remove unnecessary modifications

190546b

fix

9607b2d

Merge branch 'dev' into dev

932d5ea

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DSIP-76] Support ssl in netty #16673

[DSIP-76] Support ssl in netty #16673

xdu-chenrj commented Sep 27, 2024 •

edited by SbloodyS

Loading

SbloodyS left a comment

wangxj3 commented Sep 28, 2024

SbloodyS left a comment

xdu-chenrj commented Oct 12, 2024

xdu-chenrj commented Oct 12, 2024

SbloodyS commented Oct 12, 2024 •

edited

Loading

xdu-chenrj commented Oct 12, 2024

SbloodyS commented Oct 17, 2024

SbloodyS left a comment

SbloodyS Oct 17, 2024

xdu-chenrj Oct 17, 2024

SbloodyS Oct 17, 2024

SbloodyS Oct 18, 2024

xdu-chenrj Oct 18, 2024

SbloodyS Oct 18, 2024

xdu-chenrj commented Oct 18, 2024

SbloodyS Oct 18, 2024

xdu-chenrj Oct 18, 2024

Gallardot Oct 18, 2024

ruanwenjun left a comment

ruanwenjun Oct 18, 2024

ruanwenjun Oct 18, 2024

ruanwenjun Oct 18, 2024

ruanwenjun Oct 18, 2024

ruanwenjun Oct 18, 2024

Gallardot left a comment •

edited

Loading

Gallardot Oct 18, 2024

sonarcloud bot commented Oct 19, 2024

		- name: SslShellTaskE2ETest
		class: org.apache.dolphinscheduler.e2e.cases.ssl.SslShellTaskE2ETest

	RegistryConfiguration.class,
	NettySslConfig.class})
	RegistryConfiguration.class})

[DSIP-76] Support ssl in netty #16673

Are you sure you want to change the base?

[DSIP-76] Support ssl in netty #16673

Conversation

xdu-chenrj commented Sep 27, 2024 • edited by SbloodyS Loading

SbloodyS left a comment

Choose a reason for hiding this comment

wangxj3 commented Sep 28, 2024

SbloodyS left a comment

Choose a reason for hiding this comment

xdu-chenrj commented Oct 12, 2024

xdu-chenrj commented Oct 12, 2024

SbloodyS commented Oct 12, 2024 • edited Loading

xdu-chenrj commented Oct 12, 2024

SbloodyS commented Oct 17, 2024

SbloodyS left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xdu-chenrj commented Oct 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruanwenjun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Gallardot left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Oct 19, 2024

Quality Gate passed

xdu-chenrj commented Sep 27, 2024 •

edited by SbloodyS

Loading

SbloodyS commented Oct 12, 2024 •

edited

Loading

Gallardot left a comment •

edited

Loading