Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

希望增加熔断扩展能力 #1030

Open
Hookers opened this issue Jul 20, 2023 · 3 comments
Open

希望增加熔断扩展能力 #1030

Hookers opened this issue Jul 20, 2023 · 3 comments
Assignees

Comments

@Hookers
Copy link

Hookers commented Jul 20, 2023

https://github.com/cloudwego/kitex/blob/develop/pkg/circuitbreak/cbsuite.go 中的 ServicePanel() 函数,初始化 CBSuite.serviceControl 时,对于熔断判断的 Control.GetErrorType(https://github.com/cloudwego/kitex/blob/develop/pkg/circuitbreak/circuitbreak.go
目前只有 ErrorTypeOnServiceLevel ,希望能够扩展 GetErrorType 传入自定义函数,实现自定义熔断的失败判断,如自定义根据 resp 的 StatusCode 进行失败判断并熔断
即在获取 CBSuite.ServicePanel 时,希望能多一个扩展函数的传入参数,传入 函数

GetErrorType func(ctx context.Context, request, response interface{}, err error) ErrorType

从而替换 ErrorTypeOnServiceLevel

对应的源码为:

cbs := circuitbreak.NewCBSuite(cp.RPCInfo2Key)
cbs.ServicePanel()

// ServicePanel return return cb Panel of service
func (s *CBSuite) ServicePanel() circuitbreaker.Panel {
	if s.servicePanel == nil {
		s.initServiceCB()
	}
	return s.servicePanel
}

func (s *CBSuite) initServiceCB() {
	if s.servicePanel != nil && s.serviceControl != nil {
		return
	}
	if s.genServiceCBKey == nil {
		s.genServiceCBKey = RPCInfo2Key
	}
	opts := circuitbreaker.Options{
		ShouldTripWithKey: s.svcTripFunc,
	}
	s.servicePanel, _ = circuitbreaker.NewPanel(s.onServiceStateChange, opts)

	svcKey := func(ctx context.Context, request interface{}) (serviceCBKey string, enabled bool) {
		ri := rpcinfo.GetRPCInfo(ctx)
		serviceCBKey = s.genServiceCBKey(ri)
		cbConfig, _ := s.serviceCBConfig.LoadOrStore(serviceCBKey, defaultCBConfig)
		enabled = cbConfig.(CBConfig).Enable
		return
	}
	s.serviceControl = &Control{
		GetKey:       svcKey,
		GetErrorType: ErrorTypeOnServiceLevel,
		DecorateError: func(ctx context.Context, request interface{}, err error) error {
			return kerrors.ErrServiceCircuitBreak
		},
	}
}
@felix021
Copy link
Contributor

听起来是个合理需求,而且不止 serviceControl, 也应当考虑 instanceControl 的扩展;

你可以提个具体的方案建议吗?

@felix021 felix021 self-assigned this Jul 21, 2023
@Hookers
Copy link
Author

Hookers commented Jul 24, 2023

听起来是个合理需求,而且不止 serviceControl, 也应当考虑 instanceControl 的扩展;

你可以提个具体的方案建议吗?

@felix021 一点拙见:
方式一:
最为简单但并不那么优雅的方式就是提供一个 cbs.ServiceControl() 的重载函数,把

GetErrorType func(ctx context.Context, request, response interface{}, err error) ErrorType

作为参数传进去,这样在自定义 RetryContainer

retryC := retry.NewRetryContainerWithCB(cbs.ServiceControl(), cbs.ServicePanel())

的时候,能够直接将函数传入

方式二:
将函数作为option传入,参考重试的自定义函数(这种方式可能导致 serviceControl、instanceControl的GetErrorType函数都被覆盖为一种,不过好像也不重要,因为 RetryContainer 好像只能在 serviceControl 和 instanceControl 中二选一?)

func WithSpecifiedResultRetry(rr *retry.ShouldResultRetry) Option {
	return Option{F: func(o *client.Options, di *utils.Slice) {
		if rr == nil || (rr.RespRetry == nil && rr.ErrorRetry == nil) {
			panic(fmt.Errorf("WithSpecifiedResultRetry: invalid '%+v'", rr))
		}
		di.Push(fmt.Sprintf("WithSpecifiedResultRetry(%+v)", rr))
		o.RetryWithResult = rr
	}}
}

将熔断的判断逻辑的函数作为option传入,传入option之后,可对 retry.Container.cbContainer.cbCtl.GetErrorType 进行写操作,即最终实现的代码如下(实际通过反射+option方式实践过,也在生产上跑过,完全没问题,但是反射太trick了,希望能得到官方的支持):

type GetErrorType func(ctx context.Context, request, response interface{}, err error) circuitbreak.ErrorType
func WithRetryGetErrorType(getErrorTypeFunc GetErrorType) Option {
	return Option{F: func(o *client.Options, di *utils.Slice) {
                 if getErrorTypeFunc == nil {
			panic(fmt.Errorf("WithRetryGetErrorType: invalid '%+v'", getErrorTypeFunc))
		}
                 if o.RetryContainer == nil || o.RetryContainer.cbContainer == nil || o.RetryContainer.cbContainer.cbCtl == nil { // 原来就有才设置
			return
		}
                di.Push(fmt.Sprintf("WithRetryGetErrorType(%+v)", getErrorTypeFunc))
                o.RetryContainer.cbContainer.cbCtl.GetErrorType = getErrorTypeFunc // 通过某种方法使 GetErrorType 可设置
	}}
}

另外想探讨的是:在开启mesh的时候,instanceControl 其实打开也没意义了吧?因为kitex在调用之前,也拿不到下游的 ip,无法针对特定ip做熔断了?

@felix021
Copy link
Contributor

关于字节内部的相关能力(mesh),我们飞书上聊吧(felix.fengmin),

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants