Salesforce Destination Using Pubsub #146
Conversation
anna-cross
force-pushed
the
destination_pubsub
branch
from
5da8a78
to
8a7b2ee
Compare
anna-cross
force-pushed
the
destination_pubsub
branch
from
c326cf2
to
55fea9d
Compare
anna-cross
changed the title
cmd/connector/main.go
Outdated
| @@ -16,14 +16,16 @@ package main | |||
|
|
|||
| import ( | |||
| sf "github.com/conduitio-labs/conduit-connector-salesforce" | |||
| sfDesinationPubsub "github.com/conduitio-labs/conduit-connector-salesforce/destination" | |||
There was a problem hiding this comment.
you don't need to override the imports.
| sfDesinationPubsub "github.com/conduitio-labs/conduit-connector-salesforce/destination" | |
| "github.com/conduitio-labs/conduit-connector-salesforce/destination" |
cmd/connector/main.go
Outdated
| sdk "github.com/conduitio/conduit-connector-sdk" | ||
| ) | ||
|
|
||
| func main() { | ||
| sdk.Serve(sdk.Connector{ | ||
| NewSpecification: sf.Specification, | ||
| NewSource: sfSourcePubsub.NewSource, | ||
| NewDestination: nil, | ||
| NewDestination: sfDesinationPubsub.NewDestination, |
There was a problem hiding this comment.
| NewDestination: sfDesinationPubsub.NewDestination, | |
| NewDestination: destination.NewDestination, |
| config.Config | ||
|
|
||
| // Topic is Salesforce event or topic to write record | ||
| TopicName string `json:"topicName"` |
There was a problem hiding this comment.
| TopicName string `json:"topicName"` | |
| TopicNames string `json:"topicNames"` |
There was a problem hiding this comment.
Right now we only allow one topic for destination, we can add multi event/topic like we did for source later
There was a problem hiding this comment.
we can have the name topicNames now, and only support one name, so we don't have to deprecate another parameter later.
pubsub/client.go
Outdated
| for _, r := range records { | ||
| data := r.Payload.After | ||
|
|
||
| var dataMap map[string]interface{} |
There was a problem hiding this comment.
You don't need to do this, r.Payload.After needs to be opencdc.StructuredData which is essentially map[string]any and already serialized to go types. Doing this JSON unmarshal from bytes all kind of bad things can happen.
pubsub/mock_pub_sub_client_test.go
Outdated
| @@ -0,0 +1,480 @@ | |||
| // Code generated by mockery. DO NOT EDIT. | |||
There was a problem hiding this comment.
This mock and the subscribe mock in this folder are the same as the ones in pubsub/source, so likely you can remove the pubsub/source folder
| func (c Config) Validate(_ context.Context) (Config, error) { | ||
| var errs []error | ||
|
|
||
| if len(c.TopicName) == 0 { |
There was a problem hiding this comment.
we can add the "required" validation to the config instead
| if err := sdk.Util.ParseConfig( | ||
| ctx, | ||
| cfg, | ||
| &c, |
There was a problem hiding this comment.
you can use &d.config here instead, no need for c
| @@ -100,25 +101,25 @@ func (a oauth) Login() (*LoginResponse, error) { | |||
| strings.NewReader(body.Encode()), | |||
| ) | |||
| if err != nil { | |||
| return nil, fmt.Errorf("failed to make login req: %w", err) | |||
| return nil, errors.Errorf("failed to make login req: %s", err) | |||
There was a problem hiding this comment.
might be missing something here, but why are we using %s instead of wrapping the error with %w?
| events = append(events, &event) | ||
| } | ||
|
|
||
| publishRequest := eventbusv1.PublishRequest{ |
There was a problem hiding this comment.
Is it possible that this method would publish some events and then fail, so some are published and some are not?
Description
Per customers request, we are adding a destination for single event publishing using pubsub api. The code is also refactor so that destination can also use pubsub client we utilized on source.
Fixes # (https://github.com/meroxa/mdpx/issues/2958)
Quick checks: