NFR - ContractNegotiationManager - StateMachine delay as config value

[marcingajek-zf]

Feature Request

We need to decrease the delay introduced by the ContractNegotiationManager StateMachine of EDC.
The org.eclipse.dataspaceconnector.spi.retry.ExponentialWaitStrategy is given 5000 miliseconds as a default, hardcoded value.
As a consequence, the process of retrieving the asset through EDC can take several seconds.

Which Areas Would Be Affected?

ContractServiceExtension

Why Is the Feature Desired?

Huge delay introduced by StateMachine of the ContractServiceExtension

Solution Proposal

Make the value of org.eclipse.dataspaceconnector.contract.ContractServiceExtension#DEFAULT_ITERATION_WAIT configurable as one of the config values, similar to the org.eclipse.dataspaceconnector.contract.ContractServiceExtension#NEGOTIATION_CONSUMER_STATE_MACHINE_BATCH_SIZE.

Or maybe there is some other way to easly introduce the new, custom NegotiationWaitStrategy, that could be used in services like org.eclipse.dataspaceconnector.contract.negotiation.ConsumerContractNegotiationManagerImpl ?

[jimmarino]

The iteration should be configurable, but your description brings up another potential issue.

You indicated the process of retrieving an asset could take several seconds. The contract negotiation and transfer process are decoupled. Contract negotiation should be done upfront, on an infrequent basis. If a system is performing a contract negotiation for every asset request made by a client, and those requests are frequent, you probably should reconsider that design. In general, contract negotiation latency should not matter (within reason) as long as the system can scale out (which it is designed to do).

Similarly, the transfer process was designed with the same principles. If your asset requests are latency sensitive and frequent, you should consider using the synchronous API pattern supported by the data plane or modeling the assets as a single, non-finite transfer. Doing so will avoid introducing unnecessary overhead.

[jimmarino]

What does "short" mean, and why does the end-to-end process have to be "synchronous"?

With limited information, my guess is the issue is with how the problem is being modeled. If you need to retrieve data with low latencies, e.g., 99% of requests on the order of N milliseconds, you need to account for that in how you design an asset. Contract negotiation is, by nature, a relatively chatty protocol, and you need to perform that setup beforehand. Model the "asset" as a data stream (or generic non-finite data transfer) that the provider grants access to based on a single negotiation and transfer request sequence. Requests made by the client to transfer data would be part of the "on-going" transfer process.

Changing the delay will not solve your issue, no matter how short it is.

[marcingajek-zf]

Jim, sorry for late answer.
We have a task to create sth similar to search engine for the Assets available through EDC, served by multiple providers.
"Short" menas, that a user will wait for it in front of the GUI interface.

We have a registry services on the provider side, to look up for Assets. We can negotiate contracts uprfont for those services. Then we have a "search result" page with available items.
If user clicks in one of the items, we wan to retrieve it through EDC and show it to the user.

Keep in mind, that it is just a concept, a prototype. It may change in time.

When I've changed the harcoded values (ContractServiceExtension#DEFAULT_ITERATION_WAIT), and rebuild the EDC, I've been able to retrieve an asset in less then 500ms (locally), full process with contract negotiation (while before the change, the process took around 10-20 seconds).
And that is a why, having the "ContractServiceExtension#DEFAULT_ITERATION_WAIT" value configurable, may be helpfull.

[jimmarino]

Having the value configurable will certainly be helpful in some scenarios. However, as I mentioned, I would urge you to reconsider your design. What you described is not a realistic setup (you are running a single instance locally), and it is highly dependent on the latency of the provider connector. The complexity and latency will multiply as you add additional providers.

Also, the use case you described appears to be exactly what the CatalogCrawler is intended to handle. Is there a reason you are not using it? Have you looked at the MVD? There is no reason the UI should require synchronous interactions since it can easily be built on the existing asynchronous architecture.

[marcingajek-zf]

Thanks for the reply Jim,
I understand Your concerns and I will pass them to the authors of this design.

Is this the MVD You are referring to: https://github.com/eclipse-dataspaceconnector/MinimumViableDataspace ?
Can You pase a link to the CatalogCrawler repo/docu page? Never heard about it and I can not find any information.

[jimmarino]

Yes, as well as the IdentityHub repo. The CatalogCrawler is in the codebase, and there are a number of videos (Hackathon and tutorials) that show it in use.

[ndr-brt]

Actually, you can provide your own custom NegotiationWaitStrategy service implementation, so, from an extension you can do:

context.registerService(NegotiationWaitStrategy.class, new ExponentialWaitStrategy(customWaitPeriodInMillis)

I agree that the value should be configurable as well, issue and PR would be appreciated (maybe also for the TransferWaitStrategy)

[marcingajek-zf]

Hi, thanks for the hint. Looks promising.
Just one question: what happens if more than one strategy of the same type is define by multiple extensions?
Fist one found will be applied?

I can create an issue page, and also create a PR to configure the ContractServiceExtension#DEFAULT_ITERATION_WAIT as one of the "edc.negotiation.xxx" values.

[ndr-brt]

If multiple are defined, one will be used, there's no way to determine the order upfront, however for multiple services registered of the same type, a warning log will be printed to the console.
As a rule of thumb, better to avoid that

[marcingajek-zf]

Clear, thank You.