DTS - Failed activities return a broken error message

[andystaples]

🐛 Describe the bug
In AzureStorage or MSSQL, if an activity function fails, the orchestrator gets back an exception with relevant information about how the activity failed.
In DTS, you get back None

🤔 Expected behavior
When an activity function fails, the yield statement from the orchestrator should throw an exception with relevant information about how the activity failed, including the message from the exception thrown by the activity.

☕ Steps to reproduce
Orchestrator and activity:

@bp.orchestration_trigger(context_name="context", orchestration="CatchActivityException")
def catch_activity_exception(context: df.DurableOrchestrationContext):
    try:
        yield context.call_activity('raise_exception', context.instance_id)
    except Exception as e:
        logging.error(f"Caught exception: {e}")
        return f"Caught exception: {e}"


@bp.activity_trigger(input_name="instance")
def raise_exception(instance: str) -> str:
    global attempt_count
    if instance not in attempt_count:
        attempt_count[instance] = 1
        raise InvalidOperationException(f"This activity failed")
    return "This activity succeeded"

The return value for Azure Storage/MSSQL:

Caught exception: Activity function 'raise_exception' failed: 

The return value for DTS:

Caught exception: None \n None

⚡If deployed to Azure
N/A

[philliphoff]

I ran into this as well, so took a deeper look. It appears that Python activity errors are encoded in the DT Core TaskFailedEvent using its Result and Details properties instead of the FailureDetails property (as does .NET (Isolated)), which is the property that is propagated to DTS. In fact, the gRPC flavor of the TaskFailedEvent defines only the FailureDetails property and not the Result or Details property. And, because those properties aren't propagated to DTS, they aren't propagated back in an orchestration's work history, from which the Python orchestration failure message is derived, and so ends up with None values.

A couple of options (and both is an option, too):

• This SDK should propagate FailureDetails in addition to Result and Details
• The AzureManagedBackend DT extension should, upon TaskFailedEvent, if FailureDetails is not populated, generate a gRPC FailureDetails property based on the Details and Result properties. Similarly, when a TaskFailedEvent is returned by DTS in event history, the extension should populate the Details and Result properties based on the contents in FailureDetails, if any.

[philliphoff]

The AzureManagedBackend DT extension should, upon TaskFailedEvent, if FailureDetails is not populated, generate a gRPC FailureDetails property based on the Details and Result properties. Similarly, when a TaskFailedEvent is returned by DTS in event history, the extension should populate the Details and Result properties based on the contents in FailureDetails, if any.

I was able to verify that this allows both the activity and orchestration failure to be viewed in the DTS dashboard.