feat: add automatic recovery for streaming ASR/TTS - #189
Open
pgowda1107 wants to merge 2 commits into
Open
Conversation
Add resilient streaming wrappers that automatically reconnect on transient gRPC failures (UNAVAILABLE, DEADLINE_EXCEEDED, INTERNAL, etc.). - riva/client/retry.py: shared retry utilities with exponential backoff - riva/client/asr.py: ResilientStreamingASR with audio lookback buffer and final-transcript deduplication - riva/client/tts.py: ResilientStreamingTTS with segment-level retry - riva/client/auth.py: default gRPC keepalive for faster dead-connection detection - scripts/asr/transcribe_file.py: --auto-recover, --max-retries, --lookback-seconds - scripts/tts/talk.py: --auto-recover, --max-retries - tests/unit/test_retry.py: unit tests for retry logic
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
| uri: str = "localhost:50051", | ||
| metadata: Optional[List[Tuple[str, str]]] = None, | ||
| options: Optional[List[Tuple[str, str]]] = [], | ||
| options: Optional[List[Tuple[str, Union[str, int]]]] = None, |
Contributor
There was a problem hiding this comment.
didn't get why this change is needed?
| "--auto-recover", | ||
| action="store_true", | ||
| help="Retry retryable streaming gRPC failures using a bounded audio lookback.", | ||
| ) |
Contributor
There was a problem hiding this comment.
instead of adding new arguments to the script, can we utilize generic custom_configuration argument to pass these as key value pairs?
|
|
||
|
|
||
|
|
||
| class ResilientStreamingASR: |
Contributor
There was a problem hiding this comment.
instead of new wrapper classes, is it possible to add this logic into existing recognize/synthesize functions? default disabled and only take effect when custom_configuration arguments are sent
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds optional automatic recovery for streaming ASR and TTS when a transient gRPC failure occurs (
UNAVAILABLE,DEADLINE_EXCEEDED,INTERNAL,RESOURCE_EXHAUSTED, orABORTED).Changes
File | Change -- | -- riva/client/retry.py | Shared retryable-status detection and exponential backoff with jitter. riva/client/asr.py | Adds ResilientStreamingASR, which reconnects and replays a bounded recent PCM-audio lookback buffer. Recovery is best effort; callers that require exactly-once transcript output should handle duplicate transcripts after reconnect. riva/client/tts.py | Adds ResilientStreamingTTS, which retries a failed text segment and buffers that segment’s responses before delivery so retrying does not duplicate already delivered audio. Segment size is the latency/recovery trade-off. scripts/asr/transcribe_file.py | Adds --auto-recover, --max-retries, and --lookback-seconds. scripts/tts/talk.py | Adds --auto-recover and --max-retries. tests/unit/test_retry.py | Adds unit coverage for retryable status detection, backoff behaviour, and TTS partial-stream failure recovery.Usage
Validation