Skip to content

feat(test): add LLM test support abstraction for graceful skipping - #26

Merged
cursor[bot] merged 1 commit into
mainfrom
cursor/llm-test-support-4ca0
Aug 8, 2026
Merged

feat(test): add LLM test support abstraction for graceful skipping#26
cursor[bot] merged 1 commit into
mainfrom
cursor/llm-test-support-4ca0

Conversation

@venkateshsakamuri-lab

Copy link
Copy Markdown
Contributor

Summary

This PR introduces a test support abstraction for LLM-dependent tests, allowing them to skip gracefully when LLM credentials are not configured instead of failing with cryptic errors.

Problem

Previously, integration tests that required LLM configuration (like ChatPromptIntegrationTest) would fail with "agent runtime hit an internal execution failure" when LLM credentials weren't set. This made it impossible to get a clean test run without full LLM configuration.

Solution

Added a new com.dbaagent.support test package with three components:

1. @RequiresLlm Annotation

@RequiresLlm(chat = true, embedding = false)
class ChatPromptIntegrationTest extends BaseIntegrationTest {
    // Tests will skip if chat LLM is not configured
}
  • Works at class or method level
  • Supports independent chat and embedding requirements
  • Tests are skipped (not failed) when LLM is not configured

2. LlmAvailabilityCondition

JUnit 5 ExecutionCondition that checks DEEPSQL_CHAT_* and DEEPSQL_EMBEDDING_* environment variables using the same resolution logic as production LlmConfigResolver.

3. LlmTestSupport

Spring component for use within integration tests that need more control:

@Autowired
private LlmTestSupport llmTestSupport;

@Test
void testWithOptionalLlm() {
    if (llmTestSupport.isChatAvailable()) {
        // test with LLM
    } else {
        // fallback behavior
    }
}

Changes

  • New files:

    • backend/src/test/java/com/dbaagent/support/RequiresLlm.java
    • backend/src/test/java/com/dbaagent/support/LlmAvailabilityCondition.java
    • backend/src/test/java/com/dbaagent/support/LlmTestSupport.java
    • backend/src/test/java/com/dbaagent/support/LlmAvailabilityConditionTest.java
  • Updated files:

    • backend/src/test/java/com/dbaagent/integration/BaseIntegrationTest.java - Added LlmTestSupport integration and convenience methods
    • backend/src/test/java/com/dbaagent/integration/ChatPromptIntegrationTest.java - Added @RequiresLlm annotation

Test Results

Before this change, ChatPromptIntegrationTest had 20+ failures when LLM wasn't configured.

After this change:

Tests run: 23, Failures: 0, Errors: 0, Skipped: 23

All tests skip cleanly with a clear message about the missing configuration.

How to Use

For new tests requiring LLM:

@RequiresLlm  // Requires chat LLM by default
class MyLlmTest extends BaseIntegrationTest {
    @Test
    void myTest() {
        // This test will skip if chat LLM is not configured
    }
}

For tests requiring both chat and embedding:

@RequiresLlm(chat = true, embedding = true)
class MyFullLlmTest { ... }

For programmatic checks:

@Test
void testWithFallback() {
    if (llmTestSupport.isChatAvailable()) {
        // Full test path
    } else {
        // Degraded test path
    }
}
Open in Web Open in Cursor 

Add a new test support package with utilities for handling LLM-dependent tests:

- @RequiresLlm annotation: marks tests that need LLM configuration
  - Supports chat and embedding requirements independently
  - Tests are skipped (not failed) when LLM is not configured
  - Works at class or method level

- LlmAvailabilityCondition: JUnit 5 ExecutionCondition
  - Checks DEEPSQL_CHAT_* and DEEPSQL_EMBEDDING_* environment variables
  - Uses the same resolution logic as production LlmConfigResolver

- LlmTestSupport: Spring component for integration tests
  - Uses actual LlmConfigResolver to check both env vars and DB config
  - Provides requireChat(), requireEmbedding() methods for test assertions
  - Includes diagnostic describeConfiguration() method

Updated BaseIntegrationTest:
- Autowires LlmTestSupport for integration tests
- Adds convenience methods: isChatLlmAvailable(), requireChatLlm(), etc.

Updated ChatPromptIntegrationTest:
- Added @RequiresLlm(chat = true) annotation
- All 23 tests now skip gracefully when LLM is not configured
- Improved Javadoc documentation

This allows the test suite to run cleanly in environments without LLM
credentials, while still validating LLM-dependent functionality when
credentials are available.

Co-authored-by: Venkat SF <venkatesh.sakamuri@stayflexi.com>
@cursor
cursor Bot merged commit ddbea0b into main Aug 8, 2026
7 checks passed
@cursor
cursor Bot deleted the cursor/llm-test-support-4ca0 branch August 8, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants