Skip to content

❌ AI Integration Tests Failed #5343

New issue
New issue
Open
Open
❌ AI Integration Tests Failed#5343
Labels
ai-integration-test-failureautomated
@github-actions

Description

@github-actions
github-actions
bot
opened on Jan 20, 2026

AI Integration Test Failure

Date: January 21, 2026 at 03:57 AM UTC
Workflow Run: https://github.com/getsentry/sentry-python/actions/runs/21196586966

📊 Summary

Metric Value
Total Tests 32
✓ Passed 15
✗ Failed 17
Duration 147.36s

🧪 Test Matrix

SDK 1-simple 10-binary-content-redaction 2-multi-step 9-message-truncation
py/anthropic ✓ ✓ ✓ ✗
py/google-genai ✓ ✓ ✓ ✗
py/langchain ✓ ✓ ✓ ✗
py/langgraph ✗ ✗ ✗ ✗
py/litellm ✓ ✓ ✓ ✗
py/openai ✓ ✗ ✓ ✗
py/openai-agents ✓ ✗ ✗ ✗
py/pydantic-ai ✗ ✗ ✗ ✗

❌ Failed Tests Details

py/anthropic :: 9-message-truncation (2163ms) 
Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/google-genai :: 9-message-truncation (1500ms) 
Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/langchain :: 9-message-truncation (8765ms) 
Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/langgraph :: 1-simple (2968ms) 
Error: Fixture validation failed:
    invoke_agent (gen_ai.invoke_agent):
       gen_ai.request.model: missing
       gen_ai.request.messages: missing
    llm_call (gen_ai.chat):
       gen_ai.agent.name: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/langgraph :: 2-multi-step (4660ms) 
Error: Fixture validation failed:
    invoke_agent_1 (gen_ai.invoke_agent):
       gen_ai.request.model: missing
       gen_ai.request.messages: missing
    llm_call_1 (gen_ai.chat):
       gen_ai.agent.name: missing
    invoke_agent_2 (gen_ai.invoke_agent):
       gen_ai.request.model: missing
       gen_ai.request.messages: missing
    llm_call_2 (gen_ai.chat):
       gen_ai.agent.name: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/langgraph :: 9-message-truncation (9773ms) 
Error: Fixture validation failed:
    invoke_agent (gen_ai.invoke_agent):
       gen_ai.request.model: missing
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/langgraph :: 10-binary-content-redaction (3939ms) 
Error: Fixture validation failed:
    invoke_agent (gen_ai.invoke_agent):
       gen_ai.request.model: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/litellm :: 9-message-truncation (8908ms) 
Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/openai :: 9-message-truncation (9492ms) 
Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/openai :: 10-binary-content-redaction (2239ms) 
Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages: mismatch (expected: {'type': 'json_array', 'min_length': 1, 'contains': '[Blob substitute]'}, got: '[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAIAAAACUFjqAAAAE0lEQVR4nGP8z4APMOGVZRip0gBBLAETee26JgAAAABJRU5ErkJggg=="}}, {"type": "text", "text": "What color is this image? Answer in one word."}]}]')
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/openai-agents :: 2-multi-step (3749ms) 
Error: 'RunResult' object has no attribute 'messages'
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/openai-agents :: 9-message-truncation (6076ms) 
Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/openai-agents :: 10-binary-content-redaction (4104ms) 
Error: Fixture validation failed:
    llm_call (gen_ai.chat):
       gen_ai.request.messages: mismatch (expected: {'type': 'json_array', 'min_length': 1, 'contains': '[Blob substitute]'}, got: '[{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant that analyzes images."}]}, {"role": "user", "content": [{"type": "input_image", "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAIAAAACUFjqAAAAE0lEQVR4nGP8z4APMOGVZRip0gBBLAETee26JgAAAABJRU5ErkJggg==", "detail": "auto"}, {"type": "input_text", "text": "What color is this image? Answer in one word."}]}]')
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/pydantic-ai :: 1-simple (5365ms) 
Error: Fixture validation failed:
    invoke_agent (gen_ai.invoke_agent):
       gen_ai.agent.name: missing
       gen_ai.request.model: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/pydantic-ai :: 2-multi-step (6377ms) 
Error: Fixture validation failed:
    invoke_agent_1 (gen_ai.invoke_agent):
       gen_ai.agent.name: missing
       gen_ai.request.model: missing
    invoke_agent_2 (gen_ai.invoke_agent):
       gen_ai.agent.name: missing
       gen_ai.request.model: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/pydantic-ai :: 9-message-truncation (7810ms) 
Error: Fixture validation failed:
    invoke_agent (gen_ai.invoke_agent):
       gen_ai.request.model: missing
    llm_call (gen_ai.chat):
       gen_ai.request.messages.original_length: missing
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)
py/pydantic-ai :: 10-binary-content-redaction (8781ms) 
Error: Fixture validation failed:
    invoke_agent (gen_ai.invoke_agent):
       gen_ai.request.model: missing
    llm_call (gen_ai.chat):
       gen_ai.request.messages: mismatch (expected: {'type': 'json_array', 'min_length': 1, 'contains': '[Blob substitute]'}, got: '[{"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant that analyzes images."}]}, {"role": "user", "content": [{"type": "text", "text": "What color is this image? Answer in one word."}]}]')
    at ChildProcess.<anonymous> (/home/runner/work/_actions/getsentry/testing-ai-sdk-integrations/dba21cbfb57482556338983d8c35e6b09b534667/shared/orchestration/src/runner.ts:230:16)
    at ChildProcess.emit (node:events:524:28)
    at maybeClose (node:internal/child_process:1104:16)
    at ChildProcess._handle.onexit (node:internal/child_process:304:5)

This issue was automatically created by the AI Integration Testing framework.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ai-integration-test-failureautomated

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions