Evals From Traces Deployed on AWS

[afarntrog]

This pull request introduces the initial infrastructure and core evaluation logic for the Strands Evals project. It provides AWS CDK stacks for deploying the dashboard and evaluation pipeline, implements Lambda functions for dashboard authentication and evaluation execution, and establishes a modular system for configuring and registering new evaluators. The changes include extensive documentation and a well-structured approach to adding new evaluation types.

Evaluation Flow

→ SQS Message (session_id, eval_type)
→ Lambda Trigger
→ Fetch session from Langfuse
→ Run evaluators (based on eval_type config)
→ Export results to S3

Infrastructure and Deployment (CDK & Lambda):

• Added a comprehensive README.md detailing the architecture, setup, deployment, and usage of the Strands Evals infrastructure, including stack outputs and instructions for updating credentials and adding new evaluators.
• Introduced cdk.json to configure the AWS CDK app, including watch settings and context for best practices and resource policies.
• Implemented a Lambda@Edge function (lambda/basic-auth/index.js) that enforces Basic Auth for the dashboard, with credentials injected at deploy time from AWS Secrets Manager.

Evaluation Pipeline Core Logic:

• Added eval_configs.py to map evaluation types (e.g., github_issue, release_notes) to their evaluator configurations, with a registry for easy extension and error handling for unknown types.
• Created an evaluators module with an __init__.py that registers custom evaluators for agent evaluation, enabling modular addition and import.

Custom Evaluators Implementation:

• Implemented ConciseResponseEvaluator (concise_response.py), which uses an LLM judge to assess if agent responses are appropriately concise, providing structured output and normalization of scores.
• Added ExpectedTrajectoryEvaluator (expected_trajectory.py), a pure Python evaluator that compares actual tool usage against the expected trajectory using recall, precision, and F1 metrics, with detailed reasoning in the output.

[lizradway]

might be nice to break apart eval scripts from cdk infra code for ease of review/legibility

[afarntrog]

Would consider refactoring this a bit later on when we introduce other Mapper types. We can handle a case where one agent is sending traces to langfuse and one to cloudwatch etc. Just not over optimizing now

[afarntrog]

Also, this will trigger per run (the script sends the message to SQS after the agent is done) so for a multi-turn conversation we will have an evaluation run per run.

There are some ways I'm thinking to get around this:

• We leave it. Maybe it's even helpful to see how the evaluations progress after each complete interaction
• Use the session id as the id stored in S3 so that each run will overwrite the previous. Will still run each time but we will only have one evaluation per agent
• we provide the agent with a tool to send the message to SQS once it's complete. So if the agent has follow ups then it ideally won't put the message into SQS. The obvious downsides is it's non-deterministic. So it may call too many times or none etc