feat: realtime agents

[ramyak-mehra]

Adds a new RealtimeAgent class (packages/agents/src/realtime/realtime-agent.ts) to orchestrate the backend realtime pipeline for voice-first agents.

• Uses RealtimeKit as the low-latency transport/communication layer
• Wires an ordered pipeline of components (transport → STT → agent → TTS → transport)
• Includes provider convenience components for STT/TTS (e.g. Deepgram + ElevenLabs) with AI Gateway support
  Extras
• Built-in HTTP endpoints to start/stop pipelines and manage transcript history
• Persists transcripts (Durable Object SQLite) and supports streaming SpeakResponse back into the meeting
  Docs: @docs/realtime-agents.md
  Example @examples/realtime-agent/src/index.ts

[changeset-bot]

⚠️ No Changeset found

Latest commit: 64d81cb

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[threepointone]

what is this PR? can you describe in detail?

also remove all the pnpm lockfiles

[threepointone]

Thanks for putting this together. The core idea is solid, but this PR needs more work before it's ready to merge. Here are the main concerns:

Missing Fundamentals

No tests - A feature of this complexity needs test coverage. At minimum: pipeline validation, transcript persistence, and the WebSocket message handling.

No client example - The example shows the server-side agent, but how does someone actually connect to it? There's no frontend code showing how to join a RealtimeKit meeting and interact with the agent.

No dependencies in package.json - The example imports from @cloudflare/realtimekit, etc. but the package.json has no dependencies section.

Configuration Issues

• vite.config.ts exists but package.json scripts use wrangler dev - pick one
• README says localhost:5173 but wrangler defaults to 8787

Code Quality

• Unused imports: DataKind, RealtimePipelineComponent imported but never used
• Swallowed errors: Pipeline creation errors are caught and only logged, then execution continues with undefined flowId/token
• Dead code: #handleVideoMessage decodes video frames but does nothing with them; onRealtimeVideoFrame throws an error meaning it's mandatory for it to be defined?
• Type safety: Uses as any casts to check private properties instead of adding proper accessor methods
• Invalid JSON: Line 516 returns {'success':true} (single quotes = invalid JSON). Maybe that should be Response.json()?
• Type guard bug: isRealtimeWebsocketMessage requires context_id to be a string, but it's optional in the type definition

Missing Exports

index.ts exports MediaProcessor and TextProcessor which don't exist in components.ts

Documentation Issues

• README says transcripts are stored in "D1 database" but it's actually Durable Object SQLite
• Constructor examples show 3 arguments but the actual signature requires 4 (gatewayId). But also, RealtimeAgent extends Agent, and those (durable objects) aren't explicitly initialised, so how will this even work?
• No explanation of how to create a RealtimeKit meeting or configure AI Gateway BYOK

Hardcoded Values

The example has hardcoded internal values that won't work for anyone else:

• GATEWAY_ID = "aig-worker-testing"
• meetingId: "bbb9e53e-c839-4c84-b0cd-b8ef18ed8da2"

---

Recommendation: This needs another pass. Please add tests, fix the bugs, clean up the example to use placeholder values, and either add a client-side example or document how users are expected to connect to the agent.

[threepointone]

Blocking