Skip to main content

Module token_limit

soma_domain

Module token_limit 

Source
Expand description

Response size cap — prevents context-window exhaustion in MCP clients.

Placed in soma-domain rather than soma-application (see the module doc on actions.rs for the general reasoning): MAX_RESPONSE_BYTES is an invariant product constant read by soma-application’s provider registry and default limits (ProviderRequestLimits::default()), and directly by soma-mcp’s protocol-error rendering and response paging — putting it in soma-domain avoids every one of those call sites needing to reach into soma-application just for a usize constant.

§CUSTOMIZE: The 10K token philosophy

MCP servers communicate with AI agents that have finite context windows. A single oversized response can consume a large fraction of that window, leaving little room for the agent’s reasoning and subsequent tool calls.

Rule: no single MCP tool response may exceed ~10,000 tokens (~40KB).

§Why 40KB?

  • ~4 bytes/token on average (English prose, JSON, code)
  • 40,000 bytes / 4 bytes ≈ 10,000 tokens
  • 10K tokens is a generous upper bound that fits comfortably in any modern LLM context window without crowding out reasoning

§What to do instead of returning huge responses

  1. Paginate: add limit/offset parameters to list actions
  2. Filter: add filter or query parameters to narrow results
  3. Summarize: return counts and top-N items, with a link to get more
  4. Stream: for logs/events, return the most recent N lines

§MCP overflow handling

MCP tool responses must remain valid JSON. The RMCP adapter checks the compact serialized response against MAX_RESPONSE_BYTES. Oversized MCP results are replaced with a small structured page envelope containing a serialized JSON fragment and continuation arguments (_response_cursor, _response_offset, and _response_page_bytes) so agents can scroll through the cached response without re-running the tool action.

§Truncation is a legacy safety net, not the primary strategy

truncate_if_needed remains available for plain-text CLI or log-like outputs where partial text is acceptable. Do not use it for MCP JSON tool content. Design actions to return bounded data by default (limit=50, summary-only, etc.) so overflow handling rarely triggers.

Constants§

MAX_RESPONSE_BYTES
Maximum response size in bytes.

Functions§

truncate_if_needed
Truncate plain text to MAX_RESPONSE_BYTES if it exceeds the cap.