soma_domain/
token_limit.rs

1//! Response size cap — prevents context-window exhaustion in MCP clients.
2//!
3//! Placed in `soma-domain` rather than `soma-application` (see the module
4//! doc on `actions.rs` for the general reasoning): `MAX_RESPONSE_BYTES` is an
5//! invariant product constant read by `soma-application`'s provider registry
6//! and default limits (`ProviderRequestLimits::default()`), and directly by
7//! `soma-mcp`'s protocol-error rendering and response paging — putting it in
8//! `soma-domain` avoids every one of those call sites needing to reach into
9//! `soma-application` just for a `usize` constant.
10//!
11//! # CUSTOMIZE: The 10K token philosophy
12//!
13//! MCP servers communicate with AI agents that have finite context windows.
14//! A single oversized response can consume a large fraction of that window,
15//! leaving little room for the agent's reasoning and subsequent tool calls.
16//!
17//! **Rule**: no single MCP tool response may exceed ~10,000 tokens (~40KB).
18//!
19//! ## Why 40KB?
20//!
21//! - ~4 bytes/token on average (English prose, JSON, code)
22//! - 40,000 bytes / 4 bytes ≈ 10,000 tokens
23//! - 10K tokens is a generous upper bound that fits comfortably in any modern
24//!   LLM context window without crowding out reasoning
25//!
26//! ## What to do instead of returning huge responses
27//!
28//! 1. **Paginate**: add `limit`/`offset` parameters to list actions
29//! 2. **Filter**: add `filter` or `query` parameters to narrow results
30//! 3. **Summarize**: return counts and top-N items, with a link to get more
31//! 4. **Stream**: for logs/events, return the most recent N lines
32//!
33//! ## MCP overflow handling
34//!
35//! MCP tool responses must remain valid JSON. The RMCP adapter checks the
36//! compact serialized response against [`MAX_RESPONSE_BYTES`]. Oversized MCP
37//! results are replaced with a small structured page envelope containing a
38//! serialized JSON fragment and continuation arguments (`_response_cursor`,
39//! `_response_offset`, and `_response_page_bytes`) so agents can scroll through
40//! the cached response without re-running the tool action.
41//!
42//! ## Truncation is a legacy safety net, not the primary strategy
43//!
44//! [`truncate_if_needed`] remains available for plain-text CLI or log-like
45//! outputs where partial text is acceptable. Do not use it for MCP JSON tool
46//! content. Design actions to return bounded data by default (limit=50,
47//! summary-only, etc.) so overflow handling rarely triggers.
48
49/// Maximum response size in bytes.
50///
51/// This constant is the single source of truth for the 10K token cap.
52/// Change it here to adjust the cap for all actions simultaneously.
53///
54/// # CUSTOMIZE: Adjusting the cap
55///
56/// For services that return very dense data (e.g. binary-encoded metrics),
57/// you may want a lower cap. For services that return sparse text (e.g.
58/// configuration files), the cap may be relaxed slightly.
59///
60/// Never exceed 100KB (25K tokens) — at that size, agents start losing
61/// context from earlier in the conversation.
62pub const MAX_RESPONSE_BYTES: usize = 40_000;
63
64/// Truncate plain text to [`MAX_RESPONSE_BYTES`] if it exceeds the cap.
65///
66/// When truncation occurs, appends a clear notice telling the agent:
67/// 1. That the response was truncated (not an error)
68/// 2. The exact token limit that was hit
69/// 3. How to get the full data (use pagination/filters)
70///
71/// # Truncation boundary
72///
73/// Truncation finds the last valid UTF-8 boundary within the content budget.
74/// The returned string, including the notice, never exceeds
75/// [`MAX_RESPONSE_BYTES`].
76///
77/// # CUSTOMIZE: Returning the raw truncated string outside MCP JSON
78///
79/// This function returns a `String`, not a `Value`. The caller wraps it
80/// as appropriate:
81///
82/// ```rust,ignore
83/// // In a CLI/log helper:
84/// let raw = serde_json::to_string(&result)?;
85/// let output = token_limit::truncate_if_needed(&raw);
86/// // output is now a plain string for non-MCP presentation:
87/// Ok(json!({ "data": output }))
88/// ```
89#[must_use]
90pub fn truncate_if_needed(text: &str) -> std::borrow::Cow<'_, str> {
91    if text.len() <= MAX_RESPONSE_BYTES {
92        return std::borrow::Cow::Borrowed(text);
93    }
94
95    let notice = format!(
96        "\n\n[TRUNCATED: response exceeded {MAX_RESPONSE_BYTES} bytes (~10K tokens).\n\
97        Use limit/offset parameters or more specific filters to get a smaller result.\n\
98        Example: action=things, limit=20, offset=0]"
99    );
100    let content_budget = MAX_RESPONSE_BYTES.saturating_sub(notice.len());
101    debug_assert!(
102        notice.len() < MAX_RESPONSE_BYTES,
103        "truncation notice ({} bytes) must be smaller than MAX_RESPONSE_BYTES",
104        notice.len()
105    );
106
107    // Find the last valid UTF-8 char boundary at or before content_budget.
108    // Walks back at most 3 bytes (max UTF-8 char width is 4).
109    let boundary = {
110        let mut b = content_budget;
111        while !text.is_char_boundary(b) {
112            b -= 1;
113        }
114        b
115    };
116    let truncated = &text[..boundary];
117
118    std::borrow::Cow::Owned(format!("{truncated}{notice}"))
119}
120
121#[cfg(test)]
122#[path = "token_limit_tests.rs"]
123mod tests;
soma_domain/token_limit.rs

soma_domain/
token_limit.rs
