Reference
Evaluation and capability evidence
A route, response, or passing software test does not establish that a learned capability is operating. This page explains what the current Mpalo API evidence supports, how the deterministic mock is evaluated, and what must be demonstrated before a model-dependent capability is described as available.
The hosted customer path is a production-shaped deterministic mock. Its evidence covers backend transport, authentication, managed text persistence, lexical retrieval, stored-text rendering, usage, test-environment billing, export, deletion, and operational monitoring. It does not establish live Palo Bloom execution, customer-facing embeddings, semantic vector search, learned surprise, traversal, mapping, or personalization.
What current tests establish
The current test layers answer different questions. A green result is meaningful only within the scope of the layer that produced it.
| Evidence layer | What it can establish | What it cannot establish |
|---|---|---|
| Contract tests | Response shapes, status codes, errors, headers, idempotency semantics, and protected-route behavior. | That a model produced the result or that a configured provider is reachable. |
| Runtime tests | Tenant-scoped managed persistence, deterministic lexical ranking, stored-value rendering, export, deletion, usage, billing records, and anomaly calculations. | Semantic quality, learned retention, embedding quality, or production-scale capacity. |
| Client parity tests | HTTP, Python SDK, CLI, JSON, authentication, usage, billing, infrastructure, and idempotency surfaces preserve shared meanings. | That every future client or provider has the same behavior without its own evidence. |
| Deployment verification | Whether the public target answers the documented health, documentation, OpenAPI, and unauthenticated rejection checks. | That the target has live model readiness, universal deletion, or production billing. |
| Learned capability evaluation | Whether a named model-dependent behavior improves against fixed controls on a defined fixture. | Whether the behavior is safe or suitable for every application, modality, or physical system. |
The capability registry is the authoritative status boundary. The implementation status, provider, model execution flag, billing mode, and limitations must agree before a claim is promoted.
Evaluation method for learned memory
A learned memory result is useful only when it can be compared with simpler explanations. The first evaluation should use a frozen, held-out fixture and the same reader and context budget across conditions.
| Condition | Purpose |
|---|---|
| Lexical baseline | Measures how much the current deterministic text path can achieve without learned representations. |
| Random or shuffled control | Tests whether the reported result depends on meaningful retrieval rather than accidental ordering. |
| Untrained or ablated condition | Separates architecture and pipeline effects from learned behavior. |
| Simpler learned baseline | Tests whether the proposed mechanism improves over a less complex representation or retriever. |
| Candidate Palo Bloom condition | Measures the named representation, retention, retrieval, rendering, traversal, or mapping behavior under the same constraints. |
Capability promotion gate
A capability moves beyond mock, contract-stage, control-plane-only, or research language only when the evidence names the exact implementation and deployment that produced it.
| Required evidence | Promotion question |
|---|---|
| Versioned contract | Are the input, output, representation, model variant, dimensions, provenance, and error states explicit? |
| Execution truth | Did the named provider and model execute, and can the path prove that it did not silently fall back to mock? |
| Data and authority | Are tenant, account, application, actor, subject, consent, provider, and authorization boundaries preserved? |
| Lifecycle behavior | Do correction, conflict, replay, deletion, export, timeout, rollback, and partial failure have tested meanings? |
| Measured operation | Are latency, cost, capacity, billing, retention, and operational records independently attributable? |
| Reproducible quality | Does the frozen fixture show a defined improvement over controls, with negative results retained? |
| Review and release | Have security, privacy, legal, product, and deployment owners accepted the capability boundary? |
A model name, storage connection, API key field, charge row, or successful HTTP response does not prove that the corresponding provider, model, embedding, or learned stage executed.
How evidence is reported
Every published status should be reproducible by a reader who knows the deployment, contract, fixture, and date. Reports should preserve failures and uncertainty instead of turning incomplete evidence into a green capability label.
npm run check:api-docs npm run check:infrastructure-boundary npm run verify:deployment # The deployment verifier checks reachability and public boundaries. # It does not claim Palo Bloom readiness.
For the current runtime and its limitations, read Infrastructure. For data and model boundaries, read Data boundaries and provenance. For provider execution requirements, read Storage and providers.
Was this page helpful?
Your feedback helps us improve our documentation.