> For the complete documentation index, see [llms.txt](https://klassdocs.gitbook.io/klassapp-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://klassdocs.gitbook.io/klassapp-documentation/docs/toshi-prod-health-check.md).

# Toshi production LLM health check

## Incident class (2026-08-02)

Production Toshi never completed turns (`agent_conversations` stayed empty) because dual provider env silently resolved to an invalid model:

| Signal                            | Prod value (broken)                                |
| --------------------------------- | -------------------------------------------------- |
| `OPENAI_COMPATIBLE_URL` / default | DeepSeek (`api.deepseek.com`)                      |
| `OPENAI_COMPATIBLE_MODEL`         | **unset**                                          |
| `TOSHI_LLM_MODEL`                 | `meta/llama-3.1-8b-instruct` (NVIDIA NIM leftover) |

`config/toshi.php` resolves `model → env('OPENAI_COMPATIBLE_MODEL', env('TOSHI_LLM_MODEL', 'deepseek-chat'))`, so agents sent an NVIDIA model name to DeepSeek → HTTP 400. Live ops fixed with `OPENAI_COMPATIBLE_MODEL=deepseek-v4-flash` (OpenCode). This PR is the **structural code fix**.

## Detection logic (`ToshiLlm::assertConfigConsistent()`)

Fails loudly when **any** of these hold:

1. **Both models set and differ** — `OPENAI_COMPATIBLE_MODEL` and `TOSHI_LLM_MODEL` non-empty with different values (split-brain with `ai.providers…smartest` still on `TOSHI_LLM_MODEL`).
2. **Both URLs set with different hosts** — `OPENAI_COMPATIBLE_URL` vs `TOSHI_LLM_BASE_URL`.
3. **Model family incompatible with URL host** — e.g. `meta/llama-*` / `nvidia/*` against `*.deepseek.com` (exact incident shape when only the legacy model drives resolution).

Not a conflict: preferred `OPENAI_COMPATIBLE_*` alone; legacy `TOSHI_LLM_MODEL` alone when family matches host; identical dual model values; unknown custom hosts (no false positive).

Raw env snapshots live in `config('toshi.llm_env')` so detection works under `config:cache`.

## Failure mode

* **Throw** `AmbiguousToshiLlmConfigException` on first `ToshiLlm::model()` / `provider()` resolve (agents / `UsesToshiLlm`) and from `toshi:llm-health`.
* **Do not** fail AppServiceProvider boot — unrelated HTTP routes must keep working.
* **Alerting evidence**: no Sentry/Bugsnag package in this app; `ActivityLog` is user-action audit, not infra monitoring. Monitored signal = exception + `Log::critical` (health command) + **non-zero Artisan exit** for cron/k8s.

## Commands

| Command                              | Purpose                                                                                                                                                                          |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `php artisan toshi:llm-status`       | Config dump only (provider/model/host/key/checksum) — no live call                                                                                                               |
| `php artisan toshi:llm-health`       | **One cheap live completion** via openai-compatible `chat/completions`; verifies assistant content; exit `1` + `Log::critical` on failure                                        |
| `php artisan toshi:adversarial-live` | **In-process** live soft-refusal suite (16 scenarios); sqlite `:memory:` isolation; **no PHPUnit**; gated by `TOSHI_ADVERSARIAL_LIVE=1`; exit `1` + `Log::critical` on hard fail |

Ops (VPS):

```bash
docker exec sms-app php artisan toshi:llm-status
docker exec sms-app php artisan toshi:llm-health
# Monthly soft-refusal (only when intentionally gated on):
# TOSHI_ADVERSARIAL_LIVE=1
docker exec sms-app php artisan toshi:adversarial-live
```

## Schedule

* **`toshi:llm-health`**: **Not wired** in `app/Console/Kernel.php`. OpenCode/ops owns production cadence (cron/k8s watching exit code). Host script `.toshi-health-check.sh` every 5 min → `storage/logs/toshi-health-monitor.log` (`HEALTH OK` / `HEALTH FAIL`) + `Log::critical` on failure into laravel daily logs.
* **`toshi:adversarial-live`**: Gated monthly Kernel entry (first Sunday 02:00 Africa/Kampala, day≤7). Runs only when `TOSHI_ADVERSARIAL_LIVE=1` **and** openai-compatible key is set; `--scheduled` no-ops otherwise. Safe for `--no-dev` images (in-process runner). **Durable reports:** `storage/logs/toshi-adversarial-live.log` (Kernel `appendOutputTo` + log channel `toshi_adversarial`) and structured `Log::info` / `Log::critical` on the default stack.

## Post-deploy cleanup

If prod still has leftover `TOSHI_LLM_MODEL=meta/llama-…` alongside `OPENAI_COMPATIBLE_MODEL=deepseek-v4-flash`, assert will throw until ops **unsets or aligns** `TOSHI_LLM_MODEL`. Prefer a single source of truth: `OPENAI_COMPATIBLE_*`.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://klassdocs.gitbook.io/klassapp-documentation/docs/toshi-prod-health-check.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.