Running now · AI · human-in-the-loop triage

SORTHOUSE

An email and document triage lab with schema-validated extraction, field-level evidence, a policy outside the model, an approval queue and idempotent actions.

SORTHOUSE · approval queue
SORTHOUSE field inspector showing a photographed invoice with a low-confidence field and its evidence region.

Try it here

Open the photographed invoice, inspect the low-confidence field and its evidence, correct it, then run the prompt injection scenario in the Failure Lab. The lab interface is in English.

sorthouse.syslora.comNew tab ↗

The problem

Automating a mixed inbox is only safe when the system can tell when it lacks evidence and never repeats an action after a retry or timeout.

What it does

Open a synthetic inbox of invoices, a customer complaint, a request for quote, spam and an ambiguous message. Each message passes through real pipeline stages: sanitising, classification, extraction, validation, policy and review. Every extracted field links to its evidence: a quote in the email, a page of a PDF or a region of a photographed invoice.

The decision to automate is made by a policy outside the model, from deterministic checks. Invoices and customer replies stay approval-only; you can approve, correct or reject, and a correction keeps the original prediction in the audit. Approved actions go through a transactional outbox, so retries, duplicate deliveries and a timeout after sending are reconciled instead of repeated.

The public demo runs a deterministic baseline. LLM adapters are pluggable; reported baseline metrics are not LLM results. By default the classifier and extractor are a deterministic heuristic baseline and OCR, malware scanning and actions are simulated, all labelled in the interface; adapters for real language models exist but are not what the public demo runs. Sessions are temporary and isolated with two-hour retention, and the prompt-injection defences are layered, not a claim that the problem is solved.

The public repository includes a synthetic corpus, a tuning set, 122 evaluated cases: 100 held-out emails plus 12 adversarial and 10 failure fixtures, and measured results with their limits. On that set the baseline made 0 false automatic actions and blocked 16/16 unsafe actions, but it is a small synthetic corpus and says nothing about real inboxes or real models. This is an owned demonstration, not a client deployment.

What is in it.

  1. 01Classification and extraction with field evidence
  2. 02Deterministic validation and confidence gating
  3. 03Approval queue: approve, correct or reject
  4. 04Transactional outbox with timeout reconciliation
  5. 05Prompt-injection and failure injection lab
  6. 06Isolated sessions and measured evaluations

Built with

  • Laravel
  • Alpine.js
  • PostgreSQL
  • Redis / Horizon
  • Provider adapters
  • Playwright

SORTHOUSE · Your project

Need a system like this?

Tell us which part of this system would help your team, what you do by hand and which tools you use. We can discuss what fits your workflow.