Lessons & Patterns

Practical lessons and recurring patterns from work in AI evaluation, retrieval, workflow design, and UX measurement.

Guiding principle Use as little AI as the workflow requires.

Start with the work people are trying to do. Add AI only where it reduces effort, improves judgment, or makes a difficult step easier to complete. A useful system should not create more review, maintenance, or coordination than it removes.

Lessons from the work

Workflow

Understand the work before adding AI

Measure the existing workflow first: what users are trying to do, where effort accumulates, and which problems require a product, process, content, or training change instead.

See the evidence →
Evaluation

Evaluate the whole answer, not only factual correctness

An answer can contain no obvious factual error and still fail because it omits the context, prerequisites, consequences, or next steps a user needs.

See the evidence →
Retrieval

Put important meaning where retrieval can use it

For vector-based retrieval, product context and other essential meaning often need to appear in the embedded content itself, not only in metadata.

See the evidence →
Deployment

A successful pilot does not prove deployment readiness

A tool may work in a controlled test while licensing, security, integration, maintenance, governance, or support requirements make full adoption impractical.

See the evidence →
Usability

New users reveal complexity that experienced users route around

Long-time users learn workarounds and stop noticing some friction. Newer users expose the real learning burden and the parts of the workflow that are hardest to understand.

See the evidence →

Recurring failure patterns

These problems appeared across the case studies and help explain why an AI feature can look promising while the surrounding system still fails users.

Pattern

The completeness gap

Generated responses are factually accurate but omit information needed for a complete, useful answer.

AI Response Quality →
Pattern

Source fragmentation

Relevant information exists, but it is scattered across sources, expressed inconsistently, or unavailable to the retrieval path.

Retrieval Readiness →
Pattern

Workflow bloat

AI adds review, correction, maintenance, or coordination work because the automation boundary is too broad.

Workflow Scoping →
Pattern

Capability without readiness

The feature works, but the organization cannot yet support it reliably in the real operating environment.

Workflow Scoping →