Olivia Kelly

Content operations and knowledge systems lead

I help teams improve how people find, use, and maintain information.

Process Understand the work Identify the friction Improve the system Measure the result
Case Study · AI Response Quality

Found that most low-scoring AI responses were incomplete, not incorrect

I evaluated 949 GenAI responses across seven product teams and found that low-scoring answers were usually incomplete rather than wrong, changing how teams diagnosed and fixed them.

Failure-pattern analysis of 278 low-scoring responses

Incomplete 71% · Incorrect 13% · Correct, weak 11% · Uncategorized 5%

Created: team scorecards, a failure taxonomy, prioritized fixes, and review guidance for writers, subject-matter experts, and engineers.
Read case study →
Case Study · RAG Content Guidance

Developed shared guidance for improving RAG content

I led a cross-functional working group of 20–30 contributors across five workstreams to identify which content characteristics improved retrieval and response quality, then turned the findings into shared guidance.

First draft of shared guidance released · 20–30 contributors · 5 workstreams

Read case study →
Case Study · Measurement

Improved a workflow's usability score from 39 to 57

I designed a measurement approach that helped the platform team identify what to fix, what to validate, and which workflow changes improved the authoring experience.

CCMS pilot · SUS score: 39 → 57 across cycles 1–3

Read case study →

Browse the portfolio

Resources Use the tools

Kits, templates, and tools built from the work.

 Lessons & Patterns See the lessons

Practical lessons and recurring problems from AI evaluation, retrieval, workflow design, and UX measurement.

Resources

Browse all →
Kit AI Response Quality Audit Kit Find why AI responses fail and whether the source content is part of the problem. Includes an audit template, scoring guide, and interpretation notes.
Download →