From UX Research to UX Intelligence: What We Learned from Jen Cardello

Last week, more than 700 people signed up for our monthly free webinar (watch the webinar here), a record for the series. Our guest was Jen Cardello, Managing VP of Experience Design Research at Capital One. (Jen was also one of Jeff’s first bosses. She hired him in 2002.)

The topic struck a nerve. Open roles for design and research are near an all-time low, according to recent data from Lenny Rachitsky's newsletter, and many researchers are wondering where AI leaves them. Jen's view was that this change is coming either way, just as democratization of research and design did a decade ago. You can wait on the train platform, chase the train, or drive it. Her team chose to drive the train. Here's what that looks like for the Research team at Capital One and three things you can try this week.

From a service desk to an always-on system

For most of our careers, research has run as a service. Someone brings a question, a researcher spins up a bespoke study, and weeks later a report lands. Josh opened the session with one of his own research reports from 2002, complete with a table of contents, and admitted he always suspected nobody read them.

Capital One is replacing that model with what Jen calls a human intelligence system. Research data keeps accumulating, and anyone can query it on demand through Slack, a web interface or AI-enabled product tools. The data feeding the system has become the currency of the research team. Three principles guide the work:

  • Connected over discrete: design bodies of research to link together from the start instead of mapping them after the fact.

  • Continuous over episodic: build standing infrastructure so teams hear from customers every day without waiting for a request.

  • Programs over projects: create repeatable programs that can be lifted and shifted to other lines of business.

The emphasis here is on a continuous learning service that uses structured data formats to ensure that the most recent learnings are always available and always on for anyone in the company at any time. This is such a massive shift from how UX Research used to operate. The work goes on but the distribution and usage of that work is sped up and formatted for AI-speed teams.

What doesn't change

The fundamentals still hold. Jen's team uses Marty Cagan's four risks (desirability, viability, feasibility and usability) to explain what researchers do. The pitch to colleagues is simple. Research helps ensure product-market fit, so what teams build gets adopted and delivers meaningful value to those users and the company.

The Capital One team also worked to standardize their research processes. They built a standard ten question framework to help teams consistently request learnings and research. This made it easier for the research team to comply with each request because it was clear exactly what each team was lacking. Researchers then spend time in planning sessions with product and design partners keeping the research system tied to the business..

What the system is made of

Jobs to Be Done is the backbone. Capital One has 30 studies in progress and 10 complete, all built on Strategyn's outcome-driven innovation method and one shared taxonomy of jobs, job steps and outcome statements. That consistent structure lets AI tools synthesize across jobs and build dashboards that show results in a way that can be easily understood across the company.

On top of that data, researchers build AI skills. The first was a research planner. Next came a gap analysis skill that turns transcripts into journey maps. They also built a heuristic evaluation skill that checks designs against 37 criteria in hours instead of weeks. Finally, the Capital One team built a research coach that helps teams sharpen their questions and routes them to self-serve research or a researcher.

One of the key questions we wanted answered is what happens when the system doesn’t have an answer to a query. LLM’s are notorious for their efforts to please the user. Jen’s team surprised us because they their system to say when it doesn't know. Questions it can't answer go into a log, and that log becomes the team's learning agenda. The researcher role is growing to match as well. Program development, AI skill and agent building, data structuring and research orchestration now sit alongside traditional research methods.

A hard look at synthetic data

The liveliest part of the Q&A was about synthetic users. Capital One's AI (read: synthetic) personas kept collapsing into each other. Twenty personas would drift into one “mushy” composite offering little differentiation between potential users. The synthetic personas would also anchor on odd details (one would not stop talking about her son's soccer games) and eventually they all ended up sounding like product managers.

One of Jen's researchers called this “compelling fiction.” A single synthetic answer reads beautifully. Read hundreds and the verbal tics start jump out. Jen's rule is to never ask synthetics ethnographic questions about lived experience. Where synthetic data can help is predictions and probabilities, such as how likely a segment is to adopt a feature.

Think you can just default back to humans? Not so quickly, Jen shared. Human panels have their own problems in the AI era. One survey came back with open-ended answers written in suspiciously complete sentences. The team estimated 55% were AI-generated, and Gemini put it at 60%. Checking the quality of your human panel is now part of the job as well.

Three things you can start this week

  1. Put the research you already have to work. Capital One started by ingesting two years of existing studies, weighted by age, before running anything new. Load your recent studies into a notebook tool like NotebookLM and let stakeholders query it. Jen called this the simplest win available. Then track the questions it can't answer. That list is your learning agenda.

  2. Standardize one instrument. Pick the study you run most often, such as a concept test, and lock a core set of questions you ask every time, with room for two team-specific ones. A team Jen led at athenahealth built concept tests on validated measures so results could be compared across years. Standard questions turn one-off studies into data that compounds.

  3. Leave something interactive behind. At your next readout, share a dashboard or a PII-cleaned transcript notebook alongside your insights. Jen's teams found that reports die on the shelf, while interactive leave-behinds get used for years of planning.

Keep the conversation going

We ran out of time with more than 40 questions still in the queue, and we're working on a way to answer them.

Our last webinar of the year is on December 15. It’s an AI Practitioner Roundtable with an engineer, a designer and a product manager who all worked before and after AI. Sign up at webinar.senseandrespond.co.

And during our Black Friday sale, buy one full-price ticket to any eligible public workshop at workshops.senseandrespond.co and we'll email you a code for a second seat, free. Bring a colleague.

Next
Next

We have more decisions than ever, and fewer researchers to help us answer them