Back to all work
THE METRIC THAT WASN'T
VEDANTU · 2021–22

The Metric That Wasn't Measuring What It Said

A quarter of survey respondents said they couldn't find customer support. Seven interviews later, we had cancelled the redesign that was about to fix it.

Vedantu · Student support experience · Lead Product Designer · Jul 2021 – Apr 2022

Client engagement via Auriga IT, where I have been on payroll since 2015.

A note on what this case study is

No screens. The output here was a redirected initiative, not a shipped interface. I have included it because the most useful thing I did on this project was stop work that was already scoped and funded.

Chapter One

The Brief I Was Given

The PRD was clear and reasonable. A real number, a plausible diagnosis, a proportionate fix. Design work was ready to start.

Problem statement

Improve discoverability of customer support.

Hypothesis

Support discovery is cluttered, and more than 25% of NPS responders cite an inability to discover customer support.

Suggested solution

Redesign the entry points to the care, support and help sections.

Chapter Two

Why I Ran Research First

Two things about the brief bothered me. So before designing entry points, I wanted to watch someone actually fail to find support.

The hypothesis contained its own solution

"Discovery is cluttered" and "redesign entry points" are the same sentence twice, which usually means the problem was inferred backwards from the fix rather than the other way round.

The evidence was a survey checkbox

More than 25% selecting "could not find support" tells you what people ticked. It does not tell you what they were describing when they ticked it.

Chapter Three

The Study

Seven students, all on long-term paid courses, in classes 9 and 10, all of whom had had their Student Account Manager changed after 45 days. I moderated all seven sessions myself.

Participants were minors, so the CX team contacted parents to obtain consent before any session was scheduled. One parent joined and participated in the call.

I asked what issue they had hit, how they tried to reach support, whether they knew what Vcare was and how to reach it, and if they had not used it, why not. Every session was logged against the same structure so the pattern would be visible rather than anecdotal.

User Research
(All students have long term course and their SAM has been changed after 45 days and are from 9th, 10th)
Interviewer
Rakesh Gaur
User How much he/she rated support Feedback What was the issue you have faced? How did you try to contact support Do you understand what is Vcare and know how to reach there? Why you have not contacted Vcare? If applicable Remarks
Riya Raj 4 (customer Support)
1 (SAM)
I couldn't find how to reach out to the customer support She could not find videos related to a topic Tried calling SAM. Did not try to call Vcare. "I know if you have problem with batch and all call SAM and for other issues call Vcare." Yes, she knows what is Vcare and how to reach them. She is aware of tollfree numbers mentioned on website No reason given. "I just contact SAM for things"
Vaishnavi 4 (customer Support)
5 (SAM)
I couldn't find how to reach out to the customer support Topic related issue She used toll free number from website YES N/A Not sure why she has given this feedback about not finding support. She seems pretty aware about care and SAM.
Hansika (parent connected) 3 (customer Support)
3 (SAM)
I couldn't find how to reach out to the customer support

It took me multiple attempts to connect with the customer support

I was not able to connect to the customer support even after many attempts

My problem resolution took too long

My problem was not resolved

I was given a resolution but I am still facing problem
Course related issue He contacted everyone including SAM, email, Vcare. He did not get solution even after 6 months. Still waiting for resolution. He thinks when he wants to talk to us there is no proper response. Yes, he is well aware of toll free number and email written on website and used them. N/A He was able to contact support/SAM but he was frustrated coz no resolution/satisfactory answer has been provided to him
Raj 1 (customer Support)
1 (SAM)
I couldn't find how to reach out to the customer support

It took me multiple attempts to connect with the customer support

I was not able to connect to the customer support even after many attempts

The customer support agent(s) didn't understand my problem

My problem resolution took too long

My problem was not resolved

I was given a resolution but I am still facing problem

The customer support agent(s) was rude/unprofessional
Batch related issue He contacted Vcare via call. Because he has to wait for two days for SAM callback. Yes, but he thinks Vcare does not gurranty a resolution on the other hand if he is able to contact SAM resolution is gurrantied, So he prefers calling SAM. Although he contacted Vcare but was not happy with response still thinks SAM is more trustable source
Rishika 2 (customer Support)
2 (SAM)
I couldn't find how to reach out to the customer support

My problem resolution took too long
Batch related issue She contacted Vcare via call, because SAM was not responding. She genrally prefers calling SAM first Yes, she knows there are numbers available of Vcare in the website She does not feel comfortable calling Vcare
Niraj 1 (customer Support)
2 (SAM)
I couldn't find how to reach out to the customer support

It took me multiple attempts to connect with the customer support

I was not able to connect to the customer support even after many attempts

The customer support agent(s) didn't understand my problem

My problem resolution took too long

My problem was not resolved

I was given a resolution but I am still facing problem
Batch related issue Tried to call SAM Yes, knows what is Vcare and how to reach No reason given.
Samik 3 (customer Support)
5 (SAM)
I couldn't find how to reach out to the customer support Batch related issue Tried to call SAM Yes, knows what is Vcare and how to reach Thinks his problem was not big enough to email or contact Vcare
The research sheet — one row per participant.
Chapter Four

What Came Back

Nobody had trouble finding support.

No participant found it difficult to locate Vcare, by email, toll-free number or phone
7 of 7 understood what Vcare was and where to find it
7 of 7 preferred to call their SAM instead
They contacted Vcare only when the SAM was unresponsive
!2 of 7 actively distrusted Vcare

The stated problem did not exist. Support was discoverable and students knew exactly where it was. They were choosing not to use it.

Then two participants explained the metric

One had contacted every available channel — SAM, email and Vcare — waited six months, and never got a resolution. He still reported that he could not find support. Another was satisfied with the support she received. She also reported that she could not find support.

The metric was real. Its label was wrong.

Students were not reporting a discovery failure. They were reporting that their problem had not been solved, and "could not find support" was the nearest box on the form.

The eight research insights, as written at the time.
Chapter Five

The Diagnosis

Discoverability was never the problem. The obstacle was a mental model.

Every paid student has a named Student Account Manager. That is a person, assigned to them, with a relationship. Vcare is an institution. When something goes wrong, a fifteen-year-old calls the person they know, and falls back to the institution only when the person goes quiet.

Who they call first
The fallback
The SAM — a person
Named, assigned, a relationship
Vcare — an institution
Reached for only when the person goes quiet

That is not a navigation failure. It is a rational preference for a trusted human, and no amount of entry-point redesign moves it. We were about to spend a design and engineering cycle making something easier to find that people could already find and were deliberately not using.

The defensible version of this argument

I never disputed the 25%. That number was accurate. What I disputed was the inference drawn from it, and I disputed it with evidence about what respondents meant. Seven interviews cannot overturn a quantitative finding. They can, and did, establish that the finding was being read as something it did not say.

Chapter Six

What Changed

I wrote up the research and the recommendations. The PM took them to the stakeholders and the scope of the initiative was changed.

One support entry point, not several

A single "Get Help" destination that becomes the fixed place students go, rather than a set of touchpoints scattered across the product.

Channels presented together

With what each one is good for: chat, WhatsApp, email, phone, plus request-a-callback for paid students.

Expectations set explicitly

Published response times per channel, so a student choosing email knows it is hours and chat is immediate. If the underlying complaint is unresolved problems rather than undiscovered support, then managing expectation is closer to the real problem than improving findability.

Legacy pages deprecated

And an open decision on whether the existing FAQ page should stay.

Metrics redefined against the new framing

Request volume per channel, each channel's contribution share, and CSAT for issue resolution channel by channel. Note what these have in common — they measure whether problems get solved, not whether support gets found.

The SAM mental model, the deeper finding, went to the roadmap rather than into this phase. We fixed the channel experience first because it was tractable. Rebuilding how students relate to a named account manager is a service design problem, not a support-page problem.

The research outcomes and the revised roadmap with associated metrics.
Chapter Seven

Outcome

The scope changed. Nothing shipped.

Vedantu moved into a period of losses and layoffs, and my engagement ended in April 2022 before the redirected work was built.

So the honest outcome is this: an initiative that was scoped, funded and about to start was stopped, and replaced with one aimed at the actual problem. The value delivered was a design and engineering cycle not spent on the wrong thing, plus a set of metrics that measure resolution instead of findability.

I would rather claim that accurately than dress it up as something that launched.

What I Would Do Differently

Bring the quantitative team in, rather than working around them

I reinterpreted an NPS item from the outside. The stronger move would have been to sit with whoever owned that survey and rewrite the question, so the instrument stopped producing the misleading signal at source. Fixing one initiative is worth less than fixing the input that will misdirect the next one.

Push harder on the SAM finding

Roadmapping it was the pragmatic call and I supported it. But 7 of 7 preferring a named human over an institutional channel is the most important thing the study found, and it deserved more than a placeholder.

The generalisable version

"A number is not a finding."

"More than 25% said they could not find support" is a fact about what people clicked, and every survey instrument silently converts a range of experiences into whichever option is closest. When a brief arrives with a metric attached, the question is not whether the number is right. It is what respondents were actually describing when they produced it.

VEDANTU · STUDENT SUPPORT · 2021–22