Why You Can’t Test Your App on Friends
For the last few weeks, one line kept stopping me from bidding on Upwork: “ proven experimentation-system ownership. Not marketing funnels — a real CRO track record, depth in GA4 or Mixpanel or Amplitude, evidence that you have built an experimentation system somebody else relied on.”
I couldn’t answer that honestly, so I didn’t bid.
I understand why clients ask for it. If I were paying, I would want proof too. I would only add that there are fewer people with that proof than the job ads assume. Successful products are rare, and the people who worked on them do not always know which decision caused the result. (I wrote about that here)
So I stopped waiting for someone to give me the chance. I have my own product — GymBuddy, an Android training app I built — and no client’s money at risk. I ran a pre-registered evaluation on it.
It did not give me the result I wanted. It gave me a result I could not report at all 🙈 . Deciding that, and being able to explain why, turned out to be the real work.
What the pilot was
The pilot was a pre-registered evaluation of the first training microcycle. I wrote down the metrics, the decision thresholds and the success criteria before I recruited anyone.
It was not a usability test and not a satisfaction survey. It was also not an A/B test: there was no baseline and no change to compare against.
Four questions. Does onboarding collect enough context, and is it accurate? Is the first generated program structurally correct? Is week one actually doable? Does trust survive the first week?
Every threshold came with a decision
This is the part I would defend first. Each threshold was written together with the decision it triggers if it fails. Four of the eight:
| Threshold, written before recruiting | If not met → the decision |
|---|---|
| At least 4 of 5 primary users rate profile accuracy 4–5 | Revise the onboarding questions before releasing to anyone else |
| No more than 20% of loaded exercises need a down-correction over 20% | Recalibrate the starting weights in the generator |
| Zero critical assignments — pain, unsafe weight, an exercise the person cannot physically do with no substitute | Stop. Fix it before one more tester installs the app |
| The same missing input named by two or more primary users | Becomes a product issue automatically, even if the average scores look fine |
A number on its own is easy to explain away later. A number with a decision attached to it is not.
Two of the rules cost me something, so they are worth naming.
Up-adjustments are not a defect. Week one is meant to be conservative, so some starting weights are deliberately low. Only down-corrections count against calibration. I wrote that before the pilot started, so the report could not punish the design for doing what it was designed to do.
Counts before percentages. Every threshold is calculated for each person first, then reported as “N of M”. If I pooled all the sessions together, one active participant would cover for an inactive one. With five people, a percentage sounds more precise than it really is.
What I built to look at the evidence
I built a small dashboard that reads the pilot data usian OpenAI Codex asssitance.
Everyone in these screenshots is anonymised — P01 to P04. The file that maps those labels to real accounts stays on my machine and never reaches the dashboard.
The decisions behind it were product decisions, not technical ones.
- A separate project, not part of the app. The research tool does not live inside the product that people use.
- Local only, one computer. No public hosting and no password, because a password stored in the frontend is not real protection. The real boundary is the machine, a read-only account, and a short list of participant IDs.
- Read-only, and a test enforces it. There are no write calls, and the build fails if anyone adds one. A tool that measures something should not be able to change it.
- The data source is always shown on screen — real data or demo data. A dashboard that can quietly show fake numbers is worse than no dashboard.
Three rules are built into the screen itself.
Unknown is not zero: if there is nothing to divide yet, the metric says Pending, never 0%.
Primary cohort only: beginner and advanced testers are visible, but they never count towards a threshold.
Evidence, not engagement: the panel only shows what exists as a stored record, so it says nothing about app opens, time in the app, or retention. None of that is measured, on purpose.
Then three things went wrong
1. The panel refused to calculate one of my own thresholds.
Threshold #5 says NOT DEFINED, and the reason is printed next to it: the pilot never defined how several actual set weights become one number for the exercise.
I do have a formula that would produce something — the median of the actual weights, compared to the prescribed one. The panel does not use it. If I picked that rule now, with the data already in front of me, I would be picking the version that looks better. So the number stays uncalculated, and the medians sit there marked as candidates for review.
2. The panel broke my own rule.
For a while, threshold #3 showed 0 of 2 — FAIL. No microcycle was finished, every session was still scheduled, and my own rule says that “not reached” is not “failed”.
The arithmetic was correct. Nobody had completed a session, so the rate was zero. The mistake was calculating the threshold at all, before anyone’s week had ended.
The reason was in the app, and I had forgotten my own design. GymBuddy has no server job that marks a workout as missed. It happens on the phone: you open the app, it sees the old sessions, it marks them, and then it syncs. So if someone generates a program and never opens the app again, those sessions stay “scheduled” forever. Not because nothing happened, but because nobody was there to record that nothing happened.
The raw data was honest. My reading of it was not.
There was an easy fix: let the panel write “missed” onto those rows. One pass, everything tidy. I did not do it, and this is the rule I care about most here. A measurement tool is allowed to reinterpret data. It is not allowed to invent it. Writing “missed” would create an event the app never recorded, in the app’s own database, only to make my analysis look neater.
So the panel now shows a second status next to the original one — Past due, unresolved — and keeps the raw status visible underneath. Threshold #3 stays Pending until every session in the microcycle reaches a final state. If the week ends with nothing completed, that is a real failure, and it will say so.
There is a smaller version of the same discipline in the same data. One participant does have a completed workout, but it belongs to a later version of the program, outside the first microcycle. The scope was fixed in advance, so it does not count — even though it would have been the only completed session in the whole pilot.
3. The finding I did not expect.
The app works out your training level from a few simple questions: bodyweight squats, push-ups, how familiar the movements are, whether you remember your previous weights. All of them top out early, around the line between beginner and intermediate. So the classification reaches its ceiling and cannot tell an intermediate lifter from an advanced one.
It happened to two participants, including one I had put in the beginner group myself. One of them said he remembered his previous weights well and entered none of them. So “advanced” was based on no weight data at all.
That level feeds the starting weights. A level that is too high can mean a heavier program for someone who is not ready for it (or somebody who overestimate themself), which is exactly what one of the safety thresholds exists for.
And here is the part that belongs in a post about measurement: both of them rated the accuracy of their profile 5 out of 5. The form cannot catch this mistake, because the app recorded exactly what it had calculated from their own answers. A good question, a top score, and the fact underneath it still wrong.
The fix: “advanced” has to require real weight data, and “I remember my weights” with nothing entered should fall back to the conservative default. More generally, simple proxy questions need a ceiling that matches the decision they feed.
Why I am not reporting results
I did not invite people at random. I chose people who actually go to the gym. The filter was right, and it still did not work, because what I asked for was a favour, not a solution to their problem.
Two of the five had told me they needed a training program. They still did not finish it. One never installed the app.
Saying you need something is not the same as wanting it enough to act. Describing a problem is free. Installing an app and going to the gym is not. This is why discovery interviews mislead founders, and I would rather learn it on my own product than on a client’s.
I also thought about recruiting on Reddit. While I was working out what to write, I read a message from someone in that community: he was tired of it, everyone is building workout apps or management apps, could nobody think of anything else. I closed that channel instead.
Everyone finished onboarding. Nobody trained. The tempting conclusion is that onboarding works and the problem sits between the plan and the gym.
I cannot say that. People who never wanted a training program did not go and train. That is a fact about who I recruited, not about my product. Onboarding “worked” for the same reason: it is the cheap part of a favour — five minutes, at home, and I am watching. The gym is the expensive part, and nobody is watching there. Friends will finish the part you can see and skip the part you cannot.
All four installed the app as a favour, and none of them wanted a training program. So I cannot separate “did not want it” from “something stopped them”. My activation number is not low. It is meaningless, and reporting it as a finding would be dishonest.
What I would keep
The list of things I could not measure is not an apology. It is the reason the rest can be trusted. Limits that are not written down are where numbers quietly turn into claims.
The part that actually cost something was writing the thresholds before looking at anything, and then keeping them when the answer was inconvenient, missing, or impossible to calculate. That is the judgment I would want on a small product, where clean experiments are rare and the honest answer is often: this cannot tell us that.
Leave a Reply