Which TTS system do listeners prefer?
Paid US-based listeners compare matching audio from your candidate systems without seeing system names. Get preference rates, confidence intervals and individual judgements through the API.
Get an exact price quote before starting a study.
How it works
Audio in. Preference evidence out.
One API call creates and prices a study. Start it when the quote looks right; results stream in live as listeners answer, usually starting within minutes.
The same text rendered by system A and system B, as URLs. Label the sides; raters never see the labels.
Each pair is heard in full by several paid listeners, sides shuffled, exposure capped per person.
Win rate with a 95% interval, per-pair counts, and a CSV of every judgement for your own analysis.
Turnaround, measured rather than promised. Across the seven studies run on the platform to 6 November 2026, the first judgement arrived 0–17 minutes after start in five and under two hours in all; preference studies of 30–60-second pairs collected 150–200 judgements in their first 24 hours; a 480-judgement study completed in four days and a 384-judgement one in three. The panel is modest and growing, so a large study takes longer, and the progress endpoint reports the live rate.
The API
Two calls you will actually make.
curl -sX POST https://api.woodstitch.com/v1/evals \
-H "Authorization: Bearer $WOODSTITCH_KEY" \
-H "Content-Type: application/json" \
-d '{
"template": "tts_preference.v1",
"ratings_per_pair": 5,
"pairs": [
{"id": "p001", "passage": "p001",
"a": "https://your-bucket/cartesia/p001.wav",
"b": "https://your-bucket/eleven/p001.wav",
"system_a": "cartesia", "system_b": "elevenlabs"}
],
"dry_run": true
}'
# → {"quote": {"total_usd": 1.50}, "pair_seconds": [...], "warnings": [], "nothing_charged_yet": true}
Remove dry_run to create the study, then POST /v1/evals/{id}/start
with a spending ceiling. Signed or expiring URLs are fine: clips are fetched once, during this call.
curl -s https://api.woodstitch.com/v1/evals/7/results \
-H "Authorization: Bearer $WOODSTITCH_KEY"
{
"metrics": {
"human_pref/b_win_rate": 0.612,
"human_pref/b_win_rate_ci95_low": 0.551,
"human_pref/b_win_rate_ci95_high": 0.671,
"human_pref/tie_rate": 0.084,
"human_pref/pairs": 200,
"human_pref/judgements": 1000
},
"counts": {"a_wins": 346, "b_wins": 570, "ties": 84},
"per_pair": [{"id": "p001", "a_wins": 1, "b_wins": 4, "ties": 0}, ...]
}
Illustrative numbers for a 200-pair study. b_win_rate is the share of
preference for side b, ties counted half each way; the interval is bootstrapped
over pairs. /export?format=csv gives one row per judgement.
Full reference →
Worked price. $18 per hour of audio a listener hears, per judgement, priced on
the measured duration of your clips. 100 pairs of 30-second clips at 5 judgements each is 500 judgements ×
$0.30 = $150.00, quoted exactly before you start. Nothing is reserved up front; a study
pauses rather than overspends, and max_total_usd is a hard ceiling. Rate card ·
OpenAPI spec.
Method
Inspect the method before you trust the number.
Public research and customer studies run on the same platform under the same published rules.
No system names. Clip order is shuffled per listener, and a judgement cannot be submitted until both clips have played to the end. One playback per clip: pause and resume, no restart.
Paid $12 per hour of audio heard, nothing for deciding time, capped at $4 a day so no one person dominates a day's data. Location is checked at sign-in and on every visit; identity is verified at payout.
Per-listener caps on how often anyone hears a system, no-repeat rules per passage, and a cap on judgements per listener per study.
Listening time per clip, replays, tab focus and audio errors are recorded with every judgement and exported with it. There are no catch trials yet, and the methodology says so.
Every estimate carries a 95% interval. One that spans 0.5 does not establish a preference, and the result says so.
Stored for your study only, served only to listeners inside it, never published, never used to train anything, never shared with another client. Delete uploads any time through the API. Retention policy.
The full procedure, including what is not yet in place, is the platform methodology.
Proof
We publish our own listening studies.
Same platform, same listeners, same rules, in public.
Which TTS model is best for audiobooks?
Can listeners distinguish AI narration from human narration?
That report shows how we run a study, not how your system will score. We also build a narration product and disclose it on every report. Customer studies are never published and never enter our public research.
Questions
What engineers ask first.
Participating listeners, inside the study, with no system name, after accepting a published rater agreement. Clips are never published. Need a written retention commitment? Email us before you start.
English-speaking, US-based people paid per second of audio heard. Other languages or regions: email us and we will say plainly whether we can staff it.
Not offered: no 1–5 scale of any kind. Paired preference (forced choice, no tie) answers which system people prefer. A separate detection template asks whether a clip passes as human; it is in the docs.
Try it on your own audio.
Run a small pilot on your own clips before a larger study. Need help designing the comparison? support@woodstitch.com