The BPO podcast: 90%+ sustained accuracy in Transperfect Dataforce (transcript)

[HOST] Welcome back to The BPO podcast. Today we're talking about a number: 90%. Specifically, the 90% accuracy floor that Transperfect Dataforce set on their interpolation annotation program, and how Zipang's Indonesian pod crossed it in week 11 and has stayed above it ever since. Joining me again is Yoseph Gratika.

[GUEST] Thanks for having me back. The 90% number is the contractual floor — below it, the labeled data is too noisy to train a model on. The interesting question is what the actual sustained number is, and how it got there. End-2025, the rolling 12-week average was 92.1%. That's the number that matters.

[HOST] Walk me through week 1. The first 2,000-clip batch lands, and the accuracy is 78%. What happened?

[GUEST] Three error clusters. First, motion-blur edge cases. Operators were drawing boxes too tight, losing the object when it crossed a frame boundary. Second, automotive small-object detection. Pedestrians at distance were missed entirely; the model needs those for downstream training. Third, scene-transition frames. Operators were labeling the post-transition frame instead of the transition itself, which is a different kind of error. The error analysis ran on the gold set, and the three clusters were the input to the retraining arc.

[HOST] So 78% wasn't a surprise.

[GUEST] No. We expect week 1 to land below the floor, because the SOP is new and the operators are still learning the client's specific rubrics. The question is whether the retraining arc can land them above the floor by week 12. If week 12 is still below 90%, the program is in trouble.

[HOST] Weeks 2 through 4. What did targeted retraining look like?

[GUEST] Each cluster got a 30-minute retraining module with annotated examples from the gold set. Motion-blur got 12 examples showing the correct box width at frame boundaries. Small-object detection got 8 examples at 1080p and 4K with the correct bounding-box inflation. Scene transitions got 10 examples contrasting the transition frame versus the post-transition frame. Operators re-took the gate 3 quiz subset on those examples; the pass rate moved from 54% to 71% across the cohort. Production accuracy climbed from 78% to 82% by week 4.

[HOST] And weeks 5 to 8?

[GUEST] Peer review at frame N. A second operator labels frame N independently; any disagreement greater than 5% IoU triggers a re-annotation. This catches disagreements before they reach the client. Production accuracy climbed from 82% to 88% by week 8.

[HOST] Weeks 9 through 12 — the floor-crossing weeks.

[GUEST] Two things happened. First, supervisor review at frame N+24. A QA supervisor reviews every N+24 prediction against the source video, scoring motion-continuity and edge-case tagging. Second, a 200-case weekly gold set that the full cohort re-labels on Fridays. Scores are computed against the frozen answer key; any operator whose accuracy drifts below 89% on the gold set is pulled into 1:1 retraining before live production the following week. Production accuracy crossed 90% in week 11 and landed at 92% in week 12.

[HOST] And it's held since?

[GUEST] Yes. Oscillating between 91% and 93% on the rolling 12-week average. End-2025 was 92.1%. The contract floor is 90%; the program delivers 92.1%. That 2.1-point cushion is what makes the program defensible.

[HOST] Why three review layers? Why not one good one?

[GUEST] Each layer catches a different error class. Peer review at frame N catches disagreement between two operators on the same frame — it's a consistency check. Supervisor review at frame N+24 catches the harder prediction case, where the operator is forecasting 24 frames ahead. The weekly gold set catches drift over time, where an operator's accuracy slowly erodes as the SOP evolves or as the operator's calibration slips. One layer can't do all three. Three layers is the minimum for sustained 90%+ on a hard task.

[HOST] And the cost?

[GUEST] Supervisor time is roughly 12% of total program cost. Peer review is built into the operator rate — we pay 5% bonus for peer review work. The gold set is the most expensive line item: 200 cases × 20 operators × 12 minutes per case, every Friday, for 52 weeks. It's the bottleneck, but it's also what makes the 92.1% number real.

[HOST] The cohort math — 60 trained, 20 full-time, 33% trial-to-full-time conversion. Why so low?

[GUEST] The Transperfect task is harder than the France retail task. Interpolation annotation requires holding object state consistent across frames, and the QA reviewer and gold-set cycle is the bottleneck, not the operator pool. A larger cohort would need two full-time QA reviewers, which doubles the QA cost without doubling the throughput. The 20-operator cohort needs one QA reviewer. 33% conversion is the right number for the task.

[HOST] And the 5-gate funnel pass rates?

[GUEST] Gate 1 retained 26% of applicants, gate 2 retained 47%, gate 3 retained 54%, gate 4 retained 71%, gate 5 retained 81%. End-to-end, 60 of approximately 1,800 applicants reached the trial cohort, and 20 of those 60 converted to full-time. The aggregate funnel pass rate is 3.3%, which is at the lower end of the 6–12% Zipang range because the task is harder.

[HOST] For a buyer evaluating an Indonesian BPO in 2026, what should they ask about a 90%+ accuracy claim?

[GUEST] Three questions. First, what is the week 1 accuracy? If the vendor cannot name a week 1 number, they are not running the kind of program that produces a 92.1% sustained figure. Second, what is the gold-set cycle? If the vendor cannot name a weekly gold-set size, they are not running calibration, and the sustained number is fictional. Third, what is the failure threshold? Below 90% for two consecutive weeks triggers a remediation plan; below 88% for one week triggers escalation. A vendor who cannot name those thresholds is a vendor who has not signed a contract with a 90% floor.

[HOST] Anything else?

[GUEST] Ask for the accuracy curve, not just the sustained number. The curve from 78% in week 1 to 92% in week 12 is the operational story; the 92.1% sustained is the consequence. A vendor who can only show you the sustained number is hiding the curve.

On the 90% floor: "Below 90%, the labeled data is too noisy to train a model on. The 2.1-point cushion is what makes the program defensible."

On week 1 expectations: "We expect week 1 to land below the floor. The question is whether the retraining arc lands them above the floor by week 12."

On three review layers: "Each layer catches a different error class. One layer can't do all three. Three layers is the minimum for sustained 90%+ on a hard task."

On the gold set: "200 cases × 20 operators × 12 minutes per case, every Friday, for 52 weeks. It's the bottleneck, but it's also what makes the 92.1% number real."

On the cohort math: "A larger cohort would need two full-time QA reviewers, which doubles the QA cost without doubling throughput. 20 is the right number for the task."

On what buyers should ask: "Ask for the week 1 number, the gold-set cycle, and the failure thresholds. A vendor who cannot name those three has not signed a contract with a 90% floor."