OpenAI vs Voyage Embeddings: What We Measured in Production
In our production evals, OpenAI's text-embedding-3-large missed 74% of relevant chunks. Voyage's contextual embeddings missed 44%. Across recall, MRR, precision, and NDCG the improvement was 2–3x — on real retrieval over government documents, not a leaderboard. Re-embedding cost roughly $75 per 800,000 chunks with Voyage.
Leaderboards like MTEB tell you how models rank on benchmark corpora. They don't tell you how a model handles your domain. Ours is government procurement text — meeting minutes, contracts, budgets — retrieved as buying signals. Missing a relevant chunk isn't a decimal point on a benchmark; it's a deal our customer never saw.
The numbers
| Metric | OpenAI text-embedding-3-large | Voyage contextual |
|---|---|---|
| Relevant chunks missed | 74% | 44% |
| Recall / MRR / precision / NDCG | baseline | 2–3x better |
| Re-embedding cost | — | ~$75 per 800K chunks |
The migration trap nobody warns you about
You can't swap embedding models in place — vectors from different models live in incompatible spaces. During our transition, every query generated both an OpenAI and a Voyage embedding, searched the namespace matching each document's indexing era, then merged and ranked results. Ugly, temporary, and it worked; it retired once the backfill finished.
What to take from this
- Run your own evals on your own corpus. Our 74%-vs-44% gap doesn't appear on any public leaderboard.
- Price the backfill before committing. At ~$75 per 800K chunks, re-embedding even a 100M+ chunk corpus is a rounding error next to what better recall is worth.
- Plan the dual-model transition window — it's the actual engineering cost of switching, not the API bill.
The full context — including the vector database migration this rode along with — is in 133 Million Chunks.