Qaf · Search benchmark · 21 November 2026Mixedbread v4 finds better passages than Turbopuffer, and finds them faster
The whole library is now loaded in Mixedbread on mxbai-wholembed-v4 and matches Turbopuffer on every check we ran. On 380 real production searches, Mixedbread with no reranking beat our current best setup (Turbopuffer plus Cohere Rerank Pro) on quality and answered about nine times faster. At list price, though, Mixedbread search would cost about $24k a month, against roughly $2–5k for Cohere reranking today plus the Turbopuffer and embedding bills.
Recommendation
- Mixedbread is the better retriever. If we move, use it with its built-in reranker, or with none. Cohere on top of Mixedbread adds cost and 0.8–0.9 s and no measurable quality.
- Price is the open question. We run about 198,000 searches a day. Negotiate search pricing before committing; the list price is not workable at that volume.
- Turbopuffer is untouched. Nothing was written to or deleted from it, and no app code was switched.
Migration
All 8,583 book versions and 7,455,835 chunks were read from R2 and uploaded as 11,723 files, with zero failures. Turbopuffer's total row count is the same 7,455,835.
| Check | Result |
|---|---|
| Files indexed | 11,723 of 11,723, 0 failed |
| Tokens indexed | 3.08 billion |
| Chunk counts, 250 books: 150 spread across the library plus the 100 largest (R2 vs Turbopuffer vs Mixedbread) | 250 of 250 match |
| Field-by-field comparison, 1,547 chunks from those books | 0 mismatches, 0 missing |
| Filtered searches returning only matching results (65 queries, 3,024 results) | 0 violations |
| Upload time | 32 minutes |
| Store created to fully searchable | 3 h 46 min |
Fields compared: text, text with tashkeel, footnotes, pages, sequence number, book, version, author, author death year and categories. The one difference: Mixedbread trims trailing whitespace from chunk text (1,243 of the 1,547 chunks). For books over 2,000 chunks, the Mixedbread count was taken from the highest sequence number.
Quality
We replayed 380 search calls that the production agent made between 13 and 19 November: 315 unfiltered and 65 with book, author, genre or death-year filters. Three LLM judges graded every passage either system returned from 0 to 3, blind to its source, and we took the median grade.
Unfiltered searches (314 scored; about 95% of production searches), nDCG@10:
| Unfiltered, top 10 | nDCG | Mean grade (0–3) | Relevant passages |
|---|---|---|---|
| Turbopuffer, no rerank | 0.616 | 2.04 | 69% |
| Turbopuffer + Rerank Fast | 0.737 | 2.31 | 80% |
| Turbopuffer + Rerank Pro | 0.760 | 2.36 | 82% |
| Mixedbread, no rerank | 0.843 | 2.55 | 89% |
| Mixedbread + Rerank Fast | 0.830 | 2.53 | 89% |
| Mixedbread + Rerank Pro | 0.842 | 2.55 | 90% |
| Mixedbread + its own reranker | 0.882 | 2.62 | 91% |
"Relevant" means a median grade of 2 or 3. The ranking is the same at top 5 and top 15, except that Mixedbread with and without Rerank Pro, which are tied, swap places.
Head to head, unfiltered, nDCG at top 15 (the largest number of passages the app delivers):
| Comparison | Difference | 95% interval | Queries won / lost |
|---|---|---|---|
| Mixedbread no rerank vs Turbopuffer no rerank | +0.232 | +0.212 to +0.254 | 283 / 19 |
| Mixedbread no rerank vs Turbopuffer + Fast | +0.119 | +0.098 to +0.139 | 219 / 52 |
| Mixedbread no rerank vs Turbopuffer + Pro | +0.098 | +0.079 to +0.117 | 207 / 60 |
| Mixedbread own reranker vs Turbopuffer + Pro | +0.134 | +0.117 to +0.152 | 237 / 28 |
| Mixedbread + Pro vs Mixedbread no rerank | +0.001 | −0.011 to +0.012 | 126 / 114 |
Filtered searches (65 queries, 63 scored; two matched nothing in either system) are closer. Mixedbread with no rerank ties Turbopuffer + Pro (nDCG@10 0.776 for both) and clearly beats Turbopuffer with no rerank (0.656). Mixedbread with its own reranker scores 0.826; its lead over Turbopuffer + Pro is clear at top 15 (+0.058, interval +0.012 to +0.099) but not at top 5.
Speed
120 queries, one at a time, timed end to end as the app would run each setup. Milliseconds.
| Setup | Median | p95 |
|---|---|---|
| Turbopuffer, no rerank | 1,235 | 1,482 |
| Turbopuffer + Rerank Fast | 1,961 | 2,613 |
| Turbopuffer + Rerank Pro | 2,131 | 2,778 |
| Mixedbread, no rerank | 232 | 501 |
| Mixedbread + its own reranker | 580 | 1,003 |
| Mixedbread + Rerank Fast | 1,051 | 1,678 |
| Mixedbread + Rerank Pro | 1,162 | 1,735 |
More than half of Turbopuffer's time is the separate query-embedding call (median 681 ms); Mixedbread embeds the query itself. With 8 searches in parallel the medians barely moved (Turbopuffer 1,273 ms, Mixedbread 246 ms). The reranker rows rerank the same 48 candidates that were graded; the three Mixedbread reranker rows come from a second run an hour later.
Every query completed on both systems. The latency runs, which have no retries, saw 0 errors; the 380-query run retries transient errors silently, so first-attempt failures there are not counted.
Cost
Production made about 198,000 search calls a day in the sampled week (counted in Langfuse), roughly 6 million a month. Today only the top chat tier uses Rerank Pro; the other tiers use Fast, so current rerank spend is a blend of the two Cohere rows.
| Item | Unit price | At 6M searches / month |
|---|---|---|
| Mixedbread search | $4.00 per 1,000 | ≈ $24,000 |
| Mixedbread search with its reranker | $7.50 per 1,000 | ≈ $45,000 |
| Mixedbread storage (3.08B tokens) | $0.10 per M tokens / month | ≈ $310 |
| Mixedbread indexing, one time | $0.25 per M tokens | ≈ $770 once |
| Cohere Rerank Pro on Turbopuffer results | ≈ $0.78 per 1,000 | ≈ $4,700 |
| Cohere Rerank Fast on Turbopuffer results | ≈ $0.30 per 1,000 | ≈ $1,800 |
Mixedbread figures are public list prices; early-access pricing for v4 was not confirmed. Cohere figures use the tokens actually billed in this benchmark (15,000 to 15,500 per search) at $0.05 and $0.02 per million, and assume every search is reranked. The Turbopuffer and query-embedding bills are not included; compare against the current invoices.
What this does not show
- Judges are models, not people. They agreed with each other within one grade 95–98% of the time and gave planted irrelevant passages a relevant grade less than 0.1% of the time, but no human graded these results.
- Retrieval quality is not answer quality. Our August reranker study found better retrieval did not always produce better answers. An answer-level test is the next step before switching.
- Speed was measured from one machine, not from the production servers. Expect the gaps to hold, not the exact numbers.
- One week of queries, and only 65 filtered ones, so the filtered results have wide intervals.
- Mixedbread's passages are about 15% longer (1,062 vs 926 characters on average). Judges grade longer passages somewhat higher, so length may explain part of the gap, and it means more tokens passed to the chat model per search.
- The search volume has one source. Cross-check the 198,000 a day against Azure's rerank request count before using it in a price negotiation.
- Not tested: incremental updates and deletes when a book changes, and the app integration itself. Mixedbread renumbers chunks per uploaded file, so the app would read our sequence number and chunk id from the metadata we stored.
Method
Turbopuffer ran the production path: Gemini query embedding, vector search plus BM25, 24 candidates each, merged by reciprocal rank fusion. Mixedbread returned its top 48. Cohere Rerank v5 Pro and Fast reordered each system's full pool. Judges were GPT-6.1 Sol, Gemini 3.1 Pro and Gemini 3.7 Flash, grading the union of both pools in shuffled batches with three planted irrelevant passages per batch. One query was dropped because a judge refused it, leaving 379 scored. Intervals are paired bootstrap, 4,000 resamples.