QafQaf · Search benchmark · 21 November 2026

Mixedbread v4 finds better passages than Turbopuffer, and finds them faster

The whole library is now loaded in Mixedbread on mxbai-wholembed-v4 and matches Turbopuffer on every check we ran. On 380 real production searches, Mixedbread with no reranking beat our current best setup (Turbopuffer plus Cohere Rerank Pro) on quality and answered about nine times faster. At list price, though, Mixedbread search would cost about $24k a month, against roughly $2–5k for Cohere reranking today plus the Turbopuffer and embedding bills.

0.84 vs 0.76nDCG@10 on unfiltered searches: Mixedbread with no rerank vs Turbopuffer + Rerank Pro
0.23 s vs 2.1 smedian time per search, same comparison
$4 per 1,000Mixedbread list price per search; about $24k a month at our volume

Recommendation

Migration

All 8,583 book versions and 7,455,835 chunks were read from R2 and uploaded as 11,723 files, with zero failures. Turbopuffer's total row count is the same 7,455,835.

CheckResult
Files indexed11,723 of 11,723, 0 failed
Tokens indexed3.08 billion
Chunk counts, 250 books: 150 spread across the library plus the 100 largest (R2 vs Turbopuffer vs Mixedbread)250 of 250 match
Field-by-field comparison, 1,547 chunks from those books0 mismatches, 0 missing
Filtered searches returning only matching results (65 queries, 3,024 results)0 violations
Upload time32 minutes
Store created to fully searchable3 h 46 min

Fields compared: text, text with tashkeel, footnotes, pages, sequence number, book, version, author, author death year and categories. The one difference: Mixedbread trims trailing whitespace from chunk text (1,243 of the 1,547 chunks). For books over 2,000 chunks, the Mixedbread count was taken from the highest sequence number.

Quality

We replayed 380 search calls that the production agent made between 13 and 19 November: 315 unfiltered and 65 with book, author, genre or death-year filters. Three LLM judges graded every passage either system returned from 0 to 3, blind to its source, and we took the median grade.

Unfiltered searches (314 scored; about 95% of production searches), nDCG@10:

Turbopuffer, no rerank0.62
Turbopuffer + Rerank Fast0.74
Turbopuffer + Rerank Pro0.76
Mixedbread, no rerank0.84
Mixedbread + Rerank Fast0.83
Mixedbread + Rerank Pro0.84
Mixedbread + its own reranker0.88
Unfiltered, top 10nDCGMean grade (0–3)Relevant passages
Turbopuffer, no rerank0.6162.0469%
Turbopuffer + Rerank Fast0.7372.3180%
Turbopuffer + Rerank Pro0.7602.3682%
Mixedbread, no rerank0.8432.5589%
Mixedbread + Rerank Fast0.8302.5389%
Mixedbread + Rerank Pro0.8422.5590%
Mixedbread + its own reranker0.8822.6291%

"Relevant" means a median grade of 2 or 3. The ranking is the same at top 5 and top 15, except that Mixedbread with and without Rerank Pro, which are tied, swap places.

Head to head, unfiltered, nDCG at top 15 (the largest number of passages the app delivers):

ComparisonDifference95% intervalQueries won / lost
Mixedbread no rerank vs Turbopuffer no rerank+0.232+0.212 to +0.254283 / 19
Mixedbread no rerank vs Turbopuffer + Fast+0.119+0.098 to +0.139219 / 52
Mixedbread no rerank vs Turbopuffer + Pro+0.098+0.079 to +0.117207 / 60
Mixedbread own reranker vs Turbopuffer + Pro+0.134+0.117 to +0.152237 / 28
Mixedbread + Pro vs Mixedbread no rerank+0.001−0.011 to +0.012126 / 114

Filtered searches (65 queries, 63 scored; two matched nothing in either system) are closer. Mixedbread with no rerank ties Turbopuffer + Pro (nDCG@10 0.776 for both) and clearly beats Turbopuffer with no rerank (0.656). Mixedbread with its own reranker scores 0.826; its lead over Turbopuffer + Pro is clear at top 15 (+0.058, interval +0.012 to +0.099) but not at top 5.

The two systems mostly return different passages: of the candidate pools (48 for Mixedbread, 46 on average for Turbopuffer after its two branches overlap), about 11 passages are in common, roughly a quarter of each pool. On unfiltered queries, 82% of Mixedbread's pool was relevant against 59% of Turbopuffer's. This is why Cohere helps Turbopuffer a lot and Mixedbread not at all: Mixedbread's first-pass ordering is already good.

Speed

120 queries, one at a time, timed end to end as the app would run each setup. Milliseconds.

SetupMedianp95
Turbopuffer, no rerank1,2351,482
Turbopuffer + Rerank Fast1,9612,613
Turbopuffer + Rerank Pro2,1312,778
Mixedbread, no rerank232501
Mixedbread + its own reranker5801,003
Mixedbread + Rerank Fast1,0511,678
Mixedbread + Rerank Pro1,1621,735

More than half of Turbopuffer's time is the separate query-embedding call (median 681 ms); Mixedbread embeds the query itself. With 8 searches in parallel the medians barely moved (Turbopuffer 1,273 ms, Mixedbread 246 ms). The reranker rows rerank the same 48 candidates that were graded; the three Mixedbread reranker rows come from a second run an hour later.

Every query completed on both systems. The latency runs, which have no retries, saw 0 errors; the 380-query run retries transient errors silently, so first-attempt failures there are not counted.

Cost

Production made about 198,000 search calls a day in the sampled week (counted in Langfuse), roughly 6 million a month. Today only the top chat tier uses Rerank Pro; the other tiers use Fast, so current rerank spend is a blend of the two Cohere rows.

ItemUnit priceAt 6M searches / month
Mixedbread search$4.00 per 1,000≈ $24,000
Mixedbread search with its reranker$7.50 per 1,000≈ $45,000
Mixedbread storage (3.08B tokens)$0.10 per M tokens / month≈ $310
Mixedbread indexing, one time$0.25 per M tokens≈ $770 once
Cohere Rerank Pro on Turbopuffer results≈ $0.78 per 1,000≈ $4,700
Cohere Rerank Fast on Turbopuffer results≈ $0.30 per 1,000≈ $1,800

Mixedbread figures are public list prices; early-access pricing for v4 was not confirmed. Cohere figures use the tokens actually billed in this benchmark (15,000 to 15,500 per search) at $0.05 and $0.02 per million, and assume every search is reranked. The Turbopuffer and query-embedding bills are not included; compare against the current invoices.

What this does not show

Method

Turbopuffer ran the production path: Gemini query embedding, vector search plus BM25, 24 candidates each, merged by reciprocal rank fusion. Mixedbread returned its top 48. Cohere Rerank v5 Pro and Fast reordered each system's full pool. Judges were GPT-6.1 Sol, Gemini 3.1 Pro and Gemini 3.7 Flash, grading the union of both pools in shuffled batches with three planted irrelevant passages per batch. One query was dropped because a judge refused it, leaving 379 scored. Intervals are paired bootstrap, 4,000 resamples.