BGE Reranker Base — Cross-Encoder Ranking
16 configurations testedQuality criteria
Finite scores, deterministic descending order, and the France passage must outrank the unrelated whale passage by at least 0.25 sigmoid probability.
| Precision | Device | Result | |
|---|---|---|---|
Q4 | GPU · WebGPUAccelerated | Passed | Fast |
INT8 | GPU · WebGPUAccelerated | Passed | Fast |
UINT8 | GPU · WebGPUAccelerated | Passed | Fast |
Q8 | GPU · WebGPUAccelerated | Passed | Fast |
FP32 | GPU · WebGPUAccelerated | Passed | Fast |
BNB4 | GPU · WebGPUAccelerated | Passed | Moderate |
FP32 | CPU · WASM | Passed | Fast |
UINT8 | CPU · WASM | Passed | Fast |
Q8 | CPU · WASM | Passed | Fast |
INT8 | CPU · WASM | Passed | Fast |
BNB4 | CPU · WASM | Passed | Fast |
Q4 | CPU · WASM | Passed | Fast |
FP16 | CPU · WASM | Failed | — |
FP16 | GPU · WebGPU | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
Q4F16 | GPU · WebGPU | Failed | — |
Test setup & sample details
- Quality gate
- Finite scores, deterministic descending order, and the France passage must outrank the unrelated whale passage by at least 0.25 sigmoid probability.
- Tested
- 2026-09-14T20:55:17.075Z
- Environment
- Isolated headless Chrome on macOS; Transformers.js 4.2.0
- Runtime
- Transformers.js 4.2.0
- Artifact revision
- 280bcc27a84e0b898c251e06fddb25171bd9b101
- Fixture
{ "file": "bge-m3.json", "url": "https://f005.backblazeb2.com/file/hubters-webai-public/assets/benchmarks/bge-m3.json", "size": 500, "sha1": "75a7770d06fc312ddeb39987712a8b4bbe6f1e08", "source": "Original WebAI semantic retrieval fixture.", "license": "MIT", "licenseUrl": "https://opensource.org/license/mit/", "preprocessing": "First text is the query; remaining texts are candidate documents." }- Rerun when
- artifact revision change; runtime version change; fixture or quality-gate change; browser/backend behavior change
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "wasm",
"status": "pass",
"quality": {
"grade": "ranking-pass",
"margin": 0.9997
},
"runtime": {
"loadMs": 38416.5,
"inferenceMs": 207.90000000596046,
"totalMs": 38783.59999999404,
"pairsPerSecond": 9.62,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"grade": "ranking-pass",
"margin": 0.9997
},
"runtime": {
"loadMs": 37458.90000000596,
"inferenceMs": 437.2000000178814,
"totalMs": 38002.09999999404,
"pairsPerSecond": 4.575,
"grade": "slow"
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": "Not embedded",
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/roberta/encoder/layer.11/output/LayerNorm/Constant_output_0for node: /roberta/embeddings/LayerNorm/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "fp16",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": "Not embedded",
"failure": "quality gate failed: top=1, margin=-0.24457595644260421"
},
{
"precision": "int8",
"device": "wasm",
"status": "pass",
"quality": {
"grade": "ranking-pass",
"margin": 0.9999
},
"runtime": {
"loadMs": 24754.59999999404,
"inferenceMs": 222.7000000178814,
"totalMs": 25157.90000000596,
"pairsPerSecond": 8.981,
"grade": "good"
}
},
{
"precision": "int8",
"device": "webgpu",
"status": "pass",
"quality": {
"grade": "ranking-pass",
"margin": 0.9998
},
"runtime": {
"loadMs": 27378.40000000596,
"inferenceMs": 306.40000000596046,
"totalMs": 27784.69999998808,
"pairsPerSecond": 6.527,
"grade": "slow"
}
},
{
"precision": "uint8",
"device": "wasm",
"status": "pass",
"quality": {
"grade": "ranking-pass",
"margin": 0.9998
},
"runtime": {
"loadMs": 17714.59999999404,
"inferenceMs": 220.19999998807907,
"totalMs": 18042,
"pairsPerSecond": 9.083,
"grade": "good"
}
},
{
"precision": "uint8",
"device": "webgpu",
"status": "pass",
"quality": {
"grade": "ranking-pass",
"margin": 0.9998
},
"runtime": {
"loadMs": 14620.90000000596,
"inferenceMs": 308.2999999821186,
"totalMs": 15067.699999988079,
"pairsPerSecond": 6.487,
"grade": "slow"
}
},
{
"precision": "q8",
"device": "wasm",
"status": "pass",
"quality": {
"grade": "ranking-pass",
"margin": 0.9996
},
"runtime": {
"loadMs": 11681.600000023842,
"inferenceMs": 221.69999998807907,
"totalMs": 12031.300000011921,
"pairsPerSecond": 9.021,
"grade": "good"
}
},
{
"precision": "q8",
"device": "webgpu",
"status": "pass",
"quality": {
"grade": "ranking-pass",
"margin": 0.9996
},
"runtime": {
"loadMs": 11473.300000011921,
"inferenceMs": 370.19999998807907,
"totalMs": 11951.59999999404,
"pairsPerSecond": 5.402,
"grade": "slow"
}
},
{
"precision": "q4",
"device": "wasm",
"status": "pass",
"quality": {
"grade": "ranking-pass",
"margin": 0.9998
},
"runtime": {
"loadMs": 35621,
"inferenceMs": 322.59999999403954,
"totalMs": 36118.39999997616,
"pairsPerSecond": 6.2,
"grade": "slow"
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"grade": "ranking-pass",
"margin": 0.9998
},
"runtime": {
"loadMs": 30102.5,
"inferenceMs": 239.19999998807907,
"totalMs": 30552.90000000596,
"pairsPerSecond": 8.361,
"grade": "good"
}
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": "Not embedded",
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/roberta/embeddings/LayerNorm/Constant_output_0for node: /roberta/embeddings/LayerNorm/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": "Not embedded",
"failure": "quality gate failed: top=1, margin=-0.24198654101080747"
},
{
"precision": "bnb4",
"device": "wasm",
"status": "pass",
"quality": {
"grade": "ranking-pass",
"margin": 0.9996
},
"runtime": {
"loadMs": 30218.5,
"inferenceMs": 261.09999999403954,
"totalMs": 30566.80000001192,
"pairsPerSecond": 7.66,
"grade": "slow"
}
},
{
"precision": "bnb4",
"device": "webgpu",
"status": "pass",
"quality": {
"grade": "ranking-pass",
"margin": 0.9996
},
"runtime": {
"loadMs": 27562.19999998808,
"inferenceMs": 516,
"totalMs": 28174.29999998212,
"pairsPerSecond": 3.876,
"grade": "slow"
}
}
]