DistilBERT SST-2 English Sentiment
16 configurations testedQuality criteria
Both expected sentiment labels correct with confidence >=0.8; finite probabilities in [0,1] summing to one and distinct predictions.
| Precision | Device | Result | |
|---|---|---|---|
Q4 | GPU · WebGPUAccelerated | Passed | Fast |
Q4F16 | GPU · WebGPUAccelerated | Passed | Fast |
BNB4 | GPU · WebGPUAccelerated | Passed | Fast |
INT8 | GPU · WebGPUAccelerated | Passed | Fast |
UINT8 | GPU · WebGPUAccelerated | Passed | Fast |
FP32 | GPU · WebGPUAccelerated | Passed | Fast |
FP16 | GPU · WebGPUAccelerated | Passed | Fast |
Q8 | GPU · WebGPUAccelerated | Passed | Fast |
FP32 | CPU · WASM | Passed | Very fast |
Q8 | CPU · WASM | Passed | Very fast |
INT8 | CPU · WASM | Passed | Very fast |
UINT8 | CPU · WASM | Passed | Very fast |
BNB4 | CPU · WASM | Passed | Fast |
Q4 | CPU · WASM | Passed | Fast |
FP16 | CPU · WASM | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
Test setup & sample details
- Quality gate
- Both expected sentiment labels correct with confidence >=0.8; finite probabilities in [0,1] summing to one and distinct predictions.
- Tested
- 2026-09-15T18:11:44.541Z
- Environment
- Isolated headless Chrome 151 on macOS; Transformers.js 4.2.0
- Runtime
- Transformers.js 4.2.0
- Artifact revision
- 0b6928efcb76139cae2c6881d49cda67fe119f42
- Fixture
{ "file": "distilbert-sst2.json", "url": "https://f005.backblazeb2.com/file/hubters-webai-public/assets/benchmarks/distilbert-sst2.json", "size": 486, "sha1": "226199feb31ba0ad6ffa6629585a76930b240a3a", "source": "Xenova model-card positive example and WebAI-authored negative review", "license": "Apache-2.0", "preprocessing": "Pinned tokenizer truncates English input at max_length." }- Rerun when
- artifact revision change; runtime version change; fixture or quality-gate change; browser/backend behavior change
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "wasm",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.9998068605929036,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9995625767894342,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 37308.40000000596,
"inferenceMs": 58.80000001192093,
"totalMs": 38643.30000001192,
"grade": "excellent",
"textsPerSecond": 34.01360543528106
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.9998068606849819,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9995625754342574,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 34159.10000002384,
"inferenceMs": 239.09999999403954,
"totalMs": 35713.30000001192,
"grade": "good",
"textsPerSecond": 8.364700962149133
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 23291,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/distilbert/transformer/layer.5/output_layer_norm/Constant_output_0for node: /distilbert/embeddings/LayerNorm/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "fp16",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.9997507674754179,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9990359080961361,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 21414,
"inferenceMs": 248.59999999403954,
"totalMs": 22897,
"grade": "good",
"textsPerSecond": 8.045052293032793
}
},
{
"precision": "int8",
"device": "wasm",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.9997704615574663,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9995241222127412,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 14183.59999999404,
"inferenceMs": 74.2000000178814,
"totalMs": 15508.90000000596,
"grade": "excellent",
"textsPerSecond": 26.95417789107846
}
},
{
"precision": "int8",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.9997704615574663,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9995241225529534,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 14261.799999982119,
"inferenceMs": 198.40000000596046,
"totalMs": 15788.200000017881,
"grade": "good",
"textsPerSecond": 10.080645160987473
}
},
{
"precision": "uint8",
"device": "wasm",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.999823115266032,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9996465835611709,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 12392.899999976158,
"inferenceMs": 74.5,
"totalMs": 13833.5,
"grade": "excellent",
"textsPerSecond": 26.845637583892618
}
},
{
"precision": "uint8",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.999823115266032,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.999646582718857,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 11884.90000000596,
"inferenceMs": 205.90000000596046,
"totalMs": 13292.40000000596,
"grade": "good",
"textsPerSecond": 9.713453132307446
}
},
{
"precision": "q8",
"device": "wasm",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.9997887322507517,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9995639365188616,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 12910.800000011921,
"inferenceMs": 73.5,
"totalMs": 14227,
"grade": "excellent",
"textsPerSecond": 27.210884353741495
}
},
{
"precision": "q8",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.9997887157321852,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9995639338169253,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 13694.300000011921,
"inferenceMs": 384.90000000596046,
"totalMs": 15297.59999999404,
"grade": "slow",
"textsPerSecond": 5.196154845333926
}
},
{
"precision": "q4",
"device": "wasm",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.9998047775897084,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9995598592883355,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 19400.5,
"inferenceMs": 185.5,
"totalMs": 20935.80000001192,
"grade": "good",
"textsPerSecond": 10.781671159029651
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.9998047778689217,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9995598596030099,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 21880.59999999404,
"inferenceMs": 143,
"totalMs": 23247.29999998212,
"grade": "good",
"textsPerSecond": 13.986013986013987
}
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 15357.800000011921,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/distilbert/transformer/layer.5/output_layer_norm/Constant_output_0for node: /distilbert/embeddings/LayerNorm/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.999731571162093,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9990302480322174,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 11828.199999988079,
"inferenceMs": 186.10000002384186,
"totalMs": 13364.5,
"grade": "good",
"textsPerSecond": 10.74691026192248
}
},
{
"precision": "bnb4",
"device": "wasm",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.9997535227530785,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9996075217275017,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 13846.699999988079,
"inferenceMs": 116.30000001192093,
"totalMs": 15101,
"grade": "good",
"textsPerSecond": 17.196904555417
}
},
{
"precision": "bnb4",
"device": "webgpu",
"status": "pass",
"quality": {
"cases": [
{
"expected": "POSITIVE",
"actual": "POSITIVE",
"score": 0.9997535227530785,
"finite": true,
"pass": true
},
{
"expected": "NEGATIVE",
"actual": "NEGATIVE",
"score": 0.9996075218210392,
"finite": true,
"pass": true
}
],
"accuracy": 1,
"grade": "sentiment-regression-pass"
},
"runtime": {
"loadMs": 13052.5,
"inferenceMs": 193.19999998807907,
"totalMs": 14387.90000000596,
"grade": "good",
"textsPerSecond": 10.351966874344747
}
}
]