Distil-Whisper Large v3.5
16 configurations testedQuality criteria
{ "maximumWordErrorRate": 0.2, "rejectEmpty": true, "rejectGibberish": true, "requireValidSegmentTimestamps": true }
| Precision | Device | Result | |
|---|---|---|---|
Q4 | GPU · WebGPUAccelerated | Passed | Fast |
Q4 | CPU · WASM | Passed | Very slow |
BNB4 | GPU · WebGPU | Failed | — |
BNB4 | CPU · WASM | Failed | — |
Q4F16 | GPU · WebGPU | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
Q8 | GPU · WebGPU | Failed | — |
Q8 | CPU · WASM | Failed | — |
INT8 | GPU · WebGPU | Failed | — |
INT8 | CPU · WASM | Failed | — |
UINT8 | GPU · WebGPU | Failed | — |
UINT8 | CPU · WASM | Failed | — |
FP16 | GPU · WebGPU | Failed | — |
FP16 | CPU · WASM | Failed | — |
FP32 | GPU · WebGPU | Failed | — |
FP32 | CPU · WASM | Failed | — |
Test setup & sample details
- Quality gate
- { "maximumWordErrorRate": 0.2, "rejectEmpty": true, "rejectGibberish": true, "requireValidSegmentTimestamps": true }
- Tested
- 2026-11-02T14:56:28.712Z
- Environment
- { "browser": "156.0.8078.79", "os": "Darwin", "release": "27.2.0", "arch": "arm64", "cpu": "Apple M3 Max", "runtime": { "name": "Transformers.js", "version": "4.2.0" }, "workerType": "module", "cache": "fresh disk-backed profile per case", "artifactTransport": "checksum-verified pinned local mirror", "wasmThreads": 1 }
- Runtime
- Transformers.js 4.2.0
- Artifact revision
- d908cd7fba0fb226f14b14ad955f22797ecfe5d5
- Fixture
{ "file": "model-samples/vosk-jfk.flac", "url": "https://assets.localai.tools/assets/benchmarks/vosk/63a4b1e4c1dc655ac70961ffbf518acd249df237e5a0152faae9a4a836949715/vosk-jfk.flac", "size": 1152693, "sha1": "537ddf75dff344f8d31c1cd6ebee23ee8362d921", "sha256": "63a4b1e4c1dc655ac70961ffbf518acd249df237e5a0152faae9a4a836949715", "durationSeconds": 11 }- Rerun when
- model artifact revision change; runtime version change; fixture or quality-gate change; browser/backend behavior change
Per-case quality & failure details
[
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": true,
"grade": "excellent",
"cases": [
{
"fixtureId": "jfk",
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": true,
"grade": "excellent"
},
{
"fixtureId": "standard",
"wordErrorRate": 0.11940298507462686,
"referenceWords": 67,
"outputWords": 67,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": true,
"grade": "good"
}
]
},
"runtime": {
"loadMs": 3959.2000000029802,
"inferenceMs": 1309.6000000089407,
"totalMs": 10187.29999999702,
"grade": "excellent",
"realTimeFactor": 0.11905454545535825
}
},
{
"precision": "q4",
"device": "wasm",
"status": "pass",
"quality": {
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": true,
"grade": "excellent",
"cases": [
{
"fixtureId": "jfk",
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": true,
"grade": "excellent"
},
{
"fixtureId": "standard",
"wordErrorRate": 0.11940298507462686,
"referenceWords": 67,
"outputWords": 67,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": true,
"grade": "good"
}
]
},
"runtime": {
"loadMs": 7603.20000000298,
"inferenceMs": 53786.59999999404,
"totalMs": 179649.39999999106,
"grade": "very-slow",
"realTimeFactor": 4.889690909090367
}
},
{
"precision": "bnb4",
"device": "webgpu",
"status": "fail",
"quality": {
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "excellent",
"cases": [
{
"fixtureId": "jfk",
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": true,
"grade": "excellent"
},
{
"fixtureId": "standard",
"wordErrorRate": 0.208955223880597,
"referenceWords": 67,
"outputWords": 74,
"longestRepeatedWordRun": 2,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "failed"
}
]
},
"runtime": {
"loadMs": 3141.2000000029802,
"inferenceMs": 42814.59999999404,
"totalMs": 136188.19999998808,
"grade": "very-slow",
"realTimeFactor": 3.892236363635822
},
"failure": "Transcription quality/timestamp gate failed"
},
{
"precision": "bnb4",
"device": "wasm",
"status": "fail",
"quality": {
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "excellent",
"cases": [
{
"fixtureId": "jfk",
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": true,
"grade": "excellent"
},
{
"fixtureId": "standard",
"wordErrorRate": 0.208955223880597,
"referenceWords": 67,
"outputWords": 74,
"longestRepeatedWordRun": 2,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "failed"
}
]
},
"runtime": {
"loadMs": 4102,
"inferenceMs": 51673.29999999702,
"totalMs": 161451.5,
"grade": "very-slow",
"realTimeFactor": 4.697572727272457
},
"failure": "Transcription quality/timestamp gate failed"
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "fail",
"quality": {
"wordErrorRate": 0.7727272727272727,
"referenceWords": 22,
"outputWords": 5,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "failed",
"cases": [
{
"fixtureId": "jfk",
"wordErrorRate": 0.7727272727272727,
"referenceWords": 22,
"outputWords": 5,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "failed"
},
{
"fixtureId": "standard",
"wordErrorRate": 0.7611940298507462,
"referenceWords": 67,
"outputWords": 34,
"longestRepeatedWordRun": 10,
"gibberish": true,
"validTimestamps": false,
"passed": false,
"grade": "failed"
}
]
},
"runtime": {
"loadMs": 2436.0999999940395,
"inferenceMs": 1904.5999999940395,
"totalMs": 8336.29999999702,
"grade": "excellent",
"realTimeFactor": 0.17314545454491267
},
"failure": "Transcription quality/timestamp gate failed"
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 3986.3999999910593,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/layers.0/self_attn_layer_norm/Constant_output_0for node: /layers.0/self_attn_layer_norm/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "q8",
"device": "webgpu",
"status": "fail",
"quality": {
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "excellent",
"cases": [
{
"fixtureId": "jfk",
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": true,
"grade": "excellent"
},
{
"fixtureId": "standard",
"wordErrorRate": 0.208955223880597,
"referenceWords": 67,
"outputWords": 74,
"longestRepeatedWordRun": 2,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "failed"
}
]
},
"runtime": {
"loadMs": 3290.2999999970198,
"inferenceMs": 46085,
"totalMs": 142943.79999999702,
"grade": "very-slow",
"realTimeFactor": 4.1895454545454545
},
"failure": "Transcription quality/timestamp gate failed"
},
{
"precision": "q8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 4654.29999999702,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: qdq_actions.cc:137 TransposeDQWeightsForMatMulNBits Missing required scale: model.decoder.embed_tokens.weight_merged_0_scale for node: model.decoder.embed_tokens.weight_transposed_DequantizeLinear"
},
{
"precision": "int8",
"device": "webgpu",
"status": "fail",
"quality": {
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "excellent",
"cases": [
{
"fixtureId": "jfk",
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": true,
"grade": "excellent"
},
{
"fixtureId": "standard",
"wordErrorRate": 0.208955223880597,
"referenceWords": 67,
"outputWords": 74,
"longestRepeatedWordRun": 2,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "failed"
}
]
},
"runtime": {
"loadMs": 4951,
"inferenceMs": 44543.10000000894,
"totalMs": 140080.5,
"grade": "very-slow",
"realTimeFactor": 4.04937272727354
},
"failure": "Transcription quality/timestamp gate failed"
},
{
"precision": "int8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 4709.899999991059,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: qdq_actions.cc:137 TransposeDQWeightsForMatMulNBits Missing required scale: model.decoder.embed_tokens.weight_merged_0_scale for node: model.decoder.embed_tokens.weight_transposed_DequantizeLinear"
},
{
"precision": "uint8",
"device": "webgpu",
"status": "fail",
"quality": {
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "excellent",
"cases": [
{
"fixtureId": "jfk",
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": true,
"grade": "excellent"
},
{
"fixtureId": "standard",
"wordErrorRate": 0.208955223880597,
"referenceWords": 67,
"outputWords": 74,
"longestRepeatedWordRun": 2,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "failed"
}
]
},
"runtime": {
"loadMs": 3111.7999999970198,
"inferenceMs": 43913,
"totalMs": 134863.40000000596,
"grade": "very-slow",
"realTimeFactor": 3.992090909090909
},
"failure": "Transcription quality/timestamp gate failed"
},
{
"precision": "uint8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 4539.100000008941,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: qdq_actions.cc:137 TransposeDQWeightsForMatMulNBits Missing required scale: model.decoder.embed_tokens.weight_merged_0_scale for node: model.decoder.embed_tokens.weight_transposed_DequantizeLinear"
},
{
"precision": "fp16",
"device": "webgpu",
"status": "fail",
"quality": {
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": false,
"grade": "excellent",
"cases": [
{
"fixtureId": "jfk",
"wordErrorRate": 0,
"referenceWords": 22,
"outputWords": 22,
"longestRepeatedWordRun": 1,
"gibberish": false,
"validTimestamps": true,
"passed": true,
"grade": "excellent"
},
{
"fixtureId": "standard",
"wordErrorRate": 0.7910447761194029,
"referenceWords": 67,
"outputWords": 29,
"longestRepeatedWordRun": 5,
"gibberish": true,
"validTimestamps": false,
"passed": false,
"grade": "failed"
}
]
},
"runtime": {
"loadMs": 3865,
"inferenceMs": 1006.2000000029802,
"totalMs": 8840.70000000298,
"grade": "excellent",
"realTimeFactor": 0.0914727272729982
},
"failure": "Transcription quality/timestamp gate failed"
},
{
"precision": "fp16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 5079.5999999940395,
"grade": "failed"
},
"failure": "Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc"
},
{
"precision": "fp32",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 3783.053,
"grade": "failed"
},
"failure": "Failed to construct 'Response': The ArrayBuffer/ArrayBufferView size exceeds the supported range."
},
{
"precision": "fp32",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 3731.0329170000005,
"grade": "failed"
},
"failure": "Failed to construct 'Response': The ArrayBuffer/ArrayBufferView size exceeds the supported range."
}
]