SmolLM3 3B Base
32 configurations testedPassed6
Failed26
TestedNov 7, 2026
Runtimetransformers.js 4.2.0
Quality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | |
|---|---|---|---|
Q4F16 | GPU · WebGPUAccelerated | Passed | Fast |
Q4F16 | GPU · WebGPUAccelerated | Passed | Fast |
Q4 | GPU · WebGPUAccelerated | Passed | Fast |
Q4 | GPU · WebGPUAccelerated | Passed | Fast |
Q4F16 | GPU · WebGPUAccelerated | Passed | Slow |
Q4 | GPU · WebGPUAccelerated | Passed | Slow |
Q4F16 | CPU · WASM | Failed | — |
Q4 | CPU · WASM | Failed | — |
BNB4 | GPU · WebGPU | Failed | — |
BNB4 | GPU · WebGPU | Failed | — |
BNB4 | GPU · WebGPU | Failed | — |
BNB4 | CPU · WASM | Failed | — |
FP16 | GPU · WebGPU | Failed | — |
FP16 | GPU · WebGPU | Failed | — |
FP16 | GPU · WebGPU | Failed | — |
FP16 | CPU · WASM | Failed | — |
FP32 | GPU · WebGPU | Failed | — |
FP32 | GPU · WebGPU | Failed | — |
FP32 | GPU · WebGPU | Failed | — |
FP32 | CPU · WASM | Failed | — |
INT8 | GPU · WebGPU | Failed | — |
INT8 | GPU · WebGPU | Failed | — |
INT8 | GPU · WebGPU | Failed | — |
INT8 | CPU · WASM | Failed | — |
UINT8 | GPU · WebGPU | Failed | — |
UINT8 | GPU · WebGPU | Failed | — |
UINT8 | GPU · WebGPU | Failed | — |
UINT8 | CPU · WASM | Failed | — |
Q8 | GPU · WebGPU | Failed | — |
Q8 | GPU · WebGPU | Failed | — |
Q8 | GPU · WebGPU | Failed | — |
Q8 | CPU · WASM | Failed | — |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-07T12:06:00.559Z
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
- Runtime
- transformers.js 4.2.0
- Artifact revision
- d78a42f79198603e614095753484a04c10c2b940
- Fixture
{ "file": "model-samples/smollm3/smollm3-3b-base-geography.json", "size": 152, "sha1": "f033f82b9e5c7821b37f56df58bd3f18c26c24ef", "sha256": "48f3547630afad495ce68672989f85e41ccbc27ef56df0d2805f36097901dd21", "url": "https://assets.localai.tools/samples/v1/files/48f3547630afad495ce68672989f85e41ccbc27ef56df0d2805f36097901dd21/smollm3-3b-base-geography.json" }- Rerun when
- Artifact/runtime revisions; Generation/preprocessing changes; Fixture/quality scope; OS/backend behavior
Per-case quality & failure details
[
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"expectedPrefix": " France",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": true
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 889.7999999970198,
"inputTokens": 8,
"outputTokens": 8,
"tokensPerSecond": 8.990784445973022,
"loadMs": 3100.8000000044703,
"totalMs": 3990.60000000149,
"grade": "acceptable",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"expectedPrefix": " Celsius",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": true
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 163.79999999701977,
"inputTokens": 11,
"outputTokens": 8,
"tokensPerSecond": 48.84004884093745,
"loadMs": 3100.8000000044703,
"totalMs": 3264.60000000149,
"grade": "fast",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"expectedPrefix": " x",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": true
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 162.79999999701977,
"inputTokens": 16,
"outputTokens": 8,
"tokensPerSecond": 49.1400491409487,
"loadMs": 3100.8000000044703,
"totalMs": 3263.60000000149,
"grade": "fast",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"grade": "failed"
},
"failure": "Error: page.evaluate: Error: Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc\n at worker.onmessage (eval at evaluate (:291:30), <anonymous>:19:54)"
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"expectedPrefix": " France",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": true
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 1151.1000000014901,
"inputTokens": 8,
"outputTokens": 8,
"tokensPerSecond": 6.949874033524146,
"loadMs": 3934.2999999970198,
"totalMs": 5085.39999999851,
"grade": "acceptable",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"expectedPrefix": " Celsius",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": true
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 206.30000000447035,
"inputTokens": 11,
"outputTokens": 8,
"tokensPerSecond": 38.77847794390037,
"loadMs": 3934.2999999970198,
"totalMs": 4140.60000000149,
"grade": "fast",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"expectedPrefix": " x",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": true
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 205.69999999552965,
"inputTokens": 16,
"outputTokens": 8,
"tokensPerSecond": 38.89158969457394,
"loadMs": 3934.2999999970198,
"totalMs": 4139.999999992549,
"grade": "fast",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "q4",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"grade": "failed"
},
"failure": "Error: page.evaluate: Error: Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc\n at worker.onmessage (eval at evaluate (:291:30), <anonymous>:19:54)"
},
{
"precision": "bnb4",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " France",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 17132,
"inputTokens": 8,
"outputTokens": 8,
"tokensPerSecond": 0.46696240952603313,
"loadMs": 3545.10000000149,
"totalMs": 20677.10000000149,
"grade": "slow",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "bnb4",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " Celsius",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 16968.20000000298,
"inputTokens": 11,
"outputTokens": 8,
"tokensPerSecond": 0.47147016183205026,
"loadMs": 3545.10000000149,
"totalMs": 20513.30000000447,
"grade": "slow",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "bnb4",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " x",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 16820.5,
"inputTokens": 16,
"outputTokens": 8,
"tokensPerSecond": 0.47561011860527336,
"loadMs": 3545.10000000149,
"totalMs": 20365.60000000149,
"grade": "slow",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "bnb4",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"grade": "failed"
},
"failure": "Error: page.evaluate: Error: Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc\n at worker.onmessage (eval at evaluate (:291:30), <anonymous>:19:54)"
},
{
"precision": "fp16",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " France",
"prefixPassed": false,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 998.1999999955297,
"inputTokens": 8,
"outputTokens": 8,
"tokensPerSecond": 8.014425966776024,
"loadMs": 6923.30000000447,
"totalMs": 7921.5,
"grade": "acceptable",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "fp16",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " Celsius",
"prefixPassed": false,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 582.9000000059605,
"inputTokens": 11,
"outputTokens": 8,
"tokensPerSecond": 13.72448104292022,
"loadMs": 6923.30000000447,
"totalMs": 7506.200000010431,
"grade": "acceptable",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "fp16",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " x",
"prefixPassed": false,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 574.3000000044703,
"inputTokens": 16,
"outputTokens": 8,
"tokensPerSecond": 13.930001741141787,
"loadMs": 6923.30000000447,
"totalMs": 7497.600000008941,
"grade": "acceptable",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"grade": "failed"
},
"failure": "Error: page.evaluate: Error: Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc\n at worker.onmessage (eval at evaluate (:291:30), <anonymous>:19:54)"
},
{
"precision": "fp32",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " France",
"prefixPassed": false,
"repetitionPassed": false,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 1546.5,
"inputTokens": 8,
"outputTokens": 8,
"tokensPerSecond": 5.172971225347559,
"loadMs": 25900.89999999851,
"totalMs": 27447.39999999851,
"grade": "acceptable",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " Celsius",
"prefixPassed": false,
"repetitionPassed": false,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 824.0999999940395,
"inputTokens": 11,
"outputTokens": 8,
"tokensPerSecond": 9.707559762234999,
"loadMs": 25900.89999999851,
"totalMs": 26724.99999999255,
"grade": "acceptable",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " x",
"prefixPassed": false,
"repetitionPassed": false,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 803.8000000044703,
"inputTokens": 16,
"outputTokens": 8,
"tokensPerSecond": 9.952724558292495,
"loadMs": 25900.89999999851,
"totalMs": 26704.70000000298,
"grade": "acceptable",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "fp32",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"grade": "failed"
},
"failure": "Error: page.evaluate: Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/common/safeint.h:17 static void SafeIntExceptionHandler<onnxruntime::OnnxRuntimeException>::SafeIntOnOverflow() Integer overflow\n\n at worker.onmessage (eval at evaluate (:291:30), <anonymous>:19:54)"
},
{
"precision": "int8",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " France",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 6208.79999999702,
"inputTokens": 8,
"outputTokens": 8,
"tokensPerSecond": 1.288493750805927,
"loadMs": 3785.89999999851,
"totalMs": 9994.69999999553,
"grade": "slow",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "int8",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " Celsius",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 6647.70000000298,
"inputTokens": 11,
"outputTokens": 8,
"tokensPerSecond": 1.203423740541302,
"loadMs": 3785.89999999851,
"totalMs": 10433.60000000149,
"grade": "slow",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "int8",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " x",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 6707.70000000298,
"inputTokens": 16,
"outputTokens": 8,
"tokensPerSecond": 1.1926591827297652,
"loadMs": 3785.89999999851,
"totalMs": 10493.60000000149,
"grade": "slow",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "int8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"grade": "failed"
},
"failure": "Error: page.evaluate: Error: failed to call OrtRun(). ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc\n at worker.onmessage (eval at evaluate (:291:30), <anonymous>:19:54)"
},
{
"precision": "uint8",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " France",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 5921.79999999702,
"inputTokens": 8,
"outputTokens": 8,
"tokensPerSecond": 1.3509405923881297,
"loadMs": 3704.2999999970198,
"totalMs": 9626.09999999404,
"grade": "slow",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "uint8",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " Celsius",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 6258.20000000298,
"inputTokens": 11,
"outputTokens": 8,
"tokensPerSecond": 1.2783228404327427,
"loadMs": 3704.2999999970198,
"totalMs": 9962.5,
"grade": "slow",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "uint8",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " x",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 6808.10000000149,
"inputTokens": 16,
"outputTokens": 8,
"tokensPerSecond": 1.175070871461678,
"loadMs": 3704.2999999970198,
"totalMs": 10512.39999999851,
"grade": "slow",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "uint8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"grade": "failed"
},
"failure": "Error: page.evaluate: Error: failed to call OrtRun(). ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc\n at worker.onmessage (eval at evaluate (:291:30), <anonymous>:19:54)"
},
{
"precision": "q8",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " France",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 6263.80000000447,
"inputTokens": 8,
"outputTokens": 8,
"tokensPerSecond": 1.2771799865886988,
"loadMs": 4335.70000000298,
"totalMs": 10599.50000000745,
"grade": "slow",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "q8",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " Celsius",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 6916.60000000149,
"inputTokens": 11,
"outputTokens": 8,
"tokensPerSecond": 1.156637654338588,
"loadMs": 4335.70000000298,
"totalMs": 11252.30000000447,
"grade": "slow",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "q8",
"device": "webgpu",
"status": "fail",
"quality": {
"expectedPrefix": " x",
"prefixPassed": true,
"repetitionPassed": true,
"streamParity": true,
"finalParity": true,
"passed": false
},
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"inferenceMs": 6882.10000000149,
"inputTokens": 16,
"outputTokens": 8,
"tokensPerSecond": 1.1624358843954996,
"loadMs": 4335.70000000298,
"totalMs": 11217.80000000447,
"grade": "slow",
"loadTimingScope": "Load from pinned local snapshot. Network transfer and runtime provisioning excluded."
}
},
{
"precision": "q8",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"engine": "transformers.js",
"version": "4.2.0",
"artifactRevision": "d78a42f79198603e614095753484a04c10c2b940",
"browser": "157.0.8103.39",
"grade": "failed"
},
"failure": "Error: page.evaluate: Error: failed to call OrtRun(). ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc\n at worker.onmessage (eval at evaluate (:291:30), <anonymous>:19:54)"
}
]