SmolLM3 3B
22 configurations testedQuality criteria
No explicit quality gate recorded. Review per-case evidence; full certification is pending.
| Precision | Device | Result | |
|---|---|---|---|
Q4F16 | GPU · WebGPUAccelerated | Passed | Moderate |
Q4F16 | GPU · WebGPUAccelerated | Passed | Moderate |
Q4F16 | GPU · WebGPUAccelerated | Passed | Moderate |
Q4F16 | GPU · WebGPUAccelerated | Passed | Moderate |
Q4F16 | GPU · WebGPUAccelerated | Passed | Moderate |
Q4F16 | GPU · WebGPUAccelerated | Passed | Moderate |
Q4F16 | GPU · WebGPUAccelerated | Passed | Moderate |
Q4 | GPU · WebGPUAccelerated | Passed | Slow |
Q4 | GPU · WebGPUAccelerated | Passed | Slow |
Q4 | GPU · WebGPUAccelerated | Passed | Slow |
Q4 | GPU · WebGPUAccelerated | Passed | Slow |
Q4 | GPU · WebGPUAccelerated | Passed | Slow |
Q4 | GPU · WebGPUAccelerated | Passed | Slow |
Q4 | GPU · WebGPUAccelerated | Passed | Slow |
Q4F16 | GPU · WebGPUAccelerated | Passed | Slow |
Q4 | GPU · WebGPUAccelerated | Passed | Very slow |
Q4F16 | GPU · WebGPU | Failed | — |
Q4F16 | GPU · WebGPU | Failed | — |
Q4 | GPU · WebGPU | Failed | — |
Q4 | GPU · WebGPU | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
Q4 | CPU · WASM | Failed | — |
Test setup & sample details
- Quality gate
- No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
- Tested
- 2026-11-04T09:42:40.703Z
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
- Runtime
- transformers.js 4.2.0
- Artifact revision
- 583cadea149c60c0a737786f61ace5775e165d99
- Fixture
{ "file": "model-samples/smollm3/smollm3-3b-capital.json", "size": 270, "sha1": "a9284d58037c583b301efa43bdb20daf825a8773", "sha256": "a4eec49e0b559051ab33c74ad15ab7f256bce4dfcc0c5403fdc6af8b16712527", "url": "https://assets.localai.tools/samples/v1/files/a4eec49e0b559051ab33c74ad15ab7f256bce4dfcc0c5403fdc6af8b16712527/smollm3-3b-capital.json" }- Rerun when
- Artifact or runtime revision; Fixture/quality gates; Generation/preprocessing implementation; Browser/backend behavior
Per-case quality & failure details
[
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 302.40000000596046,
"outputTokens": 2,
"inputTokens": 82,
"tokensPerSecond": 6.613756613626253,
"loadMs": 5855.200000017881,
"durationMs": 314.09999999403954,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 6169.300000011921,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "56",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 166.2999999821186,
"outputTokens": 2,
"inputTokens": 82,
"tokensPerSecond": 12.02645820935087,
"loadMs": 5855.200000017881,
"durationMs": 167.30000001192093,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 6022.500000029802,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "chat",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 165.19999998807907,
"outputTokens": 2,
"inputTokens": 87,
"tokensPerSecond": 12.106537531139958,
"loadMs": 5855.200000017881,
"durationMs": 166.2999999821186,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 6021.5,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "hola",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 164.09999999403954,
"outputTokens": 2,
"inputTokens": 88,
"tokensPerSecond": 12.187690433105693,
"loadMs": 5855.200000017881,
"durationMs": 165,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 6020.200000017881,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "fail",
"quality": {
"passed": false,
"exactNormalized": false,
"expected": "2026-11-04",
"scope": "diagnostic-strict-format"
},
"runtime": {
"inferenceMs": 379.5,
"outputTokens": 15,
"inputTokens": 91,
"tokensPerSecond": 39.52569169960474,
"loadMs": 5855.200000017881,
"durationMs": 380.30000001192093,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 6235.500000029802,
"grade": "fast",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 163.59999999403954,
"outputTokens": 2,
"inputTokens": 82,
"tokensPerSecond": 12.224938875751016,
"loadMs": 5855.200000017881,
"durationMs": 164.40000000596046,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 6019.600000023842,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "56",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 162.2000000178814,
"outputTokens": 2,
"inputTokens": 82,
"tokensPerSecond": 12.330456225521049,
"loadMs": 5855.200000017881,
"durationMs": 163.09999999403954,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 6018.300000011921,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "chat",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 162.09999999403954,
"outputTokens": 2,
"inputTokens": 87,
"tokensPerSecond": 12.338062924574587,
"loadMs": 5855.200000017881,
"durationMs": 162.90000000596046,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 6018.100000023842,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "hola",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 162.09999999403954,
"outputTokens": 2,
"inputTokens": 88,
"tokensPerSecond": 12.338062924574587,
"loadMs": 5855.200000017881,
"durationMs": 163,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 6018.200000017881,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "fail",
"quality": {
"passed": false,
"exactNormalized": false,
"expected": "2026-11-04",
"scope": "diagnostic-strict-format"
},
"runtime": {
"inferenceMs": 372.80000001192093,
"outputTokens": 15,
"inputTokens": 91,
"tokensPerSecond": 40.236051500859304,
"loadMs": 5855.200000017881,
"durationMs": 373.89999997615814,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 6229.0999999940395,
"grade": "fast",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 747.0999999940395,
"outputTokens": 2,
"inputTokens": 82,
"tokensPerSecond": 2.677017802189742,
"loadMs": 5337.100000023842,
"durationMs": 759.8999999761581,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 6097,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "56",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 239.30000001192093,
"outputTokens": 2,
"inputTokens": 82,
"tokensPerSecond": 8.35770998704709,
"loadMs": 5337.100000023842,
"durationMs": 240.30000001192093,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 5577.400000035763,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "chat",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 238.5,
"outputTokens": 2,
"inputTokens": 87,
"tokensPerSecond": 8.385744234800839,
"loadMs": 5337.100000023842,
"durationMs": 239.40000000596046,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 5576.500000029802,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "hola",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 238.90000000596046,
"outputTokens": 2,
"inputTokens": 88,
"tokensPerSecond": 8.371703641482213,
"loadMs": 5337.100000023842,
"durationMs": 240.19999998807907,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 5577.300000011921,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "fail",
"quality": {
"passed": false,
"exactNormalized": false,
"expected": "2026-11-04",
"scope": "diagnostic-strict-format"
},
"runtime": {
"inferenceMs": 492.80000001192093,
"outputTokens": 15,
"inputTokens": 91,
"tokensPerSecond": 30.43831168757538,
"loadMs": 5337.100000023842,
"durationMs": 493.5,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 5830.600000023842,
"grade": "fast",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "Paris",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 235.7000000178814,
"outputTokens": 2,
"inputTokens": 82,
"tokensPerSecond": 8.485362748613788,
"loadMs": 5337.100000023842,
"durationMs": 236.40000000596046,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 5573.500000029802,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "56",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 233.80000001192093,
"outputTokens": 2,
"inputTokens": 82,
"tokensPerSecond": 8.554319931129276,
"loadMs": 5337.100000023842,
"durationMs": 234.69999998807907,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 5571.800000011921,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "chat",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 238.7000000178814,
"outputTokens": 2,
"inputTokens": 87,
"tokensPerSecond": 8.378718055509749,
"loadMs": 5337.100000023842,
"durationMs": 239.7000000178814,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 5576.800000041723,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"exactNormalized": true,
"expected": "hola",
"scope": "general-chat"
},
"runtime": {
"inferenceMs": 238.60000002384186,
"outputTokens": 2,
"inputTokens": 88,
"tokensPerSecond": 8.382229672255457,
"loadMs": 5337.100000023842,
"durationMs": 239.40000000596046,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 5576.500000029802,
"grade": "acceptable",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "fail",
"quality": {
"passed": false,
"exactNormalized": false,
"expected": "2026-11-04",
"scope": "diagnostic-strict-format"
},
"runtime": {
"inferenceMs": 494.59999999403954,
"outputTokens": 15,
"inputTokens": 91,
"tokensPerSecond": 30.32753740432828,
"loadMs": 5337.100000023842,
"durationMs": 495.2999999821186,
"engine": "transformers.js",
"version": "4.2.0",
"browser": "157.0.8103.39",
"artifactRevision": "583cadea149c60c0a737786f61ace5775e165d99",
"totalMs": 5832.4000000059605,
"grade": "fast",
"loadTimingScope": "Local immutable snapshot load; native load includes one-token backend warmup. Excludes network/runtime provisioning."
}
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": null,
"grade": "failed"
},
"failure": "Error: page.evaluate: Error: Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc\n at w.onmessage (eval at evaluate (:291:30), <anonymous>:1:526)"
},
{
"precision": "q4",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": null,
"grade": "failed"
},
"failure": "Error: page.evaluate: Error: Can't create a session. ERROR_CODE: 6, ERROR_MESSAGE: std::bad_alloc\n at w.onmessage (eval at evaluate (:291:30), <anonymous>:1:526)"
}
]