Florence-2 Base FT — Image + Text
17 configurations testedQuality criteria
Non-empty, non-repetitive caption that identifies a person in the checked-in portrait; streamed text must equal non-streamed text.
| Precision | Device | Result | |
|---|---|---|---|
Q4 | GPU · WebGPUAccelerated | Passed | Moderate |
FP32 | GPU · WebGPUAccelerated | Passed | Moderate |
FP32 | GPU · WebGPUAccelerated | Passed | Moderate |
BNB4 | GPU · WebGPUAccelerated | Passed | Very slow |
UINT8 | GPU · WebGPUAccelerated | Passed | Very slow |
Q8 | GPU · WebGPUAccelerated | Passed | Very slow |
INT8 | GPU · WebGPUAccelerated | Passed | Very slow |
Q8 | CPU · WASM | Passed | Very slow |
UINT8 | CPU · WASM | Passed | Very slow |
BNB4 | CPU · WASM | Passed | Very slow |
FP32 | CPU · WASM | Passed | Very slow |
INT8 | CPU · WASM | Passed | Very slow |
Q4 | CPU · WASM | Passed | Very slow |
FP16 | CPU · WASM | Failed | — |
FP16 | GPU · WebGPU | Failed | — |
Q4F16 | CPU · WASM | Failed | — |
Q4F16 | GPU · WebGPU | Failed | — |
Test setup & sample details
- Quality gate
- Non-empty, non-repetitive caption that identifies a person in the checked-in portrait; streamed text must equal non-streamed text.
- Tested
- 2026-09-12T18:20:09.043Z
- Environment
- Isolated headless Chrome on macOS; Transformers.js 4.2.0
- Runtime
- Transformers.js 4.2.0
- Artifact revision
- e88a44eaf3791a35eae0c5a47b3dbcd36e67eb6f
- Fixture
{ "file": "modnet-portrait-pexels-5965592.jpg", "url": "https://f005.backblazeb2.com/file/hubters-webai-public/assets/benchmarks/modnet-portrait-pexels-5965592.jpg", "size": 78124, "sha1": "38212739a4a4f0be25d84863cf991d7fb6c6d818", "width": 1024, "height": 683, "license": "Pexels License", "licenseUrl": "https://www.pexels.com/license/" }- Rerun when
- artifact revision change; runtime version change; fixture or quality-gate change; browser/backend behavior change
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "wasm",
"status": "pass",
"quality": {
"identifiesPerson": true,
"finite": true,
"nonRepetitive": true,
"streamMatches": true,
"wordCount": 12,
"grade": "caption-pass"
},
"runtime": {
"loadMs": 108049.29999999702,
"inferenceMs": 11079.20000000298,
"totalMs": 131064.69999999553,
"grade": "very-slow",
"streamInferenceMs": 10583.89999999851,
"imagesPerSecond": 0.09025922449271888
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"identifiesPerson": true,
"finite": true,
"nonRepetitive": true,
"streamMatches": true,
"wordCount": 12,
"grade": "caption-pass"
},
"runtime": {
"loadMs": 108948.60000000149,
"inferenceMs": 617.6999999955297,
"totalMs": 111278,
"grade": "excellent",
"streamInferenceMs": 454.20000000298023,
"imagesPerSecond": 1.6189088554431554
}
},
{
"precision": "fp16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 32635.5,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/layers.5/self_attn_layer_norm/Constant_output_0for node: /layernorm_embedding/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "fp16",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 60980,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph.cc:1491 void onnxruntime::Graph::InitializeStateFromModelFileGraphProto() This is an invalid model. Subgraph output (logits) is an outer scope value being returned directly. Please update the model to add an Identity node between the outer scope value and the subgraph output.\n"
},
{
"precision": "int8",
"device": "wasm",
"status": "pass",
"quality": {
"identifiesPerson": true,
"finite": true,
"nonRepetitive": true,
"streamMatches": true,
"wordCount": 12,
"grade": "caption-pass"
},
"runtime": {
"loadMs": 33050.10000000149,
"inferenceMs": 11720.39999999851,
"totalMs": 57501.60000000149,
"grade": "very-slow",
"streamInferenceMs": 11499.20000000298,
"imagesPerSecond": 0.0853213200914753
}
},
{
"precision": "int8",
"device": "webgpu",
"status": "pass",
"quality": {
"identifiesPerson": true,
"finite": true,
"nonRepetitive": true,
"streamMatches": true,
"wordCount": 12,
"grade": "caption-pass"
},
"runtime": {
"loadMs": 32247.70000000298,
"inferenceMs": 11916.89999999851,
"totalMs": 57049.30000000447,
"grade": "very-slow",
"streamInferenceMs": 11538.80000000447,
"imagesPerSecond": 0.08391444083613397
}
},
{
"precision": "uint8",
"device": "wasm",
"status": "pass",
"quality": {
"identifiesPerson": true,
"finite": true,
"nonRepetitive": true,
"streamMatches": true,
"wordCount": 12,
"grade": "caption-pass"
},
"runtime": {
"loadMs": 31642.10000000149,
"inferenceMs": 10279,
"totalMs": 53303.39999999851,
"grade": "very-slow",
"streamInferenceMs": 10117.20000000298,
"imagesPerSecond": 0.09728572818367545
}
},
{
"precision": "uint8",
"device": "webgpu",
"status": "pass",
"quality": {
"identifiesPerson": true,
"finite": true,
"nonRepetitive": true,
"streamMatches": true,
"wordCount": 12,
"grade": "caption-pass"
},
"runtime": {
"loadMs": 31365.29999999702,
"inferenceMs": 10366.5,
"totalMs": 53327.5,
"grade": "very-slow",
"streamInferenceMs": 10168.30000000447,
"imagesPerSecond": 0.0964645733854242
}
},
{
"precision": "q8",
"device": "wasm",
"status": "pass",
"quality": {
"identifiesPerson": true,
"finite": true,
"nonRepetitive": true,
"streamMatches": true,
"wordCount": 12,
"grade": "caption-pass"
},
"runtime": {
"loadMs": 30896.20000000298,
"inferenceMs": 10247.89999999851,
"totalMs": 52584.10000000149,
"grade": "very-slow",
"streamInferenceMs": 10127.60000000149,
"imagesPerSecond": 0.0975809678080529
}
},
{
"precision": "q8",
"device": "webgpu",
"status": "pass",
"quality": {
"identifiesPerson": true,
"finite": true,
"nonRepetitive": true,
"streamMatches": true,
"wordCount": 12,
"grade": "caption-pass"
},
"runtime": {
"loadMs": 32917.60000000149,
"inferenceMs": 10382.60000000149,
"totalMs": 54766.90000000596,
"grade": "very-slow",
"streamInferenceMs": 10149.89999999851,
"imagesPerSecond": 0.09631498853850254
}
},
{
"precision": "q4",
"device": "wasm",
"status": "pass",
"quality": {
"identifiesPerson": true,
"finite": true,
"nonRepetitive": true,
"streamMatches": true,
"wordCount": 12,
"grade": "caption-pass"
},
"runtime": {
"loadMs": 42147.69999999553,
"inferenceMs": 11727.30000000447,
"totalMs": 66847.5,
"grade": "very-slow",
"streamInferenceMs": 11659.39999999851,
"imagesPerSecond": 0.08527111952449573
}
},
{
"precision": "q4",
"device": "webgpu",
"status": "pass",
"quality": {
"identifiesPerson": true,
"finite": true,
"nonRepetitive": true,
"streamMatches": true,
"wordCount": 12,
"grade": "caption-pass"
},
"runtime": {
"loadMs": 41324.70000000298,
"inferenceMs": 543.9000000059605,
"totalMs": 43678.60000000149,
"grade": "excellent",
"streamInferenceMs": 440,
"imagesPerSecond": 1.8385732671245472
}
},
{
"precision": "q4f16",
"device": "wasm",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 19000.10000000149,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph_utils.cc:30 int onnxruntime::graph_utils::GetIndexFromName(const Node &, const std::string &, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_/layers.5/self_attn_layer_norm/Constant_output_0for node: /layernorm_embedding/Mul/SimplifiedLayerNormFusion/\n"
},
{
"precision": "q4f16",
"device": "webgpu",
"status": "fail",
"quality": "Not embedded",
"runtime": {
"loadMs": null,
"inferenceMs": null,
"totalMs": 47430.10000000149,
"grade": "failed"
},
"failure": "Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: /mnt/vss/_work/1/s/onnxruntime/core/graph/graph.cc:1491 void onnxruntime::Graph::InitializeStateFromModelFileGraphProto() This is an invalid model. Subgraph output (logits) is an outer scope value being returned directly. Please update the model to add an Identity node between the outer scope value and the subgraph output.\n"
},
{
"precision": "bnb4",
"device": "wasm",
"status": "pass",
"quality": {
"identifiesPerson": true,
"finite": true,
"nonRepetitive": true,
"streamMatches": true,
"wordCount": 12,
"grade": "caption-pass"
},
"runtime": {
"loadMs": 40501.5,
"inferenceMs": 10628.29999999702,
"totalMs": 63013.19999999553,
"grade": "very-slow",
"streamInferenceMs": 10512.30000000447,
"imagesPerSecond": 0.09408842430118461
}
},
{
"precision": "bnb4",
"device": "webgpu",
"status": "pass",
"quality": {
"identifiesPerson": true,
"finite": true,
"nonRepetitive": true,
"streamMatches": true,
"wordCount": 12,
"grade": "caption-pass"
},
"runtime": {
"loadMs": 40026.30000000447,
"inferenceMs": 10309.29999999702,
"totalMs": 61939.60000000149,
"grade": "very-slow",
"streamInferenceMs": 10240.60000000149,
"imagesPerSecond": 0.09699979630045581
}
},
{
"precision": "fp32",
"device": "webgpu",
"status": "pass",
"quality": {
"passed": true,
"method": "Normalized exact authored printed text"
},
"runtime": {
"loadMs": 24522.519999980927,
"inferenceMs": 554.3650000095367,
"totalMs": 25079.469999969006,
"pagesPerSecond": 1.8038656841301255,
"grade": "fast"
}
}
]