A model is still running

A model is still running or downloading. Leaving will stop the current operation. Stay on this page to let it finish.

Skip to content
LocalAI.toolsBeta

Join the waitlist

Get an email when the new apps and model runs launch.

No account needed.

Qwen3.5 Desktop

Run local text questions, still-image questions using Qwen3.5 4B Desktop. Answers can hallucinate or miss visual details.

Desktop inferencePrivate · Runs on your computer
About this model

Run local text questions, still-image questions using Qwen3.5 4B Desktop. Answers can hallucinate or miss visual details.

License
Apache-2.0
Model released
Mar 2, 2026
Source repository created
Feb 27, 2026
Model weights · FP16
9.3 GB

Best for

  • Local text chat
  • Grounded text extraction and summarization
  • Instruction following with human review

Keep in mind

  • Can hallucinate, make reasoning errors or emit unsafe/biased text. No safety classifier or tool execution.
  • Adapter limits input plus output budget to8192 tokens and output to2048; this is a resource policy, not the original checkpoint context limit.
  • CPU FP32 requires substantial memory and can be slow. Only Apple M3 Max/macOS27.2 has measured native evidence. Other Mac hardware/OS and Windows remain unverified.
  • The app downloads its own pinned CPython and isolated packages on request; installed Python, user packages and Python/pip configuration are not used. Automatic model-package provisioning is verified on Apple Silicon Mac; other platform stacks remain unverified.
  • Video and sampled video-frame inputs are not supported. Use a still image instead.

Processed on your deviceYour input is processed locally in the desktop app. Model and runtime files download from external hosts.

Target platforms · verification required

  • macos · aarch64, x86_64
  • windows · x86_64

Recommended configuration

FP16 · mps (verified platform only)

Sources & technical details
Original model
https://huggingface.co/Qwen/Qwen3.5-4B
Release verification
Verified
Release evidence
Official Qwen repository News explicitly lists Qwen3.5-4B among the small checkpoints publicly released2026-03-02; repository creation is recorded separately.
Source updated
Mar 2, 2026
Native model files
https://huggingface.co/Qwen/Qwen3.5-4B
Files updated
Mar 2, 2026
Artifact revision
851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a

Desktop app

Sample result

Qwen3.5 4B Desktop — Text, Image Desktop

Run Qwen3.5 Desktop on your computer

Download the desktop app to load this model, provide input and view results locally.

Public desktop downloads are coming soon.

Device requirements

macOS Apple SiliconCPU / FP32Tested32.0 GiBRAM estimate
macOS Apple SiliconMPS / FP16Tested24.0 GiBRAM estimate
Windows Intel / AMDCPU / FP32Unverified32.0 GiBRAM estimate
macOS Intel / AMDCPU / FP32Unverified32.0 GiBRAM estimate
If your device falls short

MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.

GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.

PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.

Benchmark evidence

Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.

Qwen3.5 4B Desktop — Text, Image Desktop

47 configurations tested
Passed47
Failed0
TestedNov 3, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
FP16mpsAcceleratedPassedFastmacos · aarch64889 ms / 2945 ms
FP16mpsAcceleratedPassedFastmacos · aarch64951 ms / 4310 ms
FP16mpsAcceleratedPassedFastmacos · aarch64966 ms / 3049 ms
FP16mpsAcceleratedPassedFastmacos · aarch64413 ms / 4310 ms
FP16mpsAcceleratedPassedFastmacos · aarch64414 ms / 2945 ms
FP16mpsAcceleratedPassedFastmacos · aarch64431 ms / 3049 ms
FP16mpsAcceleratedPassedFastmacos · aarch64527 ms / 2945 ms
FP16mpsAcceleratedPassedFastmacos · aarch64558 ms / 3049 ms
FP16mpsAcceleratedPassedFastmacos · aarch64490 ms / 2945 ms
FP16mpsAcceleratedPassedFastmacos · aarch64504 ms / 3327 ms
FP16mpsAcceleratedPassedFastmacos · aarch641040 ms / 3327 ms
FP16mpsAcceleratedPassedFastmacos · aarch641180 ms / 3327 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64613 ms / 4310 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64529 ms / 3049 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64438 ms / 2945 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64449 ms / 3049 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64555 ms / 4310 ms
FP16mpsAcceleratedPassedModeratemacos · aarch64739 ms / 4310 ms
FP16mpsAcceleratedPassedSlowmacos · aarch644487 ms / 3327 ms
FP16mpsAcceleratedPassedVery slowmacos · aarch644489 ms / 3327 ms
FP16mpsAcceleratedPassedVery slowmacos · aarch645042 ms / 3327 ms
FP32cpuPassedSlowmacos · aarch647214 ms / 7221 ms
FP32cpuPassedSlowmacos · aarch642529 ms / 7221 ms
FP32cpuPassedSlowmacos · aarch642051 ms / 7221 ms
FP32cpuPassedSlowmacos · aarch642568 ms / 7221 ms
FP32cpuPassedSlowmacos · aarch643179 ms / 7221 ms
FP32cpuPassedSlowmacos · aarch6411867 ms / 3092 ms
FP32cpuPassedSlowmacos · aarch6440629 ms / 2693 ms
FP32cpuPassedSlowmacos · aarch6411094 ms / 3092 ms
FP32cpuPassedSlowmacos · aarch6417068 ms / 5198 ms
FP32cpuPassedSlowmacos · aarch6417375 ms / 2693 ms
FP32cpuPassedSlowmacos · aarch6414845 ms / 5198 ms
FP32cpuPassedSlowmacos · aarch6412034 ms / 2693 ms
FP32cpuPassedSlowmacos · aarch6412366 ms / 5198 ms
FP32cpuPassedSlowmacos · aarch6416533 ms / 2693 ms
FP32cpuPassedSlowmacos · aarch6417779 ms / 2693 ms
FP32cpuPassedSlowmacos · aarch6454743 ms / 5198 ms
FP32cpuPassedSlowmacos · aarch6422952 ms / 3092 ms
FP32cpuPassedSlowmacos · aarch6420502 ms / 5198 ms
FP32cpuPassedSlowmacos · aarch6463265 ms / 3092 ms
FP32cpuPassedSlowmacos · aarch6422823 ms / 3092 ms
FP32cpuPassedVery slowmacos · aarch6451851 ms / 6714 ms
FP32cpuPassedVery slowmacos · aarch6433455 ms / 6714 ms
FP32cpuPassedVery slowmacos · aarch6423105 ms / 6714 ms
FP32cpuPassedVery slowmacos · aarch6434926 ms / 6714 ms
FP32cpuPassedVery slowmacos · aarch6423994 ms / 6714 ms
FP32cpuPassedVery slowmacos · aarch6424220 ms / 6714 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-03T14:46:28.967787+00:00
Environment
{ "platform": "macOS-27.2-arm64-arm-64bit-Mach-O", "hardware": "Apple M3 Max" }
Runtime
transformers 5.22.1
Artifact revision
851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Fixture
{
  "file": "localai-desktop/crates/desktopai/src/models/qwen35/fixtures.json",
  "size": 944,
  "sha1": "ef3eb28d89d7289b84660b6e2db5d6768bcf6d22",
  "sha256": "2e2f8f9dfe0f61f0e7715a7bd2ba15f7e5c7f5ab6cb7cdb3e594b744a0d96f0b",
  "url": "https://assets.localai.tools/samples/v1/files/2e2f8f9dfe0f61f0e7715a7bd2ba15f7e5c7f5ab6cb7cdb3e594b744a0d96f0b/fixtures.json"
}
Rerun when
Artifact revision; Transformers/Torch/runtime lock; Adapter inference or input/output semantics; Fixture checksum or quality gate; Device/backend/OS behavior
Per-case quality & failure details
[
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4310.408624995034,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 414.86554199946113,
      "inferenceMs": 413.3240829978604,
      "tokensPerSecond": 12.09704492352526,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9522413568,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 4726.7882080050185,
      "requestMs": 416.37958300998434,
      "grade": "fast",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4310.408624995034,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 557.4632080097217,
      "inferenceMs": 555.3242920141201,
      "tokensPerSecond": 7.202998423663939,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9522757632,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 4869.358084019041,
      "requestMs": 558.9494590240065,
      "grade": "moderate",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4310.408624995034,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 740.9979999938514,
      "inferenceMs": 738.9543329773005,
      "tokensPerSecond": 6.766317994042525,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9524363264,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 5052.681416011183,
      "requestMs": 742.2727910161484,
      "grade": "moderate",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4310.408624995034,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 614.4294170080684,
      "inferenceMs": 612.762709002709,
      "tokensPerSecond": 9.791718575311466,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9525067776,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 4926.251208002213,
      "requestMs": 615.8425830071792,
      "grade": "moderate",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4310.408624995034,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 952.905499987537,
      "inferenceMs": 951.1543750122655,
      "tokensPerSecond": 15.770310681488027,
      "inputTokens": 40,
      "outputTokens": 15,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9525641216,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 5264.713249984197,
      "requestMs": 954.3046249891631,
      "grade": "fast",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2944.73950000247,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 415.4665830137674,
      "inferenceMs": 413.8022919942159,
      "tokensPerSecond": 12.083065020988066,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9522413568,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3361.750874988502,
      "requestMs": 417.011374986032,
      "grade": "fast",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2944.73950000247,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 440.1672499952838,
      "inferenceMs": 438.3782090153545,
      "tokensPerSecond": 9.124541133065073,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9522757632,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3386.3159170141444,
      "requestMs": 441.5764170116745,
      "grade": "moderate",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2944.73950000247,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 492.0912079978734,
      "inferenceMs": 490.3657920076512,
      "tokensPerSecond": 10.196469822107789,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9524363264,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3438.2337080023717,
      "requestMs": 493.49420799990185,
      "grade": "fast",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2944.73950000247,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 528.3934580220375,
      "inferenceMs": 526.6213750001043,
      "tokensPerSecond": 11.393384858331684,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9525067776,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3474.47524999734,
      "requestMs": 529.7357499948703,
      "grade": "fast",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2944.73950000247,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 890.5567080073524,
      "inferenceMs": 888.9555830101017,
      "tokensPerSecond": 16.873733948785546,
      "inputTokens": 40,
      "outputTokens": 15,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9525641216,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3836.76833298523,
      "requestMs": 892.0288329827599,
      "grade": "fast",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3049.4302499864716,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 432.6420839934144,
      "inferenceMs": 431.46120800520293,
      "tokensPerSecond": 11.58852732813863,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9522413568,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3483.514124964131,
      "requestMs": 434.08387497765943,
      "grade": "fast",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3049.4302499864716,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 451.076916011516,
      "inferenceMs": 449.2849580128677,
      "tokensPerSecond": 8.903035654012344,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9522757632,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3501.7676669813227,
      "requestMs": 452.3374169948511,
      "grade": "moderate",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3049.4302499864716,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 531.1215000110678,
      "inferenceMs": 529.4120839971583,
      "tokensPerSecond": 9.44443874844919,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9524363264,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3581.7654589773156,
      "requestMs": 532.335208990844,
      "grade": "moderate",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3049.4302499864716,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 559.5353340031579,
      "inferenceMs": 557.9937919974327,
      "tokensPerSecond": 10.752807801896845,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9525067776,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3610.198958980618,
      "requestMs": 560.7687089941464,
      "grade": "fast",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3049.4302499864716,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 723
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 967.7480840182398,
      "inferenceMs": 965.7638749922626,
      "tokensPerSecond": 15.531746825920752,
      "inputTokens": 40,
      "outputTokens": 15,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 9525641216,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 4018.417374987621,
      "requestMs": 968.9871250011493,
      "grade": "fast",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2692.5151250034105,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 16533.240624994505,
      "inferenceMs": 16532.844458008185,
      "tokensPerSecond": 0.3024282973628351,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 19226.991541014286,
      "requestMs": 16534.476416010875,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2692.5151250034105,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 12034.509832999902,
      "inferenceMs": 12034.152250009356,
      "tokensPerSecond": 0.3323873520045328,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 14728.24170900276,
      "requestMs": 12035.726583999349,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2692.5151250034105,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 17779.490042012185,
      "inferenceMs": 17778.98712499882,
      "tokensPerSecond": 0.28123086905043987,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 20473.26645901194,
      "requestMs": 17780.751334008528,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2692.5151250034105,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 17375.299708975945,
      "inferenceMs": 17374.757292011054,
      "tokensPerSecond": 0.3453285648346185,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 20069.065457995748,
      "requestMs": 17376.550332992338,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2692.5151250034105,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 40629.073374992004,
      "inferenceMs": 40628.72970898752,
      "tokensPerSecond": 0.36919687392249023,
      "inputTokens": 40,
      "outputTokens": 15,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 43322.79112498509,
      "requestMs": 40630.27599998168,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3092.4598749843426,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 11867.062500008615,
      "inferenceMs": 11866.75399998785,
      "tokensPerSecond": 0.4213452136957688,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 14960.780082998099,
      "requestMs": 11868.320208013756,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3092.4598749843426,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 11094.28725001635,
      "inferenceMs": 11093.930166010978,
      "tokensPerSecond": 0.36055752471337865,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 14188.063290988794,
      "requestMs": 11095.603416004451,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3092.4598749843426,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 22823.575167014496,
      "inferenceMs": 22823.029707971727,
      "tokensPerSecond": 0.21907696147166553,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 25917.844915995374,
      "requestMs": 22825.38504101103,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3092.4598749843426,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 22954.013583017513,
      "inferenceMs": 22952.175249985885,
      "tokensPerSecond": 0.2614131312021805,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 26048.110750009073,
      "requestMs": 22955.65087502473,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3092.4598749843426,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 63265.78374998644,
      "inferenceMs": 63264.64683399536,
      "tokensPerSecond": 0.23709924500739846,
      "inputTokens": 40,
      "outputTokens": 15,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 66359.52704196097,
      "requestMs": 63267.06716697663,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5198.398041015025,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 20502.498666988686,
      "inferenceMs": 20502.160375006497,
      "tokensPerSecond": 0.24387673828243653,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 25702.14154102723,
      "requestMs": 20503.743500012206,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5198.398041015025,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 12366.548625024734,
      "inferenceMs": 12366.199374984717,
      "tokensPerSecond": 0.32346235724546885,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 17566.163082025014,
      "requestMs": 12367.765041009989,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5198.398041015025,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 14845.581332978327,
      "inferenceMs": 14845.202457974665,
      "tokensPerSecond": 0.3368091485551994,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 20045.197291008662,
      "requestMs": 14846.799249993637,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5198.398041015025,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 17068.991250009276,
      "inferenceMs": 17068.413042026805,
      "tokensPerSecond": 0.3515265294568665,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 22268.679000029806,
      "requestMs": 17070.28095901478,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5198.398041015025,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 54743.846916011535,
      "inferenceMs": 54743.488874984905,
      "tokensPerSecond": 0.27400518871303187,
      "inputTokens": 40,
      "outputTokens": 15,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 59943.58316602302,
      "requestMs": 54745.18512500799,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7220.5192080000415,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 2529.6247500227764,
      "inferenceMs": 2529.2326670023613,
      "tokensPerSecond": 1.9768841614425234,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 9751.414750004187,
      "requestMs": 2530.8955420041457,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7220.5192080000415,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 2051.846290996764,
      "inferenceMs": 2051.380416000029,
      "tokensPerSecond": 1.9499064965237258,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 9273.638749000384,
      "requestMs": 2053.119541000342,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7220.5192080000415,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 2568.103750003502,
      "inferenceMs": 2567.6854589837603,
      "tokensPerSecond": 1.947279010560313,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 9789.937000023201,
      "requestMs": 2569.4177920231596,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7220.5192080000415,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 3179.395041981479,
      "inferenceMs": 3178.9161670021713,
      "tokensPerSecond": 1.887435743754831,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 10401.232458010782,
      "requestMs": 3180.71325001074,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 7220.5192080000415,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 723
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 7214.029541995842,
      "inferenceMs": 7213.586333004059,
      "tokensPerSecond": 2.079409506942621,
      "inputTokens": 40,
      "outputTokens": 15,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 14435.75133298873,
      "requestMs": 7215.232124988688,
      "grade": "slow",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 24039.163874986116,
      "inferenceMs": 23993.61454101745,
      "tokensPerSecond": 0.1667110219330205,
      "inputTokens": 830,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 723
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "loadMs": 6713.745333021507,
      "totalMs": 30760.3046669974,
      "wallMs": 24045.84541602526,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 51858.392958005425,
      "inferenceMs": 51851.46454203641,
      "tokensPerSecond": 0.1928585834232882,
      "inputTokens": 795,
      "outputTokens": 10,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 723
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "loadMs": 6713.745333021507,
      "totalMs": 82620.61337503837,
      "wallMs": 51860.1997080259,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 24226.0877090157,
      "inferenceMs": 24219.806374982,
      "tokensPerSecond": 0.16515408662109807,
      "inputTokens": 828,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 723
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "loadMs": 6713.745333021507,
      "totalMs": 106848.3936249977,
      "wallMs": 24227.617750002537,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 34932.200959010515,
      "inferenceMs": 34925.99641595734,
      "tokensPerSecond": 0.17179180598147972,
      "inputTokens": 154,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 723
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "loadMs": 6713.745333021507,
      "totalMs": 141782.3175000376,
      "wallMs": 34933.83512500441,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 33461.776458017994,
      "inferenceMs": 33455.48454200616,
      "tokensPerSecond": 0.17934279183631305,
      "inputTokens": 154,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 723
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "loadMs": 6713.745333021507,
      "totalMs": 175245.62566704117,
      "wallMs": 33463.22249999503,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 23105.48004100565,
      "inferenceMs": 23104.981457989197,
      "tokensPerSecond": 0.17312283964707045,
      "inputTokens": 25,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 723
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "loadMs": 6713.745333021507,
      "totalMs": 198352.38491703058,
      "wallMs": 23106.679999968037,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 5107.982166984584,
      "inferenceMs": 5042.1748330118135,
      "tokensPerSecond": 0.7933084695539411,
      "inputTokens": 830,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float16": 723
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "driverAllocatedMemoryBytes": 12799688704,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "loadMs": 3327.418458997272,
      "totalMs": 8444.734667020384,
      "wallMs": 5114.763375022449,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 4499.359332956374,
      "inferenceMs": 4486.562667007092,
      "tokensPerSecond": 2.228877816315185,
      "inputTokens": 795,
      "outputTokens": 10,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float16": 723
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "driverAllocatedMemoryBytes": 14129397760,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "loadMs": 3327.418458997272,
      "totalMs": 12946.045083983336,
      "wallMs": 4501.26845802879,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 4503.86937498115,
      "inferenceMs": 4489.467957988381,
      "tokensPerSecond": 0.8909741727597272,
      "inputTokens": 828,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float16": 723
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "driverAllocatedMemoryBytes": 15569272832,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "loadMs": 3327.418458997272,
      "totalMs": 17451.63041696651,
      "wallMs": 4505.496167053934,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 1191.7708750115708,
      "inferenceMs": 1179.6049160184339,
      "tokensPerSecond": 5.0864487918989285,
      "inputTokens": 154,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float16": 723
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "driverAllocatedMemoryBytes": 15598862336,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "loadMs": 3327.418458997272,
      "totalMs": 18645.038958988152,
      "wallMs": 1193.3839159901254,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "good"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 1048.1105000362732,
      "inferenceMs": 1040.370874979999,
      "tokensPerSecond": 5.767174134046524,
      "inputTokens": 154,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float16": 723
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "driverAllocatedMemoryBytes": 14931968000,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "loadMs": 3327.418458997272,
      "totalMs": 19694.801083998755,
      "wallMs": 1049.7360409935936,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "good"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 506.46420900011435,
      "inferenceMs": 504.0246249991469,
      "tokensPerSecond": 7.936120184617508,
      "inputTokens": 25,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float16": 723
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "driverAllocatedMemoryBytes": 14940749824,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "loadMs": 3327.418458997272,
      "totalMs": 20202.642416988965,
      "wallMs": 507.81404197914526,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "good"
    },
    "failure": null
  }
]

Qwen3.5 2B Desktop — Text, Image Desktop

47 configurations tested
Passed47
Failed0
TestedNov 3, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
FP16mpsAcceleratedPassedFastmacos · aarch64219 ms / 2747 ms
FP16mpsAcceleratedPassedFastmacos · aarch64220 ms / 2845 ms
FP16mpsAcceleratedPassedFastmacos · aarch64224 ms / 3323 ms
FP16mpsAcceleratedPassedFastmacos · aarch64574 ms / 2747 ms
FP16mpsAcceleratedPassedFastmacos · aarch64620 ms / 3323 ms
FP16mpsAcceleratedPassedFastmacos · aarch64358 ms / 3323 ms
FP16mpsAcceleratedPassedFastmacos · aarch64365 ms / 2845 ms
FP16mpsAcceleratedPassedFastmacos · aarch64321 ms / 2845 ms
FP16mpsAcceleratedPassedFastmacos · aarch64780 ms / 2845 ms
FP16mpsAcceleratedPassedFastmacos · aarch64391 ms / 2747 ms
FP16mpsAcceleratedPassedFastmacos · aarch64335 ms / 3323 ms
FP16mpsAcceleratedPassedFastmacos · aarch64335 ms / 2747 ms
FP16mpsAcceleratedPassedFastmacos · aarch64279 ms / 2747 ms
FP16mpsAcceleratedPassedFastmacos · aarch64292 ms / 3323 ms
FP16mpsAcceleratedPassedFastmacos · aarch64297 ms / 2845 ms
FP16mpsAcceleratedPassedFastmacos · aarch64334 ms / 5352 ms
FP16mpsAcceleratedPassedFastmacos · aarch64503 ms / 5352 ms
FP16mpsAcceleratedPassedFastmacos · aarch64819 ms / 5352 ms
FP16mpsAcceleratedPassedSlowmacos · aarch643316 ms / 5352 ms
FP16mpsAcceleratedPassedSlowmacos · aarch643322 ms / 5352 ms
FP16mpsAcceleratedPassedSlowmacos · aarch643878 ms / 5352 ms
FP32cpuPassedModeratemacos · aarch643381 ms / 2230 ms
FP32cpuPassedModeratemacos · aarch641178 ms / 2230 ms
FP32cpuPassedModeratemacos · aarch64947 ms / 2230 ms
FP32cpuPassedModeratemacos · aarch641431 ms / 2230 ms
FP32cpuPassedModeratemacos · aarch641217 ms / 2230 ms
FP32cpuPassedSlowmacos · aarch644890 ms / 2430 ms
FP32cpuPassedSlowmacos · aarch646158 ms / 2430 ms
FP32cpuPassedSlowmacos · aarch6422845 ms / 2223 ms
FP32cpuPassedSlowmacos · aarch646134 ms / 2223 ms
FP32cpuPassedSlowmacos · aarch6423081 ms / 2683 ms
FP32cpuPassedSlowmacos · aarch647696 ms / 2223 ms
FP32cpuPassedSlowmacos · aarch6423206 ms / 2430 ms
FP32cpuPassedSlowmacos · aarch647823 ms / 2430 ms
FP32cpuPassedSlowmacos · aarch649562 ms / 2430 ms
FP32cpuPassedSlowmacos · aarch6411576 ms / 2223 ms
FP32cpuPassedSlowmacos · aarch649912 ms / 2223 ms
FP32cpuPassedSlowmacos · aarch6411090 ms / 2683 ms
FP32cpuPassedSlowmacos · aarch6413635 ms / 2683 ms
FP32cpuPassedSlowmacos · aarch6413325 ms / 2683 ms
FP32cpuPassedSlowmacos · aarch6414388 ms / 2683 ms
FP32cpuPassedVery slowmacos · aarch6423946 ms / 4817 ms
FP32cpuPassedVery slowmacos · aarch6415604 ms / 4817 ms
FP32cpuPassedVery slowmacos · aarch6410831 ms / 4817 ms
FP32cpuPassedVery slowmacos · aarch6417335 ms / 4817 ms
FP32cpuPassedVery slowmacos · aarch6411917 ms / 4817 ms
FP32cpuPassedVery slowmacos · aarch6412747 ms / 4817 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-03T15:08:12.825374+00:00
Environment
{ "platform": "macOS-27.2-arm64-arm-64bit-Mach-O", "hardware": "Apple M3 Max" }
Runtime
transformers 5.22.1
Artifact revision
15852e8c16360a2fea060d615a32b45270f8a8fc
Fixture
{
  "file": "localai-desktop/crates/desktopai/src/models/qwen35/variants/2b/qwen35-2b-chat-fixtures.json",
  "size": 997,
  "sha1": "32ddc9a166a9b21ff8d131624c9399f364b49d48",
  "sha256": "91faa64e06f53ff062dde09f7b5c092dec0c9ff459669c8fc8c0b29b4aa83805",
  "url": "https://assets.localai.tools/samples/v1/files/91faa64e06f53ff062dde09f7b5c092dec0c9ff459669c8fc8c0b29b4aa83805/qwen35-2b-chat-fixtures.json"
}
Rerun when
Artifact revision; Transformers/Torch/runtime lock; Adapter inference or input/output semantics; Fixture checksum or quality gate; Device/backend/OS behavior
Per-case quality & failure details
[
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3323.292291985126,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 225.66925000865012,
      "inferenceMs": 224.43504197872244,
      "tokensPerSecond": 22.27816100336963,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5011357696,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3550.083583977539,
      "requestMs": 226.79129199241288,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3323.292291985126,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 293.8959579914808,
      "inferenceMs": 292.0674580091145,
      "tokensPerSecond": 13.695466202452355,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5011636224,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3618.3377919951454,
      "requestMs": 295.04550001001917,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3323.292291985126,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 336.7240829975344,
      "inferenceMs": 334.9137080076616,
      "tokensPerSecond": 14.9292187224705,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5013110784,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3661.212374980096,
      "requestMs": 337.92008299496956,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3323.292291985126,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 359.5950000162702,
      "inferenceMs": 357.8500410076231,
      "tokensPerSecond": 16.76679981118735,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5013635072,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3683.980708970921,
      "requestMs": 360.6884169857949,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3323.292291985126,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 621.7434579739347,
      "inferenceMs": 619.9480420036707,
      "tokensPerSecond": 19.356460843421694,
      "inputTokens": 52,
      "outputTokens": 12,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5014192128,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3946.1172509763855,
      "requestMs": 622.8249589912593,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2747.21191599383,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 220.58758299681358,
      "inferenceMs": 219.44704098859802,
      "tokensPerSecond": 22.784540531853374,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5011357696,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 2968.845750001492,
      "requestMs": 221.6338340076618,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2747.21191599383,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 281.4334169961512,
      "inferenceMs": 279.31408301810734,
      "tokensPerSecond": 14.320795989870259,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5011636224,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3029.7455829859246,
      "requestMs": 282.53366699209437,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2747.21191599383,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 336.77233298658393,
      "inferenceMs": 335.05329201580025,
      "tokensPerSecond": 14.922999174006662,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5013110784,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3085.22854099283,
      "requestMs": 338.01662499899976,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2747.21191599383,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 393.2164160069078,
      "inferenceMs": 391.37712502270006,
      "tokensPerSecond": 15.330482075701275,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5013635072,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3141.5627500100527,
      "requestMs": 394.3508340162225,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2747.21191599383,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 575.1745000015944,
      "inferenceMs": 573.905916971853,
      "tokensPerSecond": 20.909350548808742,
      "inputTokens": 52,
      "outputTokens": 12,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5014192128,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3323.5294990008697,
      "requestMs": 576.3175830070395,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2845.4672080115415,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 221.7684590141289,
      "inferenceMs": 220.10637499624863,
      "tokensPerSecond": 22.716288885704547,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5011357696,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3068.401291006012,
      "requestMs": 222.93408299447037,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2845.4672080115415,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 298.2424590154551,
      "inferenceMs": 296.60899998270907,
      "tokensPerSecond": 13.485767458954992,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5011636224,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3144.784748990787,
      "requestMs": 299.3175409792457,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2845.4672080115415,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 322.52808299381286,
      "inferenceMs": 320.9777500014752,
      "tokensPerSecond": 15.577403729626182,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5013110784,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3169.1550409886986,
      "requestMs": 323.6878329771571,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2845.4672080115415,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 367.1350420045201,
      "inferenceMs": 365.40304200025275,
      "tokensPerSecond": 16.420224547544542,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5013635072,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3213.7402910157107,
      "requestMs": 368.27308300416917,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2845.4672080115415,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float16": 617
      },
      "parameterDtype": "torch.float16",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 781.4886249834672,
      "inferenceMs": 779.6800000069197,
      "tokensPerSecond": 15.390929612011979,
      "inputTokens": 52,
      "outputTokens": 12,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 5014192128,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "precision": "fp16",
      "cpuThreads": 4,
      "totalMs": 3628.161666012602,
      "requestMs": 782.6944580010604,
      "grade": "fast",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2683.05533300736,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 11090.146917005768,
      "inferenceMs": 11089.807915996062,
      "tokensPerSecond": 0.45086443677603694,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 13774.302750010975,
      "requestMs": 11091.247417003615,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2683.05533300736,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 14388.207624986535,
      "inferenceMs": 14387.805832986487,
      "tokensPerSecond": 0.27801320412799296,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 17072.45350003359,
      "requestMs": 14389.39816702623,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2683.05533300736,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 13325.040875002742,
      "inferenceMs": 13324.686709005618,
      "tokensPerSecond": 0.3752433441171042,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 16009.35641603428,
      "requestMs": 13326.30108302692,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2683.05533300736,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 13635.83916600328,
      "inferenceMs": 13635.37383300718,
      "tokensPerSecond": 0.44003193997334983,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 16320.005375018809,
      "requestMs": 13636.95004201145,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2683.05533300736,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 23081.71179101919,
      "inferenceMs": 23081.32212498458,
      "tokensPerSecond": 0.6498761170948313,
      "inputTokens": 52,
      "outputTokens": 15,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 25765.78620800865,
      "requestMs": 23082.73087500129,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2222.9325420048553,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 7696.824666985776,
      "inferenceMs": 7696.384582988685,
      "tokensPerSecond": 0.6496556852228379,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 9920.840042002965,
      "requestMs": 7697.907499998109,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2222.9325420048553,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 6134.183541988023,
      "inferenceMs": 6133.81408402347,
      "tokensPerSecond": 0.6521227975296251,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 8358.188042009715,
      "requestMs": 6135.25550000486,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2222.9325420048553,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 9912.524874991504,
      "inferenceMs": 9912.147750001168,
      "tokensPerSecond": 0.5044315446164945,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 12136.538584018126,
      "requestMs": 9913.60604201327,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2222.9325420048553,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 11576.50095797726,
      "inferenceMs": 11576.081584003987,
      "tokensPerSecond": 0.5183100997051451,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 13800.527541985502,
      "requestMs": 11577.594999980647,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2222.9325420048553,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 22845.499291986926,
      "inferenceMs": 22845.097125013126,
      "tokensPerSecond": 0.6565960266186166,
      "inputTokens": 52,
      "outputTokens": 15,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 25069.577916990966,
      "requestMs": 22846.64537498611,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2430.0800419878215,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 6158.165667002322,
      "inferenceMs": 6157.832541997777,
      "tokensPerSecond": 0.8119740129175154,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 8589.30562500609,
      "requestMs": 6159.225583018269,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2430.0800419878215,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 4890.482459013583,
      "inferenceMs": 4890.061124984641,
      "tokensPerSecond": 0.8179856852019541,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7321.600249968469,
      "requestMs": 4891.520207980648,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2430.0800419878215,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 7823.8121250178665,
      "inferenceMs": 7823.420917004114,
      "tokensPerSecond": 0.6391066073324213,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 10254.911166994134,
      "requestMs": 7824.8311250063125,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2430.0800419878215,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 9562.146334006684,
      "inferenceMs": 9561.730207991786,
      "tokensPerSecond": 0.6275014949684673,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 11993.358459003503,
      "requestMs": 9563.278417015681,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2430.0800419878215,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 23206.297124997946,
      "inferenceMs": 23205.806417012354,
      "tokensPerSecond": 0.646389947862505,
      "inputTokens": 52,
      "outputTokens": 15,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 25637.389124982292,
      "requestMs": 23207.30908299447,
      "grade": "slow",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "56",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2230.336165986955,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1177.9553749947809,
      "inferenceMs": 1177.6362920063548,
      "tokensPerSecond": 4.245793063562463,
      "inputTokens": 26,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 3409.3155410082545,
      "requestMs": 1178.9793750212993,
      "grade": "moderate",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "READY",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2230.336165986955,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 947.3185840179212,
      "inferenceMs": 946.9735419843346,
      "tokensPerSecond": 4.223982849212666,
      "inputTokens": 29,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 3178.7328330101445,
      "requestMs": 948.3966670231894,
      "grade": "moderate",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "Oslo",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2230.336165986955,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1217.3577500216197,
      "inferenceMs": 1217.0138749934267,
      "tokensPerSecond": 4.108416594697415,
      "inputTokens": 41,
      "outputTokens": 5,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 3448.7515819782857,
      "requestMs": 1218.4154159913305,
      "grade": "moderate",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": "ORCHID",
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2230.336165986955,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 1431.6077079856768,
      "inferenceMs": 1431.2324580096174,
      "tokensPerSecond": 4.192191119214879,
      "inputTokens": 51,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 3662.962915987009,
      "requestMs": 1432.626750000054,
      "grade": "moderate",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "accuracy": 1,
      "expected": {
        "name": "Ada",
        "count": 3
      },
      "method": "Exact normalized string or exact parsed JSON; no truncation permitted"
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2230.336165986955,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 617
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; tokenization/decoding on CPU. MPS operator CPU fallback disabled.",
      "durationMs": 3381.6107919847127,
      "inferenceMs": 3381.2420410104096,
      "tokensPerSecond": 4.436239647463268,
      "inputTokens": 52,
      "outputTokens": 15,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "precision": "fp32",
      "cpuThreads": 1,
      "totalMs": 5612.975666008424,
      "requestMs": 3382.639500021469,
      "grade": "moderate",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "fallback": false
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 12791.747083014343,
      "inferenceMs": 12747.160833969247,
      "tokensPerSecond": 0.31379536605050184,
      "inputTokens": 830,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 617
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "loadMs": 4817.363833019044,
      "totalMs": 17616.298042004928,
      "wallMs": 12798.182083002757,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 23953.209292027168,
      "inferenceMs": 23946.17183302762,
      "tokensPerSecond": 0.41760328413778264,
      "inputTokens": 795,
      "outputTokens": 10,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 617
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "loadMs": 4817.363833019044,
      "totalMs": 41571.26137497835,
      "wallMs": 23954.88762500463,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 11922.951209009625,
      "inferenceMs": 11916.825708001852,
      "tokensPerSecond": 0.3356598559055957,
      "inputTokens": 828,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 617
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "loadMs": 4817.363833019044,
      "totalMs": 53495.570999977645,
      "wallMs": 11924.195749976207,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 15623.950292007066,
      "inferenceMs": 15604.197374952491,
      "tokensPerSecond": 0.3845119268762305,
      "inputTokens": 154,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 617
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "loadMs": 4817.363833019044,
      "totalMs": 69120.88779202895,
      "wallMs": 15625.252292025834,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 17341.41916700173,
      "inferenceMs": 17335.376416973304,
      "tokensPerSecond": 0.3461130497359906,
      "inputTokens": 154,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 617
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "loadMs": 4817.363833019044,
      "totalMs": 86463.65883300314,
      "wallMs": 17342.69400004996,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 10831.494499987457,
      "inferenceMs": 10831.094250024762,
      "tokensPerSecond": 0.36930709932570804,
      "inputTokens": 25,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 617
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "loadMs": 4817.363833019044,
      "totalMs": 97296.33441701299,
      "wallMs": 10832.603749993723,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 3943.6054169782437,
      "inferenceMs": 3878.1595830223523,
      "tokensPerSecond": 1.0314170715179014,
      "inputTokens": 830,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float16": 617
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "driverAllocatedMemoryBytes": 8224980992,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "loadMs": 5351.722374965902,
      "totalMs": 9303.320834005717,
      "wallMs": 3950.681500020437,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 3330.80495899776,
      "inferenceMs": 3316.3356659933925,
      "tokensPerSecond": 3.0153763090216468,
      "inputTokens": 795,
      "outputTokens": 10,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float16": 617
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "driverAllocatedMemoryBytes": 9507192832,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "loadMs": 5351.722374965902,
      "totalMs": 12636.323624989018,
      "wallMs": 3332.918916014023,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 3349.3286659941077,
      "inferenceMs": 3322.10779201705,
      "tokensPerSecond": 1.204054850240534,
      "inputTokens": 828,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float16": 617
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "image",
      "videoFrameCount": 0,
      "videoFps": null,
      "driverAllocatedMemoryBytes": 10896261120,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "loadMs": 5351.722374965902,
      "totalMs": 15987.293709011283,
      "wallMs": 3350.8538330206648,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 833.0473329988308,
      "inferenceMs": 818.858916987665,
      "tokensPerSecond": 7.327269539998649,
      "inputTokens": 154,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float16": 617
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "driverAllocatedMemoryBytes": 10914562048,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "loadMs": 5351.722374965902,
      "totalMs": 16822.04262498999,
      "wallMs": 834.6918749739416,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "good"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 510.3952910285443,
      "inferenceMs": 502.99841701053083,
      "tokensPerSecond": 11.928466963494207,
      "inputTokens": 154,
      "outputTokens": 6,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float16": 617
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "video",
      "videoFrameCount": 4,
      "videoFps": 1,
      "driverAllocatedMemoryBytes": 10914562048,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "loadMs": 5351.722374965902,
      "totalMs": 17333.99700000882,
      "wallMs": 511.88841700786725,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "good"
    },
    "failure": null
  },
  {
    "precision": "fp16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 334.9757500109263,
      "inferenceMs": 333.5396670154296,
      "tokensPerSecond": 11.992576582547706,
      "inputTokens": 25,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float16": 617
      },
      "decoding": "greedy",
      "enableThinking": false,
      "outputLimitReached": false,
      "inputModality": "text",
      "videoFrameCount": 0,
      "videoFps": null,
      "driverAllocatedMemoryBytes": 10914889728,
      "memoryMeasurementScope": "Post-inference driver allocation; not peak",
      "loadMs": 5351.722374965902,
      "totalMs": 17670.212625002023,
      "wallMs": 336.16883296053857,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "15852e8c16360a2fea060d615a32b45270f8a8fc",
      "device": "mps",
      "precision": "fp16",
      "requestedDevice": "mps",
      "requestedPrecision": "fp16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float16",
      "grade": "good"
    },
    "failure": null
  }
]

Something not quite right?

Tell us what happened with Qwen3.5 Desktop. Loading, results, or the page itself—we’re here for all of it.

Attachments Optional

Up to 3 files · 50 MB each

Browser, OS, CPU/GPU, memory, screen, storage, and network details are included automatically. Model inputs, results, and browsing history are not collected.