A model is still running

A model is still running or downloading. Leaving will stop the current operation. Stay on this page to let it finish.

Skip to content
LocalAI.toolsBeta

Join the waitlist

Get an email when the new apps and model runs launch.

No account needed.

TranslateGemma

Translate text or text visible in still images locally. Supply source and target language codes; review OCR, names, numbers and omissions.

Desktop inferencePrivate · Runs on your computer
About this model

Translate text or text visible in still images locally. Supply source and target language codes; review OCR, names, numbers and omissions.

Created by
Google
Model released
Jan 15, 2026
Source repository created
Jan 12, 2026
Model weights · BF16
8.6 GB

Best for

  • Local short-form text translation
  • Comparing CPU and Metal translation

Keep in mind

  • Translations can omit, invent or mistranslate details, names, numbers, negation and idioms. Review important outputs.
  • Five authored short cases in four languages are a compatibility suite, not validation across all55 languages or domains.
  • Text-only adapter: no image translation, automatic language detection, conversation or streaming.
  • The official translation context is2048 input-plus-output tokens; excess budgets return invalid_input, never silent input truncation.
  • Official Hugging Face model access requires accepting Google terms and local authentication. Gemma usage terms apply.
  • Metal FP16 returned empty translations in both cold probes and is excluded; requests recover to CPU FP32 by default.
  • Source weights are BF16; FP32/FP16 are casts, not extra source precision. Large inputs/output and CPU fallback require additional memory.
  • The app downloads its own pinned CPython and isolated packages on request; installed Python, user packages and Python/pip configuration are not used. Automatic model-package provisioning is verified on Apple Silicon Mac; other platform stacks remain unverified.
  • Not a safety-critical, medical or legal translation service; no dedicated content filter.

Processed on your deviceYour input is processed locally in the desktop app. Model and runtime files download from external hosts.

Target platforms · verification required

  • macos · aarch64, x86_64
  • windows · x86_64

Recommended configuration

BF16 · mps (verified platform only)

Sources & technical details
Original model
https://huggingface.co/google/translategemma-4b-it
Release verification
Verified
Release evidence
Google release log explicitly lists TranslateGemma 4B, 12B and 27B release.
Source updated
Jan 28, 2026
Native model files
https://huggingface.co/google/translategemma-4b-it
Files updated
Jan 28, 2026
Artifact revision
10042cb0e6e7fdce748996a71dc3dc432a4e0c89

Desktop app

Sample result

TranslateGemma 4B — Text & Image Translation

Run TranslateGemma on your computer

Download the desktop app to load this model, provide input and view results locally.

Public desktop downloads are coming soon.

Device requirements

macOS Apple SiliconCPU / FP32Tested32.0 GiBRAM estimate
macOS Apple SiliconMPS / BF16Tested24.0 GiBRAM estimate
macOS Apple SiliconMPS / FP32Tested32.0 GiBRAM estimate
Windows Intel / AMDCPU / FP32Unverified32.0 GiBRAM estimate
macOS Intel / AMDCPU / FP32Unverified32.0 GiBRAM estimate
If your device falls short

MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.

GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.

PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.

Benchmark evidence

Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.

TranslateGemma 4B — Text & Image Translation

48 configurations tested
Passed48
Failed0
TestedNov 1, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
FP32mpsAcceleratedPassedFastmacos · aarch64864 ms / 5030 ms
BF16mpsAcceleratedPassedFastmacos · aarch64955 ms / 7566 ms
BF16mpsAcceleratedPassedSlowmacos · aarch644108 ms / 7566 ms
FP32mpsAcceleratedPassedSlowmacos · aarch645165 ms / 5030 ms
BF16mpsAcceleratedPassedVery slowmacos · aarch644124 ms / 7566 ms
FP32mpsAcceleratedPassedVery slowmacos · aarch645799 ms / 5030 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch64931 ms / 3917 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641052 ms / 3917 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch64502 ms / 3917 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch64838 ms / 3917 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641040 ms / 3917 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch64651 ms / 3917 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch64945 ms / 4002 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch64945 ms / 4002 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch64507 ms / 4002 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch64730 ms / 4002 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch64951 ms / 4002 ms
FP32mpsAcceleratedPassedNot measuredmacos · aarch642097 ms / 3963 ms
FP32mpsAcceleratedPassedNot measuredmacos · aarch641377 ms / 3963 ms
FP32mpsAcceleratedPassedNot measuredmacos · aarch64825 ms / 3963 ms
FP32mpsAcceleratedPassedNot measuredmacos · aarch641028 ms / 3963 ms
FP32mpsAcceleratedPassedNot measuredmacos · aarch641346 ms / 3963 ms
FP32mpsAcceleratedPassedNot measuredmacos · aarch64857 ms / 3963 ms
FP32mpsAcceleratedPassedNot measuredmacos · aarch641190 ms / 3916 ms
FP32mpsAcceleratedPassedNot measuredmacos · aarch641285 ms / 3916 ms
FP32mpsAcceleratedPassedNot measuredmacos · aarch64674 ms / 3916 ms
FP32mpsAcceleratedPassedNot measuredmacos · aarch64977 ms / 3916 ms
FP32mpsAcceleratedPassedNot measuredmacos · aarch641335 ms / 3916 ms
FP32cpuPassedSlowmacos · aarch641285 ms / 6427 ms
FP32cpuPassedSlowmacos · aarch647729 ms / 6427 ms
FP32cpuPassedVery slowmacos · aarch646934 ms / 6427 ms
FP32cpuPassedNot measuredmacos · aarch642156 ms / 2805 ms
FP32cpuPassedNot measuredmacos · aarch643008 ms / 2805 ms
FP32cpuPassedNot measuredmacos · aarch641595 ms / 2805 ms
FP32cpuPassedNot measuredmacos · aarch642218 ms / 2805 ms
FP32cpuPassedNot measuredmacos · aarch643148 ms / 2805 ms
FP32cpuPassedNot measuredmacos · aarch642098 ms / 2805 ms
FP32cpuPassedNot measuredmacos · aarch642369 ms / 2982 ms
FP32cpuPassedNot measuredmacos · aarch643342 ms / 2982 ms
FP32cpuPassedNot measuredmacos · aarch641672 ms / 2982 ms
FP32cpuPassedNot measuredmacos · aarch642244 ms / 2982 ms
FP32cpuPassedNot measuredmacos · aarch643262 ms / 2982 ms
FP32cpuPassedNot measuredmacos · aarch642222 ms / 2982 ms
FP32cpuPassedNot measuredmacos · aarch642154 ms / 2950 ms
FP32cpuPassedNot measuredmacos · aarch643063 ms / 2950 ms
FP32cpuPassedNot measuredmacos · aarch641713 ms / 2950 ms
FP32cpuPassedNot measuredmacos · aarch642299 ms / 2950 ms
FP32cpuPassedNot measuredmacos · aarch643097 ms / 2950 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-01T14:34:47.539510+00:00
Environment
{ "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2" }
Runtime
transformers 5.22.1
Artifact revision
10042cb0e6e7fdce748996a71dc3dc432a4e0c89
Fixture
{
  "file": "model-samples/desktop/translategemma-translations.json",
  "size": 1366,
  "sha1": "7a7953bc58138b42b981439fff2c2b724f4cc3a3",
  "sha256": "fb5f28fdcd81bb7d0bed06aaa029cdde7870a8b4ae2fa999a7c019cebce06f57",
  "url": "https://assets.localai.tools/samples/v1/files/fb5f28fdcd81bb7d0bed06aaa029cdde7870a8b4ae2fa999a7c019cebce06f57/translategemma-translations.json"
}
Rerun when
model revision/checksums; runtime/processor/template/backend/driver; fixture or reference checksums; language codes and generation budgets; quality gates
Per-case quality & failure details
[
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2805.3540000000794,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2198.0641250011104,
      "inferenceMs": 2156.2477919997036,
      "tokensPerSecond": 5.101454499252427,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5004.863250000199,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.101454499252427,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.9420387649932294,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2805.3540000000794,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 3008.9539579985285,
      "inferenceMs": 3007.989333000296,
      "tokensPerSecond": 5.651615786497314,
      "inputTokens": 91,
      "outputTokens": 17,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5815.731374999814,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.651615786497314,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2805.3540000000794,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1595.5337079994933,
      "inferenceMs": 1594.6597500005737,
      "tokensPerSecond": 5.016744167523587,
      "inputTokens": 84,
      "outputTokens": 8,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4402.23933299967,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.016744167523587,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2805.3540000000794,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2218.7603330003185,
      "inferenceMs": 2217.9179590002605,
      "tokensPerSecond": 5.410479657872048,
      "inputTokens": 83,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5025.471625000137,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.410479657872048,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.8333179436563465,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2805.3540000000794,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 3149.1119999991497,
      "inferenceMs": 3148.2323330001236,
      "tokensPerSecond": 5.7174941669082,
      "inputTokens": 86,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5955.8498749993305,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.7174941669082,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2805.3540000000794,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2099.250833998667,
      "inferenceMs": 2098.437082999226,
      "tokensPerSecond": 5.241996574077917,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4905.948583000281,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.241996574077917,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2981.542374998753,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2412.05541599993,
      "inferenceMs": 2369.103625000207,
      "tokensPerSecond": 4.643106314101832,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5395.135166998443,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 4.643106314101832,
        "grade": "C"
      },
      "grade": "C"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.9420387649932294,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2981.542374998753,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 3342.61641699959,
      "inferenceMs": 3341.629250000551,
      "tokensPerSecond": 5.087338758480522,
      "inputTokens": 91,
      "outputTokens": 17,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 6325.648832998922,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.087338758480522,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2981.542374998753,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1673.2737499987707,
      "inferenceMs": 1672.3838340003567,
      "tokensPerSecond": 4.783590846405116,
      "inputTokens": 84,
      "outputTokens": 8,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4656.270999998014,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 4.783590846405116,
        "grade": "C"
      },
      "grade": "C"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2981.542374998753,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2244.5151659994735,
      "inferenceMs": 2243.5423329989135,
      "tokensPerSecond": 5.348684454712187,
      "inputTokens": 83,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5227.513332998569,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.348684454712187,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.8333179436563465,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2981.542374998753,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 3263.1425839990698,
      "inferenceMs": 3262.1458749999874,
      "tokensPerSecond": 5.5178403081070275,
      "inputTokens": 86,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 6246.278833999895,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.5178403081070275,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2981.542374998753,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2222.4363340010314,
      "inferenceMs": 2221.6335410012107,
      "tokensPerSecond": 4.951311634880473,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5205.436541999006,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 4.951311634880473,
        "grade": "C"
      },
      "grade": "C"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2949.97208300083,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2197.796582999217,
      "inferenceMs": 2154.196708001109,
      "tokensPerSecond": 5.106311767696907,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5149.17429200068,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.106311767696907,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.9420387649932294,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2949.97208300083,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 3063.5832079988177,
      "inferenceMs": 3062.6570000003994,
      "tokensPerSecond": 5.550735847990089,
      "inputTokens": 91,
      "outputTokens": 17,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 6015.008708000096,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.550735847990089,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2949.97208300083,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1713.566249998621,
      "inferenceMs": 1712.6290409996727,
      "tokensPerSecond": 4.671180861986521,
      "inputTokens": 84,
      "outputTokens": 8,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4664.965917001609,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 4.671180861986521,
        "grade": "C"
      },
      "grade": "C"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2949.97208300083,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2300.364292001177,
      "inferenceMs": 2299.487542000861,
      "tokensPerSecond": 5.218554039026625,
      "inputTokens": 83,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5251.81883300138,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.218554039026625,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.8333179436563465,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 2949.97208300083,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 3098.0782919996273,
      "inferenceMs": 3097.1428329994524,
      "tokensPerSecond": 5.8118081633864325,
      "inputTokens": 86,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 6049.454458001492,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.8118081633864325,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3917.3670419986593,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 985.17237499982,
      "inferenceMs": 930.6181669999205,
      "tokensPerSecond": 11.82010021947158,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8763654144,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4904.1318759973365,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 11.82010021947158,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.9420387649932294,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3917.3670419986593,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1055.1266250004119,
      "inferenceMs": 1051.6604589993221,
      "tokensPerSecond": 16.16491316615238,
      "inputTokens": 91,
      "outputTokens": 17,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8764719104,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4973.922957999093,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 16.16491316615238,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3917.3670419986593,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 504.15924999833805,
      "inferenceMs": 501.79516699972737,
      "tokensPerSecond": 15.94276016613039,
      "inputTokens": 84,
      "outputTokens": 8,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8764719104,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4423.161249998884,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 15.94276016613039,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3917.3670419986593,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 840.723582999999,
      "inferenceMs": 837.8290000000561,
      "tokensPerSecond": 14.32273172687887,
      "inputTokens": 83,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8765308928,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4759.513708999293,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 14.32273172687887,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.8333179436563465,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3917.3670419986593,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1042.914292000205,
      "inferenceMs": 1040.3773749985703,
      "tokensPerSecond": 17.301414306539236,
      "inputTokens": 86,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8765898752,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4961.727416997746,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 17.301414306539236,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3917.3670419986593,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 654.4943330009119,
      "inferenceMs": 650.5033749999711,
      "tokensPerSecond": 16.909981443217706,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8765898752,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4573.566916998971,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 16.909981443217706,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4002.4391669994657,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 990.976375000173,
      "inferenceMs": 945.4658329996164,
      "tokensPerSecond": 11.634476483513987,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8763654144,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4995.005624999976,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 11.634476483513987,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.9420387649932294,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4002.4391669994657,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 947.0597080016887,
      "inferenceMs": 944.5086249997985,
      "tokensPerSecond": 17.998776877239873,
      "inputTokens": 91,
      "outputTokens": 17,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8764719104,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4950.873709000007,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 17.998776877239873,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4002.4391669994657,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 508.7640410001768,
      "inferenceMs": 506.5085830010503,
      "tokensPerSecond": 15.794401651794736,
      "inputTokens": 84,
      "outputTokens": 8,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8764719104,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4512.937832998432,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 15.794401651794736,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4002.4391669994657,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 732.7146250008809,
      "inferenceMs": 730.4523750008229,
      "tokensPerSecond": 16.42817575887337,
      "inputTokens": 83,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8765308928,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4736.885791999157,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 16.42817575887337,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.8333179436563465,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 4002.4391669994657,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 952.740000000631,
      "inferenceMs": 950.6874999988213,
      "tokensPerSecond": 18.933666425636517,
      "inputTokens": 86,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 8765898752,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4956.709250000131,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 18.933666425636517,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3962.649416998829,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2142.499666999356,
      "inferenceMs": 2097.339500000089,
      "tokensPerSecond": 5.244739823952933,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 17907236864,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 6106.601916999352,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.244739823952933,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.9420387649932294,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3962.649416998829,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1381.1138750006648,
      "inferenceMs": 1377.2402079994208,
      "tokensPerSecond": 12.343525770783442,
      "inputTokens": 91,
      "outputTokens": 17,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 17908301824,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5345.39174999918,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 12.343525770783442,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3962.649416998829,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 831.0627080008999,
      "inferenceMs": 824.9062499999127,
      "tokensPerSecond": 9.698071750578743,
      "inputTokens": 84,
      "outputTokens": 8,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 17908301824,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4796.62720899978,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 9.698071750578743,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3962.649416998829,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1033.1501670007128,
      "inferenceMs": 1028.0356669991306,
      "tokensPerSecond": 11.672746758902235,
      "inputTokens": 83,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 17908891648,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4998.165041997709,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 11.672746758902235,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.8333179436563465,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3962.649416998829,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1349.3314169991208,
      "inferenceMs": 1345.7292500006588,
      "tokensPerSecond": 13.375647441705818,
      "inputTokens": 86,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 17909481472,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5314.326666999477,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 13.375647441705818,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3962.649416998829,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 860.4417919996195,
      "inferenceMs": 857.0996250000462,
      "tokensPerSecond": 12.833980647231536,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 17909481472,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4825.681666998207,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 12.833980647231536,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3916.1630840008,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1237.134375000096,
      "inferenceMs": 1190.4909589993622,
      "tokensPerSecond": 9.23988537405255,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 17907236864,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5155.634084001576,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 9.23988537405255,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.9420387649932294,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3916.1630840008,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1287.6327499998297,
      "inferenceMs": 1285.1419999988138,
      "tokensPerSecond": 13.228110201063922,
      "inputTokens": 91,
      "outputTokens": 17,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 17908301824,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5206.154417999642,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 13.228110201063922,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3916.1630840008,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 677.26358300024,
      "inferenceMs": 673.5777499998221,
      "tokensPerSecond": 11.876876871307154,
      "inputTokens": 84,
      "outputTokens": 8,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 17908301824,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4595.846209000229,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 11.876876871307154,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3916.1630840008,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 980.3995000002033,
      "inferenceMs": 976.9319999995787,
      "tokensPerSecond": 12.283352372534807,
      "inputTokens": 83,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 17908891648,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 4898.568459000671,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 12.283352372534807,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.8333179436563465,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 3916.1630840008,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.float32": 883
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1338.5746670010121,
      "inferenceMs": 1335.1532920005411,
      "tokensPerSecond": 13.481598036604103,
      "inputTokens": 86,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "driverAllocatedMemoryBytes": 17909481472,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "totalMs": 5257.405459000438,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 13.481598036604103,
        "grade": "B"
      },
      "grade": "B"
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 7003.267125051934,
      "inferenceMs": 6933.844999992289,
      "tokensPerSecond": 0.4326603781889177,
      "inputTokens": 373,
      "outputTokens": 3,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 883
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "image",
      "loadMs": 6426.95279198233,
      "totalMs": 13432.36724997405,
      "wallMs": 7005.117000022437,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 7733.717249997426,
      "inferenceMs": 7729.107124963775,
      "tokensPerSecond": 1.1644294553676768,
      "inputTokens": 373,
      "outputTokens": 9,
      "maxNewTokens": 64,
      "placement": {
        "cpu/torch.float32": 883
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "image",
      "loadMs": 6426.95279198233,
      "totalMs": 21167.89758298546,
      "wallMs": 7735.42995803291,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 1285.541832970921,
      "inferenceMs": 1284.7164580016397,
      "tokensPerSecond": 3.89190935389544,
      "inputTokens": 77,
      "outputTokens": 5,
      "maxNewTokens": 32,
      "placement": {
        "cpu/torch.float32": 883
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "loadMs": 6426.95279198233,
      "totalMs": 22454.928542021662,
      "wallMs": 1286.9422499788925,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "device": "cpu",
      "precision": "fp32",
      "requestedDevice": "cpu",
      "requestedPrecision": "fp32",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 5853.161042032298,
      "inferenceMs": 5798.699582985137,
      "tokensPerSecond": 0.5173573759197261,
      "inputTokens": 373,
      "outputTokens": 3,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float32": 883
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "image",
      "driverAllocatedMemoryBytes": 22195699712,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "loadMs": 5029.8561249510385,
      "totalMs": 10885.954374971334,
      "wallMs": 5855.796208023094,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 5174.41570898518,
      "inferenceMs": 5164.625249977689,
      "tokensPerSecond": 1.7426240171131253,
      "inputTokens": 373,
      "outputTokens": 9,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.float32": 883
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "image",
      "driverAllocatedMemoryBytes": 22187606016,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "loadMs": 5029.8561249510385,
      "totalMs": 16064.887749962509,
      "wallMs": 5178.74716699589,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "fp32",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 868.847250007093,
      "inferenceMs": 864.0725829754956,
      "tokensPerSecond": 5.786550920042091,
      "inputTokens": 77,
      "outputTokens": 5,
      "maxNewTokens": 32,
      "placement": {
        "mps:0/torch.float32": 883
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 22238593024,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "loadMs": 5029.8561249510385,
      "totalMs": 16938.131957955193,
      "wallMs": 873.0481659877114,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "device": "mps",
      "precision": "fp32",
      "requestedDevice": "mps",
      "requestedPrecision": "fp32",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.float32",
      "grade": "good"
    },
    "failure": null
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 4189.44291700609,
      "inferenceMs": 4123.577499995008,
      "tokensPerSecond": 0.727523612689135,
      "inputTokens": 373,
      "outputTokens": 3,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "image",
      "driverAllocatedMemoryBytes": 13874200576,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "loadMs": 7565.672459022608,
      "totalMs": 11758.405167027377,
      "wallMs": 4192.3496669624,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "device": "mps",
      "precision": "bf16",
      "requestedDevice": "mps",
      "requestedPrecision": "bf16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.bfloat16",
      "grade": "very-slow"
    },
    "failure": null
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 4118.045540992171,
      "inferenceMs": 4107.648040982895,
      "tokensPerSecond": 2.191034847729175,
      "inputTokens": 373,
      "outputTokens": 9,
      "maxNewTokens": 64,
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "image",
      "driverAllocatedMemoryBytes": 14210039808,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "loadMs": 7565.672459022608,
      "totalMs": 15881.529500009492,
      "wallMs": 4121.432750020176,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "device": "mps",
      "precision": "bf16",
      "requestedDevice": "mps",
      "requestedPrecision": "bf16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.bfloat16",
      "grade": "slow"
    },
    "failure": null
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "scope": "Predeclared fixture answer check",
      "nonEmpty": true,
      "repetitive": false,
      "grade": "fixture-pass",
      "score": 1,
      "gibberish": false
    },
    "runtime": {
      "durationMs": 959.1342080384493,
      "inferenceMs": 955.3955420269631,
      "tokensPerSecond": 5.2334345096398724,
      "inputTokens": 77,
      "outputTokens": 5,
      "maxNewTokens": 32,
      "placement": {
        "mps:0/torch.bfloat16": 883
      },
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 14219083776,
      "memoryMeasurementScope": "Post-inference driver allocation, not peak",
      "loadMs": 7565.672459022608,
      "totalMs": 16843.326749978587,
      "wallMs": 961.676582985092,
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "artifactRevision": "10042cb0e6e7fdce748996a71dc3dc432a4e0c89",
      "device": "mps",
      "precision": "bf16",
      "requestedDevice": "mps",
      "requestedPrecision": "bf16",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "fallback": false,
      "parameterDtype": "torch.bfloat16",
      "grade": "good"
    },
    "failure": null
  }
]

TranslateGemma 12B — Text & Image Translation

50 configurations tested
Passed50
Failed0
TestedNov 5, 2026
Runtimetransformers 5.22.1

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
BF16mpsAcceleratedPassedNot measuredmacos · aarch643769 ms / 10161 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch643357 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641972 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch642719 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch643616 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch644679 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch645225 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch642021 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641979 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch642052 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch642188 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch642459 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch642103 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch642526 ms / 5774 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch642052 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641964 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641904 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641929 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641991 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch642504 ms / 5430 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641921 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641986 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641925 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641928 ms / 0 ms
BF16mpsAcceleratedPassedNot measuredmacos · aarch641866 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch6418701 ms / 20814 ms
FP32cpuPassedNot measuredmacos · aarch6410917 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch645411 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch648168 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch6410816 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch6411468 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch6413791 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch647774 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch647879 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch647691 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch647779 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch647679 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch647669 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch6417586 ms / 21513 ms
FP32cpuPassedNot measuredmacos · aarch647994 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch648162 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch648221 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch648278 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch648342 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch6417672 ms / 16340 ms
FP32cpuPassedNot measuredmacos · aarch647392 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch647403 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch647187 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch647770 ms / 0 ms
FP32cpuPassedNot measuredmacos · aarch647960 ms / 0 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-05T22:29:17.343657+00:00
Environment
{ "hardware": "Apple M3 Max", "ramBytes": 103079215104, "os": "macOS27.2", "runtime": "Transformers5.22.1/Torch2.10.0" }
Runtime
transformers 5.22.1
Artifact revision
d1b225e1caa17f1ddc7e62065d8637d0923f34e2
Fixture
{
  "file": "model-samples/desktop/translategemma-translations.json",
  "size": 1366,
  "sha1": "7a7953bc58138b42b981439fff2c2b724f4cc3a3",
  "sha256": "fb5f28fdcd81bb7d0bed06aaa029cdde7870a8b4ae2fa999a7c019cebce06f57",
  "url": "https://assets.localai.tools/samples/v1/files/fb5f28fdcd81bb7d0bed06aaa029cdde7870a8b4ae2fa999a7c019cebce06f57/translategemma-translations.json"
}
Rerun when
model revision/checksums; runtime/processor/template/backend/driver; fixture or reference checksums; language codes and generation budgets; quality gates; Concrete checkpoint/template/processor or image-translation/stream implementation changes
Per-case quality & failure details
[
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 20814.108542050235,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 18750.376000010874,
      "inferenceMs": 18700.732958037406,
      "tokensPerSecond": 0.6416860786647675,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 39564.48454206111,
      "processAndInitWallMs": 23145.30645898776,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 0.6416860786647675,
        "grade": "D"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "D",
      "outputCharacters": 38,
      "charactersPerSecond": 2.0320059157717636
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 10918.024500017054,
      "inferenceMs": 10916.8608749751,
      "tokensPerSecond": 1.6488256291020156,
      "inputTokens": 91,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 10918.024500017054,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6488256291020156,
        "grade": "C"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 51,
      "charactersPerSecond": 4.671672615789044
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 5412.598291004542,
      "inferenceMs": 5411.335791985039,
      "tokensPerSecond": 1.4783780396421051,
      "inputTokens": 84,
      "outputTokens": 8,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 5412.598291004542,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.4783780396421051,
        "grade": "C"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 34,
      "charactersPerSecond": 6.283106668478947
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 8169.297959015239,
      "inferenceMs": 8168.211416981649,
      "tokensPerSecond": 1.5915356907845333,
      "inputTokens": 83,
      "outputTokens": 13,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 8169.297959015239,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.5915356907845333,
        "grade": "C"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 44,
      "charactersPerSecond": 5.386736184193805
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.8333179436563465,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 10817.268541024532,
      "inferenceMs": 10816.100290976465,
      "tokensPerSecond": 1.6641857523285761,
      "inputTokens": 86,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 10817.268541024532,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6641857523285761,
        "grade": "C"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 61,
      "charactersPerSecond": 5.639740605113508
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.9726275669616796,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 11507.813916017767,
      "inferenceMs": 11467.677082982846,
      "tokensPerSecond": 0.34880647327746034,
      "inputTokens": 373,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "image",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 11507.813916017767,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 0.34880647327746034,
        "grade": "D"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "D",
      "outputCharacters": 11,
      "charactersPerSecond": 0.9592178015130158
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true,
      "previousReferenceScore": 0.7241705887255625,
      "referenceRevision": 2,
      "referenceCorrection": "Image says arrives, not leaves; complete original output rescored without new inference or changed measurement timestamps."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 13796.391999989282,
      "inferenceMs": 13791.45849996712,
      "tokensPerSecond": 0.6525778256173165,
      "inputTokens": 373,
      "outputTokens": 9,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "image",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 13796.391999989282,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 0.6525778256173165,
        "grade": "D"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "D",
      "outputCharacters": 26,
      "charactersPerSecond": 1.8852248295611362
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 7774.467874958646,
      "inferenceMs": 7773.622500011697,
      "tokensPerSecond": 1.543681854885794,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7774.467874958646,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.543681854885794,
        "grade": "C"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.888325873805014
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 7880.37454104051,
      "inferenceMs": 7879.408999986481,
      "tokensPerSecond": 1.5229568613611235,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7880.37454104051,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.5229568613611235,
        "grade": "C"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.8226967276435575
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 7692.187207983807,
      "inferenceMs": 7691.100208030548,
      "tokensPerSecond": 1.5602449162566336,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7692.187207983807,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.5602449162566336,
        "grade": "C"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.940775568146006
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 7780.111416999716,
      "inferenceMs": 7779.03137501562,
      "tokensPerSecond": 1.5426085101727596,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7780.111416999716,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.5426085101727596,
        "grade": "C"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.8849269488804055
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 7679.749250004534,
      "inferenceMs": 7678.825040988158,
      "tokensPerSecond": 1.5627390826000336,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7679.749250004534,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.5627390826000336,
        "grade": "C"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.948673761566773
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 7669.532208004966,
      "inferenceMs": 7668.669958016835,
      "tokensPerSecond": 1.5648085086065269,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7669.532208004966,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.5648085086065269,
        "grade": "C"
      },
      "observedPeakRssBytes": 62635360256,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.955226943920668
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 21513.48520902684,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 17628.67908400949,
      "inferenceMs": 17585.603832965717,
      "tokensPerSecond": 0.6823763411242653,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 39142.16429303633,
      "processAndInitWallMs": 24455.030084005557,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 0.6823763411242653,
        "grade": "D"
      },
      "observedPeakRssBytes": 51748388864,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "D",
      "outputCharacters": 38,
      "charactersPerSecond": 2.1608584135601734
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 7994.840208033565,
      "inferenceMs": 7993.631374964025,
      "tokensPerSecond": 1.5011950685621909,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7994.840208033565,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.5011950685621909,
        "grade": "C"
      },
      "observedPeakRssBytes": 51748388864,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.753784383780271
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 8162.989042000845,
      "inferenceMs": 8162.122417008504,
      "tokensPerSecond": 1.47020583457484,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 8162.989042000845,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.47020583457484,
        "grade": "C"
      },
      "observedPeakRssBytes": 51748388864,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.655651809486994
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 8221.425041963812,
      "inferenceMs": 8220.546249998733,
      "tokensPerSecond": 1.4597570082404012,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 8221.425041963812,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.4597570082404012,
        "grade": "C"
      },
      "observedPeakRssBytes": 51748388864,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.622563859427937
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 8279.021791007835,
      "inferenceMs": 8277.875832980499,
      "tokensPerSecond": 1.4496472575959536,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 8279.021791007835,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.4496472575959536,
        "grade": "C"
      },
      "observedPeakRssBytes": 51748388864,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.590549649053853
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 8342.972583021037,
      "inferenceMs": 8342.029083985835,
      "tokensPerSecond": 1.4384989406278095,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 8342.972583021037,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.4384989406278095,
        "grade": "C"
      },
      "observedPeakRssBytes": 51748388864,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.555246645321397
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 16340.036750014406,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 17714.265708986204,
      "inferenceMs": 17671.76549998112,
      "tokensPerSecond": 0.6790493004229159,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 34054.30245900061,
      "processAndInitWallMs": 21317.035000014585,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 0.6790493004229159,
        "grade": "D"
      },
      "observedPeakRssBytes": 62673731584,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "D",
      "outputCharacters": 38,
      "charactersPerSecond": 2.150322784672567
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 7393.179874983616,
      "inferenceMs": 7392.221041955054,
      "tokensPerSecond": 1.6233280812212165,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7393.179874983616,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6233280812212165,
        "grade": "C"
      },
      "observedPeakRssBytes": 62673731584,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 5.140538923867186
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 7404.267500038259,
      "inferenceMs": 7403.3715420519,
      "tokensPerSecond": 1.6208831249166389,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7404.267500038259,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6208831249166389,
        "grade": "C"
      },
      "observedPeakRssBytes": 62673731584,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 5.132796562236023
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 7188.012374972459,
      "inferenceMs": 7187.1529999771155,
      "tokensPerSecond": 1.6696458249933193,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7188.012374972459,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6696458249933193,
        "grade": "C"
      },
      "observedPeakRssBytes": 62673731584,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 5.287211779145511
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 7771.199208975304,
      "inferenceMs": 7770.266625040676,
      "tokensPerSecond": 1.544348550579779,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7771.199208975304,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.544348550579779,
        "grade": "C"
      },
      "observedPeakRssBytes": 62673731584,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.890437076835967
    }
  },
  {
    "precision": "fp32",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "cpu",
      "provider": "PyTorch CPU",
      "acceleration": "cpu",
      "placement": {
        "cpu/torch.float32": 1065
      },
      "parameterDtype": "torch.float32",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 7960.980458010454,
      "inferenceMs": 7960.001999977976,
      "tokensPerSecond": 1.5075373096681637,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "precision": "fp32",
      "cpuThreads": 4,
      "totalMs": 7960.980458010454,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.5075373096681637,
        "grade": "C"
      },
      "observedPeakRssBytes": 62673731584,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 4.773868147282518
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 10161.214666964952,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 3815.0306250317954,
      "inferenceMs": 3768.6234999564476,
      "tokensPerSecond": 3.184186480856652,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 13976.245291996747,
      "processAndInitWallMs": 12546.888916986063,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 3.184186480856652,
        "grade": "C"
      },
      "observedPeakRssBytes": 813088768,
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 10.083257189379397
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 3364.3570840358734,
      "inferenceMs": 3357.381042034831,
      "tokensPerSecond": 5.361321748898248,
      "inputTokens": 91,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25589104640,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 3364.3570840358734,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.361321748898248,
        "grade": "B"
      },
      "observedPeakRssBytes": 848052224,
      "grade": "B",
      "outputCharacters": 51,
      "charactersPerSecond": 15.190411621878368
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1977.1592919714749,
      "inferenceMs": 1972.1942499745637,
      "tokensPerSecond": 4.0563955604794915,
      "inputTokens": 84,
      "outputTokens": 8,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25589104640,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 1977.1592919714749,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 4.0563955604794915,
        "grade": "C"
      },
      "observedPeakRssBytes": 854573056,
      "grade": "C",
      "outputCharacters": 34,
      "charactersPerSecond": 17.239681132037838
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2725.083708995953,
      "inferenceMs": 2719.0627499949187,
      "tokensPerSecond": 4.781059208738119,
      "inputTokens": 83,
      "outputTokens": 13,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25590136832,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 2725.083708995953,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 4.781059208738119,
        "grade": "C"
      },
      "observedPeakRssBytes": 882737152,
      "grade": "C",
      "outputCharacters": 44,
      "charactersPerSecond": 16.182046552652096
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.8333179436563465,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 3621.0609159898013,
      "inferenceMs": 3615.866374981124,
      "tokensPerSecond": 4.978060064538188,
      "inputTokens": 86,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25591201792,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 3621.0609159898013,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 4.978060064538188,
        "grade": "C"
      },
      "observedPeakRssBytes": 911720448,
      "grade": "C",
      "outputCharacters": 61,
      "charactersPerSecond": 16.87009244093497
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.9726275669616796,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 4703.023584035691,
      "inferenceMs": 4679.290916013997,
      "tokensPerSecond": 0.8548303731898244,
      "inputTokens": 373,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "image",
      "driverAllocatedMemoryBytes": 28896903168,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 4703.023584035691,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 0.8548303731898244,
        "grade": "D"
      },
      "observedPeakRssBytes": 994590720,
      "grade": "D",
      "outputCharacters": 11,
      "charactersPerSecond": 2.350783526272017
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true,
      "previousReferenceScore": 0.7241705887255625,
      "referenceRevision": 2,
      "referenceCorrection": "Image says arrives, not leaves; complete original output rescored without new inference or changed measurement timestamps."
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 5232.437041995581,
      "inferenceMs": 5224.547833960969,
      "tokensPerSecond": 1.7226371134928795,
      "inputTokens": 373,
      "outputTokens": 9,
      "maxNewTokens": 64,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "image",
      "driverAllocatedMemoryBytes": 29434134528,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 5232.437041995581,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.7226371134928795,
        "grade": "C"
      },
      "observedPeakRssBytes": 1014022144,
      "grade": "C",
      "outputCharacters": 26,
      "charactersPerSecond": 4.9765072167572075
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2027.8654170106165,
      "inferenceMs": 2020.978250016924,
      "tokensPerSecond": 5.937718528093764,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 28874194944,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 2027.8654170106165,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.937718528093764,
        "grade": "B"
      },
      "observedPeakRssBytes": 1020035072,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 18.802775338963585
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1982.5050000217743,
      "inferenceMs": 1979.1662080097012,
      "tokensPerSecond": 6.063159299828334,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 28874194944,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 1982.5050000217743,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 6.063159299828334,
        "grade": "B"
      },
      "observedPeakRssBytes": 1022476288,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 19.200004449456394
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2056.177791033406,
      "inferenceMs": 2051.8050839891657,
      "tokensPerSecond": 5.848508756333389,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 28861612032,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 2056.177791033406,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.848508756333389,
        "grade": "B"
      },
      "observedPeakRssBytes": 1024819200,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 18.520277728389065
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2192.2904169769026,
      "inferenceMs": 2188.469749991782,
      "tokensPerSecond": 5.4832834678409705,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 28861612032,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 2192.2904169769026,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.4832834678409705,
        "grade": "B"
      },
      "observedPeakRssBytes": 1026965504,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 17.363730981496406
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2462.28074998362,
      "inferenceMs": 2458.858333004173,
      "tokensPerSecond": 4.880313696372533,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 28861612032,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 2462.28074998362,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 4.880313696372533,
        "grade": "C"
      },
      "observedPeakRssBytes": 1029603328,
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 15.454326705179689
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2105.3553750389256,
      "inferenceMs": 2102.65158402035,
      "tokensPerSecond": 5.7070796185146095,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 28861612032,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 2105.3553750389256,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.7070796185146095,
        "grade": "B"
      },
      "observedPeakRssBytes": 1031798784,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 18.07241879196293
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5773.994749994017,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2570.9800419863313,
      "inferenceMs": 2525.681708997581,
      "tokensPerSecond": 4.751192502701651,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 8344.974791980349,
      "processAndInitWallMs": 8422.221083019394,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 4.751192502701651,
        "grade": "C"
      },
      "observedPeakRssBytes": 823869440,
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 15.045442925221893
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2058.9742090087384,
      "inferenceMs": 2051.988707971759,
      "tokensPerSecond": 5.847985397473811,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 2058.9742090087384,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 5.847985397473811,
        "grade": "B"
      },
      "observedPeakRssBytes": 823869440,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 18.518620425333737
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1969.0306659904309,
      "inferenceMs": 1963.9066669624299,
      "tokensPerSecond": 6.1102700051221746,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 1969.0306659904309,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 6.1102700051221746,
        "grade": "B"
      },
      "observedPeakRssBytes": 823869440,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 19.349188349553554
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1907.2257080115378,
      "inferenceMs": 1903.5225420375355,
      "tokensPerSecond": 6.304101861150102,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 1907.2257080115378,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 6.304101861150102,
        "grade": "B"
      },
      "observedPeakRssBytes": 823869440,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 19.96298922697532
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1932.3398749693297,
      "inferenceMs": 1929.2572080157697,
      "tokensPerSecond": 6.220010452801124,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 1932.3398749693297,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 6.220010452801124,
        "grade": "B"
      },
      "observedPeakRssBytes": 823869440,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 19.696699767203558
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1994.610959023703,
      "inferenceMs": 1990.7710829866119,
      "tokensPerSecond": 6.027815102677328,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 1994.610959023703,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 6.027815102677328,
        "grade": "B"
      },
      "observedPeakRssBytes": 823869440,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 19.088081158478207
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 5429.80866698781,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 2548.2318329741247,
      "inferenceMs": 2504.100916965399,
      "tokensPerSecond": 4.792139134129718,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 7978.040499961935,
      "processAndInitWallMs": 7114.893625024706,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 4.792139134129718,
        "grade": "C"
      },
      "observedPeakRssBytes": 798932992,
      "grade": "C",
      "outputCharacters": 38,
      "charactersPerSecond": 15.17510725807744
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1924.7918340261094,
      "inferenceMs": 1921.4215420070104,
      "tokensPerSecond": 6.24537600815356,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 1924.7918340261094,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 6.24537600815356,
        "grade": "B"
      },
      "observedPeakRssBytes": 804028416,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 19.777024025819607
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1989.8510829661973,
      "inferenceMs": 1985.629250004422,
      "tokensPerSecond": 6.0434242696481615,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 1989.8510829661973,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 6.0434242696481615,
        "grade": "B"
      },
      "observedPeakRssBytes": 806551552,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 19.137510187219178
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1929.9311669892631,
      "inferenceMs": 1924.9550419626758,
      "tokensPerSecond": 6.2339118256833945,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 1929.9311669892631,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 6.2339118256833945,
        "grade": "B"
      },
      "observedPeakRssBytes": 809140224,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 19.74072078133075
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1932.1624580188654,
      "inferenceMs": 1927.6004580315202,
      "tokensPerSecond": 6.225356478828859,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 1932.1624580188654,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 6.225356478828859,
        "grade": "B"
      },
      "observedPeakRssBytes": 811663360,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 19.71362884962472
    }
  },
  {
    "precision": "bf16",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "transformers",
      "version": "5.22.1",
      "torchVersion": "2.10.0",
      "loadMs": 0,
      "device": "mps",
      "provider": "Metal/MPS",
      "acceleration": "partial",
      "placement": {
        "mps:0/torch.bfloat16": 1065
      },
      "parameterDtype": "torch.bfloat16",
      "accelerationDetails": "All model parameters and input tensors on selected device; text tokenization/decoding on CPU",
      "durationMs": 1870.280749979429,
      "inferenceMs": 1866.0280000185594,
      "tokensPerSecond": 6.430771671100674,
      "inputTokens": 84,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "decoding": "greedy",
      "outputLimitReached": false,
      "inputModality": "text",
      "driverAllocatedMemoryBytes": 25562333184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "precision": "bf16",
      "cpuThreads": 4,
      "totalMs": 1870.280749979429,
      "processAndInitWallMs": null,
      "artifactRevision": "d1b225e1caa17f1ddc7e62065d8637d0923f34e2",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 6.430771671100674,
        "grade": "B"
      },
      "observedPeakRssBytes": 813596672,
      "grade": "B",
      "outputCharacters": 38,
      "charactersPerSecond": 20.3641102918188
    }
  }
]

TranslateGemma 27B — Text & Image Translation

50 configurations tested
Passed50
Failed0
TestedNov 5, 2026
Runtimellama.cpp 0.3.36

Quality criteria

No explicit quality gate recorded. Review per-case evidence; full certification is pending.

PrecisionDeviceResultPlatformInference / load
Q4mpsAcceleratedPassedNot measuredmacos · aarch641522 ms / 1308 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641914 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641260 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641543 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641970 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch644476 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch644876 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641488 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641486 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641497 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641491 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641491 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641497 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641482 ms / 1393 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641475 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641479 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641486 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641463 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641471 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641486 ms / 1350 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641476 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641471 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641464 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641480 ms / 0 ms
Q4mpsAcceleratedPassedNot measuredmacos · aarch641472 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646566 ms / 15568 ms
Q4cpuPassedNot measuredmacos · aarch648020 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646136 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646891 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch648084 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch6435062 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch6435558 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646579 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646544 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646535 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646566 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646526 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646573 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646552 ms / 15610 ms
Q4cpuPassedNot measuredmacos · aarch646537 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646556 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646534 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646553 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646544 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646581 ms / 15650 ms
Q4cpuPassedNot measuredmacos · aarch646527 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646569 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646566 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646560 ms / 0 ms
Q4cpuPassedNot measuredmacos · aarch646613 ms / 0 ms
Test setup & sample details
Quality gate
No explicit quality gate recorded. Review the per-case quality evidence; full certification is pending.
Tested
2026-11-05T23:18:51.490317+00:00
Environment
{ "hardware": "Apple M3 Max", "ramBytes": 103079215104, "os": "macOS27.2", "runtime": "llama-cpp-python0.3.36/Metal+Accelerate builtfrompinnedofficialsource" }
Runtime
llama.cpp 0.3.36
Artifact revision
22fb50537007ccd41c41a0f0f0482e0fdb3335ab
Fixture
{
  "file": "model-samples/desktop/translategemma-translations.json",
  "size": 1366,
  "sha1": "7a7953bc58138b42b981439fff2c2b724f4cc3a3",
  "sha256": "fb5f28fdcd81bb7d0bed06aaa029cdde7870a8b4ae2fa999a7c019cebce06f57",
  "url": "https://assets.localai.tools/samples/v1/files/fb5f28fdcd81bb7d0bed06aaa029cdde7870a8b4ae2fa999a7c019cebce06f57/translategemma-translations.json"
}
Rerun when
model revision/checksums; runtime/processor/template/backend/driver; fixture or reference checksums; language codes and generation budgets; quality gates; Concrete checkpoint/template/processor or image-translation/stream implementation changes
Per-case quality & failure details
[
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 15567.559500050265,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6566.401416028384,
      "inferenceMs": 6566.264791996218,
      "tokensPerSecond": 1.6752294262345573,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 22133.96091607865,
      "processAndInitWallMs": 15681.74083402846,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6752294262345573,
        "grade": "C"
      },
      "observedPeakRssBytes": 35217981440,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.634862615516239
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 8020.276417024434,
      "inferenceMs": 8020.1812500017695,
      "tokensPerSecond": 2.119652844503514,
      "inputTokens": 91,
      "outputTokens": 17,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 8020.276417024434,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 2.119652844503514,
        "grade": "C"
      },
      "observedPeakRssBytes": 35229515776,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 50,
      "charactersPerSecond": 6.2342730720691595
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6136.322458041832,
      "inferenceMs": 6136.230542033445,
      "tokensPerSecond": 1.303731980928626,
      "inputTokens": 84,
      "outputTokens": 8,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6136.322458041832,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.303731980928626,
        "grade": "C"
      },
      "observedPeakRssBytes": 35232727040,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 34,
      "charactersPerSecond": 5.54086091894666
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6891.37899997877,
      "inferenceMs": 6891.2759170052595,
      "tokensPerSecond": 1.7413321051894317,
      "inputTokens": 83,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6891.37899997877,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.7413321051894317,
        "grade": "C"
      },
      "observedPeakRssBytes": 35232759808,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 43,
      "charactersPerSecond": 6.239773376928797
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.8772489985666848,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 8084.52633296838,
      "inferenceMs": 8084.430291026365,
      "tokensPerSecond": 2.2265019737976854,
      "inputTokens": 86,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 8084.52633296838,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 2.2265019737976854,
        "grade": "C"
      },
      "observedPeakRssBytes": 35239133184,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 59,
      "charactersPerSecond": 7.297978691892414
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.9726275669616796,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 35067.605208954774,
      "inferenceMs": 35061.637542035896,
      "tokensPerSecond": 0.11408480266229275,
      "inputTokens": 371,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "inputModality": "image",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 35067.605208954774,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 0.11408480266229275,
        "grade": "D"
      },
      "observedPeakRssBytes": 35508092928,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "D",
      "outputCharacters": 11,
      "charactersPerSecond": 0.31373320732130505
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 35559.80512499809,
      "inferenceMs": 35558.050959021784,
      "tokensPerSecond": 0.25310723611853425,
      "inputTokens": 371,
      "outputTokens": 9,
      "maxNewTokens": 64,
      "inputModality": "image",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 35559.80512499809,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 0.25310723611853425,
        "grade": "D"
      },
      "observedPeakRssBytes": 35522199552,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "D",
      "outputCharacters": 26,
      "charactersPerSecond": 0.7311986821202101
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6578.7065839977,
      "inferenceMs": 6578.569291043095,
      "tokensPerSecond": 1.672096091619314,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6578.7065839977,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.672096091619314,
        "grade": "C"
      },
      "observedPeakRssBytes": 35531505664,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.624323217264965
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6543.994582956657,
      "inferenceMs": 6543.892500048969,
      "tokensPerSecond": 1.6809567088575623,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6543.994582956657,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6809567088575623,
        "grade": "C"
      },
      "observedPeakRssBytes": 35534716928,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.654127111611801
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6534.671999979764,
      "inferenceMs": 6534.565916052088,
      "tokensPerSecond": 1.6833558864221758,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6534.671999979764,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6833558864221758,
        "grade": "C"
      },
      "observedPeakRssBytes": 35534716928,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.6621970725109545
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6566.134915978182,
      "inferenceMs": 6566.039333003573,
      "tokensPerSecond": 1.6752869488170052,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6566.134915978182,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6752869488170052,
        "grade": "C"
      },
      "observedPeakRssBytes": 35534733312,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.63505610056629
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6525.719166966155,
      "inferenceMs": 6525.6216249545105,
      "tokensPerSecond": 1.6856631647068074,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6525.719166966155,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6856631647068074,
        "grade": "C"
      },
      "observedPeakRssBytes": 35534733312,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.669957917650171
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.007 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.012 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6572.640958009288,
      "inferenceMs": 6572.546250012238,
      "tokensPerSecond": 1.673628390211711,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6572.640958009288,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.673628390211711,
        "grade": "C"
      },
      "observedPeakRssBytes": 35534749696,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.629477312530301
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 15610.001625027508,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.004 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6552.596584020648,
      "inferenceMs": 6552.472041978035,
      "tokensPerSecond": 1.67875573211593,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 22162.598209048156,
      "processAndInitWallMs": 15710.861333005596,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.67875573211593,
        "grade": "C"
      },
      "observedPeakRssBytes": 35223830528,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.646723826208128
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.004 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6537.497500015888,
      "inferenceMs": 6537.397958978545,
      "tokensPerSecond": 1.6826266458036965,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6537.497500015888,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6826266458036965,
        "grade": "C"
      },
      "observedPeakRssBytes": 35227140096,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.659744172248797
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.004 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6556.452957971487,
      "inferenceMs": 6556.355791981332,
      "tokensPerSecond": 1.6777612974349883,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6556.452957971487,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6777612974349883,
        "grade": "C"
      },
      "observedPeakRssBytes": 35230334976,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.643378909554052
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.004 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6534.330042020883,
      "inferenceMs": 6534.23816699069,
      "tokensPerSecond": 1.6834403214087303,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6534.330042020883,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6834403214087303,
        "grade": "C"
      },
      "observedPeakRssBytes": 35233529856,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.662481081102093
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.004 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6553.418290975969,
      "inferenceMs": 6553.290333016776,
      "tokensPerSecond": 1.678546110582011,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6553.418290975969,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.678546110582011,
        "grade": "C"
      },
      "observedPeakRssBytes": 35233562624,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.646018735594037
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.004 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6544.112917035818,
      "inferenceMs": 6544.00033398997,
      "tokensPerSecond": 1.6809290095639624,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6544.112917035818,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6809290095639624,
        "grade": "C"
      },
      "observedPeakRssBytes": 35233562624,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.654033941260601
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 15650.095625023823,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6581.056124996394,
      "inferenceMs": 6580.926374997944,
      "tokensPerSecond": 1.671497198599709,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 22231.151750020217,
      "processAndInitWallMs": 15756.894916994497,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.671497198599709,
        "grade": "C"
      },
      "observedPeakRssBytes": 35223961600,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.622308758926294
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6526.783832989167,
      "inferenceMs": 6526.688374986406,
      "tokensPerSecond": 1.685387652665876,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6526.783832989167,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.685387652665876,
        "grade": "C"
      },
      "observedPeakRssBytes": 35227238400,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.669031195330674
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6568.672249966767,
      "inferenceMs": 6568.572666030377,
      "tokensPerSecond": 1.6746408328382996,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6568.672249966767,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6746408328382996,
        "grade": "C"
      },
      "observedPeakRssBytes": 35230433280,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.632882801365189
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6566.041917016264,
      "inferenceMs": 6565.9467499936,
      "tokensPerSecond": 1.6753105711694543,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6566.041917016264,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6753105711694543,
        "grade": "C"
      },
      "observedPeakRssBytes": 35233611776,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.635135557569982
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6559.85833297018,
      "inferenceMs": 6559.759541996755,
      "tokensPerSecond": 1.6768907350301534,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6559.85833297018,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6768907350301534,
        "grade": "C"
      },
      "observedPeakRssBytes": 35233660928,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.640450654192334
    }
  },
  {
    "precision": "q4",
    "device": "cpu",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "cpu",
      "precision": "q4",
      "provider": "llama.cpp CPU",
      "acceleration": "cpu",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nload_tensors: offloading 0 repeating layers to GPU\nload_tensors: offloaded 0/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size = 15773.89 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nclip_ctx: CLIP using CPU backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 0,
      "totalLayers": 63,
      "imageEncoderUseGpu": false,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 6613.030667009298,
      "inferenceMs": 6612.940749968402,
      "tokensPerSecond": 1.6634051953440774,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 6613.030667009298,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.6634051953440774,
        "grade": "C"
      },
      "observedPeakRssBytes": 35233660928,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 37,
      "charactersPerSecond": 5.595090202520987
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 1308.1281250342727,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1522.6184169878252,
      "inferenceMs": 1522.496499994304,
      "tokensPerSecond": 7.224975558263124,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 2830.746542022098,
      "processAndInitWallMs": 1409.2495419899933,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.224975558263124,
        "grade": "B"
      },
      "observedPeakRssBytes": 18675580928,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 24.302190514157783
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1914.3679999979213,
      "inferenceMs": 1914.2458330024965,
      "tokensPerSecond": 8.880782032752546,
      "inputTokens": 91,
      "outputTokens": 17,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1914.3679999979213,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 8.880782032752546,
        "grade": "B"
      },
      "observedPeakRssBytes": 18681741312,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 50,
      "charactersPerSecond": 26.119947155154545
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1260.48204198014,
      "inferenceMs": 1260.326666990295,
      "tokensPerSecond": 6.347560683695034,
      "inputTokens": 84,
      "outputTokens": 8,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1260.48204198014,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 6.347560683695034,
        "grade": "B"
      },
      "observedPeakRssBytes": 18684936192,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 34,
      "charactersPerSecond": 26.977132905703893
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1542.663583997637,
      "inferenceMs": 1542.5692080170847,
      "tokensPerSecond": 7.779229572088732,
      "inputTokens": 83,
      "outputTokens": 12,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1542.663583997637,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.779229572088732,
        "grade": "B"
      },
      "observedPeakRssBytes": 18685018112,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 43,
      "charactersPerSecond": 27.875572633317955
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.8772489985666848,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1969.7164589888416,
      "inferenceMs": 1969.6094170212746,
      "tokensPerSecond": 9.138867759488162,
      "inputTokens": 86,
      "outputTokens": 18,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1969.7164589888416,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 9.138867759488162,
        "grade": "B"
      },
      "observedPeakRssBytes": 18691391488,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 59,
      "charactersPerSecond": 29.955177656100087
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 0.9726275669616796,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 4481.337374949362,
      "inferenceMs": 4475.781124958303,
      "tokensPerSecond": 0.8936987507487343,
      "inputTokens": 371,
      "outputTokens": 4,
      "maxNewTokens": 64,
      "inputModality": "image",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 4481.337374949362,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 0.8936987507487343,
        "grade": "D"
      },
      "observedPeakRssBytes": 18767233024,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "D",
      "outputCharacters": 11,
      "charactersPerSecond": 2.4576715645590195
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 4877.424417005386,
      "inferenceMs": 4875.6625420064665,
      "tokensPerSecond": 1.8459029767667754,
      "inputTokens": 371,
      "outputTokens": 9,
      "maxNewTokens": 64,
      "inputModality": "image",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 4877.424417005386,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 1.8459029767667754,
        "grade": "C"
      },
      "observedPeakRssBytes": 18789384192,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "C",
      "outputCharacters": 26,
      "charactersPerSecond": 5.332608599548463
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1488.4135000174865,
      "inferenceMs": 1488.3152910042554,
      "tokensPerSecond": 7.390907065516771,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1488.4135000174865,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.390907065516771,
        "grade": "B"
      },
      "observedPeakRssBytes": 18794954752,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 24.860323765829136
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1486.2614999874495,
      "inferenceMs": 1486.1637919675559,
      "tokensPerSecond": 7.401606780795625,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1486.2614999874495,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.401606780795625,
        "grade": "B"
      },
      "observedPeakRssBytes": 18798166016,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 24.89631371722165
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1496.6248330310918,
      "inferenceMs": 1496.5173339587636,
      "tokensPerSecond": 7.350399324076993,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1496.6248330310918,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.350399324076993,
        "grade": "B"
      },
      "observedPeakRssBytes": 18798166016,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 24.724070453713523
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1490.628875035327,
      "inferenceMs": 1490.5314169591293,
      "tokensPerSecond": 7.3799182458303205,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1490.628875035327,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.3799182458303205,
        "grade": "B"
      },
      "observedPeakRssBytes": 18798182400,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 24.82336137233835
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1490.8686659764498,
      "inferenceMs": 1490.770792006515,
      "tokensPerSecond": 7.378733242549286,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1490.8686659764498,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.378733242549286,
        "grade": "B"
      },
      "observedPeakRssBytes": 18798182400,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 24.819375452211236
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1497.307334037032,
      "inferenceMs": 1497.210500005167,
      "tokensPerSecond": 7.346996297422465,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1497.307334037032,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.346996297422465,
        "grade": "B"
      },
      "observedPeakRssBytes": 18798198784,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 24.71262390951193
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 1392.8450829698704,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.004 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.014 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1482.280374970287,
      "inferenceMs": 1482.1459160302766,
      "tokensPerSecond": 7.421671429937198,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 2875.1254579401575,
      "processAndInitWallMs": 1504.0638750069775,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.421671429937198,
        "grade": "B"
      },
      "observedPeakRssBytes": 18678284288,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 24.963803900697847
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.004 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.014 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1475.2059160382487,
      "inferenceMs": 1475.105916033499,
      "tokensPerSecond": 7.4570916436824835,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1475.2059160382487,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.4570916436824835,
        "grade": "B"
      },
      "observedPeakRssBytes": 18681266176,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 25.082944619659262
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.004 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.014 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1479.315874981694,
      "inferenceMs": 1479.2226669960655,
      "tokensPerSecond": 7.436338183174459,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1479.315874981694,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.436338183174459,
        "grade": "B"
      },
      "observedPeakRssBytes": 18684411904,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 25.013137525223183
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.004 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.014 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1486.4379999926314,
      "inferenceMs": 1486.3432919955812,
      "tokensPerSecond": 7.40071291688697,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1486.4379999926314,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.40071291688697,
        "grade": "B"
      },
      "observedPeakRssBytes": 18687574016,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 24.893307084074355
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.004 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.014 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1463.5885420138948,
      "inferenceMs": 1463.4912080364302,
      "tokensPerSecond": 7.51627337396767,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1463.5885420138948,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.51627337396767,
        "grade": "B"
      },
      "observedPeakRssBytes": 18687574016,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 25.282010439709435
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.004 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.014 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1471.5414999518543,
      "inferenceMs": 1471.443333022762,
      "tokensPerSecond": 7.475653158455568,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1471.5414999518543,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.475653158455568,
        "grade": "B"
      },
      "observedPeakRssBytes": 18687590400,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 25.145378805714184
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 1350.2731249900535,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1486.625040997751,
      "inferenceMs": 1486.4965840242803,
      "tokensPerSecond": 7.399949732962405,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 2836.8981659878045,
      "processAndInitWallMs": 1455.829459009692,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.399949732962405,
        "grade": "B"
      },
      "observedPeakRssBytes": 18672779264,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 24.890740010873543
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1475.9553329786286,
      "inferenceMs": 1475.8428339846432,
      "tokensPerSecond": 7.453368168140903,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1475.9553329786286,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.453368168140903,
        "grade": "B"
      },
      "observedPeakRssBytes": 18675761152,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 25.070420201928496
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1471.4906250010245,
      "inferenceMs": 1471.3910840218887,
      "tokensPerSecond": 7.475918618408838,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1471.4906250010245,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.475918618408838,
        "grade": "B"
      },
      "observedPeakRssBytes": 18678841344,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 25.14627171646609
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1463.9222080004402,
      "inferenceMs": 1463.8203749782406,
      "tokensPerSecond": 7.514583201619606,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1463.9222080004402,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.514583201619606,
        "grade": "B"
      },
      "observedPeakRssBytes": 18682150912,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 25.276325314538678
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1479.7302079969086,
      "inferenceMs": 1479.6328750089742,
      "tokensPerSecond": 7.434276559942806,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1479.7302079969086,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.434276559942806,
        "grade": "B"
      },
      "observedPeakRssBytes": 18682150912,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 25.006202974353073
    }
  },
  {
    "precision": "q4",
    "device": "mps",
    "status": "pass",
    "quality": {
      "passed": true,
      "score": 1,
      "threshold": 0.55,
      "metric": "character-ngram F2 (orders1–6), best authored reference",
      "anchorsPassed": true,
      "repetitionRejected": false,
      "repeatParity": true
    },
    "runtime": {
      "engine": "llama.cpp",
      "version": "0.3.36",
      "loadMs": 0,
      "device": "mps",
      "precision": "q4",
      "provider": "llama.cpp Metal",
      "acceleration": "partial",
      "placement": "ggml_metal_device_init: tensor API disabled for pre-M5 and pre-A19 devices\nggml_metal_library_init: using embedded metal library\nggml_metal_library_compile_all: compiled 'fa_aux' library in 0.002 sec\nggml_metal_library_compile_all: compiled 'fa_f16' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_f32' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q4_0' library in 0.003 sec\nggml_metal_library_compile_all: compiled 'fa_q4_1' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_q5_0' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_q5_1' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_q8_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f16' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'fa_vec_f32' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_0' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q4_1' library in 0.012 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_0' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q5_1' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'fa_vec_q8_0' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'mul_mv' library in 0.006 sec\nggml_metal_library_compile_all: compiled 'mul_mm' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'quantize' library in 0.013 sec\nggml_metal_library_compile_all: compiled 'softmax' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'norm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'unary' library in 0.004 sec\nggml_metal_library_compile_all: compiled 'binbcast' library in 0.011 sec\nggml_metal_library_compile_all: compiled 'reduce' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'tri' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'ssm' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'wkv' library in 0.008 sec\nggml_metal_library_compile_all: compiled 'gated_delta_net' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'solve_tri' library in 0.007 sec\nggml_metal_library_compile_all: compiled 'rope' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'conv' library in 0.010 sec\nggml_metal_library_compile_all: compiled 'upscale' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'argsort' library in 0.009 sec\nggml_metal_library_compile_all: compiled 'pool' library in 0.005 sec\nggml_metal_library_compile_all: compiled 'misc' library in 0.005 sec\nggml_metal_library_compile_all: loaded 34 libraries from embedded data in 0.013 sec (max single = 0.013 sec)\nggml_metal_rsets_init: creating a residency set collection (keep_alive = 180 s)\nggml_metal_device_init: GPU name:   MTL0 (Apple M3 Max)\nggml_metal_device_init: GPU family: MTLGPUFamilyApple9  (1009)\nggml_metal_device_init: GPU family: MTLGPUFamilyMetal4  (5002)\nggml_metal_device_init: simdgroup reduction   = true\nggml_metal_device_init: simdgroup matrix mul. = true\nggml_metal_device_init: has unified memory    = true\nggml_metal_device_init: has bfloat            = true\nggml_metal_device_init: has tensor            = false\nggml_metal_device_init: use residency sets    = true\nggml_metal_device_init: use shared buffers    = true\nggml_metal_device_init: recommendedMaxWorkingSetSize  = 83494.17 MB\nggml_metal_log_allocated_size: allocated buffer, size = 15773.91 MiB, (15774.36 / 79626.25)\nload_tensors: offloading output layer to GPU\nload_tensors: offloading 61 repeating layers to GPU\nload_tensors: offloaded 63/63 layers to GPU\nload_tensors:   CPU_Mapped model buffer size =  1102.77 MiB\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nllama_context: enumerating backends\nllama_context: backend_ptrs.size() = 3\nggml_metal_init: allocating\nggml_metal_init: found device: Apple M3 Max\nggml_metal_init: picking default device: Apple M3 Max\nggml_metal_init: use fusion         = true\nggml_metal_init: use concurrency    = true\nggml_metal_init: use graph optimize = true\nclip_ctx: CLIP using MTL0 backend",
      "parameterDtype": "Q4_K_M language / F16 vision projector",
      "accelerationDetails": "Native layer-offload logs and strict n_gpu_layers selection; native mtmd image encoder requestedGPU only forMPS, CPU forCPU. PIL orientation/alpha and tokenization onCPU.",
      "offloadedLayers": 63,
      "totalLayers": 63,
      "imageEncoderUseGpu": true,
      "modelQuantization": {
        "language": "Q4_K_M",
        "vision": "F16"
      },
      "cpuThreads": 4,
      "cpuThreadsApplied": true,
      "durationMs": 1471.6865830123425,
      "inferenceMs": 1471.5864580357447,
      "tokensPerSecond": 7.474926083977874,
      "inputTokens": 84,
      "outputTokens": 11,
      "maxNewTokens": 128,
      "inputModality": "text",
      "decoding": "greedy",
      "outputLimitReached": false,
      "totalMs": 1471.6865830123425,
      "processAndInitWallMs": null,
      "artifactRevision": "22fb50537007ccd41c41a0f0f0482e0fdb3335ab",
      "fallback": false,
      "performance": {
        "metric": "tokensPerSecond",
        "value": 7.474926083977874,
        "grade": "B"
      },
      "observedPeakRssBytes": 18682167296,
      "memoryMeasurementScope": "500ms sampled child RSS, not OS peak/minimumRAM",
      "grade": "B",
      "outputCharacters": 37,
      "charactersPerSecond": 25.14293319156194
    }
  }
]

Something not quite right?

Tell us what happened with TranslateGemma. Loading, results, or the page itself—we’re here for all of it.

Attachments Optional

Up to 3 files · 50 MB each

Browser, OS, CPU/GPU, memory, screen, storage, and network details are included automatically. Model inputs, results, and browsing history are not collected.