HY-MT 1.5 Desktop
Translate short multilingual text locally with the original HY-MT1.5 1.8B checkpoint and official source-aware translation prompt.
Desktop app
Run HY-MT 1.5 Desktop on your computer
Download the desktop app to load this model, provide input and view results locally.
Public desktop downloads are coming soon.Device requirements
If your device falls short
MemoryAdvisory only: close other apps, reduce image/tile/token/batch settings or choose a smaller verified model/precision. CPU can need more RAM than GPU.
GPUIf the selected GPU runtime fails, existing CPU fallback may recover when enabled, but CPU has its own memory/runtime needs.
PlatformUse a compatible OS/architecture/runtime or another model. These estimates do not add hardware detection or block runs.
Benchmark evidence
Saved desktop measurements apply to the recorded hardware and runtime. Fallback cases report the device that actually ran; they do not prove GPU support on the requested device.
HY-MT1.5 1.8B Desktop
165 configurations testedQuality criteria
{ "minimumCharacterF2": 0.55, "requireSemanticAnchors": true, "rejectRepetitionAndTruncation": true, "frozenPolicy": "research/hy-mt15-family-expansion/quality-gates.json" }
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 224 ms / 5024 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 224 ms / 7881 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 225 ms / 5071 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 225 ms / 5074 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 237 ms / 5506 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 241 ms / 5063 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 248 ms / 11266 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 252 ms / 5157 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 293 ms / 5071 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 293 ms / 5071 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 294 ms / 7881 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 294 ms / 7881 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 295 ms / 5024 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 295 ms / 5071 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 295 ms / 5024 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 295 ms / 5157 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 296 ms / 5157 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 299 ms / 7881 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 299 ms / 5157 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 300 ms / 5024 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 301 ms / 5074 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 301 ms / 5063 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 316 ms / 5074 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 324 ms / 5063 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 326 ms / 5074 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 334 ms / 5071 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 335 ms / 5074 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 336 ms / 5533 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 340 ms / 5024 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 340 ms / 5063 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 342 ms / 5676 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 346 ms / 5063 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 346 ms / 7881 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 357 ms / 6149 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 364 ms / 5063 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 365 ms / 5071 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 367 ms / 5071 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 369 ms / 5506 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 371 ms / 5157 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 372 ms / 7881 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 373 ms / 5157 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 376 ms / 5063 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 378 ms / 5208 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 379 ms / 5024 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 379 ms / 11266 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 380 ms / 5157 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 384 ms / 5024 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 385 ms / 7881 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 388 ms / 5074 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 398 ms / 5506 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 400 ms / 5506 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 403 ms / 5074 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 414 ms / 11266 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 430 ms / 11266 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 479 ms / 5533 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 479 ms / 5676 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 479 ms / 5533 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 483 ms / 5533 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 487 ms / 5208 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 502 ms / 5208 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 508 ms / 5676 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 512 ms / 5676 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 515 ms / 5208 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 525 ms / 5208 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 526 ms / 5533 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 528 ms / 5676 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 534 ms / 5071 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 538 ms / 5024 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 549 ms / 5024 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 551 ms / 5071 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 555 ms / 6149 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 562 ms / 7881 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 571 ms / 5074 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 582 ms / 5208 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 584 ms / 5063 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 585 ms / 5533 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 595 ms / 5063 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 599 ms / 5533 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 600 ms / 5074 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 601 ms / 5533 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 603 ms / 5157 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 606 ms / 6149 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 607 ms / 5676 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 615 ms / 5208 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 623 ms / 6149 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 625 ms / 5506 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 626 ms / 5676 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 631 ms / 5208 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 633 ms / 7881 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 642 ms / 5157 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 655 ms / 6149 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 656 ms / 11266 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 688 ms / 5676 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 707 ms / 5506 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 715 ms / 5533 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 745 ms / 5676 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 761 ms / 6149 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 769 ms / 5208 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 832 ms / 11266 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 684 ms / 4764 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 705 ms / 5158 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 714 ms / 8264 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 719 ms / 7025 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 741 ms / 5279 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 779 ms / 4773 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 798 ms / 5507 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 816 ms / 4703 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1121 ms / 4764 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1123 ms / 4773 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1143 ms / 7025 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1144 ms / 5158 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1145 ms / 5158 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1148 ms / 7025 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1148 ms / 5158 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1150 ms / 7025 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1151 ms / 7025 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1152 ms / 7025 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1153 ms / 4703 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1163 ms / 4773 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1172 ms / 4773 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1177 ms / 4773 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1189 ms / 8264 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1191 ms / 5158 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1205 ms / 5158 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1211 ms / 8264 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1221 ms / 4764 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1230 ms / 4764 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1232 ms / 4773 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1274 ms / 4764 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1280 ms / 4773 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1293 ms / 5279 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1294 ms / 4703 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1296 ms / 5158 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1301 ms / 7025 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1327 ms / 8264 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1328 ms / 8264 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1337 ms / 4703 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1343 ms / 5279 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1358 ms / 8264 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1363 ms / 5507 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1365 ms / 4764 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1366 ms / 4764 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1370 ms / 4703 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1388 ms / 8264 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1418 ms / 4773 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1421 ms / 5507 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1431 ms / 5158 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1460 ms / 4703 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1464 ms / 5507 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1480 ms / 8264 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1541 ms / 7025 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1558 ms / 4764 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1626 ms / 4703 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1643 ms / 5279 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1662 ms / 7025 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1666 ms / 5507 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1667 ms / 4764 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1678 ms / 4773 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1682 ms / 5279 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1683 ms / 5158 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1685 ms / 4703 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1735 ms / 8264 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 1937 ms / 5507 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 2006 ms / 4703 ms |
FP32 | cpu | Passed | Fast | macos · aarch64 | 2038 ms / 5279 ms |
Test setup & sample details
- Quality gate
- { "minimumCharacterF2": 0.55, "requireSemanticAnchors": true, "rejectRepetitionAndTruncation": true, "frozenPolicy": "research/hy-mt15-family-expansion/quality-gates.json" }
- Tested
- 2026-11-13T09:50:29.956212+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2", "runtime": "App-owned CPython3.13.18; Transformers5.22.1/Torch2.10.0; MPS op fallback disabled", "concurrentUploadActivity": "none; uploads excluded", "transport": "Verified exact local original snapshots, not Internet transfer timing" }
- Runtime
- transformers 5.22.1
- Artifact revision
- dbad03788f49709801014c95d481a514c272ca52
- Fixture
{ "file": "model-samples/hunyuan-translation.json", "url": "https://assets.localai.tools/samples/v1/files/b63cba0b5f61f8423d4234d4d28e9cdde538b8345b4ccdc8b2044b1bc88e2fa8/hunyuan-translation.json", "size": 3099, "sha1": "dd6e22e1c2966dd5a249b11eb2fb48b076505088", "sha256": "b63cba0b5f61f8423d4234d4d28e9cdde538b8345b4ccdc8b2044b1bc88e2fa8" }- Rerun when
- model artifact revision change; runtime version change; fixture or quality-gate change; backend/platform behavior change
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8263.978708011564,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1387.271250016056,
"inferenceMs": 1388.024875021074,
"tokensPerSecond": 10.117697432306798,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9652.003583032638,
"residentMemoryBytes": 7643840512,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8263.978708011564,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1734.4194170436822,
"inferenceMs": 1735.2955000242218,
"tokensPerSecond": 12.110392280955145,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9999.274208035786,
"residentMemoryBytes": 7646429184,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8263.978708011564,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 712.7875829464756,
"inferenceMs": 713.5959169827402,
"tokensPerSecond": 11.230421991009774,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 8977.574624994304,
"residentMemoryBytes": 7646609408,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8263.978708011564,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1210.2975830202922,
"inferenceMs": 1211.079917033203,
"tokensPerSecond": 11.570203307373081,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9475.058625044767,
"residentMemoryBytes": 7646642176,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8263.978708011564,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1479.0292080142535,
"inferenceMs": 1479.877415986266,
"tokensPerSecond": 12.172845618601855,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9743.85612399783,
"residentMemoryBytes": 7646674944,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8263.978708011564,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1326.3028329820372,
"inferenceMs": 1327.0972919999622,
"tokensPerSecond": 12.066958418006541,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9591.076000011526,
"residentMemoryBytes": 7646691328,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8263.978708011564,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1187.947541999165,
"inferenceMs": 1188.7292079627514,
"tokensPerSecond": 11.787773356206403,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9452.707915974315,
"residentMemoryBytes": 7648116736,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8263.978708011564,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1356.8150830105878,
"inferenceMs": 1357.6546250260435,
"tokensPerSecond": 10.320433173374267,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9621.633333037607,
"residentMemoryBytes": 7649722368,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 8263.978708011564,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1326.8528750049882,
"inferenceMs": 1327.6774999685585,
"tokensPerSecond": 10.553857986467804,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9591.656207980122,
"residentMemoryBytes": 7649722368,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4764.446957968175,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1220.6698340014555,
"inferenceMs": 1221.3670830242336,
"tokensPerSecond": 11.49945068223331,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5985.814040992409,
"residentMemoryBytes": 7645315072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4764.446957968175,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1666.2329169921577,
"inferenceMs": 1667.0011659734882,
"tokensPerSecond": 12.605575446357383,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6431.448123941664,
"residentMemoryBytes": 7646412800,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4764.446957968175,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 683.5750839672983,
"inferenceMs": 684.324000030756,
"tokensPerSecond": 11.707565490660604,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5448.770957998931,
"residentMemoryBytes": 7646461952,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4764.446957968175,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1228.9830839727074,
"inferenceMs": 1229.8393750097603,
"tokensPerSecond": 11.39406871920328,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5994.286332977936,
"residentMemoryBytes": 7646445568,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4764.446957968175,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1556.8414999870583,
"inferenceMs": 1557.7023340156302,
"tokensPerSecond": 11.564255016329199,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6322.149291983806,
"residentMemoryBytes": 7650033664,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4764.446957968175,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1364.224709046539,
"inferenceMs": 1365.3255419922061,
"tokensPerSecond": 11.730892954797708,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6129.7724999603815,
"residentMemoryBytes": 7650672640,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4764.446957968175,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1365.5403750017285,
"inferenceMs": 1366.4232079754584,
"tokensPerSecond": 10.25508423272915,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6130.870165943634,
"residentMemoryBytes": 7650787328,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4764.446957968175,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1273.0131249991246,
"inferenceMs": 1273.801208997611,
"tokensPerSecond": 11.000298020063758,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6038.248166965786,
"residentMemoryBytes": 7650787328,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4764.446957968175,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1119.8608329868875,
"inferenceMs": 1120.5989169538952,
"tokensPerSecond": 12.504482634121628,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5885.045874922071,
"residentMemoryBytes": 7650787328,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4773.289375007153,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1122.5874170195311,
"inferenceMs": 1123.2837090501562,
"tokensPerSecond": 12.50581557289394,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5896.573084057309,
"residentMemoryBytes": 7641219072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4773.289375007153,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1676.7585829948075,
"inferenceMs": 1677.5338339502923,
"tokensPerSecond": 12.526364701740365,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6450.823208957445,
"residentMemoryBytes": 7644545024,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4773.289375007153,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 778.050999972038,
"inferenceMs": 778.8369169575162,
"tokensPerSecond": 10.285611608682006,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5552.126291964669,
"residentMemoryBytes": 7647625216,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4773.289375007153,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1162.605417019222,
"inferenceMs": 1163.382124970667,
"tokensPerSecond": 12.044827837910708,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5936.67149997782,
"residentMemoryBytes": 7647625216,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4773.289375007153,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1417.7510830340907,
"inferenceMs": 1418.4957920224406,
"tokensPerSecond": 12.698674358658712,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6191.785167029593,
"residentMemoryBytes": 7647690752,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4773.289375007153,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1279.326832969673,
"inferenceMs": 1280.0621669739485,
"tokensPerSecond": 12.509123675777886,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6053.351541981101,
"residentMemoryBytes": 7647690752,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4773.289375007153,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1176.3042910024524,
"inferenceMs": 1177.1112089627422,
"tokensPerSecond": 11.904139781843815,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5950.400583969895,
"residentMemoryBytes": 7647723520,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4773.289375007153,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1231.083624996245,
"inferenceMs": 1231.8806250113994,
"tokensPerSecond": 11.374900081368686,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6005.170000018552,
"residentMemoryBytes": 7647789056,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4773.289375007153,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1171.1564169963822,
"inferenceMs": 1171.9409169745632,
"tokensPerSecond": 11.956896249935811,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5945.230291981716,
"residentMemoryBytes": 7647789056,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4703.418584016617,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1152.4710000376217,
"inferenceMs": 1153.209749958478,
"tokensPerSecond": 12.181029677660005,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5856.628333975095,
"residentMemoryBytes": 7644004352,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4703.418584016617,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2005.2745830034837,
"inferenceMs": 2006.3368330011144,
"tokensPerSecond": 10.474389897003107,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6709.755417017732,
"residentMemoryBytes": 7649804288,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4703.418584016617,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 814.7995830513537,
"inferenceMs": 815.6343330047093,
"tokensPerSecond": 9.822601255010614,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5519.052917021327,
"residentMemoryBytes": 7649968128,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4703.418584016617,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1336.2069579889067,
"inferenceMs": 1337.1341250021942,
"tokensPerSecond": 10.479879938276742,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6040.552709018812,
"residentMemoryBytes": 7649951744,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4703.418584016617,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1624.6733749867417,
"inferenceMs": 1625.6098339799792,
"tokensPerSecond": 11.082657359953087,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6329.0284179965965,
"residentMemoryBytes": 7649968128,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4703.418584016617,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1683.774542005267,
"inferenceMs": 1685.0952500244603,
"tokensPerSecond": 9.504622536810956,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6388.513834041078,
"residentMemoryBytes": 7649984512,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4703.418584016617,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1459.1619999846444,
"inferenceMs": 1460.051374975592,
"tokensPerSecond": 9.596808924024284,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6163.469958992209,
"residentMemoryBytes": 7650017280,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4703.418584016617,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1293.5474169789813,
"inferenceMs": 1294.3749169935472,
"tokensPerSecond": 10.825791365203733,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5997.793501010165,
"residentMemoryBytes": 7650017280,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 4703.418584016617,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1369.1406659781933,
"inferenceMs": 1369.9909169808961,
"tokensPerSecond": 10.227605311559993,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6073.4095009975135,
"residentMemoryBytes": 7650017280,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7024.655959045049,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1151.3842080021277,
"inferenceMs": 1152.069333998952,
"tokensPerSecond": 12.193826527036636,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 8176.725293044001,
"residentMemoryBytes": 7646937088,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7024.655959045049,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1660.8542089816183,
"inferenceMs": 1661.6177090327255,
"tokensPerSecond": 12.646347919099275,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 8686.273668077774,
"residentMemoryBytes": 7649427456,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7024.655959045049,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 718.4776659705676,
"inferenceMs": 719.2265000194311,
"tokensPerSecond": 11.13861794331888,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 7743.88245906448,
"residentMemoryBytes": 7649460224,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7024.655959045049,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1146.771208033897,
"inferenceMs": 1147.5826670066454,
"tokensPerSecond": 12.211149968049083,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 8172.238626051694,
"residentMemoryBytes": 7649525760,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7024.655959045049,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1540.0422499515116,
"inferenceMs": 1540.8637499785982,
"tokensPerSecond": 11.690325544116302,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 8565.519709023647,
"residentMemoryBytes": 7649525760,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7024.655959045049,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1300.3304169978946,
"inferenceMs": 1301.1392499902286,
"tokensPerSecond": 12.30733058558182,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 8325.795209035277,
"residentMemoryBytes": 7649525760,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7024.655959045049,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1141.7172910296358,
"inferenceMs": 1142.518165986985,
"tokensPerSecond": 12.264918740441033,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 8167.174125032034,
"residentMemoryBytes": 7649525760,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7024.655959045049,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1150.2280000131577,
"inferenceMs": 1150.9865829721093,
"tokensPerSecond": 12.174488240265118,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 8175.642542017158,
"residentMemoryBytes": 7649591296,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7024.655959045049,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1148.7836670130491,
"inferenceMs": 1149.5374589576386,
"tokensPerSecond": 12.189531978372779,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 8174.193418002687,
"residentMemoryBytes": 7649591296,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5157.901541970205,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1190.6079579493962,
"inferenceMs": 1191.3031250005588,
"tokensPerSecond": 11.79565543180523,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6349.204666970763,
"residentMemoryBytes": 7645626368,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5157.901541970205,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1682.2435419890098,
"inferenceMs": 1683.0069580464624,
"tokensPerSecond": 12.48554377687322,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6840.908500016667,
"residentMemoryBytes": 7649492992,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5157.901541970205,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 704.7104589873925,
"inferenceMs": 705.466957995668,
"tokensPerSecond": 11.356451214522684,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 5863.368499965873,
"residentMemoryBytes": 7649492992,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5157.901541970205,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1143.7126249657013,
"inferenceMs": 1144.4560409872793,
"tokensPerSecond": 12.243675826296593,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6302.357582957484,
"residentMemoryBytes": 7649542144,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5157.901541970205,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1430.274999991525,
"inferenceMs": 1431.0327499988489,
"tokensPerSecond": 12.587652096132437,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6588.934291969053,
"residentMemoryBytes": 7649542144,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5157.901541970205,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1295.199457963463,
"inferenceMs": 1295.9629999822937,
"tokensPerSecond": 12.355795558179524,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6453.864541952498,
"residentMemoryBytes": 7649542144,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5157.901541970205,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1147.1890829852782,
"inferenceMs": 1147.9585419874638,
"tokensPerSecond": 12.206668928285149,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6305.860083957668,
"residentMemoryBytes": 7649542144,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5157.901541970205,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1143.9718329929747,
"inferenceMs": 1144.7157500078902,
"tokensPerSecond": 12.241006617950555,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6302.617291978095,
"residentMemoryBytes": 7649542144,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5157.901541970205,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1203.9907090365887,
"inferenceMs": 1204.7665840364061,
"tokensPerSecond": 11.63032713766157,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6362.668126006611,
"residentMemoryBytes": 7649542144,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7880.543457984459,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 632.3576249997132,
"inferenceMs": 633.113416028209,
"tokensPerSecond": 22.309835567945015,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4365647872,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8513.656874012668,
"residentMemoryBytes": 618708992,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7880.543457984459,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 561.3809159840457,
"inferenceMs": 562.1828750008717,
"tokensPerSecond": 37.512735906480316,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366237696,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8442.72633298533,
"residentMemoryBytes": 678658048,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7880.543457984459,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 223.46566698979586,
"inferenceMs": 224.1855000029318,
"tokensPerSecond": 36.02621127090479,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366385152,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8104.7289579873905,
"residentMemoryBytes": 691814400,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7880.543457984459,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 345.6239169463515,
"inferenceMs": 346.4380000368692,
"tokensPerSecond": 40.69255330941452,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8226.981458021328,
"residentMemoryBytes": 708771840,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7880.543457984459,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 371.51108297985047,
"inferenceMs": 372.3467909730971,
"tokensPerSecond": 48.66557742610281,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8252.890248957556,
"residentMemoryBytes": 710148096,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7880.543457984459,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 384.365624981001,
"inferenceMs": 385.1803339784965,
"tokensPerSecond": 41.80268421553306,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8265.723791962955,
"residentMemoryBytes": 727351296,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7880.543457984459,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 298.001250019297,
"inferenceMs": 298.8052080036141,
"tokensPerSecond": 47.22712146324092,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8179.348665988073,
"residentMemoryBytes": 728399872,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7880.543457984459,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 293.61837496981025,
"inferenceMs": 294.35687500517815,
"tokensPerSecond": 47.90236813120964,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8174.900332989637,
"residentMemoryBytes": 729448448,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7880.543457984459,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 293.51229197345674,
"inferenceMs": 294.2489159759134,
"tokensPerSecond": 47.93178622113254,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8174.792373960372,
"residentMemoryBytes": 730480640,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5024.133917002473,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 537.315750028938,
"inferenceMs": 538.0639170180075,
"tokensPerSecond": 26.274511229104654,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4365647872,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5562.1978340204805,
"residentMemoryBytes": 620658688,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5024.133917002473,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 548.2407920062542,
"inferenceMs": 549.0343749988824,
"tokensPerSecond": 38.41000271798246,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366237696,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5573.168292001355,
"residentMemoryBytes": 680722432,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5024.133917002473,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 223.20000000763685,
"inferenceMs": 223.9007079624571,
"tokensPerSecond": 36.07069867416962,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366385152,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5248.03462496493,
"residentMemoryBytes": 694075392,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5024.133917002473,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 339.01808300288394,
"inferenceMs": 339.8197910282761,
"tokensPerSecond": 41.50159690628121,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5363.953708030749,
"residentMemoryBytes": 711049216,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5024.133917002473,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 382.68066599266604,
"inferenceMs": 383.5970830405131,
"tokensPerSecond": 47.23279258632707,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5407.731000042986,
"residentMemoryBytes": 712605696,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5024.133917002473,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 377.7784169651568,
"inferenceMs": 378.5880830255337,
"tokensPerSecond": 42.472999298514324,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5402.722000028007,
"residentMemoryBytes": 729792512,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5024.133917002473,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 294.24187500262633,
"inferenceMs": 294.98245799914,
"tokensPerSecond": 47.81271073594829,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5319.116375001613,
"residentMemoryBytes": 730890240,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5024.133917002473,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 294.21795799862593,
"inferenceMs": 294.9019999941811,
"tokensPerSecond": 47.80346623780305,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5319.035916996654,
"residentMemoryBytes": 732004352,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5024.133917002473,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 298.8083749660291,
"inferenceMs": 299.5647080242634,
"tokensPerSecond": 47.084805960124534,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5323.698625026736,
"residentMemoryBytes": 733184000,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5071.107458032202,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 533.1033749971539,
"inferenceMs": 533.830999978818,
"tokensPerSecond": 26.479776190249822,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4365647872,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5604.93845801102,
"residentMemoryBytes": 627965952,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5071.107458032202,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 550.4545000148937,
"inferenceMs": 551.2365000322461,
"tokensPerSecond": 38.25766978320208,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366237696,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5622.343958064448,
"residentMemoryBytes": 686997504,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5071.107458032202,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 224.57174997543916,
"inferenceMs": 225.32304102787748,
"tokensPerSecond": 35.839731328308034,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366385152,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5296.4304990600795,
"residentMemoryBytes": 700153856,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5071.107458032202,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 333.23200000450015,
"inferenceMs": 333.99387501413,
"tokensPerSecond": 42.208446496642274,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5405.101333046332,
"residentMemoryBytes": 716832768,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5071.107458032202,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 363.8015409815125,
"inferenceMs": 364.53045799862593,
"tokensPerSecond": 49.68461201773095,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5435.637916030828,
"residentMemoryBytes": 718258176,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5071.107458032202,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 365.82524998812005,
"inferenceMs": 366.5543330134824,
"tokensPerSecond": 43.913372104114494,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5437.661791045684,
"residentMemoryBytes": 735182848,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5071.107458032202,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 292.32804203638807,
"inferenceMs": 293.03983302088454,
"tokensPerSecond": 48.14217076224888,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5364.1472910530865,
"residentMemoryBytes": 736329728,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5071.107458032202,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 292.50758298439905,
"inferenceMs": 293.21437497856095,
"tokensPerSecond": 48.08478721944285,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5364.321833010763,
"residentMemoryBytes": 737476608,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5071.107458032202,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 294.1706670098938,
"inferenceMs": 294.9345000088215,
"tokensPerSecond": 47.834874011318,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5366.0419580410235,
"residentMemoryBytes": 738590720,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5074.395167001057,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 599.6171670267358,
"inferenceMs": 600.3997080260888,
"tokensPerSecond": 23.643999047687334,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4365664256,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5674.794875027146,
"residentMemoryBytes": 755974144,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5074.395167001057,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 569.933167018462,
"inferenceMs": 570.7815419882536,
"tokensPerSecond": 36.936739459283764,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366237696,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5645.176708989311,
"residentMemoryBytes": 816054272,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5074.395167001057,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 224.59795797476545,
"inferenceMs": 225.3797500161454,
"tokensPerSecond": 35.877027112724605,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366385152,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5299.774917017203,
"residentMemoryBytes": 829341696,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5074.395167001057,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 334.7275420092046,
"inferenceMs": 335.47591598471627,
"tokensPerSecond": 42.00825466533489,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5409.871082985774,
"residentMemoryBytes": 846381056,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5074.395167001057,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 387.09595799446106,
"inferenceMs": 388.00475001335144,
"tokensPerSecond": 46.69679106461968,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5462.399917014409,
"residentMemoryBytes": 847888384,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5074.395167001057,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 402.3788750055246,
"inferenceMs": 403.41575001366436,
"tokensPerSecond": 39.92469204903762,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5477.810917014722,
"residentMemoryBytes": 865173504,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5074.395167001057,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 325.04354196134955,
"inferenceMs": 325.9589999797754,
"tokensPerSecond": 43.28680855815733,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5400.354166980833,
"residentMemoryBytes": 866287616,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5074.395167001057,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 315.57270901976153,
"inferenceMs": 316.42429099883884,
"tokensPerSecond": 44.51595841122479,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5390.819457999896,
"residentMemoryBytes": 867434496,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5074.395167001057,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 299.8407499981113,
"inferenceMs": 300.70900003192946,
"tokensPerSecond": 46.93805342183249,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5375.104167032987,
"residentMemoryBytes": 868564992,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5156.663457979448,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 602.1203750278801,
"inferenceMs": 602.9193340218626,
"tokensPerSecond": 23.454675472213218,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4365664256,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5759.582792001311,
"residentMemoryBytes": 766853120,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5156.663457979448,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 640.718333015684,
"inferenceMs": 641.6353330132551,
"tokensPerSecond": 32.86330543876525,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366237696,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5798.298790992703,
"residentMemoryBytes": 826753024,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5156.663457979448,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 251.0462910286151,
"inferenceMs": 252.02825001906604,
"tokensPerSecond": 32.12467044515411,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366385152,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5408.691707998514,
"residentMemoryBytes": 839958528,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5156.663457979448,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 378.85370798176154,
"inferenceMs": 379.838542023208,
"tokensPerSecond": 37.11137839410482,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5536.502000002656,
"residentMemoryBytes": 856915968,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5156.663457979448,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 371.87054101377726,
"inferenceMs": 372.64445796608925,
"tokensPerSecond": 48.548293553688936,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5529.307915945537,
"residentMemoryBytes": 858259456,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5156.663457979448,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 370.26295805117115,
"inferenceMs": 371.05224997503683,
"tokensPerSecond": 43.39801970530571,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5527.715707954485,
"residentMemoryBytes": 875266048,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5156.663457979448,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 295.02229200443253,
"inferenceMs": 295.76854198239744,
"tokensPerSecond": 47.68399809056053,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5452.431999961846,
"residentMemoryBytes": 876314624,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5156.663457979448,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 298.36954199709,
"inferenceMs": 299.0900410222821,
"tokensPerSecond": 47.15320918895512,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5455.75349900173,
"residentMemoryBytes": 877363200,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5156.663457979448,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 294.27825001766905,
"inferenceMs": 295.00270797871053,
"tokensPerSecond": 47.82928368131091,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5451.666165958159,
"residentMemoryBytes": 878428160,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5063.365708047058,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 583.6201659985818,
"inferenceMs": 584.4836250180379,
"tokensPerSecond": 24.214476294526765,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4365664256,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5647.849333065096,
"residentMemoryBytes": 756203520,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5063.365708047058,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 593.7089169747196,
"inferenceMs": 594.560542027466,
"tokensPerSecond": 35.46730640999211,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366237696,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5657.926250074524,
"residentMemoryBytes": 815972352,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5063.365708047058,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 240.21170800551772,
"inferenceMs": 241.02458398556337,
"tokensPerSecond": 33.527771078237286,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366385152,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5304.390292032622,
"residentMemoryBytes": 829079552,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5063.365708047058,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 345.4891250003129,
"inferenceMs": 346.4089159970172,
"tokensPerSecond": 40.71464875806509,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5409.774624044076,
"residentMemoryBytes": 846004224,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5063.365708047058,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 363.4190419688821,
"inferenceMs": 364.1935000196099,
"tokensPerSecond": 49.74820718794639,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5427.559208066668,
"residentMemoryBytes": 847331328,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5063.365708047058,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 374.83474996406585,
"inferenceMs": 375.646417029202,
"tokensPerSecond": 42.854019754646544,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5439.01212507626,
"residentMemoryBytes": 864223232,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5063.365708047058,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 300.4352909629233,
"inferenceMs": 301.31387495202944,
"tokensPerSecond": 46.76462561318818,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5364.679582999088,
"residentMemoryBytes": 865304576,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5063.365708047058,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 339.2296249512583,
"inferenceMs": 340.32745799049735,
"tokensPerSecond": 41.47975568015052,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5403.693166037556,
"residentMemoryBytes": 866451456,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5063.365708047058,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 323.45166604500264,
"inferenceMs": 324.40583303105086,
"tokensPerSecond": 43.4530706378336,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5387.771541078109,
"residentMemoryBytes": 867467264,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5676.194750005379,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 687.2474999981932,
"inferenceMs": 688.2867499953136,
"tokensPerSecond": 20.532396506806585,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7477821440,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6364.481500000693,
"residentMemoryBytes": 627490816,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5676.194750005379,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 744.0054580220021,
"inferenceMs": 745.0323750381358,
"tokensPerSecond": 28.322089832768786,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478411264,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6421.227125043515,
"residentMemoryBytes": 685293568,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5676.194750005379,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 340.79741704044864,
"inferenceMs": 341.67854103725404,
"tokensPerSecond": 23.593777116698973,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478558720,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6017.873291042633,
"residentMemoryBytes": 697794560,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5676.194750005379,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 527.1212079678662,
"inferenceMs": 528.017416014336,
"tokensPerSecond": 26.65537408102582,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478706176,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6204.212166019715,
"residentMemoryBytes": 713981952,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5676.194750005379,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 624.746999994386,
"inferenceMs": 625.8827499696054,
"tokensPerSecond": 28.892788240430985,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478706176,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6302.077499974985,
"residentMemoryBytes": 714915840,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5676.194750005379,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 605.4836249677464,
"inferenceMs": 606.5831669839099,
"tokensPerSecond": 26.51308041385848,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6282.777916989289,
"residentMemoryBytes": 731054080,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5676.194750005379,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 506.9645419716835,
"inferenceMs": 508.17041704431176,
"tokensPerSecond": 27.73375594321377,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6184.365167049691,
"residentMemoryBytes": 731824128,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5676.194750005379,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 511.3586249644868,
"inferenceMs": 512.4484999687411,
"tokensPerSecond": 27.44454303146364,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6188.64324997412,
"residentMemoryBytes": 732594176,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5676.194750005379,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 478.20354101713747,
"inferenceMs": 479.0973329800181,
"tokensPerSecond": 29.40663195001654,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6155.292082985397,
"residentMemoryBytes": 733282304,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5208.352167042904,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 581.2358330003917,
"inferenceMs": 582.1381250279956,
"tokensPerSecond": 24.28690645932201,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7477821440,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5790.490292070899,
"residentMemoryBytes": 628883456,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5208.352167042904,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 768.2614589575678,
"inferenceMs": 769.2001659888774,
"tokensPerSecond": 27.41981051827274,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478411264,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5977.552333031781,
"residentMemoryBytes": 687456256,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5208.352167042904,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 377.204124990385,
"inferenceMs": 378.0499999993481,
"tokensPerSecond": 21.277336149968782,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478558720,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5586.402167042252,
"residentMemoryBytes": 700055552,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5208.352167042904,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 524.3797079892829,
"inferenceMs": 525.2519160276279,
"tokensPerSecond": 26.77990311541379,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478706176,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5733.6040830705315,
"residentMemoryBytes": 716455936,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5208.352167042904,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 629.9519159947522,
"inferenceMs": 630.7920420076698,
"tokensPerSecond": 28.64953835476948,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478706176,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5839.144209050573,
"residentMemoryBytes": 717406208,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5208.352167042904,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 613.5167910251766,
"inferenceMs": 614.641375024803,
"tokensPerSecond": 26.151350643262944,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5822.993542067707,
"residentMemoryBytes": 733708288,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5208.352167042904,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 513.7178750010207,
"inferenceMs": 514.589625003282,
"tokensPerSecond": 27.353822886532225,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5722.941792046186,
"residentMemoryBytes": 734511104,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5208.352167042904,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 501.5064590261318,
"inferenceMs": 502.36112502170727,
"tokensPerSecond": 28.007466342005383,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5710.713292064611,
"residentMemoryBytes": 735232000,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5208.352167042904,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 486.3355839625001,
"inferenceMs": 487.34508303459734,
"tokensPerSecond": 28.848734560929262,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5695.697250077501,
"residentMemoryBytes": 735936512,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5533.444667002186,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 598.3142919722013,
"inferenceMs": 599.1157499956898,
"tokensPerSecond": 23.604496557481085,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7477821440,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6132.560416997876,
"residentMemoryBytes": 628473856,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5533.444667002186,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 713.729584007524,
"inferenceMs": 714.5576669718139,
"tokensPerSecond": 29.49284251272467,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478411264,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6248.002333974,
"residentMemoryBytes": 686440448,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5533.444667002186,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 334.75841698236763,
"inferenceMs": 335.6024580425583,
"tokensPerSecond": 24.024126178893333,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478558720,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 5869.0471250447445,
"residentMemoryBytes": 699154432,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5533.444667002186,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 524.8386249877512,
"inferenceMs": 525.7356250076555,
"tokensPerSecond": 26.755961503859982,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478706176,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6059.180292009842,
"residentMemoryBytes": 715522048,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5533.444667002186,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 599.5375000056811,
"inferenceMs": 600.5027079954743,
"tokensPerSecond": 30.105419143414366,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478706176,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6133.9473749976605,
"residentMemoryBytes": 716488704,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5533.444667002186,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 583.6932079982944,
"inferenceMs": 584.7257919958793,
"tokensPerSecond": 27.51343859529672,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6118.170458998065,
"residentMemoryBytes": 732610560,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5533.444667002186,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 482.1305410005152,
"inferenceMs": 483.16487501142547,
"tokensPerSecond": 29.158966879588377,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6016.609542013612,
"residentMemoryBytes": 733396992,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5533.444667002186,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 477.67274995567277,
"inferenceMs": 478.7277089781128,
"tokensPerSecond": 29.376681914457688,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6012.172375980299,
"residentMemoryBytes": 734216192,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5533.444667002186,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 478.1302909832448,
"inferenceMs": 479.18066603597254,
"tokensPerSecond": 29.41205567723355,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 6012.625333038159,
"residentMemoryBytes": 734904320,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 11265.841416025069,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 830.5907909525558,
"inferenceMs": 831.5384170273319,
"tokensPerSecond": 16.972629449260808,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4365647872,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 12097.3798330524,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 11265.841416025069,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 654.7289160080254,
"inferenceMs": 655.7486249948852,
"tokensPerSecond": 32.14018015937715,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366237696,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 11921.590041019954,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 11265.841416025069,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 246.59404199337587,
"inferenceMs": 247.56270903162658,
"tokensPerSecond": 32.64910215293826,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366385152,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 11513.404125056695,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 11265.841416025069,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 377.81195796560496,
"inferenceMs": 378.651249979157,
"tokensPerSecond": 37.20381445521205,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 11644.492666004226,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 11265.841416025069,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 413.0434169783257,
"inferenceMs": 413.8617910211906,
"tokensPerSecond": 43.74527611619332,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 11679.70320704626,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 11265.841416025069,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 429.31795900221914,
"inferenceMs": 430.1434999797493,
"tokensPerSecond": 37.365626114010205,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 11695.984916004818,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5506.533750041854,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1420.5479579977691,
"inferenceMs": 1421.3865419733338,
"tokensPerSecond": 9.883411919568951,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6927.920292015187,
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5506.533750041854,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1936.2250409903936,
"inferenceMs": 1936.9679170195013,
"tokensPerSecond": 10.8478206823294,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 7443.501667061355,
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5506.533750041854,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 796.8812920153141,
"inferenceMs": 797.682625008747,
"tokensPerSecond": 10.042700836310381,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6304.216375050601,
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5506.533750041854,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1362.7162500051782,
"inferenceMs": 1363.4600000223145,
"tokensPerSecond": 10.275803483254277,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6869.993750064168,
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5506.533750041854,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1665.011958044488,
"inferenceMs": 1665.759916009847,
"tokensPerSecond": 10.812746603950236,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 7172.293666051701,
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5506.533750041854,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1462.9188750404865,
"inferenceMs": 1463.6539579951204,
"tokensPerSecond": 10.93912222041694,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6970.187708036974,
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5279.279334004968,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1341.8798329657875,
"inferenceMs": 1342.626957979519,
"tokensPerSecond": 10.460440903105097,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6621.906291984487,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5279.279334004968,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2037.523582985159,
"inferenceMs": 2038.2564999745227,
"tokensPerSecond": 10.308815713863662,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 7317.535833979491,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5279.279334004968,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 740.0127500295639,
"inferenceMs": 740.7512920326553,
"tokensPerSecond": 10.81606110201872,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6020.030626037624,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5279.279334004968,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1292.0497499872,
"inferenceMs": 1292.7978750085458,
"tokensPerSecond": 10.838469832861005,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6572.077209013514,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5279.279334004968,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1642.42866699351,
"inferenceMs": 1643.2602079585195,
"tokensPerSecond": 10.962221975500949,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6922.539541963488,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5279.279334004968,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1681.7243340192363,
"inferenceMs": 1682.458332972601,
"tokensPerSecond": 9.515857931454187,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 6961.7376669775695,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5505.996083957143,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 706.1722920043394,
"inferenceMs": 707.081708998885,
"tokensPerSecond": 19.96414058253828,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4365647872,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 6213.077792956028,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5505.996083957143,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 624.4930000393651,
"inferenceMs": 625.2789160353132,
"tokensPerSecond": 33.684305080729054,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366237696,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 6131.274999992456,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5505.996083957143,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 236.51220800820738,
"inferenceMs": 237.28250002022833,
"tokensPerSecond": 33.981260399245144,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366385152,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 5743.278583977371,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5505.996083957143,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 368.4866669937037,
"inferenceMs": 369.3971249740571,
"tokensPerSecond": 38.104146379886785,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 5875.3932089312,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5505.996083957143,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 396.7707910342142,
"inferenceMs": 397.59266696637496,
"tokensPerSecond": 45.49961420237002,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366532608,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 5903.588750923518,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5505.996083957143,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 399.5512499823235,
"inferenceMs": 400.3573330119252,
"tokensPerSecond": 40.188198013315045,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 4366680064,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 5906.353416969068,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6148.680582991801,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 654.540125047788,
"inferenceMs": 655.3363329730928,
"tokensPerSecond": 21.561073035180677,
"inputTokens": 23,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7477821440,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 6804.016915964894,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6148.680582991801,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 759.6767920185812,
"inferenceMs": 760.6320839840919,
"tokensPerSecond": 27.702062762299857,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478411264,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 6909.312666975893,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6148.680582991801,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 356.5952089847997,
"inferenceMs": 357.3718749685213,
"tokensPerSecond": 22.527458837035663,
"inputTokens": 26,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478558720,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 6506.0524579603225,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6148.680582991801,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 554.4056249782443,
"inferenceMs": 555.145583988633,
"tokensPerSecond": 25.297274962376353,
"inputTokens": 22,
"outputTokens": 14,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478706176,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 6703.826166980434,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7626300625142021,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6148.680582991801,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 622.5916250259615,
"inferenceMs": 623.3744579949416,
"tokensPerSecond": 28.958631491242773,
"inputTokens": 29,
"outputTokens": 18,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478706176,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 6772.055040986743,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6148.680582991801,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 605.1942909834906,
"inferenceMs": 606.1259579728357,
"tokensPerSecond": 26.50773986670098,
"inputTokens": 21,
"outputTokens": 16,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 7478853632,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 6754.806540964637,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "dbad03788f49709801014c95d481a514c272ca52",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "excellent"
}
}
]HY-MT1.5 7B Desktop
138 configurations testedQuality criteria
{ "minimumCharacterF2": 0.55, "requireSemanticAnchors": true, "rejectRepetitionAndTruncation": true, "frozenPolicy": "research/hy-mt15-family-expansion/quality-gates.json" }
| Precision | Device | Result | Platform | Inference / load | |
|---|---|---|---|---|---|
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 619 ms / 14371 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 624 ms / 7061 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 624 ms / 13975 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 625 ms / 6786 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 625 ms / 6494 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 634 ms / 14750 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 644 ms / 6628 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 648 ms / 7149 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 732 ms / 13975 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 740 ms / 6786 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 752 ms / 7149 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 753 ms / 7061 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 802 ms / 6494 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 807 ms / 14750 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 809 ms / 6786 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 812 ms / 7149 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 812 ms / 6494 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 812 ms / 13975 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 812 ms / 6786 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 812 ms / 13975 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 813 ms / 13975 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 815 ms / 6786 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 816 ms / 14371 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 816 ms / 14750 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 817 ms / 6628 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 817 ms / 7149 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 818 ms / 14750 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 818 ms / 14750 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 821 ms / 6494 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 822 ms / 6494 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 823 ms / 6628 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 824 ms / 7149 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 825 ms / 6628 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 862 ms / 6628 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 930 ms / 13975 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 954 ms / 7149 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 997 ms / 6628 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1001 ms / 6786 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1003 ms / 6494 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1050 ms / 7061 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1059 ms / 21440 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1072 ms / 14371 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1097 ms / 13975 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1101 ms / 6786 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1102 ms / 7061 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1102 ms / 14750 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1105 ms / 7149 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1107 ms / 14750 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1109 ms / 14371 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1111 ms / 6494 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1131 ms / 6628 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1285 ms / 21440 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1366 ms / 6494 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1367 ms / 7061 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1369 ms / 13975 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1372 ms / 14750 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1381 ms / 7149 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1388 ms / 6786 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1401 ms / 6628 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1416 ms / 14371 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1548 ms / 7061 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1548 ms / 13975 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1553 ms / 14371 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1554 ms / 6786 ms |
FP16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1555 ms / 7149 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1568 ms / 6494 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1571 ms / 14750 ms |
BF16 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1682 ms / 6628 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 1991 ms / 21440 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 2447 ms / 21440 ms |
FP32 | mpsAccelerated | Passed | Fast | macos · aarch64 | 2801 ms / 21440 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 2611 ms / 13188 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 2655 ms / 13543 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 2667 ms / 16399 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 2695 ms / 7213 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 2707 ms / 5703 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 2731 ms / 16329 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 2838 ms / 14972 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3091 ms / 14643 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3176 ms / 5703 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3254 ms / 13543 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3268 ms / 16399 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3314 ms / 7213 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3320 ms / 13188 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3443 ms / 16329 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3591 ms / 14972 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3748 ms / 13543 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3756 ms / 5703 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3759 ms / 13188 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3767 ms / 13188 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3778 ms / 16399 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3784 ms / 16399 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3792 ms / 13543 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3797 ms / 5703 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3824 ms / 13543 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3863 ms / 14643 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3879 ms / 16399 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3887 ms / 14643 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3902 ms / 13188 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3964 ms / 5703 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3978 ms / 16329 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 3987 ms / 14643 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4073 ms / 5703 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4096 ms / 14643 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4121 ms / 16329 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4144 ms / 16329 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4209 ms / 13543 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4258 ms / 13188 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4564 ms / 14972 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4737 ms / 14643 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 4817 ms / 16399 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5162 ms / 13188 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5254 ms / 13543 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5260 ms / 7213 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5328 ms / 16329 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5346 ms / 16329 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5374 ms / 16399 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5496 ms / 7213 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5609 ms / 14972 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5642 ms / 5703 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 5800 ms / 14643 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 6363 ms / 13188 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 6543 ms / 13543 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 6612 ms / 16329 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 6804 ms / 5703 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 6937 ms / 16399 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 6965 ms / 7213 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 6971 ms / 14972 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7080 ms / 14643 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7263 ms / 13188 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7421 ms / 14643 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7438 ms / 16399 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7468 ms / 13543 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7527 ms / 7213 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7592 ms / 16329 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7612 ms / 5703 ms |
FP32 | cpu | Passed | Moderate | macos · aarch64 | 7639 ms / 14972 ms |
FP32 | mps | Failed | — | macos · aarch64 | 2511 ms / 21440 ms |
Test setup & sample details
- Quality gate
- { "minimumCharacterF2": 0.55, "requireSemanticAnchors": true, "rejectRepetitionAndTruncation": true, "frozenPolicy": "research/hy-mt15-family-expansion/quality-gates.json" }
- Tested
- 2026-11-13T09:50:29.956212+00:00
- Environment
- { "os": "macos", "arch": "aarch64", "hardware": "Apple M3 Max", "osVersion": "27.2", "runtime": "App-owned CPython3.13.18; Transformers5.22.1/Torch2.10.0; built-in HunYuanDenseV1ForCausalLM, SDPA; MPS CPU op fallback disabled", "concurrentUploadActivity": "none; uploads excluded", "transport": "Verified exact local original snapshots, not Internet transfer timing" }
- Runtime
- transformers 5.22.1
- Artifact revision
- 397085d46158ad0b437ffd74228499e8c2a35c76
- Fixture
{ "file": "model-samples/hunyuan-translation.json", "url": "https://assets.localai.tools/samples/v1/files/b63cba0b5f61f8423d4234d4d28e9cdde538b8345b4ccdc8b2044b1bc88e2fa8/hunyuan-translation.json", "size": 3099, "sha1": "dd6e22e1c2966dd5a249b11eb2fb48b076505088", "sha256": "b63cba0b5f61f8423d4234d4d28e9cdde538b8345b4ccdc8b2044b1bc88e2fa8" }- Rerun when
- model artifact revision change; runtime version change; fixture or quality-gate change; backend/platform behavior change
Per-case quality & failure details
[
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14643.35041603772,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4736.64562497288,
"inferenceMs": 4737.382166960742,
"tokensPerSecond": 2.5353305350718878,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 19380.732582998462,
"residentMemoryBytes": 30583832576,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14643.35041603772,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7079.10550001543,
"inferenceMs": 7079.876542033162,
"tokensPerSecond": 2.9666150387476358,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 21723.22695807088,
"residentMemoryBytes": 30589140992,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14643.35041603772,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3090.144124988001,
"inferenceMs": 3091.0674589686096,
"tokensPerSecond": 2.58922398449713,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 17734.41787500633,
"residentMemoryBytes": 30589599744,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14643.35041603772,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4094.9658330064267,
"inferenceMs": 4095.905166992452,
"tokensPerSecond": 2.44233355325552,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 18739.25558303017,
"residentMemoryBytes": 30589648896,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14643.35041603772,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5798.689000017475,
"inferenceMs": 5799.518915999215,
"tokensPerSecond": 2.931896212061948,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 20442.869332036935,
"residentMemoryBytes": 30590025728,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14643.35041603772,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7419.755999988411,
"inferenceMs": 7420.618749980349,
"tokensPerSecond": 3.2347895870971493,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 22063.96916601807,
"residentMemoryBytes": 30590074880,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14643.35041603772,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3985.92175001977,
"inferenceMs": 3986.69987503672,
"tokensPerSecond": 3.010862470703743,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 18630.05029107444,
"residentMemoryBytes": 30590074880,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14643.35041603772,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3862.4612080166116,
"inferenceMs": 3863.2483340334147,
"tokensPerSecond": 3.107049674281603,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 18506.598750071134,
"residentMemoryBytes": 30590074880,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14643.35041603772,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3886.4384579937905,
"inferenceMs": 3887.221042008605,
"tokensPerSecond": 3.0878979809913436,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 18530.571458046325,
"residentMemoryBytes": 30590074880,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16328.98649998242,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5345.375541015528,
"inferenceMs": 5346.141750051174,
"tokensPerSecond": 2.2468486541984154,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 21675.128250033595,
"residentMemoryBytes": 30594039808,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16328.98649998242,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 6611.507290974259,
"inferenceMs": 6612.341791973449,
"tokensPerSecond": 3.176453124564577,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 22941.32829195587,
"residentMemoryBytes": 30601330688,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16328.98649998242,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2729.684541991446,
"inferenceMs": 2730.5405419901945,
"tokensPerSecond": 2.931110252408365,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 19059.527041972615,
"residentMemoryBytes": 30601510912,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16328.98649998242,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3441.9345410424285,
"inferenceMs": 3442.737208039034,
"tokensPerSecond": 2.9056117823504453,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 19771.723708021455,
"residentMemoryBytes": 30602493952,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16328.98649998242,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5327.264249965083,
"inferenceMs": 5328.064290981274,
"tokensPerSecond": 3.1913661659900407,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 21657.050790963694,
"residentMemoryBytes": 30602838016,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16328.98649998242,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7591.606832982507,
"inferenceMs": 7592.434207967017,
"tokensPerSecond": 3.1615230525309155,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 23921.420707949437,
"residentMemoryBytes": 30602821632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16328.98649998242,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4120.128333044704,
"inferenceMs": 4121.082666970324,
"tokensPerSecond": 2.9128469811524864,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 20450.069166952744,
"residentMemoryBytes": 30606082048,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16328.98649998242,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4142.697083996609,
"inferenceMs": 4143.56300001964,
"tokensPerSecond": 2.896872952389996,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 20472.54950000206,
"residentMemoryBytes": 30606098432,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16328.98649998242,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3977.4257910321467,
"inferenceMs": 3978.2299159560353,
"tokensPerSecond": 3.0172594785409443,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 20307.216415938456,
"residentMemoryBytes": 30606114816,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16398.717332980596,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4816.455000021961,
"inferenceMs": 4817.4360420089215,
"tokensPerSecond": 2.4940373459185805,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 21216.153374989517,
"residentMemoryBytes": 30561910784,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16398.717332980596,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 6936.230708961375,
"inferenceMs": 6937.038542004302,
"tokensPerSecond": 3.027736480994778,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 23335.755874984898,
"residentMemoryBytes": 30572199936,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16398.717332980596,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2666.007207997609,
"inferenceMs": 2666.7731249472126,
"tokensPerSecond": 3.001066317136659,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 19065.490457927808,
"residentMemoryBytes": 30572314624,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16398.717332980596,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3267.5524589722045,
"inferenceMs": 3268.3158749714494,
"tokensPerSecond": 3.060631804692108,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 19667.033207952045,
"residentMemoryBytes": 30572298240,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16398.717332980596,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5373.106334009208,
"inferenceMs": 5373.982458957471,
"tokensPerSecond": 3.1641011017791683,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 21772.699791938066,
"residentMemoryBytes": 30572314624,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16398.717332980596,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7436.94783304818,
"inferenceMs": 7437.779209052678,
"tokensPerSecond": 3.2272976847853783,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 23836.496542033274,
"residentMemoryBytes": 30572347392,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16398.717332980596,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3783.2388340029866,
"inferenceMs": 3784.0559579781257,
"tokensPerSecond": 3.1721187678391494,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 20182.77329095872,
"residentMemoryBytes": 30572363776,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16398.717332980596,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3777.0158330095,
"inferenceMs": 3777.7542080148123,
"tokensPerSecond": 3.177331294302294,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 20176.471540995408,
"residentMemoryBytes": 30572380160,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 16398.717332980596,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3877.921375038568,
"inferenceMs": 3878.6919999984093,
"tokensPerSecond": 3.0946441463492493,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 20277.409332979005,
"residentMemoryBytes": 30572396544,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13543.445665971376,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4208.0137499724515,
"inferenceMs": 4208.732833038084,
"tokensPerSecond": 2.8543154707815748,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 17752.17849900946,
"residentMemoryBytes": 30575886336,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13543.445665971376,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 6542.252958985046,
"inferenceMs": 6543.048000021372,
"tokensPerSecond": 3.2100408443683577,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 20086.493665992748,
"residentMemoryBytes": 30584406016,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13543.445665971376,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2654.4713340117596,
"inferenceMs": 2655.2497089724056,
"tokensPerSecond": 3.014114722475746,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 16198.695374943782,
"residentMemoryBytes": 30584487936,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13543.445665971376,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3252.9463329701684,
"inferenceMs": 3253.725249960553,
"tokensPerSecond": 3.074417308568935,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 16797.17091593193,
"residentMemoryBytes": 30584619008,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13543.445665971376,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5252.987666986883,
"inferenceMs": 5253.785000008065,
"tokensPerSecond": 3.2364442150910633,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 18797.23066597944,
"residentMemoryBytes": 30584635392,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13543.445665971376,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7466.854624974076,
"inferenceMs": 7467.611791973468,
"tokensPerSecond": 3.2143165659826036,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 21011.057457944844,
"residentMemoryBytes": 30586437632,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13543.445665971376,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3790.886834030971,
"inferenceMs": 3791.644707962405,
"tokensPerSecond": 3.165693311502654,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 17335.09037393378,
"residentMemoryBytes": 30586454016,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13543.445665971376,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3747.506042011082,
"inferenceMs": 3748.306291992776,
"tokensPerSecond": 3.2023454336485337,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 17291.751957964152,
"residentMemoryBytes": 30586454016,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13543.445665971376,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3823.5879170242697,
"inferenceMs": 3824.393374961801,
"tokensPerSecond": 3.138625435081547,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 17367.839040933177,
"residentMemoryBytes": 30586486784,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13187.68395902589,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4257.692749961279,
"inferenceMs": 4258.468667045236,
"tokensPerSecond": 2.820844797173472,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 17446.152626071125,
"residentMemoryBytes": 30576836608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13187.68395902589,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 6362.028416013345,
"inferenceMs": 6362.837957974989,
"tokensPerSecond": 3.300993422183347,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 19550.52191700088,
"residentMemoryBytes": 30585356288,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13187.68395902589,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2610.004917019978,
"inferenceMs": 2610.765582998283,
"tokensPerSecond": 3.0654595844773906,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 15798.449542024173,
"residentMemoryBytes": 30585470976,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13187.68395902589,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3319.3914580042474,
"inferenceMs": 3320.153374981601,
"tokensPerSecond": 3.0128271864800493,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 16507.83733400749,
"residentMemoryBytes": 30585454592,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13187.68395902589,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5161.509750003461,
"inferenceMs": 5162.286750040948,
"tokensPerSecond": 3.2937932027297907,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 18349.970709066838,
"residentMemoryBytes": 30585487360,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13187.68395902589,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7262.650666991249,
"inferenceMs": 7263.428167032544,
"tokensPerSecond": 3.3046945215034813,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 20451.112126058433,
"residentMemoryBytes": 30585651200,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13187.68395902589,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3757.741999987047,
"inferenceMs": 3758.5494999657385,
"tokensPerSecond": 3.193613659260168,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 16946.233458991628,
"residentMemoryBytes": 30586617856,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13187.68395902589,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3900.912708020769,
"inferenceMs": 3901.7003750195727,
"tokensPerSecond": 3.076407121640068,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 17089.384334045462,
"residentMemoryBytes": 30586667008,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13187.68395902589,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3766.034291998949,
"inferenceMs": 3766.826167004183,
"tokensPerSecond": 3.186598899368618,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 16954.510126030073,
"residentMemoryBytes": 30586667008,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5703.028291987721,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4072.6116249570623,
"inferenceMs": 4073.330375016667,
"tokensPerSecond": 2.949155816427165,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9776.358667004388,
"residentMemoryBytes": 30575460352,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5703.028291987721,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 6802.5834579602815,
"inferenceMs": 6803.526541043539,
"tokensPerSecond": 3.0872136002952653,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 12506.55483303126,
"residentMemoryBytes": 30590566400,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5703.028291987721,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2706.3983340049163,
"inferenceMs": 2707.2866249945946,
"tokensPerSecond": 2.9564776915384905,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 8410.314916982315,
"residentMemoryBytes": 30590894080,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5703.028291987721,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3174.877249985002,
"inferenceMs": 3175.6475000292994,
"tokensPerSecond": 3.1499888864538876,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 8878.67579201702,
"residentMemoryBytes": 30590910464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5703.028291987721,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5641.4926670258865,
"inferenceMs": 5642.3134169890545,
"tokensPerSecond": 3.013610150561966,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 11345.341708976775,
"residentMemoryBytes": 30592253952,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5703.028291987721,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7611.619082977995,
"inferenceMs": 7612.41208401043,
"tokensPerSecond": 3.153201445442238,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 13315.44037599815,
"residentMemoryBytes": 30595137536,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5703.028291987721,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3963.0806670174934,
"inferenceMs": 3963.8495839899406,
"tokensPerSecond": 3.0281536590486993,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9666.877875977661,
"residentMemoryBytes": 30595137536,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5703.028291987721,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3755.355665984098,
"inferenceMs": 3756.121499987785,
"tokensPerSecond": 3.195646393696911,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9459.149791975506,
"residentMemoryBytes": 30595268608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 5703.028291987721,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3796.069457952399,
"inferenceMs": 3796.8450839980505,
"tokensPerSecond": 3.161377035232006,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9499.873375985771,
"residentMemoryBytes": 30595268608,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14750.21604099311,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1100.6679159472696,
"inferenceMs": 1101.7945000203326,
"tokensPerSecond": 10.954121900291895,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088369664,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 15852.010541013442,
"residentMemoryBytes": 709066752,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14750.21604099311,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1370.7772920024581,
"inferenceMs": 1371.9755409983918,
"tokensPerSecond": 15.343869352091632,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089172480,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 16122.191581991501,
"residentMemoryBytes": 779190272,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14750.21604099311,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 632.7812920208089,
"inferenceMs": 633.5513329831883,
"tokensPerSecond": 12.683743345368104,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089385472,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 15383.767373976298,
"residentMemoryBytes": 792657920,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7376087954634903,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14750.21604099311,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 805.7605000212789,
"inferenceMs": 806.5663749584928,
"tokensPerSecond": 13.684004610993481,
"inputTokens": 21,
"outputTokens": 11,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 15556.782415951602,
"residentMemoryBytes": 809517056,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14750.21604099311,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1105.965540977195,
"inferenceMs": 1107.1145420428365,
"tokensPerSecond": 15.391719840726267,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 15857.330583035946,
"residentMemoryBytes": 810893312,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14750.21604099311,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1569.7225000476465,
"inferenceMs": 1570.6152910133824,
"tokensPerSecond": 15.310607978720878,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 16320.831332006492,
"residentMemoryBytes": 828964864,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14750.21604099311,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 815.1705419877544,
"inferenceMs": 816.2963750073686,
"tokensPerSecond": 14.749428722001607,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 15566.512416000478,
"residentMemoryBytes": 829915136,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14750.21604099311,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 817.4676250200719,
"inferenceMs": 818.2562080328353,
"tokensPerSecond": 14.71498378652433,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 15568.472249025945,
"residentMemoryBytes": 830881792,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14750.21604099311,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 817.3582500312477,
"inferenceMs": 818.4575829654932,
"tokensPerSecond": 14.718565698217962,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 15568.673623958603,
"residentMemoryBytes": 831946752,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6493.908666016068,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1001.9266249728389,
"inferenceMs": 1002.6766659575514,
"tokensPerSecond": 12.03455723083486,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088369664,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7496.58533197362,
"residentMemoryBytes": 689127424,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6493.908666016068,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1364.6453329711221,
"inferenceMs": 1365.840542013757,
"tokensPerSecond": 15.406880584288151,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089172480,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7859.749208029825,
"residentMemoryBytes": 758874112,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6493.908666016068,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 624.3911670171656,
"inferenceMs": 625.1689579803497,
"tokensPerSecond": 12.856627437454712,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089385472,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7119.077623996418,
"residentMemoryBytes": 772161536,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7376087954634903,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6493.908666016068,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 800.9874589624815,
"inferenceMs": 802.150083007291,
"tokensPerSecond": 13.766501147529143,
"inputTokens": 21,
"outputTokens": 11,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7296.058749023359,
"residentMemoryBytes": 788971520,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6493.908666016068,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1109.4199999934062,
"inferenceMs": 1110.5947499745525,
"tokensPerSecond": 15.353086563860014,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7604.503415990621,
"residentMemoryBytes": 790380544,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6493.908666016068,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1566.3357919547707,
"inferenceMs": 1567.568083992228,
"tokensPerSecond": 15.344786152979676,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8061.476750008296,
"residentMemoryBytes": 808189952,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6493.908666016068,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 820.0086249853484,
"inferenceMs": 820.7826249999925,
"tokensPerSecond": 14.67297156073545,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7314.691291016061,
"residentMemoryBytes": 809238528,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6493.908666016068,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 810.6508330092765,
"inferenceMs": 811.8872919585556,
"tokensPerSecond": 14.839482413745237,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7305.795957974624,
"residentMemoryBytes": 810188800,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6493.908666016068,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 820.7072909572162,
"inferenceMs": 822.0411249785684,
"tokensPerSecond": 14.659235389816,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7315.949790994637,
"residentMemoryBytes": 811237376,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6627.832625003066,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 996.2332919822074,
"inferenceMs": 997.3602080135606,
"tokensPerSecond": 12.105075578699417,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088369664,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7625.1928330166265,
"residentMemoryBytes": 688488448,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6627.832625003066,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1400.1311250030994,
"inferenceMs": 1401.1626670253463,
"tokensPerSecond": 15.022719631505524,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089172480,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8028.995292028412,
"residentMemoryBytes": 757628928,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6627.832625003066,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 643.0982079473324,
"inferenceMs": 644.205792050343,
"tokensPerSecond": 12.483144497350485,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089385472,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7272.038417053409,
"residentMemoryBytes": 770752512,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7376087954634903,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6627.832625003066,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 821.7291660257615,
"inferenceMs": 822.8254579589702,
"tokensPerSecond": 13.422626337944987,
"inputTokens": 21,
"outputTokens": 11,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7450.658082962036,
"residentMemoryBytes": 787382272,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6627.832625003066,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1130.3457079920918,
"inferenceMs": 1131.4977079746313,
"tokensPerSecond": 15.071065053667711,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7759.330332977697,
"residentMemoryBytes": 788774912,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6627.832625003066,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1680.7207079837099,
"inferenceMs": 1681.6629580571316,
"tokensPerSecond": 14.299832527501323,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8309.495583060198,
"residentMemoryBytes": 806551552,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6627.832625003066,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 861.4235839922912,
"inferenceMs": 862.3916250071488,
"tokensPerSecond": 13.96106616370498,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7490.224250010215,
"residentMemoryBytes": 807567360,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6627.832625003066,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 823.9528750418685,
"inferenceMs": 825.100084010046,
"tokensPerSecond": 14.600716790325444,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7452.932709013112,
"residentMemoryBytes": 808550400,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6627.832625003066,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 815.3278750251047,
"inferenceMs": 816.5720410179347,
"tokensPerSecond": 14.760097675191803,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7444.404666021001,
"residentMemoryBytes": 809549824,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6785.672083962709,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1000.2681670011953,
"inferenceMs": 1001.413166988641,
"tokensPerSecond": 12.056788486566065,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088369664,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7787.08525095135,
"residentMemoryBytes": 698220544,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6785.672083962709,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1386.9164580246434,
"inferenceMs": 1388.1149999797344,
"tokensPerSecond": 15.165191813070155,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089172480,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8173.787083942443,
"residentMemoryBytes": 768245760,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6785.672083962709,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 623.7327080452815,
"inferenceMs": 624.5209580520168,
"tokensPerSecond": 12.870833344778635,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089385472,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7410.1930420147255,
"residentMemoryBytes": 781647872,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6785.672083962709,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 739.0788340126164,
"inferenceMs": 739.9030829546973,
"tokensPerSecond": 13.566047573929175,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7525.575166917406,
"residentMemoryBytes": 798474240,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6785.672083962709,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1100.000916980207,
"inferenceMs": 1101.1598749901168,
"tokensPerSecond": 15.482541887986288,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7886.8319589528255,
"residentMemoryBytes": 799817728,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6785.672083962709,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1553.4557909704745,
"inferenceMs": 1554.4695829739794,
"tokensPerSecond": 15.470982222115397,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8340.141666936688,
"residentMemoryBytes": 817774592,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6785.672083962709,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 807.5257909949869,
"inferenceMs": 808.6852499982342,
"tokensPerSecond": 14.897707072694653,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7594.357333960943,
"residentMemoryBytes": 818757632,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6785.672083962709,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 814.2863340326585,
"inferenceMs": 815.3080830234103,
"tokensPerSecond": 14.774520811406727,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7600.980166986119,
"residentMemoryBytes": 819724288,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 6785.672083962709,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 811.467542022001,
"inferenceMs": 812.4139999854378,
"tokensPerSecond": 14.826727302693396,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7598.086083948147,
"residentMemoryBytes": 820723712,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13974.678416969255,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 928.8477079826407,
"inferenceMs": 929.9632499460131,
"tokensPerSecond": 12.99955765830744,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088369664,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 14904.641666915268,
"residentMemoryBytes": 691339264,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13974.678416969255,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1367.747749958653,
"inferenceMs": 1368.9910419634543,
"tokensPerSecond": 15.37766774301404,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089172480,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 15343.669458932709,
"residentMemoryBytes": 760201216,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13974.678416969255,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 623.4392080223188,
"inferenceMs": 624.2779580061324,
"tokensPerSecond": 12.87897990121978,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089385472,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 14598.956374975387,
"residentMemoryBytes": 773193728,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13974.678416969255,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 731.5755839808844,
"inferenceMs": 732.3209579917602,
"tokensPerSecond": 13.699513737248424,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 14706.999374961015,
"residentMemoryBytes": 789676032,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13974.678416969255,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1096.1347909760661,
"inferenceMs": 1097.3373330198228,
"tokensPerSecond": 15.541776032038864,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 15072.015749989077,
"residentMemoryBytes": 791068672,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13974.678416969255,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1546.990457980428,
"inferenceMs": 1548.2812499976717,
"tokensPerSecond": 15.536445135481786,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 15522.959666966926,
"residentMemoryBytes": 808747008,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13974.678416969255,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 811.4250000216998,
"inferenceMs": 812.454583006911,
"tokensPerSecond": 14.826804391977772,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 14787.132999976166,
"residentMemoryBytes": 809730048,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13974.678416969255,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 812.0097079663537,
"inferenceMs": 812.9156249924563,
"tokensPerSecond": 14.81748485315118,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 14787.59404196171,
"residentMemoryBytes": 810696704,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 13974.678416969255,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 811.4127499866299,
"inferenceMs": 812.3217500397004,
"tokensPerSecond": 14.828934340186464,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 14787.000167008955,
"residentMemoryBytes": 811679744,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7149.194958037697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 953.1380420085043,
"inferenceMs": 954.2899999651127,
"tokensPerSecond": 12.655754208659385,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088369664,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8103.484958002809,
"residentMemoryBytes": 697139200,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7149.194958037697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1379.8312500002794,
"inferenceMs": 1380.7465419522487,
"tokensPerSecond": 15.243624887219482,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089172480,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8529.941499989945,
"residentMemoryBytes": 767033344,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7149.194958037697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 647.2411670256406,
"inferenceMs": 648.1717500137165,
"tokensPerSecond": 12.402534334285262,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089385472,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7797.366708051413,
"residentMemoryBytes": 780500992,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7149.194958037697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 750.6643750239164,
"inferenceMs": 751.8726249691099,
"tokensPerSecond": 13.350525838317095,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7901.067583006807,
"residentMemoryBytes": 797310976,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7149.194958037697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1103.6318749538623,
"inferenceMs": 1104.7870829934254,
"tokensPerSecond": 15.430885922360698,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8253.982041031122,
"residentMemoryBytes": 798736384,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7149.194958037697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1554.0143339894712,
"inferenceMs": 1555.2298330003396,
"tokensPerSecond": 15.465225393425694,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 8704.424791038036,
"residentMemoryBytes": 816578560,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7149.194958037697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 810.3481250000186,
"inferenceMs": 811.5132499951869,
"tokensPerSecond": 14.847119526313909,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7960.708208032884,
"residentMemoryBytes": 817545216,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7149.194958037697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 823.1972909998149,
"inferenceMs": 824.4241249631159,
"tokensPerSecond": 14.613592251768287,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7973.619083000813,
"residentMemoryBytes": 818511872,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7149.194958037697,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 815.9192079911008,
"inferenceMs": 817.1553749707527,
"tokensPerSecond": 14.744012446837322,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference process RSS and MPS driver allocation, not peak or minimum",
"totalMs": 7966.3503330084495,
"residentMemoryBytes": 819494912,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14371.49866699474,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1071.35704200482,
"inferenceMs": 1072.4655829835683,
"tokensPerSecond": 11.25510612463049,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088369664,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 15443.964249978308,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14371.49866699474,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1414.808583969716,
"inferenceMs": 1415.6923749833368,
"tokensPerSecond": 14.867838635466956,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089172480,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 15787.191041978076,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14371.49866699474,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 617.4583750544116,
"inferenceMs": 618.7747920048423,
"tokensPerSecond": 12.989609327290848,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089385472,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 14990.273458999582,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7376087954634903,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14371.49866699474,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 814.8602499859408,
"inferenceMs": 815.9732079948299,
"tokensPerSecond": 13.533151679258026,
"inputTokens": 21,
"outputTokens": 11,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 15187.47187498957,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14371.49866699474,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1108.2339160493575,
"inferenceMs": 1109.1525420197286,
"tokensPerSecond": 15.363671655535644,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 15480.651209014468,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "bf16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14371.49866699474,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.bfloat16": 354
},
"parameterDtype": "torch.bfloat16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1552.4208749993704,
"inferenceMs": 1553.4910839633085,
"tokensPerSecond": 15.477121355406737,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 15924.989750958048,
"cpuThreads": 4,
"precision": "bf16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "bf16",
"actualPrecision": "bf16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7213.158041005954,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5259.16504196357,
"inferenceMs": 5260.12275001267,
"tokensPerSecond": 2.2834153757356246,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 12473.280791018624,
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7213.158041005954,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 6964.37516604783,
"inferenceMs": 6965.1402499875985,
"tokensPerSecond": 3.0155016346001746,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 14178.298290993553,
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7213.158041005954,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2694.291624997277,
"inferenceMs": 2695.027374953497,
"tokensPerSecond": 2.969691238420049,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 9908.185415959451,
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7213.158041005954,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3313.6002920218743,
"inferenceMs": 3314.298708050046,
"tokensPerSecond": 3.0181147624974365,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 10527.456749056,
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7213.158041005954,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5495.09375001071,
"inferenceMs": 5495.834124973044,
"tokensPerSecond": 3.093919896872152,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 12708.992165978998,
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7213.158041005954,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7526.038707990665,
"inferenceMs": 7526.787166018039,
"tokensPerSecond": 3.1890521964102443,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 14739.945207023993,
"cpuThreads": 2,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14972.291791986208,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 4562.858375022188,
"inferenceMs": 4563.663333014119,
"tokensPerSecond": 2.632221417189013,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 19535.955125000328,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14972.291791986208,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 6970.68525000941,
"inferenceMs": 6971.417874970939,
"tokensPerSecond": 3.0127650856647485,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 21943.709666957147,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14972.291791986208,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2836.880667018704,
"inferenceMs": 2837.6411669887602,
"tokensPerSecond": 2.820321876948597,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 17809.93295897497,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14972.291791986208,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 3589.772167033516,
"inferenceMs": 3590.518750017509,
"tokensPerSecond": 2.7859089235012915,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 18562.810542003717,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14972.291791986208,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 5608.144541969523,
"inferenceMs": 5608.870083000511,
"tokensPerSecond": 3.0315189703860046,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 20581.16187498672,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "cpu",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 14972.291791986208,
"device": "cpu",
"provider": "PyTorch CPU",
"acceleration": "cpu",
"placement": {
"cpu/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 7638.016375014558,
"inferenceMs": 7638.72591604013,
"tokensPerSecond": 3.142285930948345,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"totalMs": 22611.01770802634,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "cpu",
"actualDevice": "cpu",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7060.721374989953,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1048.951584030874,
"inferenceMs": 1050.0572910532355,
"tokensPerSecond": 11.494328115728372,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15088369664,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 8110.778666043188,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7060.721374989953,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1366.174582974054,
"inferenceMs": 1367.287374974694,
"tokensPerSecond": 15.393515994771525,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089172480,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 8428.008749964647,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7060.721374989953,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 622.6567919948138,
"inferenceMs": 623.8601670484059,
"tokensPerSecond": 12.8851251356781,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089385472,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 7684.581542038359,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7060.721374989953,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 751.4759170007892,
"inferenceMs": 752.5361250154674,
"tokensPerSecond": 13.339973687356897,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 7813.25750000542,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7060.721374989953,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1100.4484579898417,
"inferenceMs": 1101.509874977637,
"tokensPerSecond": 15.474931341151343,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089598464,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 8162.23124996759,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp16",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 7060.721374989953,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float16": 354
},
"parameterDtype": "torch.float16",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1546.888459008187,
"inferenceMs": 1548.019458015915,
"tokensPerSecond": 15.532644302572256,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 15089795072,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 8608.740833005868,
"cpuThreads": 4,
"precision": "fp16",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp16",
"actualPrecision": "fp16",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "fail",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": false
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 21440.188082982786,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2510.550415958278,
"inferenceMs": 2511.3039170391858,
"tokensPerSecond": 4.790001748869071,
"inputTokens": 22,
"outputTokens": 12,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 30114463744,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 23951.49200002197,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "acceptable"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 21440.188082982786,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2446.162957989145,
"inferenceMs": 2446.887083002366,
"tokensPerSecond": 8.59086657829708,
"inputTokens": 29,
"outputTokens": 21,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 30132043776,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 23887.075165985152,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 21440.188082982786,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1058.4159999853,
"inferenceMs": 1059.0835000039078,
"tokensPerSecond": 7.566080551617521,
"inputTokens": 25,
"outputTokens": 8,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 30132256768,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 22499.271582986694,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7380931729310785,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 21440.188082982786,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1284.632208000403,
"inferenceMs": 1285.285040969029,
"tokensPerSecond": 7.789900415674963,
"inputTokens": 21,
"outputTokens": 10,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 30132469760,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 22725.473123951815,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 0.7613222236973526,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 21440.188082982786,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 1990.059875010047,
"inferenceMs": 1990.7416659989394,
"tokensPerSecond": 8.547164140701268,
"inputTokens": 29,
"outputTokens": 17,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 30132469760,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 23430.929748981725,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
},
{
"precision": "fp32",
"device": "mps",
"status": "pass",
"quality": {
"passed": true,
"characterF2": 1,
"threshold": 0.55,
"anchorsPassed": true,
"repetitive": false,
"metric": "best authored reference character-ngram F2 orders1–6",
"speedPassed": true
},
"runtime": {
"engine": "transformers",
"version": "5.22.1",
"torchVersion": "2.10.0",
"loadMs": 21440.188082982786,
"device": "mps",
"provider": "Metal/MPS",
"acceleration": "partial",
"placement": {
"mps:0/torch.float32": 354
},
"parameterDtype": "torch.float32",
"accelerationDetails": "All parameters and model inputs on selected device; tokenization/decoding on CPU; MPS CPU fallback disabled",
"durationMs": 2799.802042020019,
"inferenceMs": 2800.5338750081137,
"tokensPerSecond": 8.57523600381525,
"inputTokens": 20,
"outputTokens": 24,
"maxNewTokens": 128,
"decoding": "greedy",
"outputLimitReached": false,
"driverAllocatedMemoryBytes": 30132666368,
"memoryMeasurementScope": "Post-inference driver allocation, not peak",
"totalMs": 24240.7219579909,
"cpuThreads": 4,
"precision": "fp32",
"artifactRevision": "397085d46158ad0b437ffd74228499e8c2a35c76",
"requestedDevice": "mps",
"actualDevice": "mps",
"requestedPrecision": "fp32",
"actualPrecision": "fp32",
"fallback": false,
"grade": "good"
}
}
]Release validation still required
- Sample publication excluded/pending
- Complete model/runtime backup excluded/pending
- Forced model-source and runtime-source recovery excluded/pending