developer_uid: chai_backend_admin
submission_id: chaiml-02f4-69d4-linear-w01_v47
model_name: chaiml-02f4-69d4-linear-w01_v47
model_group: ChaiML/02f4-69d4-linear-
status: inactive
timestamp: 2026-01-26T16:05:29+00:00
num_battles: 8970
num_wins: 4826
celo_rating: 1319.95
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/02f4-69d4-linear-w01
model_architecture: MistralForCausalLM
model_num_parameters: 24096691200.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
reward_model: chaiml-prm-kimi-v1-300k_33741_v7
display_name: chaiml-02f4-69d4-linear-w01_v47
ineligible_reason: model is not deployable
is_internal_developer: True
language_model: ChaiML/02f4-69d4-linear-w01
model_size: 24B
ranking_group: single
us_pacific_date: 2025-12-23
win_ratio: 0.538015607580825
generation_params: {'temperature': 0.7, 'top_p': 0.95, 'min_p': 0.025, 'top_k': 80, 'presence_penalty': 0.4, 'frequency_penalty': 0.4, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '<|im_start|>system\n{memory}<|im_end|>\n', 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-02f4-69d4-linear-w01-v47-mkmlizer
Waiting for job on chaiml-02f4-69d4-linear-w01-v47-mkmlizer to finish
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: bash: no job control in this shell
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-02f4-69d4-linear-w01-v47-mkmlizer:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║                                                                     ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║                                                                     ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║  Version: 0.30.6+torch280                                           ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║                                                                     ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║  belonging to:                                                      ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║                                                                     ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ║                                                                     ║
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: Downloaded to shared memory in 52.892s
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: Checking if ChaiML/02f4-69d4-linear-w01 already exists in ChaiML
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmplfa21tcb, device:0
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Unable to record family friendly update due to error: Invalid JSON input: Expecting value: line 1 column 1 (char 0)
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: 
Loading 0:   0%|          | 0.00/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 4.00/363 [00:02<03:30, 1.70it/s]
Loading 0:   1%|          | 4.00/363 [00:02<03:30, 1.70it/s]
Loading 0:   1%|▏         | 5.00/363 [00:04<05:54, 1.01it/s]
Loading 0:   1%|▏         | 5.00/363 [00:04<05:54, 1.01it/s]
Loading 0:   2%|▏         | 6.00/363 [00:06<07:57, 1.34s/it]
Loading 0:   2%|▏         | 6.00/363 [00:06<07:57, 1.34s/it]
Loading 0:   4%|▎         | 13.0/363 [00:09<03:54, 1.49it/s]
Loading 0:   4%|▎         | 13.0/363 [00:09<03:54, 1.49it/s]
Loading 0:   4%|▍         | 14.0/363 [00:11<05:02, 1.15it/s]
Loading 0:   4%|▍         | 14.0/363 [00:11<05:02, 1.15it/s]
Loading 0:   4%|▍         | 15.0/363 [00:14<06:16, 1.08s/it]
Loading 0:   4%|▍         | 15.0/363 [00:14<06:16, 1.08s/it]
Loading 0:   6%|▌         | 22.0/363 [00:17<03:56, 1.44it/s]
Loading 0:   6%|▌         | 22.0/363 [00:17<03:56, 1.44it/s]
Loading 0:   6%|▋         | 23.0/363 [00:19<04:49, 1.17it/s]
Loading 0:   6%|▋         | 23.0/363 [00:19<04:49, 1.17it/s]
Loading 0:   7%|▋         | 24.0/363 [00:21<05:52, 1.04s/it]
Loading 0:   7%|▋         | 24.0/363 [00:21<05:52, 1.04s/it]
Loading 0:   9%|▊         | 31.0/363 [00:24<03:46, 1.47it/s]
Loading 0:   9%|▊         | 31.0/363 [00:24<03:46, 1.47it/s]
Loading 0:   9%|▉         | 32.0/363 [00:26<04:36, 1.20it/s]
Loading 0:   9%|▉         | 32.0/363 [00:26<04:36, 1.20it/s]
Loading 0:   9%|▉         | 33.0/363 [00:28<05:36, 1.02s/it]
Loading 0:   9%|▉         | 33.0/363 [00:28<05:36, 1.02s/it]
Loading 0:  11%|█         | 40.0/363 [00:31<03:38, 1.48it/s]
Loading 0:  11%|█         | 40.0/363 [00:31<03:38, 1.48it/s]
Loading 0:  11%|█▏        | 41.0/363 [00:33<04:27, 1.21it/s]
Loading 0:  11%|█▏        | 41.0/363 [00:33<04:27, 1.21it/s]
Loading 0:  12%|█▏        | 42.0/363 [00:36<05:25, 1.01s/it]
Loading 0:  12%|█▏        | 42.0/363 [00:36<05:25, 1.01s/it]
Loading 0:  13%|█▎        | 49.0/363 [00:39<03:32, 1.48it/s]
Loading 0:  13%|█▎        | 49.0/363 [00:39<03:32, 1.48it/s]
Loading 0:  14%|█▍        | 50.0/363 [00:41<04:18, 1.21it/s]
Loading 0:  14%|█▍        | 50.0/363 [00:41<04:18, 1.21it/s]
Loading 0:  14%|█▍        | 51.0/363 [00:43<05:15, 1.01s/it]
Loading 0:  14%|█▍        | 51.0/363 [00:43<05:15, 1.01s/it]
Loading 0:  16%|█▌        | 58.0/363 [00:46<03:26, 1.48it/s]
Loading 0:  16%|█▌        | 58.0/363 [00:46<03:26, 1.48it/s]
Loading 0:  16%|█▋        | 59.0/363 [00:48<04:18, 1.18it/s]
Loading 0:  16%|█▋        | 59.0/363 [00:48<04:18, 1.18it/s]
Loading 0:  17%|█▋        | 60.0/363 [00:51<05:12, 1.03s/it]
Loading 0:  17%|█▋        | 60.0/363 [00:51<05:12, 1.03s/it]
Loading 0:  18%|█▊        | 67.0/363 [00:54<03:23, 1.46it/s]
Loading 0:  18%|█▊        | 67.0/363 [00:54<03:23, 1.46it/s]
Loading 0:  19%|█▊        | 68.0/363 [00:56<04:06, 1.20it/s]
Loading 0:  19%|█▊        | 68.0/363 [00:56<04:06, 1.20it/s]
Loading 0:  19%|█▉        | 69.0/363 [00:58<05:03, 1.03s/it]
Loading 0:  19%|█▉        | 69.0/363 [00:58<05:03, 1.03s/it]
Loading 0:  21%|██        | 76.0/363 [01:01<03:15, 1.46it/s]
Loading 0:  21%|██        | 76.0/363 [01:01<03:15, 1.46it/s]
Loading 0:  21%|██        | 77.0/363 [01:03<03:57, 1.20it/s]
Loading 0:  21%|██        | 77.0/363 [01:03<03:57, 1.20it/s]
Loading 0:  21%|██▏       | 78.0/363 [01:05<04:49, 1.01s/it]
Loading 0:  21%|██▏       | 78.0/363 [01:05<04:49, 1.01s/it]
Loading 0:  23%|██▎       | 85.0/363 [01:08<03:07, 1.48it/s]
Loading 0:  23%|██▎       | 85.0/363 [01:08<03:07, 1.48it/s]
Loading 0:  24%|██▎       | 86.0/363 [01:10<03:48, 1.21it/s]
Loading 0:  24%|██▎       | 86.0/363 [01:10<03:48, 1.21it/s]
Loading 0:  24%|██▍       | 87.0/363 [01:13<04:38, 1.01s/it]
Loading 0:  24%|██▍       | 87.0/363 [01:13<04:38, 1.01s/it]
Loading 0:  26%|██▌       | 94.0/363 [01:15<03:01, 1.49it/s]
Loading 0:  26%|██▌       | 94.0/363 [01:15<03:01, 1.49it/s]
Loading 0:  26%|██▌       | 95.0/363 [01:18<03:40, 1.21it/s]
Loading 0:  26%|██▌       | 95.0/363 [01:18<03:40, 1.21it/s]
Loading 0:  26%|██▋       | 96.0/363 [01:20<04:36, 1.04s/it]
Loading 0:  26%|██▋       | 96.0/363 [01:20<04:36, 1.04s/it]
Loading 0:  28%|██▊       | 103/363 [01:23<02:58, 1.46it/s] 
Loading 0:  28%|██▊       | 103/363 [01:23<02:58, 1.46it/s]
Loading 0:  29%|██▊       | 104/363 [01:25<03:36, 1.20it/s]
Loading 0:  29%|██▊       | 104/363 [01:25<03:36, 1.20it/s]
Loading 0:  29%|██▉       | 105/363 [01:27<04:24, 1.03s/it]
Loading 0:  29%|██▉       | 105/363 [01:27<04:24, 1.03s/it]
Loading 0:  31%|███       | 112/363 [01:30<02:51, 1.47it/s]
Loading 0:  31%|███       | 112/363 [01:30<02:51, 1.47it/s]
Loading 0:  31%|███       | 113/363 [01:32<03:27, 1.20it/s]
Loading 0:  31%|███       | 113/363 [01:32<03:27, 1.20it/s]
Loading 0:  31%|███▏      | 114/363 [01:35<04:12, 1.01s/it]
Loading 0:  31%|███▏      | 114/363 [01:35<04:12, 1.01s/it]
Loading 0:  33%|███▎      | 121/363 [01:38<02:43, 1.48it/s]
Loading 0:  33%|███▎      | 121/363 [01:38<02:43, 1.48it/s]
Loading 0:  34%|███▎      | 122/363 [01:40<03:19, 1.21it/s]
Loading 0:  34%|███▎      | 122/363 [01:40<03:19, 1.21it/s]
Loading 0:  34%|███▍      | 123/363 [01:42<04:02, 1.01s/it]
Loading 0:  34%|███▍      | 123/363 [01:42<04:02, 1.01s/it]
Loading 0:  36%|███▌      | 130/363 [01:45<02:37, 1.48it/s]
Loading 0:  36%|███▌      | 130/363 [01:45<02:37, 1.48it/s]
Loading 0:  36%|███▌      | 131/363 [01:47<03:11, 1.21it/s]
Loading 0:  36%|███▌      | 131/363 [01:47<03:11, 1.21it/s]
Loading 0:  36%|███▋      | 132/363 [01:49<03:52, 1.01s/it]
Loading 0:  36%|███▋      | 132/363 [01:49<03:52, 1.01s/it]
Loading 0:  38%|███▊      | 139/363 [01:52<02:33, 1.45it/s]
Loading 0:  38%|███▊      | 139/363 [01:52<02:33, 1.45it/s]
Loading 0:  39%|███▊      | 140/363 [01:54<03:06, 1.20it/s]
Loading 0:  39%|███▊      | 140/363 [01:54<03:06, 1.20it/s]
Loading 0:  39%|███▉      | 141/363 [01:57<03:49, 1.03s/it]
Loading 0:  39%|███▉      | 141/363 [01:57<03:49, 1.03s/it]
Loading 0:  41%|████      | 148/363 [02:00<02:27, 1.46it/s]
Loading 0:  41%|████      | 148/363 [02:00<02:27, 1.46it/s]
Loading 0:  41%|████      | 149/363 [02:02<02:58, 1.20it/s]
Loading 0:  41%|████      | 149/363 [02:02<02:58, 1.20it/s]
Loading 0:  41%|████▏     | 150/363 [02:04<03:36, 1.02s/it]
Loading 0:  41%|████▏     | 150/363 [02:04<03:36, 1.02s/it]
Loading 0:  43%|████▎     | 157/363 [02:07<02:19, 1.47it/s]
Loading 0:  43%|████▎     | 157/363 [02:07<02:19, 1.47it/s]
Loading 0:  44%|████▎     | 158/363 [02:09<02:49, 1.21it/s]
Loading 0:  44%|████▎     | 158/363 [02:09<02:49, 1.21it/s]
Loading 0:  44%|████▍     | 159/363 [02:12<03:31, 1.04s/it]
Loading 0:  44%|████▍     | 159/363 [02:12<03:31, 1.04s/it]
Loading 0:  46%|████▌     | 166/363 [02:14<02:14, 1.46it/s]
Loading 0:  46%|████▌     | 166/363 [02:14<02:14, 1.46it/s]
Loading 0:  46%|████▌     | 167/363 [02:17<02:43, 1.20it/s]
Loading 0:  46%|████▌     | 167/363 [02:17<02:43, 1.20it/s]
Loading 0:  46%|████▋     | 168/363 [02:19<03:17, 1.02s/it]
Loading 0:  46%|████▋     | 168/363 [02:19<03:17, 1.02s/it]
Loading 0:  48%|████▊     | 175/363 [02:22<02:07, 1.47it/s]
Loading 0:  48%|████▊     | 175/363 [02:22<02:07, 1.47it/s]
Loading 0:  48%|████▊     | 176/363 [02:24<02:38, 1.18it/s]
Loading 0:  48%|████▊     | 176/363 [02:24<02:38, 1.18it/s]
Loading 0:  49%|████▉     | 177/363 [02:26<03:11, 1.03s/it]
Loading 0:  49%|████▉     | 177/363 [02:26<03:11, 1.03s/it]
Loading 0:  51%|█████     | 184/363 [02:29<02:02, 1.46it/s]
Loading 0:  51%|█████     | 184/363 [02:29<02:02, 1.46it/s]
Loading 0:  51%|█████     | 185/363 [02:31<02:28, 1.20it/s]
Loading 0:  51%|█████     | 185/363 [02:31<02:28, 1.20it/s]
Loading 0:  51%|█████     | 186/363 [02:34<03:00, 1.02s/it]
Loading 0:  51%|█████     | 186/363 [02:34<03:00, 1.02s/it]
Loading 0:  53%|█████▎    | 193/363 [02:37<01:55, 1.47it/s]
Loading 0:  53%|█████▎    | 193/363 [02:37<01:55, 1.47it/s]
Loading 0:  53%|█████▎    | 194/363 [02:39<02:20, 1.20it/s]
Loading 0:  53%|█████▎    | 194/363 [02:39<02:20, 1.20it/s]
Loading 0:  54%|█████▎    | 195/363 [02:41<02:53, 1.04s/it]
Loading 0:  54%|█████▎    | 195/363 [02:41<02:53, 1.04s/it]
Loading 0:  56%|█████▌    | 202/363 [02:44<01:50, 1.45it/s]
Loading 0:  56%|█████▌    | 202/363 [02:44<01:50, 1.45it/s]
Loading 0:  56%|█████▌    | 203/363 [02:46<02:13, 1.20it/s]
Loading 0:  56%|█████▌    | 203/363 [02:46<02:13, 1.20it/s]
Loading 0:  56%|█████▌    | 204/363 [02:49<02:42, 1.02s/it]
Loading 0:  56%|█████▌    | 204/363 [02:49<02:42, 1.02s/it]
Loading 0:  58%|█████▊    | 211/363 [02:51<01:43, 1.47it/s]
Loading 0:  58%|█████▊    | 211/363 [02:51<01:43, 1.47it/s]
Loading 0:  58%|█████▊    | 212/363 [02:54<02:05, 1.21it/s]
Loading 0:  58%|█████▊    | 212/363 [02:54<02:05, 1.21it/s]
Loading 0:  59%|█████▊    | 213/363 [02:56<02:39, 1.06s/it]
Loading 0:  59%|█████▊    | 213/363 [02:56<02:39, 1.06s/it]
Loading 0:  61%|██████    | 220/363 [02:59<01:40, 1.43it/s]
Loading 0:  61%|██████    | 220/363 [02:59<01:40, 1.43it/s]
Loading 0:  61%|██████    | 221/363 [03:01<02:00, 1.18it/s]
Loading 0:  61%|██████    | 221/363 [03:01<02:00, 1.18it/s]
Loading 0:  61%|██████    | 222/363 [03:04<02:25, 1.03s/it]
Loading 0:  61%|██████    | 222/363 [03:04<02:25, 1.03s/it]
Loading 0:  63%|██████▎   | 229/363 [03:06<01:31, 1.46it/s]
Loading 0:  63%|██████▎   | 229/363 [03:06<01:31, 1.46it/s]
Loading 0:  63%|██████▎   | 230/363 [03:09<01:50, 1.20it/s]
Loading 0:  63%|██████▎   | 230/363 [03:09<01:50, 1.20it/s]
Loading 0:  64%|██████▎   | 231/363 [03:25<06:18, 2.87s/it]
Loading 0:  64%|██████▎   | 231/363 [03:25<06:18, 2.87s/it]
Loading 0:  66%|██████▌   | 238/363 [03:28<03:06, 1.49s/it]
Loading 0:  66%|██████▌   | 238/363 [03:28<03:06, 1.49s/it]
Loading 0:  66%|██████▌   | 239/363 [03:30<03:12, 1.55s/it]
Loading 0:  66%|██████▌   | 239/363 [03:30<03:12, 1.55s/it]
Loading 0:  66%|██████▌   | 240/363 [03:33<03:22, 1.64s/it]
Loading 0:  66%|██████▌   | 240/363 [03:33<03:22, 1.64s/it]
Loading 0:  68%|██████▊   | 247/363 [03:35<01:50, 1.05it/s]
Loading 0:  68%|██████▊   | 247/363 [03:35<01:50, 1.05it/s]
Loading 0:  68%|██████▊   | 248/363 [03:38<02:03, 1.07s/it]
Loading 0:  68%|██████▊   | 248/363 [03:38<02:03, 1.07s/it]
Loading 0:  69%|██████▊   | 249/363 [03:40<02:19, 1.22s/it]
Loading 0:  69%|██████▊   | 249/363 [03:40<02:19, 1.22s/it]
Loading 0:  71%|███████   | 256/363 [03:43<01:23, 1.29it/s]
Loading 0:  71%|███████   | 256/363 [03:43<01:23, 1.29it/s]
Loading 0:  71%|███████   | 257/363 [03:45<01:37, 1.09it/s]
Loading 0:  71%|███████   | 257/363 [03:45<01:37, 1.09it/s]
Loading 0:  71%|███████   | 258/363 [03:47<01:54, 1.09s/it]
Loading 0:  71%|███████   | 258/363 [03:47<01:54, 1.09s/it]
Loading 0:  73%|███████▎  | 265/363 [03:50<01:09, 1.42it/s]
Loading 0:  73%|███████▎  | 265/363 [03:50<01:09, 1.42it/s]
Loading 0:  73%|███████▎  | 266/363 [03:52<01:22, 1.17it/s]
Loading 0:  73%|███████▎  | 266/363 [03:52<01:22, 1.17it/s]
Loading 0:  74%|███████▎  | 267/363 [03:54<01:39, 1.03s/it]
Loading 0:  74%|███████▎  | 267/363 [03:54<01:39, 1.03s/it]
Loading 0:  75%|███████▌  | 274/363 [03:57<01:00, 1.47it/s]
Loading 0:  75%|███████▌  | 274/363 [03:57<01:00, 1.47it/s]
Loading 0:  76%|███████▌  | 275/363 [03:59<01:13, 1.20it/s]
Loading 0:  76%|███████▌  | 275/363 [03:59<01:13, 1.20it/s]
Loading 0:  76%|███████▌  | 276/363 [04:02<01:27, 1.01s/it]
Loading 0:  76%|███████▌  | 276/363 [04:02<01:27, 1.01s/it]
Loading 0:  78%|███████▊  | 283/363 [04:04<00:53, 1.49it/s]
Loading 0:  78%|███████▊  | 283/363 [04:04<00:53, 1.49it/s]
Loading 0:  78%|███████▊  | 284/363 [04:07<01:04, 1.22it/s]
Loading 0:  78%|███████▊  | 284/363 [04:07<01:04, 1.22it/s]
Loading 0:  79%|███████▊  | 285/363 [04:09<01:18, 1.01s/it]
Loading 0:  79%|███████▊  | 285/363 [04:09<01:18, 1.01s/it]
Loading 0:  80%|████████  | 292/363 [04:12<00:47, 1.49it/s]
Loading 0:  80%|████████  | 292/363 [04:12<00:47, 1.49it/s]
Loading 0:  81%|████████  | 293/363 [04:14<00:59, 1.19it/s]
Loading 0:  81%|████████  | 293/363 [04:14<00:59, 1.19it/s]
Loading 0:  81%|████████  | 294/363 [04:16<01:10, 1.02s/it]
Loading 0:  81%|████████  | 294/363 [04:16<01:10, 1.02s/it]
Loading 0:  83%|████████▎ | 301/363 [04:19<00:42, 1.47it/s]
Loading 0:  83%|████████▎ | 301/363 [04:19<00:42, 1.47it/s]
Loading 0:  83%|████████▎ | 302/363 [04:21<00:50, 1.21it/s]
Loading 0:  83%|████████▎ | 302/363 [04:21<00:50, 1.21it/s]
Loading 0:  83%|████████▎ | 303/363 [04:24<01:00, 1.01s/it]
Loading 0:  83%|████████▎ | 303/363 [04:24<01:00, 1.01s/it]
Loading 0:  85%|████████▌ | 310/363 [04:26<00:35, 1.49it/s]
Loading 0:  85%|████████▌ | 310/363 [04:26<00:35, 1.49it/s]
Loading 0:  86%|████████▌ | 311/363 [04:29<00:42, 1.21it/s]
Loading 0:  86%|████████▌ | 311/363 [04:29<00:42, 1.21it/s]
Loading 0:  86%|████████▌ | 312/363 [04:31<00:51, 1.01s/it]
Loading 0:  86%|████████▌ | 312/363 [04:31<00:51, 1.01s/it]
Loading 0:  88%|████████▊ | 319/363 [04:34<00:29, 1.49it/s]
Loading 0:  88%|████████▊ | 319/363 [04:34<00:29, 1.49it/s]
Loading 0:  88%|████████▊ | 320/363 [04:36<00:35, 1.22it/s]
Loading 0:  88%|████████▊ | 320/363 [04:36<00:35, 1.22it/s]
Loading 0:  88%|████████▊ | 321/363 [04:38<00:42, 1.00s/it]
Loading 0:  88%|████████▊ | 321/363 [04:38<00:42, 1.00s/it]
Loading 0:  90%|████████▉ | 326/363 [04:39<00:24, 1.52it/s]
Loading 0:  90%|████████▉ | 326/363 [04:39<00:24, 1.52it/s]
Loading 0:  90%|█████████ | 328/363 [04:42<00:26, 1.34it/s]
Loading 0:  90%|█████████ | 328/363 [04:42<00:26, 1.34it/s]
Loading 0:  91%|█████████ | 329/363 [04:44<00:31, 1.08it/s]
Loading 0:  91%|█████████ | 329/363 [04:44<00:31, 1.08it/s]
Loading 0:  91%|█████████ | 330/363 [04:46<00:38, 1.17s/it]
Loading 0:  91%|█████████ | 330/363 [04:46<00:38, 1.17s/it]
Loading 0:  93%|█████████▎| 337/363 [04:49<00:18, 1.42it/s]
Loading 0:  93%|█████████▎| 337/363 [04:49<00:18, 1.42it/s]
Loading 0:  93%|█████████▎| 338/363 [04:51<00:21, 1.16it/s]
Loading 0:  93%|█████████▎| 338/363 [04:51<00:21, 1.16it/s]
Loading 0:  93%|█████████▎| 339/363 [04:53<00:25, 1.05s/it]
Loading 0:  93%|█████████▎| 339/363 [04:53<00:25, 1.05s/it]
Loading 0:  95%|█████████▌| 346/363 [04:56<00:11, 1.47it/s]
Loading 0:  95%|█████████▌| 346/363 [04:56<00:11, 1.47it/s]
Loading 0:  96%|█████████▌| 347/363 [04:58<00:13, 1.20it/s]
Loading 0:  96%|█████████▌| 347/363 [04:58<00:13, 1.20it/s]
Loading 0:  96%|█████████▌| 348/363 [05:01<00:15, 1.02s/it]
Loading 0:  96%|█████████▌| 348/363 [05:01<00:15, 1.02s/it]
Loading 0:  98%|█████████▊| 355/363 [05:04<00:05, 1.48it/s]
Loading 0:  98%|█████████▊| 355/363 [05:04<00:05, 1.48it/s]
Loading 0:  98%|█████████▊| 356/363 [05:06<00:05, 1.21it/s]
Loading 0:  98%|█████████▊| 356/363 [05:06<00:05, 1.21it/s]
Loading 0:  98%|█████████▊| 357/363 [05:08<00:06, 1.01s/it]
Loading 0:  98%|█████████▊| 357/363 [05:08<00:06, 1.01s/it]
Loading 0: 100%|██████████| 363/363 [05:09<00:00, 1.78it/s]
Loading 0: 100%|██████████| 363/363 [05:09<00:00, 1.78it/s]
                                                           

Loading 0: 100%|██████████| 363/363 [05:09<00:00, 1.17it/s]
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: The tokenizer you are loading from '/tmp/tmplfa21tcb' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: quantized model in 316.408s
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: Processed model ChaiML/02f4-69d4-linear-w01 in 369.301s
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: creating bucket guanaco-mkml-models
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-02f4-69d4-linear-w01-v47/nvidia
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-02f4-69d4-linear-w01-v47/nvidia/config.json
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-02f4-69d4-linear-w01-v47/nvidia/special_tokens_map.json
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-02f4-69d4-linear-w01-v47/nvidia/tokenizer_config.json
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-02f4-69d4-linear-w01-v47/nvidia/tokenizer.json
chaiml-02f4-69d4-linear-w01-v47-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-02f4-69d4-linear-w01-v47/nvidia/flywheel_model.0.safetensors
Job chaiml-02f4-69d4-linear-w01-v47-mkmlizer completed after 420.02s with status: succeeded
Stopping job with name chaiml-02f4-69d4-linear-w01-v47-mkmlizer
Pipeline stage MKMLizer completed in 420.94s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-02f4-69d4-linear-w01-v47
Waiting for inference service chaiml-02f4-69d4-linear-w01-v47 to be ready
Inference service chaiml-02f4-69d4-linear-w01-v47 ready after 151.00398468971252s
Pipeline stage MKMLDeployer completed in 151.69s
run pipeline stage %s
Running pipeline stage StressChecker
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 14.052395343780518
10th percentile: 14.063743591308594
20th percentile: 14.086440086364746
30th percentile: 14.116229248046874
40th percentile: 14.15311107635498
50th percentile: 14.189992904663086
60th percentile: 14.21321611404419
70th percentile: 14.236439323425293
80th percentile: 14.505027151107788
90th percentile: 15.018979597091676
95th percentile: 15.275955820083619
99th percentile: 15.481536798477173
mean time: 14.421962261199951
%s, retrying in %s seconds...
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 13.792136192321777
10th percentile: 13.802680015563965
20th percentile: 13.82376766204834
30th percentile: 13.890816450119019
40th percentile: 14.003826379776001
50th percentile: 14.116836309432983
60th percentile: 14.123515176773072
70th percentile: 14.130194044113159
80th percentile: 14.226705360412598
90th percentile: 14.413049125671387
95th percentile: 14.50622100830078
99th percentile: 14.580758514404296
mean time: 14.093133306503296
%s, retrying in %s seconds...
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
{"detail":"HTTPConnectionPool(host='chaiml-prm-kimi-v1-300k-33741-v7-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
5 requests
5 failed requests
5th percentile: 13.849879026412964
10th percentile: 13.884500980377197
20th percentile: 13.953744888305664
30th percentile: 14.006207132339478
40th percentile: 14.041887712478637
50th percentile: 14.077568292617798
60th percentile: 14.190055322647094
70th percentile: 14.302542352676392
80th percentile: 14.471518945693969
90th percentile: 14.696985101699829
95th percentile: 14.80971817970276
99th percentile: 14.899904642105103
mean time: 14.23248586654663
clean up pipeline due to error=DeploymentChecksError('Unacceptable number of predict errors: 100.0%')
Shutdown handler de-registered
chaiml-02f4-69d4-linear-w01_v47 status is now failed due to DeploymentManager action
chaiml-02f4-69d4-linear-w01_v47 status is now inactive due to auto deactivation removed underperforming models
chaiml-02f4-69d4-linear-w01_v47 status is now protected due to ABTestQueueItem
chaiml-02f4-69d4-linear-w01_v47 status is now torndown due to DeploymentManager action
chaiml-02f4-69d4-linear-w01_v47 status is now inactive due to ABTestQueueItem