developer_uid: richhx
submission_id: chaiml-ocean-life25051_60844_v45
model_name: chaiml-ocean-life25051_60844_v45
model_group: ChaiML/ocean-life2505130
status: inactive
timestamp: 2026-01-21T02:16:46+00:00
num_battles: 14545
num_wins: 6647
celo_rating: 1275.56
family_friendly_score: 0.5464
family_friendly_standard_error: 0.007040554523615309
submission_type: basic
model_repo: ChaiML/ocean-life250513094323_sft
model_architecture: MistralForCausalLM
model_num_parameters: 24096691200.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
reward_model: default
display_name: chaiml-ocean-life25051_60844_v45
is_internal_developer: True
language_model: ChaiML/ocean-life250513094323_sft
model_size: 24B
ranking_group: single
us_pacific_date: 2026-01-20
win_ratio: 0.4569955311103472
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '####', 'You:', '####\n', '</s>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}</s>\n', 'user_template': 'You: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-ocean-life25051-60844-v45-mkmlizer
Waiting for job on chaiml-ocean-life25051-60844-v45-mkmlizer to finish
Stopping job with name chaiml-ocean-life25051-60844-v45-mkmlizer
%s, retrying in %s seconds...
Stopping job with name chaiml-ocean-life25051-60844-v45-mkmlizer
%s, retrying in %s seconds...
Stopping job with name chaiml-ocean-life25051-60844-v45-mkmlizer
%s, retrying in %s seconds...
Starting job with name chaiml-ocean-life25051-60844-v45-mkmlizer
Waiting for job on chaiml-ocean-life25051-60844-v45-mkmlizer to finish
Stopping job with name chaiml-ocean-life25051-60844-v45-mkmlizer
%s, retrying in %s seconds...
Stopping job with name chaiml-ocean-life25051-60844-v45-mkmlizer
%s, retrying in %s seconds...
Stopping job with name chaiml-ocean-life25051-60844-v45-mkmlizer
%s, retrying in %s seconds...
Starting job with name chaiml-ocean-life25051-60844-v45-mkmlizer
Waiting for job on chaiml-ocean-life25051-60844-v45-mkmlizer to finish
chaiml-ocean-life25051-60844-v45-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-ocean-life25051-60844-v45-mkmlizer: bash: no job control in this shell
chaiml-ocean-life25051-60844-v45-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-ocean-life25051-60844-v45-mkmlizer: ║                                                                     ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║                                                                     ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║  Version: 0.30.6+torch280-gfx942                                    ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║  Features: FLYWHEEL, CUDA, ROCM                                     ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║                                                                     ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║  belonging to:                                                      ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║                                                                     ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║  Expiration: 2028-03-31 00:00:00                                    ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ║                                                                     ║
chaiml-ocean-life25051-60844-v45-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-ocean-life25051-60844-v45-mkmlizer: Downloaded to shared memory in 64.901s
chaiml-ocean-life25051-60844-v45-mkmlizer: Checking if ChaiML/ocean-life250513094323_sft already exists in ChaiML
chaiml-ocean-life25051-60844-v45-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4z, folder:/tmp/tmpsqouqqyu, device:0
chaiml-ocean-life25051-60844-v45-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-ocean-life25051-60844-v45-mkmlizer: 
Loading 0:   0%|          | 0.00/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 3.00/363 [00:02<05:00, 1.20it/s]
Loading 0:   1%|          | 3.00/363 [00:02<05:00, 1.20it/s]
Loading 0:   1%|          | 4.00/363 [00:05<08:13, 1.37s/it]
Loading 0:   1%|          | 4.00/363 [00:05<08:13, 1.37s/it]
Loading 0:   1%|▏         | 5.00/363 [00:07<10:53, 1.83s/it]
Loading 0:   1%|▏         | 5.00/363 [00:07<10:53, 1.83s/it]
Loading 0:   3%|▎         | 10.0/363 [00:09<04:20, 1.36it/s]
Loading 0:   3%|▎         | 10.0/363 [00:09<04:20, 1.36it/s]
Loading 0:   3%|▎         | 12.0/363 [00:11<05:29, 1.06it/s]
Loading 0:   3%|▎         | 12.0/363 [00:11<05:29, 1.06it/s]
Loading 0:   4%|▎         | 13.0/363 [00:14<07:01, 1.20s/it]
Loading 0:   4%|▎         | 13.0/363 [00:14<07:01, 1.20s/it]
Loading 0:   4%|▍         | 14.0/363 [00:17<08:40, 1.49s/it]
Loading 0:   4%|▍         | 14.0/363 [00:17<08:40, 1.49s/it]
Loading 0:   5%|▌         | 19.0/363 [00:18<04:27, 1.29it/s]
Loading 0:   5%|▌         | 19.0/363 [00:18<04:27, 1.29it/s]
Loading 0:   6%|▌         | 21.0/363 [00:21<05:27, 1.04it/s]
Loading 0:   6%|▌         | 21.0/363 [00:21<05:27, 1.04it/s]
Loading 0:   6%|▌         | 22.0/363 [00:24<06:51, 1.21s/it]
Loading 0:   6%|▌         | 22.0/363 [00:24<06:51, 1.21s/it]
Loading 0:   6%|▋         | 23.0/363 [00:26<08:16, 1.46s/it]
Loading 0:   6%|▋         | 23.0/363 [00:26<08:16, 1.46s/it]
Loading 0:   8%|▊         | 28.0/363 [00:28<04:26, 1.26it/s]
Loading 0:   8%|▊         | 28.0/363 [00:28<04:26, 1.26it/s]
Loading 0:   9%|▊         | 31.0/363 [00:29<03:37, 1.53it/s]
Loading 0:   9%|▊         | 31.0/363 [00:29<03:37, 1.53it/s]
Loading 0:   9%|▉         | 34.0/363 [00:32<04:29, 1.22it/s]
Loading 0:   9%|▉         | 34.0/363 [00:32<04:29, 1.22it/s]
Loading 0:  10%|▉         | 35.0/363 [00:35<05:40, 1.04s/it]
Loading 0:  10%|▉         | 35.0/363 [00:35<05:40, 1.04s/it]
Loading 0:  10%|▉         | 36.0/363 [00:37<07:01, 1.29s/it]
Loading 0:  10%|▉         | 36.0/363 [00:37<07:01, 1.29s/it]
Loading 0:  11%|█         | 39.0/363 [00:40<06:05, 1.13s/it]
Loading 0:  11%|█         | 39.0/363 [00:40<06:05, 1.13s/it]
Loading 0:  11%|█         | 40.0/363 [00:43<07:25, 1.38s/it]
Loading 0:  11%|█         | 40.0/363 [00:43<07:25, 1.38s/it]
Loading 0:  11%|█▏        | 41.0/363 [00:46<08:52, 1.65s/it]
Loading 0:  11%|█▏        | 41.0/363 [00:46<08:52, 1.65s/it]
Loading 0:  13%|█▎        | 46.0/363 [00:47<04:30, 1.17it/s]
Loading 0:  13%|█▎        | 46.0/363 [00:47<04:30, 1.17it/s]
Loading 0:  13%|█▎        | 48.0/363 [00:50<05:21, 1.02s/it]
Loading 0:  13%|█▎        | 48.0/363 [00:50<05:21, 1.02s/it]
Loading 0:  13%|█▎        | 49.0/363 [00:53<06:39, 1.27s/it]
Loading 0:  13%|█▎        | 49.0/363 [00:53<06:39, 1.27s/it]
Loading 0:  14%|█▍        | 50.0/363 [00:55<08:04, 1.55s/it]
Loading 0:  14%|█▍        | 50.0/363 [00:55<08:04, 1.55s/it]
Loading 0:  15%|█▌        | 55.0/363 [00:57<04:11, 1.22it/s]
Loading 0:  15%|█▌        | 55.0/363 [00:57<04:11, 1.22it/s]
Loading 0:  16%|█▌        | 57.0/363 [01:00<05:04, 1.01it/s]
Loading 0:  16%|█▌        | 57.0/363 [01:00<05:04, 1.01it/s]
Loading 0:  16%|█▌        | 58.0/363 [01:02<06:23, 1.26s/it]
Loading 0:  16%|█▌        | 58.0/363 [01:02<06:23, 1.26s/it]
Loading 0:  16%|█▋        | 59.0/363 [01:05<07:46, 1.53s/it]
Loading 0:  16%|█▋        | 59.0/363 [01:05<07:46, 1.53s/it]
Loading 0:  18%|█▊        | 64.0/363 [01:07<04:06, 1.21it/s]
Loading 0:  18%|█▊        | 64.0/363 [01:07<04:06, 1.21it/s]
Loading 0:  18%|█▊        | 65.0/363 [01:10<05:34, 1.12s/it]
Loading 0:  18%|█▊        | 65.0/363 [01:10<05:34, 1.12s/it]
Loading 0:  19%|█▉        | 69.0/363 [01:11<03:43, 1.31it/s]
Loading 0:  19%|█▉        | 69.0/363 [01:11<03:43, 1.31it/s]
Loading 0:  20%|█▉        | 71.0/363 [01:14<04:47, 1.01it/s]
Loading 0:  20%|█▉        | 71.0/363 [01:14<04:47, 1.01it/s]
Loading 0:  20%|█▉        | 72.0/363 [01:17<05:59, 1.24s/it]
Loading 0:  20%|█▉        | 72.0/363 [01:17<05:59, 1.24s/it]
Loading 0:  20%|██        | 73.0/363 [01:20<07:17, 1.51s/it]
Loading 0:  20%|██        | 73.0/363 [01:20<07:17, 1.51s/it]
Loading 0:  21%|██▏       | 78.0/363 [01:21<03:52, 1.22it/s]
Loading 0:  21%|██▏       | 78.0/363 [01:21<03:52, 1.22it/s]
Loading 0:  22%|██▏       | 79.0/363 [01:24<05:18, 1.12s/it]
Loading 0:  22%|██▏       | 79.0/363 [01:24<05:18, 1.12s/it]
Loading 0:  22%|██▏       | 80.0/363 [01:27<06:34, 1.39s/it]
Loading 0:  22%|██▏       | 80.0/363 [01:27<06:34, 1.39s/it]
Loading 0:  23%|██▎       | 84.0/363 [01:28<04:00, 1.16it/s]
Loading 0:  23%|██▎       | 84.0/363 [01:28<04:00, 1.16it/s]
Loading 0:  24%|██▎       | 86.0/363 [01:31<04:43, 1.02s/it]
Loading 0:  24%|██▎       | 86.0/363 [01:31<04:43, 1.02s/it]
Loading 0:  24%|██▍       | 87.0/363 [01:34<06:03, 1.32s/it]
Loading 0:  24%|██▍       | 87.0/363 [01:34<06:03, 1.32s/it]
Loading 0:  25%|██▍       | 90.0/363 [01:37<05:13, 1.15s/it]
Loading 0:  25%|██▍       | 90.0/363 [01:37<05:13, 1.15s/it]
Loading 0:  25%|██▌       | 91.0/363 [01:39<06:19, 1.39s/it]
Loading 0:  25%|██▌       | 91.0/363 [01:39<06:19, 1.39s/it]
Loading 0:  25%|██▌       | 92.0/363 [01:42<07:30, 1.66s/it]
Loading 0:  25%|██▌       | 92.0/363 [01:42<07:30, 1.66s/it]
Loading 0:  27%|██▋       | 97.0/363 [01:43<03:45, 1.18it/s]
Loading 0:  27%|██▋       | 97.0/363 [01:43<03:45, 1.18it/s]
Loading 0:  27%|██▋       | 99.0/363 [01:46<04:29, 1.02s/it]
Loading 0:  27%|██▋       | 99.0/363 [01:46<04:29, 1.02s/it]
Loading 0:  28%|██▊       | 100/363 [01:49<05:35, 1.27s/it] 
Loading 0:  28%|██▊       | 100/363 [01:49<05:35, 1.27s/it]
Loading 0:  28%|██▊       | 101/363 [01:52<06:47, 1.56s/it]
Loading 0:  28%|██▊       | 101/363 [01:52<06:47, 1.56s/it]
Loading 0:  29%|██▉       | 106/363 [01:53<03:31, 1.21it/s]
Loading 0:  29%|██▉       | 106/363 [01:53<03:31, 1.21it/s]
Loading 0:  30%|██▉       | 108/363 [01:57<04:30, 1.06s/it]
Loading 0:  30%|██▉       | 108/363 [01:57<04:30, 1.06s/it]
Loading 0:  31%|███       | 111/363 [02:00<04:13, 1.01s/it]
Loading 0:  31%|███       | 111/363 [02:00<04:13, 1.01s/it]
Loading 0:  31%|███       | 112/363 [02:02<05:11, 1.24s/it]
Loading 0:  31%|███       | 112/363 [02:02<05:11, 1.24s/it]
Loading 0:  31%|███       | 113/363 [02:05<06:16, 1.51s/it]
Loading 0:  31%|███       | 113/363 [02:05<06:16, 1.51s/it]
Loading 0:  33%|███▎      | 118/363 [02:06<03:22, 1.21it/s]
Loading 0:  33%|███▎      | 118/363 [02:06<03:22, 1.21it/s]
Loading 0:  33%|███▎      | 120/363 [02:09<03:59, 1.02it/s]
Loading 0:  33%|███▎      | 120/363 [02:09<03:59, 1.02it/s]
Loading 0:  33%|███▎      | 121/363 [02:12<04:58, 1.23s/it]
Loading 0:  33%|███▎      | 121/363 [02:12<04:58, 1.23s/it]
Loading 0:  34%|███▎      | 122/363 [02:15<06:03, 1.51s/it]
Loading 0:  34%|███▎      | 122/363 [02:15<06:03, 1.51s/it]
Loading 0:  35%|███▍      | 127/363 [02:16<03:13, 1.22it/s]
Loading 0:  35%|███▍      | 127/363 [02:16<03:13, 1.22it/s]
Loading 0:  36%|███▌      | 129/363 [02:19<03:46, 1.03it/s]
Loading 0:  36%|███▌      | 129/363 [02:19<03:46, 1.03it/s]
Loading 0:  36%|███▌      | 130/363 [02:22<04:44, 1.22s/it]
Loading 0:  36%|███▌      | 130/363 [02:22<04:44, 1.22s/it]
Loading 0:  36%|███▌      | 131/363 [02:25<05:47, 1.50s/it]
Loading 0:  36%|███▌      | 131/363 [02:25<05:47, 1.50s/it]
Loading 0:  37%|███▋      | 136/363 [02:26<03:04, 1.23it/s]
Loading 0:  37%|███▋      | 136/363 [02:26<03:04, 1.23it/s]
Loading 0:  38%|███▊      | 138/363 [02:29<03:39, 1.02it/s]
Loading 0:  38%|███▊      | 138/363 [02:29<03:39, 1.02it/s]
Loading 0:  38%|███▊      | 139/363 [02:31<04:35, 1.23s/it]
Loading 0:  38%|███▊      | 139/363 [02:31<04:35, 1.23s/it]
Loading 0:  39%|███▊      | 140/363 [02:34<05:35, 1.51s/it]
Loading 0:  39%|███▊      | 140/363 [02:34<05:35, 1.51s/it]
Loading 0:  40%|███▉      | 145/363 [02:35<02:55, 1.24it/s]
Loading 0:  40%|███▉      | 145/363 [02:35<02:55, 1.24it/s]
Loading 0:  41%|████      | 148/363 [02:37<02:22, 1.51it/s]
Loading 0:  41%|████      | 148/363 [02:37<02:22, 1.51it/s]
Loading 0:  42%|████▏     | 151/363 [02:40<03:01, 1.17it/s]
Loading 0:  42%|████▏     | 151/363 [02:40<03:01, 1.17it/s]
Loading 0:  42%|████▏     | 152/363 [02:43<03:48, 1.08s/it]
Loading 0:  42%|████▏     | 152/363 [02:43<03:48, 1.08s/it]
Loading 0:  42%|████▏     | 153/363 [02:46<04:47, 1.37s/it]
Loading 0:  42%|████▏     | 153/363 [02:46<04:47, 1.37s/it]
Loading 0:  43%|████▎     | 156/363 [02:49<04:06, 1.19s/it]
Loading 0:  43%|████▎     | 156/363 [02:49<04:06, 1.19s/it]
Loading 0:  43%|████▎     | 157/363 [02:51<04:54, 1.43s/it]
Loading 0:  43%|████▎     | 157/363 [02:51<04:54, 1.43s/it]
Loading 0:  44%|████▎     | 158/363 [02:54<05:46, 1.69s/it]
Loading 0:  44%|████▎     | 158/363 [02:54<05:46, 1.69s/it]
Loading 0:  45%|████▍     | 163/363 [02:56<02:54, 1.15it/s]
Loading 0:  45%|████▍     | 163/363 [02:56<02:54, 1.15it/s]
Loading 0:  45%|████▌     | 165/363 [02:58<03:23, 1.03s/it]
Loading 0:  45%|████▌     | 165/363 [02:58<03:23, 1.03s/it]
Loading 0:  46%|████▌     | 166/363 [03:01<04:11, 1.28s/it]
Loading 0:  46%|████▌     | 166/363 [03:01<04:11, 1.28s/it]
Loading 0:  46%|████▌     | 167/363 [03:04<05:04, 1.55s/it]
Loading 0:  46%|████▌     | 167/363 [03:04<05:04, 1.55s/it]
Loading 0:  47%|████▋     | 172/363 [03:05<02:36, 1.22it/s]
Loading 0:  47%|████▋     | 172/363 [03:05<02:36, 1.22it/s]
Loading 0:  48%|████▊     | 174/363 [03:08<03:05, 1.02it/s]
Loading 0:  48%|████▊     | 174/363 [03:08<03:05, 1.02it/s]
Loading 0:  48%|████▊     | 175/363 [03:11<03:51, 1.23s/it]
Loading 0:  48%|████▊     | 175/363 [03:11<03:51, 1.23s/it]
Loading 0:  48%|████▊     | 176/363 [03:14<04:42, 1.51s/it]
Loading 0:  48%|████▊     | 176/363 [03:14<04:42, 1.51s/it]
Loading 0:  50%|████▉     | 181/363 [03:15<02:26, 1.24it/s]
Loading 0:  50%|████▉     | 181/363 [03:15<02:26, 1.24it/s]
Loading 0:  50%|█████     | 182/363 [03:18<03:19, 1.10s/it]
Loading 0:  50%|█████     | 182/363 [03:18<03:19, 1.10s/it]
Loading 0:  51%|█████     | 186/363 [03:19<02:13, 1.33it/s]
Loading 0:  51%|█████     | 186/363 [03:19<02:13, 1.33it/s]
Loading 0:  52%|█████▏    | 188/363 [03:23<02:53, 1.01it/s]
Loading 0:  52%|█████▏    | 188/363 [03:23<02:53, 1.01it/s]
Loading 0:  52%|█████▏    | 189/363 [03:26<03:41, 1.27s/it]
Loading 0:  52%|█████▏    | 189/363 [03:26<03:41, 1.27s/it]
Loading 0:  53%|█████▎    | 192/363 [03:28<03:12, 1.13s/it]
Loading 0:  53%|█████▎    | 192/363 [03:28<03:12, 1.13s/it]
Loading 0:  53%|█████▎    | 193/363 [03:31<03:52, 1.37s/it]
Loading 0:  53%|█████▎    | 193/363 [03:31<03:52, 1.37s/it]
Loading 0:  53%|█████▎    | 194/363 [03:34<04:36, 1.63s/it]
Loading 0:  53%|█████▎    | 194/363 [03:34<04:36, 1.63s/it]
Loading 0:  55%|█████▍    | 199/363 [03:35<02:20, 1.17it/s]
Loading 0:  55%|█████▍    | 199/363 [03:35<02:20, 1.17it/s]
Loading 0:  55%|█████▌    | 201/363 [03:38<02:42, 1.00s/it]
Loading 0:  55%|█████▌    | 201/363 [03:38<02:42, 1.00s/it]
Loading 0:  56%|█████▌    | 202/363 [03:41<03:19, 1.24s/it]
Loading 0:  56%|█████▌    | 202/363 [03:41<03:19, 1.24s/it]
Loading 0:  56%|█████▌    | 203/363 [03:43<04:02, 1.51s/it]
Loading 0:  56%|█████▌    | 203/363 [03:43<04:02, 1.51s/it]
Loading 0:  57%|█████▋    | 208/363 [03:45<02:06, 1.23it/s]
Loading 0:  57%|█████▋    | 208/363 [03:45<02:06, 1.23it/s]
Loading 0:  58%|█████▊    | 210/363 [03:47<02:28, 1.03it/s]
Loading 0:  58%|█████▊    | 210/363 [03:47<02:28, 1.03it/s]
Loading 0:  58%|█████▊    | 211/363 [03:50<03:05, 1.22s/it]
Loading 0:  58%|█████▊    | 211/363 [03:50<03:05, 1.22s/it]
Loading 0:  58%|█████▊    | 212/363 [03:53<03:46, 1.50s/it]
Loading 0:  58%|█████▊    | 212/363 [03:53<03:46, 1.50s/it]
Loading 0:  60%|█████▉    | 217/363 [03:54<01:56, 1.25it/s]
Loading 0:  60%|█████▉    | 217/363 [03:54<01:56, 1.25it/s]
Loading 0:  60%|██████    | 218/363 [03:57<02:35, 1.07s/it]
Loading 0:  60%|██████    | 218/363 [03:57<02:35, 1.07s/it]
Loading 0:  60%|██████    | 219/363 [04:00<03:11, 1.33s/it]
Loading 0:  60%|██████    | 219/363 [04:00<03:11, 1.33s/it]
Loading 0:  61%|██████▏   | 223/363 [04:01<01:57, 1.19it/s]
Loading 0:  61%|██████▏   | 223/363 [04:01<01:57, 1.19it/s]
Loading 0:  62%|██████▏   | 225/363 [04:04<02:25, 1.05s/it]
Loading 0:  62%|██████▏   | 225/363 [04:04<02:25, 1.05s/it]
Loading 0:  63%|██████▎   | 228/363 [04:07<02:12, 1.02it/s]
Loading 0:  63%|██████▎   | 228/363 [04:07<02:12, 1.02it/s]
Loading 0:  63%|██████▎   | 229/363 [04:09<02:43, 1.22s/it]
Loading 0:  63%|██████▎   | 229/363 [04:09<02:43, 1.22s/it]
Loading 0:  63%|██████▎   | 230/363 [04:12<03:14, 1.46s/it]
Loading 0:  63%|██████▎   | 230/363 [04:12<03:14, 1.46s/it]
Loading 0:  65%|██████▍   | 235/363 [04:13<01:41, 1.26it/s]
Loading 0:  65%|██████▍   | 235/363 [04:13<01:41, 1.26it/s]
Loading 0:  65%|██████▌   | 237/363 [04:16<02:01, 1.04it/s]
Loading 0:  65%|██████▌   | 237/363 [04:16<02:01, 1.04it/s]
Loading 0:  66%|██████▌   | 238/363 [04:19<02:31, 1.22s/it]
Loading 0:  66%|██████▌   | 238/363 [04:19<02:31, 1.22s/it]
Loading 0:  66%|██████▌   | 239/363 [04:22<03:05, 1.50s/it]
Loading 0:  66%|██████▌   | 239/363 [04:22<03:05, 1.50s/it]
Loading 0:  67%|██████▋   | 244/363 [04:23<01:35, 1.25it/s]
Loading 0:  67%|██████▋   | 244/363 [04:23<01:35, 1.25it/s]
Loading 0:  68%|██████▊   | 246/363 [04:26<01:53, 1.03it/s]
Loading 0:  68%|██████▊   | 246/363 [04:26<01:53, 1.03it/s]
Loading 0:  68%|██████▊   | 247/363 [04:29<02:21, 1.22s/it]
Loading 0:  68%|██████▊   | 247/363 [04:29<02:21, 1.22s/it]
Loading 0:  68%|██████▊   | 248/363 [04:32<02:52, 1.50s/it]
Loading 0:  68%|██████▊   | 248/363 [04:32<02:52, 1.50s/it]
Loading 0:  70%|██████▉   | 253/363 [04:33<01:28, 1.24it/s]
Loading 0:  70%|██████▉   | 253/363 [04:33<01:28, 1.24it/s]
Loading 0:  70%|███████   | 255/363 [04:36<01:45, 1.02it/s]
Loading 0:  70%|███████   | 255/363 [04:36<01:45, 1.02it/s]
Loading 0:  71%|███████   | 256/363 [04:39<02:11, 1.23s/it]
Loading 0:  71%|███████   | 256/363 [04:39<02:11, 1.23s/it]
Loading 0:  71%|███████   | 257/363 [04:41<02:39, 1.50s/it]
Loading 0:  71%|███████   | 257/363 [04:41<02:39, 1.50s/it]
Loading 0:  72%|███████▏  | 262/363 [04:43<01:21, 1.23it/s]
Loading 0:  72%|███████▏  | 262/363 [04:43<01:21, 1.23it/s]
Loading 0:  73%|███████▎  | 265/363 [04:44<01:05, 1.49it/s]
Loading 0:  73%|███████▎  | 265/363 [04:44<01:05, 1.49it/s]
Loading 0:  74%|███████▍  | 268/363 [04:47<01:20, 1.18it/s]
Loading 0:  74%|███████▍  | 268/363 [04:47<01:20, 1.18it/s]
Loading 0:  74%|███████▍  | 269/363 [04:50<01:42, 1.09s/it]
Loading 0:  74%|███████▍  | 269/363 [04:50<01:42, 1.09s/it]
Loading 
chaiml-ocean-life25051-60844-v45-mkmlizer: 0:  74%|███████▍  | 270/363 [04:53<02:07, 1.37s/it]
Loading 0:  74%|███████▍  | 270/363 [04:53<02:07, 1.37s/it]
Loading 0:  75%|███████▌  | 273/363 [04:56<01:46, 1.19s/it]
Loading 0:  75%|███████▌  | 273/363 [04:56<01:46, 1.19s/it]
Loading 0:  75%|███████▌  | 274/363 [04:58<02:07, 1.43s/it]
Loading 0:  75%|███████▌  | 274/363 [04:58<02:07, 1.43s/it]
Loading 0:  76%|███████▌  | 275/363 [05:01<02:28, 1.69s/it]
Loading 0:  76%|███████▌  | 275/363 [05:01<02:28, 1.69s/it]
Loading 0:  77%|███████▋  | 280/363 [05:02<01:12, 1.15it/s]
Loading 0:  77%|███████▋  | 280/363 [05:02<01:12, 1.15it/s]
Loading 0:  78%|███████▊  | 282/363 [05:06<01:22, 1.02s/it]
Loading 0:  78%|███████▊  | 282/363 [05:06<01:22, 1.02s/it]
Loading 0:  78%|███████▊  | 283/363 [05:08<01:41, 1.27s/it]
Loading 0:  78%|███████▊  | 283/363 [05:08<01:41, 1.27s/it]
Loading 0:  78%|███████▊  | 284/363 [05:11<02:02, 1.55s/it]
Loading 0:  78%|███████▊  | 284/363 [05:11<02:02, 1.55s/it]
Loading 0:  80%|███████▉  | 289/363 [05:12<01:01, 1.21it/s]
Loading 0:  80%|███████▉  | 289/363 [05:12<01:01, 1.21it/s]
Loading 0:  80%|████████  | 291/363 [05:27<02:50, 2.37s/it]
Loading 0:  80%|████████  | 291/363 [05:27<02:50, 2.37s/it]
Loading 0:  80%|████████  | 292/363 [05:29<02:50, 2.40s/it]
Loading 0:  80%|████████  | 292/363 [05:29<02:50, 2.40s/it]
Loading 0:  81%|████████  | 293/363 [05:32<02:52, 2.46s/it]
Loading 0:  81%|████████  | 293/363 [05:32<02:52, 2.46s/it]
Loading 0:  82%|████████▏ | 299/363 [05:36<01:26, 1.35s/it]
Loading 0:  82%|████████▏ | 299/363 [05:36<01:26, 1.35s/it]
Loading 0:  83%|████████▎ | 303/363 [05:37<00:57, 1.04it/s]
Loading 0:  83%|████████▎ | 303/363 [05:37<00:57, 1.04it/s]
Loading 0:  84%|████████▍ | 305/363 [05:40<01:04, 1.11s/it]
Loading 0:  84%|████████▍ | 305/363 [05:40<01:04, 1.11s/it]
Loading 0:  84%|████████▍ | 306/363 [05:43<01:15, 1.32s/it]
Loading 0:  84%|████████▍ | 306/363 [05:43<01:15, 1.32s/it]
Loading 0:  85%|████████▌ | 309/363 [05:45<01:02, 1.15s/it]
Loading 0:  85%|████████▌ | 309/363 [05:45<01:02, 1.15s/it]
Loading 0:  85%|████████▌ | 310/363 [05:48<01:11, 1.35s/it]
Loading 0:  85%|████████▌ | 310/363 [05:48<01:11, 1.35s/it]
Loading 0:  86%|████████▌ | 311/363 [05:51<01:21, 1.57s/it]
Loading 0:  86%|████████▌ | 311/363 [05:51<01:21, 1.57s/it]
Loading 0:  88%|████████▊ | 318/363 [05:54<00:40, 1.11it/s]
Loading 0:  88%|████████▊ | 318/363 [05:54<00:40, 1.11it/s]
Loading 0:  88%|████████▊ | 319/363 [05:57<00:47, 1.09s/it]
Loading 0:  88%|████████▊ | 319/363 [05:57<00:47, 1.09s/it]
Loading 0:  88%|████████▊ | 320/363 [05:59<00:55, 1.30s/it]
Loading 0:  88%|████████▊ | 320/363 [05:59<00:55, 1.30s/it]
Loading 0:  90%|████████▉ | 326/363 [06:00<00:25, 1.47it/s]
Loading 0:  90%|████████▉ | 326/363 [06:00<00:25, 1.47it/s]
Loading 0:  90%|█████████ | 327/363 [06:03<00:32, 1.10it/s]
Loading 0:  90%|█████████ | 327/363 [06:03<00:32, 1.10it/s]
Loading 0:  90%|█████████ | 328/363 [06:05<00:39, 1.13s/it]
Loading 0:  90%|█████████ | 328/363 [06:05<00:39, 1.13s/it]
Loading 0:  91%|█████████ | 329/363 [06:08<00:46, 1.37s/it]
Loading 0:  91%|█████████ | 329/363 [06:08<00:46, 1.37s/it]
Loading 0:  92%|█████████▏| 335/363 [06:12<00:26, 1.08it/s]
Loading 0:  92%|█████████▏| 335/363 [06:12<00:26, 1.08it/s]
Loading 0:  93%|█████████▎| 336/363 [06:14<00:30, 1.13s/it]
Loading 0:  93%|█████████▎| 336/363 [06:14<00:30, 1.13s/it]
Loading 0:  94%|█████████▍| 341/363 [06:16<00:17, 1.28it/s]
Loading 0:  94%|█████████▍| 341/363 [06:16<00:17, 1.28it/s]
Loading 0:  94%|█████████▍| 343/363 [06:19<00:18, 1.11it/s]
Loading 0:  94%|█████████▍| 343/363 [06:19<00:18, 1.11it/s]
Loading 0:  95%|█████████▌| 346/363 [06:21<00:14, 1.14it/s]
Loading 0:  95%|█████████▌| 346/363 [06:21<00:14, 1.14it/s]
Loading 0:  96%|█████████▌| 347/363 [06:24<00:17, 1.10s/it]
Loading 0:  96%|█████████▌| 347/363 [06:24<00:17, 1.10s/it]
Loading 0:  96%|█████████▌| 348/363 [06:27<00:20, 1.36s/it]
Loading 0:  96%|█████████▌| 348/363 [06:27<00:20, 1.36s/it]
Loading 0:  97%|█████████▋| 353/363 [06:28<00:07, 1.33it/s]
Loading 0:  97%|█████████▋| 353/363 [06:28<00:07, 1.33it/s]
Loading 0:  98%|█████████▊| 355/363 [06:30<00:07, 1.14it/s]
Loading 0:  98%|█████████▊| 355/363 [06:30<00:07, 1.14it/s]
Loading 0:  98%|█████████▊| 356/363 [06:33<00:07, 1.12s/it]
Loading 0:  98%|█████████▊| 356/363 [06:33<00:07, 1.12s/it]
Loading 0:  98%|█████████▊| 357/363 [06:36<00:08, 1.36s/it]
Loading 0:  98%|█████████▊| 357/363 [06:36<00:08, 1.36s/it]
Loading 0: 100%|█████████▉| 362/363 [06:37<00:00, 1.36it/s]
Loading 0: 100%|█████████▉| 362/363 [06:37<00:00, 1.36it/s]
Loading 0: 100%|██████████| 363/363 [06:37<00:00, 1.54it/s]
Loading 0: 100%|██████████| 363/363 [06:37<00:00, 1.54it/s]
                                                           

Loading 0: 100%|██████████| 363/363 [06:37<00:00, 1.09s/it]
chaiml-ocean-life25051-60844-v45-mkmlizer: The tokenizer you are loading from '/tmp/tmpsqouqqyu' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-ocean-life25051-60844-v45-mkmlizer: quantized model in 408.886s
chaiml-ocean-life25051-60844-v45-mkmlizer: Processed model ChaiML/ocean-life250513094323_sft in 473.788s
chaiml-ocean-life25051-60844-v45-mkmlizer: creating bucket guanaco-mkml-models
chaiml-ocean-life25051-60844-v45-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-ocean-life25051-60844-v45-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-ocean-life25051-60844-v45/amd
chaiml-ocean-life25051-60844-v45-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-ocean-life25051-60844-v45/amd/config.json
chaiml-ocean-life25051-60844-v45-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-ocean-life25051-60844-v45/amd/special_tokens_map.json
chaiml-ocean-life25051-60844-v45-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-ocean-life25051-60844-v45/amd/tokenizer_config.json
chaiml-ocean-life25051-60844-v45-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-ocean-life25051-60844-v45/amd/tokenizer.json
chaiml-ocean-life25051-60844-v45-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-ocean-life25051-60844-v45/amd/flywheel_model.1.safetensors
chaiml-ocean-life25051-60844-v45-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-ocean-life25051-60844-v45/amd/flywheel_model.0.safetensors
Job chaiml-ocean-life25051-60844-v45-mkmlizer completed after 1124.02s with status: succeeded
Stopping job with name chaiml-ocean-life25051-60844-v45-mkmlizer
Pipeline stage MKMLizer completed in 1816.65s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.25s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-ocean-life25051-60844-v45
Waiting for inference service chaiml-ocean-life25051-60844-v45 to be ready
Inference service chaiml-ocean-life25051-60844-v45 ready after 555.173467874527s
Pipeline stage MKMLDeployer completed in 556.76s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.8287811279296875s
Received healthy response to inference request in 1.4488563537597656s
Received healthy response to inference request in 1.4983446598052979s
Received healthy response to inference request in 1.371039867401123s
Received healthy response to inference request in 1.4696731567382812s
5 requests
0 failed requests
5th percentile: 1.3866031646728516
10th percentile: 1.4021664619445802
20th percentile: 1.433293056488037
30th percentile: 1.4530197143554688
40th percentile: 1.461346435546875
50th percentile: 1.4696731567382812
60th percentile: 1.4811417579650878
70th percentile: 1.4926103591918944
80th percentile: 1.9644319534301762
90th percentile: 2.8966065406799317
95th percentile: 3.3626938343048094
99th percentile: 3.735563669204712
mean time: 1.923339033126831
Pipeline stage StressChecker completed in 12.10s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.45s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 1.08s
Shutdown handler de-registered
chaiml-ocean-life25051_60844_v45 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.08s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-ocean-life25051-60844-v45-profiler
Waiting for inference service chaiml-ocean-life25051-60844-v45-profiler to be ready
Tearing down inference service chaiml-ocean-life25051-60844-v45-profiler
%s, retrying in %s seconds...
Creating inference service chaiml-ocean-life25051-60844-v45-profiler
Waiting for inference service chaiml-ocean-life25051-60844-v45-profiler to be ready
Tearing down inference service chaiml-ocean-life25051-60844-v45-profiler
%s, retrying in %s seconds...
Creating inference service chaiml-ocean-life25051-60844-v45-profiler
Waiting for inference service chaiml-ocean-life25051-60844-v45-profiler to be ready
Tearing down inference service chaiml-ocean-life25051-60844-v45-profiler
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService chaiml-ocean-life25051-60844-v45-profiler.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'1\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\'}, \'creationTimestamp\': \'2026-01-21T02:05:23Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}}, \'f:labels\': {\'.\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2026-01-21T02:05:23Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2026-01-21T02:05:23Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2026-01-21T02:05:24Z\'}], \'name\': \'chaiml-ocean-life25051-60844-v45-profiler\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'312072249\', \'uid\': \'aa07c136-6187-4e2b-a0f5-ece124705cc0\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'L40S\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'L40S\', \'A100_NVLINK_80GB\', \'RTX_A6000\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'args\': [\'echo "downloading $TENSORIZER_URI to $DOWNLOAD_TO_LOCAL" && time s5cmd --log debug --credentials-file /code/guanaco/guanaco_inference_services/uploading/s5cfg --endpoint-url https://object.ord1.coreweave.com cp --concurrency 4 "$TENSORIZER_URI/*" $DOWNLOAD_TO_LOCAL && python3 /code/guanaco/guanaco_inference_services/src/mkml_inference_service/main.py\'], \'command\': [\'/bin/bash\', \'-i\', \'-c\'], \'env\': [{\'name\': \'MAX_TOKEN_INPUT\', \'value\': \'1024\'}, {\'name\': \'BEST_OF\', \'value\': \'8\'}, {\'name\': \'TEMPERATURE\', \'value\': \'1.0\'}, {\'name\': \'PRESENCE_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'FREQUENCY_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'TOP_P\', \'value\': \'1.0\'}, {\'name\': \'MIN_P\', \'value\': \'0.0\'}, {\'name\': \'TOP_K\', \'value\': \'40\'}, {\'name\': \'STOPPING_WORDS\', \'value\': \'["</s>", "####\\\\\\\\n", "You:", "####", "\\\\\\\\n"]\'}, {\'name\': \'MAX_TOKENS\', \'value\': \'64\'}, {\'name\': \'MAX_BATCH_SIZE\', \'value\': \'128\'}, {\'name\': \'MAX_CACHED_RESPONSES\', \'value\': \'-1\'}, {\'name\': \'URL_ROUTE\', \'value\': \'GPT-J-6B-lit-v2\'}, {\'name\': \'OBJ_ACCESS_KEY_ID\', \'value\': \'LETMTTRMLFFAMTBK\'}, {\'name\': \'OBJ_SECRET_ACCESS_KEY\', \'value\': \'VwwZaqefOOoaouNxUk03oUmK9pVEfruJhjBHPGdgycK\'}, {\'name\': \'OBJ_ENDPOINT\', \'value\': \'https://accel-object.ord1.coreweave.com\'}, {\'name\': \'TENSORIZER_URI\', \'value\': \'s3://guanaco-mkml-models/chaiml-ocean-life25051-60844-v45/amd\'}, {\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'1\'}, {\'name\': \'MK1_QUANTIZATION_PROFILE\', \'value\': \'q4z\'}, {\'name\': \'MK1_MKML_LICENSE_KEY\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'key\', \'name\': \'mkml-license-key\'}}}], \'image\': \'gcr.io/chai-959f8/chai-guanaco/chai-guanaco:v20260113\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'exec\': {\'command\': [\'cat\', \'/tmp/ready\']}, \'failureThreshold\': 1, \'initialDelaySeconds\': 10, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'2\', \'memory\': \'26Gi\', \'nvidia.com/gpu\': \'1\'}, \'requests\': {\'cpu\': \'2\', \'memory\': \'26Gi\', \'nvidia.com/gpu\': \'1\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 1, \'minReplicas\': 1, \'timeout\': 60, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\'}, \'name\': \'shared-memory-cache\'}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-ocean-life25051-60844-v45-profiler-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2026-01-21T02:05:24Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2026-01-21T02:05:24Z\', \'message\': \'Revision "chaiml-ocean-life25051-60844-v45-profiler-predictor-00001" failed with message: 0/61 nodes are available: 61 node(s) didn\\\'t match Pod\\\'s node affinity/selector. preemption: 0/61 nodes are available: 61 Preemption is not helpful for scheduling..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2026-01-21T02:05:24Z\', \'message\': \'Configuration "chaiml-ocean-life25051-60844-v45-profiler-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2026-01-21T02:05:24Z\', \'message\': \'Configuration "chaiml-ocean-life25051-60844-v45-profiler-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2026-01-21T02:05:24Z\', \'message\': \'Configuration "chaiml-ocean-life25051-60844-v45-profiler-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2026-01-21T02:05:24Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.09s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-ocean-life25051-60844-v45-profiler
Waiting for inference service chaiml-ocean-life25051-60844-v45-profiler to be ready
Tearing down inference service chaiml-ocean-life25051-60844-v45-profiler
%s, retrying in %s seconds...
Creating inference service chaiml-ocean-life25051-60844-v45-profiler
Waiting for inference service chaiml-ocean-life25051-60844-v45-profiler to be ready
Tearing down inference service chaiml-ocean-life25051-60844-v45-profiler
%s, retrying in %s seconds...
Creating inference service chaiml-ocean-life25051-60844-v45-profiler
Waiting for inference service chaiml-ocean-life25051-60844-v45-profiler to be ready
Tearing down inference service chaiml-ocean-life25051-60844-v45-profiler
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService chaiml-ocean-life25051-60844-v45-profiler.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'1\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\'}, \'creationTimestamp\': \'2026-01-21T02:36:51Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}}, \'f:labels\': {\'.\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2026-01-21T02:36:51Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2026-01-21T02:36:51Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2026-01-21T02:36:54Z\'}], \'name\': \'chaiml-ocean-life25051-60844-v45-profiler\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'312107058\', \'uid\': \'ffd4c27a-2af4-46bf-8b8f-ac8445c2d05f\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'L40S\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'L40S\', \'A100_NVLINK_80GB\', \'RTX_A6000\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'args\': [\'echo "downloading $TENSORIZER_URI to $DOWNLOAD_TO_LOCAL" && time s5cmd --log debug --credentials-file /code/guanaco/guanaco_inference_services/uploading/s5cfg --endpoint-url https://object.ord1.coreweave.com cp --concurrency 4 "$TENSORIZER_URI/*" $DOWNLOAD_TO_LOCAL && python3 /code/guanaco/guanaco_inference_services/src/mkml_inference_service/main.py\'], \'command\': [\'/bin/bash\', \'-i\', \'-c\'], \'env\': [{\'name\': \'MAX_TOKEN_INPUT\', \'value\': \'1024\'}, {\'name\': \'BEST_OF\', \'value\': \'8\'}, {\'name\': \'TEMPERATURE\', \'value\': \'1.0\'}, {\'name\': \'PRESENCE_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'FREQUENCY_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'TOP_P\', \'value\': \'1.0\'}, {\'name\': \'MIN_P\', \'value\': \'0.0\'}, {\'name\': \'TOP_K\', \'value\': \'40\'}, {\'name\': \'STOPPING_WORDS\', \'value\': \'["####\\\\\\\\n", "</s>", "\\\\\\\\n", "You:", "####"]\'}, {\'name\': \'MAX_TOKENS\', \'value\': \'64\'}, {\'name\': \'MAX_BATCH_SIZE\', \'value\': \'128\'}, {\'name\': \'MAX_CACHED_RESPONSES\', \'value\': \'-1\'}, {\'name\': \'URL_ROUTE\', \'value\': \'GPT-J-6B-lit-v2\'}, {\'name\': \'OBJ_ACCESS_KEY_ID\', \'value\': \'LETMTTRMLFFAMTBK\'}, {\'name\': \'OBJ_SECRET_ACCESS_KEY\', \'value\': \'VwwZaqefOOoaouNxUk03oUmK9pVEfruJhjBHPGdgycK\'}, {\'name\': \'OBJ_ENDPOINT\', \'value\': \'https://accel-object.ord1.coreweave.com\'}, {\'name\': \'TENSORIZER_URI\', \'value\': \'s3://guanaco-mkml-models/chaiml-ocean-life25051-60844-v45/amd\'}, {\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'1\'}, {\'name\': \'MK1_QUANTIZATION_PROFILE\', \'value\': \'q4z\'}, {\'name\': \'MK1_MKML_LICENSE_KEY\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'key\', \'name\': \'mkml-license-key\'}}}], \'image\': \'gcr.io/chai-959f8/chai-guanaco/chai-guanaco:v20260113\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'exec\': {\'command\': [\'cat\', \'/tmp/ready\']}, \'failureThreshold\': 1, \'initialDelaySeconds\': 10, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'2\', \'memory\': \'26Gi\', \'nvidia.com/gpu\': \'1\'}, \'requests\': {\'cpu\': \'2\', \'memory\': \'26Gi\', \'nvidia.com/gpu\': \'1\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 1, \'minReplicas\': 1, \'timeout\': 60, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\'}, \'name\': \'shared-memory-cache\'}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-ocean-life25051-60844-v45-profiler-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2026-01-21T02:36:54Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2026-01-21T02:36:54Z\', \'message\': \'Revision "chaiml-ocean-life25051-60844-v45-profiler-predictor-00001" failed with message: 0/61 nodes are available: 61 node(s) didn\\\'t match Pod\\\'s node affinity/selector. preemption: 0/61 nodes are available: 61 Preemption is not helpful for scheduling..\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2026-01-21T02:36:54Z\', \'message\': \'Configuration "chaiml-ocean-life25051-60844-v45-profiler-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2026-01-21T02:36:54Z\', \'message\': \'Configuration "chaiml-ocean-life25051-60844-v45-profiler-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2026-01-21T02:36:54Z\', \'message\': \'Configuration "chaiml-ocean-life25051-60844-v45-profiler-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2026-01-21T02:36:54Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
Shutdown handler de-registered