developer_uid: zonemercy
submission_id: chaiml-kimid-v2-cosine-_80852_v5
model_name: test
model_group: ChaiML/kimid-v2_cosine-l
status: torndown
timestamp: 2025-12-12T18:27:56+00:00
num_battles: 7532
num_wins: 3809
celo_rating: 1304.97
family_friendly_score: 0.5174000000000001
family_friendly_standard_error: 0.007066784841779181
submission_type: basic
model_repo: ChaiML/kimid-v2_cosine-lr2e6g16
model_architecture: MistralForCausalLM
model_num_parameters: 24096691200.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 72
reward_model: default
latencies: [{'batch_size': 1, 'throughput': 0.42488197350118917, 'latency_mean': 2.353516048192978, 'latency_p50': 2.3355013132095337, 'latency_p90': 2.5890088796615602}, {'batch_size': 2, 'throughput': 0.630964116507844, 'latency_mean': 3.1645771718025206, 'latency_p50': 3.1558653116226196, 'latency_p90': 3.394986319541931}, {'batch_size': 3, 'throughput': 0.7682542171679472, 'latency_mean': 3.892492957115173, 'latency_p50': 3.8979198932647705, 'latency_p90': 4.298365879058838}, {'batch_size': 4, 'throughput': 0.8641367987716607, 'latency_mean': 4.601756418943405, 'latency_p50': 4.591245651245117, 'latency_p90': 5.211831259727478}, {'batch_size': 5, 'throughput': 0.9234715204372735, 'latency_mean': 5.389990454912185, 'latency_p50': 5.3801082372665405, 'latency_p90': 6.1155890941619875}]
gpu_counts: {'NVIDIA L40S': 1}
display_name: test
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/kimid-v2_cosine-lr2e6g16
model_size: 24B
ranking_group: single
throughput_3p7s: 0.74
us_pacific_date: 2025-12-08
win_ratio: 0.50570897503983
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '</s>', 'User:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 72}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '####\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-kimid-v2-cosine-80852-v5-mkmlizer
Waiting for job on chaiml-kimid-v2-cosine-80852-v5-mkmlizer to finish
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: bash: no job control in this shell
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v2-cosine-80852-v5-mkmlizer:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║                                                                     ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║                                                                     ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║  Version: 0.30.6+torch280                                           ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║                                                                     ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║  belonging to:                                                      ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║                                                                     ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ║                                                                     ║
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: Downloaded to shared memory in 53.300s
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: Checking if ChaiML/kimid-v2_cosine-lr2e6g16 already exists in ChaiML
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmpn3w1hnij, device:0
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: 
Loading 0:   0%|          | 0.00/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 3.00/363 [00:01<03:48, 1.57it/s]
Loading 0:   1%|          | 3.00/363 [00:01<03:48, 1.57it/s]
Loading 0:   1%|          | 4.00/363 [00:03<06:11, 1.04s/it]
Loading 0:   1%|          | 4.00/363 [00:03<06:11, 1.04s/it]
Loading 0:   1%|▏         | 5.00/363 [00:05<08:00, 1.34s/it]
Loading 0:   1%|▏         | 5.00/363 [00:05<08:00, 1.34s/it]
Loading 0:   3%|▎         | 12.0/363 [00:08<03:32, 1.65it/s]
Loading 0:   3%|▎         | 12.0/363 [00:08<03:32, 1.65it/s]
Loading 0:   4%|▎         | 13.0/363 [00:10<04:31, 1.29it/s]
Loading 0:   4%|▎         | 13.0/363 [00:10<04:31, 1.29it/s]
Loading 0:   4%|▍         | 14.0/363 [00:12<05:39, 1.03it/s]
Loading 0:   4%|▍         | 14.0/363 [00:12<05:39, 1.03it/s]
Loading 0:   6%|▌         | 21.0/363 [00:14<03:25, 1.67it/s]
Loading 0:   6%|▌         | 21.0/363 [00:14<03:25, 1.67it/s]
Loading 0:   6%|▌         | 22.0/363 [00:16<04:13, 1.35it/s]
Loading 0:   6%|▌         | 22.0/363 [00:16<04:13, 1.35it/s]
Loading 0:   6%|▋         | 23.0/363 [00:18<05:10, 1.09it/s]
Loading 0:   6%|▋         | 23.0/363 [00:18<05:10, 1.09it/s]
Loading 0:   8%|▊         | 30.0/363 [00:19<02:36, 2.13it/s]
Loading 0:   8%|▊         | 30.0/363 [00:19<02:36, 2.13it/s]
Loading 0:   9%|▉         | 34.0/363 [00:22<02:51, 1.92it/s]
Loading 0:   9%|▉         | 34.0/363 [00:22<02:51, 1.92it/s]
Loading 0:  10%|▉         | 35.0/363 [00:24<03:36, 1.52it/s]
Loading 0:  10%|▉         | 35.0/363 [00:24<03:36, 1.52it/s]
Loading 0:  10%|▉         | 36.0/363 [00:26<04:34, 1.19it/s]
Loading 0:  10%|▉         | 36.0/363 [00:26<04:34, 1.19it/s]
Loading 0:  11%|█         | 39.0/363 [00:28<04:07, 1.31it/s]
Loading 0:  11%|█         | 39.0/363 [00:28<04:07, 1.31it/s]
Loading 0:  11%|█         | 40.0/363 [00:29<04:58, 1.08it/s]
Loading 0:  11%|█         | 40.0/363 [00:29<04:58, 1.08it/s]
Loading 0:  11%|█▏        | 41.0/363 [00:31<05:57, 1.11s/it]
Loading 0:  11%|█▏        | 41.0/363 [00:31<05:57, 1.11s/it]
Loading 0:  13%|█▎        | 48.0/363 [00:34<03:21, 1.56it/s]
Loading 0:  13%|█▎        | 48.0/363 [00:34<03:21, 1.56it/s]
Loading 0:  13%|█▎        | 49.0/363 [00:36<04:05, 1.28it/s]
Loading 0:  13%|█▎        | 49.0/363 [00:36<04:05, 1.28it/s]
Loading 0:  14%|█▍        | 50.0/363 [00:38<04:58, 1.05it/s]
Loading 0:  14%|█▍        | 50.0/363 [00:38<04:58, 1.05it/s]
Loading 0:  16%|█▌        | 57.0/363 [00:40<03:06, 1.64it/s]
Loading 0:  16%|█▌        | 57.0/363 [00:40<03:06, 1.64it/s]
Loading 0:  16%|█▌        | 58.0/363 [00:42<03:46, 1.34it/s]
Loading 0:  16%|█▌        | 58.0/363 [00:42<03:46, 1.34it/s]
Loading 0:  16%|█▋        | 59.0/363 [00:44<04:36, 1.10it/s]
Loading 0:  16%|█▋        | 59.0/363 [00:44<04:36, 1.10it/s]
Loading 0:  18%|█▊        | 65.0/363 [00:47<03:14, 1.53it/s]
Loading 0:  18%|█▊        | 65.0/363 [00:47<03:14, 1.53it/s]
Loading 0:  20%|█▉        | 71.0/363 [00:50<02:45, 1.77it/s]
Loading 0:  20%|█▉        | 71.0/363 [00:50<02:45, 1.77it/s]
Loading 0:  20%|█▉        | 72.0/363 [00:51<03:20, 1.45it/s]
Loading 0:  20%|█▉        | 72.0/363 [00:51<03:20, 1.45it/s]
Loading 0:  20%|██        | 73.0/363 [00:53<04:04, 1.19it/s]
Loading 0:  20%|██        | 73.0/363 [00:53<04:04, 1.19it/s]
Loading 0:  22%|██▏       | 79.0/363 [00:56<02:59, 1.58it/s]
Loading 0:  22%|██▏       | 79.0/363 [00:56<02:59, 1.58it/s]
Loading 0:  22%|██▏       | 80.0/363 [00:58<03:39, 1.29it/s]
Loading 0:  22%|██▏       | 80.0/363 [00:58<03:39, 1.29it/s]
Loading 0:  24%|██▎       | 86.0/363 [01:01<02:48, 1.64it/s]
Loading 0:  24%|██▎       | 86.0/363 [01:01<02:48, 1.64it/s]
Loading 0:  24%|██▍       | 87.0/363 [01:03<03:27, 1.33it/s]
Loading 0:  24%|██▍       | 87.0/363 [01:03<03:27, 1.33it/s]
Loading 0:  25%|██▍       | 90.0/363 [01:04<03:14, 1.40it/s]
Loading 0:  25%|██▍       | 90.0/363 [01:04<03:14, 1.40it/s]
Loading 0:  25%|██▌       | 91.0/363 [01:06<03:54, 1.16it/s]
Loading 0:  25%|██▌       | 91.0/363 [01:06<03:54, 1.16it/s]
Loading 0:  25%|██▌       | 92.0/363 [01:08<04:40, 1.03s/it]
Loading 0:  25%|██▌       | 92.0/363 [01:08<04:40, 1.03s/it]
Loading 0:  27%|██▋       | 99.0/363 [01:11<02:46, 1.59it/s]
Loading 0:  27%|██▋       | 99.0/363 [01:11<02:46, 1.59it/s]
Loading 0:  28%|██▊       | 100/363 [01:13<03:21, 1.30it/s] 
Loading 0:  28%|██▊       | 100/363 [01:13<03:21, 1.30it/s]
Loading 0:  28%|██▊       | 101/363 [01:15<04:04, 1.07it/s]
Loading 0:  28%|██▊       | 101/363 [01:15<04:04, 1.07it/s]
Loading 0:  30%|██▉       | 108/363 [01:17<02:37, 1.62it/s]
Loading 0:  30%|██▉       | 108/363 [01:17<02:37, 1.62it/s]
Loading 0:  31%|███       | 111/363 [01:19<02:36, 1.61it/s]
Loading 0:  31%|███       | 111/363 [01:19<02:36, 1.61it/s]
Loading 0:  31%|███       | 112/363 [01:21<03:10, 1.31it/s]
Loading 0:  31%|███       | 112/363 [01:21<03:10, 1.31it/s]
Loading 0:  31%|███       | 113/363 [01:23<03:52, 1.08it/s]
Loading 0:  31%|███       | 113/363 [01:23<03:52, 1.08it/s]
Loading 0:  33%|███▎      | 120/363 [01:26<02:26, 1.66it/s]
Loading 0:  33%|███▎      | 120/363 [01:26<02:26, 1.66it/s]
Loading 0:  33%|███▎      | 121/363 [01:28<02:58, 1.35it/s]
Loading 0:  33%|███▎      | 121/363 [01:28<02:58, 1.35it/s]
Loading 0:  34%|███▎      | 122/363 [01:30<03:37, 1.11it/s]
Loading 0:  34%|███▎      | 122/363 [01:30<03:37, 1.11it/s]
Loading 0:  36%|███▌      | 129/363 [01:32<02:19, 1.68it/s]
Loading 0:  36%|███▌      | 129/363 [01:32<02:19, 1.68it/s]
Loading 0:  36%|███▌      | 130/363 [01:34<02:49, 1.37it/s]
Loading 0:  36%|███▌      | 130/363 [01:34<02:49, 1.37it/s]
Loading 0:  36%|███▌      | 131/363 [01:36<03:27, 1.12it/s]
Loading 0:  36%|███▌      | 131/363 [01:36<03:27, 1.12it/s]
Loading 0:  38%|███▊      | 138/363 [01:39<02:14, 1.68it/s]
Loading 0:  38%|███▊      | 138/363 [01:39<02:14, 1.68it/s]
Loading 0:  38%|███▊      | 139/363 [01:40<02:43, 1.37it/s]
Loading 0:  38%|███▊      | 139/363 [01:40<02:43, 1.37it/s]
Loading 0:  39%|███▊      | 140/363 [01:42<03:18, 1.12it/s]
Loading 0:  39%|███▊      | 140/363 [01:42<03:18, 1.12it/s]
Loading 0:  41%|████      | 148/363 [01:44<01:36, 2.22it/s]
Loading 0:  41%|████      | 148/363 [01:44<01:36, 2.22it/s]
Loading 0:  42%|████▏     | 151/363 [01:46<01:50, 1.92it/s]
Loading 0:  42%|████▏     | 151/363 [01:46<01:50, 1.92it/s]
Loading 0:  42%|████▏     | 152/363 [01:48<02:19, 1.52it/s]
Loading 0:  42%|████▏     | 152/363 [01:48<02:19, 1.52it/s]
Loading 0:  42%|████▏     | 153/363 [01:50<02:54, 1.20it/s]
Loading 0:  42%|████▏     | 153/363 [01:50<02:54, 1.20it/s]
Loading 0:  43%|████▎     | 156/363 [01:52<02:37, 1.32it/s]
Loading 0:  43%|████▎     | 156/363 [01:52<02:37, 1.32it/s]
Loading 0:  43%|████▎     | 157/363 [01:54<03:10, 1.08it/s]
Loading 0:  43%|████▎     | 157/363 [01:54<03:10, 1.08it/s]
Loading 0:  44%|████▎     | 158/363 [01:56<03:46, 1.10s/it]
Loading 0:  44%|████▎     | 158/363 [01:56<03:46, 1.10s/it]
Loading 0:  45%|████▌     | 165/363 [01:58<02:06, 1.57it/s]
Loading 0:  45%|████▌     | 165/363 [01:58<02:06, 1.57it/s]
Loading 0:  46%|████▌     | 166/363 [02:00<02:33, 1.28it/s]
Loading 0:  46%|████▌     | 166/363 [02:00<02:33, 1.28it/s]
Loading 0:  46%|████▌     | 167/363 [02:02<03:05, 1.05it/s]
Loading 0:  46%|████▌     | 167/363 [02:02<03:05, 1.05it/s]
Loading 0:  48%|████▊     | 174/363 [02:05<01:55, 1.64it/s]
Loading 0:  48%|████▊     | 174/363 [02:05<01:55, 1.64it/s]
Loading 0:  48%|████▊     | 175/363 [02:06<02:19, 1.34it/s]
Loading 0:  48%|████▊     | 175/363 [02:06<02:19, 1.34it/s]
Loading 0:  48%|████▊     | 176/363 [02:08<02:49, 1.10it/s]
Loading 0:  48%|████▊     | 176/363 [02:08<02:49, 1.10it/s]
Loading 0:  50%|█████     | 182/363 [02:11<01:58, 1.53it/s]
Loading 0:  50%|█████     | 182/363 [02:11<01:58, 1.53it/s]
Loading 0:  52%|█████▏    | 188/363 [02:14<01:38, 1.77it/s]
Loading 0:  52%|█████▏    | 188/363 [02:14<01:38, 1.77it/s]
Loading 0:  52%|█████▏    | 189/363 [02:16<02:01, 1.44it/s]
Loading 0:  52%|█████▏    | 189/363 [02:16<02:01, 1.44it/s]
Loading 0:  53%|█████▎    | 192/363 [02:17<01:55, 1.48it/s]
Loading 0:  53%|█████▎    | 192/363 [02:17<01:55, 1.48it/s]
Loading 0:  53%|█████▎    | 193/363 [02:19<02:19, 1.22it/s]
Loading 0:  53%|█████▎    | 193/363 [02:19<02:19, 1.22it/s]
Loading 0:  53%|█████▎    | 194/363 [02:21<02:47, 1.01it/s]
Loading 0:  53%|█████▎    | 194/363 [02:21<02:47, 1.01it/s]
Loading 0:  55%|█████▌    | 201/363 [02:24<01:40, 1.62it/s]
Loading 0:  55%|█████▌    | 201/363 [02:24<01:40, 1.62it/s]
Loading 0:  56%|█████▌    | 202/363 [02:26<02:01, 1.33it/s]
Loading 0:  56%|█████▌    | 202/363 [02:26<02:01, 1.33it/s]
Loading 0:  56%|█████▌    | 203/363 [02:28<02:26, 1.09it/s]
Loading 0:  56%|█████▌    | 203/363 [02:28<02:26, 1.09it/s]
Loading 0:  58%|█████▊    | 210/363 [02:30<01:32, 1.66it/s]
Loading 0:  58%|█████▊    | 210/363 [02:30<01:32, 1.66it/s]
Loading 0:  58%|█████▊    | 211/363 [02:32<01:51, 1.36it/s]
Loading 0:  58%|█████▊    | 211/363 [02:32<01:51, 1.36it/s]
Loading 0:  58%|█████▊    | 212/363 [02:34<02:15, 1.11it/s]
Loading 0:  58%|█████▊    | 212/363 [02:34<02:15, 1.11it/s]
Loading 0:  60%|██████    | 218/363 [02:37<01:34, 1.54it/s]
Loading 0:  60%|██████    | 218/363 [02:37<01:34, 1.54it/s]
Loading 0:  60%|██████    | 219/363 [02:39<01:55, 1.25it/s]
Loading 0:  60%|██████    | 219/363 [02:39<01:55, 1.25it/s]
Loading 0:  62%|██████▏   | 225/363 [02:41<01:26, 1.60it/s]
Loading 0:  62%|██████▏   | 225/363 [02:41<01:26, 1.60it/s]
Loading 0:  63%|██████▎   | 228/363 [02:43<01:24, 1.60it/s]
Loading 0:  63%|██████▎   | 228/363 [02:43<01:24, 1.60it/s]
Loading 0:  63%|██████▎   | 229/363 [02:45<01:42, 1.31it/s]
Loading 0:  63%|██████▎   | 229/363 [02:45<01:42, 1.31it/s]
Loading 0:  63%|██████▎   | 230/363 [02:47<02:03, 1.08it/s]
Loading 0:  63%|██████▎   | 230/363 [02:47<02:03, 1.08it/s]
Loading 0:  65%|██████▌   | 237/363 [02:50<01:15, 1.66it/s]
Loading 0:  65%|██████▌   | 237/363 [02:50<01:15, 1.66it/s]
Loading 0:  66%|██████▌   | 238/363 [02:52<01:32, 1.36it/s]
Loading 0:  66%|██████▌   | 238/363 [02:52<01:32, 1.36it/s]
Loading 0:  66%|██████▌   | 239/363 [02:54<01:51, 1.11it/s]
Loading 0:  66%|██████▌   | 239/363 [02:54<01:51, 1.11it/s]
Loading 0:  68%|██████▊   | 246/363 [02:56<01:09, 1.67it/s]
Loading 0:  68%|██████▊   | 246/363 [02:56<01:09, 1.67it/s]
Loading 0:  68%|██████▊   | 247/363 [02:58<01:24, 1.37it/s]
Loading 0:  68%|██████▊   | 247/363 [02:58<01:24, 1.37it/s]
Loading 0:  68%|██████▊   | 248/363 [03:00<01:42, 1.12it/s]
Loading 0:  68%|██████▊   | 248/363 [03:00<01:42, 1.12it/s]
Loading 0:  70%|███████   | 255/363 [03:02<01:04, 1.68it/s]
Loading 0:  70%|███████   | 255/363 [03:02<01:04, 1.68it/s]
Loading 0:  71%|███████   | 256/363 [03:04<01:17, 1.37it/s]
Loading 0:  71%|███████   | 256/363 [03:04<01:17, 1.37it/s]
Loading 0:  71%|███████   | 257/363 [03:06<01:34, 1.13it/s]
Loading 0:  71%|███████   | 257/363 [03:06<01:34, 1.13it/s]
Loading 0:  73%|███████▎  | 265/363 [03:08<00:43, 2.24it/s]
Loading 0:  73%|███████▎  | 265/363 [03:08<00:43, 2.24it/s]
Loading 0:  74%|███████▍  | 268/363 [03:10<00:48, 1.94it/s]
Loading 0:  74%|███████▍  | 268/363 [03:10<00:48, 1.94it/s]
Loading 0:  74%|███████▍  | 269/363 [03:12<01:01, 1.53it/s]
Loading 0:  74%|███████▍  | 269/363 [03:12<01:01, 1.53it/s]
Loading 0:  74%|███████▍  | 270/363 [03:14<01:16, 1.21it/s]
Loading 0:  74%|███████▍  | 270/363 [03:14<01:16, 1.21it/s]
Loading 0:  75%|███████▌  | 273/363 [03:28<03:26, 2.29s/it]
Loading 0:  75%|███████▌  | 273/363 [03:28<03:26, 2.29s/it]
Loading 0:  75%|███████▌  | 274/363 [03:30<03:18, 2.23s/it]
Loading 0:  75%|███████▌  | 274/363 [03:30<03:18, 2.23s/it]
Loading 0:  76%|███████▌  | 275/363 [03:32<03:13, 2.20s/it]
Loading 0:  76%|███████▌  | 275/363 [03:32<03:13, 2.20s/it]
Loading 0:  78%|███████▊  | 282/363 [03:35<01:23, 1.03s/it]
Loading 0:  78%|███████▊  | 282/363 [03:35<01:23, 1.03s/it]
Loading 0:  78%|███████▊  | 283/363 [03:37<01:28, 1.11s/it]
Loading 0:  78%|███████▊  | 283/363 [03:37<01:28, 1.11s/it]
Loading 0:  78%|███████▊  | 284/363 [03:38<01:36, 1.22s/it]
Loading 0:  78%|███████▊  | 284/363 [03:38<01:36, 1.22s/it]
Loading 0:  80%|████████  | 291/363 [03:41<00:51, 1.41it/s]
Loading 0:  80%|████████  | 291/363 [03:41<00:51, 1.41it/s]
Loading 0:  80%|████████  | 292/363 [03:43<00:58, 1.22it/s]
Loading 0:  80%|████████  | 292/363 [03:43<00:58, 1.22it/s]
Loading 0:  81%|████████  | 293/363 [03:45<01:07, 1.03it/s]
Loading 0:  81%|████████  | 293/363 [03:45<01:07, 1.03it/s]
Loading 0:  82%|████████▏ | 299/363 [03:47<00:42, 1.51it/s]
Loading 0:  82%|████████▏ | 299/363 [03:47<00:42, 1.51it/s]
Loading 0:  84%|████████▍ | 305/363 [03:49<00:32, 1.79it/s]
Loading 0:  84%|████████▍ | 305/363 [03:49<00:32, 1.79it/s]
Loading 0:  84%|████████▍ | 306/363 [03:51<00:39, 1.45it/s]
Loading 0:  84%|████████▍ | 306/363 [03:51<00:39, 1.45it/s]
Loading 0:  85%|████████▌ | 309/363 [03:53<00:36, 1.49it/s]
Loading 0:  85%|████████▌ | 309/363 [03:53<00:36, 1.49it/s]
Loading 0:  85%|████████▌ | 310/363 [03:55<00:43, 1.23it/s]
Loading 0:  85%|████████▌ | 310/363 [03:55<00:43, 1.23it/s]
Loading 0:  86%|████████▌ | 311/363 [03:57<00:51, 1.01it/s]
Loading 0:  86%|████████▌ | 311/363 [03:57<00:51, 1.01it/s]
Loading 0:  88%|████████▊ | 318/363 [04:00<00:27, 1.63it/s]
Loading 0:  88%|████████▊ | 318/363 [04:00<00:27, 1.63it/s]
Loading 0:  88%|████████▊ | 319/363 [04:01<00:32, 1.34it/s]
Loading 0:  88%|████████▊ | 319/363 [04:01<00:32, 1.34it/s]
Loading 0:  88%|████████▊ | 320/363 [04:03<00:39, 1.10it/s]
Loading 0:  88%|████████▊ | 320/363 [04:03<00:39, 1.10it/s]
Loading 0:  90%|█████████ | 327/363 [04:06<00:21, 1.68it/s]
Loading 0:  90%|█████████ | 327/363 [04:06<00:21, 1.68it/s]
Loading 0:  90%|█████████ | 328/363 [04:08<00:25, 1.37it/s]
Loading 0:  90%|█████████ | 328/363 [04:08<00:25, 1.37it/s]
Loading 0:  91%|█████████ | 329/363 [04:10<00:30, 1.12it/s]
Loading 0:  91%|█████████ | 329/363 [04:10<00:30, 1.12it/s]
Loading 0:  92%|█████████▏| 335/363 [04:12<00:17, 1.56it/s]
Loading 0:  92%|█████████▏| 335/363 [04:12<00:17, 1.56it/s]
Loading 0:  93%|█████████▎| 336/363 [04:14<00:21, 1.27it/s]
Loading 0:  93%|█████████▎| 336/363 [04:14<00:21, 1.27it/s]
Loading 0:  94%|█████████▍| 343/363 [04:17<00:11, 1.71it/s]
Loading 0:  94%|█████████▍| 343/363 [04:17<00:11, 1.71it/s]
Loading 0:  95%|█████████▌| 346/363 [04:19<00:10, 1.68it/s]
Loading 0:  95%|█████████▌| 346/363 [04:19<00:10, 1.68it/s]
Loading 0:  96%|█████████▌| 347/363 [04:21<00:11, 1.37it/s]
Loading 0:  96%|█████████▌| 347/363 [04:21<00:11, 1.37it/s]
Loading 0:  96%|█████████▌| 348/363 [04:23<00:13, 1.12it/s]
Loading 0:  96%|█████████▌| 348/363 [04:23<00:13, 1.12it/s]
Loading 0:  98%|█████████▊| 355/363 [04:25<00:04, 1.68it/s]
Loading 0:  98%|█████████▊| 355/363 [04:25<00:04, 1.68it/s]
Loading 0:  98%|█████████▊| 356/363 [04:27<00:05, 1.38it/s]
Loading 0:  98%|█████████▊| 356/363 [04:27<00:05, 1.38it/s]
Loading 0:  98%|█████████▊| 357/363 [04:29<00:05, 1.13it/s]
Loading 0:  98%|█████████▊| 357/363 [04:29<00:05, 1.13it/s]
Loading 0: 100%|██████████| 363/363 [04:30<00:00, 2.07it/s]
Loading 0: 100%|██████████| 363/363 [04:30<00:00, 2.07it/s]
                                                           

Loading 0: 100%|██████████| 363/363 [04:30<00:00, 1.34it/s]
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: The tokenizer you are loading from '/tmp/tmpn3w1hnij' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: quantized model in 276.558s
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: Processed model ChaiML/kimid-v2_cosine-lr2e6g16 in 329.859s
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: creating bucket guanaco-mkml-models
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-kimid-v2-cosine-80852-v5/nvidia
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-kimid-v2-cosine-80852-v5/nvidia/config.json
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-kimid-v2-cosine-80852-v5/nvidia/special_tokens_map.json
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-kimid-v2-cosine-80852-v5/nvidia/tokenizer_config.json
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-kimid-v2-cosine-80852-v5/nvidia/tokenizer.json
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-kimid-v2-cosine-80852-v5/nvidia/flywheel_model.0.safetensors
chaiml-kimid-v2-cosine-80852-v5-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-kimid-v2-cosine-80852-v5/nvidia/flywheel_model.1.safetensors
Job chaiml-kimid-v2-cosine-80852-v5-mkmlizer completed after 410.21s with status: succeeded
Stopping job with name chaiml-kimid-v2-cosine-80852-v5-mkmlizer
Pipeline stage MKMLizer completed in 410.74s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-kimid-v2-cosine-80852-v5
Waiting for inference service chaiml-kimid-v2-cosine-80852-v5 to be ready
Inference service chaiml-kimid-v2-cosine-80852-v5 ready after 211.13235783576965s
Pipeline stage MKMLDeployer completed in 211.87s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.549753427505493s
Failed to get response for submission blend_sagol_2025-12-08: HTTPConnectionPool(host='chaiml-4d70-fd43-linear-w01-v17-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Received healthy response to inference request in 2.52763032913208s
Received healthy response to inference request in 2.454883337020874s
Received healthy response to inference request in 2.34828519821167s
5 requests
1 failed requests
5th percentile: 2.3696048259735107
10th percentile: 2.3909244537353516
20th percentile: 2.433563709259033
30th percentile: 2.469432735443115
40th percentile: 2.498531532287598
50th percentile: 2.52763032913208
60th percentile: 2.5364795684814454
70th percentile: 2.5453288078308107
80th percentile: 6.080127859115604
90th percentile: 13.140876722335818
95th percentile: 16.67125115394592
99th percentile: 19.49555069923401
mean time: 6.01643557548523
%s, retrying in %s seconds...
Received healthy response to inference request in 2.0428833961486816s
Failed to get response for submission chaiml-4d70-fd43-linear-w01_v17: HTTPConnectionPool(host='chaiml-4d70-fd43-linear-w01-v17-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Received healthy response to inference request in 2.8873629570007324s
Received healthy response to inference request in 1.9747090339660645s
Received healthy response to inference request in 2.284400701522827s
Received healthy response to inference request in 2.559349536895752s
5 requests
0 failed requests
5th percentile: 1.9883439064025878
10th percentile: 2.0019787788391112
20th percentile: 2.029248523712158
30th percentile: 2.091186857223511
40th percentile: 2.1877937793731688
50th percentile: 2.284400701522827
60th percentile: 2.394380235671997
70th percentile: 2.504359769821167
80th percentile: 2.624952220916748
90th percentile: 2.7561575889587404
95th percentile: 2.8217602729797364
99th percentile: 2.8742424201965333
mean time: 2.3497411251068114
Pipeline stage StressChecker completed in 45.37s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.67s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.69s
Shutdown handler de-registered
chaiml-kimid-v2-cosine-_80852_v5 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Generating Leaderboard row for %s
Generated Leaderboard row for %s
Pipeline stage OfflineFamilyFriendlyScorer completed in 2658.32s
Shutdown handler de-registered
chaiml-kimid-v2-cosine-_80852_v5 status is now torndown due to DeploymentManager action