chaiml-2fe5-c13f-linear-w01

developer_uid: richhx
submission_id: chaiml-2fe5-c13f-linear-w01_v38
model_name: chaiml-2fe5-c13f-linear-w01_v38
model_group: ChaiML/2fe5-c13f-linear-
status: torndown
timestamp: 2026-01-16T23:29:02+00:00
num_battles: 11925
num_wins: 5929
celo_rating: 1294.62
family_friendly_score: 0.5318
family_friendly_standard_error: 0.0070567522274768875
submission_type: basic
model_repo: ChaiML/2fe5-c13f-linear-w01
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 10
max_input_tokens: 1024
max_output_tokens: 64
reward_model: default
latencies: [{'batch_size': 1, 'throughput': 0.9217145494735581, 'latency_mean': 1.0848705434799195, 'latency_p50': 1.0718039274215698, 'latency_p90': 1.2055843114852904}, {'batch_size': 4, 'throughput': 2.1005777534864953, 'latency_mean': 1.8891857409477233, 'latency_p50': 1.8952864408493042, 'latency_p90': 2.1413010120391847}, {'batch_size': 5, 'throughput': 2.3162383622428337, 'latency_mean': 2.1457563138008116, 'latency_p50': 2.167786121368408, 'latency_p90': 2.4073142766952516}, {'batch_size': 8, 'throughput': 2.6842976443001882, 'latency_mean': 2.954561823606491, 'latency_p50': 2.9619449377059937, 'latency_p90': 3.3439955949783324}, {'batch_size': 10, 'throughput': 2.8077165413001755, 'latency_mean': 3.5361532092094423, 'latency_p50': 3.556805968284607, 'latency_p90': 4.045292854309082}, {'batch_size': 12, 'throughput': 2.8420501652037062, 'latency_mean': 4.181497187614441, 'latency_p50': 4.220760703086853, 'latency_p90': 4.726145577430725}, {'batch_size': 15, 'throughput': 2.9149799776795713, 'latency_mean': 5.0735227847099305, 'latency_p50': 5.062867283821106, 'latency_p90': 5.872950387001038}]
gpu_counts: {'NVIDIA L40S': 1}
display_name: chaiml-2fe5-c13f-linear-w01_v38
is_internal_developer: True
language_model: ChaiML/2fe5-c13f-linear-w01
model_size: 13B
ranking_group: single
throughput_3p7s: 2.84
us_pacific_date: 2026-01-13
win_ratio: 0.49719077568134173
generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['Bot:', '</s>', '\n', '####', '<|eot_id|>', 'You:', 'User:', '<|im_end|>'], 'max_input_tokens': 1024, 'best_of': 10, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-2fe5-c13f-linear-w01-v38-mkmlizer
Shutdown handler not registered because Python interpreter is not running in the main thread
Waiting for job on chaiml-2fe5-c13f-linear-w01-v38-mkmlizer to finish
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-ca18-c13f-linear-w01-v38-mkmlizer
Waiting for job on chaiml-ca18-c13f-linear-w01-v38-mkmlizer to finish
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: bash: no job control in this shell
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: Downloaded to shared memory in 28.619s
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: Checking if ChaiML/2fe5-c13f-linear-w01 already exists in ChaiML
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmph5nvjogg, device:0
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Stopping job with name chaiml-02f4-69d4-linear-w01-v51-mkmlizer
%s, retrying in %s seconds...
Starting job with name chaiml-02f4-69d4-linear-w01-v51-mkmlizer
Waiting for job on chaiml-02f4-69d4-linear-w01-v51-mkmlizer to finish
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-4d70-fd43-linear-w01-v29-mkmlizer
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
Waiting for job on chaiml-4d70-fd43-linear-w01-v29-mkmlizer to finish
run pipeline stage %s
Running pipeline stage MKMLizer
Shutdown handler not registered because Python interpreter is not running in the main thread
Starting job with name chaiml-2a6f-69d4-linear-w01-v24-mkmlizer
run pipeline %s
Waiting for job on chaiml-2a6f-69d4-linear-w01-v24-mkmlizer to finish
run pipeline stage %s
Running pipeline stage MKMLizer
Shutdown handler not registered because Python interpreter is not running in the main thread
Starting job with name chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer
run pipeline %s
Waiting for job on chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer to finish
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer
Waiting for job on chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer to finish
Stopping job with name chaiml-02f4-69d4-linear-w01-v51-mkmlizer
%s, retrying in %s seconds...
Starting job with name chaiml-02f4-69d4-linear-w01-v51-mkmlizer
Waiting for job on chaiml-02f4-69d4-linear-w01-v51-mkmlizer to finish
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: bash: no job control in this shell
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: bash: no job control in this shell
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║                                                                     ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║                                                                     ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║                                                                     ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║  Version: 0.30.6+torch280                                           ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║                                                                     ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║  Version: 0.30.6+torch280                                           ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║                                                                     ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║  belonging to:                                                      ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║                                                                     ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║                                                                     ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║  belonging to:                                                      ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║                                                                     ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ║                                                                     ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ║                                                                     ║
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: Downloaded to shared memory in 49.218s
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: Checking if ChaiML/2a6f-69d4-linear-w01 already exists in ChaiML
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmpwlstf01d, device:0
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: Downloaded to shared memory in 45.605s
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: Checking if ChaiML/ssnew-v5-dpo-lr5e6b01-lora already exists in ChaiML
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmpv80yc7l_, device:0
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: 
Loading 0:   0%|          | 0.00/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 4.00/363 [00:01<01:42, 3.52it/s]
Loading 0:   1%|          | 4.00/363 [00:01<01:42, 3.52it/s]
Loading 0:   2%|▏         | 6.00/363 [00:03<03:30, 1.70it/s]
Loading 0:   2%|▏         | 6.00/363 [00:03<03:30, 1.70it/s]
Loading 0:   4%|▎         | 13.0/363 [00:04<02:06, 2.76it/s]
Loading 0:   4%|▎         | 13.0/363 [00:04<02:06, 2.76it/s]
Loading 0:   4%|▍         | 15.0/363 [00:06<02:53, 2.01it/s]
Loading 0:   4%|▍         | 15.0/363 [00:06<02:53, 2.01it/s]
Loading 0:   6%|▌         | 22.0/363 [00:08<02:05, 2.72it/s]
Loading 0:   6%|▌         | 22.0/363 [00:08<02:05, 2.72it/s]
Loading 0:   7%|▋         | 24.0/363 [00:10<02:42, 2.09it/s]
Loading 0:   7%|▋         | 24.0/363 [00:10<02:42, 2.09it/s]
Loading 0:   9%|▊         | 31.0/363 [00:12<02:02, 2.70it/s]
Loading 0:   9%|▊         | 31.0/363 [00:12<02:02, 2.70it/s]
Loading 0:   9%|▉         | 33.0/363 [00:14<02:35, 2.13it/s]
Loading 0:   9%|▉         | 33.0/363 [00:14<02:35, 2.13it/s]
Loading 0:  11%|█         | 40.0/363 [00:16<02:05, 2.57it/s]
Loading 0:  11%|█         | 40.0/363 [00:16<02:05, 2.57it/s]
Loading 0:  12%|█▏        | 42.0/363 [00:18<02:35, 2.07it/s]
Loading 0:  12%|█▏        | 42.0/363 [00:18<02:35, 2.07it/s]
Loading 0:  13%|█▎        | 49.0/363 [00:20<01:59, 2.63it/s]
Loading 0:  13%|█▎        | 49.0/363 [00:20<01:59, 2.63it/s]
Loading 0:  14%|█▍        | 51.0/363 [00:22<02:27, 2.11it/s]
Loading 0:  14%|█▍        | 51.0/363 [00:22<02:27, 2.11it/s]
Loading 0:  16%|█▌        | 58.0/363 [00:23<01:54, 2.66it/s]
Loading 0:  16%|█▌        | 58.0/363 [00:23<01:54, 2.66it/s]
Loading 0:  17%|█▋        | 60.0/363 [00:25<02:21, 2.14it/s]
Loading 0:  17%|█▋        | 60.0/363 [00:25<02:21, 2.14it/s]
Loading 0:  18%|█▊        | 67.0/363 [00:27<01:50, 2.68it/s]
Loading 0:  18%|█▊        | 67.0/363 [00:27<01:50, 2.68it/s]
Loading 0:  19%|█▉        | 69.0/363 [00:29<02:17, 2.14it/s]
Loading 0:  19%|█▉        | 69.0/363 [00:29<02:17, 2.14it/s]
Loading 0:  21%|██        | 76.0/363 [00:31<01:46, 2.68it/s]
Loading 0:  21%|██        | 76.0/363 [00:31<01:46, 2.68it/s]
Loading 0:  21%|██▏       | 78.0/363 [00:33<02:12, 2.15it/s]
Loading 0:  21%|██▏       | 78.0/363 [00:33<02:12, 2.15it/s]
Loading 0:  23%|██▎       | 85.0/363 [00:34<01:43, 2.69it/s]
Loading 0:  23%|██▎       | 85.0/363 [00:34<01:43, 2.69it/s]
Loading 0:  24%|██▍       | 87.0/363 [00:36<02:08, 2.15it/s]
Loading 0:  24%|██▍       | 87.0/363 [00:36<02:08, 2.15it/s]
Loading 0:  26%|██▌       | 94.0/363 [00:38<01:40, 2.69it/s]
Loading 0:  26%|██▌       | 94.0/363 [00:38<01:40, 2.69it/s]
Loading 0:  26%|██▋       | 96.0/363 [00:40<02:04, 2.15it/s]
Loading 0:  26%|██▋       | 96.0/363 [00:40<02:04, 2.15it/s]
Loading 0:  28%|██▊       | 103/363 [00:42<01:36, 2.69it/s] 
Loading 0:  28%|██▊       | 103/363 [00:42<01:36, 2.69it/s]
Loading 0:  29%|██▉       | 105/363 [00:44<01:59, 2.16it/s]
Loading 0:  29%|██▉       | 105/363 [00:44<01:59, 2.16it/s]
Loading 0:  31%|███       | 112/363 [00:46<01:32, 2.70it/s]
Loading 0:  31%|███       | 112/363 [00:46<01:32, 2.70it/s]
Loading 0:  31%|███▏      | 114/363 [00:48<01:55, 2.16it/s]
Loading 0:  31%|███▏      | 114/363 [00:48<01:55, 2.16it/s]
Loading 0:  33%|███▎      | 121/363 [00:50<01:32, 2.62it/s]
Loading 0:  33%|███▎      | 121/363 [00:50<01:32, 2.62it/s]
Loading 0:  34%|███▍      | 123/363 [00:52<01:53, 2.11it/s]
Loading 0:  34%|███▍      | 123/363 [00:52<01:53, 2.11it/s]
Loading 0:  36%|███▌      | 130/363 [00:53<01:27, 2.67it/s]
Loading 0:  36%|███▌      | 130/363 [00:53<01:27, 2.67it/s]
Loading 0:  36%|███▋      | 132/363 [00:55<01:47, 2.15it/s]
Loading 0:  36%|███▋      | 132/363 [00:55<01:47, 2.15it/s]
Loading 0:  38%|███▊      | 139/363 [00:57<01:23, 2.69it/s]
Loading 0:  38%|███▊      | 139/363 [00:57<01:23, 2.69it/s]
Loading 0:  39%|███▉      | 141/363 [00:59<01:42, 2.16it/s]
Loading 0:  39%|███▉      | 141/363 [00:59<01:42, 2.16it/s]
Loading 0:  41%|████      | 148/363 [01:01<01:19, 2.70it/s]
Loading 0:  41%|████      | 148/363 [01:01<01:19, 2.70it/s]
Loading 0:  41%|████▏     | 150/363 [01:03<01:38, 2.16it/s]
Loading 0:  41%|████▏     | 150/363 [01:03<01:38, 2.16it/s]
Loading 0:  43%|████▎     | 157/363 [01:04<01:16, 2.70it/s]
Loading 0:  43%|████▎     | 157/363 [01:04<01:16, 2.70it/s]
Loading 0:  44%|████▍     | 159/363 [01:06<01:34, 2.17it/s]
Loading 0:  44%|████▍     | 159/363 [01:06<01:34, 2.17it/s]
Loading 0:  46%|████▌     | 166/363 [01:08<01:12, 2.70it/s]
Loading 0:  46%|████▌     | 166/363 [01:08<01:12, 2.70it/s]
Loading 0:  46%|████▋     | 168/363 [01:10<01:30, 2.16it/s]
Loading 0:  46%|████▋     | 168/363 [01:10<01:30, 2.16it/s]
Loading 0:  48%|████▊     | 175/363 [01:12<01:09, 2.71it/s]
Loading 0:  48%|████▊     | 175/363 [01:12<01:09, 2.71it/s]
Loading 0:  49%|████▉     | 177/363 [01:14<01:25, 2.18it/s]
Loading 0:  49%|████▉     | 177/363 [01:14<01:25, 2.18it/s]
Loading 0:  51%|█████     | 184/363 [01:15<01:05, 2.72it/s]
Loading 0:  51%|█████     | 184/363 [01:15<01:05, 2.72it/s]
Loading 0:  51%|█████     | 186/363 [01:17<01:21, 2.18it/s]
Loading 0:  51%|█████     | 186/363 [01:17<01:21, 2.18it/s]
Loading 0:  53%|█████▎    | 193/363 [01:19<01:02, 2.72it/s]
Loading 0:  53%|█████▎    | 193/363 [01:19<01:02, 2.72it/s]
Loading 0:  54%|█████▎    | 195/363 [01:21<01:17, 2.17it/s]
Loading 0:  54%|█████▎    | 195/363 [01:21<01:17, 2.17it/s]
Loading 0:  56%|█████▌    | 202/363 [01:23<01:01, 2.63it/s]
Loading 0:  56%|█████▌    | 202/363 [01:23<01:01, 2.63it/s]
Loading 0:  56%|█████▌    | 204/363 [01:25<01:15, 2.12it/s]
Loading 0:  56%|█████▌    | 204/363 [01:25<01:15, 2.12it/s]
Loading 0:  58%|█████▊    | 211/363 [01:27<00:56, 2.67it/s]
Loading 0:  58%|█████▊    | 211/363 [01:27<00:56, 2.67it/s]
Loading 0:  58%|█████▊    | 212/363 [01:28<01:06, 2.27it/s]
Loading 0:  58%|█████▊    | 212/363 [01:28<01:06, 2.27it/s]
Loading 0:  59%|█████▊    | 213/363 [01:29<01:15, 1.99it/s]
Loading 0:  59%|█████▊    | 213/363 [01:29<01:15, 1.99it/s]
Loading 0:  61%|██████    | 220/363 [01:31<00:53, 2.68it/s]
Loading 0:  61%|██████    | 220/363 [01:31<00:53, 2.68it/s]
Loading 0:  61%|██████    | 222/363 [01:33<01:07, 2.09it/s]
Loading 0:  61%|██████    | 222/363 [01:33<01:07, 2.09it/s]
Loading 0:  63%|██████▎   | 229/363 [01:34<00:49, 2.70it/s]
Loading 0:  63%|██████▎   | 229/363 [01:34<00:49, 2.70it/s]
Loading 0:  64%|██████▎   | 231/363 [01:36<01:01, 2.13it/s]
Loading 0:  64%|██████▎   | 231/363 [01:36<01:01, 2.13it/s]
Loading 0:  66%|██████▌   | 238/363 [01:38<00:46, 2.71it/s]
Loading 0:  66%|██████▌   | 238/363 [01:38<00:46, 2.71it/s]
Loading 0:  66%|██████▌   | 240/363 [01:40<00:56, 2.16it/s]
Loading 0:  66%|██████▌   | 240/363 [01:40<00:56, 2.16it/s]
Loading 0:  68%|██████▊   | 247/363 [01:42<00:42, 2.72it/s]
Loading 0:  68%|██████▊   | 247/363 [01:42<00:42, 2.72it/s]
Loading 0:  69%|██████▊   | 249/363 [01:44<00:52, 2.17it/s]
Loading 0:  69%|██████▊   | 249/363 [01:44<00:52, 2.17it/s]
Loading 0:  71%|███████   | 256/363 [01:45<00:39, 2.72it/s]
Loading 0:  71%|███████   | 256/363 [01:45<00:39, 2.72it/s]
Loading 0:  71%|███████   | 258/363 [01:47<00:48, 2.17it/s]
Loading 0:  71%|███████   | 258/363 [01:47<00:48, 2.17it/s]
Loading 0:  73%|███████▎  | 265/363 [01:49<00:36, 2.71it/s]
Loading 0:  73%|███████▎  | 265/363 [01:49<00:36, 2.71it/s]
Loading 0:  74%|███████▎  | 267/363 [01:51<00:44, 2.17it/s]
Loading 0:  74%|███████▎  | 267/363 [01:51<00:44, 2.17it/s]
Loading 0:  75%|███████▌  | 274/363 [01:53<00:32, 2.70it/s]
Loading 0:  75%|███████▌  | 274/363 [01:53<00:32, 2.70it/s]
Loading 0:  76%|███████▌  | 276/363 [01:55<00:40, 2.17it/s]
Loading 0:  76%|███████▌  | 276/363 [01:55<00:40, 2.17it/s]
Loading 0:  78%|███████▊  | 283/363 [01:57<00:30, 2.63it/s]
Loading 0:  78%|███████▊  | 283/363 [01:57<00:30, 2.63it/s]
Loading 0:  78%|███████▊  | 284/363 [01:58<00:34, 2.32it/s]
Loading 0:  78%|███████▊  | 284/363 [01:58<00:34, 2.32it/s]
Loading 0:  79%|███████▊  | 285/363 [01:59<00:38, 2.02it/s]
Loading 0:  79%|███████▊  | 285/363 [01:59<00:38, 2.02it/s]
Loading 0:  80%|████████  | 292/363 [02:00<00:26, 2.72it/s]
Loading 0:  80%|████████  | 292/363 [02:00<00:26, 2.72it/s]
Loading 0:  81%|████████  | 294/363 [02:02<00:32, 2.10it/s]
Loading 0:  81%|████████  | 294/363 [02:02<00:32, 2.10it/s]
Loading 0:  83%|████████▎ | 301/363 [02:04<00:22, 2.71it/s]
Loading 0:  83%|████████▎ | 301/363 [02:04<00:22, 2.71it/s]
Loading 0:  83%|████████▎ | 303/363 [02:06<00:28, 2.13it/s]
Loading 0:  83%|████████▎ | 303/363 [02:06<00:28, 2.13it/s]
Loading 0:  85%|████████▌ | 310/363 [02:08<00:19, 2.70it/s]
Loading 0:  85%|████████▌ | 310/363 [02:08<00:19, 2.70it/s]
Loading 0:  86%|████████▌ | 312/363 [02:10<00:23, 2.15it/s]
Loading 0:  86%|████████▌ | 312/363 [02:10<00:23, 2.15it/s]
Loading 0:  88%|████████▊ | 319/363 [02:12<00:16, 2.70it/s]
Loading 0:  88%|████████▊ | 319/363 [02:12<00:16, 2.70it/s]
Loading 0:  88%|████████▊ | 321/363 [02:14<00:19, 2.16it/s]
Loading 0:  88%|████████▊ | 321/363 [02:14<00:19, 2.16it/s]
Loading 0:  90%|█████████ | 328/363 [02:15<00:12, 2.70it/s]
Loading 0:  90%|█████████ | 328/363 [02:15<00:12, 2.70it/s]
Loading 0:  91%|█████████ | 330/363 [02:17<00:15, 2.16it/s]
Loading 0:  91%|█████████ | 330/363 [02:17<00:15, 2.16it/s]
Loading 0:  93%|█████████▎| 337/363 [02:19<00:09, 2.70it/s]
Loading 0:  93%|█████████▎| 337/363 [02:19<00:09, 2.70it/s]
Loading 0:  93%|█████████▎| 339/363 [02:21<00:11, 2.16it/s]
Loading 0:  93%|█████████▎| 339/363 [02:21<00:11, 2.16it/s]
Loading 0:  95%|█████████▌| 346/363 [02:23<00:06, 2.70it/s]
Loading 0:  95%|█████████▌| 346/363 [02:23<00:06, 2.70it/s]
Loading 0:  96%|█████████▌| 348/363 [02:25<00:06, 2.16it/s]
Loading 0:  96%|█████████▌| 348/363 [02:25<00:06, 2.16it/s]
Loading 0:  98%|█████████▊| 355/363 [02:26<00:02, 2.71it/s]
Loading 0:  98%|█████████▊| 355/363 [02:26<00:02, 2.71it/s]
Loading 0:  98%|█████████▊| 357/363 [02:28<00:02, 2.16it/s]
Loading 0:  98%|█████████▊| 357/363 [02:28<00:02, 2.16it/s]
Loading 0: 100%|██████████| 363/363 [02:29<00:00, 2.94it/s]
Loading 0: 100%|██████████| 363/363 [02:29<00:00, 2.94it/s]
                                                           

Loading 0: 100%|██████████| 363/363 [02:29<00:00, 2.42it/s]
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: The tokenizer you are loading from '/tmp/tmph5nvjogg' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: quantized model in 162.337s
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: Processed model ChaiML/2fe5-c13f-linear-w01 in 190.956s
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: creating bucket guanaco-mkml-models
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-2fe5-c13f-linear-w01-v38/nvidia
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-mkml-models/chaiml-2fe5-c13f-linear-w01-v38/nvidia/chat_template.jinja
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-2fe5-c13f-linear-w01-v38/nvidia/special_tokens_map.json
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-2fe5-c13f-linear-w01-v38/nvidia/config.json
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-2fe5-c13f-linear-w01-v38/nvidia/tokenizer_config.json
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-2fe5-c13f-linear-w01-v38/nvidia/tokenizer.json
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: bash: no job control in this shell
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-7b07-69d4-linear-w01-v33-mkmlizer:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-2fe5-c13f-linear-w01-v38-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-2fe5-c13f-linear-w01-v38/nvidia/flywheel_model.0.safetensors
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║                                                                     ║
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║                                                                     ║
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║  Version: 0.30.6+torch280                                           ║
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
Job chaiml-2fe5-c13f-linear-w01-v38-mkmlizer completed after 294.38s with status: succeeded
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: bash: no job control in this shell
Stopping job with name chaiml-2fe5-c13f-linear-w01-v38-mkmlizer
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║                                                                     ║
Pipeline stage MKMLizer completed in 297.71s
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║  The license key for the current software has been verified as      ║
run pipeline stage %s
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║  belonging to:                                                      ║
Running pipeline stage MKMLTemplater
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║                                                                     ║
Pipeline stage MKMLTemplater completed in 1.00s
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║  Chai Research Corp.                                                ║
run pipeline stage %s
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
Running pipeline stage MKMLDeployer
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
Creating inference service chaiml-2fe5-c13f-linear-w01-v38
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ║                                                                     ║
Waiting for inference service chaiml-2fe5-c13f-linear-w01-v38 to be ready
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: Downloaded to shared memory in 45.925s
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: Checking if ChaiML/7b07-69d4-linear-w01 already exists in ChaiML
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: Downloaded to shared memory in 23.899s
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmpfkv5l4a_, device:0
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: Checking if ChaiML/ca18-c13f-linear-w01 already exists in ChaiML
chaiml-7b07-69d4-linear-w01-v33-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmpr07yf5pi, device:0
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: bash: no job control in this shell
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: bash: no job control in this shell
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-02f4-69d4-linear-w01-v51-mkmlizer:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║                                                                     ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: bash: no job control in this shell
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║                                                                     ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║                                                                     ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║  Version: 0.30.6+torch280                                           ║
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: Downloaded to shared memory in 24.021s
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: Checking if ChaiML/4d70-fd43-linear-w01 already exists in ChaiML
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmpuh2m92ld, device:0
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║                                                                     ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║  belonging to:                                                      ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║                                                                     ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║                                                                     ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║  Version: 0.30.6+torch280                                           ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ║                                                                     ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║                                                                     ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║  belonging to:                                                      ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║                                                                     ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ║                                                                     ║
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: Downloaded to shared memory in 44.352s
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: Checking if ChaiML/02f4-69d4-linear-w01 already exists in ChaiML
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmpaxwec01f, device:0
chaiml-02f4-69d4-linear-w01-v51-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: Downloaded to shared memory in 47.960s
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: Checking if ChaiML/csfs-v3-3-dpo-lr5e6b01-lora already exists in ChaiML
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmpjuyoa30u, device:0
chaiml-csfs-v3-3-dpo-l-86358-v21-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: 
Loading 0:   0%|          | 0.00/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 4.00/363 [00:01<01:33, 3.85it/s]
Loading 0:   1%|          | 4.00/363 [00:01<01:33, 3.85it/s]
Loading 0:   2%|▏         | 6.00/363 [00:02<03:05, 1.92it/s]
Loading 0:   2%|▏         | 6.00/363 [00:02<03:05, 1.92it/s]
Loading 0:   4%|▎         | 13.0/363 [00:04<01:59, 2.93it/s]
Loading 0:   4%|▎         | 13.0/363 [00:04<01:59, 2.93it/s]
Loading 0:   4%|▍         | 15.0/363 [00:06<02:38, 2.19it/s]
Loading 0:   4%|▍         | 15.0/363 [00:06<02:38, 2.19it/s]
Loading 0:   6%|▌         | 22.0/363 [00:07<01:51, 3.05it/s]
Loading 0:   6%|▌         | 22.0/363 [00:07<01:51, 3.05it/s]
Loading 0:   7%|▋         | 24.0/363 [00:09<02:23, 2.37it/s]
Loading 0:   7%|▋         | 24.0/363 [00:09<02:23, 2.37it/s]
Loading 0:   9%|▊         | 31.0/363 [00:11<01:47, 3.09it/s]
Loading 0:   9%|▊         | 31.0/363 [00:11<01:47, 3.09it/s]
Loading 0:   9%|▉         | 33.0/363 [00:12<02:15, 2.43it/s]
Loading 0:   9%|▉         | 33.0/363 [00:12<02:15, 2.43it/s]
Loading 0:  11%|█         | 40.0/363 [00:14<01:47, 3.02it/s]
Loading 0:  11%|█         | 40.0/363 [00:14<01:47, 3.02it/s]
Loading 0:  12%|█▏        | 42.0/363 [00:16<02:13, 2.41it/s]
Loading 0:  12%|█▏        | 42.0/363 [00:16<02:13, 2.41it/s]
Loading 0:  13%|█▎        | 49.0/363 [00:17<01:42, 3.05it/s]
Loading 0:  13%|█▎        | 49.0/363 [00:17<01:42, 3.05it/s]
Loading 0:  14%|█▍        | 51.0/363 [00:19<02:07, 2.44it/s]
Loading 0:  14%|█▍        | 51.0/363 [00:19<02:07, 2.44it/s]
Loading 0:  16%|█▌        | 58.0/363 [00:20<01:39, 3.08it/s]
Loading 0:  16%|█▌        | 58.0/363 [00:20<01:39, 3.08it/s]
Loading 0:  17%|█▋        | 60.0/363 [00:22<02:03, 2.46it/s]
Loading 0:  17%|█▋        | 60.0/363 [00:22<02:03, 2.46it/s]
Loading 0:  18%|█▊        | 67.0/363 [00:24<01:36, 3.07it/s]
Loading 0:  18%|█▊        | 67.0/363 [00:24<01:36, 3.07it/s]
Loading 0:  19%|█▉        | 69.0/363 [00:25<01:59, 2.46it/s]
Loading 0:  19%|█▉        | 69.0/363 [00:25<01:59, 2.46it/s]
Loading 0:  21%|██        | 76.0/363 [00:27<01:32, 3.10it/s]
Loading 0:  21%|██        | 76.0/363 [00:27<01:32, 3.10it/s]
Loading 0:  21%|██▏       | 78.0/363 [00:29<01:55, 2.47it/s]
Loading 0:  21%|██▏       | 78.0/363 [00:29<01:55, 2.47it/s]
Loading 0:  23%|██▎       | 85.0/363 [00:30<01:29, 3.10it/s]
Loading 0:  23%|██▎       | 85.0/363 [00:30<01:29, 3.10it/s]
Loading 0:  24%|██▍       | 87.0/363 [00:32<01:52, 2.45it/s]
Loading 0:  24%|██▍       | 87.0/363 [00:32<01:52, 2.45it/s]
Loading 0:  26%|██▌       | 94.0/363 [00:33<01:27, 3.07it/s]
Loading 0:  26%|██▌       | 94.0/363 [00:33<01:27, 3.07it/s]
Loading 0:  26%|██▋       | 96.0/363 [00:35<01:48, 2.46it/s]
Loading 0:  26%|██▋       | 96.0/363 [00:35<01:48, 2.46it/s]
Loading 0:  28%|██▊       | 103/363 [00:37<01:24, 3.08it/s] 
Loading 0:  28%|██▊       | 103/363 [00:37<01:24, 3.08it/s]
Loading 0:  29%|██▉       | 105/363 [00:38<01:44, 2.47it/s]
Loading 0:  29%|██▉       | 105/363 [00:38<01:44, 2.47it/s]
Loading 0:  31%|███       | 112/363 [00:40<01:21, 3.09it/s]
Loading 0:  31%|███       | 112/363 [00:40<01:21, 3.09it/s]
Loading 0:  31%|███▏      | 114/363 [00:42<01:40, 2.47it/s]
Loading 0:  31%|███▏      | 114/363 [00:42<01:40, 2.47it/s]
Loading 0:  33%|███▎      | 121/363 [00:43<01:20, 3.02it/s]
Loading 0:  33%|███▎      | 121/363 [00:43<01:20, 3.02it/s]
Loading 0:  34%|███▍      | 123/363 [00:45<01:38, 2.44it/s]
Loading 0:  34%|███▍      | 123/363 [00:45<01:38, 2.44it/s]
Loading 0:  36%|███▌      | 130/363 [00:46<01:16, 3.06it/s]
Loading 0:  36%|███▌      | 130/363 [00:46<01:16, 3.06it/s]
Loading 0:  36%|███▋      | 132/363 [00:48<01:33, 2.46it/s]
Loading 0:  36%|███▋      | 132/363 [00:48<01:33, 2.46it/s]
Loading 0:  38%|███▊      | 139/363 [00:50<01:12, 3.08it/s]
Loading 0:  38%|███▊      | 139/363 [00:50<01:12, 3.08it/s]
Loading 0:  39%|███▉      | 141/363 [00:51<01:30, 2.47it/s]
Loading 0:  39%|███▉      | 141/363 [00:51<01:30, 2.47it/s]
Loading 0:  41%|████      | 148/363 [00:53<01:09, 3.09it/s]
Loading 0:  41%|████      | 148/363 [00:53<01:09, 3.09it/s]
Loading 0:  41%|████▏     | 150/363 [00:55<01:26, 2.47it/s]
Loading 0:  41%|████▏     | 150/363 [00:55<01:26, 2.47it/s]
Loading 0:  43%|████▎     | 157/363 [00:56<01:06, 3.10it/s]
Loading 0:  43%|████▎     | 157/363 [00:56<01:06, 3.10it/s]
Loading 0:  44%|████▍     | 159/363 [00:58<01:22, 2.48it/s]
Loading 0:  44%|████▍     | 159/363 [00:58<01:22, 2.48it/s]
Loading 0:  46%|████▌     | 166/363 [00:59<01:03, 3.11it/s]
Loading 0:  46%|████▌     | 166/363 [00:59<01:03, 3.11it/s]
Loading 0:  46%|████▋     | 168/363 [01:01<01:18, 2.49it/s]
Loading 0:  46%|████▋     | 168/363 [01:01<01:18, 2.49it/s]
Loading 0:  48%|████▊     | 175/363 [01:03<01:00, 3.13it/s]
Loading 0:  48%|████▊     | 175/363 [01:03<01:00, 3.13it/s]
Loading 0:  49%|████▉     | 177/363 [01:04<01:14, 2.50it/s]
Loading 0:  49%|████▉     | 177/363 [01:04<01:14, 2.50it/s]
Loading 0:  51%|█████     | 184/363 [01:06<00:57, 3.12it/s]
Loading 0:  51%|█████     | 184/363 [01:06<00:57, 3.12it/s]
Loading 0:  51%|█████     | 186/363 [01:07<01:11, 2.49it/s]
Loading 0:  51%|█████     | 186/363 [01:07<01:11, 2.49it/s]
Loading 0:  53%|█████▎    | 193/363 [01:09<00:54, 3.11it/s]
Loading 0:  53%|█████▎    | 193/363 [01:09<00:54, 3.11it/s]
Loading 0:  54%|█████▎    | 195/363 [01:11<01:07, 2.49it/s]
Loading 0:  54%|█████▎    | 195/363 [01:11<01:07, 2.49it/s]
Loading 0:  56%|█████▌    | 202/363 [01:12<00:53, 3.02it/s]
Loading 0:  56%|█████▌    | 202/363 [01:12<00:53, 3.02it/s]
Loading 0:  56%|█████▌    | 204/363 [01:14<01:05, 2.44it/s]
Loading 0:  56%|█████▌    | 204/363 [01:14<01:05, 2.44it/s]
Loading 0:  58%|█████▊    | 211/363 [01:16<00:49, 3.07it/s]
Loading 0:  58%|█████▊    | 211/363 [01:16<00:49, 3.07it/s]
Loading 0:  59%|█████▊    | 213/363 [01:17<01:00, 2.47it/s]
Loading 0:  59%|█████▊    | 213/363 [01:17<01:00, 2.47it/s]
Loading 0:  61%|██████    | 220/363 [01:19<00:46, 3.09it/s]
Loading 0:  61%|██████    | 220/363 [01:19<00:46, 3.09it/s]
Loading 0:  61%|██████    | 222/363 [01:21<00:56, 2.47it/s]
Loading 0:  61%|██████    | 222/363 [01:21<00:56, 2.47it/s]
Loading 0:  63%|██████▎   | 229/363 [01:22<00:43, 3.09it/s]
Loading 0:  63%|██████▎   | 229/363 [01:22<00:43, 3.09it/s]
Loading 0:  64%|██████▎   | 231/363 [01:24<00:53, 2.48it/s]
Loading 0:  64%|██████▎   | 231/363 [01:24<00:53, 2.48it/s]
Loading 0:  66%|██████▌   | 238/363 [01:25<00:40, 3.10it/s]
Loading 0:  66%|██████▌   | 238/363 [01:25<00:40, 3.10it/s]
Loading 0:  66%|██████▌   | 240/363 [01:27<00:49, 2.48it/s]
Loading 0:  66%|██████▌   | 240/363 [01:27<00:49, 2.48it/s]
Loading 0:  68%|██████▊   | 247/363 [01:28<00:37, 3.11it/s]
Loading 0:  68%|██████▊   | 247/363 [01:28<00:37, 3.11it/s]
Loading 0:  69%|██████▊   | 249/363 [01:30<00:45, 2.49it/s]
Loading 0:  69%|██████▊   | 249/363 [01:30<00:45, 2.49it/s]
Loading 0:  71%|███████   | 256/363 [01:32<00:34, 3.12it/s]
Loading 0:  71%|███████   | 256/363 [01:32<00:34, 3.12it/s]
Loading 0:  71%|███████   | 258/363 [01:33<00:42, 2.49it/s]
Loading 0:  71%|███████   | 258/363 [01:33<00:42, 2.49it/s]
Loading 0:  73%|███████▎  | 265/363 [01:35<00:31, 3.11it/s]
Loading 0:  73%|███████▎  | 265/363 [01:35<00:31, 3.11it/s]
Loading 0:  74%|███████▎  | 267/363 [01:37<00:38, 2.48it/s]
Loading 0:  74%|███████▎  | 267/363 [01:37<00:38, 2.48it/s]
Loading 0:  75%|███████▌  | 274/363 [01:38<00:28, 3.10it/s]
Loading 0:  75%|███████▌  | 274/363 [01:38<00:28, 3.10it/s]
Loading 0:  76%|███████▌  | 276/363 [01:40<00:35, 2.48it/s]
Loading 0:  76%|███████▌  | 276/363 [01:40<00:35, 2.48it/s]
Loading 0:  78%|███████▊  | 283/363 [01:42<00:26, 3.02it/s]
Loading 0:  78%|███████▊  | 283/363 [01:42<00:26, 3.02it/s]
Loading 0:  79%|███████▊  | 285/363 [01:43<00:31, 2.44it/s]
Loading 0:  79%|███████▊  | 285/363 [01:43<00:31, 2.44it/s]
Loading 0:  80%|████████  | 292/363 [01:45<00:23, 3.06it/s]
Loading 0:  80%|████████  | 292/363 [01:45<00:23, 3.06it/s]
Loading 0:  81%|████████  | 294/363 [01:46<00:28, 2.45it/s]
Loading 0:  81%|████████  | 294/363 [01:46<00:28, 2.45it/s]
Loading 0:  83%|████████▎ | 301/363 [01:48<00:20, 3.05it/s]
Loading 0:  83%|████████▎ | 301/363 [01:48<00:20, 3.05it/s]
Loading 0:  83%|████████▎ | 303/363 [01:50<00:24, 2.45it/s]
Loading 0:  83%|████████▎ | 303/363 [01:50<00:24, 2.45it/s]
Loading 0:  85%|████████▌ | 310/363 [01:51<00:17, 3.02it/s]
Loading 0:  85%|████████▌ | 310/363 [01:51<00:17, 3.02it/s]
Loading 0:  86%|████████▌ | 312/363 [01:53<00:20, 2.44it/s]
Loading 0:  86%|████████▌ | 312/363 [01:53<00:20, 2.44it/s]
Loading 0:  88%|████████▊ | 319/363 [01:55<00:14, 3.06it/s]
Loading 0:  88%|████████▊ | 319/363 [01:55<00:14, 3.06it/s]
Loading 0:  88%|████████▊ | 321/363 [01:56<00:17, 2.45it/s]
Loading 0:  88%|████████▊ | 321/363 [01:56<00:17, 2.45it/s]
Loading 0:  90%|█████████ | 328/363 [01:58<00:11, 3.07it/s]
Loading 0:  90%|█████████ | 328/363 [01:58<00:11, 3.07it/s]
Loading 0:  91%|█████████ | 330/363 [02:00<00:13, 2.47it/s]
Loading 0:  91%|█████████ | 330/363 [02:00<00:13, 2.47it/s]
Loading 0:  93%|█████████▎| 337/363 [02:01<00:08, 3.08it/s]
Loading 0:  93%|█████████▎| 337/363 [02:01<00:08, 3.08it/s]
Loading 0:  93%|█████████▎| 339/363 [02:03<00:09, 2.47it/s]
Loading 0:  93%|█████████▎| 339/363 [02:03<00:09, 2.47it/s]
Loading 0:  95%|█████████▌| 346/363 [02:04<00:05, 3.10it/s]
Loading 0:  95%|█████████▌| 346/363 [02:04<00:05, 3.10it/s]
Loading 0:  96%|█████████▌| 348/363 [02:06<00:06, 2.48it/s]
Loading 0:  96%|█████████▌| 348/363 [02:06<00:06, 2.48it/s]
Loading 0:  98%|█████████▊| 355/363 [02:08<00:02, 3.10it/s]
Loading 0:  98%|█████████▊| 355/363 [02:08<00:02, 3.10it/s]
Loading 0:  98%|█████████▊| 357/363 [02:09<00:02, 2.48it/s]
Loading 0:  98%|█████████▊| 357/363 [02:09<00:02, 2.48it/s]
Loading 0: 100%|██████████| 363/363 [02:10<00:00, 3.41it/s]
Loading 0: 100%|██████████| 363/363 [02:10<00:00, 3.41it/s]
                                                           

Loading 0: 100%|██████████| 363/363 [02:10<00:00, 2.78it/s]
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: The tokenizer you are loading from '/tmp/tmpr07yf5pi' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: quantized model in 141.785s
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: Processed model ChaiML/ca18-c13f-linear-w01 in 165.684s
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: creating bucket guanaco-mkml-models
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-ca18-c13f-linear-w01-v38/nvidia
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-ca18-c13f-linear-w01-v38/nvidia/config.json
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-mkml-models/chaiml-ca18-c13f-linear-w01-v38/nvidia/chat_template.jinja
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-ca18-c13f-linear-w01-v38/nvidia/special_tokens_map.json
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-ca18-c13f-linear-w01-v38/nvidia/tokenizer_config.json
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-ca18-c13f-linear-w01-v38/nvidia/tokenizer.json
Inference service chaiml-2fe5-c13f-linear-w01-v38 ready after 182.57392191886902s
Pipeline stage MKMLDeployer completed in 187.85s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.8155004978179932s
Received healthy response to inference request in 1.5893335342407227s
Received healthy response to inference request in 1.1920185089111328s
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: The tokenizer you are loading from '/tmp/tmpv80yc7l_' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: quantized model in 277.543s
Received healthy response to inference request in 1.3988900184631348s
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: Processed model ChaiML/ssnew-v5-dpo-lr5e6b01-lora in 323.148s
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: creating bucket guanaco-mkml-models
Received healthy response to inference request in 1.5542151927947998s
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
5 requests
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-ssnew-v5-dpo-lr-24415-v22/nvidia
0 failed requests
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-ssnew-v5-dpo-lr-24415-v22/nvidia/config.json
5th percentile: 1.2333928108215333
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-ssnew-v5-dpo-lr-24415-v22/nvidia/special_tokens_map.json
10th percentile: 1.2747671127319335
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-ssnew-v5-dpo-lr-24415-v22/nvidia/tokenizer_config.json
20th percentile: 1.3575157165527343
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-ssnew-v5-dpo-lr-24415-v22/nvidia/tokenizer.json
30th percentile: 1.4299550533294678
40th percentile: 1.4920851230621337
50th percentile: 1.5542151927947998
60th percentile: 1.568262529373169
70th percentile: 1.582309865951538
80th percentile: 1.634566926956177
90th percentile: 1.725033712387085
95th percentile: 1.770267105102539
99th percentile: 1.8064538192749022
mean time: 1.5099915504455566
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: 
Loading 0:   0%|          | 0.00/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5.00/363 [00:01<02:09, 2.77it/s]
Loading 0:   1%|▏         | 5.00/363 [00:01<02:09, 2.77it/s]
Loading 0:   2%|▏         | 8.00/363 [00:02<02:05, 2.84it/s]
Loading 0:   2%|▏         | 8.00/363 [00:02<02:05, 2.84it/s]
Loading 0:   4%|▎         | 13.0/363 [00:04<01:52, 3.12it/s]
Loading 0:   4%|▎         | 13.0/363 [00:04<01:52, 3.12it/s]
Loading 0:   4%|▍         | 15.0/363 [00:06<02:35, 2.24it/s]
Loading 0:   4%|▍         | 15.0/363 [00:06<02:35, 2.24it/s]
Loading 0:   6%|▌         | 22.0/363 [00:07<01:49, 3.12it/s]
Loading 0:   6%|▌         | 22.0/363 [00:07<01:49, 3.12it/s]
Loading 0:   7%|▋         | 24.0/363 [00:09<02:22, 2.38it/s]
Loading 0:   7%|▋         | 24.0/363 [00:09<02:22, 2.38it/s]
Loading 0:   9%|▊         | 31.0/363 [00:10<01:46, 3.11it/s]
Loading 0:   9%|▊         | 31.0/363 [00:10<01:46, 3.11it/s]
Loading 0:   9%|▉         | 33.0/363 [00:12<02:15, 2.44it/s]
Loading 0:   9%|▉         | 33.0/363 [00:12<02:15, 2.44it/s]
Loading 0:  11%|█         | 40.0/363 [00:14<01:47, 3.01it/s]
Loading 0:  11%|█         | 40.0/363 [00:14<01:47, 3.01it/s]
Loading 0:  12%|█▏        | 42.0/363 [00:15<02:13, 2.40it/s]
Loading 0:  12%|█▏        | 42.0/363 [00:15<02:13, 2.40it/s]
Loading 0:  13%|█▎        | 49.0/363 [00:17<01:43, 3.05it/s]
Loading 0:  13%|█▎        | 49.0/363 [00:17<01:43, 3.05it/s]
Loading 0:  14%|█▍        | 51.0/363 [00:19<02:08, 2.43it/s]
Loading 0:  14%|█▍        | 51.0/363 [00:19<02:08, 2.43it/s]
Loading 0:  16%|█▌        | 58.0/363 [00:20<01:39, 3.07it/s]
Loading 0:  16%|█▌        | 58.0/363 [00:20<01:39, 3.07it/s]
Loading 0:  17%|█▋        | 60.0/363 [00:22<02:03, 2.45it/s]
Loading 0:  17%|█▋        | 60.0/363 [00:22<02:03, 2.45it/s]
Loading 0:  18%|█▊        | 67.0/363 [00:23<01:36, 3.07it/s]
Loading 0:  18%|█▊        | 67.0/363 [00:23<01:36, 3.07it/s]
Loading 0:  19%|█▉        | 69.0/363 [00:25<01:59, 2.46it/s]
Loading 0:  19%|█▉        | 69.0/363 [00:25<01:59, 2.46it/s]
Loading 0:  21%|██        | 76.0/363 [00:27<01:33, 3.08it/s]
Loading 0:  21%|██        | 76.0/363 [00:27<01:33, 3.08it/s]
Loading 0:  21%|██▏       | 78.0/363 [00:28<01:56, 2.44it/s]
Loading 0:  21%|██▏       | 78.0/363 [00:28<01:56, 2.44it/s]
Loading 0:  23%|██▎       | 85.0/363 [00:30<01:30, 3.06it/s]
Loading 0:  23%|██▎       | 85.0/363 [00:30<01:30, 3.06it/s]
Loading 0:  24%|██▍       | 87.0/363 [00:32<01:52, 2.45it/s]
Loading 0:  24%|██▍       | 87.0/363 [00:32<01:52, 2.45it/s]
Loading 0:  26%|██▌       | 94.0/363 [00:33<01:27, 3.07it/s]
Loading 0:  26%|██▌       | 94.0/363 [00:33<01:27, 3.07it/s]
Loading 0:  26%|██▋       | 96.0/363 [00:35<01:48, 2.46it/s]
Loading 0:  26%|██▋       | 96.0/363 [00:35<01:48, 2.46it/s]
Loading 0:  28%|██▊       | 103/363 [00:36<01:24, 3.07it/s] 
Loading 0:  28%|██▊       | 103/363 [00:36<01:24, 3.07it/s]
Loading 0:  29%|██▉       | 105/363 [00:38<01:44, 2.47it/s]
Loading 0:  29%|██▉       | 105/363 [00:38<01:44, 2.47it/s]
Loading 0:  31%|███       | 112/363 [00:40<01:21, 3.08it/s]
Loading 0:  31%|███       | 112/363 [00:40<01:21, 3.08it/s]
Loading 0:  31%|███▏      | 114/363 [00:41<01:40, 2.47it/s]
Loading 0:  31%|███▏      | 114/363 [00:41<01:40, 2.47it/s]
Loading 0:  33%|███▎      | 121/363 [00:43<01:20, 3.00it/s]
Loading 0:  33%|███▎      | 121/363 [00:43<01:20, 3.00it/s]
Loading 0:  34%|███▍      | 123/363 [00:45<01:38, 2.42it/s]
Loading 0:  34%|███▍      | 123/363 [00:45<01:38, 2.42it/s]
Loading 0:  36%|███▌      | 130/363 [00:46<01:16, 3.06it/s]
Loading 0:  36%|███▌      | 130/363 [00:46<01:16, 3.06it/s]
Loading 0:  36%|███▋      | 132/363 [00:48<01:34, 2.45it/s]
Loading 0:  36%|███▋      | 132/363 [00:48<01:34, 2.45it/s]
Loading 0:  38%|███▊      | 139/363 [00:50<01:12, 3.07it/s]
Loading 0:  38%|███▊      | 139/363 [00:50<01:12, 3.07it/s]
Loading 0:  39%|███▉      | 141/363 [00:51<01:30, 2.47it/s]
Loading 0:  39%|███▉      | 141/363 [00:51<01:30, 2.47it/s]
Loading 0:  41%|████      | 148/363 [00:53<01:09, 3.09it/s]
Loading 0:  41%|████      | 148/363 [00:53<01:09, 3.09it/s]
Loading 0:  41%|████▏     | 150/363 [00:54<01:26, 2.47it/s]
Loading 0:  41%|████▏     | 150/363 [00:54<01:26, 2.47it/s]
Loading 0:  43%|████▎     | 157/363 [00:56<01:06, 3.09it/s]
Loading 0:  43%|████▎     | 157/363 [00:56<01:06, 3.09it/s]
Loading 0:  44%|████▍     | 159/363 [00:58<01:22, 2.48it/s]
Loading 0:  44%|████▍     | 159/363 [00:58<01:22, 2.48it/s]
Loading 0:  46%|████▌     | 166/363 [00:59<01:03, 3.10it/s]
Loading 0:  46%|████▌     | 166/363 [00:59<01:03, 3.10it/s]
Loading 0:  46%|████▋     | 168/363 [01:01<01:18, 2.48it/s]
Loading 0:  46%|████▋     | 168/363 [01:01<01:18, 2.48it/s]
Loading 0:  48%|████▊     | 175/363 [01:02<01:00, 3.10it/s]
Loading 0:  48%|████▊     | 175/363 [01:02<01:00, 3.10it/s]
Loading 0:  49%|████▉     | 177/363 [01:04<01:14, 2.48it/s]
Loading 0:  49%|████▉     | 177/363 [01:04<01:14, 2.48it/s]
Loading 0:  51%|█████     | 184/363 [01:06<00:57, 3.11it/s]
Loading 0:  51%|█████     | 184/363 [01:06<00:57, 3.11it/s]
Loading 0:  51%|█████     | 186/363 [01:07<01:11, 2.49it/s]
Loading 0:  51%|█████     | 186/363 [01:07<01:11, 2.49it/s]
Loading 0:  53%|█████▎    | 193/363 [01:09<00:54, 3.11it/s]
Loading 0:  53%|█████▎    | 193/363 [01:09<00:54, 3.11it/s]
Loading 0:  54%|█████▎    | 195/363 [01:11<01:07, 2.48it/s]
Loading 0:  54%|█████▎    | 195/363 [01:11<01:07, 2.48it/s]
Loading 0:  56%|█████▌    | 202/363 [01:12<00:53, 3.02it/s]
Loading 0:  56%|█████▌    | 202/363 [01:12<00:53, 3.02it/s]
Loading 0:  56%|█████▌    | 204/363 [01:14<01:05, 2.43it/s]
Loading 0:  56%|█████▌    | 204/363 [01:14<01:05, 2.43it/s]
Loading 0:  58%|█████▊    | 211/363 [01:15<00:49, 3.06it/s]
Loading 0:  58%|█████▊    | 211/363 [01:15<00:49, 3.06it/s]
Loading 0:  59%|█████▊    | 213/363 [01:17<01:00, 2.46it/s]
Loading 0:  59%|█████▊    | 213/363 [01:17<01:00, 2.46it/s]
Loading 0:  61%|██████    | 220/363 [01:19<00:46, 3.09it/s]
Loading 0:  61%|██████    | 220/363 [01:19<00:46, 3.09it/s]
Loading 0:  61%|██████    | 222/363 [01:20<00:57, 2.47it/s]
Loading 0:  61%|██████    | 222/363 [01:20<00:57, 2.47it/s]
Loading 0:  63%|██████▎   | 229/363 [01:22<00:43, 3.10it/s]
Loading 0:  63%|██████▎   | 229/363 [01:22<00:43, 3.10it/s]
Loading 0:  64%|██████▎   | 231/363 [01:24<00:53, 2.48it/s]
Loading 0:  64%|██████▎   | 231/363 [01:24<00:53, 2.48it/s]
Loading 0:  66%|██████▌   | 238/363 [01:25<00:40, 3.10it/s]
Loading 0:  66%|██████▌   | 238/363 [01:25<00:40, 3.10it/s]
Loading 0:  66%|██████▌   | 240/363 [01:27<00:49, 2.48it/s]
Loading 0:  66%|██████▌   | 240/363 [01:27<00:49, 2.48it/s]
Loading 0:  68%|██████▊   | 247/363 [01:28<00:37, 3.10it/s]
Loading 0:  68%|██████▊   | 247/363 [01:28<00:37, 3.10it/s]
Loading 0:  69%|██████▊   | 249/363 [01:30<00:45, 2.48it/s]
Loading 0:  69%|██████▊   | 249/363 [01:30<00:45, 2.48it/s]
Loading 0:  71%|███████   | 256/363 [01:32<00:34, 3.10it/s]
Loading 0:  71%|███████   | 256/363 [01:32<00:34, 3.10it/s]
Loading 0:  71%|███████   | 258/363 [01:33<00:42, 2.48it/s]
Loading 0:  71%|███████   | 258/363 [01:33<00:42, 2.48it/s]
Loading 0:  73%|███████▎  | 265/363 [01:35<00:31, 3.10it/s]
Loading 0:  73%|███████▎  | 265/363 [01:35<00:31, 3.10it/s]
Loading 0:  74%|███████▎  | 267/363 [01:37<00:38, 2.48it/s]
Loading 0:  74%|███████▎  | 267/363 [01:37<00:38, 2.48it/s]
Loading 0:  75%|███████▌  | 274/363 [01:38<00:28, 3.10it/s]
Loading 0:  75%|███████▌  | 274/363 [01:38<00:28, 3.10it/s]
Loading 0:  76%|███████▌  | 276/363 [01:40<00:35, 2.48it/s]
Loading 0:  76%|███████▌  | 276/363 [01:40<00:35, 2.48it/s]
Loading 0:  78%|███████▊  | 283/363 [01:41<00:26, 3.01it/s]
Loading 0:  78%|███████▊  | 283/363 [01:41<00:26, 3.01it/s]
Loading 0:  79%|███████▊  | 285/363 [01:43<00:32, 2.44it/s]
Loading 0:  79%|███████▊  | 285/363 [01:43<00:32, 2.44it/s]
Loading 0:  80%|████████  | 292/363 [01:45<00:23, 3.06it/s]
Loading 0:  80%|████████  | 292/363 [01:45<00:23, 3.06it/s]
Loading 0:  81%|████████  | 294/363 [01:46<00:28, 2.46it/s]
Loading 0:  81%|████████  | 294/363 [01:46<00:28, 2.46it/s]
Loading 0:  83%|████████▎ | 301/363 [01:48<00:20, 3.09it/s]
Loading 0:  83%|████████▎ | 301/363 [01:48<00:20, 3.09it/s]
Loading 0:  83%|████████▎ | 303/363 [01:50<00:24, 2.47it/s]
Loading 0:  83%|████████▎ | 303/363 [01:50<00:24, 2.47it/s]
Loading 0:  85%|████████▌ | 310/363 [01:51<00:17, 3.09it/s]
Loading 0:  85%|████████▌ | 310/363 [01:51<00:17, 3.09it/s]
Loading 0:  86%|████████▌ | 312/363 [01:53<00:20, 2.47it/s]
Loading 0:  86%|████████▌ | 312/363 [01:53<00:20, 2.47it/s]
Loading 0:  88%|████████▊ | 319/363 [01:54<00:14, 3.09it/s]
Loading 0:  88%|████████▊ | 319/363 [01:54<00:14, 3.09it/s]
Loading 0:  88%|████████▊ | 321/363 [01:56<00:16, 2.47it/s]
Loading 0:  88%|████████▊ | 321/363 [01:56<00:16, 2.47it/s]
Loading 0:  90%|█████████ | 328/363 [01:58<00:11, 3.09it/s]
Loading 0:  90%|█████████ | 328/363 [01:58<00:11, 3.09it/s]
Loading 0:  91%|█████████ | 330/363 [01:59<00:13, 2.48it/s]
Loading 0:  91%|█████████ | 330/363 [01:59<00:13, 2.48it/s]
Loading 0:  93%|█████████▎| 337/363 [02:01<00:08, 3.09it/s]
Loading 0:  93%|█████████▎| 337/363 [02:01<00:08, 3.09it/s]
Loading 0:  93%|█████████▎| 339/363 [02:03<00:09, 2.48it/s]
Loading 0:  93%|█████████▎| 339/363 [02:03<00:09, 2.48it/s]
Loading 0:  95%|█████████▌| 346/363 [02:04<00:05, 3.08it/s]
Loading 0:  95%|█████████▌| 346/363 [02:04<00:05, 3.08it/s]
Loading 0:  96%|█████████▌| 348/363 [02:06<00:06, 2.47it/s]
Loading 0:  96%|█████████▌| 348/363 [02:06<00:06, 2.47it/s]
Loading 0:  98%|█████████▊| 355/363 [02:07<00:02, 3.10it/s]
Loading 0:  98%|█████████▊| 355/363 [02:07<00:02, 3.10it/s]
Loading 0:  98%|█████████▊| 357/363 [02:09<00:02, 2.47it/s]
Loading 0:  98%|█████████▊| 357/363 [02:09<00:02, 2.47it/s]
Loading 0: 100%|██████████| 363/363 [02:10<00:00, 3.40it/s]
Loading 0: 100%|██████████| 363/363 [02:10<00:00, 3.40it/s]
                                                           

Loading 0: 100%|██████████| 363/363 [02:10<00:00, 2.79it/s]
Pipeline stage StressChecker completed in 28.19s
chaiml-ca18-c13f-linear-w01-v38-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-ca18-c13f-linear-w01-v38/nvidia/flywheel_model.0.safetensors
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 5.03s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: The tokenizer you are loading from '/tmp/tmpuh2m92ld' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
run_pipeline:run_in_cloud %s
chaiml-ssnew-v5-dpo-lr-24415-v22-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-ssnew-v5-dpo-lr-24415-v22/nvidia/flywheel_model.1.safetensors
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: quantized model in 141.669s
Job chaiml-ca18-c13f-linear-w01-v38-mkmlizer completed after 530.13s with status: succeeded
starting trigger_guanaco_pipeline args=%s
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: Processed model ChaiML/4d70-fd43-linear-w01 in 165.690s
Stopping job with name chaiml-ca18-c13f-linear-w01-v38-mkmlizer
triggered trigger_guanaco_pipeline args=%s
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: creating bucket guanaco-mkml-models
Pipeline stage MKMLizer completed in 534.35s
Pipeline stage TriggerMKMLProfilingPipeline completed in 5.87s
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
run pipeline stage %s
Shutdown handler de-registered
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-4d70-fd43-linear-w01-v29/nvidia
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: The tokenizer you are loading from '/tmp/tmpwlstf01d' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
Running pipeline stage MKMLTemplater
chaiml-4d70-fd43-linear-w01-v29-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-4d70-fd43-linear-w01-v29/nvidia
chaiml-2a6f-69d4-linear-w01-v24-mkmlizer: The tokenizer you are loading from '/tmp/tmpwlstf01d' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
Running pipeline stage MKMLTemplater
chaiml-2fe5-c13f-linear-w01_v38 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-2fe5-c13f-linear-w01-v38-profiler
Waiting for inference service chaiml-2fe5-c13f-linear-w01-v38-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Generating Leaderboard row for %s
Generated Leaderboard row for %s
Pipeline stage OfflineFamilyFriendlyScorer completed in 2120.87s
Shutdown handler de-registered
chaiml-2fe5-c13f-linear-w01_v38 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-2fe5-c13f-linear-w01_v38
Shutdown handler not registered because Python interpreter is not running in the main thread
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
run pipeline %s
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
run pipeline stage %s
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Running pipeline stage MKMLDeleter
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Checking if service chaiml-2fe5-c13f-linear-w01-v38 is running
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Tearing down inference service chaiml-2fe5-c13f-linear-w01-v38
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Service chaiml-2fe5-c13f-linear-w01-v38 has been torndown
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Pipeline stage MKMLDeleter completed in 63.24s
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
run pipeline stage %s
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Running pipeline stage MKMLModelDeleter
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Cleaning model data from S3
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Cleaning model data from model cache
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Deleting key chaiml-2fe5-c13f-linear-w01-v38/nvidia/chat_template.jinja from bucket guanaco-mkml-models
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Deleting key chaiml-2fe5-c13f-linear-w01-v38/nvidia/config.json from bucket guanaco-mkml-models
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Deleting key chaiml-2fe5-c13f-linear-w01-v38/nvidia/flywheel_model.0.safetensors from bucket guanaco-mkml-models
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Deleting key chaiml-2fe5-c13f-linear-w01-v38/nvidia/special_tokens_map.json from bucket guanaco-mkml-models
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Deleting key chaiml-2fe5-c13f-linear-w01-v38/nvidia/tokenizer.json from bucket guanaco-mkml-models
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Deleting key chaiml-2fe5-c13f-linear-w01-v38/nvidia/tokenizer_config.json from bucket guanaco-mkml-models
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Pipeline stage MKMLModelDeleter completed in 184.47s
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Shutdown handler de-registered
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation
Falling back to EndpointApi.from_submission implementation