submission_id: chaiml-cai-synth-v4-cos_41572_v1
developer_uid: chai_backend_admin
status: failed
model_repo: ChaiML/cai-synth-v4_cosine-lr5e6g32
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '</s>', 'User:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 128}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '####\n{bot_name}:', 'truncate_by_message': True}
timestamp: 2025-12-05T01:00:51+00:00
model_name: training123
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-cai-synth-v4-cos-41572-v1-mkmlizer
Waiting for job on chaiml-cai-synth-v4-cos-41572-v1-mkmlizer to finish
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: bash: no job control in this shell
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║                                                                     ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║                                                                     ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║  Version: 0.30.6+torch280                                           ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║                                                                     ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║  belonging to:                                                      ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║                                                                     ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ║                                                                     ║
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: Downloaded to shared memory in 89.970s
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: Checking if ChaiML/cai-synth-v4_cosine-lr5e6g32 already exists in ChaiML
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpl3jkty9q, device:0
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: 
Loading 0:   0%|          | 0.00/363 [00:00<?, ?it/s]
Loading 0:   9%|▉         | 32.0/363 [00:01<00:10, 31.3it/s]
Loading 0:   9%|▉         | 32.0/363 [00:01<00:10, 31.3it/s]
Loading 0:  15%|█▌        | 55.0/363 [00:02<00:11, 26.4it/s]
Loading 0:  15%|█▌        | 55.0/363 [00:02<00:11, 26.4it/s]
Loading 0:  22%|██▏       | 80.0/363 [00:03<00:11, 25.0it/s]
Loading 0:  22%|██▏       | 80.0/363 [00:03<00:11, 25.0it/s]
Loading 0:  29%|██▉       | 107/363 [00:04<00:10, 25.5it/s] 
Loading 0:  29%|██▉       | 107/363 [00:04<00:10, 25.5it/s]
Loading 0:  38%|███▊      | 138/363 [00:05<00:08, 27.2it/s]
Loading 0:  38%|███▊      | 138/363 [00:05<00:08, 27.2it/s]
Loading 0:  44%|████▍     | 160/363 [00:06<00:07, 25.4it/s]
Loading 0:  44%|████▍     | 160/363 [00:06<00:07, 25.4it/s]
Loading 0:  52%|█████▏    | 187/363 [00:07<00:06, 25.8it/s]
Loading 0:  52%|█████▏    | 187/363 [00:07<00:06, 25.8it/s]
Loading 0:  52%|█████▏    | 187/363 [00:20<00:06, 25.8it/s]
Loading 0:  55%|█████▌    | 201/363 [00:20<00:36, 4.45it/s]
Loading 0:  55%|█████▌    | 201/363 [00:20<00:36, 4.45it/s]
Loading 0:  62%|██████▏   | 224/363 [00:21<00:23, 5.93it/s]
Loading 0:  62%|██████▏   | 224/363 [00:21<00:23, 5.93it/s]
Loading 0:  70%|███████   | 255/363 [00:22<00:12, 8.55it/s]
Loading 0:  70%|███████   | 255/363 [00:22<00:12, 8.55it/s]
Loading 0:  77%|███████▋  | 280/363 [00:23<00:07, 10.7it/s]
Loading 0:  77%|███████▋  | 280/363 [00:23<00:07, 10.7it/s]
Loading 0:  84%|████████▍ | 306/363 [00:24<00:04, 13.0it/s]
Loading 0:  84%|████████▍ | 306/363 [00:24<00:04, 13.0it/s]
Loading 0:  93%|█████████▎| 336/363 [00:26<00:01, 15.8it/s]
Loading 0:  93%|█████████▎| 336/363 [00:26<00:01, 15.8it/s]
Loading 0:  98%|█████████▊| 357/363 [00:27<00:00, 16.8it/s]
Loading 0:  98%|█████████▊| 357/363 [00:27<00:00, 16.8it/s]
Loading 0: 100%|██████████| 363/363 [00:27<00:00, 17.7it/s]
Loading 0: 100%|██████████| 363/363 [00:27<00:00, 17.7it/s]
                                                           

Loading 0: 100%|██████████| 363/363 [00:27<00:00, 13.3it/s]
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: The tokenizer you are loading from '/tmp/tmpl3jkty9q' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: quantized model in 44.340s
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: Processed model ChaiML/cai-synth-v4_cosine-lr5e6g32 in 134.310s
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-41572-v1/nvidia
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-41572-v1/nvidia/config.json
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-41572-v1/nvidia/special_tokens_map.json
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-41572-v1/nvidia/tokenizer_config.json
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-41572-v1/nvidia/tokenizer.json
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-41572-v1/nvidia/flywheel_model.1.safetensors
chaiml-cai-synth-v4-cos-41572-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-41572-v1/nvidia/flywheel_model.0.safetensors
Job chaiml-cai-synth-v4-cos-41572-v1-mkmlizer completed after 216.0s with status: succeeded
Stopping job with name chaiml-cai-synth-v4-cos-41572-v1-mkmlizer
Pipeline stage MKMLizer completed in 216.56s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-cai-synth-v4-cos-41572-v1
Waiting for inference service chaiml-cai-synth-v4-cos-41572-v1 to be ready
Inference service chaiml-cai-synth-v4-cos-41572-v1 ready after 191.18554878234863s
Pipeline stage MKMLDeployer completed in 191.73s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.404439926147461s
Received healthy response to inference request in 4.430876016616821s
Received healthy response to inference request in 4.267982721328735s
Received healthy response to inference request in 4.209833145141602s
Received healthy response to inference request in 4.439373254776001s
5 requests
0 failed requests
5th percentile: 4.221463060379028
10th percentile: 4.233092975616455
20th percentile: 4.256352806091309
30th percentile: 4.295274162292481
40th percentile: 4.349857044219971
50th percentile: 4.404439926147461
60th percentile: 4.415014362335205
70th percentile: 4.425588798522949
80th percentile: 4.432575464248657
90th percentile: 4.435974359512329
95th percentile: 4.437673807144165
99th percentile: 4.439033365249633
mean time: 4.350501012802124
%s, retrying in %s seconds...
Received healthy response to inference request in 4.321398973464966s
Received healthy response to inference request in 4.2751383781433105s
Received healthy response to inference request in 4.082649230957031s
Received healthy response to inference request in 4.190748453140259s
Received healthy response to inference request in 4.44582724571228s
5 requests
0 failed requests
5th percentile: 4.104269075393677
10th percentile: 4.125888919830322
20th percentile: 4.1691286087036135
30th percentile: 4.207626438140869
40th percentile: 4.24138240814209
50th percentile: 4.2751383781433105
60th percentile: 4.293642616271972
70th percentile: 4.312146854400635
80th percentile: 4.346284627914429
90th percentile: 4.3960559368133545
95th percentile: 4.420941591262817
99th percentile: 4.440850114822387
mean time: 4.263152456283569
%s, retrying in %s seconds...
Received healthy response to inference request in 4.136300802230835s
Received healthy response to inference request in 4.0342652797698975s
Received healthy response to inference request in 4.227530002593994s
Received healthy response to inference request in 4.113095998764038s
Received healthy response to inference request in 4.054659366607666s
5 requests
0 failed requests
5th percentile: 4.038344097137451
10th percentile: 4.042422914505005
20th percentile: 4.050580549240112
30th percentile: 4.06634669303894
40th percentile: 4.08972134590149
50th percentile: 4.113095998764038
60th percentile: 4.122377920150757
70th percentile: 4.131659841537475
80th percentile: 4.154546642303467
90th percentile: 4.191038322448731
95th percentile: 4.209284162521362
99th percentile: 4.223880834579468
mean time: 4.1131702899932865
clean up pipeline due to error=DeploymentChecksError('Unacceptable 70th percentile latency 4.131659841537475s')
Shutdown handler de-registered
chaiml-cai-synth-v4-cos_41572_v1 status is now failed due to DeploymentManager action