submission_id: chaiml-cai-synth-v4-cos_97956_v1
developer_uid: chai_backend_admin
status: failed
model_repo: ChaiML/cai-synth-v4_cosine-lr2e6g16
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['User:', '</s>', 'You:'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 128}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '####\n{bot_name}:', 'truncate_by_message': True}
timestamp: 2025-12-05T01:00:08+00:00
model_name: training123
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-cai-synth-v4-cos-97956-v1-mkmlizer
Waiting for job on chaiml-cai-synth-v4-cos-97956-v1-mkmlizer to finish
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: bash: no job control in this shell
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║                                                                     ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║                                                                     ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║  Version: 0.30.6+torch280                                           ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║                                                                     ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║  belonging to:                                                      ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║                                                                     ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ║                                                                     ║
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: Downloaded to shared memory in 140.118s
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: Checking if ChaiML/cai-synth-v4_cosine-lr2e6g16 already exists in ChaiML
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp1jy57yb2, device:0
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: 
Loading 0:   0%|          | 0.00/363 [00:00<?, ?it/s]
Loading 0:   9%|▉         | 32.0/363 [00:01<00:10, 30.3it/s]
Loading 0:   9%|▉         | 32.0/363 [00:01<00:10, 30.3it/s]
Loading 0:  15%|█▌        | 55.0/363 [00:02<00:11, 25.8it/s]
Loading 0:  15%|█▌        | 55.0/363 [00:02<00:11, 25.8it/s]
Loading 0:  22%|██▏       | 80.0/363 [00:03<00:11, 24.6it/s]
Loading 0:  22%|██▏       | 80.0/363 [00:03<00:11, 24.6it/s]
Loading 0:  29%|██▉       | 107/363 [00:04<00:10, 25.1it/s] 
Loading 0:  29%|██▉       | 107/363 [00:04<00:10, 25.1it/s]
Loading 0:  38%|███▊      | 138/363 [00:05<00:08, 26.9it/s]
Loading 0:  38%|███▊      | 138/363 [00:05<00:08, 26.9it/s]
Loading 0:  45%|████▍     | 163/363 [00:06<00:07, 26.2it/s]
Loading 0:  45%|████▍     | 163/363 [00:06<00:07, 26.2it/s]
Loading 0:  52%|█████▏    | 189/363 [00:07<00:06, 25.5it/s]
Loading 0:  52%|█████▏    | 189/363 [00:07<00:06, 25.5it/s]
Loading 0:  52%|█████▏    | 189/363 [00:20<00:06, 25.5it/s]
Loading 0:  55%|█████▌    | 201/363 [00:20<00:36, 4.43it/s]
Loading 0:  55%|█████▌    | 201/363 [00:20<00:36, 4.43it/s]
Loading 0:  62%|██████▏   | 224/363 [00:21<00:23, 5.93it/s]
Loading 0:  62%|██████▏   | 224/363 [00:21<00:23, 5.93it/s]
Loading 0:  70%|███████   | 255/363 [00:22<00:12, 8.56it/s]
Loading 0:  70%|███████   | 255/363 [00:22<00:12, 8.56it/s]
Loading 0:  77%|███████▋  | 280/363 [00:23<00:07, 10.7it/s]
Loading 0:  77%|███████▋  | 280/363 [00:23<00:07, 10.7it/s]
Loading 0:  84%|████████▍ | 306/363 [00:24<00:04, 12.9it/s]
Loading 0:  84%|████████▍ | 306/363 [00:24<00:04, 12.9it/s]
Loading 0:  91%|█████████ | 329/363 [00:25<00:02, 14.7it/s]
Loading 0:  91%|█████████ | 329/363 [00:25<00:02, 14.7it/s]
Loading 0:  97%|█████████▋| 353/363 [00:27<00:00, 16.4it/s]
Loading 0:  97%|█████████▋| 353/363 [00:27<00:00, 16.4it/s]
Loading 0: 100%|██████████| 363/363 [00:27<00:00, 17.4it/s]
Loading 0: 100%|██████████| 363/363 [00:27<00:00, 17.4it/s]
                                                           

Loading 0: 100%|██████████| 363/363 [00:27<00:00, 13.3it/s]
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: The tokenizer you are loading from '/tmp/tmp1jy57yb2' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: quantized model in 44.337s
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: Processed model ChaiML/cai-synth-v4_cosine-lr2e6g16 in 184.456s
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-97956-v1/nvidia
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-97956-v1/nvidia/config.json
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-97956-v1/nvidia/special_tokens_map.json
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-97956-v1/nvidia/tokenizer_config.json
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-97956-v1/nvidia/tokenizer.json
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-97956-v1/nvidia/flywheel_model.1.safetensors
chaiml-cai-synth-v4-cos-97956-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-cai-synth-v4-cos-97956-v1/nvidia/flywheel_model.0.safetensors
Job chaiml-cai-synth-v4-cos-97956-v1-mkmlizer completed after 287.69s with status: succeeded
Stopping job with name chaiml-cai-synth-v4-cos-97956-v1-mkmlizer
Pipeline stage MKMLizer completed in 288.15s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-cai-synth-v4-cos-97956-v1
Waiting for inference service chaiml-cai-synth-v4-cos-97956-v1 to be ready
Inference service chaiml-cai-synth-v4-cos-97956-v1 ready after 190.86187553405762s
Pipeline stage MKMLDeployer completed in 191.39s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.986063241958618s
Received healthy response to inference request in 4.616195917129517s
Received healthy response to inference request in 4.096690893173218s
Received healthy response to inference request in 4.3785927295684814s
Received healthy response to inference request in 4.168109893798828s
5 requests
0 failed requests
5th percentile: 4.11097469329834
10th percentile: 4.125258493423462
20th percentile: 4.153826093673706
30th percentile: 4.210206460952759
40th percentile: 4.29439959526062
50th percentile: 4.3785927295684814
60th percentile: 4.4736340045928955
70th percentile: 4.56867527961731
80th percentile: 4.690169382095337
90th percentile: 4.838116312026978
95th percentile: 4.912089776992798
99th percentile: 4.971268548965454
mean time: 4.449130535125732
%s, retrying in %s seconds...
Received healthy response to inference request in 4.2370898723602295s
Received healthy response to inference request in 4.237483501434326s
Received healthy response to inference request in 4.529986143112183s
Received healthy response to inference request in 4.122598171234131s
Received healthy response to inference request in 4.037091493606567s
5 requests
0 failed requests
5th percentile: 4.05419282913208
10th percentile: 4.071294164657592
20th percentile: 4.105496835708618
30th percentile: 4.145496511459351
40th percentile: 4.19129319190979
50th percentile: 4.2370898723602295
60th percentile: 4.237247323989868
70th percentile: 4.237404775619507
80th percentile: 4.2959840297698975
90th percentile: 4.41298508644104
95th percentile: 4.471485614776611
99th percentile: 4.518286037445068
mean time: 4.232849836349487
%s, retrying in %s seconds...
Received healthy response to inference request in 4.542205572128296s
Received healthy response to inference request in 4.3494672775268555s
Received healthy response to inference request in 4.41740083694458s
Received healthy response to inference request in 4.489711046218872s
Received healthy response to inference request in 4.22675633430481s
5 requests
0 failed requests
5th percentile: 4.251298522949218
10th percentile: 4.275840711593628
20th percentile: 4.324925088882447
30th percentile: 4.363053989410401
40th percentile: 4.39022741317749
50th percentile: 4.41740083694458
60th percentile: 4.4463249206542965
70th percentile: 4.475249004364014
80th percentile: 4.500209951400757
90th percentile: 4.5212077617645265
95th percentile: 4.531706666946411
99th percentile: 4.540105791091919
mean time: 4.405108213424683
clean up pipeline due to error=DeploymentChecksError('Unacceptable 70th percentile latency 4.475249004364014s')
Shutdown handler de-registered
chaiml-cai-synth-v4-cos_97956_v1 status is now failed due to DeploymentManager action