Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v8b-kimid-63800-v19-uploader
Waiting for job on chaiml-kimid-v8b-kimid-63800-v19-uploader to finish
chaiml-kimid-v8b-kimid-63800-v19-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v8b-kimid-63800-v19-uploader: bash: no job control in this shell
chaiml-kimid-v8b-kimid-63800-v19-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v8b-kimid-63800-v19-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v8b-kimid-63800-v19-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ https://mk1.ai ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ belonging to: ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v19-uploader: ╚═════════════════════════════════════════════════════════════════════╝
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v8b-kimid-63800-v19-uploader: Downloaded to shared memory in 128.401s
chaiml-kimid-v8b-kimid-63800-v19-uploader: Processed model ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-int4-mixed in 222.283s
chaiml-kimid-v8b-kimid-63800-v19-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v8b-kimid-63800-v19-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v8b-kimid-63800-v19-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/README.md
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/.gitattributes
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/chat_template.jinja
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/generation_config.json
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/special_tokens_map.json
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/added_tokens.json
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/tokenizer_config.json
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/quantization_config.json
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/config.json
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/merges.txt
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/vocab.json
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/tokenizer.json
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model.safetensors.index.json
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00027-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00003-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00022-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00020-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00019-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00013-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00010-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00012-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00024-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00005-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00025-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00009-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00018-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00007-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00023-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00021-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00014-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00015-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00001-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00004-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00026-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00008-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00017-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00006-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00016-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v19-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v19/model-00002-of-00027.safetensors
Job chaiml-kimid-v8b-kimid-63800-v19-uploader completed after 439.77s with status: succeeded
Stopping job with name chaiml-kimid-v8b-kimid-63800-v19-uploader
Pipeline stage VLLMUploader completed in 440.27s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v8b-kimid-63800-v19
Waiting for inference service chaiml-kimid-v8b-kimid-63800-v19 to be ready
Inference service chaiml-kimid-v8b-kimid-63800-v19 ready after 452.4974021911621s
Pipeline stage VLLMDeployer completed in 453.03s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.9129536151885986s
Received healthy response to inference request in 1.7640001773834229s
Received healthy response to inference request in 1.7814574241638184s
Received healthy response to inference request in 2.184041738510132s
Received healthy response to inference request in 1.7295591831207275s
Received healthy response to inference request in 1.9413714408874512s
Received healthy response to inference request in 1.9131076335906982s
Received healthy response to inference request in 1.8063349723815918s
Received healthy response to inference request in 2.3015553951263428s
Received healthy response to inference request in 1.9150505065917969s
Received healthy response to inference request in 1.930614948272705s
Received healthy response to inference request in 1.7791588306427002s
Received healthy response to inference request in 1.7789645195007324s
Received healthy response to inference request in 1.7263708114624023s
Received healthy response to inference request in 1.8778905868530273s
Received healthy response to inference request in 1.6859135627746582s
Received healthy response to inference request in 1.9513304233551025s
Received healthy response to inference request in 1.8038499355316162s
Received healthy response to inference request in 2.0574803352355957s
Received healthy response to inference request in 1.7783286571502686s
Received healthy response to inference request in 1.874622106552124s
Received healthy response to inference request in 1.8943493366241455s
Received healthy response to inference request in 2.1705081462860107s
Received healthy response to inference request in 1.903987169265747s
Received healthy response to inference request in 1.8780276775360107s
Received healthy response to inference request in 2.1041605472564697s
Received healthy response to inference request in 1.9312448501586914s
Received healthy response to inference request in 1.8107469081878662s
Received healthy response to inference request in 1.704892873764038s
Received healthy response to inference request in 1.9090607166290283s
30 requests
0 failed requests
5th percentile: 1.714557945728302
10th percentile: 1.729240345954895
20th percentile: 1.7788373470306396
30th percentile: 1.797132182121277
40th percentile: 1.849072027206421
50th percentile: 1.8861885070800781
60th percentile: 1.9106178760528565
70th percentile: 1.9197198390960692
80th percentile: 1.9433632373809815
90th percentile: 2.110795307159424
95th percentile: 2.177951622009277
99th percentile: 2.2674764347076417
mean time: 1.8933645009994506
Pipeline stage StressChecker completed in 65.63s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.69s
Shutdown handler de-registered
chaiml-kimid-v8b-kimid_63800_v19 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Generating Leaderboard row for %s
Generated Leaderboard row for %s
Pipeline stage OfflineFamilyFriendlyScorer completed in 2681.87s
Shutdown handler de-registered
chaiml-kimid-v8b-kimid_63800_v19 status is now inactive due to auto deactivation removed underperforming models