Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v11-noname-44244-v3-uploader
Waiting for job on chaiml-kimid-v11-noname-44244-v3-uploader to finish
chaiml-kimid-v11-noname-44244-v3-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v11-noname-44244-v3-uploader: bash: no job control in this shell
chaiml-kimid-v11-noname-44244-v3-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v11-noname-44244-v3-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v11-noname-44244-v3-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v11-noname-44244-v3-uploader: ║ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ https://mk1.ai ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ belonging to: ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v11-noname-44244-v3-uploader: ║ ║
chaiml-kimid-v11-noname-44244-v3-uploader: ╚═════════════════════════════════════════════════════════════════════╝
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v11-noname-44244-v3-uploader: Downloaded to shared memory in 134.476s
chaiml-kimid-v11-noname-44244-v3-uploader: Processed model ChaiML/kimid-v11-noname-kimidv4ann-lr1e5ep2r64g8b01-int4-mixed in 194.674s
chaiml-kimid-v11-noname-44244-v3-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v11-noname-44244-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v11-noname-44244-v3-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/.gitattributes
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/added_tokens.json
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/generation_config.json
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/chat_template.jinja
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/tokenizer_config.json
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/special_tokens_map.json
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/config.json
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/quantization_config.json
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/vocab.json
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/merges.txt
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model.safetensors.index.json
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00027-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00009-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00007-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00016-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00006-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00011-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00026-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00018-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00002-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00024-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00021-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00020-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00019-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00025-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00008-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00003-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00004-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00001-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00023-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00013-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00010-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00005-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00022-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00017-of-00027.safetensors
chaiml-kimid-v11-noname-44244-v3-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v11-noname-44244-v3/model-00012-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
Job chaiml-kimid-v11-noname-44244-v3-uploader completed after 704.33s with status: succeeded
Stopping job with name chaiml-kimid-v11-noname-44244-v3-uploader
Pipeline stage VLLMUploader completed in 706.63s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.30s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v11-noname-44244-v3
Waiting for inference service chaiml-kimid-v11-noname-44244-v3 to be ready
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission blend_samot_2026-01-22: HTTPConnectionPool(host='guanaco-model-mesh-load-balancer.model-mesh.k2.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service chaiml-kimid-v11-noname-44244-v3 ready after 462.5588629245758s
Pipeline stage VLLMDeployer completed in 463.20s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.890700101852417s
Received healthy response to inference request in 1.884331464767456s
Received healthy response to inference request in 1.8507483005523682s
Received healthy response to inference request in 2.0775671005249023s
Received healthy response to inference request in 2.0143916606903076s
Received healthy response to inference request in 1.9390039443969727s
Received healthy response to inference request in 2.0102334022521973s
Received healthy response to inference request in 1.9569034576416016s
Received healthy response to inference request in 1.8739893436431885s
Received healthy response to inference request in 2.536411762237549s
Received healthy response to inference request in 1.9003756046295166s
Received healthy response to inference request in 1.9293937683105469s
Received healthy response to inference request in 1.8222033977508545s
Received healthy response to inference request in 2.087493896484375s
Received healthy response to inference request in 1.9390270709991455s
Received healthy response to inference request in 1.9442427158355713s
Received healthy response to inference request in 2.381523609161377s
Received healthy response to inference request in 2.011610984802246s
Received healthy response to inference request in 1.8317713737487793s
Received healthy response to inference request in 2.018869161605835s
Received healthy response to inference request in 1.7767069339752197s
Received healthy response to inference request in 1.8649296760559082s
Received healthy response to inference request in 1.9432215690612793s
Received healthy response to inference request in 1.8351337909698486s
Received healthy response to inference request in 2.627718925476074s
Received healthy response to inference request in 1.8935766220092773s
Received healthy response to inference request in 1.88167405128479s
Received healthy response to inference request in 1.995851993560791s
Received healthy response to inference request in 2.807971477508545s
Received healthy response to inference request in 1.7775640487670898s
30 requests
0 failed requests
5th percentile: 1.797651755809784
10th percentile: 1.830814576148987
20th percentile: 1.8620934009552002
30th percentile: 1.8835342407226563
40th percentile: 1.917786502838135
50th percentile: 1.9411243200302124
60th percentile: 1.9724828720092773
70th percentile: 2.0124451875686646
80th percentile: 2.0795524597167967
90th percentile: 2.5455424785614014
95th percentile: 2.7268578290939325
99th percentile: 2.866708800792694
mean time: 2.0435047070185344
Pipeline stage StressChecker completed in 65.51s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.48s
Shutdown handler de-registered
chaiml-kimid-v11-noname_44244_v3 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
Generating Leaderboard row for %s
Generated Leaderboard row for %s
Pipeline stage OfflineFamilyFriendlyScorer completed in 5304.11s
Shutdown handler de-registered