Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v8b-kimid-63800-v21-uploader
Waiting for job on chaiml-kimid-v8b-kimid-63800-v21-uploader to finish
chaiml-kimid-v8b-kimid-63800-v21-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v8b-kimid-63800-v21-uploader: bash: no job control in this shell
chaiml-kimid-v8b-kimid-63800-v21-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v8b-kimid-63800-v21-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v8b-kimid-63800-v21-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ https://mk1.ai ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ belonging to: ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ║ ║
chaiml-kimid-v8b-kimid-63800-v21-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v8b-kimid-63800-v21-uploader: Downloaded to shared memory in 135.074s
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v8b-kimid-63800-v21-uploader: Processed model ChaiML/kimid-v8b-kimidv5a-lr5e6ep2r64g4b01-int4-mixed in 194.884s
chaiml-kimid-v8b-kimid-63800-v21-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v8b-kimid-63800-v21-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v8b-kimid-63800-v21-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/.gitattributes
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/chat_template.jinja
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/generation_config.json
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/added_tokens.json
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/README.md
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/special_tokens_map.json
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/quantization_config.json
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/merges.txt
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/vocab.json
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/tokenizer_config.json
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/config.json
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/tokenizer.json
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00027-of-00027.safetensors
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00023-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00026-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00012-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00015-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00007-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00018-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00003-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00013-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00016-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00010-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00022-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00017-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00006-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00020-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00009-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00001-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00011-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00025-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00004-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00005-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00008-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00021-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00019-of-00027.safetensors
chaiml-kimid-v8b-kimid-63800-v21-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v8b-kimid-63800-v21/model-00014-of-00027.safetensors
Job chaiml-kimid-v8b-kimid-63800-v21-uploader completed after 625.48s with status: succeeded
Stopping job with name chaiml-kimid-v8b-kimid-63800-v21-uploader
Pipeline stage VLLMUploader completed in 625.99s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v8b-kimid-63800-v21
Waiting for inference service chaiml-kimid-v8b-kimid-63800-v21 to be ready
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-kimid-v8b-kimid-63800-v21 ready after 503.5118293762207s
Pipeline stage VLLMDeployer completed in 504.63s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.008976697921753s
Received healthy response to inference request in 2.083447217941284s
Received healthy response to inference request in 2.1081390380859375s
Received healthy response to inference request in 2.672551393508911s
Received healthy response to inference request in 2.140209913253784s
Received healthy response to inference request in 2.5048599243164062s
Received healthy response to inference request in 2.132399082183838s
Received healthy response to inference request in 2.6267592906951904s
Received healthy response to inference request in 1.9265995025634766s
Received healthy response to inference request in 2.1591217517852783s
Received healthy response to inference request in 2.285539150238037s
Received healthy response to inference request in 1.9559473991394043s
Received healthy response to inference request in 2.3483643531799316s
Received healthy response to inference request in 2.247300386428833s
Received healthy response to inference request in 2.2481746673583984s
Received healthy response to inference request in 2.031101942062378s
Received healthy response to inference request in 2.4162817001342773s
Received healthy response to inference request in 1.9541563987731934s
Received healthy response to inference request in 1.9333531856536865s
Received healthy response to inference request in 2.2019646167755127s
Received healthy response to inference request in 1.9386050701141357s
Received healthy response to inference request in 1.996016025543213s
Received healthy response to inference request in 2.0426619052886963s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.1251842975616455s
Received healthy response to inference request in 1.9628887176513672s
Received healthy response to inference request in 2.0253794193267822s
Received healthy response to inference request in 2.1183273792266846s
Received healthy response to inference request in 2.0779733657836914s
Received healthy response to inference request in 1.9290204048156738s
Received healthy response to inference request in 2.3464157581329346s
30 requests
0 failed requests
5th percentile: 1.9309701561927795
10th percentile: 1.9380798816680909
20th percentile: 1.9615004539489747
30th percentile: 2.0204586029052733
40th percentile: 2.0638487815856936
50th percentile: 2.113233208656311
60th percentile: 2.1355234146118165
70th percentile: 2.2155653476715087
80th percentile: 2.2977144718170166
90th percentile: 2.4251395225524903
95th percentile: 2.571904575824737
99th percentile: 2.659271683692932
mean time: 2.151590665181478
Pipeline stage StressChecker completed in 67.88s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.67s
Shutdown handler de-registered
chaiml-kimid-v8b-kimid_63800_v21 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
chaiml-kimid-v8b-kimid_63800_v21 status is now inactive due to auto deactivation removed underperforming models