Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-83165-v14-uploader
Waiting for job on chaiml-kimid-v9-opusdv-83165-v14-uploader to finish
chaiml-kimid-v9-opusdv-83165-v14-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v9-opusdv-83165-v14-uploader: bash: no job control in this shell
chaiml-kimid-v9-opusdv-83165-v14-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v9-opusdv-83165-v14-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v9-opusdv-83165-v14-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ https://mk1.ai ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ belonging to: ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v14-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v9-opusdv-83165-v14-uploader: Downloaded to shared memory in 140.479s
chaiml-kimid-v9-opusdv-83165-v14-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-int4-mixed in 196.745s
chaiml-kimid-v9-opusdv-83165-v14-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-83165-v14-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-83165-v14-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/added_tokens.json
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/README.md
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/chat_template.jinja
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/tokenizer_config.json
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/special_tokens_map.json
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/quantization_config.json
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/.gitattributes
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/vocab.json
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/tokenizer.json
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/generation_config.json
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model.safetensors.index.json
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/merges.txt
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00027-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00011-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00001-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00016-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00006-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00026-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00025-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00021-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00004-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00007-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00012-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00013-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00015-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00017-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00024-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00010-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00022-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v14-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v14/model-00019-of-00027.safetensors
Job chaiml-kimid-v9-opusdv-83165-v14-uploader completed after 616.05s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-83165-v14-uploader
Pipeline stage VLLMUploader completed in 616.69s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-83165-v14
Waiting for inference service chaiml-kimid-v9-opusdv-83165-v14 to be ready
Inference service chaiml-kimid-v9-opusdv-83165-v14 ready after 483.44777059555054s
Pipeline stage VLLMDeployer completed in 484.07s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.300020694732666s
Received healthy response to inference request in 2.1628856658935547s
Received healthy response to inference request in 2.4196577072143555s
Received healthy response to inference request in 1.9998188018798828s
Received healthy response to inference request in 2.047323226928711s
Received healthy response to inference request in 2.618746280670166s
Received healthy response to inference request in 1.972116470336914s
Retrying (%r) after connection broken by '%r': %s
Received healthy response to inference request in 2.2277350425720215s
Received healthy response to inference request in 2.3600821495056152s
{"detail":"HTTPConnectionPool(host='chaiml-kimid-v9-opusdv-83165-v14-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Max retries exceeded with url: /v1/completions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x77f54e7e60d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))"}
Received unhealthy response to inference request!
Received healthy response to inference request in 2.54753041267395s
Received healthy response to inference request in 1.886629581451416s
Received healthy response to inference request in 2.2489843368530273s
Received healthy response to inference request in 1.8891842365264893s
Received healthy response to inference request in 1.9131958484649658s
Received healthy response to inference request in 2.119241714477539s
Received healthy response to inference request in 2.081878662109375s
Received healthy response to inference request in 2.1939289569854736s
Received healthy response to inference request in 1.9325284957885742s
Received healthy response to inference request in 2.020509958267212s
Received healthy response to inference request in 2.2500345706939697s
Received healthy response to inference request in 2.375450849533081s
Received healthy response to inference request in 2.4355738162994385s
Received healthy response to inference request in 2.0314571857452393s
Received healthy response to inference request in 2.1851608753204346s
Received healthy response to inference request in 2.3677282333374023s
Received healthy response to inference request in 2.1283349990844727s
Received healthy response to inference request in 2.058851957321167s
Received healthy response to inference request in 2.1339640617370605s
Received healthy response to inference request in 2.000969886779785s
30 requests
1 failed requests
5th percentile: 1.887779176235199
10th percentile: 1.9107946872711181
20th percentile: 1.994278335571289
30th percentile: 2.028173017501831
40th percentile: 2.0726679801940917
50th percentile: 2.1311495304107666
60th percentile: 2.1886681079864503
70th percentile: 2.24929940700531
80th percentile: 2.3616113662719727
90th percentile: 2.4212493181228636
95th percentile: 2.4971499443054195
99th percentile: 2.5980936789512636
mean time: 2.102643855412801
%s, retrying in %s seconds...
Received healthy response to inference request in 1.9816792011260986s
Received healthy response to inference request in 2.2959275245666504s
Received healthy response to inference request in 2.029853582382202s
Received healthy response to inference request in 2.23750901222229s
Received healthy response to inference request in 2.153313398361206s
Received healthy response to inference request in 1.9690630435943604s
Received healthy response to inference request in 2.261641025543213s
Received healthy response to inference request in 2.0570695400238037s
Received healthy response to inference request in 2.118555784225464s
Received healthy response to inference request in 1.9320530891418457s
Received healthy response to inference request in 2.1699023246765137s
Received healthy response to inference request in 2.2963225841522217s
Received healthy response to inference request in 1.9888076782226562s
Received healthy response to inference request in 2.4766571521759033s
Received healthy response to inference request in 1.9738192558288574s
Received healthy response to inference request in 2.2147774696350098s
Received healthy response to inference request in 1.9728529453277588s
Received healthy response to inference request in 2.074040412902832s
Received healthy response to inference request in 2.0697431564331055s
Received healthy response to inference request in 2.062497854232788s
Received healthy response to inference request in 2.106858730316162s
Received healthy response to inference request in 1.9952702522277832s
Received healthy response to inference request in 1.922095775604248s
Received healthy response to inference request in 2.635096311569214s
Received healthy response to inference request in 1.8943171501159668s
Received healthy response to inference request in 2.220737934112549s
Received healthy response to inference request in 2.297045946121216s
Received healthy response to inference request in 2.424701690673828s
Received healthy response to inference request in 2.085299253463745s
Received healthy response to inference request in 1.9904015064239502s
30 requests
0 failed requests
5th percentile: 1.926576566696167
10th percentile: 1.965362048149109
20th percentile: 1.9801072120666503
30th percentile: 1.9938096284866333
40th percentile: 2.0603265285491945
50th percentile: 2.0796698331832886
60th percentile: 2.1324588298797607
70th percentile: 2.2165656089782715
80th percentile: 2.2684983253479003
90th percentile: 2.3098115205764773
95th percentile: 2.4532771944999694
99th percentile: 2.589148955345154
mean time: 2.1302636861801147
Pipeline stage StressChecker completed in 133.08s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.71s
Shutdown handler de-registered
chaiml-kimid-v9-opusdv_83165_v14 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
chaiml-kimid-v9-opusdv_83165_v14 status is now inactive due to auto deactivation removed underperforming models