Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Connection pool is full, discarding connection: %s. Connection pool size: %s
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-kimid-v9-opusdv-83165-v27-uploader
Waiting for job on chaiml-kimid-v9-opusdv-83165-v27-uploader to finish
Stopping job with name chaiml-kimid-v9-opusdv-83165-v27-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-kimid-v9-opusdv-83165-v27-uploader
Waiting for job on chaiml-kimid-v9-opusdv-83165-v27-uploader to finish
chaiml-kimid-v9-opusdv-83165-v27-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-kimid-v9-opusdv-83165-v27-uploader: bash: no job control in this shell
chaiml-kimid-v9-opusdv-83165-v27-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-kimid-v9-opusdv-83165-v27-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-kimid-v9-opusdv-83165-v27-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ https://mk1.ai ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ The license key for the current software has been verified as ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ belonging to: ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ Chai Research Corp. ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ║ ║
chaiml-kimid-v9-opusdv-83165-v27-uploader: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-kimid-v9-opusdv-83165-v27-uploader: Downloaded to shared memory in 122.005s
chaiml-kimid-v9-opusdv-83165-v27-uploader: Processed model ChaiML/kimid-v9-opusdv1-lr5e6ep2r64g4b01-int4-mixed in 181.618s
chaiml-kimid-v9-opusdv-83165-v27-uploader: creating bucket guanaco-vllm-models
chaiml-kimid-v9-opusdv-83165-v27-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-kimid-v9-opusdv-83165-v27-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/README.md s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/README.md
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/generation_config.json
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/added_tokens.json
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/.gitattributes
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/special_tokens_map.json
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/chat_template.jinja
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/merges.txt
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/config.json
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/tokenizer_config.json
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/vocab.json
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/quantization_config.json
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model.safetensors.index.json
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/tokenizer.json
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00027-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00019-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00024-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00012-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00001-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00006-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00023-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00016-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00018-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00005-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00007-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00009-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00004-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00022-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00025-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00021-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00013-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00008-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00002-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00017-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00020-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00003-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00011-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00014-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00010-of-00027.safetensors
chaiml-kimid-v9-opusdv-83165-v27-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-kimid-v9-opusdv-83165-v27/model-00015-of-00027.safetensors
Job chaiml-kimid-v9-opusdv-83165-v27-uploader completed after 555.02s with status: succeeded
Stopping job with name chaiml-kimid-v9-opusdv-83165-v27-uploader
Pipeline stage VLLMUploader completed in 556.22s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-kimid-v9-opusdv-83165-v27
Waiting for inference service chaiml-kimid-v9-opusdv-83165-v27 to be ready
Failed to get response for submission blend_beren_2026-01-30: ('http://chaiml-2a6f-69d4-linear-w01-v4-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:42996->127.0.0.1:8080: read: connection reset by peer\n')
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-kimid-v9-opusdv-83165-v27 ready after 520.3122749328613s
Pipeline stage VLLMDeployer completed in 520.74s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.6419382095336914s
Received healthy response to inference request in 2.212266206741333s
Received healthy response to inference request in 2.2164435386657715s
Received healthy response to inference request in 2.7752485275268555s
Received healthy response to inference request in 1.9629545211791992s
Received healthy response to inference request in 2.2123541831970215s
Received healthy response to inference request in 2.2417538166046143s
Received healthy response to inference request in 1.930206060409546s
Received healthy response to inference request in 1.9784886837005615s
Received healthy response to inference request in 2.2448854446411133s
Received healthy response to inference request in 2.1892027854919434s
Received healthy response to inference request in 2.1209042072296143s
Received healthy response to inference request in 2.2752203941345215s
Received healthy response to inference request in 1.964120864868164s
Received healthy response to inference request in 1.9995715618133545s
Received healthy response to inference request in 1.9626660346984863s
Received healthy response to inference request in 2.484325647354126s
Received healthy response to inference request in 1.9750370979309082s
Received healthy response to inference request in 2.79986310005188s
Received healthy response to inference request in 2.388806104660034s
Received healthy response to inference request in 2.0312581062316895s
Received healthy response to inference request in 2.2144792079925537s
Received healthy response to inference request in 1.9414989948272705s
Received healthy response to inference request in 1.9943513870239258s
Received healthy response to inference request in 2.011052131652832s
Received healthy response to inference request in 1.9468517303466797s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.437875986099243s
Received healthy response to inference request in 2.386744976043701s
Received healthy response to inference request in 2.001350164413452s
Received healthy response to inference request in 1.929687261581421s
30 requests
0 failed requests
5th percentile: 1.935287880897522
10th percentile: 1.9463164567947389
20th percentile: 1.963887596130371
30th percentile: 1.9895925760269164
40th percentile: 2.0071713447570803
50th percentile: 2.155053496360779
60th percentile: 2.213204193115234
70th percentile: 2.242693305015564
80th percentile: 2.3871572017669678
90th percentile: 2.500086903572083
95th percentile: 2.7152588844299315
99th percentile: 2.7927248740196227
mean time: 2.182380231221517
Pipeline stage StressChecker completed in 69.00s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.71s
Shutdown handler de-registered
chaiml-kimid-v9-opusdv_83165_v27 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
Generating Leaderboard row for %s
Generated Leaderboard row for %s
Pipeline stage OfflineFamilyFriendlyScorer completed in 4361.99s
Shutdown handler de-registered
chaiml-kimid-v9-opusdv_83165_v27 status is now torndown due to DeploymentManager action