Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-bol-v3-opusdv1b-38416-v2-uploader
Waiting for job on chaiml-bol-v3-opusdv1b-38416-v2-uploader to finish
chaiml-bol-v3-opusdv1b-38416-v2-uploader: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-bol-v3-opusdv1b-38416-v2-uploader: bash: no job control in this shell
chaiml-bol-v3-opusdv1b-38416-v2-uploader: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-bol-v3-opusdv1b-38416-v2-uploader: __import__('pkg_resources').declare_namespace(__name__)
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ ██████ ██████ █████ ████ ████ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ ░░██████ ██████ ░░███ ███░ ░░███ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ ░███░█████░███ ░███ ███ ░███ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ ░███░░███ ░███ ░███████ ░███ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ ░███ ░░░ ░███ ░███░░███ ░███ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ ░███ ░███ ░███ ░░███ ░███ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ █████ █████ █████ ░░████ █████ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ ░░░░░ ░░░░░ ░░░░░ ░░░░ ░░░░░ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ Version: 0.30.6+torch280 ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ Features: FLYWHEEL, CUDA ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ Copyright 2023-2025 MK ONE TECHNOLOGIES Inc. ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ https://mk1.ai ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ The license key for the current software has been verified as ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ belonging to: ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ Chai Research Corp. ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ Expiration: 2028-03-31 23:59:59 ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ║ ║
chaiml-bol-v3-opusdv1b-38416-v2-uploader: ╚═════════════════════════════════════════════════════════════════════╝
HTTP Request: %s %s "%s %d %s"
chaiml-bol-v3-opusdv1b-38416-v2-uploader: Downloaded to shared memory in 102.825s
chaiml-bol-v3-opusdv1b-38416-v2-uploader: Processed model ChaiML/bol-v3-opusdv1b-lr1e5ep2r64g4b01-int4-mixed in 174.839s
chaiml-bol-v3-opusdv1b-38416-v2-uploader: creating bucket guanaco-vllm-models
chaiml-bol-v3-opusdv1b-38416-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-bol-v3-opusdv1b-38416-v2-uploader: uploading /dev/shm/model_cache to s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/added_tokens.json
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/.gitattributes s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/.gitattributes
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/generation_config.json s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/generation_config.json
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/chat_template.jinja s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/chat_template.jinja
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/tokenizer_config.json
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/special_tokens_map.json
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/config.json s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/config.json
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/quantization_config.json s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/quantization_config.json
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/vocab.json s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/vocab.json
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/merges.txt s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/merges.txt
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/tokenizer.json
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model.safetensors.index.json
HTTP Request: %s %s "%s %d %s"
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00027-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00008-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00018-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00016-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00021-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00024-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00023-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00025-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00004-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00001-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00012-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00011-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00022-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00002-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00010-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00007-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00020-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00017-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00009-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00019-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00026-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00005-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00015-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00014-of-00027.safetensors
chaiml-bol-v3-opusdv1b-38416-v2-uploader: cp /dev/shm/model_cache/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-bol-v3-opusdv1b-38416-v2/model-00003-of-00027.safetensors
Job chaiml-bol-v3-opusdv1b-38416-v2-uploader completed after 710.62s with status: succeeded
Stopping job with name chaiml-bol-v3-opusdv1b-38416-v2-uploader
Pipeline stage VLLMUploader completed in 710.98s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-bol-v3-opusdv1b-38416-v2
Waiting for inference service chaiml-bol-v3-opusdv1b-38416-v2 to be ready
HTTP Request: %s %s "%s %d %s"
Inference service chaiml-bol-v3-opusdv1b-38416-v2 ready after 512.1431150436401s
Pipeline stage VLLMDeployer completed in 512.61s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.9467568397521973s
Received healthy response to inference request in 2.1718194484710693s
Received healthy response to inference request in 2.034548044204712s
Received healthy response to inference request in 1.7477223873138428s
Received healthy response to inference request in 1.9019029140472412s
Received healthy response to inference request in 1.8150031566619873s
Received healthy response to inference request in 1.8221497535705566s
Received healthy response to inference request in 1.7722864151000977s
Received healthy response to inference request in 1.7755258083343506s
Received healthy response to inference request in 1.9445910453796387s
Received healthy response to inference request in 2.031362771987915s
Received healthy response to inference request in 1.7752151489257812s
Received healthy response to inference request in 2.047088384628296s
Received healthy response to inference request in 1.9119501113891602s
Received healthy response to inference request in 1.7509112358093262s
Received healthy response to inference request in 1.7631566524505615s
Received healthy response to inference request in 2.002084255218506s
Received healthy response to inference request in 1.8414638042449951s
Received healthy response to inference request in 1.8204963207244873s
Received healthy response to inference request in 2.1914260387420654s
Received healthy response to inference request in 1.7714567184448242s
Received healthy response to inference request in 1.7339744567871094s
Received healthy response to inference request in 1.884796142578125s
Received healthy response to inference request in 1.7763745784759521s
Received healthy response to inference request in 1.816282033920288s
Received healthy response to inference request in 2.1372568607330322s
Received healthy response to inference request in 1.9062836170196533s
Received healthy response to inference request in 2.3549323081970215s
Received healthy response to inference request in 2.1553170680999756s
Received healthy response to inference request in 1.783412218093872s
30 requests
0 failed requests
5th percentile: 1.7491573691368103
10th percentile: 1.761932110786438
20th percentile: 1.7746294021606446
30th percentile: 1.781300926208496
40th percentile: 1.8188106060028075
50th percentile: 1.86312997341156
60th percentile: 1.908550214767456
70th percentile: 1.9633550643920896
80th percentile: 2.0370561122894286
90th percentile: 2.156967306137085
95th percentile: 2.182603073120117
99th percentile: 2.3075154900550845
mean time: 1.912918217976888
Pipeline stage StressChecker completed in 60.82s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.62s
Shutdown handler de-registered
chaiml-bol-v3-opusdv1b-_38416_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
Generating Leaderboard row for %s
Generated Leaderboard row for %s
Pipeline stage OfflineFamilyFriendlyScorer completed in 4209.00s
Shutdown handler de-registered
chaiml-bol-v3-opusdv1b-_38416_v2 status is now torndown due to DeploymentManager action