Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-q235b-kimid-v3-j-86095-v3-uploader
Waiting for job on chaiml-q235b-kimid-v3-j-86095-v3-uploader to finish
chaiml-q235b-kimid-v3-j-86095-v3-uploader: Using quantization_mode: w4a16
chaiml-q235b-kimid-v3-j-86095-v3-uploader: Checking if ChaiML/q235b_kimid_v3_judging-step403-merged-W4A16 already exists in ChaiML
chaiml-q235b-kimid-v3-j-86095-v3-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-q235b-kimid-v3-j-86095-v3-uploader: Downloading snapshot of ChaiML/q235b_kimid_v3_judging-step403-merged-W4A16...
2026-04-02T22:17:36.437553+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
chaiml-q235b-kimid-v3-j-86095-v3-uploader: Downloaded in 52.566s
chaiml-q235b-kimid-v3-j-86095-v3-uploader: Processed model ChaiML/q235b_kimid_v3_judging-step403-merged in 55.240s
chaiml-q235b-kimid-v3-j-86095-v3-uploader: creating bucket guanaco-vllm-models
chaiml-q235b-kimid-v3-j-86095-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-kimid-v3-j-86095-v3-uploader: RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-q235b-kimid-v3-j-86095-v3-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-q235b-kimid-v3-j-86095-v3-uploader: RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-q235b-kimid-v3-j-86095-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-kimid-v3-j-86095-v3-uploader: invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-q235b-kimid-v3-j-86095-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-kimid-v3-j-86095-v3-uploader: invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-q235b-kimid-v3-j-86095-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-kimid-v3-j-86095-v3-uploader: if re.search("-\.", bucket, re.UNICODE):
chaiml-q235b-kimid-v3-j-86095-v3-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-q235b-kimid-v3-j-86095-v3-uploader: if re.search("\.\.", bucket, re.UNICODE):
chaiml-q235b-kimid-v3-j-86095-v3-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-q235b-kimid-v3-j-86095-v3-uploader: _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-q235b-kimid-v3-j-86095-v3-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-q235b-kimid-v3-j-86095-v3-uploader: wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-q235b-kimid-v3-j-86095-v3-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-q235b-kimid-v3-j-86095-v3-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/generation_config.json
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/tokenizer_config.json
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/.gitattributes
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/chat_template.jinja
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/config.json
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/quantization_config.json
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/tokenizer.json
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model.safetensors.index.json
2026-04-02T22:18:36.532017+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00025-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00025-of-00025.safetensors
2026-04-02T22:19:36.626604+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00006-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00006-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00015-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00015-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00012-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00012-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00021-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00021-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00001-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00001-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00009-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00009-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00022-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00022-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00002-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00002-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00014-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00014-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00008-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00008-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00020-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00020-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00018-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00018-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00005-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00005-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00019-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00019-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00004-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00004-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00003-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00003-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00023-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00023-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00017-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00017-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00007-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00007-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00011-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00011-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00016-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00016-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00024-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00024-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00010-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00010-of-00025.safetensors
chaiml-q235b-kimid-v3-j-86095-v3-uploader: cp /dev/shm/model_output/model-00013-of-00025.safetensors s3://guanaco-vllm-models/chaiml-q235b-kimid-v3-j-86095-v3/default/model-00013-of-00025.safetensors
Job chaiml-q235b-kimid-v3-j-86095-v3-uploader completed after 225.41s with status: succeeded
Stopping job with name chaiml-q235b-kimid-v3-j-86095-v3-uploader
Pipeline stage VLLMUploader completed in 232.14s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.11s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.27s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-q235b-kimid-v3-j-86095-v3
Waiting for inference service chaiml-q235b-kimid-v3-j-86095-v3 to be ready
2026-04-02T22:20:36.741960+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:21:36.839281+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:22:36.931052+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:23:37.043815+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:24:37.159441+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:25:37.297986+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:26:37.401986+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:27:37.495268+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:28:37.595831+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:29:37.696495+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:30:37.822558+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:31:38.018263+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:32:38.159606+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:33:38.607415+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:34:38.815426+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:35:39.030232+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
2026-04-02T22:36:39.169566+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
Inference service chaiml-q235b-kimid-v3-j-86095-v3 ready after 1013.5169882774353s
Pipeline stage VLLMDeployer completed in 1014.07s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.5084574222564697s
Received healthy response to inference request in 1.5464789867401123s
Received healthy response to inference request in 1.478945255279541s
Received healthy response to inference request in 1.4420466423034668s
Received healthy response to inference request in 1.4594311714172363s
Received healthy response to inference request in 1.5838818550109863s
Received healthy response to inference request in 1.5366528034210205s
2026-04-02T22:37:39.306274+00:00 monitor updated for chaiml-q235b-kimid-v3-j_86095_v3
Received healthy response to inference request in 1.696540117263794s
Received healthy response to inference request in 1.611680507659912s
Received healthy response to inference request in 1.4900789260864258s
Received healthy response to inference request in 1.4845678806304932s
Received healthy response to inference request in 1.5294857025146484s
Received healthy response to inference request in 1.636608362197876s
Received healthy response to inference request in 1.5672025680541992s
Received healthy response to inference request in 1.5302400588989258s
Received healthy response to inference request in 1.518315315246582s
Received healthy response to inference request in 1.5011680126190186s
Received healthy response to inference request in 1.461083173751831s
Received healthy response to inference request in 1.468144178390503s
Received healthy response to inference request in 1.6934947967529297s
Received healthy response to inference request in 1.5342497825622559s
Received healthy response to inference request in 1.4439754486083984s
Received healthy response to inference request in 1.7384021282196045s
Received healthy response to inference request in 1.574005126953125s
Received healthy response to inference request in 1.4675922393798828s
Received healthy response to inference request in 1.4630963802337646s
Received healthy response to inference request in 1.4635460376739502s
Received healthy response to inference request in 1.4839515686035156s
Received healthy response to inference request in 1.4937002658843994s
Received healthy response to inference request in 1.467787742614746s
30 requests
0 failed requests
5th percentile: 1.4509305238723755
10th percentile: 1.4609179735183715
20th percentile: 1.4667829990386962
30th percentile: 1.4757049322128295
40th percentile: 1.4878745079040527
50th percentile: 1.5097416639328003
60th percentile: 1.5318439483642579
70th percentile: 1.5526960611343383
80th percentile: 1.5894415855407715
90th percentile: 1.693799328804016
95th percentile: 1.7195642232894897
99th percentile: 2.9951413869857806
mean time: 1.5958270152409872
Pipeline stage StressChecker completed in 50.98s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.19s
Shutdown handler de-registered
chaiml-q235b-kimid-v3-j_86095_v3 status is now deployed due to DeploymentManager action
chaiml-q235b-kimid-v3-j_86095_v3 status is now inactive due to auto deactivation removed underperforming models
chaiml-q235b-kimid-v3-j_86095_v3 status is now torndown due to DeploymentManager action