chaiml-grpo-q235b-kimid_37540

developer_uid: acehao-chai
submission_id: chaiml-grpo-q235b-kimid_37540_v2
model_name: chaiml-grpo-q235b-kimid_37540_v2
model_group: ChaiML/grpo-q235b-kimid-
status: torndown
timestamp: 2026-02-22T01:01:52+00:00
num_battles: 11119
num_wins: 6003
celo_rating: 1327.95
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-300
model_architecture: Qwen3MoeForCausalLM
model_num_parameters: 18790207488.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-q235b-kimid_37540_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: False
language_model: ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-300
model_size: 19B
ranking_group: single
us_pacific_date: 2026-02-18
win_ratio: 0.5398866804568756
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|assistant|>', '<|im_end|>', '####', '<|user|>', '</think>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-q235b-kimid-37540-v2-uploader
Waiting for job on chaiml-grpo-q235b-kimid-37540-v2-uploader to finish
chaiml-grpo-q235b-kimid-37540-v2-uploader: Using quantization_mode: w4a16
chaiml-grpo-q235b-kimid-37540-v2-uploader: Checking if ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-300-W4A16 already exists in ChaiML
chaiml-grpo-q235b-kimid-37540-v2-uploader: Model already exists. Downloading to /dev/shm/model_output...
chaiml-grpo-q235b-kimid-37540-v2-uploader: Downloading snapshot of ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-300-W4A16...
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-grpo-q235b-kimid-37540-v2-uploader: Downloaded in 53.578s
chaiml-grpo-q235b-kimid-37540-v2-uploader: Processed model ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-300 in 54.131s
chaiml-grpo-q235b-kimid-37540-v2-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-q235b-kimid-37540-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-37540-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-q235b-kimid-37540-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-q235b-kimid-37540-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-q235b-kimid-37540-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-37540-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-q235b-kimid-37540-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-37540-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-q235b-kimid-37540-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-37540-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-q235b-kimid-37540-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q235b-kimid-37540-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-q235b-kimid-37540-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-q235b-kimid-37540-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-q235b-kimid-37540-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-q235b-kimid-37540-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-q235b-kimid-37540-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-q235b-kimid-37540-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/.gitattributes
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/tokenizer_config.json
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/chat_template.jinja
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/generation_config.json
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/config.json
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/added_tokens.json
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/special_tokens_map.json
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/quantization_config.json
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/merges.txt
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/tokenizer.json
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model.safetensors.index.json
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/vocab.json
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG retryable error: RequestError: send request failed
chaiml-grpo-q235b-kimid-37540-v2-uploader: caused by: Put "https://object.ord1.coreweave.com/guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00023-of-00027.safetensors?partNumber=7&uploadId=2~VRL7Wd51_7GndS5IBRKHncyDGlwF3IB": write tcp 10.0.23.131:59552->216.153.53.63:443: write: connection reset by peer
chaiml-grpo-q235b-kimid-37540-v2-uploader: ERROR "cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00023-of-00027.safetensors": MultipartUpload: upload multipart failed upload id: 2~VRL7Wd51_7GndS5IBRKHncyDGlwF3IB caused by: SignatureDoesNotMatch:  status code: 403, request id: tx000001b33f062f106832b-00699639dd-15698d1afd-default, host id:
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00027-of-00027.safetensors
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00013-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00024-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00025-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00005-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00001-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00026-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00006-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00008-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00011-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00020-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00007-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00016-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00004-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00014-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00022-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00009-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00002-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00017-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00012-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00015-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00019-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00010-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00003-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00018-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00021-of-00027.safetensors
chaiml-grpo-q235b-kimid-37540-v2-uploader: Retry 1/5 exited 1, retrying in 2 seconds...
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/.gitattributes": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/added_tokens.json": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/chat_template.jinja": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/config.json": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/generation_config.json": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/merges.txt": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00001-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00002-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00003-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00004-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00005-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00006-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00006-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00007-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00008-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00009-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00010-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00011-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00012-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00013-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00014-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00015-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00016-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00017-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00018-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00019-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00020-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00021-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00022-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00024-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00025-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00026-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00027-of-00027.safetensors": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model.safetensors.index.json": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/quantization_config.json": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/special_tokens_map.json": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/tokenizer.json": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/tokenizer_config.json": object size matches
chaiml-grpo-q235b-kimid-37540-v2-uploader: DEBUG "sync /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/vocab.json": object size matches
Retrying (%r) after connection broken by '%r': %s
chaiml-grpo-q235b-kimid-37540-v2-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-37540-v2/default/model-00023-of-00027.safetensors
Job chaiml-grpo-q235b-kimid-37540-v2-uploader completed after 333.59s with status: succeeded
Stopping job with name chaiml-grpo-q235b-kimid-37540-v2-uploader
Pipeline stage VLLMUploader completed in 334.21s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-q235b-kimid-37540-v2
Waiting for inference service chaiml-grpo-q235b-kimid-37540-v2 to be ready
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Inference service chaiml-grpo-q235b-kimid-37540-v2 ready after 884.977219581604s
Pipeline stage VLLMDeployer completed in 885.60s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.1209051609039307s
Received healthy response to inference request in 2.0315287113189697s
Received healthy response to inference request in 1.9604852199554443s
Received healthy response to inference request in 1.9997994899749756s
Received healthy response to inference request in 2.043409824371338s
Received healthy response to inference request in 2.0385541915893555s
Received healthy response to inference request in 1.9309725761413574s
Received healthy response to inference request in 2.117677927017212s
Received healthy response to inference request in 1.8538970947265625s
Received healthy response to inference request in 2.029266834259033s
Received healthy response to inference request in 2.3633604049682617s
Received healthy response to inference request in 2.038455009460449s
Received healthy response to inference request in 1.9626953601837158s
Received healthy response to inference request in 1.9180059432983398s
Received healthy response to inference request in 1.9634909629821777s
Received healthy response to inference request in 1.897362470626831s
Received healthy response to inference request in 2.053513288497925s
Received healthy response to inference request in 1.9639348983764648s
Received healthy response to inference request in 2.145308494567871s
Received healthy response to inference request in 2.4218943119049072s
Received healthy response to inference request in 2.1681740283966064s
Received healthy response to inference request in 2.2182059288024902s
Received healthy response to inference request in 1.9910118579864502s
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Received healthy response to inference request in 2.065615653991699s
Received healthy response to inference request in 2.267490863800049s
Received healthy response to inference request in 2.293494701385498s
Received healthy response to inference request in 2.471073627471924s
Received healthy response to inference request in 2.2027671337127686s
Received healthy response to inference request in 2.079540967941284s
Received healthy response to inference request in 1.9924037456512451s
30 requests
0 failed requests
5th percentile: 1.90665203332901
10th percentile: 1.9296759128570558
20th percentile: 1.9633318424224853
30th percentile: 1.9919861793518066
40th percentile: 2.030623960494995
50th percentile: 2.0409820079803467
60th percentile: 2.0711857795715334
70th percentile: 2.128226161003113
80th percentile: 2.205854892730713
90th percentile: 2.3004812717437746
95th percentile: 2.3955540537834166
99th percentile: 2.456811625957489
mean time: 2.0868098894755045
Pipeline stage StressChecker completed in 66.44s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.68s
Shutdown handler de-registered
chaiml-grpo-q235b-kimid_37540_v2 status is now deployed due to DeploymentManager action
chaiml-grpo-q235b-kimid_37540_v2 status is now inactive due to auto deactivation removed underperforming models
chaiml-grpo-q235b-kimid_37540_v2 status is now torndown due to DeploymentManager action