developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm50-combi_11380_v2
model_name: chaiml-grpo-glm50-combi_11380_v2
model_group: ChaiML/grpo-glm50-combin
status: torndown
timestamp: 2026-03-29T02:17:16+00:00
num_battles: 8967
num_wins: 4672
celo_rating: 1319.46
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm50-combine-rm82-zy-data-step200-int4
model_architecture: GlmMoeDsaForCausalLM
model_num_parameters: 31348144128.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm50-combi_11380_v2
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm50-combine-rm82-zy-data-step200-int4
model_size: 31B
ranking_group: single
us_pacific_date: 2026-03-25
win_ratio: 0.5210215233634438
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-combi-11380-v2-uploader
Waiting for job on chaiml-grpo-glm50-combi-11380-v2-uploader to finish
chaiml-grpo-glm50-combi-11380-v2-uploader: Using quantization_mode: none
chaiml-grpo-glm50-combi-11380-v2-uploader: Downloading snapshot of ChaiML/grpo-glm50-combine-rm82-zy-data-step200-int4...
2026-03-25T23:02:30.077809+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
2026-03-25T23:03:30.182276+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
chaiml-grpo-glm50-combi-11380-v2-uploader: Downloaded in 116.725s
2026-03-25T23:04:30.308541+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
2026-03-25T23:05:30.396171+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
2026-03-25T23:06:30.484727+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
2026-03-25T23:07:30.573040+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
Failed to get response for submission chaiml-grpo-glm50-chai-_10243_v1: ('http://chaiml-grpo-glm50-chai-10243-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
2026-03-25T23:08:30.685371+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
chaiml-grpo-glm50-combi-11380-v2-uploader: Processed model ChaiML/grpo-glm50-combine-rm82-zy-data-step200-int4 in 430.150s
chaiml-grpo-glm50-combi-11380-v2-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-combi-11380-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-11380-v2-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-glm50-combi-11380-v2-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-glm50-combi-11380-v2-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-glm50-combi-11380-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-11380-v2-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-11380-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-11380-v2-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-glm50-combi-11380-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-11380-v2-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-11380-v2-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-glm50-combi-11380-v2-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-glm50-combi-11380-v2-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-glm50-combi-11380-v2-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-glm50-combi-11380-v2-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-glm50-combi-11380-v2-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-glm50-combi-11380-v2-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-glm50-combi-11380-v2-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/chat_template.jinja
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/generation_config.json
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/.gitattributes
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/config.json
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/quantization_config.json
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/tokenizer_config.json
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model.safetensors.index.json
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/tokenizer.json
2026-03-25T23:09:30.794369+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00077-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00077-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00078-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00078-of-00078.safetensors
2026-03-25T23:10:30.925323+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00059-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00059-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00005-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00005-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00018-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00018-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00051-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00051-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00037-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00037-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00054-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00054-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00035-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00035-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00060-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00060-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00038-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00038-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00017-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00017-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00032-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00032-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00052-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00052-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00012-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00012-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00072-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00072-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00010-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00010-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00071-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00071-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00046-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00046-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00053-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00053-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00030-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00030-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00006-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00006-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00047-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00047-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00068-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00068-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00056-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00056-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00024-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00024-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00049-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00049-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00041-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00041-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00048-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00048-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00058-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00058-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00039-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00039-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00028-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00028-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00008-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00008-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00075-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00075-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00021-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00021-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00061-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00061-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00057-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00057-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00009-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00009-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00055-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00055-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00033-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00033-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00034-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00034-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00027-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00027-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00043-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00043-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00040-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00040-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00002-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00002-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00022-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00022-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00063-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00063-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00025-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00025-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00073-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00073-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00016-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00016-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00031-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00031-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00019-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00019-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00026-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00026-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00064-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00064-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00001-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00001-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00015-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00015-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00020-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00020-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00036-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00036-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00066-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00066-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00065-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00065-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00013-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00013-of-00078.safetensors
chaiml-grpo-glm50-combi-11380-v2-uploader: cp /dev/shm/model_output/model-00074-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-combi-11380-v2/default/model-00074-of-00078.safetensors
Job chaiml-grpo-glm50-combi-11380-v2-uploader completed after 591.38s with status: succeeded
Stopping job with name chaiml-grpo-glm50-combi-11380-v2-uploader
Pipeline stage VLLMUploader completed in 592.14s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 2.56s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm50-combi-11380-v2
Waiting for inference service chaiml-grpo-glm50-combi-11380-v2 to be ready
2026-03-25T23:11:31.048160+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
Failed to get response for submission blend_jenib_2026-03-25: ('http://chaiml-grpo-glm50-combi-46874-v4-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
2026-03-25T23:12:31.174719+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
Failed to get response for submission chaiml-grpo-glm50-chai-_10243_v1: ('http://chaiml-grpo-glm50-chai-10243-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
2026-03-25T23:13:31.314420+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
2026-03-25T23:14:31.847299+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
2026-03-25T23:15:32.016831+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
2026-03-25T23:16:32.184024+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
2026-03-25T23:17:32.361498+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
2026-03-25T23:18:32.494639+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
2026-03-25T23:19:32.702098+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-25T23:20:32.803308+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
2026-03-25T23:21:32.911863+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
2026-03-25T23:22:33.017764+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
Inference service chaiml-grpo-glm50-combi-11380-v2 ready after 693.826423406601s
Pipeline stage VLLMDeployer completed in 694.32s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 10.113731384277344s
Received healthy response to inference request in 1.6275608539581299s
Received healthy response to inference request in 1.8512914180755615s
Received healthy response to inference request in 1.5857946872711182s
Received healthy response to inference request in 1.9733686447143555s
Received healthy response to inference request in 10.037160873413086s
Received healthy response to inference request in 1.573432207107544s
Received healthy response to inference request in 1.735337257385254s
Received healthy response to inference request in 1.8014018535614014s
2026-03-25T23:23:33.146400+00:00 monitor updated for chaiml-grpo-glm50-combi_11380_v2
Received healthy response to inference request in 1.746598720550537s
Received healthy response to inference request in 1.8472177982330322s
Received healthy response to inference request in 1.7265243530273438s
Received healthy response to inference request in 1.588794469833374s
Received healthy response to inference request in 1.9372284412384033s
Received healthy response to inference request in 1.6341750621795654s
Received healthy response to inference request in 1.9404258728027344s
Received healthy response to inference request in 1.8810760974884033s
Received healthy response to inference request in 2.142033100128174s
Received healthy response to inference request in 1.5255281925201416s
Received healthy response to inference request in 1.9250555038452148s
Received healthy response to inference request in 1.7306444644927979s
Received healthy response to inference request in 1.5644803047180176s
Received healthy response to inference request in 1.529341697692871s
Received healthy response to inference request in 2.0860788822174072s
Received healthy response to inference request in 1.5983693599700928s
Received healthy response to inference request in 1.5318114757537842s
Received healthy response to inference request in 1.6444950103759766s
Received healthy response to inference request in 1.55324387550354s
Received healthy response to inference request in 1.6516468524932861s
Received healthy response to inference request in 1.544262170791626s
30 requests
0 failed requests
5th percentile: 1.530453097820282
10th percentile: 1.5430171012878418
20th percentile: 1.5716418266296386
30th percentile: 1.5954968929290771
40th percentile: 1.640367031097412
50th percentile: 1.7285844087600708
60th percentile: 1.7685199737548827
70th percentile: 1.860226821899414
80th percentile: 1.9378679275512696
90th percentile: 2.091674304008484
95th percentile: 6.4843533754348535
99th percentile: 10.09152593612671
mean time: 2.2876036961873374
Pipeline stage StressChecker completed in 71.73s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.47s
Shutdown handler de-registered
chaiml-grpo-glm50-combi_11380_v2 status is now deployed due to DeploymentManager action
chaiml-grpo-glm50-combi_11380_v2 status is now inactive due to system request
chaiml-grpo-glm50-combi_11380_v2 status is now torndown due to DeploymentManager action