chaiml-grpo-glm50-chai-_10243

developer_uid: chai_backend_admin
submission_id: chaiml-grpo-glm50-chai-_10243_v1
model_name: chaiml-grpo-glm50-chai-_10243_v1
model_group: ChaiML/grpo-glm50-chai-r
status: inactive
timestamp: 2026-03-26T03:54:05+00:00
num_battles: 17292
num_wins: 10221
celo_rating: 1359.54
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/grpo-glm50-chai-rm-kimi-data-step280-int4-fast
model_architecture: GlmMoeDsaForCausalLM
model_num_parameters: 31348144128.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-grpo-glm50-chai-_10243_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/grpo-glm50-chai-rm-kimi-data-step280-int4-fast
model_size: 31B
ranking_group: single
us_pacific_date: 2026-03-25
win_ratio: 0.5910825815405968
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '####', '<|assistant|>', '<|im_end|>', '</s>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-glm50-chai-10243-v1-uploader
Waiting for job on chaiml-grpo-glm50-chai-10243-v1-uploader to finish
chaiml-grpo-glm50-chai-10243-v1-uploader: Using quantization_mode: none
chaiml-grpo-glm50-chai-10243-v1-uploader: Downloading snapshot of ChaiML/grpo-glm50-chai-rm-kimi-data-step280-int4-fast...
2026-03-25T21:27:15.982195+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
2026-03-25T21:28:16.070332+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
chaiml-grpo-glm50-chai-10243-v1-uploader: Downloaded in 114.828s
2026-03-25T21:29:16.167086+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
2026-03-25T21:30:16.251474+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
2026-03-25T21:31:16.426050+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
Failed to get request counts for guanaco-submitter. Falling back to default
Failed to get response for submission chaiml-merged-qwen-35-d_39140_v2: HTTPConnectionPool(host='chaiml-merged-qwen-35-d-39140-v2-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
2026-03-25T21:32:16.516855+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
2026-03-25T21:33:16.629754+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
2026-03-25T21:34:16.723570+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
chaiml-grpo-glm50-chai-10243-v1-uploader: Processed model ChaiML/grpo-glm50-chai-rm-kimi-data-step280-int4-fast in 466.783s
chaiml-grpo-glm50-chai-10243-v1-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/config.json
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/.gitattributes
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/generation_config.json
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/chat_template.jinja
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/quantization_config.json
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/tokenizer_config.json
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model.safetensors.index.json
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/tokenizer.json
2026-03-25T21:35:16.855798+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00077-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00077-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00078-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00078-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00015-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00015-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00020-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00020-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00049-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00049-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00003-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00003-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00028-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00028-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00067-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00067-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00062-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00062-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00008-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00008-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00055-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00055-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00059-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00059-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00023-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00023-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00043-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00043-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00009-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00009-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00075-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00075-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00026-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00026-of-00078.safetensors
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00013-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00013-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00046-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00046-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00004-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00004-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00021-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00021-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00073-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00073-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00027-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00027-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00069-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00069-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00025-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00025-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00061-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00061-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00002-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00002-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00038-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00038-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00047-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00047-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00042-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00042-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00036-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00036-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00041-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00041-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00057-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00057-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00031-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00031-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00001-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00001-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00019-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00019-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00034-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00034-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00012-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00012-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00016-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00016-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00005-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00005-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00058-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00058-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00070-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00070-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00052-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00052-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00045-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00045-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00044-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00044-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00022-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00022-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00006-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00006-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00072-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00072-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00018-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00018-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00060-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00060-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00017-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00017-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00011-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00011-of-00078.safetensors
2026-03-25T21:36:16.956570+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00053-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00053-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00039-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00039-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00054-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00054-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00048-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00048-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00033-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00033-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00024-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00024-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00071-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00071-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00037-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00037-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00074-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00074-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00063-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00063-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00035-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00035-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00030-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00030-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00068-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00068-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00050-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00050-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00064-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00064-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00051-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00051-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00076-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00076-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00014-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00014-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00056-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00056-of-00078.safetensors
chaiml-grpo-glm50-chai-10243-v1-uploader: cp /dev/shm/model_output/model-00065-of-00078.safetensors s3://guanaco-vllm-models/chaiml-grpo-glm50-chai-10243-v1/default/model-00065-of-00078.safetensors
Job chaiml-grpo-glm50-chai-10243-v1-uploader completed after 621.69s with status: succeeded
Stopping job with name chaiml-grpo-glm50-chai-10243-v1-uploader
Pipeline stage VLLMUploader completed in 622.33s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 1.61s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-glm50-chai-10243-v1
Waiting for inference service chaiml-grpo-glm50-chai-10243-v1 to be ready
2026-03-25T21:37:17.061005+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
2026-03-25T21:38:17.201917+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
Failed to get response for submission chaiml-qwen-bobo-19k-re_33926_v2: HTTPConnectionPool(host='chaiml-qwen-bobo-19k-re-33926-v2-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=20.0)
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
2026-03-25T21:39:17.320010+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
2026-03-25T21:40:17.445645+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
2026-03-25T21:41:28.188350+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
2026-03-25T21:42:28.291927+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
Inference service chaiml-grpo-glm50-chai-10243-v1 ready after 393.48516154289246s
Pipeline stage VLLMDeployer completed in 394.00s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 10.643840789794922s
2026-03-25T21:43:28.394363+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
Received healthy response to inference request in 11.034342050552368s
Received healthy response to inference request in 11.077829837799072s
Received healthy response to inference request in 2.6289710998535156s
Received healthy response to inference request in 2.6884231567382812s
Received healthy response to inference request in 2.703131914138794s
Received healthy response to inference request in 2.5808966159820557s
Received healthy response to inference request in 2.612934112548828s
Received healthy response to inference request in 2.5980734825134277s
Received healthy response to inference request in 2.749044418334961s
Received healthy response to inference request in 3.0382988452911377s
Received healthy response to inference request in 3.248539686203003s
Received healthy response to inference request in 2.636137008666992s
Received healthy response to inference request in 2.582385301589966s
Received healthy response to inference request in 2.711843967437744s
Received healthy response to inference request in 2.6551225185394287s
Received healthy response to inference request in 2.5472412109375s
2026-03-25T21:44:28.548972+00:00 monitor updated for chaiml-grpo-glm50-chai-_10243_v1
Received healthy response to inference request in 2.7367520332336426s
Failed to get response for submission chaiml-merged-qwen-35-d_39140_v2: ('http://chaiml-merged-qwen-35-d-39140-v2-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/completions', 'request timeout')
Received healthy response to inference request in 2.7219669818878174s
Received healthy response to inference request in 2.5564498901367188s
Received healthy response to inference request in 2.650127410888672s
Received healthy response to inference request in 2.4160046577453613s
Received healthy response to inference request in 2.7023565769195557s
Received healthy response to inference request in 2.454477310180664s
Received healthy response to inference request in 2.526012420654297s
Received healthy response to inference request in 2.543605089187622s
Received healthy response to inference request in 2.6701276302337646s
Received healthy response to inference request in 2.653820753097534s
Received healthy response to inference request in 2.6934101581573486s
Received healthy response to inference request in 2.741542100906372s
30 requests
0 failed requests
5th percentile: 2.486668109893799
10th percentile: 2.5418458223342895
20th percentile: 2.576007270812988
30th percentile: 2.608475923538208
40th percentile: 2.64453125
50th percentile: 2.6626250743865967
60th percentile: 2.6969887256622314
70th percentile: 2.714880871772766
80th percentile: 2.74304256439209
90th percentile: 3.9880697965622054
95th percentile: 10.858616483211517
99th percentile: 11.065218379497528
mean time: 3.493456967671712
Pipeline stage StressChecker completed in 108.45s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.25s
Shutdown handler de-registered
chaiml-grpo-glm50-chai-_10243_v1 status is now deployed due to DeploymentManager action
chaiml-grpo-glm50-chai-_10243_v1 status is now inactive due to system request