chaiml-grpo-q3b-merged-_37630

developer_uid: acehao-chai

submission_id: chaiml-grpo-q3b-merged-_37630_v1

model_name: chaiml-grpo-q3b-merged-_37630_v1

model_group: ChaiML/grpo-q3b-merged-n

status: torndown

timestamp: 2026-02-22T22:21:54+00:00

num_battles: 10573

num_wins: 3834

celo_rating: 1198.52

family_friendly_score: 0.0

family_friendly_standard_error: 0.0

submission_type: basic

model_repo: ChaiML/grpo-q3b-merged-nemo70b-chai-rm-step-1591

model_architecture: Qwen2ForCausalLM

model_num_parameters: 3397011456.0

best_of: 8

max_input_tokens: 2048

max_output_tokens: 64

reward_model: default

display_name: chaiml-grpo-q3b-merged-_37630_v1

is_internal_developer: False

language_model: ChaiML/grpo-q3b-merged-nemo70b-chai-rm-step-1591

model_size: 3B

ranking_group: single

us_pacific_date: 2026-02-16

win_ratio: 0.362621772439232

generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': '### Instruction:\n{memory}\n', 'prompt_template': '### Input:\n{prompt}\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '### Response:\n{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-q3b-merged-37630-v1-uploader
Waiting for job on chaiml-grpo-q3b-merged-37630-v1-uploader to finish
chaiml-grpo-q3b-merged-37630-v1-uploader: Using quantization_mode: none
chaiml-grpo-q3b-merged-37630-v1-uploader: Downloading snapshot of ChaiML/grpo-q3b-merged-nemo70b-chai-rm-step-1591...
chaiml-grpo-q3b-merged-37630-v1-uploader: Downloaded in 5.193s
chaiml-grpo-q3b-merged-37630-v1-uploader: Processed model ChaiML/grpo-q3b-merged-nemo70b-chai-rm-step-1591 in 7.776s
chaiml-grpo-q3b-merged-37630-v1-uploader: creating bucket guanaco-vllm-models
chaiml-grpo-q3b-merged-37630-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q3b-merged-37630-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-grpo-q3b-merged-37630-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-grpo-q3b-merged-37630-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-grpo-q3b-merged-37630-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q3b-merged-37630-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-grpo-q3b-merged-37630-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q3b-merged-37630-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-grpo-q3b-merged-37630-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q3b-merged-37630-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-grpo-q3b-merged-37630-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-grpo-q3b-merged-37630-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-grpo-q3b-merged-37630-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-q3b-merged-37630-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-q3b-merged-37630-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-q3b-merged-37630-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-q3b-merged-37630-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-q3b-merged-37630-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/.gitattributes
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/added_tokens.json
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/chat_template.jinja
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/config.json
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/tokenizer_config.json
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/args.json
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/latest_checkpointed_iteration.txt s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/latest_checkpointed_iteration.txt
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/special_tokens_map.json
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/model.safetensors.index.json
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/iter_0001591/common.pt s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/iter_0001591/common.pt
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/merges.txt
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/tokenizer.json
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/vocab.json
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/model-00002-of-00002.safetensors
chaiml-grpo-q3b-merged-37630-v1-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-grpo-q3b-merged-37630-v1/default/model-00001-of-00002.safetensors
Job chaiml-grpo-q3b-merged-37630-v1-uploader completed after 83.56s with status: succeeded
Stopping job with name chaiml-grpo-q3b-merged-37630-v1-uploader
Pipeline stage VLLMUploader completed in 84.20s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-q3b-merged-37630-v1
Waiting for inference service chaiml-grpo-q3b-merged-37630-v1 to be ready
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Inference service chaiml-grpo-q3b-merged-37630-v1 ready after 161.17510557174683s
Pipeline stage VLLMDeployer completed in 161.84s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 0.9558110237121582s
Received healthy response to inference request in 0.9972476959228516s
Received healthy response to inference request in 0.8245415687561035s
Received healthy response to inference request in 1.1113178730010986s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.5798301696777344s
Received healthy response to inference request in 0.8108770847320557s
Received healthy response to inference request in 0.8516736030578613s
Received healthy response to inference request in 0.9058902263641357s
Received healthy response to inference request in 0.8383884429931641s
Received healthy response to inference request in 0.9201974868774414s
Received healthy response to inference request in 0.857837438583374s
Received healthy response to inference request in 1.0362520217895508s
Received healthy response to inference request in 0.9438021183013916s
Received healthy response to inference request in 0.837425947189331s
Received healthy response to inference request in 0.697070837020874s
Received healthy response to inference request in 0.9541964530944824s
Received healthy response to inference request in 0.9130637645721436s
Received healthy response to inference request in 0.6173591613769531s
Received healthy response to inference request in 1.0406224727630615s
Received healthy response to inference request in 1.678253173828125s
Received healthy response to inference request in 0.7810325622558594s
Received healthy response to inference request in 0.8460776805877686s
Received healthy response to inference request in 0.7247416973114014s
Received healthy response to inference request in 0.9984869956970215s
Received healthy response to inference request in 1.2019078731536865s
Received healthy response to inference request in 0.9977569580078125s
Received healthy response to inference request in 1.2645163536071777s
Received healthy response to inference request in 1.130272388458252s
Received healthy response to inference request in 0.8753044605255127s
Received healthy response to inference request in 0.7142713069915771s
30 requests
0 failed requests
5th percentile: 0.7048110485076904
10th percentile: 0.7236946582794189
20th percentile: 0.8218086719512939
30th percentile: 0.8437709093093873
40th percentile: 0.8683176517486573
50th percentile: 0.9166306257247925
60th percentile: 0.9548422813415527
70th percentile: 0.9979759693145752
80th percentile: 1.054761552810669
90th percentile: 1.2081687211990357
95th percentile: 1.437938952445983
99th percentile: 1.6497105026245118
mean time: 0.9635342280069987
Pipeline stage StressChecker completed in 33.34s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.63s
Shutdown handler de-registered
chaiml-grpo-q3b-merged-_37630_v1 status is now deployed due to DeploymentManager action
chaiml-grpo-q3b-merged-_37630_v1 status is now inactive due to auto deactivation removed underperforming models
admin requested tearing down of chaiml-grpo-q3b-merged-_37630_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMDeleter
Checking if service chaiml-grpo-q3b-merged-37630-v1 is running
Tearing down inference service chaiml-grpo-q3b-merged-37630-v1
Service chaiml-grpo-q3b-merged-37630-v1 has been torndown
Pipeline stage VLLMDeleter completed in 0.71s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
%s, retrying in %s seconds...
Cleaning model data from S3
Cleaning model data from model cache
%s, retrying in %s seconds...
Cleaning model data from S3
Cleaning model data from model cache
clean up pipeline due to error=TeardownError("Got unexpected keyword argument 'request_checksum_calculation'")
Shutdown handler de-registered
chaiml-grpo-q3b-merged-_37630_v1 status is now torndown due to DeploymentManager action