chaiml-mega-v0-q35b-lr5_54181

developer_uid: zonemercy
submission_id: chaiml-mega-v0-q35b-lr5_54181_v1
model_name: chaiml-mega-v0-q35b-lr5_54181_v1
model_group: ChaiML/mega-v0-q35b-lr5e
status: torndown
timestamp: 2026-03-25T06:41:11+00:00
num_battles: 10733
num_wins: 5388
celo_rating: 1296.04
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/mega-v0-q35b-lr5e6ep2g8
model_architecture: Qwen3_5MoeForConditionalGeneration
model_num_parameters: 33753909248.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-mega-v0-q35b-lr5_54181_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/mega-v0-q35b-lr5e6ep2g8
model_size: 34B
ranking_group: single
us_pacific_date: 2026-03-21
win_ratio: 0.5020031678002422
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '####', '<|assistant|>', '<|im_end|>', '</s>'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-mega-v0-q35b-lr5-54181-v1-uploader
Waiting for job on chaiml-mega-v0-q35b-lr5-54181-v1-uploader to finish
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: Using quantization_mode: none
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: Downloading snapshot of ChaiML/mega-v0-q35b-lr5e6ep2g8...
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: Downloaded in 41.023s
2026-03-22T03:38:21.268538+00:00 monitor updated for chaiml-mega-v0-q35b-lr5_54181_v1
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: Processed model ChaiML/mega-v0-q35b-lr5e6ep2g8 in 66.720s
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: creating bucket guanaco-vllm-models
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v0-q35b-lr5-54181-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-mega-v0-q35b-lr5-54181-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v0-q35b-lr5-54181-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v0-q35b-lr5-54181-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v0-q35b-lr5-54181-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v0-q35b-lr5-54181-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-mega-v0-q35b-lr5-54181-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-mega-v0-q35b-lr5-54181-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/preprocessor_config.json s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/preprocessor_config.json
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/trainer_state.json s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/trainer_state.json
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/tokenizer_config.json
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/args.json s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/args.json
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/.gitattributes s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/.gitattributes
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/README.md s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/README.md
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/chat_template.jinja
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/config.json
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/generation_config.json
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/model.safetensors.index.json
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/processor_config.json s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/processor_config.json
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/merges.txt
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/training_args.bin s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/training_args.bin
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/vocab.json
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/special_tokens_map.json
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/added_tokens.json
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/tokenizer.json
2026-03-22T03:39:21.363561+00:00 monitor updated for chaiml-mega-v0-q35b-lr5_54181_v1
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/model-00002-of-00002.safetensors s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/model-00002-of-00002.safetensors
chaiml-mega-v0-q35b-lr5-54181-v1-uploader: cp /dev/shm/model_output/model-00001-of-00002.safetensors s3://guanaco-vllm-models/chaiml-mega-v0-q35b-lr5-54181-v1/default/model-00001-of-00002.safetensors
Job chaiml-mega-v0-q35b-lr5-54181-v1-uploader completed after 184.61s with status: succeeded
2026-03-22T03:40:21.447726+00:00 monitor updated for chaiml-mega-v0-q35b-lr5_54181_v1
Stopping job with name chaiml-mega-v0-q35b-lr5-54181-v1-uploader
Pipeline stage VLLMUploader completed in 187.41s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 4.13s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-mega-v0-q35b-lr5-54181-v1
Waiting for inference service chaiml-mega-v0-q35b-lr5-54181-v1 to be ready
2026-03-22T03:41:28.600819+00:00 monitor updated for chaiml-mega-v0-q35b-lr5_54181_v1
2026-03-22T03:42:28.716889+00:00 monitor updated for chaiml-mega-v0-q35b-lr5_54181_v1
2026-03-22T03:43:28.827604+00:00 monitor updated for chaiml-mega-v0-q35b-lr5_54181_v1
Inference service chaiml-mega-v0-q35b-lr5-54181-v1 ready after 180.46376967430115s
Pipeline stage VLLMDeployer completed in 182.08s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-22T03:44:28.924358+00:00 monitor updated for chaiml-mega-v0-q35b-lr5_54181_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-22T03:45:29.491428+00:00 monitor updated for chaiml-mega-v0-q35b-lr5_54181_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 11.409895896911621s
Received healthy response to inference request in 2.765901565551758s
{"detail":"('http://chaiml-mega-v0-q35b-lr5-54181-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'upstream connect error or disconnect/reset before headers. reset reason: connection termination')"}
Received unhealthy response to inference request!
Received healthy response to inference request in 1.1580121517181396s
Received healthy response to inference request in 3.997821569442749s
Received healthy response to inference request in 5.165168762207031s
2026-03-22T03:46:31.479460+00:00 monitor updated for chaiml-mega-v0-q35b-lr5_54181_v1
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.4214367866516113s
Received healthy response to inference request in 1.6369667053222656s
Received healthy response to inference request in 2.330329179763794s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.1474354267120361s
Received healthy response to inference request in 1.0413827896118164s
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.955246686935425s
2026-03-22T03:47:31.875304+00:00 monitor updated for chaiml-mega-v0-q35b-lr5_54181_v1
Received healthy response to inference request in 2.673746347427368s
Received healthy response to inference request in 1.4645781517028809s
Received healthy response to inference request in 1.5060179233551025s
Received healthy response to inference request in 1.6162853240966797s
Received healthy response to inference request in 1.336810827255249s
Received healthy response to inference request in 1.1673791408538818s
Received healthy response to inference request in 1.398820400238037s
Received healthy response to inference request in 1.6559453010559082s
Received healthy response to inference request in 1.0614335536956787s
30 requests
10 failed requests
5th percentile: 1.1001343965530395
10th percentile: 1.1569544792175293
20th percentile: 1.3864184856414796
30th percentile: 1.493585991859436
40th percentile: 1.6483538627624512
50th percentile: 2.719823956489563
60th percentile: 4.4647604465484605
70th percentile: 18.55570404529571
80th percentile: 20.14047703742981
90th percentile: 20.152244400978088
95th percentile: 20.18165363073349
99th percentile: 20.228648753166198
mean time: 8.274506767590841
%s, retrying in %s seconds...
Received healthy response to inference request in 1.116748571395874s
Received healthy response to inference request in 1.1242001056671143s
Received healthy response to inference request in 1.0455224514007568s
Received healthy response to inference request in 1.0489997863769531s
Received healthy response to inference request in 1.0987122058868408s
Received healthy response to inference request in 1.1161384582519531s
Received healthy response to inference request in 1.1706151962280273s
Received healthy response to inference request in 1.3016753196716309s
Received healthy response to inference request in 1.1430463790893555s
Received healthy response to inference request in 1.0715887546539307s
Received healthy response to inference request in 1.0826427936553955s
Received healthy response to inference request in 1.329308271408081s
Received healthy response to inference request in 1.424560546875s
Received healthy response to inference request in 1.4026527404785156s
Received healthy response to inference request in 1.1273574829101562s
Received healthy response to inference request in 1.0908057689666748s
Received healthy response to inference request in 1.1306779384613037s
Received healthy response to inference request in 1.2018756866455078s
Received healthy response to inference request in 1.124168872833252s
Received healthy response to inference request in 1.203479528427124s
Received healthy response to inference request in 1.2275636196136475s
Received healthy response to inference request in 1.035280704498291s
Received healthy response to inference request in 1.1733849048614502s
Received healthy response to inference request in 1.196044683456421s
Received healthy response to inference request in 1.2851362228393555s
Received healthy response to inference request in 1.2327358722686768s
Received healthy response to inference request in 1.2509584426879883s
Received healthy response to inference request in 1.0193803310394287s
Received healthy response to inference request in 1.1687421798706055s
Received healthy response to inference request in 1.1370818614959717s
30 requests
0 failed requests
5th percentile: 1.0398894906044007
10th percentile: 1.0486520528793335
20th percentile: 1.089173173904419
30th percentile: 1.1165655374526977
40th percentile: 1.1260945320129394
50th percentile: 1.1400641202926636
60th percentile: 1.1717230796813964
70th percentile: 1.2023568391799926
80th percentile: 1.236380386352539
90th percentile: 1.304438614845276
95th percentile: 1.36964772939682
99th percentile: 1.4182072830200196
mean time: 1.1693695227305094
Pipeline stage StressChecker completed in 289.71s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 1.46s
Shutdown handler de-registered
chaiml-mega-v0-q35b-lr5_54181_v1 status is now deployed due to DeploymentManager action
chaiml-mega-v0-q35b-lr5_54181_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-mega-v0-q35b-lr5_54181_v1 status is now torndown due to DeploymentManager action