developer_uid: zonemercy
submission_id: chaiml-mega-v1-top2-q27b_5145_v1
model_name: chaiml-mega-v1-top2-q27b_5145_v1
model_group: ChaiML/mega-v1-top2-q27b
status: torndown
timestamp: 2026-03-31T15:56:10+00:00
num_battles: 7861
num_wins: 4053
celo_rating: 1306.22
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/mega-v1-top2-q27b-lr5e6ep2g8
model_architecture: Qwen3_5ForConditionalGeneration
model_num_parameters: 23564784640.0
best_of: 8
max_input_tokens: 2048
max_output_tokens: 80
reward_model: default
display_name: chaiml-mega-v1-top2-q27b_5145_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/mega-v1-top2-q27b-lr5e6ep2g8
model_size: 24B
ranking_group: single
us_pacific_date: 2026-03-28
win_ratio: 0.5155832591273375
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|im_end|>', '<|user|>', '<|assistant|>', '</s>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-mega-v1-top2-q27b-5145-v1-uploader
Waiting for job on chaiml-mega-v1-top2-q27b-5145-v1-uploader to finish
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Using quantization_mode: fp8
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Checking if ChaiML/mega-v1-top2-q27b-lr5e6ep2g8-FP8 already exists in ChaiML
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Downloading snapshot of ChaiML/mega-v1-top2-q27b-lr5e6ep2g8...
2026-03-28T13:08:05.763878+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Downloaded in 47.814s
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Loading /tmp/model_input...
chaiml-mega-v1-top2-q27b-5145-v1-uploader: The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Applying quantization...
chaiml-mega-v1-top2-q27b-5145-v1-uploader: 2026-03-28T13:08:14.841256+0000 | __init__ | WARNING - Disabling tokenizer parallelism due to threading conflict between FastTokenizer and Datasets. Set TOKENIZERS_PARALLELISM=false to suppress this warning.
chaiml-mega-v1-top2-q27b-5145-v1-uploader: 2026-03-28T13:08:16.809352+0000 | reset | INFO - Compression lifecycle reset
chaiml-mega-v1-top2-q27b-5145-v1-uploader: 2026-03-28T13:08:16.811628+0000 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-mega-v1-top2-q27b-5145-v1-uploader: 2026-03-28T13:08:16.857894+0000 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-mega-v1-top2-q27b-5145-v1-uploader: 2026-03-28T13:08:16.858149+0000 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-mega-v1-top2-q27b-5145-v1-uploader: 2026-03-28T13:08:16.872408+0000 | dispatch_model | WARNING - Forced to offload modules due to insufficient gpu resources
chaiml-mega-v1-top2-q27b-5145-v1-uploader: 2026-03-28T13:08:23.587064+0000 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-mega-v1-top2-q27b-5145-v1-uploader: 2026-03-28T13:08:23.587236+0000 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Saving to /dev/shm/model_output...
chaiml-mega-v1-top2-q27b-5145-v1-uploader: /usr/local/lib/python3.12/dist-packages/transformers/modeling_utils.py:3344: UserWarning: Attempting to save a model with offloaded modules. Ensure that unallocated cpu memory exceeds the `shard_size` (50GB default)
chaiml-mega-v1-top2-q27b-5145-v1-uploader:   warnings.warn(
2026-03-28T13:09:06.403033+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Pushing to ChaiML/mega-v1-top2-q27b-lr5e6ep2g8-FP8
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Checking if ChaiML/mega-v1-top2-q27b-lr5e6ep2g8-FP8 already exists in ChaiML
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Creating repo ChaiML/mega-v1-top2-q27b-lr5e6ep2g8-FP8 and uploading /dev/shm/model_output to it
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Found 1 files larger than 20GB (recommended limit):
chaiml-mega-v1-top2-q27b-5145-v1-uploader:   - model.safetensors: 35.9GB
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Large files may slow down loading and processing.
chaiml-mega-v1-top2-q27b-5145-v1-uploader: ---------- 2026-03-28 13:09:12 (0:00:00) ----------
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Files:   hashed 5/7 (34.1K/35.9G) | pre-uploaded: 0/0 (0.0/35.9G) (+7 unsure) | committed: 0/7 (0.0/35.9G) | ignored: 0
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Workers: hashing: 2 | get upload mode: 5 | pre-uploading: 0 | committing: 0 | waiting: 57
chaiml-mega-v1-top2-q27b-5145-v1-uploader: ---------------------------------------------------
2026-03-28T13:10:06.622179+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
chaiml-mega-v1-top2-q27b-5145-v1-uploader: 







chaiml-mega-v1-top2-q27b-5145-v1-uploader: ---------- 2026-03-28 13:10:12 (0:01:00) ----------
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Files:   hashed 7/7 (35.9G/35.9G) | pre-uploaded: 1/2 (20.0M/35.9G) | committed: 0/7 (0.0/35.9G) | ignored: 0
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 1 | committing: 0 | waiting: 63
chaiml-mega-v1-top2-q27b-5145-v1-uploader: ---------------------------------------------------
2026-03-28T13:11:06.734104+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
chaiml-mega-v1-top2-q27b-5145-v1-uploader:                              Processed model ChaiML/mega-v1-top2-q27b-lr5e6ep2g8 in 220.022s
chaiml-mega-v1-top2-q27b-5145-v1-uploader: creating bucket guanaco-vllm-models
chaiml-mega-v1-top2-q27b-5145-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-top2-q27b-5145-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-mega-v1-top2-q27b-5145-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-mega-v1-top2-q27b-5145-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-mega-v1-top2-q27b-5145-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-top2-q27b-5145-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-mega-v1-top2-q27b-5145-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-top2-q27b-5145-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-mega-v1-top2-q27b-5145-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-top2-q27b-5145-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-mega-v1-top2-q27b-5145-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-mega-v1-top2-q27b-5145-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-mega-v1-top2-q27b-5145-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-mega-v1-top2-q27b-5145-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-mega-v1-top2-q27b-5145-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-mega-v1-top2-q27b-5145-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-mega-v1-top2-q27b-5145-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-mega-v1-top2-q27b-5145-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-mega-v1-top2-q27b-5145-v1/default
chaiml-mega-v1-top2-q27b-5145-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-mega-v1-top2-q27b-5145-v1/default/chat_template.jinja
chaiml-mega-v1-top2-q27b-5145-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-mega-v1-top2-q27b-5145-v1/default/config.json
chaiml-mega-v1-top2-q27b-5145-v1-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-mega-v1-top2-q27b-5145-v1/default/recipe.yaml
chaiml-mega-v1-top2-q27b-5145-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-mega-v1-top2-q27b-5145-v1/default/tokenizer_config.json
chaiml-mega-v1-top2-q27b-5145-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-mega-v1-top2-q27b-5145-v1/default/generation_config.json
chaiml-mega-v1-top2-q27b-5145-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-mega-v1-top2-q27b-5145-v1/default/tokenizer.json
Failed to get response for submission chaiml-gspo-glm47-combi_10268_v1: ('http://chaiml-gspo-glm47-combi-10268-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
chaiml-mega-v1-top2-q27b-5145-v1-uploader: cp /dev/shm/model_output/model.safetensors s3://guanaco-vllm-models/chaiml-mega-v1-top2-q27b-5145-v1/default/model.safetensors
2026-03-28T13:12:06.820202+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
Job chaiml-mega-v1-top2-q27b-5145-v1-uploader completed after 308.24s with status: succeeded
Stopping job with name chaiml-mega-v1-top2-q27b-5145-v1-uploader
Pipeline stage VLLMUploader completed in 308.69s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.11s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.36s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-mega-v1-top2-q27b-5145-v1
Waiting for inference service chaiml-mega-v1-top2-q27b-5145-v1 to be ready
Failed to get response for submission chaiml-gspo-glm47-chai-_76408_v1: ('http://chaiml-gspo-glm47-chai-76408-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
2026-03-28T13:13:06.917530+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
2026-03-28T13:14:07.019625+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
2026-03-28T13:15:07.115165+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
Failed to get response for submission chaiml-gspo-glm47-cas72_44260_v1: ('http://chaiml-gspo-glm47-cas72-44260-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
2026-03-28T13:16:07.212574+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
Inference service chaiml-mega-v1-top2-q27b-5145-v1 ready after 231.22003483772278s
Pipeline stage VLLMDeployer completed in 232.36s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T13:17:07.307262+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 12.96157169342041s
Received healthy response to inference request in 18.521953582763672s
Received healthy response to inference request in 2.0126724243164062s
Received healthy response to inference request in 1.8765223026275635s
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
2026-03-28T13:18:07.429579+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
Failed to get request counts for guanaco-submitter. Falling back to default
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Failed to get response for submission chaiml-gspo-glm47-chai-_76408_v1: ('http://chaiml-gspo-glm47-chai-76408-v1-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/completions', 'activator request timeout')
HTTPConnectionPool(host='guanaco-submitter-v2.guanaco-backend.kchai-google-us-east4.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 4.202596426010132s
Received healthy response to inference request in 1.9158387184143066s
Received healthy response to inference request in 1.8553550243377686s
Received healthy response to inference request in 1.9494431018829346s
Received healthy response to inference request in 1.9233067035675049s
Received healthy response to inference request in 2.3598575592041016s
Received healthy response to inference request in 2.009852170944214s
Received healthy response to inference request in 4.455955743789673s
2026-03-28T13:19:07.546856+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
Received healthy response to inference request in 1.9360692501068115s
Received healthy response to inference request in 8.967830657958984s
Received healthy response to inference request in 2.038426399230957s
Received healthy response to inference request in 2.0143141746520996s
Received healthy response to inference request in 2.4229066371917725s
Received healthy response to inference request in 2.01887845993042s
Received healthy response to inference request in 1.9661622047424316s
Received healthy response to inference request in 2.2199974060058594s
Received healthy response to inference request in 2.1349010467529297s
Received healthy response to inference request in 2.1436116695404053s
Received healthy response to inference request in 2.0499801635742188s
Received healthy response to inference request in 2.3133816719055176s
30 requests
6 failed requests
5th percentile: 1.894214689731598
10th percentile: 1.922559905052185
20th percentile: 1.9628183841705322
30th percentile: 2.0138216495513914
40th percentile: 2.045358657836914
50th percentile: 2.1818045377731323
60th percentile: 2.3850771903991697
70th percentile: 5.809518218040454
80th percentile: 18.849644613265998
90th percentile: 20.185755443572997
95th percentile: 20.439347517490386
99th percentile: 20.639282035827637
mean time: 7.009964028994243
%s, retrying in %s seconds...
Received healthy response to inference request in 1.6899526119232178s
Received healthy response to inference request in 2.022792100906372s
Received healthy response to inference request in 2.094999313354492s
Received healthy response to inference request in 2.152872085571289s
Received healthy response to inference request in 1.8281166553497314s
Received healthy response to inference request in 1.953995704650879s
Received healthy response to inference request in 2.126018762588501s
Received healthy response to inference request in 1.9070589542388916s
Received healthy response to inference request in 1.8323349952697754s
Received healthy response to inference request in 1.7377326488494873s
Received healthy response to inference request in 1.9765069484710693s
Received healthy response to inference request in 1.8944075107574463s
Received healthy response to inference request in 2.2092673778533936s
2026-03-28T13:20:07.650956+00:00 monitor updated for chaiml-mega-v1-top2-q27b_5145_v1
Received healthy response to inference request in 1.902787685394287s
Received healthy response to inference request in 1.9063794612884521s
Received healthy response to inference request in 1.9090025424957275s
Received healthy response to inference request in 1.9864938259124756s
Received healthy response to inference request in 1.9704508781433105s
Received healthy response to inference request in 1.930074691772461s
Received healthy response to inference request in 2.033071756362915s
Received healthy response to inference request in 1.898395299911499s
Received healthy response to inference request in 2.0108587741851807s
Received healthy response to inference request in 2.320932626724243s
Received healthy response to inference request in 2.8500819206237793s
Received healthy response to inference request in 1.9346647262573242s
Received healthy response to inference request in 2.0125019550323486s
Received healthy response to inference request in 1.9213905334472656s
Received healthy response to inference request in 2.3146612644195557s
Received healthy response to inference request in 2.01717472076416s
Received healthy response to inference request in 2.384347915649414s
30 requests
0 failed requests
5th percentile: 1.778405451774597
10th percentile: 1.831913161277771
20th percentile: 1.9019092082977296
30th percentile: 1.9084194660186768
40th percentile: 1.932828712463379
50th percentile: 1.97347891330719
60th percentile: 2.011516046524048
70th percentile: 2.025875997543335
80th percentile: 2.131389427185059
90th percentile: 2.3152884006500245
95th percentile: 2.355811035633087
99th percentile: 2.7150190591812136
mean time: 2.0243108749389647
Pipeline stage StressChecker completed in 279.19s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.66s
Shutdown handler de-registered
chaiml-mega-v1-top2-q27b_5145_v1 status is now deployed due to DeploymentManager action
chaiml-mega-v1-top2-q27b_5145_v1 status is now inactive due to admin request
chaiml-mega-v1-top2-q27b_5145_v1 status is now torndown due to DeploymentManager action