developer_uid: NischayDnk
submission_id: chaiml-qwen32b-simpoexp_21972_v8
model_name: chaiml-qwen32b-simpoexp_21972_v8
model_group: ChaiML/qwen32b-simpoexp1
status: torndown
timestamp: 2025-09-29T20:33:27+00:00
num_battles: 5853
num_wins: 2776
celo_rating: 1255.53
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/qwen32b-simpoexp1-s2-ftsimpoexp4-1330pref
model_architecture: Qwen2ForCausalLM
model_num_parameters: 32759331840.0
best_of: 5
max_input_tokens: 768
max_output_tokens: 64
reward_model: default
latencies: [{'batch_size': 1, 'throughput': 0.41106958158214135, 'latency_mean': 2.4325566804409027, 'latency_p50': 2.427756428718567, 'latency_p90': 2.7247370958328245}, {'batch_size': 3, 'throughput': 0.8895812725876078, 'latency_mean': 3.365063601732254, 'latency_p50': 3.3487322330474854, 'latency_p90': 3.7557350635528564}, {'batch_size': 5, 'throughput': 1.1997365278924017, 'latency_mean': 4.142294427156449, 'latency_p50': 4.1444772481918335, 'latency_p90': 4.69084677696228}, {'batch_size': 6, 'throughput': 1.320656258636561, 'latency_mean': 4.49461419582367, 'latency_p50': 4.511002779006958, 'latency_p90': 5.06336612701416}, {'batch_size': 8, 'throughput': 1.4789126397831682, 'latency_mean': 5.378047839403153, 'latency_p50': 5.432672381401062, 'latency_p90': 6.002550864219666}, {'batch_size': 10, 'throughput': 1.6117777908795148, 'latency_mean': 6.140884531736374, 'latency_p50': 6.1505690813064575, 'latency_p90': 6.91472282409668}]
gpu_counts: {'NVIDIA L40S': 1}
display_name: chaiml-qwen32b-simpoexp_21972_v8
is_internal_developer: False
language_model: ChaiML/qwen32b-simpoexp1-s2-ftsimpoexp4-1330pref
model_size: 33B
ranking_group: single
throughput_3p7s: 1.04
us_pacific_date: 2025-09-29
win_ratio: 0.47428669058602424
generation_params: {'temperature': 0.45, 'top_p': 0.95, 'min_p': 0.025, 'top_k': 40, 'presence_penalty': 0.35, 'frequency_penalty': 0.35, 'stopping_words': ['<|im_end|>', '\n', '<|im_start|>'], 'max_input_tokens': 768, 'best_of': 5, 'max_output_tokens': 64}
formatter: {'memory_template': '<|system|>Family Friendly{memory}\n', 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{message}<|im_end|>\n', 'user_template': '<|im_start|>user\nYou:{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-qwen32b-simpoexp-21972-v8-mkmlizer
Waiting for job on chaiml-qwen32b-simpoexp-21972-v8-mkmlizer to finish
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║                                                                     ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║                                                                     ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║  Version: 0.30.2                                                    ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║                                                                     ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║  belonging to:                                                      ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║                                                                     ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ║                                                                     ║
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: Downloaded to shared memory in 71.475s
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: Checking if ChaiML/qwen32b-simpoexp1-s2-ftsimpoexp4-1330pref already exists in ChaiML
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: quantizing model to /dev/shm/model_cache, profile:q4, folder:/tmp/tmp3vtij0ee, device:0
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission chaiml-llama38b-1800seqt_5646_v1: HTTPConnectionPool(host='chaiml-llama38b-1800seqt-5646-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-smallbase-nis-mu_14714_v2: HTTPConnectionPool(host='chaiml-smallbase-nis-mu-14714-v2-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission rirv938-grpo-20250926-c_74384_v2: HTTPConnectionPool(host='rirv938-grpo-20250926-c-74384-v2-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-high-school-life_46597_v2: HTTPConnectionPool(host='chaiml-high-school-life-46597-v2-predictor.creator-studio.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-mistral31-24b-s_69496_v30: HTTPConnectionPool(host='chaiml-llama31-mer-v2-t-44570-v4-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: quantized model in 407.738s
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: Processed model ChaiML/qwen32b-simpoexp1-s2-ftsimpoexp4-1330pref in 479.214s
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: creating bucket guanaco-mkml-models
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-qwen32b-simpoexp-21972-v8/nvidia
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-qwen32b-simpoexp-21972-v8/nvidia/config.json
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-qwen32b-simpoexp-21972-v8/nvidia/tokenizer_config.json
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: cp /dev/shm/model_cache/added_tokens.json s3://guanaco-mkml-models/chaiml-qwen32b-simpoexp-21972-v8/nvidia/added_tokens.json
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-qwen32b-simpoexp-21972-v8/nvidia/special_tokens_map.json
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: cp /dev/shm/model_cache/merges.txt s3://guanaco-mkml-models/chaiml-qwen32b-simpoexp-21972-v8/nvidia/merges.txt
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: cp /dev/shm/model_cache/vocab.json s3://guanaco-mkml-models/chaiml-qwen32b-simpoexp-21972-v8/nvidia/vocab.json
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-qwen32b-simpoexp-21972-v8/nvidia/tokenizer.json
chaiml-qwen32b-simpoexp-21972-v8-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-qwen32b-simpoexp-21972-v8/nvidia/flywheel_model.0.safetensors
Job chaiml-qwen32b-simpoexp-21972-v8-mkmlizer completed after 542.03s with status: succeeded
Stopping job with name chaiml-qwen32b-simpoexp-21972-v8-mkmlizer
Pipeline stage MKMLizer completed in 542.50s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-qwen32b-simpoexp-21972-v8
Waiting for inference service chaiml-qwen32b-simpoexp-21972-v8 to be ready
Inference service chaiml-qwen32b-simpoexp-21972-v8 ready after 70.3426764011383s
Pipeline stage MKMLDeployer completed in 70.78s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.965850591659546s
Failed to get response for submission chaiml-llama38b-1800seqt_5646_v1: HTTPConnectionPool(host='chaiml-llama38b-1800seqt-5646-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Received healthy response to inference request in 2.763892889022827s
Received healthy response to inference request in 2.848867893218994s
Received healthy response to inference request in 2.5974130630493164s
5 requests
1 failed requests
5th percentile: 2.6307090282440186
10th percentile: 2.664004993438721
20th percentile: 2.730596923828125
30th percentile: 2.7808878898620604
40th percentile: 2.8148778915405273
50th percentile: 2.848867893218994
60th percentile: 3.295660972595215
70th percentile: 3.7424540519714355
80th percentile: 7.253497552871707
90th percentile: 13.828791475296022
95th percentile: 17.116438436508176
99th percentile: 19.746556005477906
mean time: 6.516021966934204
%s, retrying in %s seconds...
Received healthy response to inference request in 2.437375783920288s
Received healthy response to inference request in 2.8385391235351562s
Failed to get response for submission chaiml-smallbase-nis-mu_14714_v1: HTTPConnectionPool(host='chaiml-smallbase-nis-mu-14714-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Received healthy response to inference request in 2.5071029663085938s
Received healthy response to inference request in 2.2479805946350098s
Received healthy response to inference request in 2.932647228240967s
5 requests
0 failed requests
5th percentile: 2.2858596324920653
10th percentile: 2.323738670349121
20th percentile: 2.3994967460632326
30th percentile: 2.4513212203979493
40th percentile: 2.4792120933532713
50th percentile: 2.5071029663085938
60th percentile: 2.6396774291992187
70th percentile: 2.7722518920898436
80th percentile: 2.8573607444763183
90th percentile: 2.8950039863586428
95th percentile: 2.9138256072998048
99th percentile: 2.928882904052734
mean time: 2.5927291393280028
Pipeline stage StressChecker completed in 48.76s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.65s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.66s
Shutdown handler de-registered
chaiml-qwen32b-simpoexp_21972_v8 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-qwen32b-simpoexp-21972-v8-profiler
Waiting for inference service chaiml-qwen32b-simpoexp-21972-v8-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
Received signal 15, running shutdown handler
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3384.74s
Shutdown handler de-registered
chaiml-qwen32b-simpoexp_21972_v8 status is now torndown due to DeploymentManager action