developer_uid: NischayDnk
submission_id: chaiml-devstral24b-simp_15867_v1
model_name: chaiml-devstral24b-simp_15867_v1
model_group: ChaiML/devstral24b-simpo
status: torndown
timestamp: 2025-06-05T09:56:21+00:00
num_battles: 14400
num_wins: 7638
celo_rating: 1291.8
family_friendly_score: 0.49360000000000004
family_friendly_standard_error: 0.007070488526261817
submission_type: basic
model_repo: ChaiML/devstral24b-simpos1-sfts1ftnomask-b02-2e-1
model_architecture: MistralForCausalLM
model_num_parameters: 24096691200.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
reward_model: default
display_name: chaiml-devstral24b-simp_15867_v1
is_internal_developer: False
language_model: ChaiML/devstral24b-simpos1-sfts1ftnomask-b02-2e-1
model_size: 24B
ranking_group: single
us_pacific_date: 2025-06-05
win_ratio: 0.5304166666666666
generation_params: {'temperature': 0.45, 'top_p': 0.95, 'min_p': 0.025, 'top_k': 60, 'presence_penalty': 0.4, 'frequency_penalty': 0.4, 'stopping_words': ['<|im_end|>', '\n', '<|im_start|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '<|system|>Family Friendly{memory}\n', 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{message}<|im_end|>\n', 'user_template': '<|im_start|>user\nYou:{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-devstral24b-simp-15867-v1-mkmlizer
Waiting for job on chaiml-devstral24b-simp-15867-v1-mkmlizer to finish
chaiml-devstral24b-simp-15867-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║                                                                     ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║                                                                     ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║  Version: 0.27.1+vampire_v3                                         ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║                                                                     ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║  belonging to:                                                      ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║                                                                     ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ║                                                                     ║
chaiml-devstral24b-simp-15867-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission cycy233-4096t-pk3-clean_43009_v1: HTTPConnectionPool(host='cycy233-4096t-pk3-clean-43009-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission cycy233-4096t-pk3-clean_43009_v1: HTTPConnectionPool(host='cycy233-4096t-pk3-clean-43009-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiml-devstral24b-simp-15867-v1-mkmlizer: Downloaded to shared memory in 75.643s
chaiml-devstral24b-simp-15867-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpjkuoa5j9, device:0
chaiml-devstral24b-simp-15867-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission cycy233-4096t-pk3-clean_43009_v1: HTTPConnectionPool(host='cycy233-4096t-pk3-clean-43009-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiml-devstral24b-simp-15867-v1-mkmlizer: quantized model in 47.995s
chaiml-devstral24b-simp-15867-v1-mkmlizer: Processed model ChaiML/devstral24b-simpos1-sfts1ftnomask-b02-2e-1 in 123.639s
chaiml-devstral24b-simp-15867-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-devstral24b-simp-15867-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-devstral24b-simp-15867-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-devstral24b-simp-15867-v1
chaiml-devstral24b-simp-15867-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-devstral24b-simp-15867-v1/config.json
chaiml-devstral24b-simp-15867-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-devstral24b-simp-15867-v1/special_tokens_map.json
chaiml-devstral24b-simp-15867-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-devstral24b-simp-15867-v1/tokenizer_config.json
chaiml-devstral24b-simp-15867-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-devstral24b-simp-15867-v1/tokenizer.json
chaiml-devstral24b-simp-15867-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-devstral24b-simp-15867-v1/flywheel_model.0.safetensors
chaiml-devstral24b-simp-15867-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/chaiml-devstral24b-simp-15867-v1/flywheel_model.1.safetensors
Failed to get response for submission cycy233-4096t-pk3-clean_43009_v1: HTTPConnectionPool(host='cycy233-4096t-pk3-clean-43009-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Job chaiml-devstral24b-simp-15867-v1-mkmlizer completed after 156.48s with status: succeeded
Stopping job with name chaiml-devstral24b-simp-15867-v1-mkmlizer
Pipeline stage MKMLizer completed in 157.25s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.18s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-devstral24b-simp-15867-v1
Waiting for inference service chaiml-devstral24b-simp-15867-v1 to be ready
Failed to get response for submission cycy233-4096t-pk3-clean_43009_v1: HTTPConnectionPool(host='cycy233-4096t-pk3-clean-43009-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission cycy233-4096t-pk3-clean_43009_v1: HTTPConnectionPool(host='cycy233-4096t-pk3-clean-43009-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service chaiml-devstral24b-simp-15867-v1 ready after 110.48207950592041s
Pipeline stage MKMLDeployer completed in 111.08s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.7943100929260254s
Received healthy response to inference request in 2.426884651184082s
Received healthy response to inference request in 2.770947217941284s
Received healthy response to inference request in 1.879683494567871s
5 requests
1 failed requests
5th percentile: 1.9891237258911132
10th percentile: 2.0985639572143553
20th percentile: 2.31744441986084
30th percentile: 2.4956971645355224
40th percentile: 2.6333221912384035
50th percentile: 2.770947217941284
60th percentile: 2.7802923679351808
70th percentile: 2.7896375179290773
80th percentile: 6.26804089546204
90th percentile: 13.21550250053406
95th percentile: 16.689233303070065
99th percentile: 19.468217945098875
mean time: 6.006957912445069
%s, retrying in %s seconds...
Received healthy response to inference request in 1.9464714527130127s
Received healthy response to inference request in 1.9641873836517334s
Received healthy response to inference request in 1.846435785293579s
Received healthy response to inference request in 1.8127501010894775s
Received healthy response to inference request in 2.2114124298095703s
5 requests
0 failed requests
5th percentile: 1.8194872379302978
10th percentile: 1.8262243747711182
20th percentile: 1.8396986484527589
30th percentile: 1.8664429187774658
40th percentile: 1.9064571857452393
50th percentile: 1.9464714527130127
60th percentile: 1.953557825088501
70th percentile: 1.9606441974639892
80th percentile: 2.013632392883301
90th percentile: 2.1125224113464354
95th percentile: 2.161967420578003
99th percentile: 2.2015234279632567
mean time: 1.9562514305114747
Pipeline stage StressChecker completed in 42.57s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.73s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.70s
Shutdown handler de-registered
chaiml-devstral24b-simp_15867_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 4904.95s
Shutdown handler de-registered
chaiml-devstral24b-simp_15867_v1 status is now inactive due to auto deactivation removed underperforming models
chaiml-devstral24b-simp_15867_v1 status is now torndown due to DeploymentManager action