developer_uid: azuruce
submission_id: chaiml-test-configurabl_44178_v7
model_name: chaiml-test-configurabl_44178_v7
model_group: ChaiML/test_configurable
status: torndown
timestamp: 2025-01-05T08:57:49+00:00
num_battles: 11661
num_wins: 5348
celo_rating: 1229.62
family_friendly_score: 0.5962000000000001
family_friendly_standard_error: 0.006938956117457438
submission_type: basic
model_repo: ChaiML/test_configurable_dpo_trainer_
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 4
max_input_tokens: 1024
max_output_tokens: 64
latencies: [{'batch_size': 1, 'throughput': 0.6346953334712856, 'latency_mean': 1.5754961264133454, 'latency_p50': 1.5706982612609863, 'latency_p90': 1.7365094423294067}, {'batch_size': 3, 'throughput': 1.2646131529049502, 'latency_mean': 2.3631171083450315, 'latency_p50': 2.3497923612594604, 'latency_p90': 2.59402391910553}, {'batch_size': 5, 'throughput': 1.6085576859384756, 'latency_mean': 3.0974833142757414, 'latency_p50': 3.1007648706436157, 'latency_p90': 3.523596167564392}, {'batch_size': 6, 'throughput': 1.714405478066159, 'latency_mean': 3.48491101026535, 'latency_p50': 3.4915201663970947, 'latency_p90': 3.9019840240478514}, {'batch_size': 8, 'throughput': 1.853551066784617, 'latency_mean': 4.28264611363411, 'latency_p50': 4.285318732261658, 'latency_p90': 4.815673685073852}, {'batch_size': 10, 'throughput': 1.9601482926500993, 'latency_mean': 5.070223833322525, 'latency_p50': 5.0550538301467896, 'latency_p90': 5.760363459587097}]
gpu_counts: {'NVIDIA RTX A5000': 1}
display_name: chaiml-test-configurabl_44178_v7
is_internal_developer: True
language_model: ChaiML/test_configurable_dpo_trainer_
model_size: 13B
ranking_group: single
throughput_3p7s: 1.77
us_pacific_date: 2025-01-05
win_ratio: 0.4586227596261041
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 100, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', 'Me', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 4, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-test-configurabl-44178-v7-mkmlizer
Waiting for job on chaiml-test-configurabl-44178-v7-mkmlizer to finish
chaiml-test-configurabl-44178-v7-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-test-configurabl-44178-v7-mkmlizer: ║     _____            __           __                                ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║       /___/                                                         ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║                                                                     ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║  Version: 0.11.12                                                   ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║                                                                     ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║  belonging to:                                                      ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║                                                                     ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║  Expiration: 2025-04-15 23:59:59                                    ║
chaiml-test-configurabl-44178-v7-mkmlizer: ║                                                                     ║
chaiml-test-configurabl-44178-v7-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission rirv938-viral-20k-624-b_93859_v1: HTTPConnectionPool(host='rirv938-viral-20k-624-b-93859-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
chaiml-test-configurabl-44178-v7-mkmlizer: Downloaded to shared memory in 46.031s
chaiml-test-configurabl-44178-v7-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpdxye4y9b, device:0
chaiml-test-configurabl-44178-v7-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-test-configurabl-44178-v7-mkmlizer: quantized model in 35.457s
chaiml-test-configurabl-44178-v7-mkmlizer: Processed model ChaiML/test_configurable_dpo_trainer_ in 81.488s
chaiml-test-configurabl-44178-v7-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-test-configurabl-44178-v7-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-test-configurabl-44178-v7
chaiml-test-configurabl-44178-v7-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-test-configurabl-44178-v7/config.json
chaiml-test-configurabl-44178-v7-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-test-configurabl-44178-v7/special_tokens_map.json
chaiml-test-configurabl-44178-v7-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-test-configurabl-44178-v7/tokenizer_config.json
chaiml-test-configurabl-44178-v7-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-test-configurabl-44178-v7/tokenizer.json
chaiml-test-configurabl-44178-v7-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-test-configurabl-44178-v7/flywheel_model.0.safetensors
Job chaiml-test-configurabl-44178-v7-mkmlizer completed after 115.3s with status: succeeded
Stopping job with name chaiml-test-configurabl-44178-v7-mkmlizer
Pipeline stage MKMLizer completed in 115.79s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-test-configurabl-44178-v7
Waiting for inference service chaiml-test-configurabl-44178-v7 to be ready
Failed to get response for submission rirv938-viral-20k-624-b_93859_v1: HTTPConnectionPool(host='rirv938-viral-20k-624-b-93859-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-sma_53415_v100: HTTPConnectionPool(host='mistralai-mistral-sma-53415-v100-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission rirv938-viral-20k-624-b_93859_v1: HTTPConnectionPool(host='rirv938-viral-20k-624-b-93859-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission rirv938-viral-20k-624-b_93859_v1: HTTPConnectionPool(host='rirv938-viral-20k-624-b-93859-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-sma_53415_v100: HTTPConnectionPool(host='mistralai-mistral-sma-53415-v100-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-sma_53415_v100: HTTPConnectionPool(host='mistralai-mistral-sma-53415-v100-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-sma_53415_v100: HTTPConnectionPool(host='mistralai-mistral-sma-53415-v100-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission rirv938-viral-20k-624-b_93859_v1: HTTPConnectionPool(host='rirv938-viral-20k-624-b-93859-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission rirv938-viral-20k-624-b_93859_v1: HTTPConnectionPool(host='rirv938-viral-20k-624-b-93859-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service chaiml-test-configurabl-44178-v7 ready after 341.5960512161255s
Pipeline stage MKMLDeployer completed in 342.09s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.08125638961792s
Received healthy response to inference request in 1.0772955417633057s
dial tcp 127.0.0.1:8080: connect: connection refused

Received unhealthy response to inference request!
Received healthy response to inference request in 1.3646130561828613s
Received healthy response to inference request in 1.7445483207702637s
5 requests
1 failed requests
5th percentile: 0.3162510395050049
10th percentile: 0.5065121650695801
20th percentile: 0.8870344161987305
30th percentile: 1.0780877113342284
40th percentile: 1.0796720504760742
50th percentile: 1.08125638961792
60th percentile: 1.1945990562438964
70th percentile: 1.307941722869873
80th percentile: 1.440600109100342
90th percentile: 1.5925742149353028
95th percentile: 1.668561267852783
99th percentile: 1.7293509101867675
mean time: 1.078740644454956
%s, retrying in %s seconds...
Received healthy response to inference request in 0.8311500549316406s
Received healthy response to inference request in 1.0524232387542725s
Received healthy response to inference request in 0.6422631740570068s
Received healthy response to inference request in 0.887763500213623s
Received healthy response to inference request in 1.0415172576904297s
5 requests
0 failed requests
5th percentile: 0.6800405502319335
10th percentile: 0.7178179264068604
20th percentile: 0.7933726787567139
30th percentile: 0.8424727439880371
40th percentile: 0.8651181221008301
50th percentile: 0.887763500213623
60th percentile: 0.9492650032043457
70th percentile: 1.0107665061950684
80th percentile: 1.0436984539031982
90th percentile: 1.0480608463287353
95th percentile: 1.050242042541504
99th percentile: 1.0519869995117188
mean time: 0.8910234451293946
Pipeline stage StressChecker completed in 12.28s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.94s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.63s
Shutdown handler de-registered
chaiml-test-configurabl_44178_v7 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2507.27s
Shutdown handler de-registered
chaiml-test-configurabl_44178_v7 status is now inactive due to auto deactivation removed underperforming models
chaiml-test-configurabl_44178_v7 status is now torndown due to DeploymentManager action
chaiml-test-configurabl_44178_v7 status is now torndown due to DeploymentManager action
chaiml-test-configurabl_44178_v7 status is now torndown due to DeploymentManager action
ChatRequest
Generation Params
Prompt Formatter
Chat History
ChatMessage 1