submission_id: cloudyu-chaiml-nemo-dpo-v9_v1
developer_uid: cloudyu
best_of: 8
celo_rating: 1264.68
display_name: cloudyu-chaiml-nemo-dpo-v9_v1
family_friendly_score: 0.5554
family_friendly_standard_error: 0.007027529295563271
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
gpu_counts: {'NVIDIA RTX A5000': 1}
is_internal_developer: False
language_model: cloudyu/ChaiML-Nemo-DPO-V9
latencies: [{'batch_size': 1, 'throughput': 0.6174712678661568, 'latency_mean': 1.6194457936286926, 'latency_p50': 1.6079362630844116, 'latency_p90': 1.7781689405441283}, {'batch_size': 3, 'throughput': 1.1473942827347627, 'latency_mean': 2.6024447429180144, 'latency_p50': 2.5845041275024414, 'latency_p90': 2.913749766349792}, {'batch_size': 5, 'throughput': 1.382339727574858, 'latency_mean': 3.599818688631058, 'latency_p50': 3.62924325466156, 'latency_p90': 4.059407162666321}, {'batch_size': 6, 'throughput': 1.4668420449875825, 'latency_mean': 4.067454805374146, 'latency_p50': 4.065704584121704, 'latency_p90': 4.5287316799163815}, {'batch_size': 8, 'throughput': 1.5162207385162092, 'latency_mean': 5.235783632993698, 'latency_p50': 5.195023536682129, 'latency_p90': 5.955360007286072}, {'batch_size': 10, 'throughput': 1.5431993423503862, 'latency_mean': 6.424742347002029, 'latency_p50': 6.453848600387573, 'latency_p90': 7.2897117137908936}]
max_input_tokens: 1024
max_output_tokens: 64
model_architecture: MistralForCausalLM
model_group: cloudyu/ChaiML-Nemo-DPO-
model_name: cloudyu-chaiml-nemo-dpo-v9_v1
model_num_parameters: 12772070400.0
model_repo: cloudyu/ChaiML-Nemo-DPO-V9
model_size: 13B
num_battles: 19698
num_wins: 9933
ranking_group: single
status: inactive
submission_type: basic
throughput_3p7s: 1.41
timestamp: 2024-11-24T15:30:18+00:00
us_pacific_date: 2024-11-24
win_ratio: 0.5042643923240938
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer
Waiting for job on cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer to finish
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ _____ __ __ ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ /___/ ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ Version: 0.11.12 ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ https://mk1.ai ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ The license key for the current software has been verified as ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ belonging to: ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ Chai Research Corp. ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ Expiration: 2025-01-15 23:59:59 ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ║ ║
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: Downloaded to shared memory in 49.235s
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp4khtjf50, device:0
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: quantized model in 36.287s
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: Processed model cloudyu/ChaiML-Nemo-DPO-V9 in 85.522s
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: creating bucket guanaco-mkml-models
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/cloudyu-chaiml-nemo-dpo-v9-v1
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/cloudyu-chaiml-nemo-dpo-v9-v1/tokenizer_config.json
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/cloudyu-chaiml-nemo-dpo-v9-v1/tokenizer.json
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/cloudyu-chaiml-nemo-dpo-v9-v1/flywheel_model.0.safetensors
Connection pool is full, discarding connection: %s. Connection pool size: %s
cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer: Loading 0: 0%| | 0/363 [00:00<?, ?it/s] Loading 0: 1%|▏ | 5/363 [00:00<00:13, 26.19it/s] Loading 0: 3%|▎ | 12/363 [00:00<00:08, 43.56it/s] Loading 0: 5%|▍ | 17/363 [00:00<00:08, 42.91it/s] Loading 0: 6%|▌ | 22/363 [00:00<00:07, 43.63it/s] Loading 0: 8%|▊ | 28/363 [00:00<00:08, 41.30it/s] Loading 0: 9%|▉ | 33/363 [00:00<00:08, 40.68it/s] Loading 0: 11%|█ | 40/363 [00:00<00:07, 46.10it/s] Loading 0: 12%|█▏ | 45/363 [00:01<00:06, 46.68it/s] Loading 0: 14%|█▍ | 50/363 [00:01<00:08, 37.50it/s] Loading 0: 15%|█▌ | 56/363 [00:01<00:07, 41.99it/s] Loading 0: 17%|█▋ | 61/363 [00:01<00:09, 31.32it/s] Loading 0: 18%|█▊ | 65/363 [00:01<00:09, 30.95it/s] Loading 0: 20%|█▉ | 72/363 [00:01<00:07, 37.87it/s] Loading 0: 21%|██ | 77/363 [00:01<00:07, 39.63it/s] Loading 0: 23%|██▎ | 82/363 [00:02<00:08, 34.29it/s] Loading 0: 25%|██▍ | 90/363 [00:02<00:06, 42.44it/s] Loading 0: 26%|██▌ | 95/363 [00:02<00:06, 43.85it/s] Loading 0: 28%|██▊ | 100/363 [00:02<00:07, 36.99it/s] Loading 0: 30%|██▉ | 108/363 [00:02<00:05, 46.32it/s] Loading 0: 31%|███▏ | 114/363 [00:02<00:05, 42.22it/s] Loading 0: 33%|███▎ | 119/363 [00:02<00:05, 41.33it/s] Loading 0: 34%|███▍ | 125/363 [00:03<00:05, 44.38it/s] Loading 0: 36%|███▌ | 130/363 [00:03<00:05, 43.06it/s] Loading 0: 37%|███▋ | 135/363 [00:03<00:05, 43.98it/s] Loading 0: 39%|███▊ | 140/363 [00:03<00:05, 44.39it/s] Loading 0: 40%|███▉ | 145/363 [00:03<00:07, 28.22it/s] Loading 0: 41%|████ | 149/363 [00:03<00:07, 28.44it/s] Loading 0: 43%|████▎ | 156/363 [00:04<00:05, 36.38it/s] Loading 0: 44%|████▍ | 161/363 [00:04<00:05, 37.76it/s] Loading 0: 46%|████▌ | 166/363 [00:04<00:04, 40.00it/s] Loading 0: 47%|████▋ | 171/363 [00:04<00:04, 41.67it/s] Loading 0: 48%|████▊ | 176/363 [00:04<00:05, 35.46it/s] Loading 0: 51%|█████ | 184/363 [00:04<00:04, 43.59it/s] Loading 0: 52%|█████▏ | 189/363 [00:04<00:03, 44.99it/s] Loading 0: 53%|█████▎ | 194/363 [00:04<00:04, 37.25it/s] Loading 0: 55%|█████▌ | 201/363 [00:05<00:03, 43.88it/s] Loading 0: 57%|█████▋ | 206/363 [00:05<00:03, 42.95it/s] Loading 0: 58%|█████▊ | 211/363 [00:05<00:03, 43.99it/s] Loading 0: 60%|█████▉ | 216/363 [00:05<00:03, 45.05it/s] Loading 0: 61%|██████ | 221/363 [00:05<00:03, 45.40it/s] Loading 0: 62%|██████▏ | 226/363 [00:05<00:05, 27.14it/s] Loading 0: 63%|██████▎ | 230/363 [00:06<00:04, 27.45it/s] Loading 0: 65%|██████▌ | 237/363 [00:06<00:03, 35.54it/s] Loading 0: 67%|██████▋ | 242/363 [00:06<00:03, 36.73it/s] Loading 0: 68%|██████▊ | 247/363 [00:06<00:03, 38.33it/s] Loading 0: 69%|██████▉ | 252/363 [00:06<00:02, 40.76it/s] Loading 0: 71%|███████ | 257/363 [00:06<00:03, 34.88it/s] Loading 0: 73%|███████▎ | 264/363 [00:06<00:02, 42.36it/s] Loading 0: 74%|███████▍ | 269/363 [00:06<00:02, 40.26it/s] Loading 0: 75%|███████▌ | 274/363 [00:07<00:02, 41.11it/s] Loading 0: 77%|███████▋ | 279/363 [00:07<00:01, 42.82it/s] Loading 0: 78%|███████▊ | 284/363 [00:07<00:02, 36.33it/s] Loading 0: 80%|████████ | 291/363 [00:07<00:01, 43.56it/s] Loading 0: 82%|████████▏ | 296/363 [00:07<00:01, 42.98it/s] Loading 0: 83%|████████▎ | 302/363 [00:07<00:01, 46.98it/s] Loading 0: 85%|████████▍ | 307/363 [00:14<00:22, 2.54it/s] Loading 0: 86%|████████▌ | 312/363 [00:14<00:14, 3.45it/s] Loading 0: 88%|████████▊ | 320/363 [00:14<00:07, 5.50it/s] Loading 0: 90%|████████▉ | 325/363 [00:14<00:05, 7.15it/s] Loading 0: 91%|█████████ | 330/363 [00:15<00:03, 8.93it/s] Loading 0: 93%|█████████▎| 338/363 [00:15<00:01, 13.41it/s] Loading 0: 95%|█████████▍| 344/363 [00:15<00:01, 16.58it/s] Loading 0: 96%|█████████▌| 349/363 [00:15<00:00, 19.52it/s] Loading 0: 98%|█████████▊| 356/363 [00:15<00:00, 25.31it/s] Loading 0: 99%|█████████▉| 361/363 [00:15<00:00, 28.62it/s]
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Job cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer completed after 114.62s with status: succeeded
Stopping job with name cloudyu-chaiml-nemo-dpo-v9-v1-mkmlizer
Pipeline stage MKMLizer completed in 115.10s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service cloudyu-chaiml-nemo-dpo-v9-v1
Waiting for inference service cloudyu-chaiml-nemo-dpo-v9-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service cloudyu-chaiml-nemo-dpo-v9-v1 ready after 120.44383001327515s
Pipeline stage MKMLDeployer completed in 120.98s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.270944595336914s
Received healthy response to inference request in 1.7498230934143066s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 1.9935097694396973s
Received healthy response to inference request in 1.6040928363800049s
Received healthy response to inference request in 2.0988802909851074s
5 requests
0 failed requests
5th percentile: 1.6332388877868653
10th percentile: 1.6623849391937255
20th percentile: 1.7206770420074462
30th percentile: 1.7985604286193848
40th percentile: 1.896035099029541
50th percentile: 1.9935097694396973
60th percentile: 2.0356579780578614
70th percentile: 2.0778061866760256
80th percentile: 2.1332931518554688
90th percentile: 2.2021188735961914
95th percentile: 2.2365317344665527
99th percentile: 2.264062023162842
mean time: 1.943450117111206
Pipeline stage StressChecker completed in 11.10s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.31s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.24s
Shutdown handler de-registered
cloudyu-chaiml-nemo-dpo-v9_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2780.47s
Shutdown handler de-registered
cloudyu-chaiml-nemo-dpo-v9_v1 status is now inactive due to auto deactivation removed underperforming models