rinen0721-mistral12b-1117

developer_uid: rinen0721

submission_id: rinen0721-mistral12b-1117_v2

model_name: rinen0721-mistral12b-1117_v1

model_group: rinen0721/Mistral12b-111

status: inactive

timestamp: 2024-11-20T03:25:29+00:00

num_battles: 9647

num_wins: 4592

celo_rating: 1232.99

family_friendly_score: 0.5682

family_friendly_standard_error: 0.0070049805138915275

submission_type: basic

model_repo: rinen0721/Mistral12b-1117

model_architecture: MistralForCausalLM

model_num_parameters: 12772070400.0

best_of: 8

max_input_tokens: 1024

max_output_tokens: 64

latencies: [{'batch_size': 1, 'throughput': 0.6252971376388523, 'latency_mean': 1.5991445302963256, 'latency_p50': 1.5901962518692017, 'latency_p90': 1.7678299427032471}, {'batch_size': 3, 'throughput': 1.147745208895531, 'latency_mean': 2.6012260341644287, 'latency_p50': 2.621272087097168, 'latency_p90': 2.8435758113861085}, {'batch_size': 5, 'throughput': 1.4157523771085208, 'latency_mean': 3.514569331407547, 'latency_p50': 3.5027483701705933, 'latency_p90': 3.8947550296783446}, {'batch_size': 6, 'throughput': 1.4904798457994652, 'latency_mean': 3.997028956413269, 'latency_p50': 4.00516951084137, 'latency_p90': 4.510163259506226}, {'batch_size': 8, 'throughput': 1.5577018211490092, 'latency_mean': 5.0924727404117585, 'latency_p50': 5.0943790674209595, 'latency_p90': 5.681336688995361}, {'batch_size': 10, 'throughput': 1.5895236563722126, 'latency_mean': 6.240562995672226, 'latency_p50': 6.300407290458679, 'latency_p90': 7.084461402893067}]

gpu_counts: {'NVIDIA RTX A5000': 1}

display_name: rinen0721-mistral12b-1117_v1

is_internal_developer: False

language_model: rinen0721/Mistral12b-1117

model_size: 13B

ranking_group: single

throughput_3p7s: 1.46

us_pacific_date: 2024-11-19

win_ratio: 0.4760029024567223

generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rinen0721-mistral12b-1117-v2-mkmlizer
Waiting for job on rinen0721-mistral12b-1117-v2-mkmlizer to finish
rinen0721-mistral12b-1117-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rinen0721-mistral12b-1117-v2-mkmlizer: ║     _____            __           __                                ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║       /___/                                                         ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║                                                                     ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║  Version: 0.11.12                                                   ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║  https://mk1.ai                                                     ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║                                                                     ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║  belonging to:                                                      ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║                                                                     ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║  Chai Research Corp.                                                ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
rinen0721-mistral12b-1117-v2-mkmlizer: ║                                                                     ║
rinen0721-mistral12b-1117-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
rinen0721-mistral12b-1117-v2-mkmlizer: Downloaded to shared memory in 34.893s
rinen0721-mistral12b-1117-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpgxj0c8__, device:0
rinen0721-mistral12b-1117-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rinen0721-mistral12b-1117-v2-mkmlizer: quantized model in 35.771s
rinen0721-mistral12b-1117-v2-mkmlizer: Processed model rinen0721/Mistral12b-1117 in 70.665s
rinen0721-mistral12b-1117-v2-mkmlizer: creating bucket guanaco-mkml-models
rinen0721-mistral12b-1117-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rinen0721-mistral12b-1117-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rinen0721-mistral12b-1117-v2
rinen0721-mistral12b-1117-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rinen0721-mistral12b-1117-v2/special_tokens_map.json
rinen0721-mistral12b-1117-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rinen0721-mistral12b-1117-v2/config.json
rinen0721-mistral12b-1117-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rinen0721-mistral12b-1117-v2/tokenizer_config.json
rinen0721-mistral12b-1117-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/rinen0721-mistral12b-1117-v2/tokenizer.json
rinen0721-mistral12b-1117-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rinen0721-mistral12b-1117-v2/flywheel_model.0.safetensors
rinen0721-mistral12b-1117-v2-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 30.04it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 50.38it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 47.56it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:07, 46.22it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 52.19it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:06, 48.29it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:06, 46.09it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:06, 50.41it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 46.56it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 36.00it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:08, 36.34it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 40.91it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:06, 41.56it/s]
Loading 0:  23%|██▎       | 83/363 [00:01<00:06, 42.33it/s]
Loading 0:  25%|██▍       | 90/363 [00:02<00:05, 47.07it/s]
Loading 0:  26%|██▋       | 96/363 [00:02<00:05, 44.53it/s]
Loading 0:  28%|██▊       | 101/363 [00:02<00:06, 41.45it/s]
Loading 0:  29%|██▉       | 106/363 [00:02<00:06, 42.18it/s]
Loading 0:  31%|███       | 112/363 [00:02<00:05, 44.51it/s]
Loading 0:  32%|███▏      | 117/363 [00:02<00:05, 41.21it/s]
Loading 0:  34%|███▎      | 122/363 [00:02<00:05, 41.73it/s]
Loading 0:  35%|███▍      | 127/363 [00:03<00:06, 36.01it/s]
Loading 0:  37%|███▋      | 135/363 [00:03<00:05, 44.22it/s]
Loading 0:  39%|███▉      | 141/363 [00:03<00:05, 42.23it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 32.80it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:06, 32.62it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 38.11it/s]
Loading 0:  44%|████▍     | 161/363 [00:03<00:05, 39.64it/s]
Loading 0:  46%|████▌     | 166/363 [00:03<00:04, 41.96it/s]
Loading 0:  47%|████▋     | 172/363 [00:04<00:04, 41.44it/s]
Loading 0:  49%|████▉     | 177/363 [00:04<00:04, 39.29it/s]
Loading 0:  50%|█████     | 183/363 [00:04<00:04, 43.80it/s]
Loading 0:  52%|█████▏    | 188/363 [00:04<00:03, 43.86it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:03, 44.95it/s]
Loading 0:  55%|█████▍    | 199/363 [00:04<00:03, 44.65it/s]
Loading 0:  56%|█████▌    | 204/363 [00:04<00:03, 44.09it/s]
Loading 0:  58%|█████▊    | 211/363 [00:04<00:03, 48.18it/s]
Loading 0:  60%|█████▉    | 217/363 [00:05<00:03, 46.64it/s]
Loading 0:  61%|██████▏   | 223/363 [00:05<00:03, 36.67it/s]
Loading 0:  63%|██████▎   | 228/363 [00:05<00:03, 36.50it/s]
Loading 0:  64%|██████▍   | 232/363 [00:05<00:03, 36.59it/s]
Loading 0:  65%|██████▌   | 237/363 [00:05<00:03, 39.68it/s]
Loading 0:  67%|██████▋   | 242/363 [00:05<00:03, 39.49it/s]
Loading 0:  68%|██████▊   | 247/363 [00:05<00:02, 41.03it/s]
Loading 0:  69%|██████▉   | 252/363 [00:06<00:02, 43.33it/s]
Loading 0:  71%|███████   | 257/363 [00:06<00:02, 38.39it/s]
Loading 0:  73%|███████▎  | 265/363 [00:06<00:02, 46.71it/s]
Loading 0:  75%|███████▍  | 271/363 [00:06<00:02, 45.76it/s]
Loading 0:  76%|███████▌  | 276/363 [00:06<00:01, 44.06it/s]
Loading 0:  78%|███████▊  | 282/363 [00:06<00:01, 46.69it/s]
Loading 0:  79%|███████▉  | 287/363 [00:06<00:01, 44.23it/s]
Loading 0:  80%|████████  | 292/363 [00:06<00:01, 44.88it/s]
Loading 0:  82%|████████▏ | 297/363 [00:07<00:01, 45.94it/s]
Loading 0:  83%|████████▎ | 302/363 [00:07<00:01, 46.34it/s]
Loading 0:  85%|████████▍ | 307/363 [00:13<00:23,  2.40it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:15,  3.33it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:07,  5.41it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:05,  7.36it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:03,  9.46it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 13.47it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:01, 17.10it/s]
Loading 0:  96%|█████████▋| 350/363 [00:14<00:00, 21.31it/s]
Loading 0:  98%|█████████▊| 356/363 [00:14<00:00, 26.35it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 30.11it/s]
                                                            
Job rinen0721-mistral12b-1117-v2-mkmlizer completed after 95.14s with status: succeeded
Stopping job with name rinen0721-mistral12b-1117-v2-mkmlizer
Pipeline stage MKMLizer completed in 95.67s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.18s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rinen0721-mistral12b-1117-v2
Waiting for inference service rinen0721-mistral12b-1117-v2 to be ready
Inference service rinen0721-mistral12b-1117-v2 ready after 211.35632467269897s
Pipeline stage MKMLDeployer completed in 212.00s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.0496840476989746s
Received healthy response to inference request in 2.014893054962158s
Received healthy response to inference request in 2.2169485092163086s
Received healthy response to inference request in 1.702925205230713s
5 requests
1 failed requests
5th percentile: 1.7653187751770019
10th percentile: 1.827712345123291
20th percentile: 1.9524994850158692
30th percentile: 2.0218512535095217
40th percentile: 2.035767650604248
50th percentile: 2.0496840476989746
60th percentile: 2.116589832305908
70th percentile: 2.183495616912842
80th percentile: 5.8008327484130895
90th percentile: 12.968601226806642
95th percentile: 16.552485466003414
99th percentile: 19.41959285736084
mean time: 5.62416410446167
%s, retrying in %s seconds...
Received healthy response to inference request in 1.6748237609863281s
Received healthy response to inference request in 1.611271619796753s
Received healthy response to inference request in 2.440702199935913s
Received healthy response to inference request in 1.6923635005950928s
Received healthy response to inference request in 1.7530403137207031s
5 requests
0 failed requests
5th percentile: 1.623982048034668
10th percentile: 1.636692476272583
20th percentile: 1.662113332748413
30th percentile: 1.6783317089080811
40th percentile: 1.685347604751587
50th percentile: 1.6923635005950928
60th percentile: 1.7166342258453369
70th percentile: 1.740904951095581
80th percentile: 1.8905726909637453
90th percentile: 2.1656374454498293
95th percentile: 2.3031698226928707
99th percentile: 2.413195724487305
mean time: 1.834440279006958
Pipeline stage StressChecker completed in 40.25s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.37s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.29s
Shutdown handler de-registered
rinen0721-mistral12b-1117_v2 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2606.35s
Shutdown handler de-registered
rinen0721-mistral12b-1117_v2 status is now inactive due to auto deactivation removed underperforming models