developer_uid: Hastagaras
submission_id: hastagaras-test12b_v1
model_name: test
model_group: Hastagaras/test12b
status: inactive
timestamp: 2024-11-20T14:45:16+00:00
num_battles: 8610
num_wins: 4275
celo_rating: 1248.88
family_friendly_score: 0.5680000000000001
family_friendly_standard_error: 0.007005369369276684
submission_type: basic
model_repo: Hastagaras/test12b
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
latencies: [{'batch_size': 1, 'throughput': 0.6175911018174984, 'latency_mean': 1.6190980410575866, 'latency_p50': 1.6204264163970947, 'latency_p90': 1.7941229343414307}, {'batch_size': 3, 'throughput': 1.1416656504065565, 'latency_mean': 2.6263377976417543, 'latency_p50': 2.6496167182922363, 'latency_p90': 2.8964445352554318}, {'batch_size': 5, 'throughput': 1.3755462605780395, 'latency_mean': 3.6168392908573153, 'latency_p50': 3.6358120441436768, 'latency_p90': 4.105085778236389}, {'batch_size': 6, 'throughput': 1.461418131234366, 'latency_mean': 4.087517476081848, 'latency_p50': 4.0863319635391235, 'latency_p90': 4.624371409416199}, {'batch_size': 8, 'throughput': 1.5154984563902538, 'latency_mean': 5.242141387462616, 'latency_p50': 5.282877445220947, 'latency_p90': 5.831767702102661}, {'batch_size': 10, 'throughput': 1.581965807646248, 'latency_mean': 6.27579606294632, 'latency_p50': 6.284051537513733, 'latency_p90': 7.154863858222961}]
gpu_counts: {'NVIDIA RTX A5000': 1}
display_name: test
is_internal_developer: False
language_model: Hastagaras/test12b
model_size: 13B
ranking_group: single
throughput_3p7s: 1.4
us_pacific_date: 2024-11-20
win_ratio: 0.4965156794425087
generation_params: {'temperature': 0.9, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 100, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': "[INST]system: You're {bot_name}\n{memory}\n", 'prompt_template': '{prompt}[/INST]', 'bot_template': '{bot_name}: {message}</s>', 'user_template': '[INST]{user_name}: {message}[/INST]', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name hastagaras-test12b-v1-mkmlizer
Waiting for job on hastagaras-test12b-v1-mkmlizer to finish
hastagaras-test12b-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
hastagaras-test12b-v1-mkmlizer: ║     _____            __           __                                ║
hastagaras-test12b-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
hastagaras-test12b-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
hastagaras-test12b-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
hastagaras-test12b-v1-mkmlizer: ║       /___/                                                         ║
hastagaras-test12b-v1-mkmlizer: ║                                                                     ║
hastagaras-test12b-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
hastagaras-test12b-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
hastagaras-test12b-v1-mkmlizer: ║  https://mk1.ai                                                     ║
hastagaras-test12b-v1-mkmlizer: ║                                                                     ║
hastagaras-test12b-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
hastagaras-test12b-v1-mkmlizer: ║  belonging to:                                                      ║
hastagaras-test12b-v1-mkmlizer: ║                                                                     ║
hastagaras-test12b-v1-mkmlizer: ║  Chai Research Corp.                                                ║
hastagaras-test12b-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
hastagaras-test12b-v1-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
hastagaras-test12b-v1-mkmlizer: ║                                                                     ║
hastagaras-test12b-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
hastagaras-test12b-v1-mkmlizer: Downloaded to shared memory in 49.381s
hastagaras-test12b-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpljag448z, device:0
hastagaras-test12b-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
hastagaras-test12b-v1-mkmlizer: quantized model in 35.991s
Connection pool is full, discarding connection: %s. Connection pool size: %s
hastagaras-test12b-v1-mkmlizer: Processed model Hastagaras/test12b in 85.373s
hastagaras-test12b-v1-mkmlizer: creating bucket guanaco-mkml-models
Connection pool is full, discarding connection: %s. Connection pool size: %s
hastagaras-test12b-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
hastagaras-test12b-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/hastagaras-test12b-v1
hastagaras-test12b-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/hastagaras-test12b-v1/config.json
hastagaras-test12b-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/hastagaras-test12b-v1/special_tokens_map.json
hastagaras-test12b-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/hastagaras-test12b-v1/tokenizer_config.json
hastagaras-test12b-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/hastagaras-test12b-v1/tokenizer.json
Failed to get response for submission blend_fufer_2024-11-13: ('http://chaiml-llama-8b-pairwis-8189-v27-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:35598->127.0.0.1:8080: read: connection reset by peer\n')
hastagaras-test12b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/hastagaras-test12b-v1/flywheel_model.0.safetensors
hastagaras-test12b-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 31.30it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 53.09it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 48.41it/s]
Loading 0:   7%|▋         | 25/363 [00:00<00:06, 48.42it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 50.05it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:06, 47.00it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:07, 45.09it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:06, 50.18it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 47.06it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 35.57it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:08, 36.04it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 40.39it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:07, 40.31it/s]
Loading 0:  23%|██▎       | 83/363 [00:01<00:07, 39.55it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:06, 43.62it/s]
Loading 0:  26%|██▌       | 94/363 [00:02<00:06, 42.59it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:06, 41.30it/s]
Loading 0:  29%|██▉       | 105/363 [00:02<00:06, 39.01it/s]
Loading 0:  30%|███       | 110/363 [00:02<00:06, 41.06it/s]
Loading 0:  32%|███▏      | 115/363 [00:02<00:05, 42.87it/s]
Loading 0:  33%|███▎      | 120/363 [00:02<00:05, 41.06it/s]
Loading 0:  35%|███▍      | 126/363 [00:02<00:05, 44.14it/s]
Loading 0:  36%|███▋      | 132/363 [00:03<00:05, 42.80it/s]
Loading 0:  38%|███▊      | 137/363 [00:03<00:05, 41.28it/s]
Loading 0:  39%|███▉      | 142/363 [00:03<00:06, 32.31it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 33.07it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:06, 30.78it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 36.21it/s]
Loading 0:  44%|████▍     | 160/363 [00:03<00:05, 36.61it/s]
Loading 0:  46%|████▌     | 166/363 [00:04<00:04, 40.67it/s]
Loading 0:  47%|████▋     | 172/363 [00:04<00:04, 40.75it/s]
Loading 0:  49%|████▉     | 177/363 [00:04<00:04, 40.78it/s]
Loading 0:  51%|█████     | 184/363 [00:04<00:03, 45.75it/s]
Loading 0:  52%|█████▏    | 190/363 [00:04<00:03, 43.99it/s]
Loading 0:  54%|█████▎    | 195/363 [00:04<00:04, 41.22it/s]
Loading 0:  55%|█████▌    | 201/363 [00:04<00:03, 43.77it/s]
Loading 0:  57%|█████▋    | 206/363 [00:05<00:03, 41.62it/s]
Loading 0:  58%|█████▊    | 211/363 [00:05<00:03, 40.75it/s]
Loading 0:  60%|█████▉    | 216/363 [00:05<00:03, 41.32it/s]
Loading 0:  61%|██████    | 221/363 [00:05<00:03, 41.83it/s]
Loading 0:  62%|██████▏   | 226/363 [00:05<00:05, 26.40it/s]
Loading 0:  63%|██████▎   | 230/363 [00:05<00:04, 26.83it/s]
Loading 0:  65%|██████▌   | 237/363 [00:05<00:03, 35.16it/s]
Loading 0:  67%|██████▋   | 242/363 [00:06<00:03, 38.00it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:02, 40.42it/s]
Loading 0:  70%|██████▉   | 253/363 [00:06<00:02, 40.16it/s]
Loading 0:  71%|███████   | 258/363 [00:06<00:02, 39.42it/s]
Loading 0:  73%|███████▎  | 265/363 [00:06<00:02, 44.98it/s]
Loading 0:  75%|███████▍  | 271/363 [00:06<00:02, 44.38it/s]
Loading 0:  76%|███████▌  | 276/363 [00:06<00:02, 42.53it/s]
Loading 0:  78%|███████▊  | 282/363 [00:06<00:01, 46.80it/s]
Loading 0:  79%|███████▉  | 287/363 [00:07<00:01, 44.63it/s]
Loading 0:  80%|████████  | 292/363 [00:07<00:01, 45.02it/s]
Loading 0:  82%|████████▏ | 297/363 [00:07<00:01, 46.18it/s]
Loading 0:  83%|████████▎ | 303/363 [00:07<00:01, 44.92it/s]
Loading 0:  85%|████████▍ | 308/363 [00:14<00:21,  2.51it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:15,  3.24it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:08,  5.34it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:05,  7.25it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:03,  9.22it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 13.09it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:01, 16.40it/s]
Loading 0:  96%|█████████▌| 349/363 [00:15<00:00, 19.53it/s]
Loading 0:  98%|█████████▊| 356/363 [00:15<00:00, 25.75it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 28.61it/s]
                                                            

Job hastagaras-test12b-v1-mkmlizer completed after 106.14s with status: succeeded
Stopping job with name hastagaras-test12b-v1-mkmlizer
Pipeline stage MKMLizer completed in 106.80s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.23s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service hastagaras-test12b-v1
Waiting for inference service hastagaras-test12b-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service hastagaras-test12b-v1 ready after 281.3969306945801s
Pipeline stage MKMLDeployer completed in 282.16s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.249411106109619s
Received healthy response to inference request in 1.9317772388458252s
Received healthy response to inference request in 1.8470633029937744s
Received healthy response to inference request in 1.959460973739624s
Received healthy response to inference request in 1.8265743255615234s
5 requests
0 failed requests
5th percentile: 1.8306721210479737
10th percentile: 1.8347699165344238
20th percentile: 1.8429655075073241
30th percentile: 1.8640060901641846
40th percentile: 1.8978916645050048
50th percentile: 1.9317772388458252
60th percentile: 1.9428507328033446
70th percentile: 1.9539242267608643
80th percentile: 2.0174510002136232
90th percentile: 2.133431053161621
95th percentile: 2.19142107963562
99th percentile: 2.237813100814819
mean time: 1.9628573894500732
Pipeline stage StressChecker completed in 11.58s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 3.40s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.44s
Shutdown handler de-registered
hastagaras-test12b_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service hastagaras-test12b-v1-profiler
Waiting for inference service hastagaras-test12b-v1-profiler to be ready
Inference service hastagaras-test12b-v1-profiler ready after 220.5020887851715s
Pipeline stage MKMLProfilerDeployer completed in 220.92s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/hastagaras-test12b-v1-profiler-predictor-00001-deployment-h8pgm:/code/chaiverse_profiler_1732114585 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/hastagaras-test12b-v1-profiler-predictor-00001-deployment-h8pgm:/code/chaiverse_profiler_1732114585 --namespace tenant-chaiml-guanaco
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/hastagaras-test12b-v1-profiler-predictor-00001-deployment-h8pgm:/code/chaiverse_profiler_1732114585 --namespace tenant-chaiml-guanaco
clean up pipeline due to error=ISVCScriptError("[Errno 8] Exec format error: '/bin/sh'")
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service hastagaras-test12b-v1-profiler is running
Tearing down inference service hastagaras-test12b-v1-profiler
Service hastagaras-test12b-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.14s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service hastagaras-test12b-v1-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 2.22s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service hastagaras-test12b-v1-profiler
Waiting for inference service hastagaras-test12b-v1-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2590.89s
Shutdown handler de-registered
hastagaras-test12b_v1 status is now inactive due to auto deactivation removed underperforming models