valentin87-xmen-rpg-id-ep2s

developer_uid: valentin87

submission_id: valentin87-xmen-rpg-id-ep2s_v1

model_name: valentin87-xmen-rpg-id-ep2s_v1

model_group: valentin87/xmen_rpg_id_e

status: torndown

timestamp: 2024-09-15T21:07:31+00:00

num_battles: 11246

num_wins: 5574

celo_rating: 1248.82

family_friendly_score: 0.0

submission_type: basic

model_repo: valentin87/xmen_rpg_id_eP2s

model_architecture: MistralForCausalLM

model_num_parameters: 12772070400.0

best_of: 8

max_input_tokens: 512

max_output_tokens: 64

latencies: [{'batch_size': 1, 'throughput': 0.6919862431193883, 'latency_mean': 1.4450165331363678, 'latency_p50': 1.447206974029541, 'latency_p90': 1.6087213516235352}, {'batch_size': 3, 'throughput': 1.3192367575174855, 'latency_mean': 2.272406930923462, 'latency_p50': 2.2627604007720947, 'latency_p90': 2.5259056806564333}, {'batch_size': 5, 'throughput': 1.5794535562478684, 'latency_mean': 3.1602878952026368, 'latency_p50': 3.1721527576446533, 'latency_p90': 3.4845013856887816}, {'batch_size': 6, 'throughput': 1.6171873846610916, 'latency_mean': 3.687465491294861, 'latency_p50': 3.6946959495544434, 'latency_p90': 4.2236439943313595}, {'batch_size': 8, 'throughput': 1.589471798746861, 'latency_mean': 5.006010060310364, 'latency_p50': 5.0596946477890015, 'latency_p90': 5.702545189857483}, {'batch_size': 10, 'throughput': 1.5397336002403044, 'latency_mean': 6.440935448408127, 'latency_p50': 6.519960522651672, 'latency_p90': 7.260689043998719}]

gpu_counts: {'NVIDIA RTX A5000': 1}

display_name: valentin87-xmen-rpg-id-ep2s_v1

is_internal_developer: True

language_model: valentin87/xmen_rpg_id_eP2s

model_size: 13B

ranking_group: single

throughput_3p7s: 1.63

us_pacific_date: 2024-09-15

win_ratio: 0.495642895251645

generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 100, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name valentin87-xmen-rpg-id-ep2s-v1-mkmlizer
Waiting for job on valentin87-xmen-rpg-id-ep2s-v1-mkmlizer to finish
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║     _____            __           __                                ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║       /___/                                                         ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║                                                                     ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║  Version: 0.10.1                                                    ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║  https://mk1.ai                                                     ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║                                                                     ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║  belonging to:                                                      ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║                                                                     ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║  Chai Research Corp.                                                ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ║                                                                     ║
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: Downloaded to shared memory in 54.180s
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpqfhanvz1, device:0
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission blend_hokok_2024-09-09: ('http://neversleep-noromaid-v0-8068-v150-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: quantized model in 35.703s
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: Processed model valentin87/xmen_rpg_id_eP2s in 89.883s
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: creating bucket guanaco-mkml-models
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/valentin87-xmen-rpg-id-ep2s-v1
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/valentin87-xmen-rpg-id-ep2s-v1/config.json
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/valentin87-xmen-rpg-id-ep2s-v1/special_tokens_map.json
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/valentin87-xmen-rpg-id-ep2s-v1/tokenizer_config.json
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/valentin87-xmen-rpg-id-ep2s-v1/tokenizer.json
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/valentin87-xmen-rpg-id-ep2s-v1/flywheel_model.0.safetensors
valentin87-xmen-rpg-id-ep2s-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:12, 28.69it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:07, 49.64it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:08, 41.81it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:08, 41.27it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 47.78it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:07, 45.71it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:07, 45.76it/s]
Loading 0:  14%|█▍        | 50/363 [00:01<00:06, 45.11it/s]
Loading 0:  15%|█▌        | 56/363 [00:01<00:06, 47.53it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 37.26it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:07, 37.31it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:06, 41.91it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:06, 42.07it/s]
Loading 0:  23%|██▎       | 83/363 [00:01<00:06, 42.78it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:05, 46.26it/s]
Loading 0:  26%|██▌       | 94/363 [00:02<00:05, 45.04it/s]
Loading 0:  28%|██▊       | 100/363 [00:02<00:06, 41.67it/s]
Loading 0:  30%|███       | 109/363 [00:02<00:04, 52.69it/s]
Loading 0:  32%|███▏      | 115/363 [00:02<00:05, 48.49it/s]
Loading 0:  33%|███▎      | 121/363 [00:02<00:05, 47.77it/s]
Loading 0:  35%|███▍      | 126/363 [00:02<00:05, 46.42it/s]
Loading 0:  36%|███▌      | 131/363 [00:02<00:04, 46.62it/s]
Loading 0:  37%|███▋      | 136/363 [00:03<00:05, 38.65it/s]
Loading 0:  39%|███▉      | 142/363 [00:03<00:06, 34.93it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 35.82it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:06, 35.15it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 40.51it/s]
Loading 0:  44%|████▍     | 161/363 [00:03<00:04, 41.21it/s]
Loading 0:  46%|████▌     | 166/363 [00:03<00:04, 42.11it/s]
Loading 0:  47%|████▋     | 171/363 [00:04<00:04, 43.20it/s]
Loading 0:  48%|████▊     | 176/363 [00:04<00:05, 34.80it/s]
Loading 0:  50%|█████     | 183/363 [00:04<00:04, 41.53it/s]
Loading 0:  52%|█████▏    | 188/363 [00:04<00:04, 41.13it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:04, 41.29it/s]
Loading 0:  55%|█████▍    | 198/363 [00:04<00:03, 42.46it/s]
Loading 0:  56%|█████▌    | 203/363 [00:04<00:04, 36.01it/s]
Loading 0:  58%|█████▊    | 210/363 [00:04<00:03, 43.32it/s]
Loading 0:  59%|█████▉    | 215/363 [00:05<00:03, 43.76it/s]
Loading 0:  61%|██████    | 220/363 [00:05<00:03, 45.11it/s]
Loading 0:  62%|██████▏   | 225/363 [00:05<00:04, 29.07it/s]
Loading 0:  63%|██████▎   | 230/363 [00:05<00:04, 31.68it/s]
Loading 0:  66%|██████▌   | 238/363 [00:05<00:03, 41.08it/s]
Loading 0:  67%|██████▋   | 244/363 [00:05<00:02, 41.26it/s]
Loading 0:  69%|██████▊   | 249/363 [00:06<00:02, 40.70it/s]
Loading 0:  71%|███████   | 256/363 [00:06<00:02, 45.78it/s]
Loading 0:  72%|███████▏  | 262/363 [00:06<00:02, 45.75it/s]
Loading 0:  74%|███████▎  | 267/363 [00:06<00:02, 44.16it/s]
Loading 0:  75%|███████▌  | 273/363 [00:06<00:01, 47.36it/s]
Loading 0:  77%|███████▋  | 278/363 [00:06<00:01, 47.44it/s]
Loading 0:  78%|███████▊  | 284/363 [00:06<00:01, 43.35it/s]
Loading 0:  80%|████████  | 292/363 [00:06<00:01, 50.45it/s]
Loading 0:  82%|████████▏ | 298/363 [00:07<00:01, 46.86it/s]
Loading 0:  84%|████████▎ | 304/363 [00:13<00:20,  2.92it/s]
Loading 0:  85%|████████▍ | 308/363 [00:13<00:15,  3.65it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:10,  4.64it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:05,  7.47it/s]
Loading 0:  90%|████████▉ | 325/363 [00:14<00:03,  9.60it/s]
Loading 0:  91%|█████████ | 330/363 [00:14<00:02, 11.58it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 17.21it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:00, 21.09it/s]
Loading 0:  96%|█████████▋| 350/363 [00:14<00:00, 25.57it/s]
Loading 0:  98%|█████████▊| 357/363 [00:14<00:00, 29.50it/s]
                                                            
Job valentin87-xmen-rpg-id-ep2s-v1-mkmlizer completed after 116.3s with status: succeeded
Stopping job with name valentin87-xmen-rpg-id-ep2s-v1-mkmlizer
Pipeline stage MKMLizer completed in 117.10s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.09s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service valentin87-xmen-rpg-id-ep2s-v1
Waiting for inference service valentin87-xmen-rpg-id-ep2s-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service valentin87-xmen-rpg-id-ep2s-v1 ready after 160.45840644836426s
Pipeline stage MKMLDeployer completed in 160.87s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.476088762283325s
Received healthy response to inference request in 2.588636636734009s
Failed to get response for submission blend_hokok_2024-09-09: ('http://neversleep-noromaid-v0-8068-v150-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Received healthy response to inference request in 2.604949474334717s
Received healthy response to inference request in 2.7328717708587646s
Received healthy response to inference request in 3.0160703659057617s
5 requests
0 failed requests
5th percentile: 2.498598337173462
10th percentile: 2.5211079120635986
20th percentile: 2.566127061843872
30th percentile: 2.59189920425415
40th percentile: 2.5984243392944335
50th percentile: 2.604949474334717
60th percentile: 2.656118392944336
70th percentile: 2.707287311553955
80th percentile: 2.7895114898681643
90th percentile: 2.902790927886963
95th percentile: 2.959430646896362
99th percentile: 3.004742422103882
mean time: 2.6837234020233156
Pipeline stage StressChecker completed in 26.63s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 6.81s
Shutdown handler de-registered
valentin87-xmen-rpg-id-ep2s_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service valentin87-xmen-rpg-id-ep2s-v1-profiler
Waiting for inference service valentin87-xmen-rpg-id-ep2s-v1-profiler to be ready
Inference service valentin87-xmen-rpg-id-ep2s-v1-profiler ready after 170.39362478256226s
Pipeline stage MKMLProfilerDeployer completed in 173.71s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/valentin87-xmen-rpg-766622b8afd05117896ea5c1bc3db0a5-deplo7c2dt:/code/chaiverse_profiler_1726434993 --namespace tenant-chaiml-guanaco
kubectl exec -it valentin87-xmen-rpg-766622b8afd05117896ea5c1bc3db0a5-deplo7c2dt --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1726434993 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1726434993/summary.json'
kubectl exec -it valentin87-xmen-rpg-766622b8afd05117896ea5c1bc3db0a5-deplo7c2dt --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1726434993/summary.json'
Pipeline stage MKMLProfilerRunner completed in 956.54s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service valentin87-xmen-rpg-id-ep2s-v1-profiler is running
Tearing down inference service valentin87-xmen-rpg-id-ep2s-v1-profiler
Service valentin87-xmen-rpg-id-ep2s-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 4.44s
Shutdown handler de-registered
valentin87-xmen-rpg-id-ep2s_v1 status is now inactive due to auto deactivation removed underperforming models
valentin87-xmen-rpg-id-ep2s_v1 status is now torndown due to DeploymentManager action