zonemercy-lexical-nemov8_5966

submission_id: zonemercy-lexical-nemov8_5966_v9

developer_uid: chai_backend_admin

best_of: 8

celo_rating: 1254.92

display_name: zonemercy-lexical-nemov8_5966_v9

family_friendly_score: 0.5508083235122244

family_friendly_standard_error: 0.002328313121624354

formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': 'Bot: {message}\n', 'user_template': 'User: {message}\n', 'response_template': 'Bot:', 'truncate_by_message': False}

generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '###', 'Bot:', 'User:', 'You:', '<|im_end|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

gpu_counts: {'NVIDIA RTX A5000': 1}

is_internal_developer: True

language_model: zonemercy/Lexical-Nemov8-1k1e5

latencies: [{'batch_size': 1, 'throughput': 0.6075860385764573, 'latency_mean': 1.6457632970809937, 'latency_p50': 1.6525061130523682, 'latency_p90': 1.809170389175415}, {'batch_size': 3, 'throughput': 1.0789403124115602, 'latency_mean': 2.773970226049423, 'latency_p50': 2.787737727165222, 'latency_p90': 3.068733477592468}, {'batch_size': 5, 'throughput': 1.2344696690410222, 'latency_mean': 4.041068782806397, 'latency_p50': 4.0542237758636475, 'latency_p90': 4.489140963554382}, {'batch_size': 6, 'throughput': 1.2790121126368965, 'latency_mean': 4.677124183177948, 'latency_p50': 4.692606091499329, 'latency_p90': 5.1760409593582155}, {'batch_size': 8, 'throughput': 1.2505401023376905, 'latency_mean': 6.3679009521007535, 'latency_p50': 6.356367707252502, 'latency_p90': 7.293629932403564}, {'batch_size': 10, 'throughput': 1.213392556388075, 'latency_mean': 8.19394871711731, 'latency_p50': 8.258911967277527, 'latency_p90': 9.25556604862213}]

max_input_tokens: 1024

max_output_tokens: 64

model_architecture: MistralForCausalLM

model_group: zonemercy/Lexical-Nemov8

model_name: zonemercy-lexical-nemov8_5966_v9

model_num_parameters: 12772070400.0

model_repo: zonemercy/Lexical-Nemov8-1k1e5

model_size: 13B

num_battles: 4426659

num_wins: 2225230

ranking_group: single

status: inactive

submission_type: basic

throughput_3p7s: 1.21

timestamp: 2024-09-18T20:51:30+00:00

us_pacific_date: 2024-09-18

win_ratio: 0.502688370619919

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-lexical-nemov8-5966-v9-mkmlizer
Waiting for job on zonemercy-lexical-nemov8-5966-v9-mkmlizer to finish
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║     _____            __           __                                ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║       /___/                                                         ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║                                                                     ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║  Version: 0.10.1                                                    ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║                                                                     ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║  belonging to:                                                      ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║                                                                     ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ║                                                                     ║
zonemercy-lexical-nemov8-5966-v9-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
zonemercy-lexical-nemov8-5966-v9-mkmlizer: Downloaded to shared memory in 91.646s
zonemercy-lexical-nemov8-5966-v9-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp6fav33kb, device:0
zonemercy-lexical-nemov8-5966-v9-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-lexical-nemov8-5966-v9-mkmlizer: quantized model in 39.313s
zonemercy-lexical-nemov8-5966-v9-mkmlizer: Processed model zonemercy/Lexical-Nemov8-1k1e5 in 130.959s
zonemercy-lexical-nemov8-5966-v9-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-lexical-nemov8-5966-v9-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-lexical-nemov8-5966-v9-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-lexical-nemov8-5966-v9
zonemercy-lexical-nemov8-5966-v9-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-lexical-nemov8-5966-v9/config.json
zonemercy-lexical-nemov8-5966-v9-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-lexical-nemov8-5966-v9/special_tokens_map.json
zonemercy-lexical-nemov8-5966-v9-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-lexical-nemov8-5966-v9/tokenizer_config.json
zonemercy-lexical-nemov8-5966-v9-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-lexical-nemov8-5966-v9/tokenizer.json
zonemercy-lexical-nemov8-5966-v9-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-lexical-nemov8-5966-v9/flywheel_model.0.safetensors
zonemercy-lexical-nemov8-5966-v9-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:14, 24.08it/s]
Loading 0:   3%|▎         | 10/363 [00:00<00:11, 30.71it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:12, 27.22it/s]
Loading 0:   6%|▌         | 21/363 [00:00<00:08, 38.75it/s]
Loading 0:   7%|▋         | 26/363 [00:01<00:14, 23.13it/s]
Loading 0:   9%|▊         | 31/363 [00:01<00:11, 27.97it/s]
Loading 0:  10%|▉         | 35/363 [00:01<00:11, 29.77it/s]
Loading 0:  11%|█         | 39/363 [00:01<00:10, 31.65it/s]
Loading 0:  12%|█▏        | 43/363 [00:01<00:10, 30.87it/s]
Loading 0:  13%|█▎        | 48/363 [00:01<00:09, 33.31it/s]
Loading 0:  14%|█▍        | 52/363 [00:01<00:09, 31.96it/s]
Loading 0:  15%|█▌        | 56/363 [00:01<00:09, 32.62it/s]
Loading 0:  17%|█▋        | 61/363 [00:02<00:10, 28.88it/s]
Loading 0:  18%|█▊        | 65/363 [00:02<00:11, 26.21it/s]
Loading 0:  20%|█▉        | 71/363 [00:02<00:09, 31.48it/s]
Loading 0:  21%|██        | 75/363 [00:02<00:09, 30.61it/s]
Loading 0:  22%|██▏       | 79/363 [00:02<00:09, 29.99it/s]
Loading 0:  23%|██▎       | 84/363 [00:02<00:08, 33.06it/s]
Loading 0:  24%|██▍       | 88/363 [00:02<00:08, 31.82it/s]
Loading 0:  26%|██▌       | 93/363 [00:03<00:07, 34.27it/s]
Loading 0:  27%|██▋       | 97/363 [00:03<00:08, 32.59it/s]
Loading 0:  28%|██▊       | 101/363 [00:03<00:09, 26.86it/s]
Loading 0:  29%|██▊       | 104/363 [00:03<00:11, 23.28it/s]
Loading 0:  31%|███       | 111/363 [00:03<00:08, 30.76it/s]
Loading 0:  32%|███▏      | 115/363 [00:03<00:08, 29.89it/s]
Loading 0:  33%|███▎      | 120/363 [00:03<00:07, 32.59it/s]
Loading 0:  34%|███▍      | 124/363 [00:04<00:07, 31.28it/s]
Loading 0:  36%|███▌      | 129/363 [00:04<00:06, 33.93it/s]
Loading 0:  37%|███▋      | 133/363 [00:04<00:07, 32.25it/s]
Loading 0:  38%|███▊      | 137/363 [00:04<00:06, 32.44it/s]
Loading 0:  39%|███▉      | 142/363 [00:04<00:07, 28.50it/s]
Loading 0:  40%|████      | 146/363 [00:04<00:07, 28.40it/s]
Loading 0:  41%|████      | 149/363 [00:05<00:08, 24.44it/s]
Loading 0:  43%|████▎     | 156/363 [00:05<00:06, 31.91it/s]
Loading 0:  44%|████▍     | 160/363 [00:05<00:06, 31.09it/s]
Loading 0:  45%|████▌     | 165/363 [00:05<00:05, 33.68it/s]
Loading 0:  47%|████▋     | 169/363 [00:05<00:06, 31.81it/s]
Loading 0:  48%|████▊     | 174/363 [00:05<00:05, 33.77it/s]
Loading 0:  49%|████▉     | 178/363 [00:05<00:05, 32.19it/s]
Loading 0:  50%|█████     | 182/363 [00:06<00:06, 26.85it/s]
Loading 0:  51%|█████     | 185/363 [00:06<00:07, 23.09it/s]
Loading 0:  53%|█████▎    | 192/363 [00:06<00:05, 30.67it/s]
Loading 0:  54%|█████▍    | 196/363 [00:06<00:05, 29.91it/s]
Loading 0:  55%|█████▌    | 201/363 [00:06<00:04, 33.15it/s]
Loading 0:  56%|█████▋    | 205/363 [00:06<00:04, 31.70it/s]
Loading 0:  58%|█████▊    | 210/363 [00:06<00:04, 33.21it/s]
Loading 0:  59%|█████▉    | 214/363 [00:07<00:04, 31.61it/s]
Loading 0:  60%|██████    | 218/363 [00:07<00:04, 32.33it/s]
Loading 0:  61%|██████▏   | 223/363 [00:07<00:04, 28.55it/s]
Loading 0:  63%|██████▎   | 227/363 [00:07<00:04, 28.42it/s]
Loading 0:  63%|██████▎   | 230/363 [00:07<00:05, 24.78it/s]
Loading 0:  65%|██████▌   | 237/363 [00:07<00:03, 31.88it/s]
Loading 0:  66%|██████▋   | 241/363 [00:07<00:03, 30.92it/s]
Loading 0:  68%|██████▊   | 246/363 [00:08<00:03, 33.82it/s]
Loading 0:  69%|██████▉   | 250/363 [00:08<00:03, 32.04it/s]
Loading 0:  70%|███████   | 255/363 [00:08<00:03, 34.45it/s]
Loading 0:  71%|███████▏  | 259/363 [00:08<00:03, 32.46it/s]
Loading 0:  72%|███████▏  | 263/363 [00:08<00:03, 27.10it/s]
Loading 0:  73%|███████▎  | 266/363 [00:08<00:04, 23.49it/s]
Loading 0:  75%|███████▌  | 273/363 [00:09<00:02, 31.06it/s]
Loading 0:  76%|███████▋  | 277/363 [00:09<00:02, 30.15it/s]
Loading 0:  78%|███████▊  | 282/363 [00:09<00:02, 33.08it/s]
Loading 0:  79%|███████▉  | 286/363 [00:09<00:02, 31.01it/s]
Loading 0:  80%|████████  | 291/363 [00:09<00:02, 33.62it/s]
Loading 0:  81%|████████▏ | 295/363 [00:09<00:02, 31.97it/s]
Loading 0:  82%|████████▏ | 299/363 [00:09<00:02, 31.94it/s]
Loading 0:  84%|████████▎ | 304/363 [00:10<00:02, 28.26it/s]
Loading 0:  85%|████████▍ | 307/363 [00:10<00:02, 26.27it/s]
Loading 0:  86%|████████▌ | 311/363 [00:10<00:02, 24.83it/s]
Loading 0:  88%|████████▊ | 318/363 [00:10<00:01, 32.40it/s]
Loading 0:  89%|████████▊ | 322/363 [00:10<00:01, 31.55it/s]
Loading 0:  90%|█████████ | 327/363 [00:10<00:01, 33.72it/s]
Loading 0:  91%|█████████ | 331/363 [00:10<00:01, 31.94it/s]
Loading 0:  93%|█████████▎| 336/363 [00:11<00:00, 34.41it/s]
Loading 0:  94%|█████████▎| 340/363 [00:11<00:00, 32.36it/s]
Loading 0:  95%|█████████▍| 344/363 [00:18<00:09,  2.01it/s]
Loading 0:  96%|█████████▌| 348/363 [00:18<00:05,  2.71it/s]
Loading 0:  97%|█████████▋| 353/363 [00:18<00:02,  3.94it/s]
Loading 0:  98%|█████████▊| 357/363 [00:18<00:01,  5.12it/s]
                                                            
Job zonemercy-lexical-nemov8-5966-v9-mkmlizer completed after 159.3s with status: succeeded
Stopping job with name zonemercy-lexical-nemov8-5966-v9-mkmlizer
Pipeline stage MKMLizer completed in 160.38s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.26s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-lexical-nemov8-5966-v9
Waiting for inference service zonemercy-lexical-nemov8-5966-v9 to be ready
Inference service zonemercy-lexical-nemov8-5966-v9 ready after 30.484674215316772s
Pipeline stage MKMLDeployer completed in 31.67s
run pipeline stage %s
Running pipeline stage StressChecker
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.7893869876861572s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 2.186316728591919s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.7297940254211426s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.8685457706451416s
HTTP Request: %s %s "%s %d %s"
Received healthy response to inference request in 1.8739430904388428s
5 requests
0 failed requests
5th percentile: 1.7575443744659425
10th percentile: 1.7852947235107421
20th percentile: 1.8407954216003417
30th percentile: 1.8696252346038817
40th percentile: 1.8717841625213623
50th percentile: 1.8739430904388428
60th percentile: 1.9988925457000732
70th percentile: 2.1238420009613037
80th percentile: 2.3069307804107666
90th percentile: 2.548158884048462
95th percentile: 2.6687729358673096
99th percentile: 2.7652641773223876
mean time: 2.0895973205566407
Pipeline stage StressChecker completed in 12.53s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 2.08s
Shutdown handler de-registered
zonemercy-lexical-nemov8_5966_v9 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service zonemercy-lexical-nemov8-5966-v9-profiler
Waiting for inference service zonemercy-lexical-nemov8-5966-v9-profiler to be ready
Inference service zonemercy-lexical-nemov8-5966-v9-profiler ready after 300.7017590999603s
Pipeline stage MKMLProfilerDeployer completed in 301.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/zonemercy-lexical-ne6ed8ec8f34e6049fa342076e0618ab1b-deplobb25k:/code/chaiverse_profiler_1726693252 --namespace tenant-chaiml-guanaco
kubectl exec -it zonemercy-lexical-ne6ed8ec8f34e6049fa342076e0618ab1b-deplobb25k --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1726693252 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 64 --summary /code/chaiverse_profiler_1726693252/summary.json'
kubectl exec -it zonemercy-lexical-ne6ed8ec8f34e6049fa342076e0618ab1b-deplobb25k --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1726693252/summary.json'
Pipeline stage MKMLProfilerRunner completed in 1164.19s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service zonemercy-lexical-nemov8-5966-v9-profiler is running
Tearing down inference service zonemercy-lexical-nemov8-5966-v9-profiler
Service zonemercy-lexical-nemov8-5966-v9-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.91s
Shutdown handler de-registered
zonemercy-lexical-nemov8_5966_v9 status is now deployed due to admin request