riverise-0910-2005-sft

submission_id: riverise-0910-2005-sft_v3

developer_uid: Riverise

best_of: 16

celo_rating: 1249.21

display_name: riverise-0910-2005-sft_v1

family_friendly_score: 0.0

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

generation_params: {'temperature': 1.15, 'top_p': 0.95, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}

gpu_counts: {'NVIDIA RTX A5000': 1}

is_internal_developer: False

language_model: Riverise/0910_2005_sft

latencies: [{'batch_size': 1, 'throughput': 0.8937784130787519, 'latency_mean': 1.1187599301338196, 'latency_p50': 1.1293067932128906, 'latency_p90': 1.22826247215271}, {'batch_size': 4, 'throughput': 1.773123049445759, 'latency_mean': 2.2420759487152098, 'latency_p50': 2.2549245357513428, 'latency_p90': 2.5229142904281616}, {'batch_size': 5, 'throughput': 1.8556613326043776, 'latency_mean': 2.675740343332291, 'latency_p50': 2.6928497552871704, 'latency_p90': 3.002635669708252}, {'batch_size': 8, 'throughput': 1.9795702858849704, 'latency_mean': 4.018562967777252, 'latency_p50': 4.01986837387085, 'latency_p90': 4.585871911048889}, {'batch_size': 10, 'throughput': 2.003215816696514, 'latency_mean': 4.941545494794846, 'latency_p50': 4.9235618114471436, 'latency_p90': 5.660477185249329}, {'batch_size': 12, 'throughput': 2.001506881332334, 'latency_mean': 5.907315639257431, 'latency_p50': 5.94504714012146, 'latency_p90': 6.717521786689758}, {'batch_size': 15, 'throughput': 2.00865553811037, 'latency_mean': 7.305866615772247, 'latency_p50': 7.38349449634552, 'latency_p90': 8.131968379020691}]

max_input_tokens: 512

max_output_tokens: 64

model_architecture: LlamaForCausalLM

model_group: Riverise/0910_2005_sft

model_name: riverise-0910-2005-sft_v1

model_num_parameters: 8030261248.0

model_repo: Riverise/0910_2005_sft

model_size: 8B

num_battles: 9644

num_wins: 4883

ranking_group: single

status: torndown

submission_type: basic

throughput_3p7s: 1.97

timestamp: 2024-09-12T07:22:06+00:00

us_pacific_date: 2024-09-12

win_ratio: 0.5063251762754044

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name riverise-0910-2005-sft-v3-mkmlizer
Waiting for job on riverise-0910-2005-sft-v3-mkmlizer to finish
riverise-0910-2005-sft-v3-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
riverise-0910-2005-sft-v3-mkmlizer: ║     _____            __           __                                ║
riverise-0910-2005-sft-v3-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
riverise-0910-2005-sft-v3-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
riverise-0910-2005-sft-v3-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
riverise-0910-2005-sft-v3-mkmlizer: ║       /___/                                                         ║
riverise-0910-2005-sft-v3-mkmlizer: ║                                                                     ║
riverise-0910-2005-sft-v3-mkmlizer: ║  Version: 0.10.1                                                    ║
riverise-0910-2005-sft-v3-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
riverise-0910-2005-sft-v3-mkmlizer: ║  https://mk1.ai                                                     ║
riverise-0910-2005-sft-v3-mkmlizer: ║                                                                     ║
riverise-0910-2005-sft-v3-mkmlizer: ║  The license key for the current software has been verified as      ║
riverise-0910-2005-sft-v3-mkmlizer: ║  belonging to:                                                      ║
riverise-0910-2005-sft-v3-mkmlizer: ║                                                                     ║
riverise-0910-2005-sft-v3-mkmlizer: ║  Chai Research Corp.                                                ║
riverise-0910-2005-sft-v3-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
riverise-0910-2005-sft-v3-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
riverise-0910-2005-sft-v3-mkmlizer: ║                                                                     ║
riverise-0910-2005-sft-v3-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
riverise-0910-2005-sft-v3-mkmlizer: Downloaded to shared memory in 24.360s
riverise-0910-2005-sft-v3-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpup42wyim, device:0
riverise-0910-2005-sft-v3-mkmlizer: Saving flywheel model at /dev/shm/model_cache
riverise-0910-2005-sft-v3-mkmlizer: quantized model in 26.002s
riverise-0910-2005-sft-v3-mkmlizer: Processed model Riverise/0910_2005_sft in 50.362s
riverise-0910-2005-sft-v3-mkmlizer: creating bucket guanaco-mkml-models
riverise-0910-2005-sft-v3-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
riverise-0910-2005-sft-v3-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/riverise-0910-2005-sft-v3
riverise-0910-2005-sft-v3-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/riverise-0910-2005-sft-v3/config.json
riverise-0910-2005-sft-v3-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/riverise-0910-2005-sft-v3/special_tokens_map.json
riverise-0910-2005-sft-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/riverise-0910-2005-sft-v3/tokenizer_config.json
riverise-0910-2005-sft-v3-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/riverise-0910-2005-sft-v3/tokenizer.json
Failed to get response for submission chaiml-llama-8b-pairwis_8189_v19: HTTPConnectionPool(host='zonemercy-virgo-edit-v1-1e5-v12-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f860c3009e0>, 'Connection to zonemercy-virgo-edit-v1-1e5-v12-predictor.tenant-chaiml-guanaco.k2.chaiverse.com timed out. (connect timeout=None)'))
riverise-0910-2005-sft-v3-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/riverise-0910-2005-sft-v3/flywheel_model.0.safetensors
riverise-0910-2005-sft-v3-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 7/291 [00:00<00:05, 51.68it/s]
Loading 0:   8%|▊         | 22/291 [00:00<00:03, 78.73it/s]
Loading 0:  11%|█         | 31/291 [00:00<00:03, 71.59it/s]
Loading 0:  14%|█▎        | 40/291 [00:00<00:03, 76.15it/s]
Loading 0:  17%|█▋        | 49/291 [00:00<00:03, 78.10it/s]
Loading 0:  20%|██        | 59/291 [00:00<00:02, 84.56it/s]
Loading 0:  24%|██▍       | 70/291 [00:00<00:02, 81.56it/s]
Loading 0:  27%|██▋       | 79/291 [00:01<00:02, 80.40it/s]
Loading 0:  30%|███       | 88/291 [00:02<00:09, 21.69it/s]
Loading 0:  35%|███▌      | 103/291 [00:02<00:05, 32.30it/s]
Loading 0:  38%|███▊      | 112/291 [00:02<00:04, 38.58it/s]
Loading 0:  42%|████▏     | 121/291 [00:02<00:03, 43.74it/s]
Loading 0:  44%|████▍     | 129/291 [00:02<00:03, 49.05it/s]
Loading 0:  47%|████▋     | 137/291 [00:02<00:02, 52.84it/s]
Loading 0:  51%|█████     | 148/291 [00:02<00:02, 58.92it/s]
Loading 0:  54%|█████▍    | 157/291 [00:03<00:02, 60.82it/s]
Loading 0:  57%|█████▋    | 166/291 [00:03<00:02, 61.99it/s]
Loading 0:  60%|██████    | 175/291 [00:03<00:01, 64.46it/s]
Loading 0:  63%|██████▎   | 184/291 [00:03<00:01, 70.11it/s]
Loading 0:  66%|██████▌   | 192/291 [00:04<00:04, 20.86it/s]
Loading 0:  68%|██████▊   | 198/291 [00:04<00:03, 23.79it/s]
Loading 0:  70%|███████   | 205/291 [00:04<00:02, 28.95it/s]
Loading 0:  74%|███████▎  | 214/291 [00:04<00:02, 37.24it/s]
Loading 0:  77%|███████▋  | 223/291 [00:04<00:01, 45.12it/s]
Loading 0:  80%|███████▉  | 232/291 [00:05<00:01, 48.33it/s]
Loading 0:  83%|████████▎ | 241/291 [00:05<00:00, 54.70it/s]
Loading 0:  86%|████████▌ | 250/291 [00:05<00:00, 61.07it/s]
Loading 0:  89%|████████▉ | 259/291 [00:05<00:00, 61.39it/s]
Loading 0:  92%|█████████▏| 268/291 [00:05<00:00, 67.23it/s]
Loading 0:  95%|█████████▌| 277/291 [00:05<00:00, 71.24it/s]
Loading 0:  98%|█████████▊| 286/291 [00:05<00:00, 73.98it/s]
                                                            
Job riverise-0910-2005-sft-v3-mkmlizer completed after 73.92s with status: succeeded
Stopping job with name riverise-0910-2005-sft-v3-mkmlizer
Pipeline stage MKMLizer completed in 74.92s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service riverise-0910-2005-sft-v3
Waiting for inference service riverise-0910-2005-sft-v3 to be ready
Inference service riverise-0910-2005-sft-v3 ready after 170.8835482597351s
Pipeline stage MKMLDeployer completed in 171.21s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.530318975448608s
Received healthy response to inference request in 8.439541816711426s
Received healthy response to inference request in 1.9010188579559326s
Received healthy response to inference request in 2.4858272075653076s
Received healthy response to inference request in 5.317116975784302s
5 requests
0 failed requests
5th percentile: 2.017980527877808
10th percentile: 2.1349421977996825
20th percentile: 2.3688655376434324
30th percentile: 2.8947255611419678
40th percentile: 3.712522268295288
50th percentile: 4.530318975448608
60th percentile: 4.845038175582886
70th percentile: 5.159757375717163
80th percentile: 5.941601943969727
90th percentile: 7.190571880340577
95th percentile: 7.815056848526001
99th percentile: 8.31464482307434
mean time: 4.534764766693115
%s, retrying in %s seconds...
Received healthy response to inference request in 1.5423591136932373s
Received healthy response to inference request in 1.7777683734893799s
Received healthy response to inference request in 1.515580654144287s
Received healthy response to inference request in 7.630360126495361s
Received healthy response to inference request in 2.326824903488159s
5 requests
0 failed requests
5th percentile: 1.5209363460540772
10th percentile: 1.5262920379638671
20th percentile: 1.5370034217834472
30th percentile: 1.5894409656524657
40th percentile: 1.6836046695709228
50th percentile: 1.7777683734893799
60th percentile: 1.9973909854888916
70th percentile: 2.2170135974884033
80th percentile: 3.3875319480896007
90th percentile: 5.508946037292481
95th percentile: 6.56965308189392
99th percentile: 7.418218717575073
mean time: 2.958578634262085
Pipeline stage StressChecker completed in 39.43s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 3.87s
Shutdown handler de-registered
riverise-0910-2005-sft_v3 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service riverise-0910-2005-sft-v3-profiler
Waiting for inference service riverise-0910-2005-sft-v3-profiler to be ready
Inference service riverise-0910-2005-sft-v3-profiler ready after 170.4076623916626s
Pipeline stage MKMLProfilerDeployer completed in 170.76s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/riverise-0910-2005-sft-v3-profiler-predictor-00001-deploymb42j5:/code/chaiverse_profiler_1726126224 --namespace tenant-chaiml-guanaco
kubectl exec -it riverise-0910-2005-sft-v3-profiler-predictor-00001-deploymb42j5 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1726126224 && python profiles.py profile --best_of_n 16 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1726126224/summary.json'
kubectl exec -it riverise-0910-2005-sft-v3-profiler-predictor-00001-deploymb42j5 --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1726126224/summary.json'
Pipeline stage MKMLProfilerRunner completed in 849.84s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service riverise-0910-2005-sft-v3-profiler is running
Tearing down inference service riverise-0910-2005-sft-v3-profiler
Service riverise-0910-2005-sft-v3-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.94s
Shutdown handler de-registered
riverise-0910-2005-sft_v3 status is now inactive due to auto deactivation removed underperforming models
riverise-0910-2005-sft_v3 status is now torndown due to DeploymentManager action