riverise-alighment-0906

submission_id: riverise-alighment-0906_v1

developer_uid: Riverise

best_of: 16

celo_rating: 1248.08

display_name: riverise-alighment-0906_v1

family_friendly_score: 0.0

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}

gpu_counts: {'NVIDIA RTX A5000': 1}

is_internal_developer: False

language_model: Riverise/alighment_0906

latencies: [{'batch_size': 1, 'throughput': 0.9169173084544289, 'latency_mean': 1.0905486357212066, 'latency_p50': 1.0869613885879517, 'latency_p90': 1.2143476009368896}, {'batch_size': 4, 'throughput': 1.8467096491453583, 'latency_mean': 2.157620149850845, 'latency_p50': 2.146966576576233, 'latency_p90': 2.387117314338684}, {'batch_size': 5, 'throughput': 1.927395968823675, 'latency_mean': 2.576580295562744, 'latency_p50': 2.594598889350891, 'latency_p90': 2.8534260034561156}, {'batch_size': 8, 'throughput': 2.0610647705583367, 'latency_mean': 3.8536030519008637, 'latency_p50': 3.88791823387146, 'latency_p90': 4.346201729774475}, {'batch_size': 10, 'throughput': 2.0486814166475615, 'latency_mean': 4.837924400568008, 'latency_p50': 4.804916620254517, 'latency_p90': 5.7247639179229735}, {'batch_size': 12, 'throughput': 2.08131650629025, 'latency_mean': 5.689735391139984, 'latency_p50': 5.7736440896987915, 'latency_p90': 6.463075304031372}, {'batch_size': 15, 'throughput': 2.080286426247177, 'latency_mean': 7.065651820898056, 'latency_p50': 7.164153456687927, 'latency_p90': 7.899716114997863}]

max_input_tokens: 512

max_output_tokens: 64

model_architecture: LlamaForCausalLM

model_group: Riverise/alighment_0906

model_name: riverise-alighment-0906_v1

model_num_parameters: 8030261248.0

model_repo: Riverise/alighment_0906

model_size: 8B

num_battles: 10873

num_wins: 5563

ranking_group: single

status: torndown

submission_type: basic

throughput_3p7s: 2.06

timestamp: 2024-09-09T09:03:09+00:00

us_pacific_date: 2024-09-09

win_ratio: 0.511634323553757

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name riverise-alighment-0906-v1-mkmlizer
Waiting for job on riverise-alighment-0906-v1-mkmlizer to finish
riverise-alighment-0906-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
riverise-alighment-0906-v1-mkmlizer: ║     _____            __           __                                ║
riverise-alighment-0906-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
riverise-alighment-0906-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
riverise-alighment-0906-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
riverise-alighment-0906-v1-mkmlizer: ║       /___/                                                         ║
riverise-alighment-0906-v1-mkmlizer: ║                                                                     ║
riverise-alighment-0906-v1-mkmlizer: ║  Version: 0.10.1                                                    ║
riverise-alighment-0906-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
riverise-alighment-0906-v1-mkmlizer: ║  https://mk1.ai                                                     ║
riverise-alighment-0906-v1-mkmlizer: ║                                                                     ║
riverise-alighment-0906-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
riverise-alighment-0906-v1-mkmlizer: ║  belonging to:                                                      ║
riverise-alighment-0906-v1-mkmlizer: ║                                                                     ║
riverise-alighment-0906-v1-mkmlizer: ║  Chai Research Corp.                                                ║
riverise-alighment-0906-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
riverise-alighment-0906-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
riverise-alighment-0906-v1-mkmlizer: ║                                                                     ║
riverise-alighment-0906-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
riverise-alighment-0906-v1-mkmlizer: Downloaded to shared memory in 34.572s
riverise-alighment-0906-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp3w3nqct0, device:0
riverise-alighment-0906-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
riverise-alighment-0906-v1-mkmlizer: quantized model in 25.633s
riverise-alighment-0906-v1-mkmlizer: Processed model Riverise/alighment_0906 in 60.205s
riverise-alighment-0906-v1-mkmlizer: creating bucket guanaco-mkml-models
riverise-alighment-0906-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
riverise-alighment-0906-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/riverise-alighment-0906-v1
riverise-alighment-0906-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/riverise-alighment-0906-v1/special_tokens_map.json
riverise-alighment-0906-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/riverise-alighment-0906-v1/config.json
riverise-alighment-0906-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/riverise-alighment-0906-v1/tokenizer_config.json
riverise-alighment-0906-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/riverise-alighment-0906-v1/tokenizer.json
riverise-alighment-0906-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/riverise-alighment-0906-v1/flywheel_model.0.safetensors
riverise-alighment-0906-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 7/291 [00:00<00:05, 52.07it/s]
Loading 0:   8%|▊         | 22/291 [00:00<00:03, 78.33it/s]
Loading 0:  11%|█         | 31/291 [00:00<00:03, 78.95it/s]
Loading 0:  14%|█▍        | 42/291 [00:00<00:02, 88.82it/s]
Loading 0:  18%|█▊        | 52/291 [00:00<00:02, 81.55it/s]
Loading 0:  21%|██        | 61/291 [00:00<00:02, 81.96it/s]
Loading 0:  24%|██▍       | 70/291 [00:00<00:02, 73.99it/s]
Loading 0:  27%|██▋       | 79/291 [00:01<00:02, 78.20it/s]
Loading 0:  30%|███       | 88/291 [00:02<00:09, 21.20it/s]
Loading 0:  35%|███▌      | 103/291 [00:02<00:05, 31.47it/s]
Loading 0:  38%|███▊      | 112/291 [00:02<00:04, 37.03it/s]
Loading 0:  42%|████▏     | 121/291 [00:02<00:03, 43.96it/s]
Loading 0:  45%|████▍     | 130/291 [00:02<00:03, 51.03it/s]
Loading 0:  49%|████▉     | 142/291 [00:02<00:02, 58.81it/s]
Loading 0:  52%|█████▏    | 151/291 [00:02<00:02, 62.87it/s]
Loading 0:  55%|█████▍    | 160/291 [00:03<00:02, 64.90it/s]
Loading 0:  58%|█████▊    | 169/291 [00:03<00:01, 67.23it/s]
Loading 0:  63%|██████▎   | 184/291 [00:03<00:01, 78.52it/s]
Loading 0:  66%|██████▋   | 193/291 [00:04<00:03, 24.55it/s]
Loading 0:  69%|██████▉   | 202/291 [00:04<00:02, 30.51it/s]
Loading 0:  73%|███████▎  | 211/291 [00:04<00:02, 37.29it/s]
Loading 0:  76%|███████▌  | 220/291 [00:04<00:01, 44.69it/s]
Loading 0:  79%|███████▊  | 229/291 [00:04<00:01, 51.66it/s]
Loading 0:  82%|████████▏ | 240/291 [00:04<00:00, 62.63it/s]
Loading 0:  86%|████████▌ | 250/291 [00:05<00:00, 63.13it/s]
Loading 0:  89%|████████▉ | 259/291 [00:05<00:00, 68.79it/s]
Loading 0:  92%|█████████▏| 268/291 [00:05<00:00, 69.71it/s]
Loading 0:  95%|█████████▌| 277/291 [00:05<00:00, 71.74it/s]
Loading 0:  99%|█████████▊| 287/291 [00:05<00:00, 42.86it/s]
                                                            
Job riverise-alighment-0906-v1-mkmlizer completed after 84.64s with status: succeeded
Stopping job with name riverise-alighment-0906-v1-mkmlizer
Pipeline stage MKMLizer completed in 86.35s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.09s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service riverise-alighment-0906-v1
Waiting for inference service riverise-alighment-0906-v1 to be ready
Inference service riverise-alighment-0906-v1 ready after 141.53224205970764s
Pipeline stage MKMLDeployer completed in 142.15s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 1.9558467864990234s
Received healthy response to inference request in 2.4215850830078125s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 1.556781530380249s
Received healthy response to inference request in 1.7771425247192383s
Received healthy response to inference request in 1.666693925857544s
5 requests
0 failed requests
5th percentile: 1.578764009475708
10th percentile: 1.600746488571167
20th percentile: 1.644711446762085
30th percentile: 1.6887836456298828
40th percentile: 1.7329630851745605
50th percentile: 1.7771425247192383
60th percentile: 1.8486242294311523
70th percentile: 1.9201059341430664
80th percentile: 2.048994445800781
90th percentile: 2.235289764404297
95th percentile: 2.3284374237060548
99th percentile: 2.402955551147461
mean time: 1.8756099700927735
Pipeline stage StressChecker completed in 10.90s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 4.83s
Shutdown handler de-registered
riverise-alighment-0906_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service riverise-alighment-0906-v1-profiler
Waiting for inference service riverise-alighment-0906-v1-profiler to be ready
Inference service riverise-alighment-0906-v1-profiler ready after 150.3379716873169s
Pipeline stage MKMLProfilerDeployer completed in 150.69s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/riverise-alighment-0906-v1-profiler-predictor-00001-deployvk4qd:/code/chaiverse_profiler_1725873027 --namespace tenant-chaiml-guanaco
kubectl exec -it riverise-alighment-0906-v1-profiler-predictor-00001-deployvk4qd --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1725873027 && python profiles.py profile --best_of_n 16 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725873027/summary.json'
kubectl exec -it riverise-alighment-0906-v1-profiler-predictor-00001-deployvk4qd --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1725873027/summary.json'
Pipeline stage MKMLProfilerRunner completed in 822.45s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service riverise-alighment-0906-v1-profiler is running
Tearing down inference service riverise-alighment-0906-v1-profiler
Service riverise-alighment-0906-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 1.59s
Shutdown handler de-registered
riverise-alighment-0906_v1 status is now inactive due to auto deactivation removed underperforming models
riverise-alighment-0906_v1 status is now torndown due to DeploymentManager action