cycy233-l3-p-v3-c3

submission_id: cycy233-l3-p-v3-c3_v1

developer_uid: shiroe40

alignment_samples: 9532

alignment_score: 0.36351666080747536

best_of: 16

celo_rating: 1248.84

display_name: auto

formatter: {'memory_template': "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{bot_name}'s Persona: {memory}\n\n", 'prompt_template': '{prompt}<|eot_id|>', 'bot_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}: {message}<|eot_id|>', 'user_template': '<|start_header_id|>user<|end_header_id|>\n\n{user_name}: {message}<|eot_id|>', 'response_template': '<|start_header_id|>assistant<|end_header_id|>\n\n{bot_name}:', 'truncate_by_message': False}

generation_params: {'temperature': 1.0, 'top_p': 0.9, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|end_header_id|>', '<|eot_id|>'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}

gpu_counts: {'NVIDIA RTX A5000': 1}

is_internal_developer: False

language_model: cycy233/L3-p-v3-c3

latencies: [{'batch_size': 1, 'throughput': 0.904314380335219, 'latency_mean': 1.1057097434997558, 'latency_p50': 1.1029020547866821, 'latency_p90': 1.2457449436187744}, {'batch_size': 4, 'throughput': 1.7634630757564755, 'latency_mean': 2.2566311705112456, 'latency_p50': 2.26652991771698, 'latency_p90': 2.5347859382629396}, {'batch_size': 5, 'throughput': 1.872329038772999, 'latency_mean': 2.6568923592567444, 'latency_p50': 2.6870994567871094, 'latency_p90': 2.9997154235839845}, {'batch_size': 8, 'throughput': 1.9823710275344542, 'latency_mean': 4.000118588209152, 'latency_p50': 4.001702666282654, 'latency_p90': 4.556585025787354}, {'batch_size': 10, 'throughput': 1.981817307775268, 'latency_mean': 4.994505189657211, 'latency_p50': 4.965648293495178, 'latency_p90': 5.779207801818847}, {'batch_size': 12, 'throughput': 2.0077867758997114, 'latency_mean': 5.899681622982025, 'latency_p50': 5.937089204788208, 'latency_p90': 6.759657406806945}, {'batch_size': 15, 'throughput': 2.004721489088285, 'latency_mean': 7.343309516906738, 'latency_p50': 7.460484623908997, 'latency_p90': 8.201191020011901}]

max_input_tokens: 512

max_output_tokens: 64

model_architecture: LlamaForCausalLM

model_group: cycy233/L3-p-v3-c3

model_name: auto

model_num_parameters: 8030261248.0

model_repo: cycy233/L3-p-v3-c3

model_size: 8B

num_battles: 9532

num_wins: 4839

propriety_score: 0.7178051511758119

propriety_total_count: 893.0

ranking_group: single

status: inactive

submission_type: basic

throughput_3p7s: 1.98

timestamp: 2024-09-12T03:13:45+00:00

us_pacific_date: 2024-09-11

win_ratio: 0.5076584137641629

Download Preference Data

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name cycy233-l3-p-v3-c3-v1-mkmlizer
Waiting for job on cycy233-l3-p-v3-c3-v1-mkmlizer to finish
cycy233-l3-p-v3-c3-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
cycy233-l3-p-v3-c3-v1-mkmlizer: ║     _____            __           __                                ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║       /___/                                                         ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║                                                                     ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║  Version: 0.10.1                                                    ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║  https://mk1.ai                                                     ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║                                                                     ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║  belonging to:                                                      ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║                                                                     ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║  Chai Research Corp.                                                ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ║                                                                     ║
cycy233-l3-p-v3-c3-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
cycy233-l3-p-v3-c3-v1-mkmlizer: Downloaded to shared memory in 38.600s
cycy233-l3-p-v3-c3-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpa71rd6bl, device:0
cycy233-l3-p-v3-c3-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
cycy233-l3-p-v3-c3-v1-mkmlizer: quantized model in 26.246s
cycy233-l3-p-v3-c3-v1-mkmlizer: Processed model cycy233/L3-p-v3-c3 in 64.846s
cycy233-l3-p-v3-c3-v1-mkmlizer: creating bucket guanaco-mkml-models
cycy233-l3-p-v3-c3-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
cycy233-l3-p-v3-c3-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/cycy233-l3-p-v3-c3-v1
cycy233-l3-p-v3-c3-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/cycy233-l3-p-v3-c3-v1/special_tokens_map.json
cycy233-l3-p-v3-c3-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/cycy233-l3-p-v3-c3-v1/config.json
cycy233-l3-p-v3-c3-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/cycy233-l3-p-v3-c3-v1/tokenizer_config.json
cycy233-l3-p-v3-c3-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/cycy233-l3-p-v3-c3-v1/tokenizer.json
cycy233-l3-p-v3-c3-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/cycy233-l3-p-v3-c3-v1/flywheel_model.0.safetensors
cycy233-l3-p-v3-c3-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 4/291 [00:00<00:08, 33.44it/s]
Loading 0:   4%|▍         | 13/291 [00:00<00:04, 55.76it/s]
Loading 0:   8%|▊         | 22/291 [00:00<00:04, 65.54it/s]
Loading 0:  11%|█         | 31/291 [00:00<00:03, 73.70it/s]
Loading 0:  14%|█▎        | 40/291 [00:00<00:03, 74.22it/s]
Loading 0:  17%|█▋        | 49/291 [00:00<00:03, 75.18it/s]
Loading 0:  21%|██        | 61/291 [00:00<00:02, 78.37it/s]
Loading 0:  24%|██▍       | 71/291 [00:00<00:02, 84.01it/s]
Loading 0:  29%|██▊       | 83/291 [00:02<00:08, 23.27it/s]
Loading 0:  31%|███       | 90/291 [00:02<00:07, 26.85it/s]
Loading 0:  33%|███▎      | 97/291 [00:02<00:06, 31.25it/s]
Loading 0:  36%|███▋      | 106/291 [00:02<00:04, 38.59it/s]
Loading 0:  40%|███▉      | 115/291 [00:02<00:04, 43.77it/s]
Loading 0:  43%|████▎     | 124/291 [00:02<00:03, 48.62it/s]
Loading 0:  46%|████▌     | 133/291 [00:02<00:02, 54.55it/s]
Loading 0:  49%|████▉     | 142/291 [00:03<00:02, 59.62it/s]
Loading 0:  52%|█████▏    | 151/291 [00:03<00:02, 62.03it/s]
Loading 0:  55%|█████▍    | 160/291 [00:03<00:02, 64.93it/s]
Loading 0:  58%|█████▊    | 169/291 [00:03<00:01, 69.39it/s]
Loading 0:  61%|██████    | 178/291 [00:03<00:01, 68.14it/s]
Loading 0:  64%|██████▍   | 187/291 [00:04<00:05, 20.74it/s]
Loading 0:  67%|██████▋   | 196/291 [00:04<00:03, 26.90it/s]
Loading 0:  73%|███████▎  | 211/291 [00:04<00:02, 38.96it/s]
Loading 0:  77%|███████▋  | 223/291 [00:05<00:01, 46.98it/s]
Loading 0:  80%|███████▉  | 232/291 [00:05<00:01, 53.56it/s]
Loading 0:  83%|████████▎ | 241/291 [00:05<00:00, 59.59it/s]
Loading 0:  88%|████████▊ | 256/291 [00:05<00:00, 71.76it/s]
Loading 0:  92%|█████████▏| 267/291 [00:05<00:00, 79.68it/s]
Loading 0:  95%|█████████▌| 277/291 [00:05<00:00, 74.33it/s]
Loading 0:  98%|█████████▊| 286/291 [00:05<00:00, 76.57it/s]
                                                            
Job cycy233-l3-p-v3-c3-v1-mkmlizer completed after 94.0s with status: succeeded
Stopping job with name cycy233-l3-p-v3-c3-v1-mkmlizer
Pipeline stage MKMLizer completed in 94.81s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.07s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service cycy233-l3-p-v3-c3-v1
Waiting for inference service cycy233-l3-p-v3-c3-v1 to be ready
Failed to get response for submission zonemercy-virgo-edit-v1-1e5_v12: HTTPConnectionPool(host='zonemercy-virgo-edit-v1-1e5-v12-predictor.tenant-chaiml-guanaco.k2.chaiverse.com', port=80): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fc2f87ce510>, 'Connection to zonemercy-virgo-edit-v1-1e5-v12-predictor.tenant-chaiml-guanaco.k2.chaiverse.com timed out. (connect timeout=None)'))
Inference service cycy233-l3-p-v3-c3-v1 ready after 160.87208604812622s
Pipeline stage MKMLDeployer completed in 161.19s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.0546798706054688s
Received healthy response to inference request in 2.4291117191314697s
Failed to get response for submission blend_henub_2024-09-09: ('http://zonemercy-lexical-nemo-1518-v18-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:36566->127.0.0.1:8080: read: connection reset by peer\n')
Received healthy response to inference request in 1.6006574630737305s
Received healthy response to inference request in 4.65236759185791s
Received healthy response to inference request in 1.5344629287719727s
5 requests
0 failed requests
5th percentile: 1.5477018356323242
10th percentile: 1.5609407424926758
20th percentile: 1.587418556213379
30th percentile: 1.691461944580078
40th percentile: 1.8730709075927734
50th percentile: 2.0546798706054688
60th percentile: 2.2044526100158692
70th percentile: 2.3542253494262693
80th percentile: 2.873762893676758
90th percentile: 3.7630652427673343
95th percentile: 4.207716417312621
99th percentile: 4.563437356948852
mean time: 2.4542559146881104
Pipeline stage StressChecker completed in 13.14s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 6.26s
Shutdown handler de-registered
cycy233-l3-p-v3-c3_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service cycy233-l3-p-v3-c3-v1-profiler
Waiting for inference service cycy233-l3-p-v3-c3-v1-profiler to be ready
Inference service cycy233-l3-p-v3-c3-v1-profiler ready after 170.40595936775208s
Pipeline stage MKMLProfilerDeployer completed in 170.84s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/cycy233-l3-p-v3-c3-v1-profiler-predictor-00001-deployment-2zlkx:/code/chaiverse_profiler_1726111319 --namespace tenant-chaiml-guanaco
kubectl exec -it cycy233-l3-p-v3-c3-v1-profiler-predictor-00001-deployment-2zlkx --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1726111319 && python profiles.py profile --best_of_n 16 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1726111319/summary.json'
kubectl exec -it cycy233-l3-p-v3-c3-v1-profiler-predictor-00001-deployment-2zlkx --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1726111319/summary.json'
Pipeline stage MKMLProfilerRunner completed in 848.16s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service cycy233-l3-p-v3-c3-v1-profiler is running
Tearing down inference service cycy233-l3-p-v3-c3-v1-profiler
Service cycy233-l3-p-v3-c3-v1-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.11s
Shutdown handler de-registered
cycy233-l3-p-v3-c3_v1 status is now inactive due to auto deactivation removed underperforming models