developer_uid: chai_backend_admin
submission_id: zonemercy-vingt-deux-v1-1e5_v9
model_name: temp-1
model_group: zonemercy/Vingt-Deux-v1-
status: torndown
timestamp: 2024-09-24T16:06:47+00:00
num_battles: 4731
num_wins: 2251
celo_rating: 1236.41
family_friendly_score: 0.0
submission_type: basic
model_repo: zonemercy/Vingt-Deux-v1-1e5
model_architecture: MistralForCausalLM
model_num_parameters: 22247282688.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
latencies: [{'batch_size': 1, 'throughput': 0.3809209789809493, 'latency_mean': 2.625156282186508, 'latency_p50': 2.62872850894928, 'latency_p90': 2.891029477119446}, {'batch_size': 2, 'throughput': 0.5969255504298934, 'latency_mean': 3.3421693336963654, 'latency_p50': 3.355453372001648, 'latency_p90': 3.6558380365371703}, {'batch_size': 3, 'throughput': 0.7542351419096032, 'latency_mean': 3.960221129655838, 'latency_p50': 3.9722386598587036, 'latency_p90': 4.347625637054444}, {'batch_size': 4, 'throughput': 0.8817468515486284, 'latency_mean': 4.502881811857224, 'latency_p50': 4.480708003044128, 'latency_p90': 5.050600385665893}, {'batch_size': 5, 'throughput': 0.9662302938034383, 'latency_mean': 5.158867874145508, 'latency_p50': 5.185339450836182, 'latency_p90': 5.8149580478668215}]
gpu_counts: {'NVIDIA RTX A6000': 1}
display_name: temp-1
ineligible_reason: num_battles<5000
is_internal_developer: True
language_model: zonemercy/Vingt-Deux-v1-1e5
model_size: 22B
ranking_group: single
throughput_3p7s: 0.7
us_pacific_date: 2024-09-24
win_ratio: 0.4757979285563306
generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '</s>', '####', 'Bot:', 'User:', 'You:', '<|im_end|>', '<|eot_id|>'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Starting job with name zonemercy-vingt-deux-v1-1e5-v9-mkmlizer
Waiting for job on zonemercy-vingt-deux-v1-1e5-v9-mkmlizer to finish
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v2: ('http://zonemercy-vingt-deux-v2-1e5-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission zonemercy-vingt-deux-v1-1e5_v5: ('http://zonemercy-vingt-deux-v1-1e5-v5-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║     _____            __           __                                ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║       /___/                                                         ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║  Version: 0.10.1                                                    ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║  https://mk1.ai                                                     ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║  The license key for the current software has been verified as      ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║  belonging to:                                                      ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║  Chai Research Corp.                                                ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ║                                                                     ║
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v2: ('http://zonemercy-vingt-deux-v2-1e5-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: Downloaded to shared memory in 51.344s
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpqx528x7n, device:0
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: quantized model in 47.534s
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: Processed model zonemercy/Vingt-Deux-v1-1e5 in 98.878s
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: creating bucket guanaco-mkml-models
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v9
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v2: ('http://zonemercy-vingt-deux-v2-1e5-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v9/config.json
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v9/special_tokens_map.json
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v9/tokenizer_config.json
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v9/tokenizer.json
zonemercy-vingt-deux-v1-1e5-v9-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/zonemercy-vingt-deux-v1-1e5-v9/flywheel_model.1.safetensors
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Job zonemercy-vingt-deux-v1-1e5-v9-mkmlizer completed after 155.46s with status: succeeded
Stopping job with name zonemercy-vingt-deux-v1-1e5-v9-mkmlizer
Pipeline stage MKMLizer completed in 156.33s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-vingt-deux-v1-1e5-v9
Waiting for inference service zonemercy-vingt-deux-v1-1e5-v9 to be ready
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-v1-1e5_v5: ('http://zonemercy-vingt-deux-v1-1e5-v5-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Failed to get response for submission zonemercy-vingt-deux-v1-1e5_v5: ('http://zonemercy-vingt-deux-v1-1e5-v5-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Inference service zonemercy-vingt-deux-v1-1e5-v9 ready after 201.10201859474182s
Pipeline stage MKMLDeployer completed in 201.51s
run pipeline stage %s
Failed to get response for submission zonemercy-vingt-deux-v2-1e5_v2: ('http://zonemercy-vingt-deux-v2-1e5-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'activator request timeout')
Running pipeline stage StressChecker
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Received healthy response to inference request in 4.5086143016815186s
Received healthy response to inference request in 3.879378318786621s
Failed to get response for submission zonemercy-vingt-deux-gfv_3432_v1: ('http://zonemercy-vingt-deux-gfv-3432-v1-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Received healthy response to inference request in 2.8934760093688965s
Received healthy response to inference request in 2.620652675628662s
Received healthy response to inference request in 2.5226004123687744s
5 requests
0 failed requests
5th percentile: 2.5422108650207518
10th percentile: 2.5618213176727296
20th percentile: 2.6010422229766847
30th percentile: 2.675217342376709
40th percentile: 2.784346675872803
50th percentile: 2.8934760093688965
60th percentile: 3.2878369331359862
70th percentile: 3.682197856903076
80th percentile: 4.005225515365601
90th percentile: 4.256919908523559
95th percentile: 4.3827671051025385
99th percentile: 4.483444862365722
mean time: 3.2849443435668944
Pipeline stage StressChecker completed in 18.87s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 5.01s
Shutdown handler de-registered
zonemercy-vingt-deux-v1-1e5_v9 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service zonemercy-vingt-deux-v1-1e5-v9-profiler
Waiting for inference service zonemercy-vingt-deux-v1-1e5-v9-profiler to be ready
Tearing down inference service zonemercy-vingt-deux-v1-1e5-v9-profiler
%s, retrying in %s seconds...
Creating inference service zonemercy-vingt-deux-v1-1e5-v9-profiler
Waiting for inference service zonemercy-vingt-deux-v1-1e5-v9-profiler to be ready
Tearing down inference service zonemercy-vingt-deux-v1-1e5-v9-profiler
%s, retrying in %s seconds...
Creating inference service zonemercy-vingt-deux-v1-1e5-v9-profiler
Waiting for inference service zonemercy-vingt-deux-v1-1e5-v9-profiler to be ready
Inference service zonemercy-vingt-deux-v1-1e5-v9-profiler ready after 120.32781887054443s
Pipeline stage MKMLProfilerDeployer completed in 1324.74s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/zonemercy-vingt-deux300a138d6cdecf8d5f1aae3f39e5c2e4-deplo4648p:/code/chaiverse_profiler_1727195776 --namespace tenant-chaiml-guanaco
kubectl exec -it zonemercy-vingt-deux300a138d6cdecf8d5f1aae3f39e5c2e4-deplo4648p --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1727195776 && python profiles.py profile --best_of_n 8 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 64 --summary /code/chaiverse_profiler_1727195776/summary.json'
kubectl exec -it zonemercy-vingt-deux300a138d6cdecf8d5f1aae3f39e5c2e4-deplo4648p --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1727195776/summary.json'
Pipeline stage MKMLProfilerRunner completed in 1563.41s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service zonemercy-vingt-deux-v1-1e5-v9-profiler is running
Tearing down inference service zonemercy-vingt-deux-v1-1e5-v9-profiler
Service zonemercy-vingt-deux-v1-1e5-v9-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 2.08s
Shutdown handler de-registered
zonemercy-vingt-deux-v1-1e5_v9 status is now inactive due to auto deactivation removed underperforming models
run pipeline %s
admin requested tearing down of zonemercy-vingt-deux-v1-1e5_v9
run pipeline stage %s
Shutdown handler not registered because Python interpreter is not running in the main thread
Running pipeline stage MKMLDeleter
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLDeleter
%s, retrying in %s seconds...
%s, retrying in %s seconds...
clean up pipeline due to error=TeardownError("module 'kubernetes.config' has no attribute 'load_kube_config'")
Shutdown handler de-registered
zonemercy-vingt-deux-v1-1e5_v9 status is now torndown due to DeploymentManager action