function_tuguk_2024-09-14

submission_id: function_tuguk_2024-09-14

developer_uid: chai_backend_admin

celo_rating: 1256.9

display_name: dpo_with_ava_reward_100k_v1

family_friendly_score: 0.0

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

generation_params: {'temperature': 0.95, 'top_p': 1.0, 'min_p': 0.08, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', '<|eot_id|>'], 'max_input_tokens': 512, 'best_of': 16, 'max_output_tokens': 64}

is_internal_developer: True

model_group:

model_name: dpo_with_ava_reward_100k_v1

num_battles: 10429

num_wins: 5246

ranking_group: single

status: torndown

submission_type: function

timestamp: 2024-09-14T21:02:05+00:00

us_pacific_date: 2024-09-14

win_ratio: 0.5030204238181992

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
HTTPSConnectionPool(host='guanaco-submitter.chai-research.com', port=443): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.7585394382476807s
Received healthy response to inference request in 3.101152181625366s
Received healthy response to inference request in 2.332153797149658s
5 requests
2 failed requests
5th percentile: 2.4859534740447997
10th percentile: 2.6397531509399412
20th percentile: 2.9473525047302247
30th percentile: 3.232629632949829
40th percentile: 3.4955845355987547
50th percentile: 3.7585394382476807
60th percentile: 10.495782899856566
70th percentile: 17.233026361465452
80th percentile: 20.604030179977418
90th percentile: 20.608794355392455
95th percentile: 20.611176443099975
99th percentile: 20.613082113265992
mean time: 10.08141040802002
%s, retrying in %s seconds...
Received healthy response to inference request in 2.333146095275879s
Received healthy response to inference request in 2.8301706314086914s
Received healthy response to inference request in 2.915898084640503s
Failed to get response for submission blend_hokok_2024-09-09: ('http://neversleep-noromaid-v0-8068-v150-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Received healthy response to inference request in 4.7208251953125s
Received healthy response to inference request in 2.8001253604888916s
5 requests
0 failed requests
5th percentile: 2.4265419483184814
10th percentile: 2.519937801361084
20th percentile: 2.706729507446289
30th percentile: 2.8061344146728517
40th percentile: 2.8181525230407716
50th percentile: 2.8301706314086914
60th percentile: 2.864461612701416
70th percentile: 2.8987525939941405
80th percentile: 3.2768835067749027
90th percentile: 3.9988543510437013
95th percentile: 4.3598397731781
99th percentile: 4.64862811088562
mean time: 3.120033073425293
Pipeline stage StressChecker completed in 67.03s
Shutdown handler de-registered
function_tuguk_2024-09-14 status is now deployed due to DeploymentManager action
function_tuguk_2024-09-14 status is now inactive due to auto deactivation removed underperforming models
function_tuguk_2024-09-14 status is now torndown due to DeploymentManager action