submission_id: function_notos_2024-12-30
developer_uid: jxlu90
status: torndown
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 100, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '</s>', '\n'], 'max_input_tokens': 1024, 'best_of': 4, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
timestamp: 2025-12-20T00:31:09+00:00
model_name: llama_405b_bo4_ctx1k
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.660979747772217s
Received healthy response to inference request in 3.23396635055542s
Received healthy response to inference request in 5.297492265701294s
Received healthy response to inference request in 3.7661821842193604s
Received healthy response to inference request in 2.897848606109619s
5 requests
0 failed requests
5th percentile: 2.9650721549987793
10th percentile: 3.0322957038879395
20th percentile: 3.1667428016662598
30th percentile: 3.3193690299987795
40th percentile: 3.490174388885498
50th percentile: 3.660979747772217
60th percentile: 3.7030607223510743
70th percentile: 3.745141696929932
80th percentile: 4.072444200515747
90th percentile: 4.6849682331085205
95th percentile: 4.991230249404907
99th percentile: 5.236239862442017
mean time: 3.771293830871582
%s, retrying in %s seconds...
Received healthy response to inference request in 2.914203643798828s
Received healthy response to inference request in 3.897469997406006s
Received healthy response to inference request in 3.9210143089294434s
Received healthy response to inference request in 5.9447386264801025s
Received healthy response to inference request in 4.307234525680542s
5 requests
0 failed requests
5th percentile: 3.1108569145202636
10th percentile: 3.307510185241699
20th percentile: 3.7008167266845704
30th percentile: 3.9021788597106934
40th percentile: 3.911596584320068
50th percentile: 3.9210143089294434
60th percentile: 4.075502395629883
70th percentile: 4.229990482330322
80th percentile: 4.634735345840454
90th percentile: 5.289736986160278
95th percentile: 5.61723780632019
99th percentile: 5.87923846244812
mean time: 4.196932220458985
%s, retrying in %s seconds...
Received healthy response to inference request in 3.4771602153778076s
Received healthy response to inference request in 4.3349928855896s
Received healthy response to inference request in 5.360929489135742s
Received healthy response to inference request in 3.0803062915802s
Received healthy response to inference request in 3.027218818664551s
5 requests
0 failed requests
5th percentile: 3.0378363132476807
10th percentile: 3.0484538078308105
20th percentile: 3.0696887969970703
30th percentile: 3.1596770763397215
40th percentile: 3.3184186458587646
50th percentile: 3.4771602153778076
60th percentile: 3.8202932834625245
70th percentile: 4.163426351547241
80th percentile: 4.540180206298828
90th percentile: 4.950554847717285
95th percentile: 5.155742168426514
99th percentile: 5.319892024993896
mean time: 3.85612154006958
clean up pipeline due to error=DeploymentChecksError('Unacceptable 70th percentile latency 4.163426351547241s')
Shutdown handler de-registered
function_notos_2024-12-30 status is now failed due to DeploymentManager action
function_notos_2024-12-30 status is now torndown due to DeploymentManager action