submission_id: function_forun_2024-12-30
developer_uid: jxlu90
status: torndown
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 100, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['You:', '\n', '</s>'], 'max_input_tokens': 1024, 'best_of': 4, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
timestamp: 2025-12-20T00:28:00+00:00
model_name: llama_405b_bo4_ctx1k
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.2468979358673096s
Received healthy response to inference request in 4.322024345397949s
Received healthy response to inference request in 4.083297252655029s
Received healthy response to inference request in 4.673069000244141s
Received healthy response to inference request in 1.9485578536987305s
5 requests
0 failed requests
5th percentile: 2.208225870132446
10th percentile: 2.467893886566162
20th percentile: 2.987229919433594
30th percentile: 3.4141777992248534
40th percentile: 3.7487375259399416
50th percentile: 4.083297252655029
60th percentile: 4.178788089752198
70th percentile: 4.274278926849365
80th percentile: 4.392233276367188
90th percentile: 4.532651138305664
95th percentile: 4.602860069274902
99th percentile: 4.659027214050293
mean time: 3.654769277572632
%s, retrying in %s seconds...
Received healthy response to inference request in 3.2435343265533447s
Received healthy response to inference request in 6.472658157348633s
Received healthy response to inference request in 5.222383975982666s
Retrying (%r) after connection broken by '%r': %s
Received healthy response to inference request in 6.337781667709351s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 3.3848540782928467s
5 requests
0 failed requests
5th percentile: 3.271798276901245
10th percentile: 3.3000622272491453
20th percentile: 3.3565901279449464
30th percentile: 3.7523600578308103
40th percentile: 4.487372016906739
50th percentile: 5.222383975982666
60th percentile: 5.66854305267334
70th percentile: 6.114702129364013
80th percentile: 6.364756965637207
90th percentile: 6.41870756149292
95th percentile: 6.445682859420776
99th percentile: 6.467263097763062
mean time: 4.932242441177368
%s, retrying in %s seconds...
Received healthy response to inference request in 3.438929557800293s
Received healthy response to inference request in 4.151821851730347s
Received healthy response to inference request in 2.832855224609375s
Received healthy response to inference request in 4.665947675704956s
Received healthy response to inference request in 1.5902202129364014s
5 requests
0 failed requests
5th percentile: 1.838747215270996
10th percentile: 2.0872742176055907
20th percentile: 2.58432822227478
30th percentile: 2.9540700912475586
40th percentile: 3.196499824523926
50th percentile: 3.438929557800293
60th percentile: 3.7240864753723146
70th percentile: 4.009243392944335
80th percentile: 4.2546470165252686
90th percentile: 4.460297346115112
95th percentile: 4.563122510910034
99th percentile: 4.645382642745972
mean time: 3.3359549045562744
clean up pipeline due to error=DeploymentChecksError('Unacceptable 70th percentile latency 4.009243392944335s')
Shutdown handler de-registered
function_forun_2024-12-30 status is now failed due to DeploymentManager action
function_forun_2024-12-30 status is now torndown due to DeploymentManager action