Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 4.550747632980347s
Received healthy response to inference request in 6.290600538253784s
Received healthy response to inference request in 4.265039920806885s
Received healthy response to inference request in 1.5733230113983154s
5 requests
1 failed requests
5th percentile: 2.1116663932800295
10th percentile: 2.6500097751617435
20th percentile: 3.726696538925171
30th percentile: 4.322181463241577
40th percentile: 4.436464548110962
50th percentile: 4.550747632980347
60th percentile: 5.246688795089722
70th percentile: 5.942629957199096
80th percentile: 9.054627370834353
90th percentile: 14.582681035995485
95th percentile: 17.346707868576047
99th percentile: 19.5579293346405
mean time: 7.35808916091919
%s, retrying in %s seconds...
Received healthy response to inference request in 3.910750389099121s
Received healthy response to inference request in 5.028837442398071s
Received healthy response to inference request in 5.293795108795166s
Received healthy response to inference request in 3.7942123413085938s
Received healthy response to inference request in 5.088320255279541s
5 requests
0 failed requests
5th percentile: 3.817519950866699
10th percentile: 3.8408275604248048
20th percentile: 3.887442779541016
30th percentile: 4.134367799758911
40th percentile: 4.581602621078491
50th percentile: 5.028837442398071
60th percentile: 5.0526305675506595
70th percentile: 5.076423692703247
80th percentile: 5.129415225982666
90th percentile: 5.211605167388916
95th percentile: 5.252700138092041
99th percentile: 5.285576114654541
mean time: 4.623183107376098
%s, retrying in %s seconds...
Received healthy response to inference request in 5.355981111526489s
Received healthy response to inference request in 5.1383020877838135s
Received healthy response to inference request in 2.1084418296813965s
Received healthy response to inference request in 4.729319095611572s
Received healthy response to inference request in 4.156328201293945s
5 requests
0 failed requests
5th percentile: 2.518019104003906
10th percentile: 2.927596378326416
20th percentile: 3.7467509269714356
30th percentile: 4.270926380157471
40th percentile: 4.500122737884522
50th percentile: 4.729319095611572
60th percentile: 4.892912292480469
70th percentile: 5.056505489349365
80th percentile: 5.181837892532348
90th percentile: 5.268909502029419
95th percentile: 5.3124453067779545
99th percentile: 5.347273950576782
mean time: 4.297674465179443
clean up pipeline due to error=DeploymentChecksError('Unacceptable 70th percentile latency 5.056505489349365s')
Shutdown handler de-registered
function_lusam_2025-11-09 status is now failed due to DeploymentManager action
function_lusam_2025-11-09 status is now inactive due to auto deactivation removed underperforming models
function_lusam_2025-11-09 status is now protected due to ABTestQueueItem