Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.790727138519287s
Received healthy response to inference request in 3.9548532962799072s
Failed to get response for submission chaiml-nemo-20241016-bre_9520_v5: ('http://chaiml-nemo-20241016-bre-9520-v5-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:55014->127.0.0.1:8080: read: connection reset by peer\n')
Received healthy response to inference request in 2.684215784072876s
Received healthy response to inference request in 3.699573278427124s
Received healthy response to inference request in 2.4927523136138916s
5 requests
0 failed requests
5th percentile: 2.5310450077056883
10th percentile: 2.5693377017974854
20th percentile: 2.6459230899810793
30th percentile: 2.705518054962158
40th percentile: 2.7481225967407226
50th percentile: 2.790727138519287
60th percentile: 3.1542655944824216
70th percentile: 3.5178040504455566
80th percentile: 3.7506292819976808
90th percentile: 3.8527412891387938
95th percentile: 3.9037972927093505
99th percentile: 3.9446420955657957
mean time: 3.124424362182617
Pipeline stage StressChecker completed in 16.90s
Shutdown handler de-registered
function_domet_2024-10-18 status is now deployed due to DeploymentManager action
function_domet_2024-10-18 status is now inactive due to auto deactivation removed underperforming models
function_domet_2024-10-18 status is now torndown due to DeploymentManager action