Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage StressChecker
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 2.6800377368927s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Received healthy response to inference request in 4.034345388412476s
Received healthy response to inference request in 5.49668288230896s
Received healthy response to inference request in 4.345855236053467s
Received healthy response to inference request in 3.064784288406372s
5 requests
0 failed requests
5th percentile: 2.7569870471954347
10th percentile: 2.833936357498169
20th percentile: 2.9878349781036375
30th percentile: 3.258696508407593
40th percentile: 3.6465209484100343
50th percentile: 4.034345388412476
60th percentile: 4.158949327468872
70th percentile: 4.283553266525269
80th percentile: 4.576020765304565
90th percentile: 5.036351823806763
95th percentile: 5.2665173530578615
99th percentile: 5.45064977645874
mean time: 3.924341106414795
%s, retrying in %s seconds...
Failed to get response for submission chaiml-nemo-20241010-tie_5991_v2: ('http://chaiml-nemo-20241010-tie-5991-v2-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:45648->127.0.0.1:8080: read: connection reset by peer\n')
Received healthy response to inference request in 3.368295192718506s
Received healthy response to inference request in 3.420031785964966s
Received healthy response to inference request in 4.7154107093811035s
Received healthy response to inference request in 3.007563352584839s
Received healthy response to inference request in 3.1352622509002686s
5 requests
0 failed requests
5th percentile: 3.0331031322479247
10th percentile: 3.0586429119110106
20th percentile: 3.1097224712371827
30th percentile: 3.181868839263916
40th percentile: 3.275082015991211
50th percentile: 3.368295192718506
60th percentile: 3.3889898300170898
70th percentile: 3.4096844673156737
80th percentile: 3.6791075706481937
90th percentile: 4.197259140014649
95th percentile: 4.456334924697876
99th percentile: 4.663595552444458
mean time: 3.5293126583099363
Pipeline stage StressChecker completed in 39.71s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.31s
Shutdown handler de-registered
function_jufub_2024-11-15 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3167.51s
Shutdown handler de-registered
function_jufub_2024-11-15 status is now inactive due to auto deactivation removed underperforming models