Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 1.979492425918579s
Received healthy response to inference request in 2.046116352081299s
Received healthy response to inference request in 2.153958320617676s
Received healthy response to inference request in 2.8522746562957764s
5 requests
1 failed requests
5th percentile: 1.992817211151123
10th percentile: 2.006141996383667
20th percentile: 2.032791566848755
30th percentile: 2.0676847457885743
40th percentile: 2.110821533203125
50th percentile: 2.153958320617676
60th percentile: 2.433284854888916
70th percentile: 2.712611389160156
80th percentile: 6.300870037078861
90th percentile: 13.19806079864502
95th percentile: 16.6466561794281
99th percentile: 19.405532484054564
mean time: 5.8254186630249025
%s, retrying in %s seconds...
Received healthy response to inference request in 2.1801321506500244s
Received healthy response to inference request in 2.0740745067596436s
Received healthy response to inference request in 2.8081881999969482s
Received healthy response to inference request in 2.4166505336761475s
Received healthy response to inference request in 1.9992337226867676s
5 requests
0 failed requests
5th percentile: 2.0142018795013428
10th percentile: 2.029170036315918
20th percentile: 2.0591063499450684
30th percentile: 2.0952860355377196
40th percentile: 2.1377090930938722
50th percentile: 2.1801321506500244
60th percentile: 2.2747395038604736
70th percentile: 2.369346857070923
80th percentile: 2.4949580669403075
90th percentile: 2.651573133468628
95th percentile: 2.729880666732788
99th percentile: 2.7925266933441164
mean time: 2.2956558227539063
Pipeline stage StressChecker completed in 43.77s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
Connection pool is full, discarding connection: %s. Connection pool size: %s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 5.61s
Shutdown handler de-registered
function_dodol_2024-11-21 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2542.72s
Shutdown handler de-registered
function_dodol_2024-11-21 status is now inactive due to auto deactivation removed underperforming models