Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.779017448425293s
Received healthy response to inference request in 3.960042715072632s
Received healthy response to inference request in 3.928415536880493s
Received healthy response to inference request in 2.4986281394958496s
Received healthy response to inference request in 4.112584114074707s
5 requests
0 failed requests
5th percentile: 2.554706001281738
10th percentile: 2.6107838630676268
20th percentile: 2.7229395866394044
30th percentile: 3.008897066116333
40th percentile: 3.468656301498413
50th percentile: 3.928415536880493
60th percentile: 3.9410664081573485
70th percentile: 3.953717279434204
80th percentile: 3.990550994873047
90th percentile: 4.051567554473877
95th percentile: 4.082075834274292
99th percentile: 4.106482458114624
mean time: 3.455737590789795
%s, retrying in %s seconds...
Received healthy response to inference request in 3.536536455154419s
Received healthy response to inference request in 3.3870162963867188s
Received healthy response to inference request in 3.756509304046631s
Received healthy response to inference request in 3.4304256439208984s
Failed to get response for submission chaiml-sft-gemma2-28b-v_83370_v5: HTTPConnectionPool(host='chaiml-sft-gemma2-28b-v-83370-v5-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Received healthy response to inference request in 3.8008646965026855s
5 requests
0 failed requests
5th percentile: 3.3956981658935548
10th percentile: 3.404380035400391
20th percentile: 3.4217437744140624
30th percentile: 3.4516478061676024
40th percentile: 3.4940921306610107
50th percentile: 3.536536455154419
60th percentile: 3.6245255947113035
70th percentile: 3.7125147342681886
80th percentile: 3.765380382537842
90th percentile: 3.7831225395202637
95th percentile: 3.7919936180114746
99th percentile: 3.7990904808044434
mean time: 3.5822704792022706
%s, retrying in %s seconds...
Received healthy response to inference request in 2.571005344390869s
Received healthy response to inference request in 3.076932191848755s
Received healthy response to inference request in 2.0058348178863525s
Received healthy response to inference request in 3.697352647781372s
Received healthy response to inference request in 2.576789140701294s
5 requests
0 failed requests
5th percentile: 2.118868923187256
10th percentile: 2.2319030284881594
20th percentile: 2.4579712390899657
30th percentile: 2.5721621036529543
40th percentile: 2.574475622177124
50th percentile: 2.576789140701294
60th percentile: 2.7768463611602785
70th percentile: 2.9769035816192626
80th percentile: 3.2010162830352784
90th percentile: 3.449184465408325
95th percentile: 3.5732685565948485
99th percentile: 3.6725358295440675
mean time: 2.7855828285217283
Pipeline stage StressChecker completed in 52.32s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.63s
Shutdown handler de-registered
function_nofun_2025-04-03 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 4008.40s
Shutdown handler de-registered
function_nofun_2025-04-03 status is now inactive due to auto deactivation removed underperforming models