rirv938-96p-4ff-rirv938_38486

developer_uid: rirv938

submission_id: rirv938-96p-4ff-rirv938_38486_v1

model_name: rirv938-96p-4ff-rirv938_38486_v1

model_group: rirv938/96p_4ff_rirv938_

status: torndown

timestamp: 2025-04-03T16:59:42+00:00

num_battles: 7582

num_wins: 4326

celo_rating: 1353.56

family_friendly_score: 0.0

family_friendly_standard_error: 0.0

submission_type: basic

model_repo: rirv938/96p_4ff_rirv938_20k_100p_0ff_ri_19485_v1_cp2250_v3

model_architecture: MistralForCausalLM

model_num_parameters: 24096691200.0

best_of: 8

max_input_tokens: 768

max_output_tokens: 64

reward_model: default

latencies: [{'batch_size': 1, 'throughput': 0.3885934728642077, 'latency_mean': 2.5732857942581178, 'latency_p50': 2.566739082336426, 'latency_p90': 2.865937876701355}, {'batch_size': 3, 'throughput': 0.7891150867284826, 'latency_mean': 3.7880757951736452, 'latency_p50': 3.8028353452682495, 'latency_p90': 4.172553944587707}, {'batch_size': 5, 'throughput': 1.062646954638084, 'latency_mean': 4.69541414141655, 'latency_p50': 4.706522822380066, 'latency_p90': 5.262800192832946}, {'batch_size': 6, 'throughput': 1.1288829311540218, 'latency_mean': 5.280676040649414, 'latency_p50': 5.221379637718201, 'latency_p90': 5.964762282371521}, {'batch_size': 10, 'throughput': 1.3009536535152375, 'latency_mean': 7.607504514455795, 'latency_p50': 7.6471558809280396, 'latency_p90': 8.624907422065736}]

gpu_counts: {'NVIDIA RTX A6000': 1}

display_name: rirv938-96p-4ff-rirv938_38486_v1

is_internal_developer: True

language_model: rirv938/96p_4ff_rirv938_20k_100p_0ff_ri_19485_v1_cp2250_v3

model_size: 24B

ranking_group: single

throughput_3p7s: 0.77

us_pacific_date: 2025-04-03

win_ratio: 0.5705618570298074

generation_params: {'temperature': 0.9, 'top_p': 0.9, 'min_p': 0.6, 'top_k': 80, 'presence_penalty': 0.5, 'frequency_penalty': 0.5, 'stopping_words': ['###', 'You:', '</s>', '\n'], 'max_input_tokens': 768, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': '[INST]', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '[/INST]{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name rirv938-96p-4ff-rirv938-38486-v1-mkmlizer
Waiting for job on rirv938-96p-4ff-rirv938-38486-v1-mkmlizer to finish
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║     _____            __           __                                ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║       /___/                                                         ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║                                                                     ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║  Version: 0.12.8                                                    ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║  https://mk1.ai                                                     ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║                                                                     ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║  belonging to:                                                      ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║                                                                     ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║  Chai Research Corp.                                                ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║  Expiration: 2025-04-15 23:59:59                                    ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ║                                                                     ║
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission chaiml-sft-gemma2-28b-v_83370_v5: HTTPConnectionPool(host='chaiml-sft-gemma2-28b-v-83370-v5-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-sft-gemma2-28b-v_83370_v5: HTTPConnectionPool(host='chaiml-sft-gemma2-28b-v-83370-v5-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: Downloaded to shared memory in 189.749s
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp_7z2eld0, device:0
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: quantized model in 65.650s
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: Processed model rirv938/96p_4ff_rirv938_20k_100p_0ff_ri_19485_v1_cp2250_v3 in 255.400s
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: creating bucket guanaco-mkml-models
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/rirv938-96p-4ff-rirv938-38486-v1
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/rirv938-96p-4ff-rirv938-38486-v1/config.json
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/rirv938-96p-4ff-rirv938-38486-v1/special_tokens_map.json
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/rirv938-96p-4ff-rirv938-38486-v1/tokenizer_config.json
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/rirv938-96p-4ff-rirv938-38486-v1/flywheel_model.1.safetensors
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/rirv938-96p-4ff-rirv938-38486-v1/flywheel_model.0.safetensors
rirv938-96p-4ff-rirv938-38486-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|          | 3/363 [00:00<00:12, 28.08it/s]
Loading 0:   2%|▏         | 6/363 [00:00<00:26, 13.71it/s]
Loading 0:   3%|▎         | 11/363 [00:00<00:16, 20.99it/s]
Loading 0:   4%|▍         | 14/363 [00:00<00:25, 13.45it/s]
Loading 0:   4%|▍         | 16/363 [00:01<00:26, 13.19it/s]
Loading 0:   6%|▌         | 21/363 [00:01<00:20, 16.78it/s]
Loading 0:   6%|▋         | 23/363 [00:01<00:25, 13.29it/s]
Loading 0:   8%|▊         | 28/363 [00:01<00:17, 19.08it/s]
Loading 0:   9%|▉         | 32/363 [00:01<00:14, 22.53it/s]
Loading 0:  10%|▉         | 35/363 [00:02<00:20, 15.68it/s]
Loading 0:  10%|█         | 38/363 [00:02<00:20, 16.08it/s]
Loading 0:  11%|█▏        | 41/363 [00:02<00:26, 12.21it/s]
Loading 0:  13%|█▎        | 46/363 [00:02<00:18, 16.85it/s]
Loading 0:  14%|█▍        | 50/363 [00:02<00:15, 19.70it/s]
Loading 0:  15%|█▍        | 53/363 [00:03<00:21, 14.36it/s]
Loading 0:  15%|█▌        | 56/363 [00:03<00:20, 15.03it/s]
Loading 0:  16%|█▌        | 58/363 [00:03<00:22, 13.38it/s]
Loading 0:  17%|█▋        | 60/363 [00:03<00:23, 12.71it/s]
Loading 0:  18%|█▊        | 64/363 [00:04<00:18, 16.09it/s]
Loading 0:  19%|█▊        | 68/363 [00:04<00:15, 19.47it/s]
Loading 0:  20%|█▉        | 71/363 [00:04<00:21, 13.85it/s]
Loading 0:  20%|██        | 73/363 [00:04<00:22, 13.17it/s]
Loading 0:  21%|██        | 75/363 [00:04<00:20, 13.86it/s]
Loading 0:  21%|██        | 77/363 [00:05<00:26, 10.95it/s]
Loading 0:  23%|██▎       | 82/363 [00:05<00:16, 16.55it/s]
Loading 0:  24%|██▎       | 86/363 [00:05<00:14, 19.59it/s]
Loading 0:  25%|██▍       | 89/363 [00:05<00:20, 13.27it/s]
Loading 0:  25%|██▌       | 91/363 [00:06<00:21, 12.62it/s]
Loading 0:  26%|██▌       | 95/363 [00:06<00:16, 16.16it/s]
Loading 0:  27%|██▋       | 99/363 [00:06<00:13, 19.25it/s]
Loading 0:  28%|██▊       | 102/363 [00:06<00:14, 17.98it/s]
Loading 0:  29%|██▉       | 105/363 [00:06<00:18, 13.88it/s]
Loading 0:  29%|██▉       | 107/363 [00:07<00:20, 12.63it/s]
Loading 0:  30%|███       | 109/363 [00:07<00:20, 12.17it/s]
Loading 0:  31%|███       | 111/363 [00:07<00:19, 12.94it/s]
Loading 0:  31%|███       | 113/363 [00:07<00:23, 10.50it/s]
Loading 0:  33%|███▎      | 118/363 [00:07<00:15, 16.01it/s]
Loading 0:  34%|███▎      | 122/363 [00:07<00:12, 19.16it/s]
Loading 0:  34%|███▍      | 125/363 [00:08<00:18, 12.96it/s]
Loading 0:  35%|███▍      | 127/363 [00:08<00:18, 12.53it/s]
Loading 0:  36%|███▌      | 129/363 [00:08<00:17, 13.53it/s]
Loading 0:  36%|███▌      | 131/363 [00:08<00:20, 11.28it/s]
Loading 0:  38%|███▊      | 137/363 [00:09<00:11, 19.07it/s]
Loading 0:  39%|███▉      | 141/363 [00:09<00:12, 18.39it/s]
Loading 0:  40%|███▉      | 144/363 [00:09<00:15, 14.03it/s]
Loading 0:  40%|████      | 147/363 [00:09<00:13, 15.98it/s]
Loading 0:  41%|████▏     | 150/363 [00:10<00:15, 14.00it/s]
Loading 0:  42%|████▏     | 154/363 [00:10<00:11, 17.65it/s]
Loading 0:  44%|████▎     | 158/363 [00:10<00:09, 21.11it/s]
Loading 0:  44%|████▍     | 161/363 [00:10<00:13, 14.85it/s]
Loading 0:  45%|████▌     | 164/363 [00:10<00:12, 15.55it/s]
Loading 0:  46%|████▌     | 167/363 [00:11<00:16, 12.10it/s]
Loading 0:  47%|████▋     | 172/363 [00:11<00:11, 16.96it/s]
Loading 0:  48%|████▊     | 176/363 [00:11<00:09, 19.91it/s]
Loading 0:  49%|████▉     | 179/363 [00:11<00:12, 14.44it/s]
Loading 0:  50%|█████     | 182/363 [00:11<00:12, 15.01it/s]
Loading 0:  51%|█████     | 184/363 [00:12<00:13, 13.37it/s]
Loading 0:  51%|█████     | 186/363 [00:12<00:13, 12.76it/s]
Loading 0:  52%|█████▏    | 190/363 [00:12<00:10, 16.77it/s]
Loading 0:  53%|█████▎    | 194/363 [00:12<00:08, 20.25it/s]
Loading 0:  54%|█████▍    | 197/363 [00:12<00:11, 14.13it/s]
Loading 0:  55%|█████▍    | 199/363 [00:13<00:12, 13.37it/s]
Loading 0:  55%|█████▌    | 201/363 [00:28<04:45,  1.76s/it]
Loading 0:  56%|█████▌    | 203/363 [00:28<03:38,  1.37s/it]
Loading 0:  57%|█████▋    | 207/363 [00:28<02:06,  1.23it/s]
Loading 0:  58%|█████▊    | 210/363 [00:28<01:28,  1.74it/s]
Loading 0:  59%|█████▊    | 213/363 [00:28<01:04,  2.33it/s]
Loading 0:  59%|█████▉    | 215/363 [00:29<00:52,  2.82it/s]
Loading 0:  60%|█████▉    | 217/363 [00:29<00:42,  3.42it/s]
Loading 0:  60%|██████    | 219/363 [00:29<00:33,  4.28it/s]
Loading 0:  61%|██████    | 221/363 [00:29<00:30,  4.69it/s]
Loading 0:  62%|██████▏   | 226/363 [00:29<00:16,  8.27it/s]
Loading 0:  63%|██████▎   | 230/363 [00:29<00:11, 11.25it/s]
Loading 0:  64%|██████▍   | 233/363 [00:30<00:13,  9.56it/s]
Loading 0:  65%|██████▍   | 235/363 [00:30<00:13,  9.58it/s]
Loading 0:  65%|██████▌   | 237/363 [00:30<00:11, 10.60it/s]
Loading 0:  66%|██████▌   | 239/363 [00:31<00:13,  9.32it/s]
Loading 0:  67%|██████▋   | 244/363 [00:31<00:08, 14.60it/s]
Loading 0:  68%|██████▊   | 248/363 [00:31<00:06, 18.19it/s]
Loading 0:  69%|██████▉   | 251/363 [00:31<00:08, 13.07it/s]
Loading 0:  70%|██████▉   | 253/363 [00:31<00:08, 12.60it/s]
Loading 0:  70%|███████   | 255/363 [00:31<00:08, 13.43it/s]
Loading 0:  71%|███████   | 257/363 [00:32<00:09, 10.98it/s]
Loading 0:  72%|███████▏  | 262/363 [00:32<00:06, 16.70it/s]
Loading 0:  73%|███████▎  | 266/363 [00:32<00:04, 20.07it/s]
Loading 0:  74%|███████▍  | 269/363 [00:32<00:06, 13.89it/s]
Loading 0:  75%|███████▍  | 271/363 [00:33<00:07, 13.12it/s]
Loading 0:  75%|███████▌  | 273/363 [00:33<00:06, 13.87it/s]
Loading 0:  76%|███████▌  | 275/363 [00:33<00:07, 11.28it/s]
Loading 0:  77%|███████▋  | 280/363 [00:33<00:04, 16.66it/s]
Loading 0:  78%|███████▊  | 284/363 [00:33<00:03, 19.89it/s]
Loading 0:  79%|███████▉  | 287/363 [00:34<00:05, 14.24it/s]
Loading 0:  80%|███████▉  | 289/363 [00:34<00:05, 13.65it/s]
Loading 0:  80%|████████  | 291/363 [00:34<00:04, 14.50it/s]
Loading 0:  81%|████████  | 293/363 [00:34<00:05, 11.86it/s]
Loading 0:  82%|████████▏ | 298/363 [00:34<00:03, 17.82it/s]
Loading 0:  83%|████████▎ | 302/363 [00:34<00:02, 21.37it/s]
Loading 0:  84%|████████▍ | 305/363 [00:35<00:03, 15.39it/s]
Loading 0:  85%|████████▍ | 308/363 [00:35<00:03, 15.82it/s]
Loading 0:  85%|████████▌ | 310/363 [00:35<00:03, 13.85it/s]
Loading 0:  86%|████████▌ | 312/363 [00:35<00:03, 13.13it/s]
Loading 0:  87%|████████▋ | 316/363 [00:35<00:02, 16.98it/s]
Loading 0:  88%|████████▊ | 320/363 [00:36<00:02, 20.27it/s]
Loading 0:  89%|████████▉ | 323/363 [00:36<00:02, 13.96it/s]
Loading 0:  90%|████████▉ | 325/363 [00:36<00:02, 13.27it/s]
Loading 0:  90%|█████████ | 327/363 [00:36<00:02, 13.89it/s]
Loading 0:  91%|█████████ | 329/363 [00:36<00:03, 11.32it/s]
Loading 0:  92%|█████████▏| 334/363 [00:37<00:01, 16.96it/s]
Loading 0:  93%|█████████▎| 338/363 [00:37<00:01, 20.07it/s]
Loading 0:  94%|█████████▍| 341/363 [00:37<00:01, 14.25it/s]
Loading 0:  94%|█████████▍| 343/363 [00:37<00:01, 13.66it/s]
Loading 0:  95%|█████████▌| 345/363 [00:37<00:01, 14.55it/s]
Loading 0:  96%|█████████▌| 347/363 [00:38<00:01, 11.77it/s]
Loading 0:  97%|█████████▋| 352/363 [00:38<00:00, 18.22it/s]
Loading 0:  98%|█████████▊| 357/363 [00:45<00:03,  1.67it/s]
Loading 0:  99%|█████████▉| 360/363 [00:45<00:01,  2.17it/s]
Loading 0: 100%|█████████▉| 362/363 [00:45<00:00,  2.57it/s]
                                                            
Job rirv938-96p-4ff-rirv938-38486-v1-mkmlizer completed after 288.81s with status: succeeded
Stopping job with name rirv938-96p-4ff-rirv938-38486-v1-mkmlizer
Pipeline stage MKMLizer completed in 289.25s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service rirv938-96p-4ff-rirv938-38486-v1
Waiting for inference service rirv938-96p-4ff-rirv938-38486-v1 to be ready
Failed to get response for submission chaiml-sft-gemma2-28b-v_83370_v5: HTTPConnectionPool(host='chaiml-sft-gemma2-28b-v-83370-v5-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission chaiml-sft-gemma2-28b-v_83370_v5: HTTPConnectionPool(host='chaiml-sft-gemma2-28b-v-83370-v5-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service rirv938-96p-4ff-rirv938-38486-v1 ready after 200.65529131889343s
Pipeline stage MKMLDeployer completed in 201.20s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.9995968341827393s
Received healthy response to inference request in 2.474003553390503s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Unable to obtain autoscaler alexdaoud-trainer-bajem_28556_v2 due to error: Failed to establish a connection: HTTPSConnectionPool(host='guanaco-autoscalers.firebaseio.com', port=443): Max retries exceeded with url: /autoscalers/alexdaoud-trainer-bajem_28556_v2/actions.json (Caused by ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))). Falling back to minimal defaults: logs=None autoscaler_id='fallback' status='deployed' metric=<function null_metric at 0x7357bac5e200> observations=[] max_observations_length=1000 target=0.0 actions=[SubmissionAutoscalerParameterAction(generation_parameter='max_input_tokens', current_scale=128, min_scale=512, max_scale=2048)] scale_up_policy=SubmissionAutoscalerPolicy(max_percent_change=0.25, period=0.1, stabilisation_window=0, tolerance=0.1) scale_down_policy=SubmissionAutoscalerPolicy(max_percent_change=0.25, period=0.1, stabilisation_window=0, tolerance=0.1) panic_policy=SubmissionAutoscalerPanicPolicy(max_percent_change=1.0, period=0.1, stabilisation_window=0, tolerance=-1e-06, z_score=3, num_observations=5, num_historical_observations=100)
Received healthy response to inference request in 2.7606022357940674s
Received healthy response to inference request in 2.6420180797576904s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Received healthy response to inference request in 2.516387939453125s
5 requests
0 failed requests
5th percentile: 2.4824804306030273
10th percentile: 2.4909573078155516
20th percentile: 2.5079110622406007
30th percentile: 2.541513967514038
40th percentile: 2.591766023635864
50th percentile: 2.6420180797576904
60th percentile: 2.689451742172241
70th percentile: 2.736885404586792
80th percentile: 2.8084011554718016
90th percentile: 2.9039989948272704
95th percentile: 2.951797914505005
99th percentile: 2.9900370502471922
mean time: 2.678521728515625
Pipeline stage StressChecker completed in 15.48s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.97s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.92s
Shutdown handler de-registered
rirv938-96p-4ff-rirv938_38486_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
rirv938-96p-4ff-rirv938_38486_v1 status is now inactive due to auto deactivation removed underperforming models
rirv938-96p-4ff-rirv938_38486_v1 status is now torndown due to DeploymentManager action