chaiml-rm-pointwise-exp_60494

developer_uid: richhx
submission_id: chaiml-rm-pointwise-exp_60494_v5
model_name: chaiml-rm-pointwise-exp_60494_v5
model_group: ChaiML/rm-pointwise-exp1
status: torndown
timestamp: 2025-06-16T17:01:52+00:00
num_battles: 10038
num_wins: 4598
celo_rating: 1259.78
family_friendly_score: 0.5571999999999999
family_friendly_standard_error: 0.007024644617345421
submission_type: basic
model_repo: ChaiML/rm-pointwise-exp1-llama3-8bins
model_architecture: LlamaForSequenceClassification
model_num_parameters: 8030261248.0
best_of: 8
max_input_tokens: 256
max_output_tokens: 64
reward_model: default
display_name: chaiml-rm-pointwise-exp_60494_v5
is_internal_developer: True
language_model: ChaiML/rm-pointwise-exp1-llama3-8bins
model_size: 8B
ranking_group: single
us_pacific_date: 2025-06-16
win_ratio: 0.458059374377366
generation_params: {'temperature': 0.55, 'top_p': 0.95, 'min_p': 0.025, 'top_k': 60, 'presence_penalty': 0.35, 'frequency_penalty': 0.35, 'stopping_words': ['<|im_end|>', '\n'], 'max_input_tokens': 256, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-rm-pointwise-exp-60494-v5-mkmlizer
Waiting for job on chaiml-rm-pointwise-exp-60494-v5-mkmlizer to finish
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║  Version: 0.27.1+vampire_v3                                         ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║  belonging to:                                                      ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: Downloaded to shared memory in 25.836s
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpfjus_0l7, device:0
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: creating bucket guanaco-mkml-models
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v5
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v5/config.json
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v5/special_tokens_map.json
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v5/tokenizer_config.json
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v5/tokenizer.json
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v5/flywheel_model.0.safetensors
chaiml-rm-pointwise-exp-60494-v5-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:07, 36.63it/s]
Loading 0:   4%|▍         | 13/291 [00:00<00:05, 55.20it/s]
Loading 0:   7%|▋         | 19/291 [00:00<00:05, 46.02it/s]
Loading 0:   8%|▊         | 24/291 [00:00<00:05, 44.60it/s]
Loading 0:  10%|█         | 30/291 [00:00<00:05, 48.67it/s]
Loading 0:  12%|█▏        | 36/291 [00:00<00:05, 49.47it/s]
Loading 0:  14%|█▍        | 42/291 [00:00<00:05, 41.63it/s]
Loading 0:  17%|█▋        | 49/291 [00:01<00:05, 46.69it/s]
Loading 0:  19%|█▉        | 55/291 [00:01<00:05, 42.77it/s]
Loading 0:  21%|██        | 60/291 [00:01<00:05, 41.67it/s]
Loading 0:  23%|██▎       | 66/291 [00:01<00:04, 45.26it/s]
Loading 0:  24%|██▍       | 71/291 [00:01<00:04, 44.33it/s]
Loading 0:  26%|██▌       | 76/291 [00:01<00:04, 43.90it/s]
Loading 0:  28%|██▊       | 81/291 [00:01<00:04, 44.82it/s]
Loading 0:  30%|██▉       | 86/291 [00:02<00:07, 26.05it/s]
Loading 0:  32%|███▏      | 94/291 [00:02<00:05, 34.81it/s]
Loading 0:  34%|███▍      | 100/291 [00:02<00:05, 35.86it/s]
Loading 0:  36%|███▌      | 105/291 [00:02<00:05, 37.10it/s]
Loading 0:  38%|███▊      | 111/291 [00:02<00:04, 42.04it/s]
Loading 0:  40%|███▉      | 116/291 [00:02<00:04, 42.54it/s]
Loading 0:  42%|████▏     | 121/291 [00:02<00:03, 43.20it/s]
Loading 0:  44%|████▎     | 127/291 [00:03<00:03, 41.12it/s]
Loading 0:  45%|████▌     | 132/291 [00:03<00:03, 41.71it/s]
Loading 0:  47%|████▋     | 138/291 [00:03<00:03, 45.02it/s]
Loading 0:  49%|████▉     | 143/291 [00:03<00:03, 45.58it/s]
Loading 0:  51%|█████     | 148/291 [00:03<00:03, 46.31it/s]
Loading 0:  53%|█████▎    | 154/291 [00:03<00:03, 43.50it/s]
Loading 0:  55%|█████▍    | 159/291 [00:03<00:03, 43.02it/s]
Loading 0:  57%|█████▋    | 166/291 [00:03<00:02, 48.09it/s]
Loading 0:  59%|█████▉    | 171/291 [00:03<00:02, 48.55it/s]
Loading 0:  60%|██████    | 176/291 [00:04<00:02, 48.50it/s]
Loading 0:  62%|██████▏   | 181/291 [00:04<00:02, 38.54it/s]
Loading 0:  64%|██████▍   | 187/291 [00:04<00:03, 30.83it/s]
Loading 0:  66%|██████▌   | 191/291 [00:04<00:03, 31.95it/s]
Loading 0:  67%|██████▋   | 195/291 [00:04<00:02, 32.41it/s]
Loading 0:  69%|██████▉   | 202/291 [00:04<00:02, 39.57it/s]
Loading 0:  71%|███████▏  | 208/291 [00:05<00:02, 38.75it/s]
Loading 0:  73%|███████▎  | 213/291 [00:05<00:01, 39.76it/s]
Loading 0:  76%|███████▌  | 220/291 [00:05<00:01, 45.31it/s]
Loading 0:  78%|███████▊  | 226/291 [00:05<00:01, 42.92it/s]
Loading 0:  79%|███████▉  | 231/291 [00:05<00:01, 42.42it/s]
Loading 0:  82%|████████▏ | 238/291 [00:05<00:01, 47.31it/s]
Loading 0:  84%|████████▎ | 243/291 [00:05<00:01, 47.08it/s]
Loading 0:  85%|████████▌ | 248/291 [00:06<00:01, 38.57it/s]
Loading 0:  88%|████████▊ | 256/291 [00:06<00:00, 46.82it/s]
Loading 0:  90%|█████████ | 262/291 [00:06<00:00, 44.08it/s]
Loading 0:  92%|█████████▏| 267/291 [00:06<00:00, 43.35it/s]
Loading 0:  94%|█████████▍| 274/291 [00:06<00:00, 48.17it/s]
Loading 0:  96%|█████████▌| 280/291 [00:06<00:00, 44.46it/s]
Loading 0:  98%|█████████▊| 285/291 [00:06<00:00, 44.06it/s]
Loading 0: 100%|█████████▉| 290/291 [00:07<00:00, 31.14it/s]
                                                            
Job chaiml-rm-pointwise-exp-60494-v5-mkmlizer completed after 74.08s with status: succeeded
Stopping job with name chaiml-rm-pointwise-exp-60494-v5-mkmlizer
Pipeline stage MKMLizer completed in 74.62s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-rm-pointwise-exp-60494-v5
Waiting for inference service chaiml-rm-pointwise-exp-60494-v5 to be ready
Inference service chaiml-rm-pointwise-exp-60494-v5 ready after 100.65087103843689s
Pipeline stage MKMLDeployer completed in 101.22s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.867512226104736s
Received healthy response to inference request in 3.6396360397338867s
Received healthy response to inference request in 6.765580892562866s
Received healthy response to inference request in 3.340139865875244s
Received healthy response to inference request in 3.7867989540100098s
5 requests
0 failed requests
5th percentile: 3.4000391006469726
10th percentile: 3.459938335418701
20th percentile: 3.5797368049621583
30th percentile: 3.669068622589111
40th percentile: 3.7279337882995605
50th percentile: 3.7867989540100098
60th percentile: 4.2190842628479
70th percentile: 4.651369571685791
80th percentile: 5.247125959396363
90th percentile: 6.006353425979614
95th percentile: 6.38596715927124
99th percentile: 6.6896581459045406
mean time: 4.479933595657348
%s, retrying in %s seconds...
Received healthy response to inference request in 2.1624021530151367s
Received healthy response to inference request in 3.343045473098755s
Received healthy response to inference request in 3.353618860244751s
Received healthy response to inference request in 3.536576747894287s
Received healthy response to inference request in 7.312796354293823s
5 requests
0 failed requests
5th percentile: 2.3985308170318604
10th percentile: 2.634659481048584
20th percentile: 3.106916809082031
30th percentile: 3.345160150527954
40th percentile: 3.3493895053863527
50th percentile: 3.353618860244751
60th percentile: 3.4268020153045655
70th percentile: 3.49998517036438
80th percentile: 4.291820669174195
90th percentile: 5.802308511734009
95th percentile: 6.557552433013916
99th percentile: 7.161747570037842
mean time: 3.9416879177093507
Pipeline stage StressChecker completed in 45.25s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.78s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.68s
Shutdown handler de-registered
chaiml-rm-pointwise-exp_60494_v5 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3608.89s
Shutdown handler de-registered
chaiml-rm-pointwise-exp_60494_v5 status is now inactive due to auto deactivation removed underperforming models
chaiml-rm-pointwise-exp_60494_v5 status is now torndown due to DeploymentManager action