developer_uid: NischayDnk
submission_id: chaiml-rm-pointwise-exp_60494_v1
model_name: chaiml-rm-pointwise-exp_60494_v1
model_group: ChaiML/rm-pointwise-exp1
status: torndown
timestamp: 2025-06-16T05:24:48+00:00
num_battles: 1
num_wins: 1
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/rm-pointwise-exp1-llama3-8bins
model_architecture: LlamaForSequenceClassification
model_num_parameters: 8030261248.0
best_of: 8
max_input_tokens: 256
max_output_tokens: 64
reward_model: default
display_name: chaiml-rm-pointwise-exp_60494_v1
ineligible_reason: num_battles<5000
is_internal_developer: False
language_model: ChaiML/rm-pointwise-exp1-llama3-8bins
model_size: 8B
ranking_group: single
us_pacific_date: 2025-06-15
win_ratio: 1.0
generation_params: {'temperature': 0.55, 'top_p': 0.95, 'min_p': 0.025, 'top_k': 60, 'presence_penalty': 0.35, 'frequency_penalty': 0.35, 'stopping_words': ['<|im_end|>', '\n'], 'max_input_tokens': 256, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-rm-pointwise-exp-60494-v1-mkmlizer
Waiting for job on chaiml-rm-pointwise-exp-60494-v1-mkmlizer to finish
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║  Version: 0.27.1+vampire_v3                                         ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║  belonging to:                                                      ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: Downloaded to shared memory in 28.049s
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpf8_detrw, device:0
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: quantized model in 19.245s
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: Processed model ChaiML/rm-pointwise-exp1-llama3-8bins in 47.294s
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: creating bucket guanaco-mkml-models
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v1
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v1/config.json
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v1/special_tokens_map.json
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v1/tokenizer_config.json
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v1/tokenizer.json
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v1/flywheel_model.0.safetensors
chaiml-rm-pointwise-exp-60494-v1-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:08, 31.79it/s]
Loading 0:   4%|▍         | 13/291 [00:00<00:05, 51.31it/s]
Loading 0:   7%|▋         | 19/291 [00:00<00:06, 41.55it/s]
Loading 0:   8%|▊         | 24/291 [00:00<00:06, 42.31it/s]
Loading 0:  10%|█         | 30/291 [00:00<00:05, 47.19it/s]
Loading 0:  12%|█▏        | 35/291 [00:00<00:05, 46.76it/s]
Loading 0:  14%|█▎        | 40/291 [00:00<00:05, 46.49it/s]
Loading 0:  16%|█▌        | 46/291 [00:01<00:05, 42.75it/s]
Loading 0:  18%|█▊        | 51/291 [00:01<00:05, 42.47it/s]
Loading 0:  20%|█▉        | 57/291 [00:01<00:04, 46.82it/s]
Loading 0:  21%|██▏       | 62/291 [00:01<00:04, 46.06it/s]
Loading 0:  23%|██▎       | 67/291 [00:01<00:04, 46.04it/s]
Loading 0:  25%|██▌       | 73/291 [00:01<00:05, 41.37it/s]
Loading 0:  27%|██▋       | 78/291 [00:01<00:05, 40.49it/s]
Loading 0:  29%|██▊       | 83/291 [00:02<00:06, 31.15it/s]
Loading 0:  30%|██▉       | 87/291 [00:02<00:06, 30.97it/s]
Loading 0:  32%|███▏      | 93/291 [00:02<00:05, 37.12it/s]
Loading 0:  34%|███▎      | 98/291 [00:02<00:05, 37.67it/s]
Loading 0:  35%|███▌      | 103/291 [00:02<00:04, 40.44it/s]
Loading 0:  37%|███▋      | 108/291 [00:02<00:04, 41.92it/s]
Loading 0:  39%|███▉      | 113/291 [00:02<00:05, 35.12it/s]
Loading 0:  42%|████▏     | 121/291 [00:02<00:03, 44.16it/s]
Loading 0:  44%|████▎     | 127/291 [00:03<00:03, 44.11it/s]
Loading 0:  45%|████▌     | 132/291 [00:03<00:03, 42.90it/s]
Loading 0:  48%|████▊     | 139/291 [00:03<00:03, 49.02it/s]
Loading 0:  50%|████▉     | 145/291 [00:03<00:03, 45.25it/s]
Loading 0:  52%|█████▏    | 150/291 [00:03<00:03, 44.77it/s]
Loading 0:  54%|█████▍    | 157/291 [00:03<00:02, 50.10it/s]
Loading 0:  56%|█████▌    | 163/291 [00:03<00:02, 46.68it/s]
Loading 0:  58%|█████▊    | 168/291 [00:03<00:02, 44.75it/s]
Loading 0:  61%|██████    | 177/291 [00:04<00:02, 48.50it/s]
Loading 0:  63%|██████▎   | 182/291 [00:04<00:02, 46.47it/s]
Loading 0:  64%|██████▍   | 187/291 [00:04<00:02, 34.79it/s]
Loading 0:  66%|██████▌   | 191/291 [00:04<00:02, 35.19it/s]
Loading 0:  67%|██████▋   | 195/291 [00:04<00:02, 34.59it/s]
Loading 0:  69%|██████▉   | 202/291 [00:04<00:02, 42.30it/s]
Loading 0:  71%|███████   | 207/291 [00:04<00:01, 42.97it/s]
Loading 0:  73%|███████▎  | 212/291 [00:05<00:02, 36.80it/s]
Loading 0:  76%|███████▌  | 220/291 [00:05<00:01, 45.54it/s]
Loading 0:  78%|███████▊  | 226/291 [00:05<00:01, 43.02it/s]
Loading 0:  79%|███████▉  | 231/291 [00:05<00:01, 43.59it/s]
Loading 0:  82%|████████▏ | 238/291 [00:05<00:01, 48.84it/s]
Loading 0:  84%|████████▍ | 244/291 [00:05<00:01, 45.21it/s]
Loading 0:  86%|████████▌ | 249/291 [00:05<00:00, 44.63it/s]
Loading 0:  88%|████████▊ | 256/291 [00:05<00:00, 50.90it/s]
Loading 0:  90%|█████████ | 262/291 [00:06<00:00, 48.33it/s]
Loading 0:  92%|█████████▏| 268/291 [00:06<00:00, 46.12it/s]
Loading 0:  94%|█████████▍| 274/291 [00:06<00:00, 48.06it/s]
Loading 0:  96%|█████████▌| 280/291 [00:06<00:00, 45.67it/s]
Loading 0:  98%|█████████▊| 285/291 [00:06<00:00, 45.56it/s]
Loading 0: 100%|█████████▉| 290/291 [00:06<00:00, 34.19it/s]
                                                            

Job chaiml-rm-pointwise-exp-60494-v1-mkmlizer completed after 73.91s with status: succeeded
Stopping job with name chaiml-rm-pointwise-exp-60494-v1-mkmlizer
Pipeline stage MKMLizer completed in 74.49s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-rm-pointwise-exp-60494-v1
Waiting for inference service chaiml-rm-pointwise-exp-60494-v1 to be ready
Inference service chaiml-rm-pointwise-exp-60494-v1 ready after 110.47871518135071s
Pipeline stage MKMLDeployer completed in 110.95s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 3.5605664253234863s
Received healthy response to inference request in 3.4535741806030273s
Received healthy response to inference request in 3.1439123153686523s
Received healthy response to inference request in 3.82320237159729s
5 requests
1 failed requests
5th percentile: 3.205844688415527
10th percentile: 3.2677770614624024
20th percentile: 3.3916418075561525
30th percentile: 3.474972629547119
40th percentile: 3.517769527435303
50th percentile: 3.5605664253234863
60th percentile: 3.665620803833008
70th percentile: 3.7706751823425293
80th percentile: 7.088618850708011
90th percentile: 13.619451808929444
95th percentile: 16.88486828804016
99th percentile: 19.497201471328736
mean time: 6.826308012008667
%s, retrying in %s seconds...
Received healthy response to inference request in 3.404179811477661s
Received healthy response to inference request in 4.09314489364624s
Received healthy response to inference request in 2.4271583557128906s
Received healthy response to inference request in 4.239295959472656s
Received healthy response to inference request in 2.97784423828125s
5 requests
0 failed requests
5th percentile: 2.5372955322265627
10th percentile: 2.6474327087402343
20th percentile: 2.867707061767578
30th percentile: 3.063111352920532
40th percentile: 3.2336455821990966
50th percentile: 3.404179811477661
60th percentile: 3.6797658443450927
70th percentile: 3.9553518772125242
80th percentile: 4.122375106811523
90th percentile: 4.18083553314209
95th percentile: 4.210065746307373
99th percentile: 4.2334499168396
mean time: 3.42832465171814
%s, retrying in %s seconds...
Received healthy response to inference request in 4.075912237167358s
Received healthy response to inference request in 2.0318493843078613s
Received healthy response to inference request in 4.980515718460083s
Received healthy response to inference request in 2.262091875076294s
Received healthy response to inference request in 4.375554800033569s
5 requests
0 failed requests
5th percentile: 2.077897882461548
10th percentile: 2.1239463806152346
20th percentile: 2.2160433769226073
30th percentile: 2.6248559474945066
40th percentile: 3.3503840923309327
50th percentile: 4.075912237167358
60th percentile: 4.195769262313843
70th percentile: 4.3156262874603275
80th percentile: 4.496546983718872
90th percentile: 4.7385313510894775
95th percentile: 4.85952353477478
99th percentile: 4.956317281723022
mean time: 3.5451848030090334
clean up pipeline due to error=DeploymentChecksError('Unacceptable 70th percentile latency 4.3156262874603275s')
Shutdown handler de-registered
chaiml-rm-pointwise-exp_60494_v1 status is now failed due to DeploymentManager action
chaiml-rm-pointwise-exp_60494_v1 status is now inactive due to system request
chaiml-rm-pointwise-exp_60494_v1 status is now torndown due to DeploymentManager action