submission_id: chaiml-rm-pointwise-exp_60494_v2
developer_uid: NischayDnk
status: failed
model_repo: ChaiML/rm-pointwise-exp1-llama3-8bins
generation_params: {'temperature': 0.55, 'top_p': 0.95, 'min_p': 0.025, 'top_k': 60, 'presence_penalty': 0.35, 'frequency_penalty': 0.35, 'stopping_words': ['<|im_end|>', '\n'], 'max_input_tokens': 224, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
timestamp: 2025-06-16T05:50:51+00:00
model_name: chaiml-rm-pointwise-exp_60494_v2
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-rm-pointwise-exp-60494-v2-mkmlizer
Waiting for job on chaiml-rm-pointwise-exp-60494-v2-mkmlizer to finish
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║  Version: 0.27.1+vampire_v3                                         ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║  belonging to:                                                      ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ║                                                                     ║
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: Downloaded to shared memory in 22.051s
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmp23hb6pfl, device:0
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: quantized model in 21.673s
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: Processed model ChaiML/rm-pointwise-exp1-llama3-8bins in 43.724s
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: creating bucket guanaco-mkml-models
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v2
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v2/config.json
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v2/special_tokens_map.json
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v2/tokenizer.json
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-rm-pointwise-exp-60494-v2/flywheel_model.0.safetensors
chaiml-rm-pointwise-exp-60494-v2-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:08, 33.47it/s]
Loading 0:   4%|▍         | 12/291 [00:00<00:05, 51.10it/s]
Loading 0:   6%|▌         | 18/291 [00:00<00:05, 49.73it/s]
Loading 0:   8%|▊         | 24/291 [00:00<00:06, 40.20it/s]
Loading 0:  11%|█         | 31/291 [00:00<00:05, 46.14it/s]
Loading 0:  12%|█▏        | 36/291 [00:00<00:05, 45.72it/s]
Loading 0:  14%|█▍        | 41/291 [00:01<00:07, 34.71it/s]
Loading 0:  16%|█▋        | 48/291 [00:01<00:05, 42.05it/s]
Loading 0:  18%|█▊        | 53/291 [00:01<00:05, 41.47it/s]
Loading 0:  20%|█▉        | 58/291 [00:01<00:05, 41.98it/s]
Loading 0:  22%|██▏       | 63/291 [00:01<00:05, 43.53it/s]
Loading 0:  23%|██▎       | 68/291 [00:01<00:06, 35.13it/s]
Loading 0:  26%|██▌       | 75/291 [00:01<00:05, 41.98it/s]
Loading 0:  27%|██▋       | 80/291 [00:01<00:05, 42.00it/s]
Loading 0:  29%|██▉       | 85/291 [00:02<00:07, 27.06it/s]
Loading 0:  31%|███       | 89/291 [00:02<00:06, 29.14it/s]
Loading 0:  32%|███▏      | 94/291 [00:02<00:06, 32.23it/s]
Loading 0:  34%|███▎      | 98/291 [00:02<00:05, 32.92it/s]
Loading 0:  35%|███▌      | 103/291 [00:02<00:05, 35.29it/s]
Loading 0:  37%|███▋      | 107/291 [00:02<00:05, 35.66it/s]
Loading 0:  38%|███▊      | 112/291 [00:02<00:04, 37.91it/s]
Loading 0:  40%|████      | 117/291 [00:03<00:04, 40.82it/s]
Loading 0:  42%|████▏     | 122/291 [00:03<00:04, 33.91it/s]
Loading 0:  44%|████▍     | 129/291 [00:03<00:03, 41.74it/s]
Loading 0:  46%|████▌     | 134/291 [00:03<00:05, 26.71it/s]
Loading 0:  47%|████▋     | 138/291 [00:03<00:05, 28.92it/s]
Loading 0:  49%|████▉     | 142/291 [00:03<00:05, 29.38it/s]
Loading 0:  51%|█████     | 147/291 [00:04<00:04, 33.51it/s]
Loading 0:  52%|█████▏    | 151/291 [00:04<00:04, 34.45it/s]
Loading 0:  54%|█████▍    | 157/291 [00:04<00:03, 39.68it/s]
Loading 0:  56%|█████▌    | 162/291 [00:04<00:03, 41.56it/s]
Loading 0:  57%|█████▋    | 167/291 [00:04<00:03, 34.78it/s]
Loading 0:  60%|██████    | 175/291 [00:04<00:02, 45.05it/s]
Loading 0:  62%|██████▏   | 181/291 [00:04<00:02, 39.26it/s]
Loading 0:  64%|██████▍   | 187/291 [00:05<00:03, 29.34it/s]
Loading 0:  66%|██████▌   | 191/291 [00:05<00:03, 30.24it/s]
Loading 0:  67%|██████▋   | 195/291 [00:05<00:03, 30.57it/s]
Loading 0:  69%|██████▉   | 201/291 [00:05<00:02, 36.46it/s]
Loading 0:  71%|███████   | 206/291 [00:05<00:02, 37.76it/s]
Loading 0:  73%|███████▎  | 211/291 [00:05<00:01, 40.27it/s]
Loading 0:  75%|███████▍  | 217/291 [00:06<00:02, 31.69it/s]
Loading 0:  76%|███████▌  | 221/291 [00:06<00:02, 31.37it/s]
Loading 0:  79%|███████▊  | 229/291 [00:06<00:01, 40.49it/s]
Loading 0:  80%|████████  | 234/291 [00:06<00:01, 42.15it/s]
Loading 0:  82%|████████▏ | 239/291 [00:06<00:01, 33.43it/s]
Loading 0:  84%|████████▍ | 244/291 [00:06<00:01, 36.83it/s]
Loading 0:  86%|████████▌ | 249/291 [00:06<00:01, 35.11it/s]
Loading 0:  87%|████████▋ | 253/291 [00:06<00:01, 36.12it/s]
Loading 0:  88%|████████▊ | 257/291 [00:07<00:01, 32.90it/s]
Loading 0:  90%|█████████ | 262/291 [00:07<00:00, 36.73it/s]
Loading 0:  91%|█████████▏| 266/291 [00:07<00:00, 34.18it/s]
Loading 0:  94%|█████████▍| 273/291 [00:07<00:00, 41.17it/s]
Loading 0:  96%|█████████▌| 278/291 [00:07<00:00, 40.25it/s]
Loading 0:  97%|█████████▋| 283/291 [00:07<00:00, 32.96it/s]
Loading 0:  99%|█████████▊| 287/291 [00:08<00:00, 22.89it/s]
                                                            

Job chaiml-rm-pointwise-exp-60494-v2-mkmlizer completed after 73.9s with status: succeeded
Stopping job with name chaiml-rm-pointwise-exp-60494-v2-mkmlizer
Pipeline stage MKMLizer completed in 74.46s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-rm-pointwise-exp-60494-v2
Waiting for inference service chaiml-rm-pointwise-exp-60494-v2 to be ready
Inference service chaiml-rm-pointwise-exp-60494-v2 ready after 100.43557214736938s
Pipeline stage MKMLDeployer completed in 101.48s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 4.359129428863525s
Received healthy response to inference request in 2.3193368911743164s
Received healthy response to inference request in 3.7781269550323486s
Received healthy response to inference request in 3.9715025424957275s
Received healthy response to inference request in 2.2603342533111572s
5 requests
0 failed requests
5th percentile: 2.272134780883789
10th percentile: 2.283935308456421
20th percentile: 2.3075363636016846
30th percentile: 2.611094903945923
40th percentile: 3.194610929489136
50th percentile: 3.7781269550323486
60th percentile: 3.8554771900177003
70th percentile: 3.932827425003052
80th percentile: 4.049027919769287
90th percentile: 4.204078674316406
95th percentile: 4.281604051589966
99th percentile: 4.3436243534088135
mean time: 3.337686014175415
%s, retrying in %s seconds...
Received healthy response to inference request in 3.5741772651672363s
Received healthy response to inference request in 5.288232326507568s
Received healthy response to inference request in 3.5119590759277344s
Received healthy response to inference request in 3.3931429386138916s
Received healthy response to inference request in 4.698882818222046s
5 requests
0 failed requests
5th percentile: 3.41690616607666
10th percentile: 3.440669393539429
20th percentile: 3.488195848464966
30th percentile: 3.524402713775635
40th percentile: 3.5492899894714354
50th percentile: 3.5741772651672363
60th percentile: 4.02405948638916
70th percentile: 4.4739417076110835
80th percentile: 4.81675271987915
90th percentile: 5.05249252319336
95th percentile: 5.170362424850464
99th percentile: 5.264658346176147
mean time: 4.093278884887695
%s, retrying in %s seconds...
Received healthy response to inference request in 2.6454436779022217s
Received healthy response to inference request in 3.307173013687134s
Received healthy response to inference request in 4.868031740188599s
Received healthy response to inference request in 4.231562614440918s
Received healthy response to inference request in 4.6834611892700195s
5 requests
0 failed requests
5th percentile: 2.7777895450592043
10th percentile: 2.9101354122161864
20th percentile: 3.174827146530151
30th percentile: 3.4920509338378904
40th percentile: 3.861806774139404
50th percentile: 4.231562614440918
60th percentile: 4.412322044372559
70th percentile: 4.593081474304199
80th percentile: 4.720375299453735
90th percentile: 4.794203519821167
95th percentile: 4.831117630004883
99th percentile: 4.860648918151855
mean time: 3.9471344470977785
clean up pipeline due to error=DeploymentChecksError('Unacceptable 70th percentile latency 4.593081474304199s')
Shutdown handler de-registered
chaiml-rm-pointwise-exp_60494_v2 status is now failed due to DeploymentManager action