servicenow-ai-apriel-ne_30954

developer_uid: fengjiayi
submission_id: servicenow-ai-apriel-ne_30954_v1
model_name: servicenow-ai-apriel-ne_30954_v1
model_group: ServiceNow-AI/Apriel-Nem
status: torndown
timestamp: 2025-05-30T11:39:41+00:00
num_battles: 7746
num_wins: 3381
celo_rating: 1240.9
family_friendly_score: 0.6194
family_friendly_standard_error: 0.006866493136965914
submission_type: basic
model_repo: ServiceNow-AI/Apriel-Nemotron-15b-Thinker
model_architecture: MistralForCausalLM
model_num_parameters: 15629542400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
reward_model: default
display_name: servicenow-ai-apriel-ne_30954_v1
is_internal_developer: False
language_model: ServiceNow-AI/Apriel-Nemotron-15b-Thinker
model_size: 16B
ranking_group: single
us_pacific_date: 2025-05-30
win_ratio: 0.4364833462432223
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name servicenow-ai-apriel-ne-30954-v1-mkmlizer
Waiting for job on servicenow-ai-apriel-ne-30954-v1-mkmlizer to finish
Failed to get response for submission chaiml-nis-qwen32b-dpoe_75458_v3: ('http://chaiml-nis-qwen32b-dpoe-75458-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:41212->127.0.0.1:8080: read: connection reset by peer\n')
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║                                                                     ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║                                                                     ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║  Version: 0.27.1+vampire_v3                                         ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║  https://mk1.ai                                                     ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║                                                                     ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║  belonging to:                                                      ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║                                                                     ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║  Chai Research Corp.                                                ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ║                                                                     ║
servicenow-ai-apriel-ne-30954-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
servicenow-ai-apriel-ne-30954-v1-mkmlizer: Downloaded to shared memory in 39.140s
servicenow-ai-apriel-ne-30954-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp_rjscnhp, device:0
servicenow-ai-apriel-ne-30954-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
servicenow-ai-apriel-ne-30954-v1-mkmlizer: quantized model in 35.444s
servicenow-ai-apriel-ne-30954-v1-mkmlizer: Processed model ServiceNow-AI/Apriel-Nemotron-15b-Thinker in 74.584s
servicenow-ai-apriel-ne-30954-v1-mkmlizer: creating bucket guanaco-mkml-models
servicenow-ai-apriel-ne-30954-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
servicenow-ai-apriel-ne-30954-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/servicenow-ai-apriel-ne-30954-v1
servicenow-ai-apriel-ne-30954-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/servicenow-ai-apriel-ne-30954-v1/config.json
servicenow-ai-apriel-ne-30954-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.1.safetensors s3://guanaco-mkml-models/servicenow-ai-apriel-ne-30954-v1/flywheel_model.1.safetensors
servicenow-ai-apriel-ne-30954-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/servicenow-ai-apriel-ne-30954-v1/flywheel_model.0.safetensors
servicenow-ai-apriel-ne-30954-v1-mkmlizer: 
Loading 0:   0%|          | 0/453 [00:00<?, ?it/s]
Loading 0:   0%|          | 2/453 [00:06<22:59,  3.06s/it]
Loading 0:   1%|▏         | 6/453 [00:06<06:08,  1.21it/s]
Loading 0:   3%|▎         | 13/453 [00:06<02:11,  3.35it/s]
Loading 0:   4%|▍         | 17/453 [00:06<01:29,  4.87it/s]
Loading 0:   5%|▌         | 23/453 [00:06<00:54,  7.93it/s]
Loading 0:   6%|▋         | 29/453 [00:06<00:37, 11.43it/s]
Loading 0:   8%|▊         | 34/453 [00:06<00:28, 14.79it/s]
Loading 0:   9%|▉         | 40/453 [00:07<00:24, 16.91it/s]
Loading 0:  10%|▉         | 44/453 [00:07<00:21, 19.43it/s]
Loading 0:  11%|█         | 50/453 [00:07<00:16, 25.11it/s]
Loading 0:  12%|█▏        | 56/453 [00:07<00:13, 29.23it/s]
Loading 0:  13%|█▎        | 61/453 [00:07<00:12, 31.66it/s]
Loading 0:  15%|█▌        | 68/453 [00:07<00:09, 38.52it/s]
Loading 0:  16%|█▌        | 73/453 [00:07<00:09, 40.03it/s]
Loading 0:  17%|█▋        | 78/453 [00:08<00:10, 35.56it/s]
Loading 0:  19%|█▉        | 86/453 [00:08<00:08, 43.82it/s]
Loading 0:  20%|██        | 92/453 [00:08<00:08, 42.57it/s]
Loading 0:  21%|██▏       | 97/453 [00:08<00:08, 42.45it/s]
Loading 0:  23%|██▎       | 104/453 [00:08<00:07, 46.99it/s]
Loading 0:  24%|██▍       | 110/453 [00:08<00:07, 44.78it/s]
Loading 0:  25%|██▌       | 115/453 [00:08<00:07, 43.92it/s]
Loading 0:  27%|██▋       | 121/453 [00:09<00:09, 33.98it/s]
Loading 0:  28%|██▊       | 125/453 [00:09<00:09, 33.83it/s]
Loading 0:  29%|██▉       | 131/453 [00:09<00:08, 38.43it/s]
Loading 0:  30%|███       | 136/453 [00:09<00:07, 39.97it/s]
Loading 0:  31%|███       | 141/453 [00:09<00:09, 33.81it/s]
Loading 0:  33%|███▎      | 148/453 [00:09<00:07, 41.18it/s]
Loading 0:  34%|███▍      | 153/453 [00:09<00:07, 41.78it/s]
Loading 0:  35%|███▍      | 158/453 [00:09<00:06, 42.86it/s]
Loading 0:  36%|███▌      | 164/453 [00:10<00:06, 41.46it/s]
Loading 0:  37%|███▋      | 169/453 [00:10<00:06, 40.78it/s]
Loading 0:  39%|███▉      | 176/453 [00:10<00:06, 45.88it/s]
Loading 0:  40%|████      | 182/453 [00:10<00:06, 43.98it/s]
Loading 0:  41%|████▏     | 187/453 [00:10<00:06, 42.25it/s]
Loading 0:  43%|████▎     | 193/453 [00:10<00:05, 46.37it/s]
Loading 0:  44%|████▎     | 198/453 [00:10<00:05, 44.59it/s]
Loading 0:  45%|████▍     | 203/453 [00:11<00:07, 31.37it/s]
Loading 0:  46%|████▌     | 207/453 [00:11<00:07, 32.72it/s]
Loading 0:  47%|████▋     | 211/453 [00:11<00:07, 34.19it/s]
Loading 0:  47%|████▋     | 215/453 [00:11<00:06, 34.77it/s]
Loading 0:  49%|████▊     | 220/453 [00:11<00:06, 37.80it/s]
Loading 0:  50%|████▉     | 225/453 [00:11<00:05, 39.05it/s]
Loading 0:  51%|█████     | 230/453 [00:11<00:05, 39.77it/s]
Loading 0:  52%|█████▏    | 235/453 [00:11<00:05, 41.79it/s]
Loading 0:  53%|█████▎    | 240/453 [00:12<00:06, 33.57it/s]
Loading 0:  55%|█████▍    | 247/453 [00:12<00:04, 41.49it/s]
Loading 0:  56%|█████▌    | 252/453 [00:12<00:04, 41.85it/s]
Loading 0:  57%|█████▋    | 257/453 [00:12<00:04, 41.69it/s]
Loading 0:  58%|█████▊    | 262/453 [00:12<00:04, 43.62it/s]
Loading 0:  59%|█████▉    | 267/453 [00:12<00:05, 34.73it/s]
Loading 0:  60%|██████    | 274/453 [00:12<00:04, 41.16it/s]
Loading 0:  62%|██████▏   | 279/453 [00:13<00:04, 40.22it/s]
Loading 0:  63%|██████▎   | 284/453 [00:13<00:05, 28.55it/s]
Loading 0:  64%|██████▎   | 288/453 [00:13<00:05, 29.99it/s]
Loading 0:  64%|██████▍   | 292/453 [00:13<00:05, 31.98it/s]
Loading 0:  65%|██████▌   | 296/453 [00:13<00:04, 32.68it/s]
Loading 0:  66%|██████▋   | 301/453 [00:13<00:04, 36.03it/s]
Loading 0:  67%|██████▋   | 305/453 [00:13<00:04, 35.52it/s]
Loading 0:  68%|██████▊   | 310/453 [00:13<00:03, 38.88it/s]
Loading 0:  70%|██████▉   | 315/453 [00:14<00:03, 40.07it/s]
Loading 0:  71%|███████   | 320/453 [00:14<00:03, 40.48it/s]
Loading 0:  72%|███████▏  | 325/453 [00:14<00:03, 42.20it/s]
Loading 0:  73%|███████▎  | 330/453 [00:14<00:03, 35.60it/s]
Loading 0:  74%|███████▍  | 337/453 [00:14<00:02, 43.08it/s]
Loading 0:  75%|███████▌  | 342/453 [00:14<00:02, 39.97it/s]
Loading 0:  77%|███████▋  | 347/453 [00:14<00:02, 40.60it/s]
Loading 0:  78%|███████▊  | 352/453 [00:15<00:02, 42.42it/s]
Loading 0:  79%|███████▉  | 357/453 [00:15<00:02, 34.92it/s]
Loading 0:  80%|████████  | 364/453 [00:15<00:02, 29.96it/s]
Loading 0:  81%|████████  | 368/453 [00:15<00:02, 30.72it/s]
Loading 0:  82%|████████▏ | 373/453 [00:15<00:02, 34.08it/s]
Loading 0:  83%|████████▎ | 377/453 [00:15<00:02, 34.93it/s]
Loading 0:  84%|████████▍ | 381/453 [00:26<00:02, 34.93it/s]
Loading 0:  84%|████████▍ | 382/453 [00:30<01:05,  1.09it/s]
Loading 0:  85%|████████▍ | 384/453 [00:30<00:54,  1.27it/s]
Loading 0:  86%|████████▌ | 389/453 [00:30<00:33,  1.92it/s]
Loading 0:  87%|████████▋ | 393/453 [00:30<00:23,  2.55it/s]
Loading 0:  89%|████████▊ | 401/453 [00:30<00:11,  4.49it/s]
Loading 0:  90%|████████▉ | 407/453 [00:31<00:07,  6.31it/s]
Loading 0:  91%|█████████ | 412/453 [00:31<00:04,  8.23it/s]
Loading 0:  92%|█████████▏| 418/453 [00:31<00:03, 11.26it/s]
Loading 0:  93%|█████████▎| 423/453 [00:31<00:02, 14.06it/s]
Loading 0:  94%|█████████▍| 428/453 [00:31<00:01, 17.02it/s]
Loading 0:  96%|█████████▌| 433/453 [00:31<00:00, 20.85it/s]
Loading 0:  97%|█████████▋| 438/453 [00:31<00:00, 22.07it/s]
Loading 0:  98%|█████████▊| 445/453 [00:32<00:00, 23.27it/s]
Loading 0:  99%|█████████▉| 449/453 [00:32<00:00, 24.89it/s]
                                                            
Job servicenow-ai-apriel-ne-30954-v1-mkmlizer completed after 94.47s with status: succeeded
Stopping job with name servicenow-ai-apriel-ne-30954-v1-mkmlizer
Pipeline stage MKMLizer completed in 94.95s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service servicenow-ai-apriel-ne-30954-v1
Waiting for inference service servicenow-ai-apriel-ne-30954-v1 to be ready
Inference service servicenow-ai-apriel-ne-30954-v1 ready after 140.63343930244446s
Pipeline stage MKMLDeployer completed in 141.20s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.525524854660034s
Received healthy response to inference request in 1.3716251850128174s
Received healthy response to inference request in 1.1563661098480225s
Received healthy response to inference request in 1.802905559539795s
Received healthy response to inference request in 1.9104113578796387s
5 requests
0 failed requests
5th percentile: 1.1994179248809815
10th percentile: 1.2424697399139404
20th percentile: 1.3285733699798583
30th percentile: 1.457881259918213
40th percentile: 1.630393409729004
50th percentile: 1.802905559539795
60th percentile: 1.8459078788757324
70th percentile: 1.8889101982116698
80th percentile: 2.0334340572357177
90th percentile: 2.279479455947876
95th percentile: 2.402502155303955
99th percentile: 2.5009203147888184
mean time: 1.7533666133880614
Pipeline stage StressChecker completed in 9.93s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.73s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.67s
Shutdown handler de-registered
servicenow-ai-apriel-ne_30954_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.11s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service servicenow-ai-apriel-ne-30954-v1-profiler
Waiting for inference service servicenow-ai-apriel-ne-30954-v1-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3239.39s
Shutdown handler de-registered
servicenow-ai-apriel-ne_30954_v1 status is now inactive due to auto deactivation removed underperforming models
servicenow-ai-apriel-ne_30954_v1 status is now torndown due to DeploymentManager action