developer_uid: cgato
submission_id: cgato-nemo-12b-humaniz_74135_v23
model_name: cgato-nemo-12b-humaniz_74135_v23
model_group: cgato/Nemo-12b-Humanize-
status: torndown
timestamp: 2024-12-29T23:52:07+00:00
num_battles: 20579
num_wins: 10225
celo_rating: 1259.63
family_friendly_score: 0.5826
family_friendly_standard_error: 0.006973911958148023
submission_type: basic
model_repo: cgato/Nemo-12b-Humanize-KTO-Experimental-Latest
model_architecture: MistralForCausalLM
model_num_parameters: 12772111360.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
latencies: [{'batch_size': 1, 'throughput': 0.6155416349716519, 'latency_mean': 1.6244928383827208, 'latency_p50': 1.6286394596099854, 'latency_p90': 1.7901229858398438}, {'batch_size': 3, 'throughput': 1.1311680102787833, 'latency_mean': 2.6421259772777557, 'latency_p50': 2.643632173538208, 'latency_p90': 2.9047838687896728}, {'batch_size': 5, 'throughput': 1.365917267411497, 'latency_mean': 3.645848693847656, 'latency_p50': 3.6316787004470825, 'latency_p90': 4.096790218353272}, {'batch_size': 6, 'throughput': 1.4216921705173786, 'latency_mean': 4.18159086227417, 'latency_p50': 4.1960813999176025, 'latency_p90': 4.678508949279785}, {'batch_size': 8, 'throughput': 1.5070170010222486, 'latency_mean': 5.278571177721023, 'latency_p50': 5.302056670188904, 'latency_p90': 5.987956213951111}, {'batch_size': 10, 'throughput': 1.5368515915666328, 'latency_mean': 6.456275273561477, 'latency_p50': 6.448912262916565, 'latency_p90': 7.2113752365112305}]
gpu_counts: {'NVIDIA RTX A5000': 1}
display_name: cgato-nemo-12b-humaniz_74135_v23
is_internal_developer: False
language_model: cgato/Nemo-12b-Humanize-KTO-Experimental-Latest
model_size: 13B
ranking_group: single
throughput_3p7s: 1.38
us_pacific_date: 2024-12-29
win_ratio: 0.4968657369162739
generation_params: {'temperature': 0.7, 'top_p': 0.9, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '<|im_start|>system\n{memory}<|im_end|>\n', 'prompt_template': '<|im_start|>user\n{prompt}<|im_end|>\n', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name cgato-nemo-12b-humaniz-74135-v23-mkmlizer
Waiting for job on cgato-nemo-12b-humaniz-74135-v23-mkmlizer to finish
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║     _____            __           __                                ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║       /___/                                                         ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║                                                                     ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║  Version: 0.11.12                                                   ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║  https://mk1.ai                                                     ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║                                                                     ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║  The license key for the current software has been verified as      ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║  belonging to:                                                      ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║                                                                     ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║  Chai Research Corp.                                                ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ║                                                                     ║
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: Downloaded to shared memory in 49.964s
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpmy71bqz_, device:0
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: Saving flywheel model at /dev/shm/model_cache
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: quantized model in 35.512s
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: Processed model cgato/Nemo-12b-Humanize-KTO-Experimental-Latest in 85.476s
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: creating bucket guanaco-mkml-models
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/cgato-nemo-12b-humaniz-74135-v23
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/cgato-nemo-12b-humaniz-74135-v23/config.json
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/cgato-nemo-12b-humaniz-74135-v23/special_tokens_map.json
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/cgato-nemo-12b-humaniz-74135-v23/tokenizer_config.json
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/cgato-nemo-12b-humaniz-74135-v23/tokenizer.json
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/cgato-nemo-12b-humaniz-74135-v23/flywheel_model.0.safetensors
cgato-nemo-12b-humaniz-74135-v23-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 31.98it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 52.43it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 48.25it/s]
Loading 0:   7%|▋         | 25/363 [00:00<00:06, 48.51it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 50.63it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:06, 48.53it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:06, 46.82it/s]
Loading 0:  13%|█▎        | 49/363 [00:00<00:06, 52.25it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 47.61it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 37.25it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:07, 37.24it/s]
Loading 0:  20%|█▉        | 71/363 [00:01<00:07, 40.01it/s]
Loading 0:  21%|██        | 76/363 [00:01<00:07, 40.60it/s]
Loading 0:  22%|██▏       | 81/363 [00:01<00:06, 41.72it/s]
Loading 0:  24%|██▎       | 86/363 [00:01<00:06, 42.24it/s]
Loading 0:  25%|██▌       | 91/363 [00:02<00:07, 37.74it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:05, 46.56it/s]
Loading 0:  29%|██▉       | 105/363 [00:02<00:05, 45.02it/s]
Loading 0:  31%|███       | 112/363 [00:02<00:05, 47.83it/s]
Loading 0:  32%|███▏      | 117/363 [00:02<00:05, 45.71it/s]
Loading 0:  34%|███▍      | 123/363 [00:02<00:05, 44.02it/s]
Loading 0:  35%|███▌      | 128/363 [00:02<00:05, 42.86it/s]
Loading 0:  37%|███▋      | 135/363 [00:03<00:04, 47.75it/s]
Loading 0:  39%|███▉      | 141/363 [00:03<00:04, 45.73it/s]
Loading 0:  40%|████      | 146/363 [00:03<00:06, 33.29it/s]
Loading 0:  41%|████▏     | 150/363 [00:03<00:06, 32.92it/s]
Loading 0:  43%|████▎     | 157/363 [00:03<00:05, 39.30it/s]
Loading 0:  45%|████▍     | 163/363 [00:03<00:05, 39.88it/s]
Loading 0:  46%|████▋     | 168/363 [00:03<00:04, 39.41it/s]
Loading 0:  48%|████▊     | 175/363 [00:04<00:04, 44.98it/s]
Loading 0:  50%|████▉     | 181/363 [00:04<00:04, 43.44it/s]
Loading 0:  51%|█████     | 186/363 [00:04<00:04, 43.13it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:03, 47.54it/s]
Loading 0:  55%|█████▍    | 199/363 [00:04<00:03, 45.80it/s]
Loading 0:  56%|█████▌    | 204/363 [00:04<00:03, 45.13it/s]
Loading 0:  58%|█████▊    | 211/363 [00:04<00:03, 49.43it/s]
Loading 0:  60%|█████▉    | 217/363 [00:04<00:03, 45.94it/s]
Loading 0:  61%|██████▏   | 223/363 [00:05<00:03, 36.34it/s]
Loading 0:  63%|██████▎   | 228/363 [00:05<00:03, 36.83it/s]
Loading 0:  64%|██████▍   | 232/363 [00:05<00:03, 36.86it/s]
Loading 0:  66%|██████▌   | 238/363 [00:05<00:03, 40.68it/s]
Loading 0:  67%|██████▋   | 243/363 [00:05<00:02, 41.89it/s]
Loading 0:  68%|██████▊   | 248/363 [00:05<00:03, 36.62it/s]
Loading 0:  71%|███████   | 256/363 [00:05<00:02, 44.82it/s]
Loading 0:  72%|███████▏  | 261/363 [00:06<00:02, 44.88it/s]
Loading 0:  73%|███████▎  | 266/363 [00:06<00:02, 35.86it/s]
Loading 0:  75%|███████▌  | 273/363 [00:06<00:02, 41.23it/s]
Loading 0:  77%|███████▋  | 278/363 [00:06<00:02, 39.47it/s]
Loading 0:  78%|███████▊  | 283/363 [00:06<00:01, 40.80it/s]
Loading 0:  80%|███████▉  | 289/363 [00:06<00:01, 40.38it/s]
Loading 0:  81%|████████  | 294/363 [00:06<00:01, 40.00it/s]
Loading 0:  83%|████████▎ | 300/363 [00:07<00:01, 44.78it/s]
Loading 0:  84%|████████▍ | 305/363 [00:13<00:22,  2.56it/s]
Loading 0:  85%|████████▌ | 309/363 [00:13<00:16,  3.30it/s]
Loading 0:  86%|████████▌ | 313/363 [00:14<00:11,  4.23it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:06,  6.66it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:04,  9.18it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:02, 11.63it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 16.48it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:00, 20.52it/s]
Loading 0:  96%|█████████▌| 349/363 [00:14<00:00, 23.92it/s]
Loading 0:  98%|█████████▊| 356/363 [00:14<00:00, 30.26it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 32.75it/s]
                                                            

Job cgato-nemo-12b-humaniz-74135-v23-mkmlizer completed after 114.6s with status: succeeded
Stopping job with name cgato-nemo-12b-humaniz-74135-v23-mkmlizer
Pipeline stage MKMLizer completed in 115.12s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service cgato-nemo-12b-humaniz-74135-v23
Waiting for inference service cgato-nemo-12b-humaniz-74135-v23 to be ready
Inference service cgato-nemo-12b-humaniz-74135-v23 ready after 311.1888153553009s
Pipeline stage MKMLDeployer completed in 311.73s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.1390953063964844s
Received healthy response to inference request in 1.7472665309906006s
Failed to get response for submission function_gidef_2024-11-26: ('http://chaiml-elo-alignment-run-3-v44-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:41304->127.0.0.1:8080: read: connection reset by peer\n')
Received healthy response to inference request in 1.7508790493011475s
Received healthy response to inference request in 1.6459035873413086s
Received healthy response to inference request in 1.1940429210662842s
5 requests
0 failed requests
5th percentile: 1.284415054321289
10th percentile: 1.374787187576294
20th percentile: 1.5555314540863037
30th percentile: 1.666176176071167
40th percentile: 1.7067213535308838
50th percentile: 1.7472665309906006
60th percentile: 1.7487115383148193
70th percentile: 1.750156545639038
80th percentile: 1.828522300720215
90th percentile: 1.9838088035583497
95th percentile: 2.061452054977417
99th percentile: 2.1235666561126707
mean time: 1.695437479019165
Pipeline stage StressChecker completed in 10.13s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.72s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.69s
Shutdown handler de-registered
cgato-nemo-12b-humaniz_74135_v23 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.08s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.09s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service cgato-nemo-12b-humaniz-74135-v23-profiler
Waiting for inference service cgato-nemo-12b-humaniz-74135-v23-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2861.30s
Shutdown handler de-registered
cgato-nemo-12b-humaniz_74135_v23 status is now inactive due to auto deactivation removed underperforming models
cgato-nemo-12b-humaniz_74135_v23 status is now torndown due to DeploymentManager action
cgato-nemo-12b-humaniz_74135_v23 status is now torndown due to DeploymentManager action
cgato-nemo-12b-humaniz_74135_v23 status is now torndown due to DeploymentManager action