cgato-nemo-12b-thespice-_4109

developer_uid: c.gato
submission_id: cgato-nemo-12b-thespice-_4109_v1
model_name: cgato-nemo-12b-thespice-_4109_v1
model_group: cgato/Nemo-12b-TheSpice-
status: torndown
timestamp: 2024-11-03T03:01:40+00:00
num_battles: 16341
num_wins: 8737
celo_rating: 1267.37
family_friendly_score: 0.5688
family_friendly_standard_error: 0.00700380696478708
submission_type: basic
model_repo: cgato/Nemo-12b-TheSpice-V0.9-All-v2-KTO-v0.2.2-E1-2547
model_architecture: MistralForCausalLM
model_num_parameters: 12772121600.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
display_name: cgato-nemo-12b-thespice-_4109_v1
is_internal_developer: False
language_model: cgato/Nemo-12b-TheSpice-V0.9-All-v2-KTO-v0.2.2-E1-2547
model_size: 13B
ranking_group: single
us_pacific_date: 2024-11-02
win_ratio: 0.5346674010158498
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '<|im_start|>system\n{memory}<|im_end|>\n', 'prompt_template': '<|im_start|>user\n{prompt}<|im_end|>\n', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{user_name}: {message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name cgato-nemo-12b-thespice-4109-v1-mkmlizer
Waiting for job on cgato-nemo-12b-thespice-4109-v1-mkmlizer to finish
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║     _____            __           __                                ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║       /___/                                                         ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║                                                                     ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║  Version: 0.11.12                                                   ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║  https://mk1.ai                                                     ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║                                                                     ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║  belonging to:                                                      ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║                                                                     ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║  Chai Research Corp.                                                ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║  Expiration: 2025-01-15 23:59:59                                    ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ║                                                                     ║
cgato-nemo-12b-thespice-4109-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Connection pool is full, discarding connection: %s. Connection pool size: %s
cgato-nemo-12b-thespice-4109-v1-mkmlizer: Downloaded to shared memory in 51.840s
cgato-nemo-12b-thespice-4109-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp7qrje0jh, device:0
cgato-nemo-12b-thespice-4109-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
cgato-nemo-12b-thespice-4109-v1-mkmlizer: quantized model in 35.854s
cgato-nemo-12b-thespice-4109-v1-mkmlizer: Processed model cgato/Nemo-12b-TheSpice-V0.9-All-v2-KTO-v0.2.2-E1-2547 in 87.694s
cgato-nemo-12b-thespice-4109-v1-mkmlizer: creating bucket guanaco-mkml-models
cgato-nemo-12b-thespice-4109-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
cgato-nemo-12b-thespice-4109-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/cgato-nemo-12b-thespice-4109-v1
cgato-nemo-12b-thespice-4109-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/cgato-nemo-12b-thespice-4109-v1/config.json
cgato-nemo-12b-thespice-4109-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/cgato-nemo-12b-thespice-4109-v1/special_tokens_map.json
cgato-nemo-12b-thespice-4109-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/cgato-nemo-12b-thespice-4109-v1/tokenizer_config.json
cgato-nemo-12b-thespice-4109-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/cgato-nemo-12b-thespice-4109-v1/tokenizer.json
Failed to get response for submission blend_gerot_2024-10-19: ('http://chaiml-virgo-edit-v1-1e5-v9-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:33032->127.0.0.1:8080: read: connection reset by peer\n')
cgato-nemo-12b-thespice-4109-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/cgato-nemo-12b-thespice-4109-v1/flywheel_model.0.safetensors
cgato-nemo-12b-thespice-4109-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 30.24it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 50.83it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 46.79it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:07, 44.39it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 50.86it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:06, 48.76it/s]
Loading 0:  12%|█▏        | 43/363 [00:00<00:06, 49.48it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:06, 51.41it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 45.26it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 33.70it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:09, 32.98it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 40.00it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:07, 39.83it/s]
Loading 0:  23%|██▎       | 83/363 [00:01<00:07, 38.88it/s]
Loading 0:  25%|██▍       | 90/363 [00:02<00:06, 44.37it/s]
Loading 0:  26%|██▋       | 96/363 [00:02<00:06, 43.28it/s]
Loading 0:  28%|██▊       | 101/363 [00:02<00:06, 42.43it/s]
Loading 0:  29%|██▉       | 106/363 [00:02<00:05, 43.88it/s]
Loading 0:  31%|███       | 112/363 [00:02<00:05, 47.37it/s]
Loading 0:  32%|███▏      | 117/363 [00:02<00:05, 45.33it/s]
Loading 0:  34%|███▍      | 123/363 [00:02<00:05, 44.04it/s]
Loading 0:  35%|███▌      | 128/363 [00:02<00:05, 43.10it/s]
Loading 0:  37%|███▋      | 134/363 [00:03<00:04, 46.78it/s]
Loading 0:  38%|███▊      | 139/363 [00:03<00:04, 45.76it/s]
Loading 0:  40%|███▉      | 144/363 [00:03<00:07, 28.00it/s]
Loading 0:  41%|████      | 149/363 [00:03<00:06, 30.78it/s]
Loading 0:  43%|████▎     | 157/363 [00:03<00:05, 39.20it/s]
Loading 0:  45%|████▍     | 163/363 [00:03<00:05, 39.54it/s]
Loading 0:  46%|████▋     | 168/363 [00:04<00:04, 39.58it/s]
Loading 0:  48%|████▊     | 175/363 [00:04<00:04, 44.89it/s]
Loading 0:  50%|████▉     | 181/363 [00:04<00:04, 44.02it/s]
Loading 0:  51%|█████     | 186/363 [00:04<00:04, 42.90it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:03, 47.15it/s]
Loading 0:  55%|█████▍    | 199/363 [00:04<00:03, 45.52it/s]
Loading 0:  56%|█████▌    | 204/363 [00:04<00:03, 44.89it/s]
Loading 0:  58%|█████▊    | 211/363 [00:04<00:03, 48.82it/s]
Loading 0:  60%|█████▉    | 217/363 [00:05<00:03, 46.75it/s]
Loading 0:  61%|██████▏   | 223/363 [00:05<00:03, 35.51it/s]
Loading 0:  63%|██████▎   | 227/363 [00:05<00:03, 35.58it/s]
Loading 0:  64%|██████▎   | 231/363 [00:05<00:03, 34.90it/s]
Loading 0:  66%|██████▌   | 238/363 [00:05<00:03, 41.31it/s]
Loading 0:  67%|██████▋   | 244/363 [00:05<00:02, 41.23it/s]
Loading 0:  69%|██████▊   | 249/363 [00:05<00:02, 39.75it/s]
Loading 0:  71%|███████   | 256/363 [00:06<00:02, 44.84it/s]
Loading 0:  72%|███████▏  | 262/363 [00:06<00:02, 43.93it/s]
Loading 0:  74%|███████▎  | 267/363 [00:06<00:02, 43.35it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:01, 48.27it/s]
Loading 0:  77%|███████▋  | 279/363 [00:06<00:01, 48.69it/s]
Loading 0:  78%|███████▊  | 284/363 [00:06<00:01, 40.28it/s]
Loading 0:  80%|████████  | 292/363 [00:06<00:01, 47.98it/s]
Loading 0:  82%|████████▏ | 298/363 [00:07<00:01, 44.75it/s]
Loading 0:  83%|████████▎ | 303/363 [00:07<00:01, 45.10it/s]
Loading 0:  85%|████████▍ | 308/363 [00:13<00:20,  2.62it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:15,  3.31it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:08,  5.37it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:05,  7.28it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:03,  9.33it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 13.21it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:01, 16.51it/s]
Loading 0:  96%|█████████▌| 349/363 [00:14<00:00, 19.24it/s]
Loading 0:  98%|█████████▊| 356/363 [00:14<00:00, 25.09it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 28.28it/s]
                                                            
Job cgato-nemo-12b-thespice-4109-v1-mkmlizer completed after 116.08s with status: succeeded
Stopping job with name cgato-nemo-12b-thespice-4109-v1-mkmlizer
Pipeline stage MKMLizer completed in 116.69s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service cgato-nemo-12b-thespice-4109-v1
Waiting for inference service cgato-nemo-12b-thespice-4109-v1 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Inference service cgato-nemo-12b-thespice-4109-v1 ready after 150.53752899169922s
Pipeline stage MKMLDeployer completed in 151.06s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.3640267848968506s
Received healthy response to inference request in 1.7620313167572021s
Received healthy response to inference request in 1.7880117893218994s
Received healthy response to inference request in 1.9284358024597168s
5 requests
1 failed requests
5th percentile: 1.7672274112701416
10th percentile: 1.772423505783081
20th percentile: 1.78281569480896
30th percentile: 1.816096591949463
40th percentile: 1.8722661972045898
50th percentile: 1.9284358024597168
60th percentile: 2.1026721954345704
70th percentile: 2.2769085884094236
80th percentile: 5.92471618652344
90th percentile: 13.046094989776613
95th percentile: 16.606784391403195
99th percentile: 19.455335912704466
mean time: 5.601995897293091
%s, retrying in %s seconds...
Received healthy response to inference request in 1.9155352115631104s
Received healthy response to inference request in 1.7182435989379883s
Received healthy response to inference request in 1.9198417663574219s
Received healthy response to inference request in 1.9355289936065674s
Received healthy response to inference request in 2.1270599365234375s
5 requests
0 failed requests
5th percentile: 1.7577019214630127
10th percentile: 1.7971602439880372
20th percentile: 1.876076889038086
30th percentile: 1.9163965225219726
40th percentile: 1.9181191444396972
50th percentile: 1.9198417663574219
60th percentile: 1.92611665725708
70th percentile: 1.9323915481567382
80th percentile: 1.9738351821899414
90th percentile: 2.0504475593566895
95th percentile: 2.0887537479400633
99th percentile: 2.1193986988067626
mean time: 1.9232419013977051
Pipeline stage StressChecker completed in 40.37s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 2.88s
Shutdown handler de-registered
cgato-nemo-12b-thespice-_4109_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2666.32s
Shutdown handler de-registered
cgato-nemo-12b-thespice-_4109_v1 status is now inactive due to auto deactivation removed underperforming models
cgato-nemo-12b-thespice-_4109_v1 status is now torndown due to DeploymentManager action