dtnewman-daniel-2025021_12303

developer_uid: dtnewman
submission_id: dtnewman-daniel-2025021_12303_v4
model_name: dtnewman-daniel-2025021_12303_v4
model_group: dtnewman/daniel-20250211
status: torndown
timestamp: 2025-02-12T18:54:17+00:00
num_battles: 7307
num_wins: 3433
celo_rating: 1246.88
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: dtnewman/daniel-20250211-c-lt4000-4epochs-embeddings
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 8
max_input_tokens: 1024
max_output_tokens: 64
reward_model: default
display_name: dtnewman-daniel-2025021_12303_v4
ineligible_reason: num_battles<10000
is_internal_developer: False
language_model: dtnewman/daniel-20250211-c-lt4000-4epochs-embeddings
model_size: 13B
ranking_group: single
us_pacific_date: 2025-02-12
win_ratio: 0.46982345695908034
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', '\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': 'You: {message}\n', 'response_template': '####\n{bot_name}:', 'truncate_by_message': False}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name dtnewman-daniel-2025021-12303-v4-mkmlizer
Waiting for job on dtnewman-daniel-2025021-12303-v4-mkmlizer to finish
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
dtnewman-daniel-2025021-12303-v4-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║     _____            __           __                                ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║       /___/                                                         ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║                                                                     ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║  Version: 0.12.8                                                    ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║  https://mk1.ai                                                     ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║                                                                     ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║  The license key for the current software has been verified as      ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║  belonging to:                                                      ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║                                                                     ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║  Chai Research Corp.                                                ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║  Expiration: 2025-04-15 23:59:59                                    ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ║                                                                     ║
dtnewman-daniel-2025021-12303-v4-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
dtnewman-daniel-2025021-12303-v4-mkmlizer: Downloaded to shared memory in 54.514s
dtnewman-daniel-2025021-12303-v4-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpukkvh1n2, device:0
dtnewman-daniel-2025021-12303-v4-mkmlizer: Saving flywheel model at /dev/shm/model_cache
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
dtnewman-daniel-2025021-12303-v4-mkmlizer: quantized model in 36.067s
dtnewman-daniel-2025021-12303-v4-mkmlizer: Processed model dtnewman/daniel-20250211-c-lt4000-4epochs-embeddings in 90.582s
dtnewman-daniel-2025021-12303-v4-mkmlizer: creating bucket guanaco-mkml-models
dtnewman-daniel-2025021-12303-v4-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
dtnewman-daniel-2025021-12303-v4-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/dtnewman-daniel-2025021-12303-v4
dtnewman-daniel-2025021-12303-v4-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/dtnewman-daniel-2025021-12303-v4/config.json
dtnewman-daniel-2025021-12303-v4-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/dtnewman-daniel-2025021-12303-v4/special_tokens_map.json
dtnewman-daniel-2025021-12303-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/dtnewman-daniel-2025021-12303-v4/tokenizer_config.json
dtnewman-daniel-2025021-12303-v4-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/dtnewman-daniel-2025021-12303-v4/tokenizer.json
dtnewman-daniel-2025021-12303-v4-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/dtnewman-daniel-2025021-12303-v4/flywheel_model.0.safetensors
dtnewman-daniel-2025021-12303-v4-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:12, 29.26it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:07, 48.53it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 43.32it/s]
Loading 0:   7%|▋         | 24/363 [00:00<00:07, 43.33it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 49.73it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:07, 46.07it/s]
Loading 0:  12%|█▏        | 42/363 [00:00<00:07, 44.50it/s]
Loading 0:  13%|█▎        | 49/363 [00:01<00:06, 49.38it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 44.90it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:09, 30.86it/s]
Loading 0:  18%|█▊        | 65/363 [00:01<00:09, 31.16it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 37.48it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:07, 38.38it/s]
Loading 0:  23%|██▎       | 83/363 [00:02<00:07, 39.10it/s]
Loading 0:  25%|██▍       | 90/363 [00:02<00:06, 44.59it/s]
Loading 0:  26%|██▋       | 96/363 [00:02<00:06, 42.39it/s]
Loading 0:  28%|██▊       | 101/363 [00:02<00:06, 41.36it/s]
Loading 0:  29%|██▉       | 106/363 [00:02<00:06, 42.39it/s]
Loading 0:  31%|███       | 112/363 [00:02<00:05, 46.64it/s]
Loading 0:  32%|███▏      | 117/363 [00:02<00:05, 44.36it/s]
Loading 0:  34%|███▎      | 122/363 [00:02<00:05, 45.48it/s]
Loading 0:  35%|███▍      | 127/363 [00:03<00:06, 37.61it/s]
Loading 0:  37%|███▋      | 134/363 [00:03<00:05, 45.16it/s]
Loading 0:  38%|███▊      | 139/363 [00:03<00:04, 45.63it/s]
Loading 0:  40%|███▉      | 144/363 [00:03<00:08, 26.37it/s]
Loading 0:  41%|████      | 149/363 [00:03<00:07, 28.82it/s]
Loading 0:  43%|████▎     | 157/363 [00:03<00:05, 37.15it/s]
Loading 0:  45%|████▍     | 163/363 [00:04<00:05, 37.31it/s]
Loading 0:  46%|████▋     | 168/363 [00:04<00:05, 38.09it/s]
Loading 0:  48%|████▊     | 174/363 [00:04<00:04, 42.91it/s]
Loading 0:  49%|████▉     | 179/363 [00:04<00:04, 42.72it/s]
Loading 0:  51%|█████     | 184/363 [00:04<00:04, 43.00it/s]
Loading 0:  52%|█████▏    | 189/363 [00:04<00:03, 44.09it/s]
Loading 0:  53%|█████▎    | 194/363 [00:04<00:04, 36.49it/s]
Loading 0:  55%|█████▌    | 201/363 [00:04<00:03, 43.69it/s]
Loading 0:  57%|█████▋    | 206/363 [00:05<00:03, 43.85it/s]
Loading 0:  58%|█████▊    | 211/363 [00:05<00:03, 44.76it/s]
Loading 0:  60%|█████▉    | 217/363 [00:05<00:03, 41.99it/s]
Loading 0:  61%|██████    | 222/363 [00:05<00:03, 43.52it/s]
Loading 0:  63%|██████▎   | 227/363 [00:05<00:04, 27.89it/s]
Loading 0:  64%|██████▎   | 231/363 [00:05<00:04, 27.91it/s]
Loading 0:  65%|██████▌   | 237/363 [00:06<00:03, 33.88it/s]
Loading 0:  67%|██████▋   | 242/363 [00:06<00:03, 35.85it/s]
Loading 0:  68%|██████▊   | 247/363 [00:06<00:03, 37.52it/s]
Loading 0:  70%|██████▉   | 253/363 [00:06<00:02, 37.46it/s]
Loading 0:  71%|███████   | 258/363 [00:06<00:02, 37.55it/s]
Loading 0:  73%|███████▎  | 264/363 [00:06<00:02, 42.52it/s]
Loading 0:  74%|███████▍  | 269/363 [00:06<00:02, 42.55it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:02, 42.21it/s]
Loading 0:  77%|███████▋  | 279/363 [00:07<00:01, 43.87it/s]
Loading 0:  78%|███████▊  | 284/363 [00:07<00:02, 35.80it/s]
Loading 0:  80%|████████  | 291/363 [00:07<00:01, 40.92it/s]
Loading 0:  82%|████████▏ | 296/363 [00:07<00:01, 39.36it/s]
Loading 0:  83%|████████▎ | 302/363 [00:07<00:01, 43.72it/s]
Loading 0:  85%|████████▍ | 307/363 [00:14<00:21,  2.57it/s]
Loading 0:  86%|████████▌ | 312/363 [00:14<00:14,  3.50it/s]
Loading 0:  88%|████████▊ | 320/363 [00:14<00:07,  5.59it/s]
Loading 0:  90%|████████▉ | 326/363 [00:14<00:04,  7.52it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:03,  9.59it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 13.50it/s]
Loading 0:  95%|█████████▍| 344/363 [00:15<00:01, 16.77it/s]
Loading 0:  96%|█████████▌| 349/363 [00:15<00:00, 19.99it/s]
Loading 0:  98%|█████████▊| 356/363 [00:15<00:00, 26.04it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 28.99it/s]
                                                            
Job dtnewman-daniel-2025021-12303-v4-mkmlizer completed after 338.84s with status: succeeded
Stopping job with name dtnewman-daniel-2025021-12303-v4-mkmlizer
Pipeline stage MKMLizer completed in 339.29s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service dtnewman-daniel-2025021-12303-v4
Waiting for inference service dtnewman-daniel-2025021-12303-v4 to be ready
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service dtnewman-daniel-2025021-12303-v4 ready after 220.819105386734s
Pipeline stage MKMLDeployer completed in 221.34s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.0399060249328613s
Received healthy response to inference request in 1.7914314270019531s
Received healthy response to inference request in 1.739363431930542s
Received healthy response to inference request in 1.6813623905181885s
Received healthy response to inference request in 1.7710065841674805s
5 requests
0 failed requests
5th percentile: 1.6929625988006591
10th percentile: 1.7045628070831298
20th percentile: 1.7277632236480713
30th percentile: 1.7456920623779297
40th percentile: 1.758349323272705
50th percentile: 1.7710065841674805
60th percentile: 1.7791765213012696
70th percentile: 1.7873464584350587
80th percentile: 1.841126346588135
90th percentile: 1.940516185760498
95th percentile: 1.9902111053466796
99th percentile: 2.029967041015625
mean time: 1.804613971710205
Pipeline stage StressChecker completed in 10.35s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.71s
run pipeline stage %s
Failed to get response for submission mistralai-mistral-nem_93303_v330: HTTPConnectionPool(host='mistralai-mistral-nem-93303-v330-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.71s
Shutdown handler de-registered
dtnewman-daniel-2025021_12303_v4 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
Evaluating %s Family Friendly Score with %s threads
clean up pipeline due to error=DeploymentChecksError('None: None')
Shutdown handler de-registered
dtnewman-daniel-2025021_12303_v4 status is now inactive due to auto deactivation removed underperforming models
dtnewman-daniel-2025021_12303_v4 status is now torndown due to DeploymentManager action
dtnewman-daniel-2025021_12303_v4 status is now torndown due to DeploymentManager action
dtnewman-daniel-2025021_12303_v4 status is now torndown due to DeploymentManager action
Failed to get response for submission rirv938-llama-8b-1024-t_67568_v3: HTTPConnectionPool(host='rirv938-llama-8b-1024-t-67568-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x72396449c7d0>, 'Connection to rirv938-llama-8b-1024-t-67568-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com timed out. (connect timeout=12.0)'))