chaiml-test-feed-convo-v1-1e5

submission_id: chaiml-test-feed-convo-v1-1e5_v5
developer_uid: zonemercy
status: torndown
model_repo: ChaiML/Test-Feed-Convo-v1-1e5
generation_params: {'temperature': 0.9, 'top_p': 1.0, 'min_p': 0.05, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</s>', 'Bot:', 'User:', 'You:'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
timestamp: 2024-09-11T17:25:00+00:00
model_name: chaiml-test-feed-convo-v1-1e5_v5
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-test-feed-convo-v1-1e5-v5-mkmlizer
Waiting for job on chaiml-test-feed-convo-v1-1e5-v5-mkmlizer to finish
Failed to get response for submission chaiml-elo-alignment-run-3_v42: ('http://chaiml-llama-8b-pairwis-8189-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'readfrom tcp 127.0.0.1:35384->127.0.0.1:8080: write tcp 127.0.0.1:35384->127.0.0.1:8080: use of closed network connection\n')
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║     _____            __           __                                ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║       /___/                                                         ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║                                                                     ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║  Version: 0.10.1                                                    ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║                                                                     ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║  belonging to:                                                      ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║                                                                     ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ║                                                                     ║
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: Downloaded to shared memory in 28.168s
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmp8jp4b5zb, device:0
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: quantized model in 35.322s
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: Processed model ChaiML/Test-Feed-Convo-v1-1e5 in 63.490s
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: creating bucket guanaco-mkml-models
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-test-feed-convo-v1-1e5-v5
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-test-feed-convo-v1-1e5-v5/special_tokens_map.json
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-test-feed-convo-v1-1e5-v5/config.json
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-test-feed-convo-v1-1e5-v5/tokenizer_config.json
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-test-feed-convo-v1-1e5-v5/tokenizer.json
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-test-feed-convo-v1-1e5-v5/flywheel_model.0.safetensors
chaiml-test-feed-convo-v1-1e5-v5-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 32.24it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 55.12it/s]
Loading 0:   6%|▌         | 20/363 [00:00<00:06, 55.74it/s]
Loading 0:   7%|▋         | 26/363 [00:00<00:06, 53.29it/s]
Loading 0:   9%|▉         | 32/363 [00:00<00:07, 46.79it/s]
Loading 0:  11%|█         | 40/363 [00:00<00:05, 55.56it/s]
Loading 0:  13%|█▎        | 46/363 [00:00<00:06, 49.08it/s]
Loading 0:  14%|█▍        | 52/363 [00:01<00:06, 48.62it/s]
Loading 0:  17%|█▋        | 60/363 [00:01<00:05, 50.74it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:08, 34.43it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:07, 37.78it/s]
Loading 0:  21%|██        | 77/363 [00:01<00:07, 40.24it/s]
Loading 0:  23%|██▎       | 82/363 [00:01<00:07, 37.28it/s]
Loading 0:  25%|██▍       | 90/363 [00:01<00:05, 46.06it/s]
Loading 0:  26%|██▋       | 96/363 [00:02<00:05, 44.64it/s]
Loading 0:  28%|██▊       | 101/363 [00:02<00:06, 41.84it/s]
Loading 0:  29%|██▉       | 106/363 [00:02<00:05, 42.93it/s]
Loading 0:  31%|███       | 113/363 [00:02<00:05, 44.17it/s]
Loading 0:  33%|███▎      | 118/363 [00:02<00:05, 44.06it/s]
Loading 0:  35%|███▍      | 126/363 [00:02<00:04, 52.44it/s]
Loading 0:  36%|███▋      | 132/363 [00:02<00:04, 49.12it/s]
Loading 0:  38%|███▊      | 138/363 [00:03<00:04, 50.14it/s]
Loading 0:  40%|███▉      | 144/363 [00:03<00:06, 33.91it/s]
Loading 0:  41%|████      | 149/363 [00:03<00:06, 35.34it/s]
Loading 0:  43%|████▎     | 157/363 [00:03<00:04, 43.30it/s]
Loading 0:  45%|████▍     | 163/363 [00:03<00:04, 43.53it/s]
Loading 0:  46%|████▋     | 168/363 [00:03<00:04, 44.41it/s]
Loading 0:  48%|████▊     | 175/363 [00:03<00:03, 50.17it/s]
Loading 0:  50%|████▉     | 181/363 [00:04<00:03, 48.74it/s]
Loading 0:  52%|█████▏    | 187/363 [00:04<00:03, 48.51it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:03, 51.30it/s]
Loading 0:  55%|█████▍    | 199/363 [00:04<00:03, 44.38it/s]
Loading 0:  56%|█████▌    | 204/363 [00:04<00:03, 41.85it/s]
Loading 0:  58%|█████▊    | 210/363 [00:04<00:03, 45.99it/s]
Loading 0:  59%|█████▉    | 215/363 [00:04<00:03, 45.92it/s]
Loading 0:  61%|██████    | 222/363 [00:04<00:03, 46.83it/s]
Loading 0:  63%|██████▎   | 227/363 [00:05<00:04, 33.82it/s]
Loading 0:  64%|██████▎   | 231/363 [00:05<00:03, 33.29it/s]
Loading 0:  66%|██████▌   | 238/363 [00:05<00:03, 40.40it/s]
Loading 0:  67%|██████▋   | 244/363 [00:05<00:03, 39.32it/s]
Loading 0:  69%|██████▊   | 249/363 [00:05<00:02, 39.59it/s]
Loading 0:  71%|███████   | 256/363 [00:05<00:02, 44.46it/s]
Loading 0:  72%|███████▏  | 261/363 [00:05<00:02, 44.38it/s]
Loading 0:  73%|███████▎  | 266/363 [00:06<00:02, 38.21it/s]
Loading 0:  75%|███████▌  | 274/363 [00:06<00:01, 46.18it/s]
Loading 0:  77%|███████▋  | 280/363 [00:06<00:01, 45.80it/s]
Loading 0:  79%|███████▊  | 285/363 [00:06<00:01, 45.68it/s]
Loading 0:  80%|████████  | 292/363 [00:06<00:01, 50.45it/s]
Loading 0:  82%|████████▏ | 298/363 [00:06<00:01, 48.21it/s]
Loading 0:  84%|████████▎ | 304/363 [00:13<00:20,  2.89it/s]
Loading 0:  85%|████████▍ | 308/363 [00:13<00:15,  3.62it/s]
Loading 0:  86%|████████▌ | 312/363 [00:13<00:11,  4.58it/s]
Loading 0:  88%|████████▊ | 320/363 [00:13<00:05,  7.39it/s]
Loading 0:  90%|████████▉ | 326/363 [00:13<00:03,  9.80it/s]
Loading 0:  91%|█████████ | 331/363 [00:14<00:02, 12.27it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 17.00it/s]
Loading 0:  94%|█████████▍| 343/363 [00:14<00:00, 20.46it/s]
Loading 0:  96%|█████████▌| 348/363 [00:14<00:00, 22.00it/s]
Loading 0:  98%|█████████▊| 356/363 [00:14<00:00, 29.76it/s]
Loading 0: 100%|█████████▉| 362/363 [00:14<00:00, 30.95it/s]
                                                            
Job chaiml-test-feed-convo-v1-1e5-v5-mkmlizer completed after 96.47s with status: succeeded
Stopping job with name chaiml-test-feed-convo-v1-1e5-v5-mkmlizer
Pipeline stage MKMLizer completed in 97.97s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.08s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-test-feed-convo-v1-1e5-v5
Waiting for inference service chaiml-test-feed-convo-v1-1e5-v5 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission chaiml-elo-alignment-run-3_v42: ('http://chaiml-elo-alignment-run-3-v42-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission chaiml-elo-alignment-run-3_v42: ('http://chaiml-elo-alignment-run-3-v42-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission chaiml-elo-alignment-run-3_v42: ('http://chaiml-elo-alignment-run-3-v42-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission chaiml-elo-alignment-run-3_v42: ('http://chaiml-elo-alignment-run-3-v42-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission chaiml-elo-alignment-run-3_v42: ('http://chaiml-elo-alignment-run-3-v42-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Failed to get response for submission chaiml-elo-alignment-run-3_v42: ('http://chaiml-elo-alignment-run-3-v42-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission chaiml-elo-alignment-run-3_v42: ('http://chaiml-llama-8b-pairwis-8189-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:47618->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission function_sabet_2024-09-10: ('http://chaiml-llama-8b-pairwis-8189-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:33672->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission blend_siken_2024-09-09: ('http://chaiml-llama-8b-pairwis-8189-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:51740->127.0.0.1:8080: read: connection reset by peer\n')
Tearing down inference service chaiml-test-feed-convo-v1-1e5-v5
%s, retrying in %s seconds...
Creating inference service chaiml-test-feed-convo-v1-1e5-v5
Waiting for inference service chaiml-test-feed-convo-v1-1e5-v5 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission blend_puheb_2024-09-09: ('http://chaiml-llama-8b-pairwis-8189-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:36226->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission chaiml-elo-alignment-run-3_v41: ('http://chaiml-llama-8b-pairwis-8189-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:38840->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission neversleep-noromaid-v0_8068_v150: ('http://neversleep-noromaid-v0-8068-v150-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '{"error":"ValueError : [TypeError(\\"\'numpy.int64\' object is not iterable\\"), TypeError(\'vars() argument must have __dict__ attribute\')]"}')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission blend_siken_2024-09-09: ('http://chaiml-elo-alignment-run-3-v34-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:54546->127.0.0.1:8080: read: connection reset by peer\n')
Tearing down inference service chaiml-test-feed-convo-v1-1e5-v5
%s, retrying in %s seconds...
Creating inference service chaiml-test-feed-convo-v1-1e5-v5
Waiting for inference service chaiml-test-feed-convo-v1-1e5-v5 to be ready
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission neversleep-noromaid-v0_8068_v150: ('http://neversleep-noromaid-v0-8068-v150-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:57618->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Failed to get response for submission chaiml-llama-8b-pairwis_8189_v19: ('http://chaiml-llama-8b-pairwis-8189-v19-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:51354->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Tearing down inference service chaiml-test-feed-convo-v1-1e5-v5
clean up pipeline due to error=%s
Shutdown handler de-registered
chaiml-test-feed-convo-v1-1e5_v5 status is now failed due to DeploymentManager action
chaiml-test-feed-convo-v1-1e5_v5 status is now torndown due to DeploymentManager action