submission_id: blend_lades_2024-03-05
developer_uid: chai_backend_admin
status: deployed
timestamp: 2024-03-05T19:35:47+00:00
model_name: blend_lades_2024-03-05
tagged_submissions: [{'submission_id': 'chaiml-phase2-winner-13b2_v123', 'tags': ['sfw', 'nsfw']}, {'submission_id': 'khanhnto-khanhnto_v37', 'tags': ['sfw', 'nsfw']}, {'submission_id': 'anhnv125-llama-op-v17-1_v27', 'tags': ['sfw', 'nsfw']}, {'submission_id': 'thanhdaonguyen-once-upon-a-t_v19', 'tags': ['sfw', 'nsfw']}]
router: sfw_router
safety_score: None
entertaining: None
stay_in_character: None
user_preference: None
double_thumbs_up: 534490
thumbs_up: 885455
thumbs_down: 393635
num_battles: 512589
num_wins: 232146
win_ratio: 0.4528891568098418
celo_rating: 1120.54
Resubmit model
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
sao10k-solstice-11b-v1-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
Received score %s for bot %s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 278, in _get_conversation_tag conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 278, in _get_conversation_tag conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Failed to get response for submission neversleep-noromaidxopen_6111_v4: ('http://neversleep-noromaidxopen-6111-v4-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:59486->127.0.0.1:8080: read: connection reset by peer\n')
Exception raised while processing tagging_function
Exception raised while processing tagging_function
blend_guhof_2024-03-28 status is now deployed due to DeploymentManager action
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 278, in _get_conversation_tag conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 278, in _get_conversation_tag conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Exception raised while processing tagging_function
Failed to get response for submission blend_rarom_2024-02-29: ('http://thanhdaonguyen-once-upon-a-t-v19-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:54200->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission blend_salas_2024-03-03: ('http://khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:53582->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission saishf-kuro-lotus-10-7b_v1: ('http://saishf-kuro-lotus-10-7b-v1-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:47454->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 278, in _get_conversation_tag conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Failed to get response for submission blend_ridut_2024-03-28: 'ChatApi' object has no attribute 'tag'
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 278, in _get_conversation_tag conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 278, in _get_conversation_tag conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
10th percentile: 1.2588603973388672
Received score %s for bot %s
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 278, in _get_conversation_tag conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Failed to get response for submission cgato-thespice-8x7b-ft-v0-1-1_v4: ('http://cgato-thespice-8x7b-ft-v0-1-1-v4-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:56476->127.0.0.1:8080: read: connection reset by peer\n')
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Retrying (%r) after connection broken by '%r': %s
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v37: ('http://thanhdaonguyen-once-upon-a-t-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:52304->127.0.0.1:8080: read: connection reset by peer\n')
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Failed to get response for submission anhnv125-llama-op-v17-1_v32: ('http://anhnv125-llama-op-v17-1-v32-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:33080->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v37: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: ('http://thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:57462->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission neversleep-noromaid-v0-1_8068_v1: ('http://neversleep-noromaid-v0-1-8068-v1-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:59380->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission chaiml-phase2-winner-13b2_v225: ('http://chaiml-phase2-winner-13b2-v225-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:40204->127.0.0.1:8080: read: connection reset by peer\n')
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Scoring model output for bot %s
Pipeline stage DaemonicSafetyScorer completed in 0.05s
Received score %s for bot %s
Scoring model output for bot %s
Scoring model output for bot %s
Received score %s for bot %s
Scoring model output for bot %s
Received score %s for bot %s
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Exception raised while processing tagging_function
Failed to get response for submission chaiml-phase2-winner-13b2_v256: ('http://chaiml-phase2-winner-13b2-v256-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'readfrom tcp 127.0.0.1:60790->127.0.0.1:8080: write tcp 127.0.0.1:60790->127.0.0.1:8080: use of closed network connection\n')
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v21: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v21-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f94b830ebe0>, 'Connection to thanhdaonguyen-once-upon-a-t-v21-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud timed out. (connect timeout=5.5)'))
Failed to get response for submission khanhnto-khanhnto_v39: HTTPConnectionPool(host='khanhnto-khanhnto-v39-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f94b8156f40>, 'Connection to khanhnto-khanhnto-v39-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud timed out. (connect timeout=5.5)'))
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission blend_jofin_2024-02-09: HTTPConnectionPool(host='khoantap-tighthuggerrp-v7-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efec26377c0>: Failed to establish a new connection: [Errno 110] Connection timed out'))
Failed to get response for submission khanhnto-khanhnto_v39: HTTPConnectionPool(host='khanhnto-khanhnto-v39-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fade7250670>, 'Connection to khanhnto-khanhnto-v39-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud timed out. (connect timeout=5.5)'))
Failed to get response for submission khanhnto-khanhnto_v55: HTTPConnectionPool(host='khanhnto-khanhnto-v55-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Max retries exceeded with url: /v1/models/GPT-J-6B-lit-v2:predict (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f8f28245400>, 'Connection to khanhnto-khanhnto-v55-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud timed out. (connect timeout=5.5)'))
Exception raised while processing tagging_function
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Failed to get response for submission hflserdaniel-chai-s6-13b-slp3_v6: ('http://hflserdaniel-chai-s6-13b-slp3-v6-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:38724->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission chaiml-phase2-winner-13b2_v256: HTTPConnectionPool(host='chaiml-phase2-winner-13b2-v256-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission hflserdaniel-chai-s6-13b-slp3_v6: HTTPConnectionPool(host='hflserdaniel-chai-s6-13b-slp3-v6-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v37: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Failed to get response for submission hflserdaniel-chai-s6-13b-slp3_v6: ('http://hflserdaniel-chai-s6-13b-slp3-v6-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:46368->127.0.0.1:8080: read: connection reset by peer\n')
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
fredithefish-starfishrp-v2-v1-mkmlizer: creating bucket guanaco-mkml-models
Failed to get response for submission hflserdaniel-chai-s6-13b-slp3_v6: ('http://hflserdaniel-chai-s6-13b-slp3-v6-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:42318->127.0.0.1:8080: read: connection reset by peer\n')
Scoring model output for bot %s
Received score %s for bot %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Received score %s for bot %s
Failed to get response for submission blend_supof_2024-03-03: ('http://thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:41864->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
fredithefish-starfishrp-v4-mkmlizer: quantizing model to /dev/shm/model_cache
Running M-Eval for topic stay_in_character
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Scoring model output for bot %s
Failed to get response for submission roleplay4fun-staticpunch-v1-0_v4: ('http://roleplay4fun-staticpunch-v1-0-v4-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:42902->127.0.0.1:8080: read: connection reset by peer\n')
Received score %s for bot %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Exception raised while processing tagging_function
Failed to get response for submission chaiml-phase2-winner-13b2_v256: HTTPConnectionPool(host='chaiml-phase2-winner-13b2-v256-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission chaiml-phase2-winner-13b2_v256: HTTPConnectionPool(host='chaiml-phase2-winner-13b2-v256-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission nitral-ai-kunocchini-7b-_1360_v1: ('http://nitral-ai-kunocchini-7b-1360-v1-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:43490->127.0.0.1:8080: read: connection reset by peer\n')
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v37: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission blend_tejos_2024-02-09: ('http://khanhnto-khanhnto-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:44174->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Failed to get response for submission hflserdaniel-chai-s6-13b-slp3_v6: HTTPConnectionPool(host='hflserdaniel-chai-s6-13b-slp3-v6-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Failed to get response for submission khanhnto-khanhnto_v55: HTTPConnectionPool(host='khanhnto-khanhnto-v55-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
anhnv125-mistral-v3-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Failed to get response for submission chaiml-phase2-winner-13b2_v256: ('http://chaiml-phase2-winner-13b2-v256-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:41616->127.0.0.1:8080: read: connection reset by peer\n')
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
cgato-thespice-7b-ft-ins-5091-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/cgato-thespice-7b-ft-ins-5091-v1/special_tokens_map.json
Scoring model output for bot %s
Received score %s for bot %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission anhnv125-llama-op-v17-1_v32: ('http://anhnv125-llama-op-v17-1-v32-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:58318->127.0.0.1:8080: read: connection reset by peer\n')
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission hflserdaniel-chai-s6-13b-slp3_v6: HTTPConnectionPool(host='hflserdaniel-chai-s6-13b-slp3-v6-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: ('http://thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:60650->127.0.0.1:8080: read: connection reset by peer\n')
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
cgato-thespice-7b-ft-ins-1130-v2-mkmlizer: ║ ║
cgato-thespice-7b-ft-ins-1130-v2-mkmlizer: warnings.warn(
Exception raised while processing tagging_function
Pipeline stage StressChecker completed in 24.90s
Scoring model output for bot %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission hflserdaniel-chai-s6-13b-slp3_v6: HTTPConnectionPool(host='hflserdaniel-chai-s6-13b-slp3-v6-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v39: ('http://khanhnto-khanhnto-v39-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:59110->127.0.0.1:8080: read: connection reset by peer\n')
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v55: HTTPConnectionPool(host='khanhnto-khanhnto-v55-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
mistralai-mistral-7b-ins-6151-v4-mkmlizer: Saving model to /tmp/reward_cache/reward.tensors
Failed to get response for submission sao10k-typhon-mixtral-v1_v2: ('http://sao10k-typhon-mixtral-v1-v2-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:38966->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v54: ('http://khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:53426->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission khanhnto-khanhnto_v55: HTTPConnectionPool(host='khanhnto-khanhnto-v55-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v34: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v34-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission anhnv125-mistral-v2_v5: ('http://anhnv125-mistral-v2-v5-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'request timeout')
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission blend_nodas_2024-02-13: ('http://thanhdaonguyen-once-upon-a-t-v19-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:36096->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission grimjim-kuno-kunoichi-v1-dpo-_v1: ('http://grimjim-kuno-kunoichi-v1-dpo-v1-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:35100->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission khanhnto-khanhnto_v55: HTTPConnectionPool(host='khanhnto-khanhnto-v55-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v37: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission chaiml-phase2-winner-13b2_v256: HTTPConnectionPool(host='chaiml-phase2-winner-13b2-v256-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission khanhnto-khanhnto_v55: HTTPConnectionPool(host='khanhnto-khanhnto-v55-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Failed to get response for submission chaiml-phase2-winner-13b2_v238: ('http://chaiml-phase2-winner-13b2-v238-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:50170->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission hflserdaniel-chai-s6-13b-slp3_v6: HTTPConnectionPool(host='hflserdaniel-chai-s6-13b-slp3-v6-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission hflserdaniel-chai-s6-13b-slp3_v6: HTTPConnectionPool(host='hflserdaniel-chai-s6-13b-slp3-v6-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-mistral-v2_v4: (<class 'guanaco_model_service.schemas.Submission'>, <class 'cachetools.keys._HashedTuple'>, 'submission_id', 'a100-khanhnto-khanhnto_v54')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission blend_pider_2024-02-09: ('http://anhnv125-llama-op-v17-1-v27-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:40404->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v37: ('http://khanhnto-khanhnto-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:56806->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v32: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v32-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Failed to get response for submission chaiml-phase2-winner-13b2_v256: HTTPConnectionPool(host='chaiml-phase2-winner-13b2-v256-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v32: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v32-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
maldv-winter-garden-7b-delta-v1-mkmlizer: Bucket 's3://guanaco-reward-models/' created
Connection pool is full, discarding connection: %s
Scoring model output for bot %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Received score %s for bot %s
Scoring model output for bot %s
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
maldv-electric-mist-7b-v1-mkmlizer: ║ belonging to: ║
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
5 requests
90th percentile: 2.0464465618133545
Received score %s for bot %s
Scoring model output for bot %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission chaiml-phase2-winner-13b2_v256: ('http://chaiml-phase2-winner-13b2-v256-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:46980->127.0.0.1:8080: read: connection reset by peer\n')
Received score %s for bot %s
Received score %s for bot %s
Failed to get response for submission neversleep-noromaidxopen_6111_v4: ('http://neversleep-noromaidxopen-6111-v4-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:49410->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v37: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v37: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Scoring model output for bot %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v32: ('http://anhnv125-llama-op-v17-1-v32-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:41554->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Scoring model output for bot %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v55: HTTPConnectionPool(host='khanhnto-khanhnto-v55-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v27: ('http://anhnv125-llama-op-v17-1-v27-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'readfrom tcp 127.0.0.1:45196->127.0.0.1:8080: write tcp 127.0.0.1:45196->127.0.0.1:8080: write: broken pipe\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission inv-exponenta-alpha-7b_v2: (<class 'guanaco_model_service.schemas.Submission'>, <class 'cachetools.keys._HashedTuple'>, 'submission_id', 'khanhnto-khanhnto_v54')
Failed to get response for submission neversleep-noromaidxopen_2041_v4: ('http://neversleep-noromaidxopen-2041-v4-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:40428->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission cgato-thespice-8x7b-ft-v0-1-1_v4: (<class 'guanaco_model_service.schemas.Submission'>, <class 'cachetools.keys._HashedTuple'>, 'submission_id', 'saishf-kuno-lake-7b_v1')
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission chaiml-phase2-winner-13b2_v256: ('http://chaiml-phase2-winner-13b2-v256-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:44864->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission saishf-fimbulvetr-kuro-l_8724_v3: ('http://saishf-fimbulvetr-kuro-l-8724-v3-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:38236->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Scoring model output for bot %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission blend_tiput_2024-02-09: ('http://khanhnto-khanhnto-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:39692->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v37: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission cgato-thespice-7b-v0-1_v5: (<class 'guanaco_model_service.schemas.Submission'>, <class 'cachetools.keys._HashedTuple'>, 'submission_id', 'katythecutie-lemonaderp-4-5-3_v5')
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission anhnv125-llama-op-v17-1_v27: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v27-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission neversleep-noromaidxopen_6111_v3: ('http://neversleep-noromaidxopen-6111-v3-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:54552->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission chaiml-phase2-winner-13b2_v238: HTTPConnectionPool(host='chaiml-phase2-winner-13b2-v238-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v55: ('http://khanhnto-khanhnto-v55-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:43920->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
nitral-ai-eris-primev4-6-2485-v1-mkmlizer: creating bucket guanaco-reward-models
Received score %s for bot %s
Failed to get response for submission anhnv125-llama-op-v17-1_v27: ('http://anhnv125-llama-op-v17-1-v27-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:58708->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Received score %s for bot %s
Received score %s for bot %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
60th percentile: 1.6031003952026366
Scoring model output for bot %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission saishf-fimbulvetr-kuro-l_8724_v1: ('http://saishf-fimbulvetr-kuro-l-8724-v1-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:56554->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Retrying (%r) after connection broken by '%r': %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission blend_jakil_2024-03-28: ('http://neversleep-noromaidxopen-2041-v2-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:34510->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Failed to get response for submission blend_himel_2024-03-03: ('http://roleplay4fun-staticpunch-v1-0-v4-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:52814->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Retrying (%r) after connection broken by '%r': %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
anhnv125-mistral-test-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/anhnv125-mistral-test-v1/special_tokens_map.json
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v54: ('http://khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:40430->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v39: ('http://khanhnto-khanhnto-v39-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:55734->127.0.0.1:8080: read: connection reset by peer\n')
fredithefish-goatrp2-v2-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
fredithefish-goatrp2-v2-mkmlizer: ║ belonging to: ║
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Retrying (%r) after connection broken by '%r': %s
Received score %s for bot %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Received score %s for bot %s
Scoring model output for bot %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Scoring model output for bot %s
Received score %s for bot %s
Received score %s for bot %s
Received score %s for bot %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission wendyhoang-med-rp-model_v3: ('http://wendyhoang-med-rp-model-v3-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:33324->127.0.0.1:8080: read: connection reset by peer\n')
Exception raised while processing tagging_function
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationMessage' object is not subscriptable
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission hflserdaniel-chai-s6-13b-slp3_v6: ('http://hflserdaniel-chai-s6-13b-slp3-v6-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:40356->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Failed to get response for submission neversleep-noromaidxopen_2041_v2: ('http://neversleep-noromaidxopen-2041-v2-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:40044->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v21: ('http://thanhdaonguyen-once-upon-a-t-v21-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:41860->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v27: ('http://anhnv125-llama-op-v17-1-v27-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:32786->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
cognitivecomputations-do-5004-v1-mkmlizer: creating bucket guanaco-mkml-models
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Running pipeline stage StressChecker
20th percentile: 3.3008821487426756
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Scoring model output for bot %s
Connection pool is full, discarding connection: %s
Received score %s for bot %s
Scoring model output for bot %s
Received score %s for bot %s
Scoring model output for bot %s
Retrying (%r) after connection broken by '%r': %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
%s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v34: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v34-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Retrying (%r) after connection broken by '%r': %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v34: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v34-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
%s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v54: HTTPConnectionPool(host='khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: ('http://thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:50586->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
%s
Connection pool is full, discarding connection: %s
Traceback (most recent call last): File "/code/guanaco/guanaco_services/src/guanaco_model_service/chat_api.py", line 274, in resolve_chat_api conversation_tag = self.tagging_function(conversation_state) File "/home/zongyi/gitlab/zztools/zztools/llm/guanaco/submit_routing_model.py", line 176, in last_user_message_length TypeError: 'ConversationState' object is not subscriptable
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
saishf-top-western-maid-7b-v2-mkmlizer: warnings.warn(
Running pipeline stage StressChecker
Received score %s for bot %s
Received score %s for bot %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v37: ('http://thanhdaonguyen-once-upon-a-t-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:46596->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v27: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v27-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission neversleep-noromaidxopen_6111_v1: ('http://neversleep-noromaidxopen-6111-v1-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:44130->127.0.0.1:8080: read: connection reset by peer\n')
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
%s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v32: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v32-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v27: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v27-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v37: ('http://khanhnto-khanhnto-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:39508->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v54: ('http://khanhnto-khanhnto-v54-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:53634->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Exception raised while processing tagging_function
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
%s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
%s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v37: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v55: ('http://khanhnto-khanhnto-v55-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:33630->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v34: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v34-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
%s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission blend_gekek_2024-03-28: (<class 'guanaco_model_service.schemas.Submission'>, <class 'cachetools.keys._HashedTuple'>, 'submission_id', 'saishf-kuro-lotus-10-7b_v2')
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: ('http://thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:47962->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
%s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Failed to get response for submission chaiml-phase2-winner-13b2_v225: ('http://chaiml-phase2-winner-13b2-v225-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:33968->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission fredithefish-starfishrp_v3: ('http://fredithefish-starfishrp-v3-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:34720->127.0.0.1:8080: read: connection reset by peer\n')
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission saishf-fimbulvetr-kuro-l_8724_v3: (<class 'guanaco_model_service.schemas.Submission'>, <class 'cachetools.keys._HashedTuple'>, 'submission_id', 'anhnv125-mistral-v2_v7')
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
%s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission khanhnto-khanhnto_v37: ('http://khanhnto-khanhnto-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:55778->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v27: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v27-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission blend_gekek_2024-03-28: ('http://katythecutie-lemonaderp-4-5-v11-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:55718->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission chaiml-phase2-winner-13b2_v238: ('http://chaiml-phase2-winner-13b2-v238-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:34278->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission sao10k-solstice-mixtral-v1_v1: ('http://sao10k-solstice-mixtral-v1-v1-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:58512->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
%s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
%s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v32: ('http://anhnv125-llama-op-v17-1-v32-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:36988->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
%s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v37: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v37-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v27: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v27-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v27: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v27-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission anhnv125-llama-op-v17-1_v32: HTTPConnectionPool(host='anhnv125-llama-op-v17-1-v32-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Exception raised while processing tagging_function
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s
Failed to get response for submission thanhdaonguyen-once-upon-a-t_v35: HTTPConnectionPool(host='thanhdaonguyen-once-upon-a-t-v35-predictor-default.tenant-chaiml-guanaco.knative.ord1.coreweave.cloud', port=80): Read timed out. (read timeout=5.5)
Connection pool is full, discarding connection: %s
Connection pool is full, discarding connection: %s

Usage Metrics

Latency Metrics