chaiml-rm-ava-3-models-_45935

developer_uid: rirv938
submission_id: chaiml-rm-ava-3-models-_45935_v6
model_name: chaiml-rm-ava-3-models-_45935_v6
model_group: ChaiML/rm_ava_3_models_c
status: inactive
timestamp: 2026-01-04T19:27:01+00:00
num_battles: 7494
num_wins: 3934
family_friendly_score: 0.5264
family_friendly_standard_error: 0.007061204429840564
submission_type: basic
model_repo: ChaiML/rm_ava_3_models_combo_788_3k_unsafe
model_architecture: LlamaForSequenceClassification
model_num_parameters: 8030261248.0
best_of: 1
max_input_tokens: 1024
max_output_tokens: 1
reward_model: default
display_name: chaiml-rm-ava-3-models-_45935_v6
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/rm_ava_3_models_combo_788_3k_unsafe
model_size: 8B
ranking_group: single
us_pacific_date: 2026-01-04
win_ratio: 0.5249532959701094
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
Running pipeline stage MKMLizer
Starting job with name chaiml-rm-ava-3-models-45935-v6-mkmlizer
Waiting for job on chaiml-rm-ava-3-models-45935-v6-mkmlizer to finish
chaiml-rm-ava-3-models-45935-v6-mkmlizer: bash: cannot set terminal process group (-1): Inappropriate ioctl for device
chaiml-rm-ava-3-models-45935-v6-mkmlizer: bash: no job control in this shell
chaiml-rm-ava-3-models-45935-v6-mkmlizer: /root/miniconda3/envs/nvidia/lib/python3.11/site-packages/mk1/__init__.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
chaiml-rm-ava-3-models-45935-v6-mkmlizer:   __import__('pkg_resources').declare_namespace(__name__)
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║                                                                     ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║                                                                     ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║  Version: 0.30.6+torch280                                           ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║                                                                     ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║  belonging to:                                                      ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║                                                                     ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ║                                                                     ║
chaiml-rm-ava-3-models-45935-v6-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-rm-ava-3-models-45935-v6-mkmlizer: Downloaded to shared memory in 15.358s
chaiml-rm-ava-3-models-45935-v6-mkmlizer: Checking if ChaiML/rm_ava_3_models_combo_788_3k_unsafe already exists in ChaiML
chaiml-rm-ava-3-models-45935-v6-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpxpm6p_dr, device:0
chaiml-rm-ava-3-models-45935-v6-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-rm-ava-3-models-45935-v6-mkmlizer: 
Loading 0:   0%|          | 0.00/291 [00:00<?, ?it/s]
Loading 0:  16%|█▌        | 46.0/291 [00:01<00:05, 44.0it/s]
Loading 0:  16%|█▌        | 46.0/291 [00:01<00:05, 44.0it/s]
Loading 0:  26%|██▋       | 77.0/291 [00:02<00:06, 34.9it/s]
Loading 0:  26%|██▋       | 77.0/291 [00:02<00:06, 34.9it/s]
Loading 0:  38%|███▊      | 112/291 [00:03<00:05, 34.8it/s] 
Loading 0:  38%|███▊      | 112/291 [00:03<00:05, 34.8it/s]
Loading 0:  54%|█████▍    | 158/291 [00:04<00:03, 38.4it/s]
Loading 0:  54%|█████▍    | 158/291 [00:04<00:03, 38.4it/s]
Loading 0:  68%|██████▊   | 199/291 [00:05<00:02, 38.5it/s]
Loading 0:  68%|██████▊   | 199/291 [00:05<00:02, 38.5it/s]
Loading 0:  85%|████████▍ | 247/291 [00:06<00:01, 41.5it/s]
Loading 0:  85%|████████▍ | 247/291 [00:06<00:01, 41.5it/s]
Loading 0:  99%|█████████▊| 287/291 [00:07<00:00, 39.9it/s]
Loading 0:  99%|█████████▊| 287/291 [00:07<00:00, 39.9it/s]
Loading 0: 100%|██████████| 291/291 [00:07<00:00, 39.9it/s]
                                                           

Loading 0: 100%|██████████| 291/291 [00:07<00:00, 39.1it/s]
chaiml-rm-ava-3-models-45935-v6-mkmlizer: quantized model in 18.192s
chaiml-rm-ava-3-models-45935-v6-mkmlizer: Processed model ChaiML/rm_ava_3_models_combo_788_3k_unsafe in 33.551s
chaiml-rm-ava-3-models-45935-v6-mkmlizer: creating bucket guanaco-mkml-models
chaiml-rm-ava-3-models-45935-v6-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-rm-ava-3-models-45935-v6-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-rm-ava-3-models-45935-v6/nvidia
chaiml-rm-ava-3-models-45935-v6-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-rm-ava-3-models-45935-v6/nvidia/config.json
chaiml-rm-ava-3-models-45935-v6-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-rm-ava-3-models-45935-v6/nvidia/special_tokens_map.json
chaiml-rm-ava-3-models-45935-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-rm-ava-3-models-45935-v6/nvidia/tokenizer_config.json
chaiml-rm-ava-3-models-45935-v6-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-rm-ava-3-models-45935-v6/nvidia/tokenizer.json
chaiml-rm-ava-3-models-45935-v6-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-rm-ava-3-models-45935-v6/nvidia/flywheel_model.0.safetensors
Job chaiml-rm-ava-3-models-45935-v6-mkmlizer completed after 103.92s with status: succeeded
Stopping job with name chaiml-rm-ava-3-models-45935-v6-mkmlizer
Pipeline stage MKMLizer completed in 104.53s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-rm-ava-3-models-45935-v6
Waiting for inference service chaiml-rm-ava-3-models-45935-v6 to be ready
Failed to get response for submission blend_jotik_2025-11-05: HTTPConnectionPool(host='guanaco-model-mesh-load-balancer.model-mesh.k2.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service chaiml-rm-ava-3-models-45935-v6 ready after 150.60363602638245s
Pipeline stage MKMLDeployer completed in 151.11s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 3.7380478382110596s
{"detail":"('http://guanaco-model-mesh-load-balancer.model-mesh.k2.chaiverse.com/models/chaiml-2fe5-c13f-linear-w01_v12/predict', '{\"detail\":\"1 validation error for RuntimeResponse\\\\npredictions\\\\n  Field required [type=missing, input_value={\\'detail\\': \\\\\"Cannot connec... (\\'10.0.32.80\\', 8080)]\\\\\"}, input_type=dict]\\\\n    For further information visit https://errors.pydantic.dev/2.11/v/missing\"}')"}
Received unhealthy response to inference request!
Received healthy response to inference request in 4.539062023162842s
Received healthy response to inference request in 3.0576345920562744s
{"detail":"HTTPConnectionPool(host='guanaco-model-mesh-load-balancer.model-mesh.k2.chaiverse.com', port=80): Read timed out. (read timeout=12.0)"}
Received unhealthy response to inference request!
5 requests
2 failed requests
5th percentile: 1.078099298477173
10th percentile: 1.5729831218719483
20th percentile: 2.562750768661499
30th percentile: 3.1937172412872314
40th percentile: 3.4658825397491455
50th percentile: 3.7380478382110596
60th percentile: 4.058453512191773
70th percentile: 4.378859186172485
80th percentile: 6.073221874237062
90th percentile: 9.1415415763855
95th percentile: 10.675701427459716
99th percentile: 11.903029308319091
mean time: 4.825564241409301
%s, retrying in %s seconds...
Received healthy response to inference request in 4.514638662338257s
Received healthy response to inference request in 4.460996866226196s
Received healthy response to inference request in 2.2232329845428467s
Received healthy response to inference request in 3.0516550540924072s
Received healthy response to inference request in 3.0772459506988525s
5 requests
0 failed requests
5th percentile: 2.388917398452759
10th percentile: 2.554601812362671
20th percentile: 2.885970640182495
30th percentile: 3.056773233413696
40th percentile: 3.0670095920562743
50th percentile: 3.0772459506988525
60th percentile: 3.6307463169097898
70th percentile: 4.184246683120727
80th percentile: 4.471725225448608
90th percentile: 4.493181943893433
95th percentile: 4.503910303115845
99th percentile: 4.512492990493774
mean time: 3.465553903579712
Pipeline stage StressChecker completed in 44.06s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.66s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.59s
Shutdown handler de-registered
chaiml-rm-ava-3-models-_45935_v6 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
%s, retrying in %s seconds...
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service chaiml-rm-ava-3-models-45935-v6-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 0.85s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.08s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-rm-ava-3-models-45935-v6-profiler
Waiting for inference service chaiml-rm-ava-3-models-45935-v6-profiler to be ready
Inference service chaiml-rm-ava-3-models-45935-v6-profiler ready after 151.3863492012024s
Pipeline stage MKMLProfilerDeployer completed in 151.88s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl --kubeconfig /code/guanaco/guanaco_services/resources/kchai_coreweave_us_east_04a.yaml cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/chaiml-rm-ava-3-mode9908d929ae238d4de1df270122506619-deplonqd95:/code/chaiverse_profiler_1767553305 --namespace tenant-chaiml-guanaco
kubectl --kubeconfig /code/guanaco/guanaco_services/resources/kchai_coreweave_us_east_04a.yaml exec -it chaiml-rm-ava-3-mode9908d929ae238d4de1df270122506619-deplonqd95 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1767553305 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 1 --summary /code/chaiverse_profiler_1767553305/summary.json'
%s, retrying in %s seconds...
kubectl --kubeconfig /code/guanaco/guanaco_services/resources/kchai_coreweave_us_east_04a.yaml cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/chaiml-rm-ava-3-mode9908d929ae238d4de1df270122506619-deplonqd95:/code/chaiverse_profiler_1767553777 --namespace tenant-chaiml-guanaco
kubectl --kubeconfig /code/guanaco/guanaco_services/resources/kchai_coreweave_us_east_04a.yaml exec -it chaiml-rm-ava-3-mode9908d929ae238d4de1df270122506619-deplonqd95 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1767553777 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 1 --summary /code/chaiverse_profiler_1767553777/summary.json'
%s, retrying in %s seconds...
kubectl --kubeconfig /code/guanaco/guanaco_services/resources/kchai_coreweave_us_east_04a.yaml cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/chaiml-rm-ava-3-mode9908d929ae238d4de1df270122506619-deplonqd95:/code/chaiverse_profiler_1767554271 --namespace tenant-chaiml-guanaco
kubectl --kubeconfig /code/guanaco/guanaco_services/resources/kchai_coreweave_us_east_04a.yaml exec -it chaiml-rm-ava-3-mode9908d929ae238d4de1df270122506619-deplonqd95 --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1767554271 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 1 --summary /code/chaiverse_profiler_1767554271/summary.json'
clean up pipeline due to error=ISVCScriptError('Command failed with error: Defaulted container "kserve-container" out of: kserve-container, queue-proxy\nUnable to use a TTY - input is not a terminal or the right kind of file\n\n  0%|          | 0/200 [00:00<?, ?it/s]\n  0%|          | 1/200 [00:00<01:13,  2.70it/s]\n  1%|          | 2/200 [00:00<01:08,  2.89it/s]\n  2%|▏         | 3/200 [00:01<01:05,  3.01it/s]\n  2%|▏         | 4/200 [00:01<01:07,  2.90it/s]\n  2%|▎         | 5/200 [00:05<05:26,  1.67s/it]\n  3%|▎         | 6/200 [00:05<03:44,  1.16s/it]\n  4%|▎         | 7/200 [00:09<06:40,  2.07s/it]\n  4%|▍         | 8/200 [00:09<04:50,  1.51s/it]\n  4%|▍         | 9/200 [00:10<03:38,  1.14s/it]\n  5%|▌         | 10/200 [00:10<02:50,  1.11it/s]\n  6%|▌         | 11/200 [00:10<02:17,  1.37it/s]\n  6%|▌         | 12/200 [00:14<05:15,  1.68s/it]\n  6%|▋         | 13/200 [00:18<07:14,  2.32s/it]\n  7%|▋         | 14/200 [00:22<08:36,  2.78s/it]\n  8%|▊         | 15/200 [00:26<09:32,  3.09s/it]\n  8%|▊         | 16/200 [00:29<10:09,  3.31s/it]\n  8%|▊         | 17/200 [00:33<10:33,  3.46s/it]\n  9%|▉         | 18/200 [00:37<10:48,  3.56s/it]\n 10%|▉         | 19/200 [00:41<10:58,  3.64s/it]\n 10%|█         | 20/200 [00:45<11:11,  3.73s/it]\n 10%|█         | 21/200 [00:45<07:55,  2.66s/it]\n 11%|█         | 22/200 [00:49<09:02,  3.05s/it]\n 12%|█▏        | 23/200 [00:49<06:34,  2.23s/it]\n 12%|█▏        | 24/200 [00:53<07:57,  2.72s/it]\n 12%|█▎        | 25/200 [00:53<05:49,  2.00s/it]\n 13%|█▎        | 26/200 [00:57<07:24,  2.56s/it]\n 14%|█▎        | 27/200 [01:01<08:34,  2.97s/it]\n 14%|█▍        | 28/200 [01:02<06:13,  2.17s/it]\n 14%|█▍        | 29/200 [01:02<04:38,  1.63s/it]\n 15%|█▌        | 30/200 [01:02<03:31,  1.24s/it]\n 16%|█▌        | 31/200 [01:06<05:41,  2.02s/it]\n 16%|█▌        | 32/200 [01:10<07:09,  2.56s/it]\n 16%|█▋        | 33/200 [01:10<05:14,  1.88s/it]\n 17%|█▋        | 34/200 [01:11<03:54,  1.41s/it]\n 18%|█▊        | 35/200 [01:15<05:59,  2.18s/it]\n 18%|█▊        | 36/200 [01:15<04:18,  1.58s/it]\n 18%|█▊        | 37/200 [01:19<06:14,  2.30s/it]\n 19%|█▉        | 38/200 [01:19<04:37,  1.71s/it]\n 20%|█▉        | 39/200 [01:19<03:30,  1.30s/it]\n 20%|██        | 40/200 [01:23<05:30,  2.07s/it]\n 20%|██        | 41/200 [01:27<06:52,  2.59s/it]\n 21%|██        | 42/200 [01:27<05:08,  1.95s/it]\n 22%|██▏       | 43/200 [01:31<06:35,  2.52s/it]\n 22%|██▏       | 44/200 [01:35<07:33,  2.91s/it]\n 22%|██▎       | 45/200 [01:39<08:11,  3.17s/it]\n 23%|██▎       | 46/200 [01:43<08:38,  3.37s/it]\n 24%|██▎       | 47/200 [01:43<06:14,  2.45s/it]\n 24%|██▍       | 48/200 [01:47<07:21,  2.90s/it]\n 24%|██▍       | 49/200 [01:51<08:00,  3.18s/it]\n 25%|██▌       | 50/200 [01:55<08:26,  3.38s/it]\n 26%|██▌       | 51/200 [01:59<08:44,  3.52s/it]\n 26%|██▌       | 52/200 [02:02<08:54,  3.61s/it]\n 26%|██▋       | 53/200 [02:06<09:09,  3.74s/it]\n 27%|██▋       | 54/200 [02:10<09:11,  3.78s/it]\n 28%|██▊       | 55/200 [02:11<06:38,  2.75s/it]\n 28%|██▊       | 56/200 [02:11<04:51,  2.03s/it]\n 28%|██▊       | 57/200 [02:15<06:09,  2.58s/it]\n 29%|██▉       | 58/200 [02:15<04:30,  1.91s/it]\n 30%|██▉       | 59/200 [02:16<03:23,  1.44s/it]\n 30%|███       | 60/200 [02:19<05:04,  2.18s/it]\n 30%|███       | 61/200 [02:23<06:13,  2.69s/it]\n 31%|███       | 62/200 [02:27<06:58,  3.03s/it]\n 32%|███▏      | 63/200 [02:27<05:03,  2.21s/it]\n 32%|███▏      | 64/200 [02:28<03:52,  1.71s/it]\n 32%|███▎      | 65/200 [02:32<05:19,  2.37s/it]\n 33%|███▎      | 66/200 [02:36<06:16,  2.81s/it]\n 34%|███▎      | 67/200 [02:36<04:34,  2.06s/it]\n 34%|███▍      | 68/200 [02:40<05:42,  2.60s/it]\n 34%|███▍      | 69/200 [02:44<06:27,  2.96s/it]\n 35%|███▌      | 70/200 [02:44<04:41,  2.17s/it]\n 36%|███▌      | 71/200 [02:48<05:44,  2.67s/it]\n 36%|███▌      | 72/200 [02:52<06:27,  3.02s/it]\n 36%|███▋      | 73/200 [02:56<06:54,  3.26s/it]\n 37%|███▋      | 74/200 [02:56<05:00,  2.39s/it]\n 38%|███▊      | 75/200 [02:56<03:40,  1.76s/it]\n 38%|███▊      | 76/200 [03:00<05:01,  2.43s/it]\n 38%|███▊      | 77/200 [03:04<05:52,  2.87s/it]\n 39%|███▉      | 78/200 [03:08<06:24,  3.15s/it]\n 40%|███▉      | 79/200 [03:08<04:38,  2.30s/it]\n 40%|████      | 80/200 [03:12<05:31,  2.76s/it]\n 40%|████      | 81/200 [03:12<04:01,  2.03s/it]\n 41%|████      | 82/200 [03:13<02:59,  1.52s/it]\n 42%|████▏     | 83/200 [03:13<02:16,  1.16s/it]\n 42%|████▏     | 84/200 [03:13<01:45,  1.09it/s]\n 42%|████▎     | 85/200 [03:14<01:24,  1.36it/s]\n 43%|████▎     | 86/200 [03:17<03:10,  1.67s/it]\n 44%|████▎     | 87/200 [03:18<02:21,  1.26s/it]\n 44%|████▍     | 88/200 [03:22<03:51,  2.07s/it]\n 44%|████▍     | 89/200 [03:26<04:48,  2.60s/it]\n 45%|████▌     | 90/200 [03:29<05:25,  2.96s/it]\n 46%|████▌     | 91/200 [03:33<05:51,  3.22s/it]\n 46%|████▌     | 92/200 [03:34<04:13,  2.35s/it]\n 46%|████▋     | 93/200 [03:34<03:06,  1.74s/it]\n 47%|████▋     | 94/200 [03:34<02:19,  1.32s/it]\n 48%|████▊     | 95/200 [03:38<03:37,  2.07s/it]\n 48%|████▊     | 96/200 [03:38<02:40,  1.54s/it]\n 48%|████▊     | 97/200 [03:39<02:01,  1.18s/it]\n 49%|████▉     | 98/200 [03:43<03:26,  2.03s/it]\n 50%|████▉     | 99/200 [03:43<02:33,  1.52s/it]\n 50%|█████     | 100/200 [03:43<01:57,  1.18s/it]\n 50%|█████     | 101/200 [03:44<01:31,  1.08it/s]\n 51%|█████     | 102/200 [03:44<01:13,  1.33it/s]\n 52%|█████▏    | 103/200 [03:48<02:43,  1.68s/it]\n 52%|█████▏    | 104/200 [03:48<02:02,  1.27s/it]\n 52%|█████▎    | 105/200 [03:52<03:15,  2.05s/it]\n 53%|█████▎    | 106/200 [03:56<04:03,  2.59s/it]\n 54%|█████▎    | 107/200 [03:56<02:57,  1.91s/it]\n 54%|█████▍    | 108/200 [03:57<02:12,  1.44s/it]\n 55%|█████▍    | 109/200 [04:01<03:17,  2.17s/it]\n 55%|█████▌    | 110/200 [04:04<04:04,  2.72s/it]\n 56%|█████▌    | 111/200 [04:08<04:31,  3.05s/it]\n 56%|█████▌    | 112/200 [04:09<03:15,  2.22s/it]\n 56%|█████▋    | 113/200 [04:13<03:59,  2.75s/it]\n 57%|█████▋    | 114/200 [04:13<02:49,  1.97s/it]\n 57%|█████▊    | 115/200 [04:13<02:09,  1.53s/it]\n 58%|█████▊    | 116/200 [04:14<01:38,  1.17s/it]\n 58%|█████▊    | 117/200 [04:17<02:43,  1.97s/it]\n 59%|█████▉    | 118/200 [04:21<03:26,  2.52s/it]\n 60%|█████▉    | 119/200 [04:25<03:55,  2.91s/it]\n 60%|██████    | 120/200 [04:29<04:17,  3.21s/it]\n 60%|██████    | 121/200 [04:33<04:29,  3.42s/it]\n 61%|██████    | 122/200 [04:37<04:36,  3.55s/it]\n 62%|██████▏   | 123/200 [04:41<04:41,  3.65s/it]\n 62%|██████▏   | 124/200 [04:41<03:21,  2.65s/it]\n 62%|██████▎   | 125/200 [04:45<03:48,  3.04s/it]\n 63%|██████▎   | 126/200 [04:49<04:02,  3.27s/it]\n 64%|██████▎   | 127/200 [04:49<02:54,  2.39s/it]\n 64%|██████▍   | 128/200 [04:53<03:23,  2.82s/it]\n 64%|██████▍   | 129/200 [04:53<02:26,  2.07s/it]\n 65%|██████▌   | 130/200 [04:53<01:48,  1.55s/it]\n 66%|██████▌   | 131/200 [04:57<02:34,  2.23s/it]\n 66%|██████▌   | 132/200 [05:01<03:04,  2.71s/it]\n 66%|██████▋   | 133/200 [05:05<03:23,  3.04s/it]\n 67%|██████▋   | 134/200 [05:09<03:35,  3.26s/it]\n 68%|██████▊   | 135/200 [05:13<03:45,  3.48s/it]\n 68%|██████▊   | 136/200 [05:13<02:38,  2.48s/it]\n 68%|██████▊   | 137/200 [05:13<01:58,  1.87s/it]\n 69%|██████▉   | 138/200 [05:14<01:27,  1.42s/it]\n 70%|██████▉   | 139/200 [05:18<02:11,  2.15s/it]\n 70%|███████   | 140/200 [05:21<02:38,  2.65s/it]\n 70%|███████   | 141/200 [05:25<02:57,  3.01s/it]\n 71%|███████   | 142/200 [05:29<03:08,  3.25s/it]\n 72%|███████▏  | 143/200 [05:33<03:14,  3.42s/it]\n 72%|███████▏  | 144/200 [05:37<03:17,  3.54s/it]\n 72%|███████▎  | 145/200 [05:37<02:21,  2.57s/it]\n 73%|███████▎  | 146/200 [05:37<01:42,  1.90s/it]\n 74%|███████▎  | 147/200 [05:38<01:15,  1.43s/it]\n 74%|███████▍  | 148/200 [05:41<01:51,  2.15s/it]\n 74%|███████▍  | 149/200 [05:45<02:14,  2.64s/it]\n 75%|███████▌  | 150/200 [05:46<01:37,  1.94s/it]\n 76%|███████▌  | 151/200 [05:50<02:04,  2.55s/it]\n 76%|███████▌  | 152/200 [05:53<02:20,  2.93s/it]\n 76%|███████▋  | 153/200 [05:57<02:30,  3.19s/it]\n 77%|███████▋  | 154/200 [06:01<02:35,  3.39s/it]\n 78%|███████▊  | 155/200 [06:05<02:38,  3.52s/it]\n 78%|███████▊  | 156/200 [06:09<02:39,  3.62s/it]\n 78%|███████▊  | 157/200 [06:12<02:38,  3.68s/it]\n 79%|███████▉  | 158/200 [06:13<01:52,  2.67s/it]\n 80%|███████▉  | 159/200 [06:13<01:20,  1.97s/it]\n 80%|████████  | 160/200 [06:17<01:41,  2.53s/it]\n 80%|████████  | 161/200 [06:21<01:53,  2.91s/it]\n 81%|████████  | 162/200 [06:25<02:01,  3.18s/it]\n 82%|████████▏ | 163/200 [06:25<01:26,  2.34s/it]\n 82%|████████▏ | 164/200 [06:29<01:40,  2.80s/it]\n 82%|████████▎ | 165/200 [06:29<01:12,  2.06s/it]\n 83%|████████▎ | 166/200 [06:33<01:29,  2.64s/it]\n 84%|████████▎ | 167/200 [06:37<01:38,  2.99s/it]\n 84%|████████▍ | 168/200 [06:37<01:09,  2.19s/it]\n 84%|████████▍ | 169/200 [06:41<01:23,  2.69s/it]\n 85%|████████▌ | 170/200 [06:45<01:30,  3.03s/it]\n 86%|████████▌ | 171/200 [06:49<01:34,  3.26s/it]\n 86%|████████▌ | 172/200 [06:49<01:06,  2.38s/it]\n 86%|████████▋ | 173/200 [06:49<00:47,  1.77s/it]\n 87%|████████▋ | 174/200 [06:50<00:34,  1.34s/it]\n 88%|████████▊ | 175/200 [06:54<00:52,  2.10s/it]\n 88%|████████▊ | 176/200 [06:54<00:37,  1.56s/it]\n 88%|████████▊ | 177/200 [06:58<00:51,  2.25s/it]\n 89%|████████▉ | 178/200 [07:02<01:00,  2.77s/it]\n 90%|████████▉ | 179/200 [07:06<01:04,  3.08s/it]\n 90%|█████████ | 180/200 [07:09<01:06,  3.30s/it]\n 90%|█████████ | 181/200 [07:10<00:45,  2.40s/it]\n 91%|█████████ | 182/200 [07:14<00:51,  2.83s/it]\n 92%|█████████▏| 183/200 [07:17<00:53,  3.12s/it]\n 92%|█████████▏| 184/200 [07:18<00:37,  2.32s/it]\n 92%|█████████▎| 185/200 [07:18<00:25,  1.67s/it]\n 93%|█████████▎| 186/200 [07:18<00:18,  1.32s/it]\n 94%|█████████▎| 187/200 [07:19<00:13,  1.03s/it]\n 94%|█████████▍| 188/200 [07:23<00:22,  1.88s/it]\n 94%|█████████▍| 189/200 [07:23<00:15,  1.41s/it]\n 95%|█████████▌| 190/200 [07:23<00:10,  1.09s/it]\n 96%|█████████▌| 191/200 [07:24<00:07,  1.16it/s]\n 96%|█████████▌| 192/200 [07:28<00:14,  1.76s/it]\n 96%|█████████▋| 193/200 [07:28<00:09,  1.33s/it]\n 97%|█████████▋| 194/200 [07:32<00:12,  2.08s/it]\n 98%|█████████▊| 195/200 [07:32<00:07,  1.55s/it]\n 98%|█████████▊| 196/200 [07:32<00:04,  1.21s/it]\n 98%|█████████▊| 197/200 [07:33<00:02,  1.06it/s]\n 99%|█████████▉| 198/200 [07:37<00:03,  1.82s/it]\n100%|█████████▉| 199/200 [07:37<00:01,  1.36s/it]\n100%|██████████| 200/200 [07:41<00:00,  2.10s/it]\n100%|██████████| 200/200 [07:41<00:00,  2.31s/it]\n/code/chaiverse_profiler_1767554271/profiles.py:401: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.11/migration/\n  return self.dict()\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 653, in <module>\n    cli()\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/click/core.py", line 1485, in __call__\n    return self.main(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/click/core.py", line 1406, in main\n    rv = self.invoke(ctx)\n         ^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/click/core.py", line 1873, in invoke\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\n                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/click/core.py", line 1269, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/click/core.py", line 824, in invoke\n    return callback(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 112, in profile_batches\n    profiles = run_batch_profile_with_auto_batch(target, batches, settings, auto_batch, output)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 182, in run_batch_profile_with_auto_batch\n    profiles = run_batch_profile(target, batches, settings, output)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 296, in run_batch_profile\n    analysis_data.write_jsonlines([batch_profile.to_dict()], path)\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/data.py", line 64, in write_jsonlines\n    f.write(json.dumps(row) + \'\\n\')\n            ^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/json/__init__.py", line 231, in dumps\n    return _default_encoder.encode(obj)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/json/encoder.py", line 200, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/json/encoder.py", line 258, in iterencode\n    return _iterencode(o, 0)\n           ^^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/json/encoder.py", line 180, in default\n    raise TypeError(f\'Object of type {o.__class__.__name__} \'\nTypeError: Object of type ResponseStats is not JSON serializable\ncommand terminated with exit code 1\n, output: waiting for startup of endpoint=\'localhost\' route=\'GPT-J-6B-lit-v2\' namespace=\'tenant-chaiml-guanaco\' reward=False url_format=\'{endpoint}.{namespace}.k.chaiverse.com\'\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767554271/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\n### Batch size: 1 ###\n\ntotal requests 200\nduration (s): 461.21393489837646\nerrors 88\nmean length: 1.68\n\nthroughput (request / second): 0.433638242183788\nthroughput (character / second): 0.7285122468687638\naverage request duration (s) 2.3059888315200805\n50%ile request duration (s) 3.8063430786132812\n75%ile request duration (s) 3.844412088394165\n90%ile request duration (s) 3.9117984294891355\n95%ile request duration (s) 3.9703969717025758\n\nmean input tokens 2.0\nmean output tokens 1.0\n\n\n')
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service chaiml-rm-ava-3-models-45935-v6-profiler is running
Tearing down inference service chaiml-rm-ava-3-models-45935-v6-profiler
Service chaiml-rm-ava-3-models-45935-v6-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 0.72s
Shutdown handler de-registered
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service chaiml-rm-ava-3-models-45935-v6-profiler is running
Skipping teardown as no inference service was found
Pipeline stage MKMLProfilerDeleter completed in 0.72s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.10s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service chaiml-rm-ava-3-models-45935-v6-profiler
Waiting for inference service chaiml-rm-ava-3-models-45935-v6-profiler to be ready
Inference service chaiml-rm-ava-3-models-45935-v6-profiler ready after 151.27271270751953s
Pipeline stage MKMLProfilerDeployer completed in 151.72s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl --kubeconfig /code/guanaco/guanaco_services/resources/kchai_coreweave_us_east_04a.yaml cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/chaiml-rm-ava-3-mode9908d929ae238d4de1df270122506619-deplofbpks:/code/chaiverse_profiler_1767554974 --namespace tenant-chaiml-guanaco
kubectl --kubeconfig /code/guanaco/guanaco_services/resources/kchai_coreweave_us_east_04a.yaml exec -it chaiml-rm-ava-3-mode9908d929ae238d4de1df270122506619-deplofbpks --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1767554974 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 1 --summary /code/chaiverse_profiler_1767554974/summary.json'
%s, retrying in %s seconds...
kubectl --kubeconfig /code/guanaco/guanaco_services/resources/kchai_coreweave_us_east_04a.yaml cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/chaiml-rm-ava-3-mode9908d929ae238d4de1df270122506619-deplofbpks:/code/chaiverse_profiler_1767555476 --namespace tenant-chaiml-guanaco
kubectl --kubeconfig /code/guanaco/guanaco_services/resources/kchai_coreweave_us_east_04a.yaml exec -it chaiml-rm-ava-3-mode9908d929ae238d4de1df270122506619-deplofbpks --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1767555476 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 1 --summary /code/chaiverse_profiler_1767555476/summary.json'
%s, retrying in %s seconds...
kubectl --kubeconfig /code/guanaco/guanaco_services/resources/kchai_coreweave_us_east_04a.yaml cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/chaiml-rm-ava-3-mode9908d929ae238d4de1df270122506619-deplofbpks:/code/chaiverse_profiler_1767555939 --namespace tenant-chaiml-guanaco
kubectl --kubeconfig /code/guanaco/guanaco_services/resources/kchai_coreweave_us_east_04a.yaml exec -it chaiml-rm-ava-3-mode9908d929ae238d4de1df270122506619-deplofbpks --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1767555939 && python profiles.py profile --best_of_n 1 --auto_batch 5 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 1024 --output_tokens 1 --summary /code/chaiverse_profiler_1767555939/summary.json'
clean up pipeline due to error=ISVCScriptError('Command failed with error: Defaulted container "kserve-container" out of: kserve-container, queue-proxy\nUnable to use a TTY - input is not a terminal or the right kind of file\n\n  0%|          | 0/200 [00:00<?, ?it/s]\n  0%|          | 1/200 [00:00<01:51,  1.78it/s]\n  1%|          | 2/200 [00:00<01:27,  2.27it/s]\n  2%|▏         | 3/200 [00:04<06:37,  2.02s/it]\n  2%|▏         | 4/200 [00:05<04:25,  1.35s/it]\n  2%|▎         | 5/200 [00:05<03:13,  1.01it/s]\n  3%|▎         | 6/200 [00:09<06:23,  1.98s/it]\n  4%|▎         | 7/200 [00:13<08:21,  2.60s/it]\n  4%|▍         | 8/200 [00:17<09:36,  3.00s/it]\n  4%|▍         | 9/200 [00:17<06:53,  2.16s/it]\n  5%|▌         | 10/200 [00:21<08:33,  2.70s/it]\n  6%|▌         | 11/200 [00:25<09:37,  3.06s/it]\n  6%|▌         | 12/200 [00:25<07:06,  2.27s/it]\n  6%|▋         | 13/200 [00:26<05:16,  1.69s/it]\n  7%|▋         | 14/200 [00:26<04:00,  1.29s/it]\n  8%|▊         | 15/200 [00:30<06:25,  2.09s/it]\n  8%|▊         | 16/200 [00:30<04:46,  1.56s/it]\n  8%|▊         | 17/200 [00:31<03:39,  1.20s/it]\n  9%|▉         | 18/200 [00:34<06:06,  2.01s/it]\n 10%|▉         | 19/200 [00:38<07:46,  2.58s/it]\n 10%|█         | 20/200 [00:42<08:54,  2.97s/it]\n 10%|█         | 21/200 [00:46<09:41,  3.25s/it]\n 11%|█         | 22/200 [00:50<10:11,  3.44s/it]\n 12%|█▏        | 23/200 [00:54<10:31,  3.57s/it]\n 12%|█▏        | 24/200 [00:58<10:44,  3.66s/it]\n 12%|█▎        | 25/200 [00:58<07:45,  2.66s/it]\n 13%|█▎        | 26/200 [01:02<08:48,  3.04s/it]\n 14%|█▎        | 27/200 [01:06<09:34,  3.32s/it]\n 14%|█▍        | 28/200 [01:06<06:56,  2.42s/it]\n 14%|█▍        | 29/200 [01:10<08:09,  2.86s/it]\n 15%|█▌        | 30/200 [01:11<05:57,  2.10s/it]\n 16%|█▌        | 31/200 [01:14<07:26,  2.64s/it]\n 16%|█▌        | 32/200 [01:15<05:26,  1.94s/it]\n 16%|█▋        | 33/200 [01:19<07:09,  2.57s/it]\n 17%|█▋        | 34/200 [01:19<05:17,  1.91s/it]\n 18%|█▊        | 35/200 [01:23<06:54,  2.51s/it]\n 18%|█▊        | 36/200 [01:27<07:59,  2.92s/it]\n 18%|█▊        | 37/200 [01:27<05:50,  2.15s/it]\n 19%|█▉        | 38/200 [01:31<07:13,  2.68s/it]\n 20%|█▉        | 39/200 [01:32<05:17,  1.97s/it]\n 20%|██        | 40/200 [01:35<06:48,  2.55s/it]\n 20%|██        | 41/200 [01:39<07:48,  2.95s/it]\n 21%|██        | 42/200 [01:40<05:41,  2.16s/it]\n 22%|██▏       | 43/200 [01:40<04:13,  1.62s/it]\n 22%|██▏       | 44/200 [01:40<03:13,  1.24s/it]\n 22%|██▎       | 45/200 [01:44<05:15,  2.04s/it]\n 23%|██▎       | 46/200 [01:45<03:54,  1.52s/it]\n 24%|██▎       | 47/200 [01:48<05:42,  2.24s/it]\n 24%|██▍       | 48/200 [01:52<06:54,  2.73s/it]\n 24%|██▍       | 49/200 [01:56<07:45,  3.08s/it]\n 25%|██▌       | 50/200 [01:57<05:38,  2.25s/it]\n 26%|██▌       | 51/200 [02:00<06:48,  2.74s/it]\n 26%|██▌       | 52/200 [02:04<07:44,  3.14s/it]\n 26%|██▋       | 53/200 [02:05<05:29,  2.24s/it]\n 27%|██▋       | 54/200 [02:09<06:47,  2.79s/it]\n 28%|██▊       | 55/200 [02:13<07:32,  3.12s/it]\n 28%|██▊       | 56/200 [02:13<05:28,  2.28s/it]\n 28%|██▊       | 57/200 [02:13<04:03,  1.70s/it]\n 29%|██▉       | 58/200 [02:14<03:04,  1.30s/it]\n 30%|██▉       | 59/200 [02:18<04:53,  2.08s/it]\n 30%|███       | 60/200 [02:21<06:07,  2.62s/it]\n 30%|███       | 61/200 [02:22<04:28,  1.93s/it]\n 31%|███       | 62/200 [02:26<05:49,  2.53s/it]\n 32%|███▏      | 63/200 [02:30<06:42,  2.94s/it]\n 32%|███▏      | 64/200 [02:33<07:17,  3.22s/it]\n 32%|███▎      | 65/200 [02:37<07:40,  3.41s/it]\n 33%|███▎      | 66/200 [02:41<07:57,  3.56s/it]\n 34%|███▎      | 67/200 [02:42<05:44,  2.59s/it]\n 34%|███▍      | 68/200 [02:42<04:12,  1.91s/it]\n 34%|███▍      | 69/200 [02:42<03:08,  1.44s/it]\n 35%|███▌      | 70/200 [02:43<02:24,  1.11s/it]\n 36%|███▌      | 71/200 [02:47<04:17,  1.99s/it]\n 36%|███▌      | 72/200 [02:50<05:27,  2.56s/it]\n 36%|███▋      | 73/200 [02:51<03:59,  1.89s/it]\n 37%|███▋      | 74/200 [02:55<05:14,  2.49s/it]\n 38%|███▊      | 75/200 [02:59<06:03,  2.91s/it]\n 38%|███▊      | 76/200 [02:59<04:24,  2.13s/it]\n 38%|███▊      | 77/200 [03:03<05:30,  2.68s/it]\n 39%|███▉      | 78/200 [03:07<06:10,  3.04s/it]\n 40%|███▉      | 79/200 [03:07<04:29,  2.22s/it]\n 40%|████      | 80/200 [03:11<05:26,  2.72s/it]\n 40%|████      | 81/200 [03:11<03:57,  2.00s/it]\n 41%|████      | 82/200 [03:15<05:08,  2.61s/it]\n 42%|████▏     | 83/200 [03:19<05:49,  2.99s/it]\n 42%|████▏     | 84/200 [03:23<06:20,  3.28s/it]\n 42%|████▎     | 85/200 [03:27<06:37,  3.46s/it]\n 43%|████▎     | 86/200 [03:27<04:47,  2.52s/it]\n 44%|████▎     | 87/200 [03:31<05:32,  2.94s/it]\n 44%|████▍     | 88/200 [03:32<04:01,  2.15s/it]\n 44%|████▍     | 89/200 [03:36<04:57,  2.68s/it]\n 45%|████▌     | 90/200 [03:36<03:37,  1.98s/it]\n 46%|████▌     | 91/200 [03:40<04:37,  2.55s/it]\n 46%|████▌     | 92/200 [03:44<05:17,  2.94s/it]\n 46%|████▋     | 93/200 [03:44<03:50,  2.15s/it]\n 47%|████▋     | 94/200 [03:48<04:43,  2.67s/it]\n 48%|████▊     | 95/200 [03:52<05:17,  3.03s/it]\n 48%|████▊     | 96/200 [03:52<03:55,  2.26s/it]\n 48%|████▊     | 97/200 [03:52<02:54,  1.69s/it]\n 49%|████▉     | 98/200 [03:56<04:00,  2.36s/it]\n 50%|████▉     | 99/200 [03:57<02:56,  1.75s/it]\n 50%|█████     | 100/200 [04:01<03:59,  2.40s/it]\n 50%|█████     | 101/200 [04:05<04:42,  2.85s/it]\n 51%|█████     | 102/200 [04:08<05:09,  3.16s/it]\n 52%|█████▏    | 103/200 [04:09<03:43,  2.31s/it]\n 52%|█████▏    | 104/200 [04:13<04:26,  2.78s/it]\n 52%|█████▎    | 105/200 [04:13<03:14,  2.04s/it]\n 53%|█████▎    | 106/200 [04:13<02:23,  1.53s/it]\n 54%|█████▎    | 107/200 [04:14<01:49,  1.17s/it]\n 54%|█████▍    | 108/200 [04:14<01:24,  1.09it/s]\n 55%|█████▍    | 109/200 [04:18<02:44,  1.81s/it]\n 55%|█████▌    | 110/200 [04:18<02:02,  1.36s/it]\n 56%|█████▌    | 111/200 [04:18<01:33,  1.05s/it]\n 56%|█████▌    | 112/200 [04:23<02:51,  1.95s/it]\n 56%|█████▋    | 113/200 [04:26<03:39,  2.53s/it]\n 57%|█████▋    | 114/200 [04:30<04:14,  2.96s/it]\n 57%|█████▊    | 115/200 [04:31<03:04,  2.17s/it]\n 58%|█████▊    | 116/200 [04:31<02:16,  1.62s/it]\n 58%|█████▊    | 117/200 [04:35<03:11,  2.30s/it]\n 59%|█████▉    | 118/200 [04:39<03:47,  2.78s/it]\n 60%|█████▉    | 119/200 [04:43<04:11,  3.10s/it]\n 60%|██████    | 120/200 [04:47<04:27,  3.35s/it]\n 60%|██████    | 121/200 [04:47<03:12,  2.44s/it]\n 61%|██████    | 122/200 [04:51<03:43,  2.87s/it]\n 62%|██████▏   | 123/200 [04:55<04:08,  3.22s/it]\n 62%|██████▏   | 124/200 [04:59<04:20,  3.42s/it]\n 62%|██████▎   | 125/200 [05:03<04:27,  3.56s/it]\n 63%|██████▎   | 126/200 [05:03<03:08,  2.54s/it]\n 64%|██████▎   | 127/200 [05:07<03:38,  3.00s/it]\n 64%|██████▍   | 128/200 [05:11<03:54,  3.26s/it]\n 64%|██████▍   | 129/200 [05:15<04:04,  3.45s/it]\n 65%|██████▌   | 130/200 [05:18<04:10,  3.57s/it]\n 66%|██████▌   | 131/200 [05:22<04:13,  3.67s/it]\n 66%|██████▌   | 132/200 [05:26<04:13,  3.73s/it]\n 66%|██████▋   | 133/200 [05:30<04:13,  3.78s/it]\n 67%|██████▋   | 134/200 [05:34<04:11,  3.81s/it]\n 68%|██████▊   | 135/200 [05:38<04:08,  3.83s/it]\n 68%|██████▊   | 136/200 [05:38<02:57,  2.78s/it]\n 68%|██████▊   | 137/200 [05:42<03:16,  3.11s/it]\n 69%|██████▉   | 138/200 [05:42<02:21,  2.27s/it]\n 70%|██████▉   | 139/200 [05:46<02:48,  2.76s/it]\n 70%|███████   | 140/200 [05:47<02:01,  2.03s/it]\n 70%|███████   | 141/200 [05:51<02:35,  2.63s/it]\n 71%|███████   | 142/200 [05:55<02:54,  3.00s/it]\n 72%|███████▏  | 143/200 [05:55<02:05,  2.20s/it]\n 72%|███████▏  | 144/200 [05:55<01:32,  1.64s/it]\n 72%|███████▎  | 145/200 [05:59<02:07,  2.32s/it]\n 73%|███████▎  | 146/200 [06:03<02:30,  2.80s/it]\n 74%|███████▎  | 147/200 [06:07<02:45,  3.12s/it]\n 74%|███████▍  | 148/200 [06:11<02:53,  3.35s/it]\n 74%|███████▍  | 149/200 [06:11<02:04,  2.44s/it]\n 75%|███████▌  | 150/200 [06:11<01:30,  1.81s/it]\n 76%|███████▌  | 151/200 [06:15<02:01,  2.49s/it]\n 76%|███████▌  | 152/200 [06:19<02:19,  2.90s/it]\n 76%|███████▋  | 153/200 [06:23<02:30,  3.20s/it]\n 77%|███████▋  | 154/200 [06:27<02:36,  3.40s/it]\n 78%|███████▊  | 155/200 [06:31<02:39,  3.55s/it]\n 78%|███████▊  | 156/200 [06:31<01:54,  2.60s/it]\n 78%|███████▊  | 157/200 [06:35<02:08,  2.99s/it]\n 79%|███████▉  | 158/200 [06:39<02:16,  3.25s/it]\n 80%|███████▉  | 159/200 [06:43<02:20,  3.43s/it]\n 80%|████████  | 160/200 [06:47<02:22,  3.57s/it]\n 80%|████████  | 161/200 [06:51<02:22,  3.65s/it]\n 81%|████████  | 162/200 [06:51<01:40,  2.65s/it]\n 82%|████████▏ | 163/200 [06:55<01:53,  3.07s/it]\n 82%|████████▏ | 164/200 [06:59<02:00,  3.34s/it]\n 82%|████████▎ | 165/200 [06:59<01:25,  2.44s/it]\n 83%|████████▎ | 166/200 [07:03<01:37,  2.87s/it]\n 84%|████████▎ | 167/200 [07:07<01:44,  3.18s/it]\n 84%|████████▍ | 168/200 [07:07<01:14,  2.32s/it]\n 84%|████████▍ | 169/200 [07:11<01:26,  2.79s/it]\n 85%|████████▌ | 170/200 [07:15<01:33,  3.12s/it]\n 86%|████████▌ | 171/200 [07:16<01:06,  2.28s/it]\n 86%|████████▌ | 172/200 [07:16<00:47,  1.70s/it]\n 86%|████████▋ | 173/200 [07:20<01:03,  2.36s/it]\n 87%|████████▋ | 174/200 [07:24<01:14,  2.86s/it]\n 88%|████████▊ | 175/200 [07:24<00:51,  2.05s/it]\n 88%|████████▊ | 176/200 [07:25<00:37,  1.58s/it]\n 88%|████████▊ | 177/200 [07:25<00:27,  1.22s/it]\n 89%|████████▉ | 178/200 [07:29<00:44,  2.03s/it]\n 90%|████████▉ | 179/200 [07:29<00:31,  1.52s/it]\n 90%|█████████ | 180/200 [07:29<00:23,  1.17s/it]\n 90%|█████████ | 181/200 [07:33<00:37,  2.00s/it]\n 91%|█████████ | 182/200 [07:37<00:46,  2.56s/it]\n 92%|█████████▏| 183/200 [07:38<00:32,  1.89s/it]\n 92%|█████████▏| 184/200 [07:42<00:40,  2.50s/it]\n 92%|█████████▎| 185/200 [07:42<00:27,  1.85s/it]\n 93%|█████████▎| 186/200 [07:46<00:34,  2.47s/it]\n 94%|█████████▎| 187/200 [07:46<00:23,  1.83s/it]\n 94%|█████████▍| 188/200 [07:50<00:29,  2.45s/it]\n 94%|█████████▍| 189/200 [07:50<00:19,  1.81s/it]\n 95%|█████████▌| 190/200 [07:54<00:24,  2.49s/it]\n 96%|█████████▌| 191/200 [07:58<00:26,  2.91s/it]\n 96%|█████████▌| 192/200 [08:02<00:25,  3.23s/it]\n 96%|█████████▋| 193/200 [08:03<00:16,  2.35s/it]\n 97%|█████████▋| 194/200 [08:03<00:10,  1.75s/it]\n 98%|█████████▊| 195/200 [08:07<00:12,  2.45s/it]\n 98%|█████████▊| 196/200 [08:11<00:11,  2.88s/it]\n 98%|█████████▊| 197/200 [08:11<00:06,  2.12s/it]\n 99%|█████████▉| 198/200 [08:15<00:05,  2.66s/it]\n100%|█████████▉| 199/200 [08:16<00:01,  1.96s/it]\n100%|██████████| 200/200 [08:19<00:00,  2.55s/it]\n100%|██████████| 200/200 [08:19<00:00,  2.50s/it]\n/code/chaiverse_profiler_1767555939/profiles.py:401: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.11/migration/\n  return self.dict()\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 653, in <module>\n    cli()\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/click/core.py", line 1485, in __call__\n    return self.main(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/click/core.py", line 1406, in main\n    rv = self.invoke(ctx)\n         ^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/click/core.py", line 1873, in invoke\n    return _process_result(sub_ctx.command.invoke(sub_ctx))\n                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/click/core.py", line 1269, in invoke\n    return ctx.invoke(self.callback, **ctx.params)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/site-packages/click/core.py", line 824, in invoke\n    return callback(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 112, in profile_batches\n    profiles = run_batch_profile_with_auto_batch(target, batches, settings, auto_batch, output)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 182, in run_batch_profile_with_auto_batch\n    profiles = run_batch_profile(target, batches, settings, output)\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 296, in run_batch_profile\n    analysis_data.write_jsonlines([batch_profile.to_dict()], path)\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/data.py", line 64, in write_jsonlines\n    f.write(json.dumps(row) + \'\\n\')\n            ^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/json/__init__.py", line 231, in dumps\n    return _default_encoder.encode(obj)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/json/encoder.py", line 200, in encode\n    chunks = self.iterencode(o, _one_shot=True)\n             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/json/encoder.py", line 258, in iterencode\n    return _iterencode(o, 0)\n           ^^^^^^^^^^^^^^^^^\n  File "/root/miniconda3/envs/guanaco/lib/python3.11/json/encoder.py", line 180, in default\n    raise TypeError(f\'Object of type {o.__class__.__name__} \'\nTypeError: Object of type ResponseStats is not JSON serializable\ncommand terminated with exit code 1\n, output: waiting for startup of endpoint=\'localhost\' route=\'GPT-J-6B-lit-v2\' namespace=\'tenant-chaiml-guanaco\' reward=False url_format=\'{endpoint}.{namespace}.k.chaiverse.com\'\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (1,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : could not broadcast input array from shape (2,) into shape (0,)"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\nRequest failed with: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\nTraceback (most recent call last):\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 606, in get_request\n    resp = get_request_causal_model(inputs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/chaiverse_profiler_1767555939/profiles.py", line 623, in get_request_causal_model\n    resp = target.predict(text, parameters, headers=headers)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/code/guanaco/guanaco_inference_services/src/inference_analysis/client.py", line 28, in predict\n    assert resp.status_code == 200, (resp.status_code, resp.text)\n           ^^^^^^^^^^^^^^^^^^^^^^^\nAssertionError: (500, \'{"error":"Exception : negative dimensions are not allowed"}\')\n\n### Batch size: 1 ###\n\ntotal requests 200\nduration (s): 499.9167447090149\nerrors 79\nmean length: 1.815\n\nthroughput (request / second): 0.4000666153249446\nthroughput (character / second): 0.7261209068147745\naverage request duration (s) 2.4995166289806368\n50%ile request duration (s) 3.87353253364563\n75%ile request duration (s) 3.900469422340393\n90%ile request duration (s) 3.936157894134521\n95%ile request duration (s) 4.037367630004883\n\nmean input tokens 2.0\nmean output tokens 1.0\n\n\n')
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service chaiml-rm-ava-3-models-45935-v6-profiler is running
Tearing down inference service chaiml-rm-ava-3-models-45935-v6-profiler
Service chaiml-rm-ava-3-models-45935-v6-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 0.74s
Shutdown handler de-registered