nousresearch-meta-llama_4939

submission_id: nousresearch-meta-llama_4939_v12
developer_uid: end_to_end_test
best_of: 4
display_name: nousresearch-meta-llama_4939_v12
family_friendly_score: 0.0
formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}
generation_params: {'temperature': 1.0, 'top_p': 0.99, 'min_p': 0.1, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 512, 'best_of': 4, 'max_output_tokens': 64}
gpu_counts: {'NVIDIA RTX A5000': 1}
ineligible_reason: model is only for e2e test
is_internal_developer: True
language_model: NousResearch/Meta-Llama-3.1-8B-Instruct
latencies: [{'batch_size': 1, 'throughput': 1.0341601589980216, 'latency_mean': 0.9669103038311004, 'latency_p50': 0.9606138467788696, 'latency_p90': 1.0909669637680053}, {'batch_size': 5, 'throughput': 3.078468274461607, 'latency_mean': 1.6153664338588714, 'latency_p50': 1.611327052116394, 'latency_p90': 1.78397376537323}, {'batch_size': 10, 'throughput': 4.131904881498681, 'latency_mean': 2.3861480212211608, 'latency_p50': 2.399808883666992, 'latency_p90': 2.732097792625427}, {'batch_size': 15, 'throughput': 4.463048448375489, 'latency_mean': 3.3115236258506773, 'latency_p50': 3.3204104900360107, 'latency_p90': 3.7515533924102784}, {'batch_size': 20, 'throughput': 4.596678907862003, 'latency_mean': 4.282260652780533, 'latency_p50': 4.357721447944641, 'latency_p90': 4.974072980880737}, {'batch_size': 25, 'throughput': 4.660731428332805, 'latency_mean': 5.255385618209839, 'latency_p50': 5.295829772949219, 'latency_p90': 5.97768394947052}]
max_input_tokens: 512
max_output_tokens: 64
model_architecture: LlamaForCausalLM
model_group: NousResearch/Meta-Llama-
model_name: nousresearch-meta-llama_4939_v12
model_num_parameters: 8030261248.0
model_repo: NousResearch/Meta-Llama-3.1-8B-Instruct
model_size: 8B
num_battles: 297
num_wins: 127
ranking_group: single
status: torndown
submission_type: basic
throughput_3p7s: 4.59
timestamp: 2024-08-29T05:21:08+00:00
us_pacific_date: 2024-08-28
win_ratio: 0.4276094276094276
Download Preference Data
Resubmit model
Deleting key nousresearch-meta-llama-4939-v11/flywheel_model.0.safetensors from bucket guanaco-mkml-models
pipeline stage MKMLizer: starting
pipeline stage MKMLizer: trying
Starting job with name nousresearch-meta-llama-4939-v12-mkmlizer
Deleting key nousresearch-meta-llama-4939-v11/special_tokens_map.json from bucket guanaco-mkml-models
Deleting key nousresearch-meta-llama-4939-v11/tokenizer.json from bucket guanaco-mkml-models
Waiting for job on nousresearch-meta-llama-4939-v12-mkmlizer to finish
Deleting key nousresearch-meta-llama-4939-v11/tokenizer_config.json from bucket guanaco-mkml-models
pipeline stage MKMLModelDeleter: completed in 7.84s
nousresearch-meta-llama_4939_v11 status is now torndown due to DeploymentManager action
nousresearch-meta-llama-4939-v12-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
nousresearch-meta-llama-4939-v12-mkmlizer: ║     _____            __           __                                ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║       /___/                                                         ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║  Version: 0.10.1                                                    ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║  https://mk1.ai                                                     ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║  The license key for the current software has been verified as      ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║  belonging to:                                                      ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║  Chai Research Corp.                                                ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║  Expiration: 2024-10-15 23:59:59                                    ║
nousresearch-meta-llama-4939-v12-mkmlizer: ║                                                                     ║
nousresearch-meta-llama-4939-v12-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
nousresearch-meta-llama-4939-v12-mkmlizer: Downloaded to shared memory in 36.796s
nousresearch-meta-llama-4939-v12-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmphzz07zk8, device:0
nousresearch-meta-llama-4939-v12-mkmlizer: Saving flywheel model at /dev/shm/model_cache
nousresearch-meta-llama-4939-v12-mkmlizer: quantized model in 25.985s
nousresearch-meta-llama-4939-v12-mkmlizer: Processed model NousResearch/Meta-Llama-3.1-8B-Instruct in 62.782s
nousresearch-meta-llama-4939-v12-mkmlizer: creating bucket guanaco-mkml-models
nousresearch-meta-llama-4939-v12-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
nousresearch-meta-llama-4939-v12-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v12
nousresearch-meta-llama-4939-v12-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v12/config.json
nousresearch-meta-llama-4939-v12-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v12/special_tokens_map.json
nousresearch-meta-llama-4939-v12-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v12/tokenizer_config.json
nousresearch-meta-llama-4939-v12-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v12/tokenizer.json
nousresearch-meta-llama-4939-v12-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/nousresearch-meta-llama-4939-v12/flywheel_model.0.safetensors
nousresearch-meta-llama-4939-v12-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:08, 35.49it/s]
Loading 0:   4%|▍         | 13/291 [00:00<00:05, 55.01it/s]
Loading 0:   7%|▋         | 19/291 [00:00<00:05, 49.60it/s]
Loading 0:   9%|▊         | 25/291 [00:00<00:05, 51.82it/s]
Loading 0:  11%|█         | 32/291 [00:00<00:05, 47.31it/s]
Loading 0:  14%|█▍        | 41/291 [00:00<00:05, 49.51it/s]
Loading 0:  17%|█▋        | 50/291 [00:01<00:04, 51.01it/s]
Loading 0:  20%|█▉        | 58/291 [00:01<00:04, 57.65it/s]
Loading 0:  22%|██▏       | 65/291 [00:01<00:04, 53.41it/s]
Loading 0:  24%|██▍       | 71/291 [00:01<00:04, 52.61it/s]
Loading 0:  26%|██▋       | 77/291 [00:01<00:04, 45.91it/s]
Loading 0:  29%|██▊       | 83/291 [00:01<00:05, 37.67it/s]
Loading 0:  30%|███       | 88/291 [00:01<00:05, 38.55it/s]
Loading 0:  32%|███▏      | 94/291 [00:02<00:04, 43.08it/s]
Loading 0:  34%|███▍      | 100/291 [00:02<00:04, 43.23it/s]
Loading 0:  36%|███▌      | 105/291 [00:02<00:04, 43.72it/s]
Loading 0:  39%|███▉      | 113/291 [00:02<00:03, 45.01it/s]
Loading 0:  42%|████▏     | 121/291 [00:02<00:03, 51.45it/s]
Loading 0:  44%|████▎     | 127/291 [00:02<00:03, 49.60it/s]
Loading 0:  46%|████▌     | 133/291 [00:02<00:03, 50.22it/s]
Loading 0:  48%|████▊     | 139/291 [00:02<00:02, 52.30it/s]
Loading 0:  50%|████▉     | 145/291 [00:03<00:03, 47.05it/s]
Loading 0:  52%|█████▏    | 150/291 [00:03<00:03, 46.16it/s]
Loading 0:  54%|█████▍    | 157/291 [00:03<00:02, 51.09it/s]
Loading 0:  56%|█████▌    | 163/291 [00:03<00:02, 48.84it/s]
Loading 0:  58%|█████▊    | 168/291 [00:03<00:02, 46.50it/s]
Loading 0:  61%|██████    | 177/291 [00:03<00:02, 50.76it/s]
Loading 0:  63%|██████▎   | 183/291 [00:03<00:02, 50.76it/s]
Loading 0:  65%|██████▍   | 189/291 [00:04<00:03, 33.72it/s]
Loading 0:  67%|██████▋   | 194/291 [00:04<00:02, 35.76it/s]
Loading 0:  69%|██████▉   | 202/291 [00:04<00:02, 44.20it/s]
Loading 0:  71%|███████▏  | 208/291 [00:04<00:01, 44.03it/s]
Loading 0:  73%|███████▎  | 213/291 [00:04<00:01, 44.63it/s]
Loading 0:  76%|███████▌  | 220/291 [00:04<00:01, 49.69it/s]
Loading 0:  78%|███████▊  | 226/291 [00:04<00:01, 47.35it/s]
Loading 0:  79%|███████▉  | 231/291 [00:04<00:01, 45.42it/s]
Loading 0:  82%|████████▏ | 238/291 [00:05<00:01, 50.71it/s]
Loading 0:  84%|████████▍ | 244/291 [00:05<00:00, 47.76it/s]
Loading 0:  86%|████████▌ | 249/291 [00:05<00:00, 44.99it/s]
Loading 0:  88%|████████▊ | 255/291 [00:05<00:00, 48.49it/s]
Loading 0:  89%|████████▉ | 260/291 [00:05<00:00, 48.35it/s]
Loading 0:  91%|█████████ | 265/291 [00:05<00:00, 48.78it/s]
Loading 0:  93%|█████████▎| 271/291 [00:05<00:00, 46.18it/s]
Loading 0:  95%|█████████▍| 276/291 [00:05<00:00, 44.70it/s]
Loading 0:  97%|█████████▋| 282/291 [00:06<00:00, 42.14it/s]
Loading 0:  99%|█████████▊| 287/291 [00:11<00:01,  3.16it/s]
                                                            
Job nousresearch-meta-llama-4939-v12-mkmlizer completed after 87.92s with status: succeeded
Stopping job with name nousresearch-meta-llama-4939-v12-mkmlizer
pipeline stage MKMLizer: completed in 89.53s
pipeline stage MKMLKubeTemplater: starting
pipeline stage MKMLKubeTemplater: trying
pipeline stage MKMLKubeTemplater: completed in 0.36s
pipeline stage ISVCDeployer: starting
pipeline stage ISVCDeployer: trying
Creating inference service nousresearch-meta-llama-4939-v12
Waiting for inference service nousresearch-meta-llama-4939-v12 to be ready
Inference service nousresearch-meta-llama-4939-v12 ready after 183.20175790786743s
pipeline stage ISVCDeployer: completed in 184.50s
pipeline stage StressChecker: starting
pipeline stage StressChecker: trying
Received healthy response to inference request in 8.030771017074585s
Received healthy response to inference request in 2.4622409343719482s
Received healthy response to inference request in 2.593817949295044s
Received healthy response to inference request in 2.8911056518554688s
Received healthy response to inference request in 2.8946540355682373s
5 requests
0 failed requests
5th percentile: 2.4885563373565676
10th percentile: 2.5148717403411864
20th percentile: 2.5675025463104246
30th percentile: 2.653275489807129
40th percentile: 2.772190570831299
50th percentile: 2.8911056518554688
60th percentile: 2.892525005340576
70th percentile: 2.8939443588256837
80th percentile: 3.9218774318695075
90th percentile: 5.976324224472046
95th percentile: 7.003547620773315
99th percentile: 7.825326337814331
mean time: 3.7745179176330566
pipeline stage StressChecker: completed in 21.07s
nousresearch-meta-llama_4939_v12 status is now deployed due to DeploymentManager action
nousresearch-meta-llama_4939_v12 status is now inactive due to admin request
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.40s
Running pipeline stage MKMLProfilerDeployer
Creating inference service nousresearch-meta-llama-4939-v12-profiler
Waiting for inference service nousresearch-meta-llama-4939-v12-profiler to be ready
Inference service nousresearch-meta-llama-4939-v12-profiler ready after 181.7974979877472s
Pipeline stage MKMLProfilerDeployer completed in 182.79s
Running pipeline stage MKMLProfilerRunner
script pods %s
Pipeline stage MKMLProfilerRunner completed in 1.43s
Running pipeline stage MKMLProfilerDeleter
Checking if service nousresearch-meta-llama-4939-v12-profiler is running
Tearing down inference service nousresearch-meta-llama-4939-v12-profiler
Service nousresearch-meta-llama-4939-v12-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 3.80s
Running pipeline stage MKMLProfilerDeployer
Creating inference service nousresearch-meta-llama-4939-v12-profiler
Waiting for inference service nousresearch-meta-llama-4939-v12-profiler to be ready
Inference service nousresearch-meta-llama-4939-v12-profiler ready after 272.37903594970703s
Pipeline stage MKMLProfilerDeployer completed in 273.42s
pipeline run %s
pipeline skip stage %s
pipeline skip stage %s
pipeline run stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
Pipeline stage MKMLProfilerRunner completed in 1.05s
pipeline skip stage %s
run pipeline %s
skip pipeline stage %s
skip pipeline stage %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
Pipeline stage MKMLProfilerRunner completed in 0.89s
skip pipeline stage %s
run pipeline %s
skip pipeline stage %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service nousresearch-meta-llama-4939-v12-profiler
Ignoring service nousresearch-meta-llama-4939-v12-profiler already deployed
Waiting for inference service nousresearch-meta-llama-4939-v12-profiler to be ready
Inference service nousresearch-meta-llama-4939-v12-profiler ready after 10.216428995132446s
Pipeline stage MKMLProfilerDeployer completed in 11.27s
skip pipeline stage %s
skip pipeline stage %s
run pipeline %s
skip pipeline stage %s
skip pipeline stage %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
Pipeline stage MKMLProfilerRunner completed in 0.91s
skip pipeline stage %s
run pipeline %s
skip pipeline stage %s
skip pipeline stage %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
Pipeline stage MKMLProfilerRunner completed in 0.99s
skip pipeline stage %s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
Pipeline stage MKMLProfilerRunner completed in 1.00s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
Pipeline stage MKMLProfilerRunner completed in 2.89s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
Pipeline stage MKMLProfilerRunner completed in 1.18s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'chmod +x /code/profiler/batch_profiler.py && /code/profiler/batch_profiler.py --best_of_n 4 --input_tokens 512 --output_tokens 64'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'chmod +x /code/profiler/batch_profiler.py && /code/profiler/batch_profiler.py --best_of_n 4 --input_tokens 512 --output_tokens 64'
%s, retrying in %s seconds...
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'chmod +x /code/profiler/batch_profiler.py && python /code/profiler/batch_profiler.py batches --best_of_n 4 --input_tokens 512 --output_tokens 64'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'chmod +x /code/profiler/batch_profiler.py && python /code/profiler/batch_profiler.py batches --best_of_n 4 --input_tokens 512 --output_tokens 64'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
Retrying (%r) after connection broken by '%r': %s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
Pipeline stage MKMLProfilerRunner completed in 1127.51s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --batches "1,10,20" --samples 10 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
Pipeline stage MKMLProfilerRunner completed in 21.70s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --batches "1,10,20" --samples 10 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
Pipeline stage MKMLProfilerRunner completed in 25.23s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --batches "1,10,20" --samples 10 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
Pipeline stage MKMLProfilerRunner completed in 21.92s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --batches "1,10,20" --samples 10 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --batches "1,10,20" --samples 10 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --batches "1,10,20" --samples 10 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
Pipeline stage MKMLProfilerRunner completed in 26.46s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --batches "1,10,20" --samples 10 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
Pipeline stage MKMLProfilerRunner completed in 22.03s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --batches "1,10,20" --samples 10 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
Pipeline stage MKMLProfilerRunner completed in 22.37s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --batches "1,10,20" --samples 10 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --batches "1,10,20" --samples 10 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --batches "1,10,20" --samples 10 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
Pipeline stage MKMLProfilerRunner completed in 22.07s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_services/src/guanaco_model_service/profiler tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x batch_profiler.py && python batch_profiler.py profile --best_of_n 4 --input_tokens 512 --output_tokens 64 --output /code/profiler/results.json'
Pipeline stage MKMLProfilerRunner completed in 2721.90s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/profiler/summary.json'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/profiler/summary.json'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/profiler/summary.json'
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/profiler/summary.json'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/profiler/summary.json'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/profiler/summary.json'
%s, retrying in %s seconds...
script pods %s
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/profiler/summary.json'
%s, retrying in %s seconds...
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/profiler/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/profiler/summary.json'
Pipeline stage MKMLProfilerRunner completed in 99.50s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
script pods %s
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/profiler/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/profiler/summary.json'
Pipeline stage MKMLProfilerRunner completed in 99.68s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler/summary.json'
Pipeline stage MKMLProfilerRunner completed in 100.95s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler/summary.json'
Pipeline stage MKMLProfilerRunner completed in 130.48s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler/summary.json'
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 1 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 1 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 1 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler/summary.json'
Pipeline stage MKMLProfilerRunner completed in 27.78s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 1 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 1 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 1 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler/summary.json'
Pipeline stage MKMLProfilerRunner completed in 27.20s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler/summary.json'
%s, retrying in %s seconds...
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler_1725154505
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler_1725154505 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725154505/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler_1725154505/summary.json'
Pipeline stage MKMLProfilerRunner completed in 100.41s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler_1725154741
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler_1725154741 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725154741/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler_1725154741/summary.json'
Pipeline stage MKMLProfilerRunner completed in 682.62s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler_1725156375
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler_1725156375 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 1,6,11,16,21,26,31,36,41,46,51,56,61,66,71,76,81,86,91,96 --samples 50 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725156375/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler_1725156375/summary.json'
Pipeline stage MKMLProfilerRunner completed in 119.22s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler_1725156572
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler_1725156572 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 1,6,11,16,21,26,31,36,41,46,51,56,61,66,71,76,81,86,91,96 --samples 50 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725156572/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler_1725156572/summary.json'
Pipeline stage MKMLProfilerRunner completed in 119.41s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler_1725156736
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler_1725156736 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 1,6,11,16,21,26,31,36,41,46,51,56,61,66,71,76,81,86,91,96 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725156736/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler_1725156736/summary.json'
Pipeline stage MKMLProfilerRunner completed in 447.47s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler_1725158565
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler_1725158565 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 1,6,11,16,21,26,31,36,41,46,51,56,61,66,71,76,81,86,91,96 --samples 50 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725158565/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler_1725158565/summary.json'
Pipeline stage MKMLProfilerRunner completed in 117.41s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler_1725158762
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler_1725158762 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 1,6,11,16,21,26,31,36,41,46,51,56,61,66,71,76,81,86,91,96 --samples 100 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725158762/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler_1725158762/summary.json'
Pipeline stage MKMLProfilerRunner completed in 230.58s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler_1725159211
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler_1725159211 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 1,6,11,16,21,26,31,36,41,46,51,56,61,66,71,76,81,86,91,96 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725159211/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- bash -c 'cat /code/chaiverse_profiler_1725159211/summary.json'
Pipeline stage MKMLProfilerRunner completed in 441.52s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns:/code/chaiverse_profiler_1725251146
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohn4ns -- sh -c 'cd /code/chaiverse_profiler_1725251146 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725251146/summary.json'
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage %s skipped, reason=%s
Pipeline stage MKMLProfilerTemplater completed in 0.17s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service nousresearch-meta-llama-4939-v12-profiler
Waiting for inference service nousresearch-meta-llama-4939-v12-profiler to be ready
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage %s skipped, reason=%s
Pipeline stage MKMLProfilerTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service nousresearch-meta-llama-4939-v12-profiler
Waiting for inference service nousresearch-meta-llama-4939-v12-profiler to be ready
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage %s skipped, reason=%s
Pipeline stage MKMLProfilerTemplater completed in 0.35s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service nousresearch-meta-llama-4939-v12-profiler
Ignoring service nousresearch-meta-llama-4939-v12-profiler already deployed
Waiting for inference service nousresearch-meta-llama-4939-v12-profiler to be ready
Inference service nousresearch-meta-llama-4939-v12-profiler ready after 376.0439910888672s
Pipeline stage MKMLProfilerDeployer completed in 377.66s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplodjwbc:/code/chaiverse_profiler_1725257542
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplodjwbc -- sh -c 'cd /code/chaiverse_profiler_1725257542 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725257542/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplodjwbc -- bash -c 'cat /code/chaiverse_profiler_1725257542/summary.json'
Pipeline stage MKMLProfilerRunner completed in 450.26s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service nousresearch-meta-llama-4939-v12-profiler is running
Tearing down inference service nousresearch-meta-llama-4939-v12-profiler
Service nousresearch-meta-llama-4939-v12-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 3.84s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage %s skipped, reason=%s
Pipeline stage MKMLProfilerTemplater completed in 0.30s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service nousresearch-meta-llama-4939-v12-profiler
Ignoring service nousresearch-meta-llama-4939-v12-profiler already deployed
Waiting for inference service nousresearch-meta-llama-4939-v12-profiler to be ready
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service nousresearch-meta-llama-4939-v12-profiler is running
Tearing down inference service nousresearch-meta-llama-4939-v12-profiler
Service nousresearch-meta-llama-4939-v12-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 3.22s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.55s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service nousresearch-meta-llama-4939-v12-profiler
Waiting for inference service nousresearch-meta-llama-4939-v12-profiler to be ready
Inference service nousresearch-meta-llama-4939-v12-profiler ready after 141.97586107254028s
Pipeline stage MKMLProfilerDeployer completed in 144.00s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohhcqc:/code/chaiverse_profiler_1725325014 --namespace tenant-chaiml-guanaco
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohhcqc --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1725325014 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725325014/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohhcqc:/code/chaiverse_profiler_1725325465 --namespace tenant-chaiml-guanaco
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohhcqc --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1725325465 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725325465/summary.json'
%s, retrying in %s seconds...
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplohhcqc:/code/chaiverse_profiler_1725325489 --namespace tenant-chaiml-guanaco
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service nousresearch-meta-llama-4939-v12-profiler is running
Tearing down inference service nousresearch-meta-llama-4939-v12-profiler
Service nousresearch-meta-llama-4939-v12-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 3.79s
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage %s skipped, reason=%s
Pipeline stage MKMLProfilerTemplater completed in 0.35s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service nousresearch-meta-llama-4939-v12-profiler
Waiting for inference service nousresearch-meta-llama-4939-v12-profiler to be ready
Inference service nousresearch-meta-llama-4939-v12-profiler ready after 142.23034405708313s
Pipeline stage MKMLProfilerDeployer completed in 143.78s
run pipeline stage %s
Running pipeline stage MKMLProfilerRunner
kubectl cp /code/guanaco/guanaco_inference_services/src/inference_scripts tenant-chaiml-guanaco/nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplo87k5r:/code/chaiverse_profiler_1725326217 --namespace tenant-chaiml-guanaco
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplo87k5r --namespace tenant-chaiml-guanaco -- sh -c 'cd /code/chaiverse_profiler_1725326217 && chmod +x profiles.py && python profiles.py profile --best_of_n 4 --batches 1,5,10,15,20,25,30,35,40,45,50,55,60,65,70,75,80,85,90,95,100,105,110,115,120,125,130,135,140,145,150,155,160,165,170,175,180,185,190,195 --samples 200 --input_tokens 512 --output_tokens 64 --summary /code/chaiverse_profiler_1725326217/summary.json'
kubectl exec -it nousresearch-meta-lld112c0e9bd507f6fa054f830d68e217f-deplo87k5r --namespace tenant-chaiml-guanaco -- bash -c 'cat /code/chaiverse_profiler_1725326217/summary.json'
Pipeline stage MKMLProfilerRunner completed in 445.80s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Checking if service nousresearch-meta-llama-4939-v12-profiler is running
Tearing down inference service nousresearch-meta-llama-4939-v12-profiler
Service nousresearch-meta-llama-4939-v12-profiler has been torndown
Pipeline stage MKMLProfilerDeleter completed in 3.34s
nousresearch-meta-llama_4939_v12 status is now torndown due to DeploymentManager action
nousresearch-meta-llama_4939_v12 status is now torndown due to DeploymentManager action