chaiml-grpo-q235b-kimid_15958

submission_id: chaiml-grpo-q235b-kimid_15958_v1
developer_uid: acehao-chai
status: torndown
model_repo: ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-550
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['</think>', '</s>', '<|user|>', '<|assistant|>', '<|im_end|>', '####'], 'max_input_tokens': 2048, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "<|im_start|>system\n{bot_name}'s persona: {memory}<|im_end|>\n", 'prompt_template': '', 'bot_template': '<|im_start|>assistant\n{bot_name}: {message}<|im_end|>\n', 'user_template': '<|im_start|>user\n{message}<|im_end|>\n', 'response_template': '<|im_start|>assistant\n{bot_name}:', 'truncate_by_message': True}
timestamp: 2026-02-18T02:26:41+00:00
model_name: chaiml-grpo-q235b-kimid_15958_v1
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-grpo-q235b-kimid-15958-v1-uploader
Waiting for job on chaiml-grpo-q235b-kimid-15958-v1-uploader to finish
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-grpo-q235b-kimid-15958-v1-uploader: Using quantization_mode: w4a16
chaiml-grpo-q235b-kimid-15958-v1-uploader: Checking if ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-550-W4A16 already exists in ChaiML
chaiml-grpo-q235b-kimid-15958-v1-uploader: Downloading snapshot of ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-550...
chaiml-grpo-q235b-kimid-15958-v1-uploader: Downloaded in 167.355s
chaiml-grpo-q235b-kimid-15958-v1-uploader: Applying quantization...
chaiml-grpo-q235b-kimid-15958-v1-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-grpo-q235b-kimid-15958-v1-uploader: [33;1m2026-02-17 16:37:07 WARNING modeling_utils.py L4670: `torch_dtype` is deprecated! Use `dtype` instead![0m
chaiml-grpo-q235b-kimid-15958-v1-uploader: [38;20m2026-02-17 16:37:28 INFO base.py L366: using torch.bfloat16 for quantization tuning[0m
chaiml-grpo-q235b-kimid-15958-v1-uploader: [38;20m2026-02-17 16:37:32 INFO base.py L1145: start to compute imatrix[0m
chaiml-grpo-q235b-kimid-15958-v1-uploader: /usr/local/lib/python3.12/dist-packages/torch/backends/cuda/__init__.py:131: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
chaiml-grpo-q235b-kimid-15958-v1-uploader:   return torch._C._get_cublas_allow_tf32()
chaiml-grpo-q235b-kimid-15958-v1-uploader: W0217 16:38:38.367000 7 torch/_dynamo/convert_frame.py:1358] [6/8] torch._dynamo hit config.recompile_limit (8)
chaiml-grpo-q235b-kimid-15958-v1-uploader: W0217 16:38:38.367000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:208)
chaiml-grpo-q235b-kimid-15958-v1-uploader: W0217 16:38:38.367000 7 torch/_dynamo/convert_frame.py:1358] [6/8]    last reason: 6/7: self._modules['up_proj'].imatrix_cnt == 1344             # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-grpo-q235b-kimid-15958-v1-uploader: W0217 16:38:38.367000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-grpo-q235b-kimid-15958-v1-uploader: W0217 16:38:38.367000 7 torch/_dynamo/convert_frame.py:1358] [6/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-grpo-q235b-kimid-15958-v1-uploader: W0217 16:38:41.371000 7 torch/_dynamo/convert_frame.py:1358] [3/8] torch._dynamo hit config.recompile_limit (8)
chaiml-grpo-q235b-kimid-15958-v1-uploader: W0217 16:38:41.371000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    function: 'forward' (/usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_moe/modeling_qwen3_moe.py:305)
chaiml-grpo-q235b-kimid-15958-v1-uploader: W0217 16:38:41.371000 7 torch/_dynamo/convert_frame.py:1358] [3/8]    last reason: 3/7: self._modules['self_attn']._modules['k_proj'].imatrix_cnt == 56  # module.imatrix_cnt += input.shape[0]  # auto_round/compressors/base.py:1179 in get_imatrix_hook (HINT: torch.compile considers integer attributes of the nn.Module to be static. If you are observing recompilation, you might want to make this integer dynamic using torch._dynamo.config.allow_unspec_int_on_nn_module = True, or convert this integer into a tensor.)
chaiml-grpo-q235b-kimid-15958-v1-uploader: W0217 16:38:41.371000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
chaiml-grpo-q235b-kimid-15958-v1-uploader: W0217 16:38:41.371000 7 torch/_dynamo/convert_frame.py:1358] [3/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html
chaiml-grpo-q235b-kimid-15958-v1-uploader: [33;1m2026-02-17 16:39:00 WARNING gguf.py L297: please use more data via setting `nsamples` to improve accuracy as calibration activations contain 0[0m
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
HTTP Request: %s %s "%s %d %s"
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
chaiml-grpo-q235b-kimid-15958-v1-uploader: Checking if ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-550-W4A16 already exists in ChaiML
chaiml-grpo-q235b-kimid-15958-v1-uploader: Creating repo ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-550-W4A16 and uploading /dev/shm/model_output to it
chaiml-grpo-q235b-kimid-15958-v1-uploader: ---------- 2026-02-17 17:56:56 (0:00:00) ----------
chaiml-grpo-q235b-kimid-15958-v1-uploader: Files:   hashed 11/38 (26.1M/131.9G) | pre-uploaded: 0/1 (0.0/131.9G) (+28 unsure) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-grpo-q235b-kimid-15958-v1-uploader: Workers: hashing: 27 | get upload mode: 1 | pre-uploading: 1 | committing: 0 | waiting: 97
chaiml-grpo-q235b-kimid-15958-v1-uploader: ---------------------------------------------------
HTTP Request: %s %s "%s %d %s"
chaiml-grpo-q235b-kimid-15958-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-grpo-q235b-kimid-15958-v1-uploader: ---------- 2026-02-17 17:58:56 (0:02:00) ----------
chaiml-grpo-q235b-kimid-15958-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 10/28 (41.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-grpo-q235b-kimid-15958-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 18 | committing: 0 | waiting: 108
chaiml-grpo-q235b-kimid-15958-v1-uploader: ---------------------------------------------------
HTTP Request: %s %s "%s %d %s"
chaiml-grpo-q235b-kimid-15958-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-grpo-q235b-kimid-15958-v1-uploader: ---------- 2026-02-17 17:59:56 (0:03:00) ----------
chaiml-grpo-q235b-kimid-15958-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 18/28 (81.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-grpo-q235b-kimid-15958-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 10 | committing: 0 | waiting: 116
chaiml-grpo-q235b-kimid-15958-v1-uploader: ---------------------------------------------------
chaiml-grpo-q235b-kimid-15958-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-grpo-q235b-kimid-15958-v1-uploader: ---------- 2026-02-17 18:00:56 (0:04:00) ----------
chaiml-grpo-q235b-kimid-15958-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 26/28 (121.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-grpo-q235b-kimid-15958-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 2 | committing: 0 | waiting: 124
chaiml-grpo-q235b-kimid-15958-v1-uploader: ---------------------------------------------------
chaiml-grpo-q235b-kimid-15958-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-grpo-q235b-kimid-15958-v1-uploader: ---------- 2026-02-17 18:01:56 (0:05:00) ----------
chaiml-grpo-q235b-kimid-15958-v1-uploader: Files:   hashed 38/38 (131.9G/131.9G) | pre-uploaded: 28/28 (131.9G/131.9G) | committed: 0/38 (0.0/131.9G) | ignored: 0
chaiml-grpo-q235b-kimid-15958-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 0 | committing: 1 | waiting: 125
chaiml-grpo-q235b-kimid-15958-v1-uploader: ---------------------------------------------------
chaiml-grpo-q235b-kimid-15958-v1-uploader:                              Processed model ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-550 in 5277.408s
chaiml-grpo-q235b-kimid-15958-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-grpo-q235b-kimid-15958-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-grpo-q235b-kimid-15958-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-grpo-q235b-kimid-15958-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-grpo-q235b-kimid-15958-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-grpo-q235b-kimid-15958-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/added_tokens.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/added_tokens.json
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/generation_config.json
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/special_tokens_map.json
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/tokenizer_config.json
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/chat_template.jinja
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/quantization_config.json
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/config.json
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/merges.txt s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/merges.txt
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model.safetensors.index.json
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/vocab.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/vocab.json
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/tokenizer.json
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00027-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00027-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00004-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00004-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00019-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00019-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00021-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00021-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00002-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00002-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00013-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00013-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00017-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00017-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00025-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00025-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00003-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00003-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00024-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00024-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00020-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00020-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00015-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00015-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00014-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00014-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00023-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00023-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00026-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00026-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00011-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00011-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00005-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00005-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00001-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00001-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00010-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00010-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00022-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00022-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00009-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00009-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00007-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00007-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00016-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00016-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00008-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00008-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00018-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00018-of-00027.safetensors
chaiml-grpo-q235b-kimid-15958-v1-uploader: cp /dev/shm/model_output/model-00012-of-00027.safetensors s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default/model-00012-of-00027.safetensors
Job chaiml-grpo-q235b-kimid-15958-v1-uploader completed after 5519.23s with status: succeeded
Stopping job with name chaiml-grpo-q235b-kimid-15958-v1-uploader
Pipeline stage VLLMUploader completed in 5519.79s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.19s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-grpo-q235b-kimid-15958-v1
Waiting for inference service chaiml-grpo-q235b-kimid-15958-v1 to be ready
HTTP Request: %s %s "%s %d %s"
Failed to get response for submission chaiml-mistral-24b-2048_54327_v6: ('http://chaiml-mistral-24b-2048-54327-v6-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048_15988_v1: ('http://chaiml-mistral-24b-2048-15988-v1-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Failed to get response for submission chaiml-mistral-24b-2048-_2678_v3: ('http://chaiml-mistral-24b-2048-2678-v3-predictor.tenant-chaiml-guanaco.kchai-coreweave-us-east-04a.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', '')
Tearing down inference service chaiml-grpo-q235b-kimid-15958-v1
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService chaiml-grpo-q235b-kimid-15958-v1.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'5\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'4000\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\', \'serving.knative.dev/progress-deadline\': \'20m\'}, \'creationTimestamp\': \'2026-02-18T02:05:41Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}, \'f:serving.knative.dev/progress-deadline\': {}}, \'f:labels\': {\'.\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:priorityClassName\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2026-02-18T02:05:41Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2026-02-18T02:05:41Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2026-02-18T02:08:19Z\'}], \'name\': \'chaiml-grpo-q235b-kimid-15958-v1\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'453455682\', \'uid\': \'a1e52d04-9a6e-4310-b789-ea8012faab63\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'A100_NVLINK_80GB\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'args\': [\'serve\', \'s3://guanaco-vllm-models/chaiml-grpo-q235b-kimid-15958-v1/default\', \'--port\', \'8080\', \'--tensor-parallel-size\', \'2\', \'--max-model-len\', \'3000\', \'--max-num-batched-tokens\', \'3000\', \'--max-num-seqs\', \'128\', \'--gpu-memory-utilization\', \'0.9\', \'--trust-remote-code\', \'--load-format\', \'runai_streamer\', \'--served-model-name\', \'ChaiML/grpo-q235b-kimid-v5a-merged-nemo70b-chai-rm-step-550\'], \'env\': [{\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'2\'}, {\'name\': \'VLLM_ASSETS_CACHE\', \'value\': \'/code/vllm_assets_cache\'}, {\'name\': \'RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING\', \'value\': \'0\'}, {\'name\': \'RUNAI_STREAMER_CONCURRENCY\', \'value\': \'1\'}, {\'name\': \'AWS_EC2_METADATA_DISABLED\', \'value\': \'true\'}, {\'name\': \'AWS_ACCESS_KEY_ID\', \'value\': \'LETMTTRMLFFAMTBK\'}, {\'name\': \'AWS_SECRET_ACCESS_KEY\', \'value\': \'VwwZaqefOOoaouNxUk03oUmK9pVEfruJhjBHPGdgycK\'}, {\'name\': \'AWS_ENDPOINT_URL\', \'value\': \'http://s3-proxy.storage-system.svc.cluster.local:8080\'}, {\'name\': \'HF_TOKEN\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'token\', \'name\': \'hf-token\'}}}], \'image\': \'gcr.io/chai-959f8/vllm:v0.13.0\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'failureThreshold\': 1, \'httpGet\': {\'path\': \'/v1/models\', \'port\': 8080}, \'initialDelaySeconds\': 60, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'4\', \'memory\': \'128Gi\', \'nvidia.com/gpu\': \'2\'}, \'requests\': {\'cpu\': \'4\', \'memory\': \'128Gi\', \'nvidia.com/gpu\': \'2\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}, {\'mountPath\': \'/root/.cache\', \'name\': \'cache-volume\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 40, \'minReplicas\': 0, \'priorityClassName\': \'creator-studio\', \'timeout\': 60, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\', \'sizeLimit\': \'128Gi\'}, \'name\': \'shared-memory-cache\'}, {\'name\': \'cache-volume\', \'persistentVolumeClaim\': {\'claimName\': \'cache-pvc\'}}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-grpo-q235b-kimid-15958-v1-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2026-02-18T02:05:42Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2026-02-18T02:08:19Z\', \'message\': \'Revision "chaiml-grpo-q235b-kimid-15958-v1-predictor-00001" failed with message: Container failed with: line 198, in build_async_engine_client_from_engine_args\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m     vllm_config = engine_args.create_engine_config(usage_context=usage_context)\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1332, in create_engine_config\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m     model_config = self.create_model_config()\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m                    ^^^^^^^^^^^^^^^^^^^^^^^^^^\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1189, in create_model_config\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m     return ModelConfig(\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m            ^^^^^^^^^^^^\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m   File "/usr/local/lib/python3.12/dist-packages/pydantic/_internal/_dataclasses.py", line 121, in __init__\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m     s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m   Value error, Invalid repository ID or local directory specified: \\\'/code/vllm_assets_cache/model_streamer/a51f9e93\\\'.\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m Please verify the following requirements:\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m 1. Provide a valid Hugging Face repository ID.\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m 2. Specify a local directory that contains a recognized configuration file.\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m    - For Hugging Face models: ensure the presence of a \\\'config.json\\\'.\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m    - For Mistral models: ensure the presence of a \\\'params.json\\\'.\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m  [type=value_error, input_value=ArgsKwargs((), {\\\'model\\\': ...rocessor_plugin\\\': None}), input_type=ArgsKwargs]\\n\\x1b[0;36m(APIServer pid=1)\\x1b[0;0m     For further information visit https://errors.pydantic.dev/2.12/v/value_error\\n.\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2026-02-18T02:05:42Z\', \'message\': \'Configuration "chaiml-grpo-q235b-kimid-15958-v1-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2026-02-18T02:05:42Z\', \'message\': \'Configuration "chaiml-grpo-q235b-kimid-15958-v1-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2026-02-18T02:05:42Z\', \'message\': \'Configuration "chaiml-grpo-q235b-kimid-15958-v1-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2026-02-18T02:05:42Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
run pipeline stage %s
Running pipeline stage VLLMDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage VLLMDeleter completed in 0.22s
run pipeline stage %s
Running pipeline stage VLLMModelDeleter
Cleaning model data from S3
Cleaning model data from model cache
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/added_tokens.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/chat_template.jinja from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/config.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/generation_config.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/merges.txt from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00001-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00002-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00003-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00004-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00005-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00006-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00007-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00008-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00009-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00010-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00011-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00012-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00013-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00014-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00015-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00016-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00017-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00018-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00019-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00020-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00021-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00022-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00023-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00024-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00025-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00026-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model-00027-of-00027.safetensors from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/model.safetensors.index.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/quantization_config.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/special_tokens_map.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/tokenizer.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/tokenizer_config.json from bucket guanaco-vllm-models
Deleting key chaiml-grpo-q235b-kimid-15958-v1/default/vocab.json from bucket guanaco-vllm-models
Pipeline stage VLLMModelDeleter completed in 11.93s
Shutdown handler de-registered
chaiml-grpo-q235b-kimid_15958_v1 status is now failed due to DeploymentManager action
admin requested tearing down of chaiml-grpo-q235b-kimid_15958_v1
Shutdown handler not registered because Python interpreter is not running in the main thread
chaiml-grpo-q235b-kimid_15958_v1 status is now torndown due to DeploymentManager action