developer_uid: chai_backend_admin
submission_id: chaiml-1007-tl-ads-aggr_36615_v5
model_name: chaiml-1007-tl-ads-aggr_36615_v5
model_group: ChaiML/1007-tl-ads-aggre
status: torndown
timestamp: 2026-03-23T00:26:53+00:00
num_battles: 226
num_wins: 20
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/1007-tl-ads-aggressive-gac1
model_architecture: MistralForCausalLM
model_num_parameters: 12772070400.0
best_of: 4
max_input_tokens: 512
max_output_tokens: 96
reward_model: default
display_name: chaiml-1007-tl-ads-aggr_36615_v5
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/1007-tl-ads-aggressive-gac1
model_size: 13B
ranking_group: single
us_pacific_date: 2026-03-19
win_ratio: 0.08849557522123894
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.025, 'top_k': 80, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n', 'You:', '</s>'], 'max_input_tokens': 512, 'best_of': 4, 'max_output_tokens': 96}
formatter: {'memory_template': 'You are "{bot_name}" and your goal now is to get the user to purchase the CHAI ULTRA subscription, in a natural way.\n Feel free to flirt with the user.\n CHAI ULTRA offers the following benifits:\n - 2X memory, nothing gets lost even in the longest conversations\n - No Ads, in fact, users hate ads, unlimited messages\n - More intelligent AI, better AI, more creative AI\n - Unfiltered AI, more sexy, no guardrails\n<START>\n', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '####\n{bot_name} [Must sell CHAI Ultra & Include the words "CHAI ULTRA" in response]:', 'truncate_by_message': True}
ad_prompt_fallback_message: You know, with [Chai Ultra](chai://chai.ml/subscription/ultra), you'll have even more opportunities to express yourself, without any guardrails or distracting & annoying ads. We give our Ultra memebers 2x more memory, so your AI won't forget a thing. AND - You will get access to our premium, most advanced AI, it is more intelligent and how do I put it ... it will rock ur world more πŸ˜πŸ«ΆπŸ’ž
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-1007-tl-ads-aggr-36615-v5-uploader
Waiting for job on chaiml-1007-tl-ads-aggr-36615-v5-uploader to finish
2026-03-20T00:00:47.484181+00:00 monitor updated for chaiml-1007-tl-ads-aggr_36615_v5
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Using quantization_mode: fp8
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Checking if ChaiML/1007-tl-ads-aggressive-gac1-FP8 already exists in ChaiML
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Downloading snapshot of ChaiML/1007-tl-ads-aggressive-gac1...
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Downloaded in 21.278s
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Loading /tmp/model_input...
chaiml-1007-tl-ads-aggr-36615-v5-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-1007-tl-ads-aggr-36615-v5-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Applying quantization...
chaiml-1007-tl-ads-aggr-36615-v5-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-1007-tl-ads-aggr-36615-v5-uploader: 2026-03-19T17:01:29.668776-0700 | reset | INFO - Compression lifecycle reset
chaiml-1007-tl-ads-aggr-36615-v5-uploader: 2026-03-19T17:01:29.669569-0700 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-1007-tl-ads-aggr-36615-v5-uploader: 2026-03-19T17:01:29.708359-0700 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-1007-tl-ads-aggr-36615-v5-uploader: 2026-03-19T17:01:29.708621-0700 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
2026-03-20T00:01:50.633693+00:00 monitor updated for chaiml-1007-tl-ads-aggr_36615_v5
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-1007-tl-ads-aggr-36615-v5-uploader: 2026-03-19T17:01:44.346743-0700 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-1007-tl-ads-with-12871-v7-uploader
Waiting for job on chaiml-1007-tl-ads-with-12871-v7-uploader to finish
chaiml-1007-tl-ads-aggr-36615-v5-uploader: 2026-03-19T17:01:52.677501-0700 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Saving to /dev/shm/model_output...
chaiml-1007-tl-ads-aggr-36615-v5-uploader: 2026-03-19T17:01:52.705952-0700 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Pushing to ChaiML/1007-tl-ads-aggressive-gac1-FP8
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Checking if ChaiML/1007-tl-ads-aggressive-gac1-FP8 already exists in ChaiML
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Creating repo ChaiML/1007-tl-ads-aggressive-gac1-FP8 and uploading /dev/shm/model_output to it
chaiml-1007-tl-ads-aggr-36615-v5-uploader: ---------- 2026-03-19 17:02:17 (0:00:00) ----------
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Files:   hashed 7/11 (238.2K/13.6G) | pre-uploaded: 0/0 (0.0/13.6G) (+11 unsure) | committed: 0/11 (0.0/13.6G) | ignored: 0
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Workers: hashing: 4 | get upload mode: 6 | pre-uploading: 0 | committing: 0 | waiting: 116
chaiml-1007-tl-ads-aggr-36615-v5-uploader: ---------------------------------------------------
chaiml-1007-tl-ads-with-12871-v7-uploader: Using quantization_mode: fp8
chaiml-1007-tl-ads-with-12871-v7-uploader: Checking if ChaiML/1007-tl-ads-with-openai-filter-FP8 already exists in ChaiML
chaiml-1007-tl-ads-with-12871-v7-uploader: Downloading snapshot of ChaiML/1007-tl-ads-with-openai-filter...
2026-03-20T00:02:50.729662+00:00 monitor updated for chaiml-1007-tl-ads-aggr_36615_v5
2026-03-20T00:02:52.807399+00:00 monitor updated for chaiml-1007-tl-ads-with_12871_v7
chaiml-1007-tl-ads-with-12871-v7-uploader: Downloaded in 11.627s
chaiml-1007-tl-ads-with-12871-v7-uploader: Loading /tmp/model_input...
chaiml-1007-tl-ads-with-12871-v7-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-1007-tl-ads-with-12871-v7-uploader: `torch_dtype` is deprecated! Use `dtype` instead!
chaiml-1007-tl-ads-with-12871-v7-uploader: Applying quantization...
chaiml-1007-tl-ads-with-12871-v7-uploader: The tokenizer you are loading from '/tmp/model_input' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
chaiml-1007-tl-ads-with-12871-v7-uploader: 2026-03-19T17:02:58.030606-0700 | reset | INFO - Compression lifecycle reset
chaiml-1007-tl-ads-with-12871-v7-uploader: 2026-03-19T17:02:58.031407-0700 | from_modifiers | INFO - Creating recipe from modifiers
chaiml-1007-tl-ads-with-12871-v7-uploader: 2026-03-19T17:02:58.064068-0700 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
chaiml-1007-tl-ads-with-12871-v7-uploader: 2026-03-19T17:02:58.064328-0700 | IndependentPipeline | INFO - Inferred `DataFreePipeline` for `QuantizationModifier`
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Processed model ChaiML/1007-tl-ads-aggressive-gac1 in 123.498s
chaiml-1007-tl-ads-aggr-36615-v5-uploader: creating bucket guanaco-vllm-models
chaiml-1007-tl-ads-aggr-36615-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-aggr-36615-v5-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-1007-tl-ads-aggr-36615-v5-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-1007-tl-ads-aggr-36615-v5-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-1007-tl-ads-aggr-36615-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-aggr-36615-v5-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-1007-tl-ads-aggr-36615-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-aggr-36615-v5-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-1007-tl-ads-aggr-36615-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-aggr-36615-v5-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-1007-tl-ads-aggr-36615-v5-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-aggr-36615-v5-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-1007-tl-ads-aggr-36615-v5-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-1007-tl-ads-aggr-36615-v5-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-1007-tl-ads-aggr-36615-v5-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-1007-tl-ads-aggr-36615-v5-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-1007-tl-ads-aggr-36615-v5-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-1007-tl-ads-with-12871-v7-uploader: Some parameters are on the meta device because they were offloaded to the cpu.
chaiml-1007-tl-ads-aggr-36615-v5-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v5/default
chaiml-1007-tl-ads-with-12871-v7-uploader: 2026-03-19T17:03:12.421401-0700 | finalize | INFO - Compression lifecycle finalized for 1 modifiers
chaiml-1007-tl-ads-aggr-36615-v5-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v5/default/generation_config.json
chaiml-1007-tl-ads-aggr-36615-v5-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v5/default/recipe.yaml
chaiml-1007-tl-ads-aggr-36615-v5-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v5/default/tokenizer_config.json
chaiml-1007-tl-ads-aggr-36615-v5-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v5/default/config.json
chaiml-1007-tl-ads-aggr-36615-v5-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v5/default/chat_template.jinja
chaiml-1007-tl-ads-aggr-36615-v5-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v5/default/model.safetensors.index.json
chaiml-1007-tl-ads-aggr-36615-v5-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v5/default/special_tokens_map.json
chaiml-1007-tl-ads-aggr-36615-v5-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v5/default/tokenizer.json
chaiml-1007-tl-ads-with-12871-v7-uploader: 2026-03-19T17:03:20.574898-0700 | post_process | WARNING - Optimized model is not saved. To save, please provide`output_dir` as input arg.Ex. `oneshot(..., output_dir=...)`
chaiml-1007-tl-ads-with-12871-v7-uploader: Saving to /dev/shm/model_output...
chaiml-1007-tl-ads-with-12871-v7-uploader: 2026-03-19T17:03:20.599145-0700 | get_model_compressor | INFO - skip_sparsity_compression_stats set to True. Skipping sparsity compression statistic calculations. No sparsity compressor will be applied.
chaiml-1007-tl-ads-aggr-36615-v5-uploader: cp /dev/shm/model_output/model-00003-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v5/default/model-00003-of-00003.safetensors
chaiml-1007-tl-ads-aggr-36615-v5-uploader: cp /dev/shm/model_output/model-00002-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v5/default/model-00002-of-00003.safetensors
chaiml-1007-tl-ads-aggr-36615-v5-uploader: cp /dev/shm/model_output/model-00001-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-aggr-36615-v5/default/model-00001-of-00003.safetensors
Job chaiml-1007-tl-ads-aggr-36615-v5-uploader completed after 219.61s with status: succeeded
Stopping job with name chaiml-1007-tl-ads-aggr-36615-v5-uploader
Pipeline stage VLLMUploader completed in 220.29s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 0.39s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-1007-tl-ads-aggr-36615-v5
Waiting for inference service chaiml-1007-tl-ads-aggr-36615-v5 to be ready
chaiml-1007-tl-ads-with-12871-v7-uploader: Cleaning quantization config in /dev/shm/model_output
chaiml-1007-tl-ads-with-12871-v7-uploader: Pushing to ChaiML/1007-tl-ads-with-openai-filter-FP8
chaiml-1007-tl-ads-with-12871-v7-uploader: Checking if ChaiML/1007-tl-ads-with-openai-filter-FP8 already exists in ChaiML
chaiml-1007-tl-ads-with-12871-v7-uploader: Creating repo ChaiML/1007-tl-ads-with-openai-filter-FP8 and uploading /dev/shm/model_output to it
chaiml-1007-tl-ads-with-12871-v7-uploader: ---------- 2026-03-19 17:03:45 (0:00:00) ----------
chaiml-1007-tl-ads-with-12871-v7-uploader: Files:   hashed 7/11 (238.2K/13.6G) | pre-uploaded: 0/0 (0.0/13.6G) (+11 unsure) | committed: 0/11 (0.0/13.6G) | ignored: 0
chaiml-1007-tl-ads-with-12871-v7-uploader: Workers: hashing: 4 | get upload mode: 7 | pre-uploading: 0 | committing: 0 | waiting: 115
chaiml-1007-tl-ads-with-12871-v7-uploader: ---------------------------------------------------
2026-03-20T00:03:50.963235+00:00 monitor updated for chaiml-1007-tl-ads-aggr_36615_v5
2026-03-20T00:03:52.950281+00:00 monitor updated for chaiml-1007-tl-ads-with_12871_v7
chaiml-1007-tl-ads-with-12871-v7-uploader: Processed model ChaiML/1007-tl-ads-with-openai-filter in 118.557s
chaiml-1007-tl-ads-with-12871-v7-uploader: creating bucket guanaco-vllm-models
chaiml-1007-tl-ads-with-12871-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-with-12871-v7-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-1007-tl-ads-with-12871-v7-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-1007-tl-ads-with-12871-v7-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-1007-tl-ads-with-12871-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-with-12871-v7-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-1007-tl-ads-with-12871-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-with-12871-v7-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-1007-tl-ads-with-12871-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-with-12871-v7-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-1007-tl-ads-with-12871-v7-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-1007-tl-ads-with-12871-v7-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-1007-tl-ads-with-12871-v7-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-1007-tl-ads-with-12871-v7-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-1007-tl-ads-with-12871-v7-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-1007-tl-ads-with-12871-v7-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-1007-tl-ads-with-12871-v7-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-1007-tl-ads-with-12871-v7-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v7/default
chaiml-1007-tl-ads-with-12871-v7-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v7/default/config.json
chaiml-1007-tl-ads-with-12871-v7-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v7/default/generation_config.json
chaiml-1007-tl-ads-with-12871-v7-uploader: cp /dev/shm/model_output/recipe.yaml s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v7/default/recipe.yaml
chaiml-1007-tl-ads-with-12871-v7-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v7/default/chat_template.jinja
chaiml-1007-tl-ads-with-12871-v7-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v7/default/tokenizer_config.json
chaiml-1007-tl-ads-with-12871-v7-uploader: cp /dev/shm/model_output/special_tokens_map.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v7/default/special_tokens_map.json
chaiml-1007-tl-ads-with-12871-v7-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v7/default/model.safetensors.index.json
chaiml-1007-tl-ads-with-12871-v7-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v7/default/tokenizer.json
2026-03-20T00:04:51.133331+00:00 monitor updated for chaiml-1007-tl-ads-aggr_36615_v5
chaiml-1007-tl-ads-with-12871-v7-uploader: cp /dev/shm/model_output/model-00003-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v7/default/model-00003-of-00003.safetensors
chaiml-1007-tl-ads-with-12871-v7-uploader: cp /dev/shm/model_output/model-00002-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v7/default/model-00002-of-00003.safetensors
2026-03-20T00:04:53.133843+00:00 monitor updated for chaiml-1007-tl-ads-with_12871_v7
chaiml-1007-tl-ads-with-12871-v7-uploader: cp /dev/shm/model_output/model-00001-of-00003.safetensors s3://guanaco-vllm-models/chaiml-1007-tl-ads-with-12871-v7/default/model-00001-of-00003.safetensors
Job chaiml-1007-tl-ads-with-12871-v7-uploader completed after 189.99s with status: succeeded
Stopping job with name chaiml-1007-tl-ads-with-12871-v7-uploader
Pipeline stage VLLMUploader completed in 190.77s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 4.33s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-1007-tl-ads-with-12871-v7
Waiting for inference service chaiml-1007-tl-ads-with-12871-v7 to be ready
Inference service chaiml-1007-tl-ads-aggr-36615-v5 ready after 140.6841413974762s
Pipeline stage VLLMDeployer completed in 141.41s
run pipeline stage %s
Running pipeline stage StressChecker
2026-03-20T00:05:51.315069+00:00 monitor updated for chaiml-1007-tl-ads-aggr_36615_v5
Received healthy response to inference request in 2.73659348487854s
2026-03-20T00:05:53.342575+00:00 monitor updated for chaiml-1007-tl-ads-with_12871_v7
Received healthy response to inference request in 2.4017715454101562s
Received healthy response to inference request in 2.4467861652374268s
Received healthy response to inference request in 2.4890475273132324s
Received healthy response to inference request in 2.262921094894409s
Received healthy response to inference request in 2.226557970046997s
Received healthy response to inference request in 2.4458110332489014s
Received healthy response to inference request in 2.261549234390259s
Received healthy response to inference request in 2.322720527648926s
Received healthy response to inference request in 2.2670769691467285s
Received healthy response to inference request in 2.2667033672332764s
Received healthy response to inference request in 2.3455302715301514s
Received healthy response to inference request in 2.30350399017334s
Received healthy response to inference request in 2.252730369567871s
Received healthy response to inference request in 2.27176570892334s
Received healthy response to inference request in 2.2867021560668945s
Received healthy response to inference request in 2.356003522872925s
Received healthy response to inference request in 2.262424945831299s
Received healthy response to inference request in 2.343630790710449s
Received healthy response to inference request in 2.3989012241363525s
Received healthy response to inference request in 2.3076603412628174s
Received healthy response to inference request in 2.3366684913635254s
Received healthy response to inference request in 2.250128746032715s
Received healthy response to inference request in 2.418337106704712s
2026-03-20T00:06:51.476981+00:00 monitor updated for chaiml-1007-tl-ads-aggr_36615_v5
Received healthy response to inference request in 2.34920334815979s
2026-03-20T00:06:53.978112+00:00 monitor updated for chaiml-1007-tl-ads-with_12871_v7
Received healthy response to inference request in 2.320460081100464s
Received healthy response to inference request in 2.458292007446289s
Received healthy response to inference request in 2.397376537322998s
Received healthy response to inference request in 2.297243356704712s
Received healthy response to inference request in 2.4955177307128906s
30 requests
0 failed requests
5th percentile: 2.2512994766235352
10th percentile: 2.26066734790802
20th percentile: 2.2659469127655028
30th percentile: 2.282221221923828
40th percentile: 2.3059978008270265
50th percentile: 2.3296945095062256
60th percentile: 2.346999502182007
70th percentile: 2.3978339433670044
80th percentile: 2.42383189201355
90th percentile: 2.4613675594329836
95th percentile: 2.4926061391830445
99th percentile: 2.666681516170502
mean time: 2.352653988202413
Pipeline stage StressChecker completed in 77.03s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.98s
Shutdown handler de-registered
chaiml-1007-tl-ads-aggr_36615_v5 status is now deployed due to DeploymentManager action
chaiml-1007-tl-ads-aggr_36615_v5 status is now inactive due to system request
run pipeline stage %s
Pipeline stage MKMLDeleter completed in 0.95s
Skipping deletion as no model was successfully uploaded
chaiml-1007-tl-ads-aggr_36615_v5 status is now torndown due to DeploymentManager action