chaiml-gspo-glm47-kimi-_69191

developer_uid: chai_backend_admin
submission_id: chaiml-gspo-glm47-kimi-_69191_v1
model_name: chaiml-gspo-glm47-kimi-_69191_v1
model_group: ChaiML/gspo-glm47-kimi-r
status: deployed
timestamp: 2026-04-03T19:21:59+00:00
num_battles: 10045
num_wins: 5741
celo_rating: 1353.41
family_friendly_score: 0.0
family_friendly_standard_error: 0.0
submission_type: basic
model_repo: ChaiML/gspo-glm47-kimi-rm-megablend-data-step400
model_architecture: Glm4MoeForCausalLM
model_num_parameters: 24110003200.0
best_of: 8
max_input_tokens: 1500
max_output_tokens: 80
reward_model: default
display_name: chaiml-gspo-glm47-kimi-_69191_v1
ineligible_reason: max_output_tokens!=64
is_internal_developer: True
language_model: ChaiML/gspo-glm47-kimi-rm-megablend-data-step400
model_size: 24B
ranking_group: single
us_pacific_date: 2026-04-03
win_ratio: 0.5715281234444998
generation_params: {'temperature': 1.0, 'top_p': 0.95, 'min_p': 0.0, 'top_k': 60, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['<|user|>', '<|assistant|>', '####', '<|im_end|>', '</s>'], 'max_input_tokens': 1500, 'best_of': 8, 'max_output_tokens': 80}
formatter: {'memory_template': "[gMASK]<sop><|system|>{bot_name}'s persona: {memory}", 'prompt_template': '', 'bot_template': '<|assistant|>{message}', 'user_template': '<|user|>{message}', 'response_template': '<|assistant|></think>', 'truncate_by_message': True}
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage VLLMUploader
Starting job with name chaiml-gspo-glm47-kimi-69191-v1-uploader
Waiting for job on chaiml-gspo-glm47-kimi-69191-v1-uploader to finish
chaiml-gspo-glm47-kimi-69191-v1-uploader: Using quantization_mode: w4a16
chaiml-gspo-glm47-kimi-69191-v1-uploader: Checking if ChaiML/gspo-glm47-kimi-rm-megablend-data-step400-W4A16 already exists in ChaiML
chaiml-gspo-glm47-kimi-69191-v1-uploader: Downloading snapshot of ChaiML/gspo-glm47-kimi-rm-megablend-data-step400...
2026-04-03T18:39:26.807883+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T18:40:26.894292+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T18:41:27.052760+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T18:42:27.155555+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T18:43:27.284159+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: Traceback (most recent call last):
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
chaiml-gspo-glm47-kimi-69191-v1-uploader:     yield
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpx/_transports/default.py", line 250, in handle_request
chaiml-gspo-glm47-kimi-69191-v1-uploader:     resp = self._pool.handle_request(req)
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
chaiml-gspo-glm47-kimi-69191-v1-uploader:     raise exc from None
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
chaiml-gspo-glm47-kimi-69191-v1-uploader:     response = connection.handle_request(
chaiml-gspo-glm47-kimi-69191-v1-uploader:                ^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/connection.py", line 103, in handle_request
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return self._connection.handle_request(request)
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/http11.py", line 136, in handle_request
chaiml-gspo-glm47-kimi-69191-v1-uploader:     raise exc
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/http11.py", line 106, in handle_request
chaiml-gspo-glm47-kimi-69191-v1-uploader:     ) = self._receive_response_headers(**kwargs)
chaiml-gspo-glm47-kimi-69191-v1-uploader:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/http11.py", line 177, in _receive_response_headers
chaiml-gspo-glm47-kimi-69191-v1-uploader:     event = self._receive_event(timeout=timeout)
chaiml-gspo-glm47-kimi-69191-v1-uploader:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpcore/_sync/http11.py", line 231, in _receive_event
chaiml-gspo-glm47-kimi-69191-v1-uploader:     raise RemoteProtocolError(msg)
chaiml-gspo-glm47-kimi-69191-v1-uploader: httpcore.RemoteProtocolError: Server disconnected without sending a response.
chaiml-gspo-glm47-kimi-69191-v1-uploader: The above exception was the direct cause of the following exception:
chaiml-gspo-glm47-kimi-69191-v1-uploader: Traceback (most recent call last):
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/code/uploading/compress.py", line 344, in <module>
chaiml-gspo-glm47-kimi-69191-v1-uploader:     cli()
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1485, in __call__
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return self.main(*args, **kwargs)
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1406, in main
chaiml-gspo-glm47-kimi-69191-v1-uploader:     rv = self.invoke(ctx)
chaiml-gspo-glm47-kimi-69191-v1-uploader:          ^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1873, in invoke
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return _process_result(sub_ctx.command.invoke(sub_ctx))
chaiml-gspo-glm47-kimi-69191-v1-uploader:                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 1269, in invoke
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return ctx.invoke(self.callback, **ctx.params)
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/click/core.py", line 824, in invoke
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return callback(*args, **kwargs)
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/code/uploading/compress.py", line 52, in process
chaiml-gspo-glm47-kimi-69191-v1-uploader:     quantize_w4a16(repo_id, download_path, output_path, revision, hf_auth_token)
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/code/uploading/compress.py", line 86, in quantize_w4a16
chaiml-gspo-glm47-kimi-69191-v1-uploader:     hf_utils.download_snapshot_to_folder(repo_id, revision, hf_auth_token, download_path)
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/code/src/inference_service_common/hf_utils.py", line 60, in download_snapshot_to_folder
chaiml-gspo-glm47-kimi-69191-v1-uploader:     snapshot_download(
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return fn(*args, **kwargs)
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/_snapshot_download.py", line 450, in snapshot_download
chaiml-gspo-glm47-kimi-69191-v1-uploader:     thread_map(
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/tqdm/std.py", line 1169, in __iter__
chaiml-gspo-glm47-kimi-69191-v1-uploader:     for obj in iterable:
chaiml-gspo-glm47-kimi-69191-v1-uploader:                ^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 619, in result_iterator
chaiml-gspo-glm47-kimi-69191-v1-uploader:     yield _result_or_cancel(fs.pop())
chaiml-gspo-glm47-kimi-69191-v1-uploader:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 317, in _result_or_cancel
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return fut.result(timeout)
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return self.__get_result()
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
chaiml-gspo-glm47-kimi-69191-v1-uploader:     raise self._exception
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run
chaiml-gspo-glm47-kimi-69191-v1-uploader:     result = self.fn(*self.args, **self.kwargs)
chaiml-gspo-glm47-kimi-69191-v1-uploader:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/_snapshot_download.py", line 430, in _inner_hf_hub_download
chaiml-gspo-glm47-kimi-69191-v1-uploader:     hf_hub_download(  # type: ignore
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return fn(*args, **kwargs)
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/file_download.py", line 965, in hf_hub_download
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return _hf_hub_download_to_local_dir(
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/file_download.py", line 1274, in _hf_hub_download_to_local_dir
chaiml-gspo-glm47-kimi-69191-v1-uploader:     (url_to_download, etag, commit_hash, expected_size, xet_file_data, head_call_error) = _get_metadata_or_catch_error(
chaiml-gspo-glm47-kimi-69191-v1-uploader:                                                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/file_download.py", line 1653, in _get_metadata_or_catch_error
chaiml-gspo-glm47-kimi-69191-v1-uploader:     metadata = get_hf_file_metadata(
chaiml-gspo-glm47-kimi-69191-v1-uploader:                ^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_validators.py", line 89, in _inner_fn
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return fn(*args, **kwargs)
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/file_download.py", line 1576, in get_hf_file_metadata
chaiml-gspo-glm47-kimi-69191-v1-uploader:     response = _httpx_follow_relative_redirects_with_backoff(
chaiml-gspo-glm47-kimi-69191-v1-uploader:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_http.py", line 685, in _httpx_follow_relative_redirects_with_backoff
chaiml-gspo-glm47-kimi-69191-v1-uploader:     response = http_backoff(
chaiml-gspo-glm47-kimi-69191-v1-uploader:                ^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_http.py", line 559, in http_backoff
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return next(
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_http.py", line 467, in _http_backoff_base
chaiml-gspo-glm47-kimi-69191-v1-uploader:     response = client.request(method=method, url=url, **kwargs)
chaiml-gspo-glm47-kimi-69191-v1-uploader:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpx/_client.py", line 825, in request
chaiml-gspo-glm47-kimi-69191-v1-uploader:     return self.send(request, auth=auth, follow_redirects=follow_redirects)
chaiml-gspo-glm47-kimi-69191-v1-uploader:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpx/_client.py", line 914, in send
chaiml-gspo-glm47-kimi-69191-v1-uploader:     response = self._send_handling_auth(
chaiml-gspo-glm47-kimi-69191-v1-uploader:                ^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpx/_client.py", line 942, in _send_handling_auth
chaiml-gspo-glm47-kimi-69191-v1-uploader:     response = self._send_handling_redirects(
chaiml-gspo-glm47-kimi-69191-v1-uploader:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpx/_client.py", line 979, in _send_handling_redirects
chaiml-gspo-glm47-kimi-69191-v1-uploader:     response = self._send_single_request(request)
chaiml-gspo-glm47-kimi-69191-v1-uploader:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpx/_client.py", line 1014, in _send_single_request
chaiml-gspo-glm47-kimi-69191-v1-uploader:     response = transport.handle_request(request)
chaiml-gspo-glm47-kimi-69191-v1-uploader:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpx/_transports/default.py", line 249, in handle_request
chaiml-gspo-glm47-kimi-69191-v1-uploader:     with map_httpcore_exceptions():
chaiml-gspo-glm47-kimi-69191-v1-uploader:          ^^^^^^^^^^^^^^^^^^^^^^^^^
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
chaiml-gspo-glm47-kimi-69191-v1-uploader:     self.gen.throw(value)
chaiml-gspo-glm47-kimi-69191-v1-uploader:   File "/usr/local/lib/python3.12/dist-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
chaiml-gspo-glm47-kimi-69191-v1-uploader:     raise mapped_exc(message) from exc
chaiml-gspo-glm47-kimi-69191-v1-uploader: httpx.RemoteProtocolError: Server disconnected without sending a response.
Job chaiml-gspo-glm47-kimi-69191-v1-uploader completed after 342.26s with status: failed
Job failed chaiml-gspo-glm47-kimi-69191-v1-uploader: 
Stopping job with name chaiml-gspo-glm47-kimi-69191-v1-uploader
%s, retrying in %s seconds...
Starting job with name chaiml-gspo-glm47-kimi-69191-v1-uploader
Waiting for job on chaiml-gspo-glm47-kimi-69191-v1-uploader to finish
2026-04-03T18:44:27.410112+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: Using quantization_mode: w4a16
2026-04-03T18:45:27.545850+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T18:46:27.678977+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T18:47:27.796019+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T18:48:27.920707+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: Downloaded in 245.463s
2026-04-03T18:49:28.045972+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T18:50:28.138532+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
Failed to get request counts for guanaco-submitter. Falling back to default
2026-04-03T18:51:28.326203+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T18:52:28.438232+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T18:53:28.947373+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: Applying quantization...
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:53:42 INFO __init__.py L202: Patched transformers.models.glm4_moe.modeling_glm4_moe.Glm4MoeMoE -> auto_round.modeling.unfused_moe.glm_moe.LinearGlm4MoeMoE[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:54:16 INFO base.py L486: using torch.bfloat16 for quantization tuning[0m
2026-04-03T18:54:29.076610+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:54:33 INFO device.py L1468:  'peak_ram': 11.94GB, 'peak_vram': 1.44GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:54:43 INFO device.py L1468:  'peak_ram': 16.4GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:54:57 INFO device.py L1468:  'peak_ram': 16.96GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:55:07 INFO device.py L1468:  'peak_ram': 19.38GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:55:23 INFO device.py L1468:  'peak_ram': 19.7GB, 'peak_vram': 1.59GB[0m
2026-04-03T18:55:29.176907+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:55:34 INFO device.py L1468:  'peak_ram': 22.12GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:55:45 INFO device.py L1468:  'peak_ram': 22.12GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:56:00 INFO device.py L1468:  'peak_ram': 22.62GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:56:12 INFO device.py L1468:  'peak_ram': 22.62GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:56:26 INFO device.py L1468:  'peak_ram': 23.36GB, 'peak_vram': 1.59GB[0m
2026-04-03T18:56:29.289766+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:56:38 INFO device.py L1468:  'peak_ram': 23.36GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:56:49 INFO device.py L1468:  'peak_ram': 23.57GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:57:04 INFO device.py L1468:  'peak_ram': 23.57GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:57:16 INFO device.py L1468:  'peak_ram': 23.57GB, 'peak_vram': 1.59GB[0m
2026-04-03T18:57:29.399263+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:57:28 INFO device.py L1468:  'peak_ram': 23.57GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:57:43 INFO device.py L1468:  'peak_ram': 23.57GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:57:55 INFO device.py L1468:  'peak_ram': 23.57GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:58:10 INFO device.py L1468:  'peak_ram': 23.57GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:58:22 INFO device.py L1468:  'peak_ram': 23.57GB, 'peak_vram': 1.59GB[0m
2026-04-03T18:58:29.628290+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:58:33 INFO device.py L1468:  'peak_ram': 23.57GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:58:44 INFO device.py L1468:  'peak_ram': 23.57GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:58:50 INFO device.py L1468:  'peak_ram': 23.57GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:58:58 INFO device.py L1468:  'peak_ram': 24.3GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:59:04 INFO device.py L1468:  'peak_ram': 24.3GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:59:08 INFO device.py L1468:  'peak_ram': 24.82GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:59:15 INFO device.py L1468:  'peak_ram': 24.82GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:59:19 INFO device.py L1468:  'peak_ram': 25.66GB, 'peak_vram': 1.59GB[0m
2026-04-03T18:59:29.746054+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:59:25 INFO device.py L1468:  'peak_ram': 25.66GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:59:36 INFO device.py L1468:  'peak_ram': 26.33GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 18:59:48 INFO device.py L1468:  'peak_ram': 26.33GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:00:12 INFO device.py L1468:  'peak_ram': 27.26GB, 'peak_vram': 1.59GB[0m
2026-04-03T19:00:29.849248+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:00:23 INFO device.py L1468:  'peak_ram': 27.26GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:00:37 INFO device.py L1468:  'peak_ram': 27.26GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:00:48 INFO device.py L1468:  'peak_ram': 27.26GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:00:59 INFO device.py L1468:  'peak_ram': 27.26GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:01:13 INFO device.py L1468:  'peak_ram': 27.26GB, 'peak_vram': 1.59GB[0m
2026-04-03T19:01:29.956000+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:01:24 INFO device.py L1468:  'peak_ram': 27.32GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:01:38 INFO device.py L1468:  'peak_ram': 27.32GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:01:50 INFO device.py L1468:  'peak_ram': 28.22GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:02:02 INFO device.py L1468:  'peak_ram': 28.22GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:02:16 INFO device.py L1468:  'peak_ram': 29.26GB, 'peak_vram': 1.59GB[0m
2026-04-03T19:02:30.075646+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:02:28 INFO device.py L1468:  'peak_ram': 29.26GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:02:42 INFO device.py L1468:  'peak_ram': 30.13GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:02:51 INFO device.py L1468:  'peak_ram': 30.13GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:02:59 INFO device.py L1468:  'peak_ram': 30.56GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:03:11 INFO device.py L1468:  'peak_ram': 30.56GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:03:19 INFO device.py L1468:  'peak_ram': 31.69GB, 'peak_vram': 1.59GB[0m
2026-04-03T19:03:30.188752+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:03:35 INFO device.py L1468:  'peak_ram': 31.69GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:03:55 INFO device.py L1468:  'peak_ram': 32.4GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:04:12 INFO device.py L1468:  'peak_ram': 32.4GB, 'peak_vram': 1.59GB[0m
2026-04-03T19:04:30.313369+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:04:30 INFO device.py L1468:  'peak_ram': 32.4GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:04:43 INFO device.py L1468:  'peak_ram': 32.4GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:04:54 INFO device.py L1468:  'peak_ram': 32.4GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:05:10 INFO device.py L1468:  'peak_ram': 32.4GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:05:21 INFO device.py L1468:  'peak_ram': 32.4GB, 'peak_vram': 1.59GB[0m
2026-04-03T19:05:30.505768+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:05:37 INFO device.py L1468:  'peak_ram': 33.37GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:05:49 INFO device.py L1468:  'peak_ram': 33.37GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:06:02 INFO device.py L1468:  'peak_ram': 33.79GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:06:17 INFO device.py L1468:  'peak_ram': 33.79GB, 'peak_vram': 1.59GB[0m
2026-04-03T19:06:30.619181+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:06:29 INFO device.py L1468:  'peak_ram': 34.43GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:06:41 INFO device.py L1468:  'peak_ram': 34.43GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:06:56 INFO device.py L1468:  'peak_ram': 35.33GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:07:07 INFO device.py L1468:  'peak_ram': 35.33GB, 'peak_vram': 1.59GB[0m
2026-04-03T19:07:30.734129+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:07:23 INFO device.py L1468:  'peak_ram': 36.54GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:07:33 INFO device.py L1468:  'peak_ram': 36.54GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:07:42 INFO device.py L1468:  'peak_ram': 37.11GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:07:50 INFO device.py L1468:  'peak_ram': 37.11GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:07:55 INFO device.py L1468:  'peak_ram': 37.89GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:08:03 INFO device.py L1468:  'peak_ram': 37.89GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:08:12 INFO device.py L1468:  'peak_ram': 37.89GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:08:22 INFO device.py L1468:  'peak_ram': 37.89GB, 'peak_vram': 1.59GB[0m
2026-04-03T19:08:31.455361+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:08:35 INFO device.py L1468:  'peak_ram': 37.89GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:08:44 INFO device.py L1468:  'peak_ram': 37.89GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:08:52 INFO device.py L1468:  'peak_ram': 37.89GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:09:04 INFO device.py L1468:  'peak_ram': 38.82GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:09:15 INFO device.py L1468:  'peak_ram': 38.82GB, 'peak_vram': 1.59GB[0m
2026-04-03T19:09:31.584716+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:09:29 INFO device.py L1468:  'peak_ram': 39.91GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:09:40 INFO device.py L1468:  'peak_ram': 39.91GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:09:49 INFO device.py L1468:  'peak_ram': 40.35GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:09:59 INFO device.py L1468:  'peak_ram': 40.35GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:10:04 INFO device.py L1468:  'peak_ram': 41.42GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:10:08 INFO device.py L1468:  'peak_ram': 41.42GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:10:16 INFO device.py L1468:  'peak_ram': 42.31GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:10:21 INFO device.py L1468:  'peak_ram': 42.31GB, 'peak_vram': 1.59GB[0m
2026-04-03T19:10:31.690385+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:10:28 INFO device.py L1468:  'peak_ram': 43.49GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:10:33 INFO device.py L1468:  'peak_ram': 43.49GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:10:37 INFO device.py L1468:  'peak_ram': 44.36GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:10:45 INFO device.py L1468:  'peak_ram': 44.36GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:10:49 INFO device.py L1468:  'peak_ram': 44.64GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:10:57 INFO device.py L1468:  'peak_ram': 44.64GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:11:02 INFO device.py L1468:  'peak_ram': 44.64GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:11:11 INFO shard_writer.py L208: model has been saved to /dev/shm/model_output/[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [33;1m2026-04-03 19:11:12 WARNING export.py L336: /dev/shm/model_output already exists, this may cause model conflict[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: [38;20m2026-04-03 19:11:12 INFO device.py L1468:  'peak_ram': 44.64GB, 'peak_vram': 1.59GB[0m
chaiml-gspo-glm47-kimi-69191-v1-uploader: Checking if ChaiML/gspo-glm47-kimi-rm-megablend-data-step400-W4A16 already exists in ChaiML
chaiml-gspo-glm47-kimi-69191-v1-uploader: Creating repo ChaiML/gspo-glm47-kimi-rm-megablend-data-step400-W4A16 and uploading /dev/shm/model_output to it
chaiml-gspo-glm47-kimi-69191-v1-uploader: ---------- 2026-04-03 19:11:13 (0:00:00) ----------
chaiml-gspo-glm47-kimi-69191-v1-uploader: Files:   hashed 7/45 (32.0M/197.4G) | pre-uploaded: 0/0 (0.0/197.4G) (+44 unsure) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-kimi-69191-v1-uploader: Workers: hashing: 38 | get upload mode: 1 | pre-uploading: 0 | committing: 0 | waiting: 25
chaiml-gspo-glm47-kimi-69191-v1-uploader: ---------------------------------------------------
2026-04-03T19:11:31.819449+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: 
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-kimi-69191-v1-uploader: ---------- 2026-04-03 19:12:13 (0:01:00) ----------
chaiml-gspo-glm47-kimi-69191-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 12/40 (47.2G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-kimi-69191-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 28 | committing: 0 | waiting: 36
chaiml-gspo-glm47-kimi-69191-v1-uploader: ---------------------------------------------------
2026-04-03T19:12:31.961799+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader:                              
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
[K[F
chaiml-gspo-glm47-kimi-69191-v1-uploader: ---------- 2026-04-03 19:13:13 (0:02:00) ----------
chaiml-gspo-glm47-kimi-69191-v1-uploader: Files:   hashed 45/45 (197.4G/197.4G) | pre-uploaded: 36/40 (175.9G/197.4G) | committed: 0/45 (0.0/197.4G) | ignored: 0
chaiml-gspo-glm47-kimi-69191-v1-uploader: Workers: hashing: 0 | get upload mode: 0 | pre-uploading: 4 | committing: 0 | waiting: 60
chaiml-gspo-glm47-kimi-69191-v1-uploader: ---------------------------------------------------
2026-04-03T19:13:32.073890+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader:                              Processed model ChaiML/gspo-glm47-kimi-rm-megablend-data-step400 in 1761.787s
chaiml-gspo-glm47-kimi-69191-v1-uploader: creating bucket guanaco-vllm-models
chaiml-gspo-glm47-kimi-69191-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:56: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-kimi-69191-v1-uploader:   RE_S3_DATESTRING = re.compile('\.[0-9]*(?:[Z\\-\\+]*?)')
chaiml-gspo-glm47-kimi-69191-v1-uploader: /usr/lib/python3/dist-packages/S3/BaseUtils.py:57: SyntaxWarning: invalid escape sequence '\s'
chaiml-gspo-glm47-kimi-69191-v1-uploader:   RE_XML_NAMESPACE = re.compile(b'^(<?[^>]+?>\s*|\s*)(<\w+) xmlns=[\'"](https?://[^\'"]+)[\'"]', re.MULTILINE)
chaiml-gspo-glm47-kimi-69191-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:240: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-kimi-69191-v1-uploader:   invalid = re.search("([^a-z0-9\.-])", bucket, re.UNICODE)
chaiml-gspo-glm47-kimi-69191-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:244: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-kimi-69191-v1-uploader:   invalid = re.search("([^A-Za-z0-9\._-])", bucket, re.UNICODE)
chaiml-gspo-glm47-kimi-69191-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:255: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-kimi-69191-v1-uploader:   if re.search("-\.", bucket, re.UNICODE):
chaiml-gspo-glm47-kimi-69191-v1-uploader: /usr/lib/python3/dist-packages/S3/Utils.py:257: SyntaxWarning: invalid escape sequence '\.'
chaiml-gspo-glm47-kimi-69191-v1-uploader:   if re.search("\.\.", bucket, re.UNICODE):
chaiml-gspo-glm47-kimi-69191-v1-uploader: /usr/lib/python3/dist-packages/S3/S3Uri.py:155: SyntaxWarning: invalid escape sequence '\w'
chaiml-gspo-glm47-kimi-69191-v1-uploader:   _re = re.compile("^(\w+://)?(.*)", re.UNICODE)
chaiml-gspo-glm47-kimi-69191-v1-uploader: /usr/lib/python3/dist-packages/S3/FileLists.py:480: SyntaxWarning: invalid escape sequence '\*'
chaiml-gspo-glm47-kimi-69191-v1-uploader:   wildcard_split_result = re.split("\*|\?", uri_str, maxsplit=1)
chaiml-gspo-glm47-kimi-69191-v1-uploader: Bucket 's3://guanaco-vllm-models/' created
chaiml-gspo-glm47-kimi-69191-v1-uploader: uploading /dev/shm/model_output to s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/config.json
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/chat_template.jinja s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/chat_template.jinja
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/tokenizer_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/tokenizer_config.json
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/generation_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/generation_config.json
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/quantization_config.json s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/quantization_config.json
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model.safetensors.index.json s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model.safetensors.index.json
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/tokenizer.json s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/tokenizer.json
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00038-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00038-of-00038.safetensors
2026-04-03T19:14:32.186327+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00036-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00036-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00037-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00037-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00010-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00010-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00016-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00016-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00028-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00028-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00034-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00034-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00011-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00011-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00025-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00025-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00026-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00026-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00009-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00009-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00032-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00032-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00019-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00019-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00035-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00035-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00005-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00005-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00015-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00015-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00001-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00001-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00007-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00007-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00014-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00014-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00022-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00022-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00021-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00021-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00012-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00012-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00003-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00003-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00023-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00023-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00004-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00004-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00030-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00030-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00013-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00013-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00029-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00029-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00024-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00024-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00002-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00002-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00018-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00018-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00020-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00020-of-00038.safetensors
chaiml-gspo-glm47-kimi-69191-v1-uploader: cp /dev/shm/model_output/model-00031-of-00038.safetensors s3://guanaco-vllm-models/chaiml-gspo-glm47-kimi-69191-v1/default/model-00031-of-00038.safetensors
Job chaiml-gspo-glm47-kimi-69191-v1-uploader completed after 1868.08s with status: succeeded
Stopping job with name chaiml-gspo-glm47-kimi-69191-v1-uploader
Pipeline stage VLLMUploader completed in 2212.44s
run pipeline stage %s
Running pipeline stage VLLMUploaderAMD
Pipeline stage vllm_upload_amd skipped, reason=not amd cluster
Pipeline stage VLLMUploaderAMD completed in 0.24s
run pipeline stage %s
Running pipeline stage VLLMTemplater
Pipeline stage VLLMTemplater completed in 3.63s
run pipeline stage %s
Running pipeline stage VLLMDeployer
Creating inference service chaiml-gspo-glm47-kimi-69191-v1
Waiting for inference service chaiml-gspo-glm47-kimi-69191-v1 to be ready
2026-04-03T19:15:32.314961+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T19:16:32.445902+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T19:17:32.596793+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T19:18:32.747242+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
2026-04-03T19:19:33.053533+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
Inference service chaiml-gspo-glm47-kimi-69191-v1 ready after 282.40013551712036s
Pipeline stage VLLMDeployer completed in 283.12s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 8.921180963516235s
Received healthy response to inference request in 2.0734424591064453s
Received healthy response to inference request in 8.836258172988892s
2026-04-03T19:20:33.356182+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
Received healthy response to inference request in 10.056238889694214s
Received healthy response to inference request in 8.940481662750244s
Received healthy response to inference request in 2.17576265335083s
Received healthy response to inference request in 2.1481401920318604s
Received healthy response to inference request in 2.1739630699157715s
Received healthy response to inference request in 8.64180302619934s
Received healthy response to inference request in 2.083364486694336s
Received healthy response to inference request in 2.2380712032318115s
Received healthy response to inference request in 2.0463781356811523s
Received healthy response to inference request in 2.308913469314575s
Received healthy response to inference request in 2.032925844192505s
Received healthy response to inference request in 2.068856954574585s
Received healthy response to inference request in 2.1710469722747803s
Received healthy response to inference request in 2.2504565715789795s
Received healthy response to inference request in 2.1800076961517334s
Received healthy response to inference request in 2.2553210258483887s
Received healthy response to inference request in 2.012854814529419s
2026-04-03T19:21:33.530859+00:00 monitor updated for chaiml-gspo-glm47-kimi-_69191_v1
Received healthy response to inference request in 2.144747257232666s
Received healthy response to inference request in 2.1036293506622314s
Received healthy response to inference request in 2.14559006690979s
Received healthy response to inference request in 2.0889716148376465s
Received healthy response to inference request in 2.2321724891662598s
Received healthy response to inference request in 2.5939443111419678s
Received healthy response to inference request in 2.187641143798828s
Received healthy response to inference request in 2.1220312118530273s
Received healthy response to inference request in 2.1193318367004395s
Received healthy response to inference request in 2.258481502532959s
30 requests
0 failed requests
5th percentile: 2.038979375362396
10th percentile: 2.0666090726852415
20th percentile: 2.0878501892089845
30th percentile: 2.121221399307251
40th percentile: 2.147120141983032
50th percentile: 2.174862861633301
60th percentile: 2.2054536819458006
70th percentile: 2.2519159078598023
80th percentile: 2.3659196376800544
90th percentile: 8.844750452041627
95th percentile: 8.93179634809494
99th percentile: 9.732669293880464
mean time: 3.3204003016153973
Pipeline stage StressChecker completed in 110.13s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.88s
Shutdown handler de-registered
chaiml-gspo-glm47-kimi-_69191_v1 status is now deployed due to DeploymentManager action