Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name yamatazen-ethereallight-12b-v1-mkmlizer
Waiting for job on yamatazen-ethereallight-12b-v1-mkmlizer to finish
yamatazen-ethereallight-12b-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
yamatazen-ethereallight-12b-v1-mkmlizer: ║ _____ __ __ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ /___/ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ Version: 0.12.8 ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ https://mk1.ai ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ The license key for the current software has been verified as ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ belonging to: ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ Chai Research Corp. ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
yamatazen-ethereallight-12b-v1-mkmlizer: Traceback (most recent call last):
yamatazen-ethereallight-12b-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 406, in hf_raise_for_status
yamatazen-ethereallight-12b-v1-mkmlizer: response.raise_for_status()
yamatazen-ethereallight-12b-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
yamatazen-ethereallight-12b-v1-mkmlizer: raise HTTPError(http_error_msg, response=self)
yamatazen-ethereallight-12b-v1-mkmlizer: requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/models/yamatazen/EtherealLight-12B/paths-info/main
yamatazen-ethereallight-12b-v1-mkmlizer: The above exception was the direct cause of the following exception:
yamatazen-ethereallight-12b-v1-mkmlizer: Traceback (most recent call last):
yamatazen-ethereallight-12b-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 193, in <module>
yamatazen-ethereallight-12b-v1-mkmlizer: cli()
yamatazen-ethereallight-12b-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
yamatazen-ethereallight-12b-v1-mkmlizer: return self.main(*args, **kwargs)
yamatazen-ethereallight-12b-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1078, in main
yamatazen-ethereallight-12b-v1-mkmlizer: rv = self.invoke(ctx)
yamatazen-ethereallight-12b-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
yamatazen-ethereallight-12b-v1-mkmlizer: return _process_result(sub_ctx.command.invoke(sub_ctx))
yamatazen-ethereallight-12b-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
yamatazen-ethereallight-12b-v1-mkmlizer: return ctx.invoke(self.callback, **ctx.params)
yamatazen-ethereallight-12b-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke
yamatazen-ethereallight-12b-v1-mkmlizer: return __callback(*args, **kwargs)
yamatazen-ethereallight-12b-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 39, in quantize
yamatazen-ethereallight-12b-v1-mkmlizer: temp_folder = download_to_shared_memory(repo_id, revision, hf_auth_token)
yamatazen-ethereallight-12b-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 81, in download_to_shared_memory
yamatazen-ethereallight-12b-v1-mkmlizer: if repo_has_model_safetensors(repo_id, revision, token):
yamatazen-ethereallight-12b-v1-mkmlizer: File "/code/uploading/mkmlize.py", line 104, in repo_has_model_safetensors
yamatazen-ethereallight-12b-v1-mkmlizer: files = [f.path for f in get_paths_info(repo_id, revision=revision, paths=["/"], token=token)]
yamatazen-ethereallight-12b-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
yamatazen-ethereallight-12b-v1-mkmlizer: return fn(*args, **kwargs)
yamatazen-ethereallight-12b-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 3259, in get_paths_info
yamatazen-ethereallight-12b-v1-mkmlizer: hf_raise_for_status(response)
yamatazen-ethereallight-12b-v1-mkmlizer: File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 477, in hf_raise_for_status
yamatazen-ethereallight-12b-v1-mkmlizer: raise _format(HfHubHTTPError, str(e), response) from e
yamatazen-ethereallight-12b-v1-mkmlizer: huggingface_hub.errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/models/yamatazen/EtherealLight-12B/paths-info/main (Request ID: Root=1-67bfb5ec-7f1a3c07337a212e71bb3413;64fada61-75e7-4f64-adef-055369bf8198)
yamatazen-ethereallight-12b-v1-mkmlizer: process spawn timed out after 10s
Job yamatazen-ethereallight-12b-v1-mkmlizer completed after 33.74s with status: failed
Stopping job with name yamatazen-ethereallight-12b-v1-mkmlizer
%s, retrying in %s seconds...
Starting job with name yamatazen-ethereallight-12b-v1-mkmlizer
Waiting for job on yamatazen-ethereallight-12b-v1-mkmlizer to finish
Failed to get response for submission nischaydnk-mistral24bas_99124_v4: HTTPConnectionPool(host='nischaydnk-mistral24bas-99124-v4-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
yamatazen-ethereallight-12b-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
yamatazen-ethereallight-12b-v1-mkmlizer: ║ _____ __ __ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ /___/ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ Version: 0.12.8 ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ https://mk1.ai ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ The license key for the current software has been verified as ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ belonging to: ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ Chai Research Corp. ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ Expiration: 2025-04-15 23:59:59 ║
yamatazen-ethereallight-12b-v1-mkmlizer: ║ ║
yamatazen-ethereallight-12b-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission mistralai-mistral-smal_78605_v18: HTTPConnectionPool(host='mistralai-mistral-smal-78605-v18-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
yamatazen-ethereallight-12b-v1-mkmlizer: Downloaded to shared memory in 47.623s
yamatazen-ethereallight-12b-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpubt6kvk0, device:0
yamatazen-ethereallight-12b-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
yamatazen-ethereallight-12b-v1-mkmlizer: quantized model in 36.885s
yamatazen-ethereallight-12b-v1-mkmlizer: Processed model yamatazen/EtherealLight-12B in 84.509s
yamatazen-ethereallight-12b-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
yamatazen-ethereallight-12b-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/yamatazen-ethereallight-12b-v1
Failed to get response for submission nischaydnk-mistral24bas_99124_v4: HTTPConnectionPool(host='nischaydnk-mistral24bas-99124-v4-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
yamatazen-ethereallight-12b-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/yamatazen-ethereallight-12b-v1/config.json
yamatazen-ethereallight-12b-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/yamatazen-ethereallight-12b-v1/special_tokens_map.json
yamatazen-ethereallight-12b-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/yamatazen-ethereallight-12b-v1/tokenizer_config.json
yamatazen-ethereallight-12b-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/yamatazen-ethereallight-12b-v1/tokenizer.json
yamatazen-ethereallight-12b-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/yamatazen-ethereallight-12b-v1/flywheel_model.0.safetensors
yamatazen-ethereallight-12b-v1-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%| | 2/363 [00:06<18:20, 3.05s/it]
Loading 0: 2%|▏ | 6/363 [00:06<04:52, 1.22it/s]
Loading 0: 3%|▎ | 11/363 [00:06<02:08, 2.74it/s]
Loading 0: 4%|▍ | 15/363 [00:06<01:21, 4.28it/s]
Loading 0: 6%|▋ | 23/363 [00:06<00:40, 8.49it/s]
Loading 0: 8%|▊ | 28/363 [00:06<00:29, 11.48it/s]
Loading 0: 9%|▉ | 33/363 [00:06<00:24, 13.74it/s]
Loading 0: 11%|█ | 40/363 [00:07<00:19, 16.17it/s]
Loading 0: 12%|█▏ | 44/363 [00:07<00:17, 18.57it/s]
Loading 0: 13%|█▎ | 49/363 [00:07<00:13, 22.81it/s]
Loading 0: 15%|█▍ | 53/363 [00:07<00:12, 24.97it/s]
Loading 0: 16%|█▌ | 58/363 [00:07<00:10, 28.74it/s]
Loading 0: 17%|█▋ | 62/363 [00:07<00:10, 29.90it/s]
Loading 0: 18%|█▊ | 67/363 [00:07<00:08, 33.53it/s]
Loading 0: 20%|█▉ | 71/363 [00:08<00:08, 33.41it/s]
Loading 0: 21%|██ | 76/363 [00:08<00:07, 36.76it/s]
Loading 0: 22%|██▏ | 81/363 [00:08<00:07, 37.95it/s]
Loading 0: 24%|██▎ | 86/363 [00:08<00:07, 39.56it/s]
Loading 0: 25%|██▌ | 91/363 [00:08<00:06, 41.84it/s]
Loading 0: 26%|██▋ | 96/363 [00:08<00:07, 34.09it/s]
Loading 0: 28%|██▊ | 103/363 [00:08<00:06, 41.53it/s]
Loading 0: 30%|██▉ | 108/363 [00:08<00:06, 40.86it/s]
Loading 0: 31%|███ | 113/363 [00:09<00:06, 40.78it/s]
Loading 0: 33%|███▎ | 118/363 [00:09<00:05, 41.95it/s]
Loading 0: 34%|███▍ | 123/363 [00:09<00:09, 25.74it/s]
Loading 0: 36%|███▌ | 130/363 [00:09<00:07, 32.36it/s]
Loading 0: 37%|███▋ | 135/363 [00:09<00:06, 34.27it/s]
Loading 0: 39%|███▊ | 140/363 [00:09<00:06, 36.43it/s]
Loading 0: 40%|███▉ | 145/363 [00:10<00:05, 39.29it/s]
Loading 0: 41%|████▏ | 150/363 [00:10<00:06, 32.67it/s]
Loading 0: 43%|████▎ | 155/363 [00:10<00:05, 36.12it/s]
Loading 0: 44%|████▍ | 160/363 [00:10<00:05, 34.99it/s]
Loading 0: 46%|████▌ | 166/363 [00:10<00:04, 39.54it/s]
Loading 0: 47%|████▋ | 171/363 [00:10<00:04, 39.15it/s]
Loading 0: 48%|████▊ | 176/363 [00:10<00:04, 38.55it/s]
Loading 0: 50%|████▉ | 181/363 [00:10<00:04, 39.97it/s]
Loading 0: 51%|█████ | 186/363 [00:11<00:05, 33.00it/s]
Loading 0: 53%|█████▎ | 191/363 [00:11<00:04, 36.62it/s]
Loading 0: 54%|█████▎ | 195/363 [00:11<00:04, 34.10it/s]
Loading 0: 56%|█████▌ | 202/363 [00:11<00:05, 28.94it/s]
Loading 0: 57%|█████▋ | 206/363 [00:11<00:05, 29.87it/s]
Loading 0: 58%|█████▊ | 211/363 [00:11<00:04, 33.75it/s]
Loading 0: 59%|█████▉ | 215/363 [00:12<00:04, 33.40it/s]
Loading 0: 61%|██████ | 220/363 [00:12<00:03, 36.46it/s]
Loading 0: 62%|██████▏ | 224/363 [00:12<00:03, 36.26it/s]
Loading 0: 63%|██████▎ | 229/363 [00:12<00:03, 39.09it/s]
Loading 0: 64%|██████▍ | 234/363 [00:12<00:03, 40.16it/s]
Loading 0: 66%|██████▌ | 239/363 [00:12<00:03, 40.56it/s]
Loading 0: 67%|██████▋ | 244/363 [00:12<00:02, 42.55it/s]
Loading 0: 69%|██████▊ | 249/363 [00:12<00:03, 34.84it/s]
Loading 0: 71%|███████ | 256/363 [00:13<00:02, 41.94it/s]
Loading 0: 72%|███████▏ | 261/363 [00:13<00:02, 42.22it/s]
Loading 0: 73%|███████▎ | 266/363 [00:13<00:02, 41.27it/s]
Loading 0: 75%|███████▍ | 271/363 [00:13<00:02, 42.70it/s]
Loading 0: 76%|███████▌ | 276/363 [00:13<00:02, 35.13it/s]
Loading 0: 78%|███████▊ | 283/363 [00:13<00:02, 30.96it/s]
Loading 0: 79%|███████▉ | 287/363 [00:13<00:02, 31.75it/s]
Loading 0: 80%|████████ | 292/363 [00:14<00:02, 34.52it/s]
Loading 0: 82%|████████▏ | 296/363 [00:14<00:01, 34.98it/s]
Loading 0: 83%|████████▎ | 301/363 [00:14<00:01, 37.62it/s]
Loading 0: 84%|████████▍ | 305/363 [00:14<00:01, 37.02it/s]
Loading 0: 85%|████████▌ | 310/363 [00:14<00:01, 39.52it/s]
Loading 0: 87%|████████▋ | 315/363 [00:14<00:01, 40.38it/s]
Loading 0: 88%|████████▊ | 320/363 [00:14<00:01, 40.78it/s]
Loading 0: 90%|████████▉ | 325/363 [00:14<00:00, 41.76it/s]
Loading 0: 91%|█████████ | 330/363 [00:15<00:00, 34.43it/s]
Loading 0: 93%|█████████▎| 337/363 [00:15<00:00, 41.83it/s]
Loading 0: 94%|█████████▍| 342/363 [00:15<00:00, 41.49it/s]
Loading 0: 96%|█████████▌| 347/363 [00:15<00:00, 41.10it/s]
Loading 0: 97%|█████████▋| 352/363 [00:15<00:00, 40.57it/s]
Loading 0: 98%|█████████▊| 357/363 [00:15<00:00, 33.96it/s]
Job yamatazen-ethereallight-12b-v1-mkmlizer completed after 151.81s with status: succeeded
Stopping job with name yamatazen-ethereallight-12b-v1-mkmlizer
Pipeline stage MKMLizer completed in 186.44s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service yamatazen-ethereallight-12b-v1
Waiting for inference service yamatazen-ethereallight-12b-v1 to be ready
Failed to get response for submission nischaydnk-mistral24bas_99124_v4: HTTPConnectionPool(host='nischaydnk-mistral24bas-99124-v4-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name yamatazen-eris-light-12b-v2-v1-mkmlizer
Waiting for job on yamatazen-eris-light-12b-v2-v1-mkmlizer to finish
Failed to get response for submission nischaydnk-mistral24bas_99124_v4: HTTPConnectionPool(host='nischaydnk-mistral24bas-99124-v4-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Inference service yamatazen-ethereallight-12b-v1 ready after 282.1760687828064s
Pipeline stage MKMLDeployer completed in 282.80s
run pipeline stage %s
Running pipeline stage StressChecker
Received healthy response to inference request in 2.0207183361053467s
Received healthy response to inference request in 1.553931474685669s
Received healthy response to inference request in 1.6360085010528564s
Retrying (%r) after connection broken by '%r': %s
Received healthy response to inference request in 1.5450663566589355s
Failed to get response for submission mistralai-mistral-smal_78605_v18: HTTPConnectionPool(host='mistralai-mistral-smal-78605-v18-predictor.tenant-chaiml-guanaco.k.chaiverse.com', port=80): Read timed out. (read timeout=12.0)
Received healthy response to inference request in 1.5029022693634033s
5 requests
0 failed requests
5th percentile: 1.5113350868225097
10th percentile: 1.5197679042816161
20th percentile: 1.5366335391998291
30th percentile: 1.5468393802642821
40th percentile: 1.5503854274749755
50th percentile: 1.553931474685669
60th percentile: 1.586762285232544
70th percentile: 1.619593095779419
80th percentile: 1.7129504680633545
90th percentile: 1.8668344020843506
95th percentile: 1.9437763690948486
99th percentile: 2.005329942703247
mean time: 1.6517253875732423
Pipeline stage StressChecker completed in 10.50s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.90s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.86s
Shutdown handler de-registered
yamatazen-ethereallight-12b_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 2405.08s
Shutdown handler de-registered
yamatazen-ethereallight-12b_v1 status is now inactive due to auto deactivation removed underperforming models
yamatazen-ethereallight-12b_v1 status is now torndown due to DeploymentManager action