Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer
Waiting for job on zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer to finish
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ _____ __ __ ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ / _/ /_ ___ __/ / ___ ___ / / ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ / _/ / // / |/|/ / _ \/ -_) -_) / ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ /_//_/\_, /|__,__/_//_/\__/\__/_/ ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ /___/ ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ Version: 0.11.12 ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ Copyright 2023 MK ONE TECHNOLOGIES Inc. ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ https://mk1.ai ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ The license key for the current software has been verified as ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ belonging to: ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ Chai Research Corp. ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ Expiration: 2025-01-15 23:59:59 ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ║ ║
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
Failed to get response for submission blend_lofer_2024-10-07: ('http://chaiml-llama-8b-pairwis-8189-v27-predictor.tenant-chaiml-guanaco.k2.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:37626->127.0.0.1:8080: read: connection reset by peer\n')
Connection pool is full, discarding connection: %s. Connection pool size: %s
Connection pool is full, discarding connection: %s. Connection pool size: %s
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: Downloaded to shared memory in 1279.061s
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmphore2zdf, device:0
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: quantized model in 35.358s
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: Processed model zonemercy/Viral-ss-v15-12b1e5 in 1314.419s
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/zonemercy-viral-ss-v15-12b1e5-v1
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/zonemercy-viral-ss-v15-12b1e5-v1/config.json
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/zonemercy-viral-ss-v15-12b1e5-v1/special_tokens_map.json
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/zonemercy-viral-ss-v15-12b1e5-v1/tokenizer_config.json
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/zonemercy-viral-ss-v15-12b1e5-v1/tokenizer.json
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/zonemercy-viral-ss-v15-12b1e5-v1/flywheel_model.0.safetensors
zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer:
Loading 0: 0%| | 0/363 [00:00<?, ?it/s]
Loading 0: 1%|▏ | 5/363 [00:00<00:12, 28.61it/s]
Loading 0: 4%|▎ | 13/363 [00:00<00:07, 49.48it/s]
Loading 0: 5%|▌ | 19/363 [00:00<00:07, 46.64it/s]
Loading 0: 7%|▋ | 24/363 [00:00<00:07, 45.64it/s]
Loading 0: 9%|▊ | 31/363 [00:00<00:06, 51.67it/s]
Loading 0: 10%|█ | 37/363 [00:00<00:06, 49.30it/s]
Loading 0: 12%|█▏ | 43/363 [00:00<00:06, 50.70it/s]
Loading 0: 13%|█▎ | 49/363 [00:00<00:05, 53.27it/s]
Loading 0: 15%|█▌ | 55/363 [00:01<00:06, 50.09it/s]
Loading 0: 17%|█▋ | 61/363 [00:01<00:08, 35.22it/s]
Loading 0: 18%|█▊ | 66/363 [00:01<00:08, 36.24it/s]
Loading 0: 20%|█▉ | 72/363 [00:01<00:07, 40.99it/s]
Loading 0: 21%|██▏ | 78/363 [00:01<00:06, 42.85it/s]
Loading 0: 23%|██▎ | 83/363 [00:01<00:06, 41.44it/s]
Loading 0: 25%|██▍ | 90/363 [00:02<00:05, 46.53it/s]
Loading 0: 26%|██▋ | 96/363 [00:02<00:05, 44.93it/s]
Loading 0: 28%|██▊ | 101/363 [00:02<00:06, 42.31it/s]
Loading 0: 30%|██▉ | 108/363 [00:02<00:05, 48.96it/s]
Loading 0: 31%|███▏ | 114/363 [00:02<00:05, 46.38it/s]
Loading 0: 33%|███▎ | 119/363 [00:02<00:05, 45.85it/s]
Loading 0: 35%|███▍ | 126/363 [00:02<00:04, 51.08it/s]
Loading 0: 36%|███▋ | 132/363 [00:02<00:04, 49.23it/s]
Loading 0: 38%|███▊ | 138/363 [00:03<00:04, 50.80it/s]
Loading 0: 40%|███▉ | 144/363 [00:03<00:06, 31.61it/s]
Loading 0: 41%|████ | 149/363 [00:03<00:06, 33.63it/s]
Loading 0: 43%|████▎ | 157/363 [00:03<00:04, 42.07it/s]
Loading 0: 45%|████▍ | 163/363 [00:03<00:04, 43.02it/s]
Loading 0: 46%|████▋ | 168/363 [00:03<00:04, 43.80it/s]
Loading 0: 48%|████▊ | 175/363 [00:03<00:03, 49.43it/s]
Loading 0: 50%|████▉ | 181/363 [00:04<00:03, 48.35it/s]
Loading 0: 52%|█████▏ | 187/363 [00:04<00:03, 48.04it/s]
Loading 0: 53%|█████▎ | 193/363 [00:04<00:03, 50.86it/s]
Loading 0: 55%|█████▍ | 199/363 [00:04<00:03, 49.23it/s]
Loading 0: 56%|█████▋ | 205/363 [00:04<00:03, 49.53it/s]
Loading 0: 58%|█████▊ | 211/363 [00:04<00:02, 51.70it/s]
Loading 0: 60%|█████▉ | 217/363 [00:04<00:02, 49.78it/s]
Loading 0: 61%|██████▏ | 223/363 [00:05<00:03, 37.44it/s]
Loading 0: 63%|██████▎ | 228/363 [00:05<00:03, 36.81it/s]
Loading 0: 64%|██████▍ | 233/363 [00:05<00:03, 39.30it/s]
Loading 0: 66%|██████▌ | 239/363 [00:05<00:03, 38.39it/s]
Loading 0: 68%|██████▊ | 247/363 [00:05<00:02, 45.69it/s]
Loading 0: 69%|██████▉ | 252/363 [00:05<00:02, 45.89it/s]
Loading 0: 71%|███████ | 257/363 [00:05<00:02, 40.29it/s]
Loading 0: 73%|███████▎ | 265/363 [00:05<00:02, 48.61it/s]
Loading 0: 75%|███████▍ | 271/363 [00:06<00:01, 47.84it/s]
Loading 0: 76%|███████▋ | 277/363 [00:06<00:01, 48.13it/s]
Loading 0: 78%|███████▊ | 283/363 [00:06<00:01, 50.89it/s]
Loading 0: 80%|███████▉ | 289/363 [00:06<00:01, 49.24it/s]
Loading 0: 81%|████████▏ | 295/363 [00:06<00:01, 49.13it/s]
Loading 0: 83%|████████▎ | 302/363 [00:06<00:01, 53.87it/s]
Loading 0: 85%|████████▍ | 308/363 [00:13<00:18, 2.92it/s]
Loading 0: 86%|████████▌ | 312/363 [00:13<00:14, 3.63it/s]
Loading 0: 88%|████████▊ | 320/363 [00:13<00:07, 5.70it/s]
Loading 0: 90%|████████▉ | 326/363 [00:13<00:04, 7.65it/s]
Loading 0: 91%|█████████ | 331/363 [00:13<00:03, 9.74it/s]
Loading 0: 93%|█████████▎| 338/363 [00:14<00:01, 13.73it/s]
Loading 0: 95%|█████████▍| 344/363 [00:14<00:01, 17.26it/s]
Loading 0: 96%|█████████▋| 350/363 [00:14<00:00, 21.43it/s]
Loading 0: 98%|█████████▊| 356/363 [00:14<00:00, 26.41it/s]
Loading 0: 100%|█████████▉| 362/363 [00:14<00:00, 30.20it/s]
Job zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer completed after 1352.38s with status: succeeded
Stopping job with name zonemercy-viral-ss-v15-12b1e5-v1-mkmlizer
Pipeline stage MKMLizer completed in 1352.94s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.19s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service zonemercy-viral-ss-v15-12b1e5-v1
Waiting for inference service zonemercy-viral-ss-v15-12b1e5-v1 to be ready
Tearing down inference service zonemercy-viral-ss-v15-12b1e5-v1
%s, retrying in %s seconds...
Creating inference service zonemercy-viral-ss-v15-12b1e5-v1
Waiting for inference service zonemercy-viral-ss-v15-12b1e5-v1 to be ready
Tearing down inference service zonemercy-viral-ss-v15-12b1e5-v1
%s, retrying in %s seconds...
Creating inference service zonemercy-viral-ss-v15-12b1e5-v1
Waiting for inference service zonemercy-viral-ss-v15-12b1e5-v1 to be ready
Failed to get response for submission zonemercy-viral-ss-v12-12b1e5_v3: ('http://zonemercy-viral-ss-v12-12b1e5-v3-predictor.tenant-chaiml-guanaco.k.chaiverse.com/v1/models/GPT-J-6B-lit-v2:predict', 'read tcp 127.0.0.1:42378->127.0.0.1:8080: read: connection reset by peer\n')
Tearing down inference service zonemercy-viral-ss-v15-12b1e5-v1
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService zonemercy-viral-ss-v15-12b1e5-v1. The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'1\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'3700\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\'}, \'creationTimestamp\': \'2024-10-17T21:45:11Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}}, \'f:labels\': {\'.\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2024-10-17T21:45:11Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2024-10-17T21:45:11Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:lastFailureInfo\': {\'.\': {}, \'f:exitCode\': {}, \'f:message\': {}, \'f:reason\': {}}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2024-10-17T21:45:38Z\'}], \'name\': \'zonemercy-viral-ss-v15-12b1e5-v1\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'127225874\', \'uid\': \'a7101842-befb-49d4-b431-8ed90abd9853\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'topology.kubernetes.io/region\', \'operator\': \'In\', \'values\': [\'ORD1\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'RTX_A5000\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'env\': [{\'name\': \'MAX_TOKEN_INPUT\', \'value\': \'1024\'}, {\'name\': \'BEST_OF\', \'value\': \'8\'}, {\'name\': \'TEMPERATURE\', \'value\': \'1.0\'}, {\'name\': \'PRESENCE_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'FREQUENCY_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'TOP_P\', \'value\': \'1.0\'}, {\'name\': \'MIN_P\', \'value\': \'0.0\'}, {\'name\': \'TOP_K\', \'value\': \'40\'}, {\'name\': \'STOPPING_WORDS\', \'value\': \'["\\\\\\\\n", "</s>", "####", "Bot:", "User:", "You:", "<|im_end|>", "<|eot_id|>"]\'}, {\'name\': \'MAX_TOKENS\', \'value\': \'64\'}, {\'name\': \'MAX_BATCH_SIZE\', \'value\': \'128\'}, {\'name\': \'URL_ROUTE\', \'value\': \'GPT-J-6B-lit-v2\'}, {\'name\': \'OBJ_ACCESS_KEY_ID\', \'value\': \'LETMTTRMLFFAMTBK\'}, {\'name\': \'OBJ_SECRET_ACCESS_KEY\', \'value\': \'VwwZaqefOOoaouNxUk03oUmK9pVEfruJhjBHPGdgycK\'}, {\'name\': \'OBJ_ENDPOINT\', \'value\': \'https://accel-object.ord1.coreweave.com\'}, {\'name\': \'TENSORIZER_URI\', \'value\': \'s3://guanaco-mkml-models/zonemercy-viral-ss-v15-12b1e5-v1\'}, {\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'1\'}, {\'name\': \'MK1_MKML_LICENSE_KEY\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'key\', \'name\': \'mkml-license-key\'}}}], \'image\': \'gcr.io/chai-959f8/chai-guanaco/mkml:make_router_health_async\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'exec\': {\'command\': [\'cat\', \'/tmp/ready\']}, \'failureThreshold\': 1, \'initialDelaySeconds\': 10, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'2\', \'memory\': \'14Gi\', \'nvidia.com/gpu\': \'1\'}, \'requests\': {\'cpu\': \'2\', \'memory\': \'14Gi\', \'nvidia.com/gpu\': \'1\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 500, \'minReplicas\': 0, \'timeout\': 60, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\'}, \'name\': \'shared-memory-cache\'}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'zonemercy-viral-ss-v15-12b1e5-v1-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2024-10-17T21:45:38Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2024-10-17T21:45:38Z\', \'message\': \'Revision "zonemercy-viral-ss-v15-12b1e5-v1-predictor-00001" failed with message: Container failed with: ║\\n║ Chai Research Corp. ║\\n║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║\\n║ Expiration: 2025-01-15 23:59:59 ║\\n║ ║\\n╚═════════════════════════════════════════════════════════════════════╝\\n\\nINFO:datasets:PyTorch version 2.3.0 available.\\nInference config: InferenceConfig(server_num_workers=1, server_port=8080, max_batch_size=128, log_level=0, reserve_memory=2048, num_gpus=1, quantization_profile=s0, all_reduce_profile=None, kv_cache_profile=None, calibration_samples=-1, sampling=SamplingParameters(temperature=1.0, top_p=1.0, min_p=0.0, top_k=40, max_input_tokens=1024, max_tokens=64, stop=[\\\'\\\\n\\\', \\\'</s>\\\', \\\'####\\\', \\\'Bot:\\\', \\\'User:\\\', \\\'You:\\\', \\\'<|im_end|>\\\', \\\'<|eot_id|>\\\'], eos_token_ids=[], frequency_penalty=0.0, presence_penalty=0.0, reward_enabled=True, num_samples=8, reward_max_token_input=256, drop_incomplete_sentences=True, profile=False), url_route=GPT-J-6B-lit-v2, tensorizer_uri=s3://guanaco-mkml-models/zonemercy-viral-ss-v15-12b1e5-v1, s3_creds=S3Credentials(s3_access_key_id=\\\'LETMTTRMLFFAMTBK\\\', s3_secret_access_key=\\\'VwwZaqefOOoaouNxUk03oUmK9pVEfruJhjBHPGdgycK\\\', s3_endpoint=\\\'https://accel-object.ord1.coreweave.com\\\', s3_uncached_endpoint=\\\'https://object.ord1.coreweave.com\\\'), local_folder=/dev/shm/model_cache)\\nTraceback (most recent call last):\\n File "/code/mkml_inference_service/main.py", line 95, in <module>\\n model.load()\\n File "/code/mkml_inference_service/main.py", line 31, in load\\n self.engine = mkml_backend.AsyncInferenceService.from_folder(settings, settings.local_folder)\\n File "/code/mkml_inference_service/mkml_backend.py", line 45, in from_folder\\n with open(model_config) as f:\\nFileNotFoundError: [Errno 2] No such file or directory: \\\'/dev/shm/model_cache/config.json\\\'\\n.\', \'reason\': \'RevisionFailed\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2024-10-17T21:45:38Z\', \'message\': \'Configuration "zonemercy-viral-ss-v15-12b1e5-v1-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2024-10-17T21:45:38Z\', \'message\': \'Configuration "zonemercy-viral-ss-v15-12b1e5-v1-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2024-10-17T21:45:38Z\', \'message\': \'Configuration "zonemercy-viral-ss-v15-12b1e5-v1-predictor" does not have any ready Revision.\', \'reason\': \'RevisionMissing\', \'status\': \'False\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2024-10-17T21:45:38Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'False\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'lastFailureInfo\': {\'exitCode\': 1, \'message\': \' ║\\n║ Chai Research Corp. ║\\n║ Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f ║\\n║ Expiration: 2025-01-15 23:59:59 ║\\n║ ║\\n╚═════════════════════════════════════════════════════════════════════╝\\n\\nINFO:datasets:PyTorch version 2.3.0 available.\\nInference config: InferenceConfig(server_num_workers=1, server_port=8080, max_batch_size=128, log_level=0, reserve_memory=2048, num_gpus=1, quantization_profile=s0, all_reduce_profile=None, kv_cache_profile=None, calibration_samples=-1, sampling=SamplingParameters(temperature=1.0, top_p=1.0, min_p=0.0, top_k=40, max_input_tokens=1024, max_tokens=64, stop=[\\\'\\\\n\\\', \\\'</s>\\\', \\\'####\\\', \\\'Bot:\\\', \\\'User:\\\', \\\'You:\\\', \\\'<|im_end|>\\\', \\\'<|eot_id|>\\\'], eos_token_ids=[], frequency_penalty=0.0, presence_penalty=0.0, reward_enabled=True, num_samples=8, reward_max_token_input=256, drop_incomplete_sentences=True, profile=False), url_route=GPT-J-6B-lit-v2, tensorizer_uri=s3://guanaco-mkml-models/zonemercy-viral-ss-v15-12b1e5-v1, s3_creds=S3Credentials(s3_access_key_id=\\\'LETMTTRMLFFAMTBK\\\', s3_secret_access_key=\\\'VwwZaqefOOoaouNxUk03oUmK9pVEfruJhjBHPGdgycK\\\', s3_endpoint=\\\'https://accel-object.ord1.coreweave.com\\\', s3_uncached_endpoint=\\\'https://object.ord1.coreweave.com\\\'), local_folder=/dev/shm/model_cache)\\nTraceback (most recent call last):\\n File "/code/mkml_inference_service/main.py", line 95, in <module>\\n model.load()\\n File "/code/mkml_inference_service/main.py", line 31, in load\\n self.engine = mkml_backend.AsyncInferenceService.from_folder(settings, settings.local_folder)\\n File "/code/mkml_inference_service/mkml_backend.py", line 45, in from_folder\\n with open(model_config) as f:\\nFileNotFoundError: [Errno 2] No such file or directory: \\\'/dev/shm/model_cache/config.json\\\'\\n\', \'reason\': \'ModelLoadFailed\'}, \'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
Shutdown handler de-registered
zonemercy-viral-ss-v15-12b1e5_v1 status is now failed due to DeploymentManager action
zonemercy-viral-ss-v15-12b1e5_v1 status is now torndown due to DeploymentManager action