chaiml-llama31-mer-v2-_44570

submission_id: chaiml-llama31-mer-v2-_44570_v12
developer_uid: NischayDnk
status: failed
model_repo: ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572
generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 576, 'best_of': 1, 'max_output_tokens': 1}
formatter: {'memory_template': '', 'prompt_template': '', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '', 'truncate_by_message': True}
timestamp: 2025-08-20T07:03:37+00:00
model_name: chaiml-llama31-mer-v2-_44570_v12
Resubmit model
Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name chaiml-llama31-mer-v2-44570-v12-mkmlizer
Waiting for job on chaiml-llama31-mer-v2-44570-v12-mkmlizer to finish
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║                                                                     ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║   ██████   ██████  █████   ████  ████                               ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║  ░░██████ ██████  ░░███   ███░  ░░███                               ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║   ░███░█████░███   ░███  ███     ░███                               ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║   ░███░░███ ░███   ░███████      ░███                               ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║   ░███ ░░░  ░███   ░███░░███     ░███                               ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║   ░███      ░███   ░███ ░░███    ░███                               ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║   █████     █████  █████ ░░████  █████                              ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║  ░░░░░     ░░░░░  ░░░░░   ░░░░  ░░░░░                               ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║                                                                     ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║  Version: 0.30.2                                                    ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║  Features: FLYWHEEL, CUDA                                           ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║  Copyright 2023-2025 MK ONE TECHNOLOGIES Inc.                       ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║  https://mk1.ai                                                     ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║                                                                     ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║  The license key for the current software has been verified as      ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║  belonging to:                                                      ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║                                                                     ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║  Chai Research Corp.                                                ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ║                                                                     ║
chaiml-llama31-mer-v2-44570-v12-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
chaiml-llama31-mer-v2-44570-v12-mkmlizer: Downloaded to shared memory in 10.928s
chaiml-llama31-mer-v2-44570-v12-mkmlizer: Checking if ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572 already exists in ChaiML
chaiml-llama31-mer-v2-44570-v12-mkmlizer: quantizing model to /dev/shm/model_cache, profile:t0, folder:/tmp/tmpeo7p95t7, device:0
chaiml-llama31-mer-v2-44570-v12-mkmlizer: Saving flywheel model at /dev/shm/model_cache
chaiml-llama31-mer-v2-44570-v12-mkmlizer: quantized model in 16.093s
chaiml-llama31-mer-v2-44570-v12-mkmlizer: Processed model ChaiML/llama31-mer-v2-try1-new8m-filterv3-full-512seq-bestep-572 in 27.022s
chaiml-llama31-mer-v2-44570-v12-mkmlizer: creating bucket guanaco-mkml-models
chaiml-llama31-mer-v2-44570-v12-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
chaiml-llama31-mer-v2-44570-v12-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/chaiml-llama31-mer-v2-44570-v12/nvidia
chaiml-llama31-mer-v2-44570-v12-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/chaiml-llama31-mer-v2-44570-v12/nvidia/config.json
chaiml-llama31-mer-v2-44570-v12-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/chaiml-llama31-mer-v2-44570-v12/nvidia/special_tokens_map.json
chaiml-llama31-mer-v2-44570-v12-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/chaiml-llama31-mer-v2-44570-v12/nvidia/tokenizer_config.json
chaiml-llama31-mer-v2-44570-v12-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/chaiml-llama31-mer-v2-44570-v12/nvidia/tokenizer.json
chaiml-llama31-mer-v2-44570-v12-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/chaiml-llama31-mer-v2-44570-v12/nvidia/flywheel_model.0.safetensors
chaiml-llama31-mer-v2-44570-v12-mkmlizer: 
Loading 0:   0%|          | 0/291 [00:00<?, ?it/s]
Loading 0:   2%|▏         | 5/291 [00:00<00:07, 40.85it/s]
Loading 0:   5%|▍         | 14/291 [00:00<00:05, 54.25it/s]
Loading 0:   8%|▊         | 23/291 [00:00<00:04, 58.07it/s]
Loading 0:  11%|█         | 32/291 [00:00<00:04, 58.58it/s]
Loading 0:  14%|█▍        | 41/291 [00:00<00:04, 58.88it/s]
Loading 0:  17%|█▋        | 50/291 [00:00<00:04, 59.55it/s]
Loading 0:  20%|██        | 59/291 [00:01<00:03, 59.85it/s]
Loading 0:  23%|██▎       | 68/291 [00:01<00:03, 59.22it/s]
Loading 0:  26%|██▋       | 77/291 [00:01<00:03, 59.58it/s]
Loading 0:  29%|██▊       | 83/291 [00:01<00:04, 49.31it/s]
Loading 0:  31%|███       | 89/291 [00:01<00:04, 50.07it/s]
Loading 0:  33%|███▎      | 95/291 [00:01<00:04, 47.25it/s]
Loading 0:  36%|███▌      | 104/291 [00:01<00:03, 51.84it/s]
Loading 0:  39%|███▉      | 113/291 [00:02<00:03, 54.74it/s]
Loading 0:  42%|████▏     | 122/291 [00:02<00:02, 56.44it/s]
Loading 0:  45%|████▌     | 131/291 [00:02<00:02, 56.52it/s]
Loading 0:  48%|████▊     | 140/291 [00:02<00:02, 57.55it/s]
Loading 0:  51%|█████     | 149/291 [00:02<00:02, 57.84it/s]
Loading 0:  54%|█████▍    | 158/291 [00:02<00:02, 59.01it/s]
Loading 0:  57%|█████▋    | 167/291 [00:02<00:02, 60.01it/s]
Loading 0:  61%|██████    | 177/291 [00:03<00:01, 64.53it/s]
Loading 0:  63%|██████▎   | 184/291 [00:03<00:01, 64.33it/s]
Loading 0:  66%|██████▌   | 191/291 [00:03<00:02, 45.89it/s]
Loading 0:  68%|██████▊   | 197/291 [00:03<00:01, 48.40it/s]
Loading 0:  70%|██████▉   | 203/291 [00:03<00:01, 46.47it/s]
Loading 0:  73%|███████▎  | 212/291 [00:03<00:01, 50.32it/s]
Loading 0:  76%|███████▌  | 221/291 [00:04<00:01, 52.72it/s]
Loading 0:  79%|███████▉  | 230/291 [00:04<00:01, 54.31it/s]
Loading 0:  82%|████████▏ | 239/291 [00:04<00:00, 56.43it/s]
Loading 0:  85%|████████▌ | 248/291 [00:04<00:00, 57.92it/s]
Loading 0:  88%|████████▊ | 257/291 [00:04<00:00, 59.17it/s]
Loading 0:  91%|█████████▏| 266/291 [00:04<00:00, 59.20it/s]
Loading 0:  95%|█████████▍| 275/291 [00:04<00:00, 59.81it/s]
Loading 0:  97%|█████████▋| 282/291 [00:05<00:00, 56.66it/s]
Loading 0:  99%|█████████▉| 288/291 [00:05<00:00, 46.07it/s]
                                                            
Job chaiml-llama31-mer-v2-44570-v12-mkmlizer completed after 53.45s with status: succeeded
Stopping job with name chaiml-llama31-mer-v2-44570-v12-mkmlizer
Pipeline stage MKMLizer completed in 53.87s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.15s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service chaiml-llama31-mer-v2-44570-v12
Waiting for inference service chaiml-llama31-mer-v2-44570-v12 to be ready
Tearing down inference service chaiml-llama31-mer-v2-44570-v12
%s, retrying in %s seconds...
Creating inference service chaiml-llama31-mer-v2-44570-v12
Waiting for inference service chaiml-llama31-mer-v2-44570-v12 to be ready
Tearing down inference service chaiml-llama31-mer-v2-44570-v12
%s, retrying in %s seconds...
Creating inference service chaiml-llama31-mer-v2-44570-v12
Waiting for inference service chaiml-llama31-mer-v2-44570-v12 to be ready
Tearing down inference service chaiml-llama31-mer-v2-44570-v12
clean up pipeline due to error=DeploymentError('Timeout to start the InferenceService chaiml-llama31-mer-v2-44570-v12.                                The InferenceService is as following: {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'kind\': \'InferenceService\', \'metadata\': {\'annotations\': {\'autoscaling.knative.dev/class\': \'hpa.autoscaling.knative.dev\', \'autoscaling.knative.dev/container-concurrency-target-percentage\': \'70\', \'autoscaling.knative.dev/initial-scale\': \'1\', \'autoscaling.knative.dev/max-scale-down-rate\': \'1.1\', \'autoscaling.knative.dev/max-scale-up-rate\': \'2\', \'autoscaling.knative.dev/metric\': \'mean_pod_latency_ms_v2\', \'autoscaling.knative.dev/panic-threshold-percentage\': \'650\', \'autoscaling.knative.dev/panic-window-percentage\': \'35\', \'autoscaling.knative.dev/scale-down-delay\': \'30s\', \'autoscaling.knative.dev/scale-to-zero-grace-period\': \'10m\', \'autoscaling.knative.dev/stable-window\': \'180s\', \'autoscaling.knative.dev/target\': \'300\', \'autoscaling.knative.dev/target-burst-capacity\': \'-1\', \'autoscaling.knative.dev/tick-interval\': \'15s\', \'features.knative.dev/http-full-duplex\': \'Enabled\', \'networking.knative.dev/ingress-class\': \'istio.ingress.networking.knative.dev\'}, \'creationTimestamp\': \'2025-08-20T07:24:38Z\', \'finalizers\': [\'inferenceservice.finalizers\'], \'generation\': 1, \'labels\': {\'istio.io/rev\': \'prod-canary\', \'knative.coreweave.cloud/ingress\': \'istio.ingress.networking.knative.dev\', \'prometheus.k.chaiverse.com\': \'true\', \'qos.coreweave.cloud/latency\': \'low\'}, \'managedFields\': [{\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:annotations\': {\'.\': {}, \'f:autoscaling.knative.dev/class\': {}, \'f:autoscaling.knative.dev/container-concurrency-target-percentage\': {}, \'f:autoscaling.knative.dev/initial-scale\': {}, \'f:autoscaling.knative.dev/max-scale-down-rate\': {}, \'f:autoscaling.knative.dev/max-scale-up-rate\': {}, \'f:autoscaling.knative.dev/metric\': {}, \'f:autoscaling.knative.dev/panic-threshold-percentage\': {}, \'f:autoscaling.knative.dev/panic-window-percentage\': {}, \'f:autoscaling.knative.dev/scale-down-delay\': {}, \'f:autoscaling.knative.dev/scale-to-zero-grace-period\': {}, \'f:autoscaling.knative.dev/stable-window\': {}, \'f:autoscaling.knative.dev/target\': {}, \'f:autoscaling.knative.dev/target-burst-capacity\': {}, \'f:autoscaling.knative.dev/tick-interval\': {}, \'f:features.knative.dev/http-full-duplex\': {}, \'f:networking.knative.dev/ingress-class\': {}}, \'f:labels\': {\'.\': {}, \'f:istio.io/rev\': {}, \'f:knative.coreweave.cloud/ingress\': {}, \'f:prometheus.k.chaiverse.com\': {}, \'f:qos.coreweave.cloud/latency\': {}}}, \'f:spec\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:affinity\': {\'.\': {}, \'f:nodeAffinity\': {\'.\': {}, \'f:tion\': {}, \'f:requiredDuringSchedulingIgnoredDuringExecution\': {}}}, \'f:containerConcurrency\': {}, \'f:containers\': {}, \'f:imagePullSecrets\': {}, \'f:maxReplicas\': {}, \'f:minReplicas\': {}, \'f:timeout\': {}, \'f:volumes\': {}}}}, \'manager\': \'OpenAPI-Generator\', \'operation\': \'Update\', \'time\': \'2025-08-20T07:24:38Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:metadata\': {\'f:finalizers\': {\'.\': {}, \'v:"inferenceservice.finalizers"\': {}}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'time\': \'2025-08-20T07:24:38Z\'}, {\'apiVersion\': \'serving.kserve.io/v1beta1\', \'fieldsType\': \'FieldsV1\', \'fieldsV1\': {\'f:status\': {\'.\': {}, \'f:components\': {\'.\': {}, \'f:predictor\': {\'.\': {}, \'f:latestCreatedRevision\': {}}}, \'f:conditions\': {}, \'f:modelStatus\': {\'.\': {}, \'f:states\': {\'.\': {}, \'f:activeModelState\': {}, \'f:targetModelState\': {}}, \'f:transitionStatus\': {}}, \'f:observedGeneration\': {}}}, \'manager\': \'manager\', \'operation\': \'Update\', \'subresource\': \'status\', \'time\': \'2025-08-20T07:24:39Z\'}], \'name\': \'chaiml-llama31-mer-v2-44570-v12\', \'namespace\': \'tenant-chaiml-guanaco\', \'resourceVersion\': \'682951889\', \'uid\': \'a7234c9b-048d-4746-b382-cfda8d19642e\'}, \'spec\': {\'predictor\': {\'affinity\': {\'nodeAffinity\': {\'tion\': [{\'preference\': {\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'RTX_A5000\']}]}, \'weight\': 5}], \'requiredDuringSchedulingIgnoredDuringExecution\': {\'nodeSelectorTerms\': [{\'matchExpressions\': [{\'key\': \'gpu.nvidia.com/class\', \'operator\': \'In\', \'values\': [\'RTX_A5000\', \'L40S\']}]}]}}}, \'containerConcurrency\': 0, \'containers\': [{\'env\': [{\'name\': \'MAX_TOKEN_INPUT\', \'value\': \'1152\'}, {\'name\': \'BEST_OF\', \'value\': \'1\'}, {\'name\': \'TEMPERATURE\', \'value\': \'1.0\'}, {\'name\': \'PRESENCE_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'FREQUENCY_PENALTY\', \'value\': \'0.0\'}, {\'name\': \'TOP_P\', \'value\': \'1.0\'}, {\'name\': \'MIN_P\', \'value\': \'0.0\'}, {\'name\': \'TOP_K\', \'value\': \'40\'}, {\'name\': \'STOPPING_WORDS\', \'value\': \'["\\\\\\\\n"]\'}, {\'name\': \'MAX_TOKENS\', \'value\': \'1\'}, {\'name\': \'MAX_BATCH_SIZE\', \'value\': \'128\'}, {\'name\': \'MAX_CACHED_RESPONSES\', \'value\': \'256\'}, {\'name\': \'URL_ROUTE\', \'value\': \'GPT-J-6B-lit-v2\'}, {\'name\': \'OBJ_ACCESS_KEY_ID\', \'value\': \'LETMTTRMLFFAMTBK\'}, {\'name\': \'OBJ_SECRET_ACCESS_KEY\', \'value\': \'VwwZaqefOOoaouNxUk03oUmK9pVEfruJhjBHPGdgycK\'}, {\'name\': \'OBJ_ENDPOINT\', \'value\': \'https://accel-object.ord1.coreweave.com\'}, {\'name\': \'TENSORIZER_URI\', \'value\': \'s3://guanaco-mkml-models/chaiml-llama31-mer-v2-44570-v12/nvidia\'}, {\'name\': \'RESERVE_MEMORY\', \'value\': \'2048\'}, {\'name\': \'DOWNLOAD_TO_LOCAL\', \'value\': \'/dev/shm/model_cache\'}, {\'name\': \'NUM_GPUS\', \'value\': \'1\'}, {\'name\': \'MK1_QUANTIZATION_PROFILE\', \'value\': \'t0\'}, {\'name\': \'MK1_MKML_LICENSE_KEY\', \'valueFrom\': {\'secretKeyRef\': {\'key\': \'key\', \'name\': \'mkml-license-key\'}}}], \'image\': \'gcr.io/chai-959f8/chai-guanaco/mkml:mkml_v0.30.2\', \'imagePullPolicy\': \'IfNotPresent\', \'name\': \'kserve-container\', \'readinessProbe\': {\'exec\': {\'command\': [\'cat\', \'/tmp/ready\']}, \'failureThreshold\': 1, \'initialDelaySeconds\': 10, \'periodSeconds\': 10, \'successThreshold\': 1, \'timeoutSeconds\': 5}, \'resources\': {\'limits\': {\'cpu\': \'2\', \'memory\': \'12Gi\', \'nvidia.com/gpu\': \'1\'}, \'requests\': {\'cpu\': \'2\', \'memory\': \'12Gi\', \'nvidia.com/gpu\': \'1\'}}, \'volumeMounts\': [{\'mountPath\': \'/dev/shm\', \'name\': \'shared-memory-cache\'}]}], \'imagePullSecrets\': [{\'name\': \'docker-creds\'}], \'maxReplicas\': 40, \'minReplicas\': 0, \'timeout\': 60, \'volumes\': [{\'emptyDir\': {\'medium\': \'Memory\'}, \'name\': \'shared-memory-cache\'}]}}, \'status\': {\'components\': {\'predictor\': {\'latestCreatedRevision\': \'chaiml-llama31-mer-v2-44570-v12-predictor-00001\'}}, \'conditions\': [{\'lastTransitionTime\': \'2025-08-20T07:24:39Z\', \'reason\': \'PredictorConfigurationReady not ready\', \'severity\': \'Info\', \'status\': \'Unknown\', \'type\': \'LatestDeploymentReady\'}, {\'lastTransitionTime\': \'2025-08-20T07:24:39Z\', \'severity\': \'Info\', \'status\': \'Unknown\', \'type\': \'PredictorConfigurationReady\'}, {\'lastTransitionTime\': \'2025-08-20T07:24:39Z\', \'message\': \'Configuration "chaiml-llama31-mer-v2-44570-v12-predictor" is waiting for a Revision to become ready.\', \'reason\': \'RevisionMissing\', \'status\': \'Unknown\', \'type\': \'PredictorReady\'}, {\'lastTransitionTime\': \'2025-08-20T07:24:39Z\', \'message\': \'Configuration "chaiml-llama31-mer-v2-44570-v12-predictor" is waiting for a Revision to become ready.\', \'reason\': \'RevisionMissing\', \'severity\': \'Info\', \'status\': \'Unknown\', \'type\': \'PredictorRouteReady\'}, {\'lastTransitionTime\': \'2025-08-20T07:24:39Z\', \'message\': \'Configuration "chaiml-llama31-mer-v2-44570-v12-predictor" is waiting for a Revision to become ready.\', \'reason\': \'RevisionMissing\', \'status\': \'Unknown\', \'type\': \'Ready\'}, {\'lastTransitionTime\': \'2025-08-20T07:24:39Z\', \'reason\': \'PredictorRouteReady not ready\', \'severity\': \'Info\', \'status\': \'Unknown\', \'type\': \'RoutesReady\'}], \'modelStatus\': {\'states\': {\'activeModelState\': \'\', \'targetModelState\': \'Pending\'}, \'transitionStatus\': \'InProgress\'}, \'observedGeneration\': 1}}')
Shutdown handler de-registered
chaiml-llama31-mer-v2-_44570_v12 status is now failed due to DeploymentManager action