cycy233-modelv-chai-step9000

developer_uid: cycy233

submission_id: cycy233-modelv-chai-step9000_v1

model_name: cycy233-modelv-chai-step1000_v1

model_group: cycy233/modelV_chai_step

status: torndown

timestamp: 2025-04-30T03:31:14+00:00

num_battles: 8898

num_wins: 3823

celo_rating: 1235.39

family_friendly_score: 0.496

family_friendly_standard_error: 0.007070841534074993

submission_type: basic

model_repo: cycy233/modelV_chai_step9000

model_architecture: MistralForCausalLM

model_num_parameters: 12772070400.0

best_of: 8

max_input_tokens: 1024

max_output_tokens: 64

reward_model: default

latencies: [{'batch_size': 1, 'throughput': 0.6047647301880228, 'latency_mean': 1.6534768915176392, 'latency_p50': 1.6443809270858765, 'latency_p90': 1.8164761781692504}, {'batch_size': 3, 'throughput': 1.102983346209897, 'latency_mean': 2.7094667625427244, 'latency_p50': 2.709179401397705, 'latency_p90': 2.9984298944473267}, {'batch_size': 5, 'throughput': 1.324125655026118, 'latency_mean': 3.7641166114807127, 'latency_p50': 3.7816940546035767, 'latency_p90': 4.168919825553894}, {'batch_size': 6, 'throughput': 1.3880068660192812, 'latency_mean': 4.30444811463356, 'latency_p50': 4.313125729560852, 'latency_p90': 4.784019970893859}, {'batch_size': 8, 'throughput': 1.4477256728732089, 'latency_mean': 5.489273487329483, 'latency_p50': 5.506035089492798, 'latency_p90': 6.185006022453308}, {'batch_size': 10, 'throughput': 1.4932112124614987, 'latency_mean': 6.64665696978569, 'latency_p50': 6.569210052490234, 'latency_p90': 7.501220464706421}]

gpu_counts: {'NVIDIA RTX A5000': 1}

display_name: cycy233-modelv-chai-step1000_v1

is_internal_developer: False

language_model: cycy233/modelV_chai_step9000

model_size: 13B

ranking_group: single

throughput_3p7s: 1.32

us_pacific_date: 2025-04-29

win_ratio: 0.42964711171049674

generation_params: {'temperature': 1.0, 'top_p': 1.0, 'min_p': 0.0, 'top_k': 40, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'stopping_words': ['\n'], 'max_input_tokens': 1024, 'best_of': 8, 'max_output_tokens': 64}

formatter: {'memory_template': "{bot_name}'s Persona: {memory}\n####\n", 'prompt_template': '{prompt}\n<START>\n', 'bot_template': '{bot_name}: {message}\n', 'user_template': '{user_name}: {message}\n', 'response_template': '{bot_name}:', 'truncate_by_message': False}

Resubmit model

Shutdown handler not registered because Python interpreter is not running in the main thread
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLizer
Starting job with name cycy233-modelv-chai-step9000-v1-mkmlizer
Waiting for job on cycy233-modelv-chai-step9000-v1-mkmlizer to finish
cycy233-modelv-chai-step9000-v1-mkmlizer: ╔═════════════════════════════════════════════════════════════════════╗
cycy233-modelv-chai-step9000-v1-mkmlizer: ║     _____            __           __                                ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║    / _/ /_ ___    __/ /  ___ ___ / /                                ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║   / _/ / // / |/|/ / _ \/ -_) -_) /                                 ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║  /_//_/\_, /|__,__/_//_/\__/\__/_/                                  ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║       /___/                                                         ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║                                                                     ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║  Version: 0.12.8                                                    ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║  Copyright 2023 MK ONE TECHNOLOGIES Inc.                            ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║  https://mk1.ai                                                     ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║                                                                     ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║  The license key for the current software has been verified as      ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║  belonging to:                                                      ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║                                                                     ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║  Chai Research Corp.                                                ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║  Account ID: 7997a29f-0ceb-4cc7-9adf-840c57b4ae6f                   ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║  Expiration: 2028-03-31 23:59:59                                    ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ║                                                                     ║
cycy233-modelv-chai-step9000-v1-mkmlizer: ╚═════════════════════════════════════════════════════════════════════╝
cycy233-modelv-chai-step9000-v1-mkmlizer: Downloaded to shared memory in 43.909s
cycy233-modelv-chai-step9000-v1-mkmlizer: quantizing model to /dev/shm/model_cache, profile:s0, folder:/tmp/tmpdsvytkx4, device:0
cycy233-modelv-chai-step9000-v1-mkmlizer: Saving flywheel model at /dev/shm/model_cache
cycy233-modelv-chai-step9000-v1-mkmlizer: quantized model in 35.495s
cycy233-modelv-chai-step9000-v1-mkmlizer: Processed model cycy233/modelV_chai_step9000 in 79.405s
cycy233-modelv-chai-step9000-v1-mkmlizer: creating bucket guanaco-mkml-models
cycy233-modelv-chai-step9000-v1-mkmlizer: Bucket 's3://guanaco-mkml-models/' created
cycy233-modelv-chai-step9000-v1-mkmlizer: uploading /dev/shm/model_cache to s3://guanaco-mkml-models/cycy233-modelv-chai-step9000-v1
cycy233-modelv-chai-step9000-v1-mkmlizer: cp /dev/shm/model_cache/config.json s3://guanaco-mkml-models/cycy233-modelv-chai-step9000-v1/config.json
cycy233-modelv-chai-step9000-v1-mkmlizer: cp /dev/shm/model_cache/special_tokens_map.json s3://guanaco-mkml-models/cycy233-modelv-chai-step9000-v1/special_tokens_map.json
cycy233-modelv-chai-step9000-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer_config.json s3://guanaco-mkml-models/cycy233-modelv-chai-step9000-v1/tokenizer_config.json
cycy233-modelv-chai-step9000-v1-mkmlizer: cp /dev/shm/model_cache/tokenizer.json s3://guanaco-mkml-models/cycy233-modelv-chai-step9000-v1/tokenizer.json
cycy233-modelv-chai-step9000-v1-mkmlizer: cp /dev/shm/model_cache/flywheel_model.0.safetensors s3://guanaco-mkml-models/cycy233-modelv-chai-step9000-v1/flywheel_model.0.safetensors
cycy233-modelv-chai-step9000-v1-mkmlizer: 
Loading 0:   0%|          | 0/363 [00:00<?, ?it/s]
Loading 0:   1%|▏         | 5/363 [00:00<00:11, 31.40it/s]
Loading 0:   4%|▎         | 13/363 [00:00<00:06, 52.82it/s]
Loading 0:   5%|▌         | 19/363 [00:00<00:07, 48.45it/s]
Loading 0:   7%|▋         | 25/363 [00:00<00:06, 49.22it/s]
Loading 0:   9%|▊         | 31/363 [00:00<00:06, 52.30it/s]
Loading 0:  10%|█         | 37/363 [00:00<00:06, 49.64it/s]
Loading 0:  12%|█▏        | 43/363 [00:00<00:06, 49.83it/s]
Loading 0:  13%|█▎        | 49/363 [00:00<00:06, 51.95it/s]
Loading 0:  15%|█▌        | 55/363 [00:01<00:06, 49.59it/s]
Loading 0:  17%|█▋        | 61/363 [00:01<00:08, 37.02it/s]
Loading 0:  18%|█▊        | 66/363 [00:01<00:07, 37.30it/s]
Loading 0:  20%|█▉        | 72/363 [00:01<00:06, 41.83it/s]
Loading 0:  21%|██▏       | 78/363 [00:01<00:06, 41.53it/s]
Loading 0:  23%|██▎       | 83/363 [00:01<00:07, 39.45it/s]
Loading 0:  25%|██▍       | 89/363 [00:02<00:06, 43.44it/s]
Loading 0:  26%|██▌       | 94/363 [00:02<00:06, 43.22it/s]
Loading 0:  27%|██▋       | 99/363 [00:02<00:05, 44.14it/s]
Loading 0:  29%|██▉       | 105/363 [00:02<00:06, 42.70it/s]
Loading 0:  31%|███       | 112/363 [00:02<00:05, 45.98it/s]
Loading 0:  32%|███▏      | 117/363 [00:02<00:05, 44.42it/s]
Loading 0:  34%|███▍      | 123/363 [00:02<00:05, 43.78it/s]
Loading 0:  35%|███▌      | 128/363 [00:02<00:05, 42.54it/s]
Loading 0:  37%|███▋      | 134/363 [00:03<00:04, 45.97it/s]
Loading 0:  38%|███▊      | 139/363 [00:03<00:05, 44.23it/s]
Loading 0:  40%|███▉      | 144/363 [00:03<00:07, 28.02it/s]
Loading 0:  41%|████      | 149/363 [00:03<00:07, 30.35it/s]
Loading 0:  43%|████▎     | 156/363 [00:03<00:05, 38.22it/s]
Loading 0:  44%|████▍     | 161/363 [00:03<00:05, 39.62it/s]
Loading 0:  46%|████▌     | 166/363 [00:03<00:04, 40.50it/s]
Loading 0:  47%|████▋     | 172/363 [00:04<00:04, 40.90it/s]
Loading 0:  49%|████▉     | 177/363 [00:04<00:04, 40.53it/s]
Loading 0:  50%|█████     | 183/363 [00:04<00:04, 44.87it/s]
Loading 0:  52%|█████▏    | 188/363 [00:04<00:04, 43.44it/s]
Loading 0:  53%|█████▎    | 193/363 [00:04<00:03, 43.92it/s]
Loading 0:  55%|█████▍    | 199/363 [00:04<00:03, 42.46it/s]
Loading 0:  56%|█████▌    | 204/363 [00:04<00:03, 41.62it/s]
Loading 0:  58%|█████▊    | 211/363 [00:04<00:03, 46.73it/s]
Loading 0:  60%|█████▉    | 217/363 [00:05<00:03, 44.94it/s]
Loading 0:  61%|██████    | 222/363 [00:05<00:03, 46.03it/s]
Loading 0:  63%|██████▎   | 227/363 [00:05<00:04, 33.16it/s]
Loading 0:  64%|██████▎   | 231/363 [00:05<00:03, 33.12it/s]
Loading 0:  65%|██████▌   | 237/363 [00:05<00:03, 39.01it/s]
Loading 0:  67%|██████▋   | 242/363 [00:05<00:02, 40.40it/s]
Loading 0:  68%|██████▊   | 247/363 [00:05<00:02, 42.73it/s]
Loading 0:  70%|██████▉   | 253/363 [00:06<00:02, 42.97it/s]
Loading 0:  71%|███████   | 258/363 [00:06<00:02, 41.81it/s]
Loading 0:  73%|███████▎  | 265/363 [00:06<00:02, 46.72it/s]
Loading 0:  74%|███████▍  | 270/363 [00:06<00:01, 46.93it/s]
Loading 0:  76%|███████▌  | 275/363 [00:06<00:02, 39.78it/s]
Loading 0:  78%|███████▊  | 283/363 [00:06<00:01, 47.19it/s]
Loading 0:  80%|███████▉  | 289/363 [00:06<00:01, 45.53it/s]
Loading 0:  81%|████████  | 294/363 [00:06<00:01, 44.52it/s]
Loading 0:  83%|████████▎ | 300/363 [00:07<00:01, 48.28it/s]
Loading 0:  84%|████████▍ | 305/363 [00:13<00:21,  2.64it/s]
Loading 0:  85%|████████▌ | 309/363 [00:13<00:16,  3.37it/s]
Loading 0:  86%|████████▌ | 313/363 [00:14<00:11,  4.33it/s]
Loading 0:  88%|████████▊ | 319/363 [00:14<00:06,  6.41it/s]
Loading 0:  89%|████████▉ | 324/363 [00:14<00:04,  8.60it/s]
Loading 0:  91%|█████████ | 330/363 [00:14<00:02, 11.55it/s]
Loading 0:  93%|█████████▎| 338/363 [00:14<00:01, 17.26it/s]
Loading 0:  95%|█████████▍| 344/363 [00:14<00:00, 20.95it/s]
Loading 0:  96%|█████████▌| 349/363 [00:14<00:00, 24.37it/s]
Loading 0:  98%|█████████▊| 356/363 [00:14<00:00, 31.22it/s]
Loading 0: 100%|█████████▉| 362/363 [00:15<00:00, 34.49it/s]
                                                            
Job cycy233-modelv-chai-step9000-v1-mkmlizer completed after 104.81s with status: succeeded
Stopping job with name cycy233-modelv-chai-step9000-v1-mkmlizer
Pipeline stage MKMLizer completed in 105.27s
run pipeline stage %s
Running pipeline stage MKMLTemplater
Pipeline stage MKMLTemplater completed in 0.14s
run pipeline stage %s
Running pipeline stage MKMLDeployer
Creating inference service cycy233-modelv-chai-step9000-v1
Waiting for inference service cycy233-modelv-chai-step9000-v1 to be ready
Inference service cycy233-modelv-chai-step9000-v1 ready after 140.54603028297424s
Pipeline stage MKMLDeployer completed in 140.98s
run pipeline stage %s
Running pipeline stage StressChecker
HTTPConnectionPool(host='guanaco-submitter.guanaco-backend.k2.chaiverse.com', port=80): Read timed out. (read timeout=20)
Received unhealthy response to inference request!
Received healthy response to inference request in 2.069108486175537s
Received healthy response to inference request in 1.7152161598205566s
Received healthy response to inference request in 1.7507736682891846s
Received healthy response to inference request in 1.9195799827575684s
5 requests
1 failed requests
5th percentile: 1.7223276615142822
10th percentile: 1.7294391632080077
20th percentile: 1.743662166595459
30th percentile: 1.7845349311828613
40th percentile: 1.8520574569702148
50th percentile: 1.9195799827575684
60th percentile: 1.979391384124756
70th percentile: 2.0392027854919434
80th percentile: 5.681100797653201
90th percentile: 12.905085420608522
95th percentile: 16.51707773208618
99th percentile: 19.40667158126831
mean time: 5.516749668121338
%s, retrying in %s seconds...
Received healthy response to inference request in 1.4836652278900146s
Received healthy response to inference request in 1.6800246238708496s
Received healthy response to inference request in 1.749607801437378s
Received healthy response to inference request in 1.9474294185638428s
Received healthy response to inference request in 1.8838555812835693s
5 requests
0 failed requests
5th percentile: 1.5229371070861817
10th percentile: 1.5622089862823487
20th percentile: 1.6407527446746826
30th percentile: 1.6939412593841552
40th percentile: 1.7217745304107666
50th percentile: 1.749607801437378
60th percentile: 1.8033069133758546
70th percentile: 1.857006025314331
80th percentile: 1.896570348739624
90th percentile: 1.9219998836517334
95th percentile: 1.9347146511077882
99th percentile: 1.9448864650726319
mean time: 1.7489165306091308
Pipeline stage StressChecker completed in 38.58s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyTriggerPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage OfflineFamilyFriendlyTriggerPipeline completed in 0.88s
run pipeline stage %s
Running pipeline stage TriggerMKMLProfilingPipeline
run_pipeline:run_in_cloud %s
starting trigger_guanaco_pipeline args=%s
triggered trigger_guanaco_pipeline args=%s
Pipeline stage TriggerMKMLProfilingPipeline completed in 0.72s
Shutdown handler de-registered
cycy233-modelv-chai-step9000_v1 status is now deployed due to DeploymentManager action
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeleter
Skipping teardown as no inference service was successfully deployed
Pipeline stage MKMLProfilerDeleter completed in 0.16s
run pipeline stage %s
Running pipeline stage MKMLProfilerTemplater
Pipeline stage MKMLProfilerTemplater completed in 0.12s
run pipeline stage %s
Running pipeline stage MKMLProfilerDeployer
Creating inference service cycy233-modelv-chai-step9000-v1-profiler
Waiting for inference service cycy233-modelv-chai-step9000-v1-profiler to be ready
Shutdown handler registered
run pipeline %s
run pipeline stage %s
Running pipeline stage OfflineFamilyFriendlyScorer
Evaluating %s Family Friendly Score with %s threads
Pipeline stage OfflineFamilyFriendlyScorer completed in 3011.41s
Shutdown handler de-registered
cycy233-modelv-chai-step9000_v1 status is now inactive due to auto deactivation removed underperforming models
cycy233-modelv-chai-step9000_v1 status is now torndown due to DeploymentManager action