Cholesky requires positive definite

[chintan-ushur]

Describe the bug
While implementing a recommended recipe to compress the LLM, facing an issue.

Expected behavior
Successful compression of the LLM

To Reproduce

Exact steps to reproduce the behavior:

from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier
from llmcompressor.transformers import oneshot


recipe = [
    SmoothQuantModifier(smoothing_strength=0.8),
    GPTQModifier(scheme="W8A8", targets="Linear", ignore=["lm_head"]),
]

oneshot(
    model=model,#"ushuradmin/usrs_extractions_16bit_llama_3_1_v1",
    dataset=load_custom_dataset(DATA_PATH),
    recipe=recipe,
    output_dir="usrs_extractions_Meta-Llama-3.1-8B-Instruct-INT8",
    max_seq_length=2048,
    num_calibration_samples=10,
    #device_map = "auto"
    #hf_token="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"
)

Errors

26, in GPTQWrapper.compress(self, blocksize, percdamp)
    124 diag = torch.arange(self.columns, device=self.dev)
    125 self.H[diag, diag] += damp
--> 126 self.H = torch.linalg.cholesky(self.H)
    127 self.H = torch.cholesky_inverse(self.H)
    128 self.H = torch.linalg.cholesky(self.H, upper=True)

_LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 3596 is not positive-definite).

[robertgshaw2-redhat]

This issue is due to numerical reasons, where the Hessian is unable to be inverted. You can set dampening_frac=0.1 in GPTQModifier to avoid this.

@dsikka - could someone write up an FAQ doc that covers what to do when seeing this error?

[eldarkurtic]

@chintan-ushur based on the command you shared above, I think your problem is the size of the calibration dataset:

num_calibration_samples=10,

Please increase this to a larger number, for example: 512, 1024, 2048, etc. If increasing num_calibration_samples still gives you this error, then please try @robertgshaw2-neuralmagic suggestion from above with larger dampening_frac=0.1.

[limertang]

@robertgshaw2-neuralmagic @eldarkurtic I set num_calibration_samples=512 or 1024, and dampening_frac=0.1,still give the error as below:
self.H = torch.linalg.cholesky(self.H, upper=True)
torch._C._LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 18941 is not positive-definite).

[eldarkurtic]

@limertang could you please share a reproducible script so we can try it as well?

[chintan-ushur]

In my case; dampening_frac=0.1 worked.

[limertang]

@eldarkurtic code as below:
`NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 512

model_path = '/work2/model/Qwen2-7B-Instruct'
quant_path = '/work/model/quantize_vllm_script/Qwen2-7B-Instruct-int8'

#Load model
model = SparseAutoModelForCausalLM.from_pretrained(
model_path, device_map="auto", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_path)

#load data
ds = Dataset.from_json("/work2/model/open_data_zh.json")
ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
def tokenize(sample):
return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
ds = ds.map(tokenize, remove_columns=ds.column_names)

#Quantize
#Configure the quantization algorithms
recipe = [
SmoothQuantModifier(smoothing_strength=0.8),
GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"], sequential_update=True, dampening_frac=0.1),
]
#Apply quantization
oneshot(
model=model,
dataset=ds,
recipe=recipe,
max_seq_length=MAX_SEQUENCE_LENGTH,
num_calibration_samples=NUM_CALIBRATION_SAMPLES,
output_dir=quant_path
)

`
The data in json file looks like this：
[
{
"text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n保持健康的三个提示。<|im_end|>\n<|im_start|>assistant\n"
},
{
"text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n三原色是什么？<|im_end|>\n<|im_start|>assistant\n"
},
{
"text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n描述原子的结构。<|im_end|>\n<|im_start|>assistant\n"
},
{
"text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n如何减少空气污染？<|im_end|>\n<|im_start|>assistant\n"
},
{
"text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n描述一次你不得不做出困难决定的经历。<|im_end|>\n<|im_start|>assistant\n"
},
{
"text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n辨认不同之处。 输入：Twitter，Instagram，Telegram。<|im_end|>\n<|im_start|>assistant\n"
},
{
"text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n解释为什么以下分数等同于1/4输入：4/16<|im_end|>\n<|im_start|>assistant\n"
},
...................................................................

[eldarkurtic]

Could you please share the entire calibration dataset so I can also try to run and reproduce the issue on my side?

Also one note: it seems that your calibration dataset doesn't have any "answers" after the assistant tag which is problematic. The calibration data should be a full data sample, with both, user and assistant parts of the conversation.

[limertang]

@eldarkurtic
Thank you at first.
I can't upload file due to company restrictions。
I templated the data as below， but errors still exist. Errors occurs at the same layer every time,
2024-09-14T08:40:21.675633+0000 | compress_module | INFO - Compressing model.layers.3.model.layers.3.mlp.down_proj...

{
"text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n保持健康的三个提示。<|im_end|>\n<|im_start|>assistant\n以下是保持>健康的三个提示：\n\n1. 保持身体活动。每天做适当的身体运动，如散步、跑步或游泳，能促进心血管健康，增强肌肉力量，并有助于减少体重。\n\n2. 均衡饮食。每天食用新鲜的>蔬菜、水果、全谷物和脂肪含量低的蛋白质食物，避免高糖、高脂肪和加工食品，以保持健康的饮食习惯。\n\n3. 睡眠充足。睡眠对人体健康至关重要，成年人每天应保证 7-8 小时>的睡眠。良好的睡眠有助于减轻压力，促进身体恢复，并提高注意力和记忆力。<|im_end|>\n"
},
{
"text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n三原色是什么？<|im_end|>\n<|im_start|>assistant\n三原色通常指的是>红色、绿色和蓝色（RGB）。它们是通过加色混合原理创建色彩的三种基础颜色。在以发光为基础的显示设备中（如电视、计算机显示器、智能手机和平板电脑显示屏）, 三原色可混合
产生大量色彩。其中红色和绿色可以混合生成黄色，红色和蓝色可以混合生成品红色，蓝色和绿色可以混合生成青色。当红色、绿色和蓝色按相等比例混合时，可以产生白色或灰色。\n\n此外，在印刷和绘画中，三原色指的是以颜料为基础的红、黄和蓝颜色（RYB）。这三种颜色用以通过减色混合原理来创建色彩。不过，三原色的具体定义并不唯一，不同的颜色系>统可能会采用不同的三原色。<|im_end|>\n"
},

[eldarkurtic]

Unfortunately, without being able to reproduce the setup, I can't provide more help beyond these few suggestions to try:

1. Use enough calibration data (≥ 512 samples; if possible, please try 2k, 3k, or 4k as well).
2. Once you have enough calibration data, try shuffling it so the samples are in a different order.
3. If steps 1 and 2 don't help, try gradually increasing dampening_frac. Be aware that this should be the last option, as increasing dampening_frac makes your GPTQ algorithms more similar to round-to-nearest quantization, which negatively impacts accuracy.

[okwinds]

I encountered the same issue:

torch._C._LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 17915 is not positive-definite).

1. Use enough calibration data (≥ 5k samples; model: Qwen2.5-7B-Instruct)
2. Shuffling data so the samples are in a different order
3. I adjusted the dampening_frac, which allowed me to complete the model quantization, but running the VLLM inference caused the WSL Ubuntu 22.04 to crash.

My solution was to switch back to the older version 0.1.0, which resolved the issue, although the quantization process is quite slow.

Unfortunately, without being able to reproduce the setup, I can't provide more help beyond these few suggestions to try:

1. Use enough calibration data (≥ 512 samples; if possible, please try 2k, 3k, or 4k as well).
2. Once you have enough calibration data, try shuffling it so the samples are in a different order.
3. If steps 1 and 2 don't help, try gradually increasing dampening_frac. Be aware that this should be the last option, as increasing dampening_frac makes your GPTQ algorithms more similar to round-to-nearest quantization, which negatively impacts accuracy.

[HelloCard]

try quant Qwen/Qwen2.5-32B-Instruct, same issues, my script:

from llmcompressor.transformers import SparseAutoModelForCausalLM
from transformers import AutoTokenizer
import torch
MODEL_ID = "/root/autodl-tmp/Qwen2.5-32B-Instruct"
model = SparseAutoModelForCausalLM.from_pretrained(
    MODEL_ID, device_map="auto", torch_dtype="auto",
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)





from datasets import load_dataset

NUM_CALIBRATION_SAMPLES = 2048
MAX_SEQUENCE_LENGTH = 8192

# Load and preprocess the dataset
ds = load_dataset("/root/autodl-tmp/ultrachat_200k", split="train_sft")
ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))

def preprocess(example):
    return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
ds = ds.map(preprocess)

def tokenize(sample):
    return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
ds = ds.map(tokenize, remove_columns=ds.column_names)






from llmcompressor.transformers import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier

# Configure the quantization algorithms
recipe = [
    SmoothQuantModifier(smoothing_strength=0.9),
    GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
]

# Apply quantization
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

# Save the compressed model
SAVE_DIR = MODEL_ID.split("/")[1] + "-W8A8-Dynamic-Per-Token"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)

llmcompressor==0.2.0:

2024-10-07T14:17:19.377770+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.mlp.down_proj...
Traceback (most recent call last):
  File "/root/autodl-tmp/quant.py", line 48, in <module>
    oneshot(
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 76, in oneshot
    main(model_args, data_args, training_args)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 364, in main
    stage_runner.one_shot()
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/transformers/finetune/runner.py", line 171, in one_shot
    self.trainer.one_shot(calibration_data=calib_data, stage=stage)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/transformers/finetune/session_mixin.py", line 401, in one_shot
    apply(
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/core/session_functions.py", line 184, in apply
    return active_session().apply(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/core/session.py", line 210, in apply
    self.initialize(**kwargs)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/core/session.py", line 156, in initialize
    mod_data = self._lifecycle.initialize(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/core/lifecycle.py", line 126, in initialize
    data = mod.initialize(state=self.state, **extras)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/stage.py", line 124, in initialize
    modifier.initialize(state, **kwargs)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/modifier.py", line 118, in initialize
    initialized = self.on_initialize(state=state, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 187, in on_initialize
    self.apply_compression(calibration_dataloader)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 292, in apply_compression
    layer_compressor.compress()
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/utils/layer_compressor.py", line 176, in compress
    self.layer.apply(compress_module)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 895, in apply
    module.apply(fn)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 895, in apply
    module.apply(fn)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 896, in apply
    fn(self)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/utils/layer_compressor.py", line 172, in compress_module
    module.compress(**self.args)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/utils/gptq_wrapper.py", line 166, in compress
    self.H = torch.linalg.cholesky(self.H)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch._C._LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 6236 is not positive-definite).

llmcompressor==0.2.0, NUM_CALIBRATION_SAMPLES = 4096 and dampening_frac=0.1:

Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17/17 [00:14<00:00,  1.15it/s]
Map: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4096/4096 [00:00<00:00, 6091.42 examples/s]
Map: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4096/4096 [00:10<00:00, 388.98 examples/s]
2024-10-07T14:18:20.792490+0800 | main | WARNING - Process rank: 0, device: cuda:0, n_gpu: 8, distributed training: True, 16-bits training: False
2024-10-07T14:18:20.793621+0800 | main | INFO - Training/evaluation parameters TrainingArguments(
_n_gpu=8,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=False,
bf16_full_eval=False,
clear_sparse_session=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=None,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_oneshot=True,
do_predict=False,
do_train=False,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=None,
eval_strategy=IntervalStrategy.NO,
eval_use_gather_object=False,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=HubStrategy.EVERY_SAVE,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=5e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=./output/runs/Oct07_14-18-20_autodl-container-02b446b599-47e89a5d,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=500,
logging_strategy=IntervalStrategy.STEPS,
lr_scheduler_kwargs={},
lr_scheduler_type=SchedulerType.LINEAR,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=3.0,
oneshot_device=cuda:0,
optim=OptimizerNames.ADAMW_TORCH,
optim_args=None,
optim_target_modules=None,
output_dir=./output,
overwrite_output_dir=False,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=8,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
recipe=[SmoothQuantModifier(index=None, group=None, start=None, end=None, update=None, initialized_structure_=False, initialized_=False, finalized_=False, started_=False, ended_=False, smoothing_strength=0.9, mappings=[[['re:.*q_proj', 're:.*k_proj', 're:.*v_proj'], 're:.*input_layernorm'], [['re:.*gate_proj', 're:.*up_proj'], 're:.*post_attention_layernorm']], ignore=None, num_calibration_steps=None, calibration_function=None, hooks_=None, resolved_mappings_=None, scales_=None), GPTQModifier(index=None, group=None, start=None, end=None, update=None, initialized_structure_=False, initialized_=False, finalized_=False, started_=False, ended_=False, sequential_update=True, targets='Linear', sequential_targets=None, block_size=128, quantize=True, dampening_frac=0.1, config_groups=None, ignore=['lm_head'], disable_quantization_observer_epoch=None, num_calibration_steps=None, scheme='W8A8', model=None, layer_compressors_=None, compressible_layers_=None, quantization_modifier_=None)],
recipe_args=None,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=./output,
run_stages=False,
save_compressed=True,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=IntervalStrategy.STEPS,
save_total_limit=None,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
)
2024-10-07T14:18:21.086676+0800 | _check_create_state | INFO - State created for compression lifecycle
2024-10-07T14:18:21.089376+0800 | pre_initialize_structure | INFO - Compression lifecycle structure pre-initialized for 0 modifiers
2024-10-07T14:18:21.090801+0800 | pre_initialize_structure | INFO - Compression lifecycle structure pre-initialized for 0 modifiers
2024-10-07T14:18:21.146974+0800 | one_shot | INFO - *** One Shot ***
2024-10-07T14:18:21.153343+0800 | from_modifiers | INFO - Creating recipe from modifiers
2024-10-07T14:18:21.180845+0800 | _check_compile_recipe | INFO - Recipe compiled and 1 modifiers created
2024-10-07T14:18:24.300367+0800 | _calibrate | INFO - Running SmoothQuantModifier calibration with 4096 samples...
  0%|                                                                                                                                                                                                | 0/4096 [00:00<?, ?it/s]Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4096/4096 [52:42<00:00,  1.30it/s]
2024-10-07T15:11:07.116400+0800 | _apply_smoothing | INFO - Smoothing activation scales...
2024-10-07T15:11:07.430726+0800 | on_initialize_structure | WARNING - GPTQ quantization is set to True without an active quantization modifier.
2024-10-07T15:11:07.430868+0800 | _build_quant_modifier | INFO - Building quantization modifier with args: {'targets': 'Linear', 'scheme': 'W8A8', 'ignore': ['lm_head']}
2024-10-07T15:11:07.476983+0800 | _check_calibration_data | INFO - Skipping QuantizationModifier calibration, it is not required for the provided quantization config.
2024-10-07T15:11:07.689353+0800 | initialize_compression | INFO - Preparing model.layers.0 for compression
2024-10-07T15:11:07.689478+0800 | initialize_compression | INFO - Preparing model.layers.1 for compression
2024-10-07T15:11:07.689516+0800 | initialize_compression | INFO - Preparing model.layers.2 for compression
2024-10-07T15:11:07.689545+0800 | initialize_compression | INFO - Preparing model.layers.3 for compression
2024-10-07T15:11:07.689568+0800 | initialize_compression | INFO - Preparing model.layers.4 for compression
2024-10-07T15:11:07.689589+0800 | initialize_compression | INFO - Preparing model.layers.5 for compression
2024-10-07T15:11:07.689609+0800 | initialize_compression | INFO - Preparing model.layers.6 for compression
2024-10-07T15:11:07.689628+0800 | initialize_compression | INFO - Preparing model.layers.7 for compression
2024-10-07T15:11:07.689646+0800 | initialize_compression | INFO - Preparing model.layers.8 for compression
2024-10-07T15:11:07.689663+0800 | initialize_compression | INFO - Preparing model.layers.9 for compression
2024-10-07T15:11:07.689682+0800 | initialize_compression | INFO - Preparing model.layers.10 for compression
2024-10-07T15:11:07.689705+0800 | initialize_compression | INFO - Preparing model.layers.11 for compression
2024-10-07T15:11:07.689724+0800 | initialize_compression | INFO - Preparing model.layers.12 for compression
2024-10-07T15:11:07.689745+0800 | initialize_compression | INFO - Preparing model.layers.13 for compression
2024-10-07T15:11:07.689768+0800 | initialize_compression | INFO - Preparing model.layers.14 for compression
2024-10-07T15:11:07.689787+0800 | initialize_compression | INFO - Preparing model.layers.15 for compression
2024-10-07T15:11:07.689805+0800 | initialize_compression | INFO - Preparing model.layers.16 for compression
2024-10-07T15:11:07.689823+0800 | initialize_compression | INFO - Preparing model.layers.17 for compression
2024-10-07T15:11:07.689840+0800 | initialize_compression | INFO - Preparing model.layers.18 for compression
2024-10-07T15:11:07.689859+0800 | initialize_compression | INFO - Preparing model.layers.19 for compression
2024-10-07T15:11:07.689877+0800 | initialize_compression | INFO - Preparing model.layers.20 for compression
2024-10-07T15:11:07.689894+0800 | initialize_compression | INFO - Preparing model.layers.21 for compression
2024-10-07T15:11:07.689910+0800 | initialize_compression | INFO - Preparing model.layers.22 for compression
2024-10-07T15:11:07.689927+0800 | initialize_compression | INFO - Preparing model.layers.23 for compression
2024-10-07T15:11:07.689945+0800 | initialize_compression | INFO - Preparing model.layers.24 for compression
2024-10-07T15:11:07.689962+0800 | initialize_compression | INFO - Preparing model.layers.25 for compression
2024-10-07T15:11:07.689978+0800 | initialize_compression | INFO - Preparing model.layers.26 for compression
2024-10-07T15:11:07.689994+0800 | initialize_compression | INFO - Preparing model.layers.27 for compression
2024-10-07T15:11:07.690010+0800 | initialize_compression | INFO - Preparing model.layers.28 for compression
2024-10-07T15:11:07.690027+0800 | initialize_compression | INFO - Preparing model.layers.29 for compression
2024-10-07T15:11:07.690043+0800 | initialize_compression | INFO - Preparing model.layers.30 for compression
2024-10-07T15:11:07.690058+0800 | initialize_compression | INFO - Preparing model.layers.31 for compression
2024-10-07T15:11:07.690076+0800 | initialize_compression | INFO - Preparing model.layers.32 for compression
2024-10-07T15:11:07.690092+0800 | initialize_compression | INFO - Preparing model.layers.33 for compression
2024-10-07T15:11:07.690109+0800 | initialize_compression | INFO - Preparing model.layers.34 for compression
2024-10-07T15:11:07.690126+0800 | initialize_compression | INFO - Preparing model.layers.35 for compression
2024-10-07T15:11:07.690142+0800 | initialize_compression | INFO - Preparing model.layers.36 for compression
2024-10-07T15:11:07.690158+0800 | initialize_compression | INFO - Preparing model.layers.37 for compression
2024-10-07T15:11:07.690175+0800 | initialize_compression | INFO - Preparing model.layers.38 for compression
2024-10-07T15:11:07.690192+0800 | initialize_compression | INFO - Preparing model.layers.39 for compression
2024-10-07T15:11:07.690212+0800 | initialize_compression | INFO - Preparing model.layers.40 for compression
2024-10-07T15:11:07.690229+0800 | initialize_compression | INFO - Preparing model.layers.41 for compression
2024-10-07T15:11:07.690246+0800 | initialize_compression | INFO - Preparing model.layers.42 for compression
2024-10-07T15:11:07.690264+0800 | initialize_compression | INFO - Preparing model.layers.43 for compression
2024-10-07T15:11:07.690283+0800 | initialize_compression | INFO - Preparing model.layers.44 for compression
2024-10-07T15:11:07.690303+0800 | initialize_compression | INFO - Preparing model.layers.45 for compression
2024-10-07T15:11:07.690321+0800 | initialize_compression | INFO - Preparing model.layers.46 for compression
2024-10-07T15:11:07.690339+0800 | initialize_compression | INFO - Preparing model.layers.47 for compression
2024-10-07T15:11:07.690356+0800 | initialize_compression | INFO - Preparing model.layers.48 for compression
2024-10-07T15:11:07.690372+0800 | initialize_compression | INFO - Preparing model.layers.49 for compression
2024-10-07T15:11:07.690388+0800 | initialize_compression | INFO - Preparing model.layers.50 for compression
2024-10-07T15:11:07.690404+0800 | initialize_compression | INFO - Preparing model.layers.51 for compression
2024-10-07T15:11:07.690420+0800 | initialize_compression | INFO - Preparing model.layers.52 for compression
2024-10-07T15:11:07.690435+0800 | initialize_compression | INFO - Preparing model.layers.53 for compression
2024-10-07T15:11:07.690451+0800 | initialize_compression | INFO - Preparing model.layers.54 for compression
2024-10-07T15:11:07.690469+0800 | initialize_compression | INFO - Preparing model.layers.55 for compression
2024-10-07T15:11:07.690486+0800 | initialize_compression | INFO - Preparing model.layers.56 for compression
2024-10-07T15:11:07.690502+0800 | initialize_compression | INFO - Preparing model.layers.57 for compression
2024-10-07T15:11:07.690519+0800 | initialize_compression | INFO - Preparing model.layers.58 for compression
2024-10-07T15:11:07.690534+0800 | initialize_compression | INFO - Preparing model.layers.59 for compression
2024-10-07T15:11:07.690550+0800 | initialize_compression | INFO - Preparing model.layers.60 for compression
2024-10-07T15:11:07.690566+0800 | initialize_compression | INFO - Preparing model.layers.61 for compression
2024-10-07T15:11:07.690582+0800 | initialize_compression | INFO - Preparing model.layers.62 for compression
2024-10-07T15:11:07.690598+0800 | initialize_compression | INFO - Preparing model.layers.63 for compression
2024-10-07T15:11:07.690682+0800 | apply_compression | INFO - Running GPTQModifier calibration with 4096 samples...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4096/4096 [00:38<00:00, 107.53it/s]
2024-10-07T15:11:45.789303+0800 | apply_compression | INFO - 
===== Compressing layer 1/64  =====
2024-10-07T15:11:45.793833+0800 | apply_compression | INFO - Calibrating model.layers.0...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4096/4096 [07:17<00:00,  9.35it/s]
2024-10-07T15:19:03.788689+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.self_attn.q_proj...
2024-10-07T15:19:05.308717+0800 | compress | METRIC - time 1.52
2024-10-07T15:19:05.309861+0800 | compress | METRIC - error 64.10
2024-10-07T15:19:05.310468+0800 | compress | METRIC - GPU 0 | usage: 48.09% | total memory: 23 GB
2024-10-07T15:19:05.310517+0800 | compress | METRIC - GPU 1 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:05.310549+0800 | compress | METRIC - GPU 2 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:05.310574+0800 | compress | METRIC - GPU 3 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:05.310596+0800 | compress | METRIC - GPU 4 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:05.310616+0800 | compress | METRIC - GPU 5 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:05.310634+0800 | compress | METRIC - GPU 6 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:05.310652+0800 | compress | METRIC - GPU 7 | usage: 34.68% | total memory: 23 GB
2024-10-07T15:19:05.310719+0800 | compress | METRIC - Compressed layer size: 50.0244140625 MB
done
2024-10-07T15:19:05.310967+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.self_attn.k_proj...
2024-10-07T15:19:06.289511+0800 | compress | METRIC - time 0.98
2024-10-07T15:19:06.289880+0800 | compress | METRIC - error 19.46
2024-10-07T15:19:06.290121+0800 | compress | METRIC - GPU 0 | usage: 48.10% | total memory: 23 GB
2024-10-07T15:19:06.290168+0800 | compress | METRIC - GPU 1 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:06.290200+0800 | compress | METRIC - GPU 2 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:06.290230+0800 | compress | METRIC - GPU 3 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:06.290256+0800 | compress | METRIC - GPU 4 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:06.290277+0800 | compress | METRIC - GPU 5 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:06.290296+0800 | compress | METRIC - GPU 6 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:06.290316+0800 | compress | METRIC - GPU 7 | usage: 34.68% | total memory: 23 GB
2024-10-07T15:19:06.290379+0800 | compress | METRIC - Compressed layer size: 10.0048828125 MB
done
2024-10-07T15:19:06.290584+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.self_attn.v_proj...
2024-10-07T15:19:07.249296+0800 | compress | METRIC - time 0.96
2024-10-07T15:19:07.250220+0800 | compress | METRIC - error 2.36
2024-10-07T15:19:07.250470+0800 | compress | METRIC - GPU 0 | usage: 48.10% | total memory: 23 GB
2024-10-07T15:19:07.250512+0800 | compress | METRIC - GPU 1 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:07.250540+0800 | compress | METRIC - GPU 2 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:07.250566+0800 | compress | METRIC - GPU 3 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:07.250591+0800 | compress | METRIC - GPU 4 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:07.250612+0800 | compress | METRIC - GPU 5 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:07.250629+0800 | compress | METRIC - GPU 6 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:07.250646+0800 | compress | METRIC - GPU 7 | usage: 34.68% | total memory: 23 GB
2024-10-07T15:19:07.250703+0800 | compress | METRIC - Compressed layer size: 10.0048828125 MB
done
2024-10-07T15:19:07.250900+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.self_attn.o_proj...
2024-10-07T15:19:08.266205+0800 | compress | METRIC - time 1.02
2024-10-07T15:19:08.267269+0800 | compress | METRIC - error 15.49
2024-10-07T15:19:08.267527+0800 | compress | METRIC - GPU 0 | usage: 48.10% | total memory: 23 GB
2024-10-07T15:19:08.267564+0800 | compress | METRIC - GPU 1 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:08.267591+0800 | compress | METRIC - GPU 2 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:08.267614+0800 | compress | METRIC - GPU 3 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:08.267633+0800 | compress | METRIC - GPU 4 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:08.267652+0800 | compress | METRIC - GPU 5 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:08.267669+0800 | compress | METRIC - GPU 6 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:08.267684+0800 | compress | METRIC - GPU 7 | usage: 34.68% | total memory: 23 GB
2024-10-07T15:19:08.267736+0800 | compress | METRIC - Compressed layer size: 50.0146484375 MB
done
2024-10-07T15:19:08.267962+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.mlp.gate_proj...
2024-10-07T15:19:09.856850+0800 | compress | METRIC - time 1.59
2024-10-07T15:19:09.857920+0800 | compress | METRIC - error 49.78
2024-10-07T15:19:09.858180+0800 | compress | METRIC - GPU 0 | usage: 55.64% | total memory: 23 GB
2024-10-07T15:19:09.858225+0800 | compress | METRIC - GPU 1 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:09.858250+0800 | compress | METRIC - GPU 2 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:09.858273+0800 | compress | METRIC - GPU 3 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:09.858303+0800 | compress | METRIC - GPU 4 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:09.858322+0800 | compress | METRIC - GPU 5 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:09.858341+0800 | compress | METRIC - GPU 6 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:09.858366+0800 | compress | METRIC - GPU 7 | usage: 34.68% | total memory: 23 GB
2024-10-07T15:19:09.858423+0800 | compress | METRIC - Compressed layer size: 270.0791015625 MB
done
2024-10-07T15:19:09.858624+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.mlp.up_proj...
2024-10-07T15:19:11.442625+0800 | compress | METRIC - time 1.58
2024-10-07T15:19:11.443849+0800 | compress | METRIC - error 32.30
2024-10-07T15:19:11.444119+0800 | compress | METRIC - GPU 0 | usage: 55.64% | total memory: 23 GB
2024-10-07T15:19:11.444162+0800 | compress | METRIC - GPU 1 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:11.444193+0800 | compress | METRIC - GPU 2 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:11.444215+0800 | compress | METRIC - GPU 3 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:11.444235+0800 | compress | METRIC - GPU 4 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:11.444255+0800 | compress | METRIC - GPU 5 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:11.444274+0800 | compress | METRIC - GPU 6 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:11.444292+0800 | compress | METRIC - GPU 7 | usage: 34.68% | total memory: 23 GB
2024-10-07T15:19:11.444349+0800 | compress | METRIC - Compressed layer size: 270.0791015625 MB
done
2024-10-07T15:19:11.444567+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.mlp.down_proj...
Traceback (most recent call last):
  File "/root/autodl-tmp/quant.py", line 48, in <module>
    oneshot(
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 76, in oneshot
    main(model_args, data_args, training_args)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 364, in main
    stage_runner.one_shot()
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/transformers/finetune/runner.py", line 171, in one_shot
    self.trainer.one_shot(calibration_data=calib_data, stage=stage)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/transformers/finetune/session_mixin.py", line 401, in one_shot
    apply(
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/core/session_functions.py", line 184, in apply
    return active_session().apply(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/core/session.py", line 210, in apply
    self.initialize(**kwargs)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/core/session.py", line 156, in initialize
    mod_data = self._lifecycle.initialize(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/core/lifecycle.py", line 126, in initialize
    data = mod.initialize(state=self.state, **extras)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/stage.py", line 124, in initialize
    modifier.initialize(state, **kwargs)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/modifier.py", line 118, in initialize
    initialized = self.on_initialize(state=state, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 187, in on_initialize
    self.apply_compression(calibration_dataloader)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 292, in apply_compression
    layer_compressor.compress()
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/utils/layer_compressor.py", line 176, in compress
    self.layer.apply(compress_module)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 895, in apply
    module.apply(fn)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 895, in apply
    module.apply(fn)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 896, in apply
    fn(self)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/utils/layer_compressor.py", line 172, in compress_module
    module.compress(**self.args)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/utils/gptq_wrapper.py", line 166, in compress
    self.H = torch.linalg.cholesky(self.H)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch._C._LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 5838 is not positive-definite).
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4096/4096 [07:17<00:00,  9.35it/s]
2024-10-07T15:19:03.788689+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.self_attn.q_proj...
2024-10-07T15:19:05.308717+0800 | compress | METRIC - time 1.52
2024-10-07T15:19:05.309861+0800 | compress | METRIC - error 64.10
2024-10-07T15:19:05.310468+0800 | compress | METRIC - GPU 0 | usage: 48.09% | total memory: 23 GB
2024-10-07T15:19:05.310517+0800 | compress | METRIC - GPU 1 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:05.310549+0800 | compress | METRIC - GPU 2 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:05.310574+0800 | compress | METRIC - GPU 3 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:05.310596+0800 | compress | METRIC - GPU 4 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:05.310616+0800 | compress | METRIC - GPU 5 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:05.310634+0800 | compress | METRIC - GPU 6 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:05.310652+0800 | compress | METRIC - GPU 7 | usage: 34.68% | total memory: 23 GB
2024-10-07T15:19:05.310719+0800 | compress | METRIC - Compressed layer size: 50.0244140625 MB
done
2024-10-07T15:19:05.310967+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.self_attn.k_proj...
2024-10-07T15:19:06.289511+0800 | compress | METRIC - time 0.98
2024-10-07T15:19:06.289880+0800 | compress | METRIC - error 19.46
2024-10-07T15:19:06.290121+0800 | compress | METRIC - GPU 0 | usage: 48.10% | total memory: 23 GB
2024-10-07T15:19:06.290168+0800 | compress | METRIC - GPU 1 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:06.290200+0800 | compress | METRIC - GPU 2 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:06.290230+0800 | compress | METRIC - GPU 3 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:06.290256+0800 | compress | METRIC - GPU 4 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:06.290277+0800 | compress | METRIC - GPU 5 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:06.290296+0800 | compress | METRIC - GPU 6 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:06.290316+0800 | compress | METRIC - GPU 7 | usage: 34.68% | total memory: 23 GB
2024-10-07T15:19:06.290379+0800 | compress | METRIC - Compressed layer size: 10.0048828125 MB
done
2024-10-07T15:19:06.290584+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.self_attn.v_proj...
2024-10-07T15:19:07.249296+0800 | compress | METRIC - time 0.96
2024-10-07T15:19:07.250220+0800 | compress | METRIC - error 2.36
2024-10-07T15:19:07.250470+0800 | compress | METRIC - GPU 0 | usage: 48.10% | total memory: 23 GB
2024-10-07T15:19:07.250512+0800 | compress | METRIC - GPU 1 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:07.250540+0800 | compress | METRIC - GPU 2 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:07.250566+0800 | compress | METRIC - GPU 3 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:07.250591+0800 | compress | METRIC - GPU 4 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:07.250612+0800 | compress | METRIC - GPU 5 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:07.250629+0800 | compress | METRIC - GPU 6 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:07.250646+0800 | compress | METRIC - GPU 7 | usage: 34.68% | total memory: 23 GB
2024-10-07T15:19:07.250703+0800 | compress | METRIC - Compressed layer size: 10.0048828125 MB
done
2024-10-07T15:19:07.250900+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.self_attn.o_proj...
2024-10-07T15:19:08.266205+0800 | compress | METRIC - time 1.02
2024-10-07T15:19:08.267269+0800 | compress | METRIC - error 15.49
2024-10-07T15:19:08.267527+0800 | compress | METRIC - GPU 0 | usage: 48.10% | total memory: 23 GB
2024-10-07T15:19:08.267564+0800 | compress | METRIC - GPU 1 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:08.267591+0800 | compress | METRIC - GPU 2 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:08.267614+0800 | compress | METRIC - GPU 3 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:08.267633+0800 | compress | METRIC - GPU 4 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:08.267652+0800 | compress | METRIC - GPU 5 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:08.267669+0800 | compress | METRIC - GPU 6 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:08.267684+0800 | compress | METRIC - GPU 7 | usage: 34.68% | total memory: 23 GB
2024-10-07T15:19:08.267736+0800 | compress | METRIC - Compressed layer size: 50.0146484375 MB
done
2024-10-07T15:19:08.267962+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.mlp.gate_proj...
2024-10-07T15:19:09.856850+0800 | compress | METRIC - time 1.59
2024-10-07T15:19:09.857920+0800 | compress | METRIC - error 49.78
2024-10-07T15:19:09.858180+0800 | compress | METRIC - GPU 0 | usage: 55.64% | total memory: 23 GB
2024-10-07T15:19:09.858225+0800 | compress | METRIC - GPU 1 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:09.858250+0800 | compress | METRIC - GPU 2 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:09.858273+0800 | compress | METRIC - GPU 3 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:09.858303+0800 | compress | METRIC - GPU 4 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:09.858322+0800 | compress | METRIC - GPU 5 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:09.858341+0800 | compress | METRIC - GPU 6 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:09.858366+0800 | compress | METRIC - GPU 7 | usage: 34.68% | total memory: 23 GB
2024-10-07T15:19:09.858423+0800 | compress | METRIC - Compressed layer size: 270.0791015625 MB
done
2024-10-07T15:19:09.858624+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.mlp.up_proj...
2024-10-07T15:19:11.442625+0800 | compress | METRIC - time 1.58
2024-10-07T15:19:11.443849+0800 | compress | METRIC - error 32.30
2024-10-07T15:19:11.444119+0800 | compress | METRIC - GPU 0 | usage: 55.64% | total memory: 23 GB
2024-10-07T15:19:11.444162+0800 | compress | METRIC - GPU 1 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:11.444193+0800 | compress | METRIC - GPU 2 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:11.444215+0800 | compress | METRIC - GPU 3 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:11.444235+0800 | compress | METRIC - GPU 4 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:11.444255+0800 | compress | METRIC - GPU 5 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:11.444274+0800 | compress | METRIC - GPU 6 | usage: 37.36% | total memory: 23 GB
2024-10-07T15:19:11.444292+0800 | compress | METRIC - GPU 7 | usage: 34.68% | total memory: 23 GB
2024-10-07T15:19:11.444349+0800 | compress | METRIC - Compressed layer size: 270.0791015625 MB
done
2024-10-07T15:19:11.444567+0800 | compress_module | INFO - Compressing model.layers.0.model.layers.0.mlp.down_proj...
Traceback (most recent call last):
  File "/root/autodl-tmp/quant.py", line 48, in <module>
    oneshot(
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 76, in oneshot
    main(model_args, data_args, training_args)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 364, in main
    stage_runner.one_shot()
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/transformers/finetune/runner.py", line 171, in one_shot
    self.trainer.one_shot(calibration_data=calib_data, stage=stage)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/transformers/finetune/session_mixin.py", line 401, in one_shot
    apply(
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/core/session_functions.py", line 184, in apply
    return active_session().apply(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/core/session.py", line 210, in apply
    self.initialize(**kwargs)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/core/session.py", line 156, in initialize
    mod_data = self._lifecycle.initialize(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/core/lifecycle.py", line 126, in initialize
    data = mod.initialize(state=self.state, **extras)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/stage.py", line 124, in initialize
    modifier.initialize(state, **kwargs)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/modifier.py", line 118, in initialize
    initialized = self.on_initialize(state=state, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 187, in on_initialize
    self.apply_compression(calibration_dataloader)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 292, in apply_compression
    layer_compressor.compress()
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/utils/layer_compressor.py", line 176, in compress
    self.layer.apply(compress_module)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 895, in apply
    module.apply(fn)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 895, in apply
    module.apply(fn)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 896, in apply
    fn(self)
  File "/root/miniconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/utils/layer_compressor.py", line 172, in compress_module
    module.compress(**self.args)
  File "/root/miniconda3/lib/python3.12/site-packages/llmcompressor/modifiers/quantization/gptq/utils/gptq_wrapper.py", line 166, in compress
    self.H = torch.linalg.cholesky(self.H)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch._C._LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 5838 is not positive-definite).

[HelloCard]

llmcompressor==0.1.0, NUM_CALIBRATION_SAMPLES = 4096 and dampening_frac=0.1:
(The second attempt set: export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True)

CPU ：128 核心
内存：640 GB
GPU ：NVIDIA GeForce RTX 4090 D, 8
存储：
  系 统 盘/               ：61% 19G/30G
  数 据 盘/root/autodl-tmp：56% 84G/150G
+----------------------------------------------------------------------------------------------------------------+
*注意:
1.系统盘较小请将大的数据存放于数据盘或网盘中，重置系统时数据盘和网盘中的数据不受影响
2.清理系统盘请参考：https://www.autodl.com/docs/qa1/
root@autodl-container-02b446b599-47e89a5d:~# ls
autodl-pub  autodl-tmp  miniconda3  tf-logs  transformers
root@autodl-container-02b446b599-47e89a5d:~# cd  autodl-tmp
root@autodl-container-02b446b599-47e89a5d:~/autodl-tmp# python3 quant.py
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████| 17/17 [00:13<00:00,  1.27it/s]2024-10-07T16:01:48.488182+0800 | main | WARNING - Process rank: 0, device: cuda:0, n_gpu: 8, distributed training: True, 16-bits training: False
2024-10-07T16:01:48.489483+0800 | main | INFO - Training/evaluation parameters TrainingArguments(
_n_gpu=8,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=False,
bf16_full_eval=False,
clear_sparse_session=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=None,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_oneshot=True,
do_predict=False,
do_train=False,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=None,
eval_strategy=IntervalStrategy.NO,
eval_use_gather_object=False,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=HubStrategy.EVERY_SAVE,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=5e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=./output/runs/Oct07_16-01-48_autodl-container-02b446b599-47e89a5d,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=500,
logging_strategy=IntervalStrategy.STEPS,
lr_scheduler_kwargs={},
lr_scheduler_type=SchedulerType.LINEAR,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=3.0,
oneshot_device=cuda:0,
optim=OptimizerNames.ADAMW_TORCH,
optim_args=None,
optim_target_modules=None,
output_dir=./output,
overwrite_output_dir=False,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=8,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
recipe=[SmoothQuantModifier(index=None, group=None, start=None, end=None, update=None, initialized_structure_=False, initialized_=False, finalized_=False, started_=False, ended_=False, smoothing_strength=0.9, mappings=[[['re:.*q_proj', 're:.*k_proj', 're:.*v_proj'], 're:.*input_layernorm'], [['re:.*gate_proj', 're:.*up_proj'], 're:.*post_attention_layernorm']], ignore=None, num_calibration_steps=None, calibration_function=None, hooks_=None, resolved_mappings_=None, scales_=None), GPTQModifier(index=None, group=None, start=None, end=None, update=None, initialized_structure_=False, initialized_=False, finalized_=False, started_=False, ended_=False, sequential_update=False, targets='Linear', sequential_targets=None, block_size=128, quantize=True, dampening_frac=0.01, config_groups=None, ignore=['lm_head'], disable_quantization_observer_epoch=None, num_calibration_steps=None, scheme='W8A8', model=None, layer_compressors_=None, compressible_layers_=None, quantization_modifier_=None)],
recipe_args=None,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=./output,
run_stages=False,
save_compressed=True,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=IntervalStrategy.STEPS,
save_total_limit=None,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
)
2024-10-07T16:01:48.775734+0800 | _check_create_state | INFO - State created for compression lifecycle
2024-10-07T16:01:48.776210+0800 | pre_initialize_structure | INFO - Compression lifecycle structure pre-initialized for 0 modifiers
2024-10-07T16:01:48.776324+0800 | pre_initialize_structure | INFO - Compression lifecycle structure pre-initialized for 0 modifiers
2024-10-07T16:01:48.815432+0800 | one_shot | INFO - *** One Shot ***
2024-10-07T16:01:48.818487+0800 | from_modifiers | INFO - Creating recipe from modifiers
2024-10-07T16:01:48.819153+0800 | create_instance | WARNING - Could not process input as a file path or zoo stub, attempting to process it as a string.
2024-10-07T16:01:48.847950+0800 | _check_compile_recipe | INFO - Recipe compiled and 1 modifiers created
2024-10-07T16:01:52.397242+0800 | _calibrate | INFO - Running SmoothQuantModifier calibration with 2048 samples...
  0%|                                                                                          | 0/2048 [00:00<?, ?it/s]Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)
100%|███████████████████████████████████████████████████████████████████████████████| 2048/2048 [26:20<00:00,  1.30it/s]2024-10-07T16:28:13.200329+0800 | _apply_smoothing | INFO - Smoothing activation scales...
2024-10-07T16:28:13.495101+0800 | on_initialize_structure | WARNING - GPTQ quantization is set to True without an active quantization modifier.
2024-10-07T16:28:13.495246+0800 | _build_quant_modifier | INFO - Building quantization modifier with args: {'targets': 'Linear', 'scheme': 'W8A8', 'ignore': ['lm_head']}
2024-10-07T16:28:13.548303+0800 | _check_calibration_data | INFO - Skipping QuantizationModifier calibration, it is not required for the provided quantization config.
2024-10-07T16:28:13.765202+0800 | initialize_compression | INFO - Preparing model.layers.0 for compression
2024-10-07T16:28:13.769142+0800 | initialize_compression | INFO - Preparing model.layers.1 for compression
2024-10-07T16:28:13.773317+0800 | initialize_compression | INFO - Preparing model.layers.2 for compression
2024-10-07T16:28:13.777429+0800 | initialize_compression | INFO - Preparing model.layers.3 for compression
2024-10-07T16:28:13.781667+0800 | initialize_compression | INFO - Preparing model.layers.4 for compression
Traceback (most recent call last):
  File "/root/autodl-tmp/quant.py", line 48, in <module>
    oneshot(
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 76, in oneshot
    main(model_args, data_args, training_args)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 359, in main
    stage_runner.one_shot()
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/transformers/finetune/runner.py", line 171, in one_shot
    self.trainer.one_shot(calibration_data=calib_data, stage=stage)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/transformers/finetune/session_mixin.py", line 401, in one_shot
    apply(
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/core/session_functions.py", line 184, in apply
    return active_session().apply(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/core/session.py", line 210, in apply
    self.initialize(**kwargs)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/core/session.py", line 156, in initialize
    mod_data = self._lifecycle.initialize(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/core/lifecycle.py", line 126, in initialize
    data = mod.initialize(state=self.state, **extras)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/stage.py", line 124, in initialize
    modifier.initialize(state, **kwargs)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/modifier.py", line 118, in initialize
    initialized = self.on_initialize(state=state, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 187, in on_initialize
    self.initialize_compression(modifiable_model, calibration_dataloader)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 246, in initialize_compression
    compressor.pre_compress()
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/utils/layer_compressor.py", line 79, in pre_compress
    wrapper = self.module_compressor_class(full_name, layer)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/quantization/gptq/utils/gptq_wrapper.py", line 45, in __init__
    "H", torch.zeros((self.columns, self.columns), device=self.dev)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.85 GiB. GPU 0 has a total capacity of 23.64 GiB of which 1.87 GiB is free. Process 856169 has 21.76 GiB memory in use. Of the allocated memory 21.23 GiB is allocated by PyTorch, and 126.51 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
root@autodl-container-02b446b599-47e89a5d:~/autodl-tmp# export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
root@autodl-container-02b446b599-47e89a5d:~/autodl-tmp# python3 quant.py
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17/17 [00:13<00:00,  1.22it/s]
2024-10-07T16:37:59.133060+0800 | main | WARNING - Process rank: 0, device: cuda:0, n_gpu: 8, distributed training: True, 16-bits training: False
2024-10-07T16:37:59.134379+0800 | main | INFO - Training/evaluation parameters TrainingArguments(
_n_gpu=8,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=False,
bf16_full_eval=False,
clear_sparse_session=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=None,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_oneshot=True,
do_predict=False,
do_train=False,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=None,
eval_strategy=IntervalStrategy.NO,
eval_use_gather_object=False,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=HubStrategy.EVERY_SAVE,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=5e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=./output/runs/Oct07_16-37-58_autodl-container-02b446b599-47e89a5d,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=500,
logging_strategy=IntervalStrategy.STEPS,
lr_scheduler_kwargs={},
lr_scheduler_type=SchedulerType.LINEAR,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=3.0,
oneshot_device=cuda:0,
optim=OptimizerNames.ADAMW_TORCH,
optim_args=None,
optim_target_modules=None,
output_dir=./output,
overwrite_output_dir=False,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=8,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
recipe=[SmoothQuantModifier(index=None, group=None, start=None, end=None, update=None, initialized_structure_=False, initialized_=False, finalized_=False, started_=False, ended_=False, smoothing_strength=0.9, mappings=[[['re:.*q_proj', 're:.*k_proj', 're:.*v_proj'], 're:.*input_layernorm'], [['re:.*gate_proj', 're:.*up_proj'], 're:.*post_attention_layernorm']], ignore=None, num_calibration_steps=None, calibration_function=None, hooks_=None, resolved_mappings_=None, scales_=None), GPTQModifier(index=None, group=None, start=None, end=None, update=None, initialized_structure_=False, initialized_=False, finalized_=False, started_=False, ended_=False, sequential_update=False, targets='Linear', sequential_targets=None, block_size=128, quantize=True, dampening_frac=0.01, config_groups=None, ignore=['lm_head'], disable_quantization_observer_epoch=None, num_calibration_steps=None, scheme='W8A8', model=None, layer_compressors_=None, compressible_layers_=None, quantization_modifier_=None)],
recipe_args=None,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=./output,
run_stages=False,
save_compressed=True,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=IntervalStrategy.STEPS,
save_total_limit=None,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
)
2024-10-07T16:37:59.407001+0800 | _check_create_state | INFO - State created for compression lifecycle
2024-10-07T16:37:59.407489+0800 | pre_initialize_structure | INFO - Compression lifecycle structure pre-initialized for 0 modifiers
2024-10-07T16:37:59.407604+0800 | pre_initialize_structure | INFO - Compression lifecycle structure pre-initialized for 0 modifiers
2024-10-07T16:37:59.444954+0800 | one_shot | INFO - *** One Shot ***
2024-10-07T16:37:59.448012+0800 | from_modifiers | INFO - Creating recipe from modifiers
2024-10-07T16:37:59.448688+0800 | create_instance | WARNING - Could not process input as a file path or zoo stub, attempting to process it as a string.
2024-10-07T16:37:59.474336+0800 | _check_compile_recipe | INFO - Recipe compiled and 1 modifiers created
2024-10-07T16:38:03.036247+0800 | _calibrate | INFO - Running SmoothQuantModifier calibration with 2048 samples...
  0%|                                                                                                                                                                                                                                                 | 0/2048 [00:00<?, ?it/s]Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2048/2048 [27:17<00:00,  1.25it/s]
2024-10-07T17:05:20.101499+0800 | _apply_smoothing | INFO - Smoothing activation scales...
2024-10-07T17:05:20.404871+0800 | on_initialize_structure | WARNING - GPTQ quantization is set to True without an active quantization modifier.
2024-10-07T17:05:20.405012+0800 | _build_quant_modifier | INFO - Building quantization modifier with args: {'targets': 'Linear', 'scheme': 'W8A8', 'ignore': ['lm_head']}
2024-10-07T17:05:20.455597+0800 | _check_calibration_data | INFO - Skipping QuantizationModifier calibration, it is not required for the provided quantization config.
2024-10-07T17:05:20.676385+0800 | initialize_compression | INFO - Preparing model.layers.0 for compression
2024-10-07T17:05:20.692998+0800 | initialize_compression | INFO - Preparing model.layers.1 for compression
2024-10-07T17:05:20.712378+0800 | initialize_compression | INFO - Preparing model.layers.2 for compression
2024-10-07T17:05:20.731872+0800 | initialize_compression | INFO - Preparing model.layers.3 for compression
2024-10-07T17:05:20.751130+0800 | initialize_compression | INFO - Preparing model.layers.4 for compression
Traceback (most recent call last):
  File "/root/autodl-tmp/quant.py", line 48, in <module>
    oneshot(
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 76, in oneshot
    main(model_args, data_args, training_args)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 359, in main
    stage_runner.one_shot()
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/transformers/finetune/runner.py", line 171, in one_shot
    self.trainer.one_shot(calibration_data=calib_data, stage=stage)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/transformers/finetune/session_mixin.py", line 401, in one_shot
    apply(
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/core/session_functions.py", line 184, in apply
    return active_session().apply(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/core/session.py", line 210, in apply
    self.initialize(**kwargs)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/core/session.py", line 156, in initialize
    mod_data = self._lifecycle.initialize(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/core/lifecycle.py", line 126, in initialize
    data = mod.initialize(state=self.state, **extras)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/stage.py", line 124, in initialize
    modifier.initialize(state, **kwargs)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/modifier.py", line 118, in initialize
    initialized = self.on_initialize(state=state, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 187, in on_initialize
    self.initialize_compression(modifiable_model, calibration_dataloader)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 246, in initialize_compression
    compressor.pre_compress()
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/utils/layer_compressor.py", line 79, in pre_compress
    wrapper = self.module_compressor_class(full_name, layer)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/quantization/gptq/utils/gptq_wrapper.py", line 45, in __init__
    "H", torch.zeros((self.columns, self.columns), device=self.dev)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.85 GiB. GPU 0 has a total capacity of 23.64 GiB of which 1.95 GiB is free. Process 863413 has 21.68 GiB memory in use. Of the allocated memory 21.23 GiB is allocated by PyTorch, and 43.51 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Video memory is not fully utilized.
Reduce the number of GPUs to 6 4090d and try again:

CPU ：96 核心
内存：480 GB
GPU ：NVIDIA GeForce RTX 4090 D, 6
存储：
  系 统 盘/               ：61% 19G/30G
  数 据 盘/root/autodl-tmp：56% 84G/150G
+----------------------------------------------------------------------------------------------------------------+
*注意:
1.系统盘较小请将大的数据存放于数据盘或网盘中，重置系统时数据盘和网盘中的数据不受影响
2.清理系统盘请参考：https://www.autodl.com/docs/qa1/
root@autodl-container-02b446b599-47e89a5d:~# cd  autodl-tmp
root@autodl-container-02b446b599-47e89a5d:~/autodl-tmp# python3 quant.py
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17/17 [00:13<00:00,  1.25it/s]
2024-10-07T17:13:06.232104+0800 | main | WARNING - Process rank: 0, device: cuda:0, n_gpu: 6, distributed training: True, 16-bits training: False
2024-10-07T17:13:06.232704+0800 | main | INFO - Training/evaluation parameters TrainingArguments(
_n_gpu=6,
accelerator_config={'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None, 'use_configured_state': False},
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
auto_find_batch_size=False,
batch_eval_metrics=False,
bf16=False,
bf16_full_eval=False,
clear_sparse_session=False,
data_seed=None,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_persistent_workers=False,
dataloader_pin_memory=True,
dataloader_prefetch_factor=None,
ddp_backend=None,
ddp_broadcast_buffers=None,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
ddp_timeout=1800,
debug=[],
deepspeed=None,
disable_tqdm=False,
dispatch_batches=None,
do_eval=False,
do_oneshot=True,
do_predict=False,
do_train=False,
eval_accumulation_steps=None,
eval_delay=0,
eval_do_concat_batches=True,
eval_on_start=False,
eval_steps=None,
eval_strategy=IntervalStrategy.NO,
eval_use_gather_object=False,
evaluation_strategy=None,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
fsdp=[],
fsdp_config={'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False},
fsdp_min_num_params=0,
fsdp_transformer_layer_cls_to_wrap=None,
full_determinism=False,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
gradient_checkpointing_kwargs=None,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_always_push=False,
hub_model_id=None,
hub_private_repo=False,
hub_strategy=HubStrategy.EVERY_SAVE,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
include_inputs_for_metrics=False,
include_num_input_tokens_seen=False,
include_tokens_per_second=False,
jit_mode_eval=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=5e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=0,
log_level=passive,
log_level_replica=warning,
log_on_each_node=True,
logging_dir=./output/runs/Oct07_17-13-06_autodl-container-02b446b599-47e89a5d,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=500,
logging_strategy=IntervalStrategy.STEPS,
lr_scheduler_kwargs={},
lr_scheduler_type=SchedulerType.LINEAR,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
neftune_noise_alpha=None,
no_cuda=False,
num_train_epochs=3.0,
oneshot_device=cuda:0,
optim=OptimizerNames.ADAMW_TORCH,
optim_args=None,
optim_target_modules=None,
output_dir=./output,
overwrite_output_dir=False,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=8,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
ray_scope=last,
recipe=[SmoothQuantModifier(index=None, group=None, start=None, end=None, update=None, initialized_structure_=False, initialized_=False, finalized_=False, started_=False, ended_=False, smoothing_strength=0.9, mappings=[[['re:.*q_proj', 're:.*k_proj', 're:.*v_proj'], 're:.*input_layernorm'], [['re:.*gate_proj', 're:.*up_proj'], 're:.*post_attention_layernorm']], ignore=None, num_calibration_steps=None, calibration_function=None, hooks_=None, resolved_mappings_=None, scales_=None), GPTQModifier(index=None, group=None, start=None, end=None, update=None, initialized_structure_=False, initialized_=False, finalized_=False, started_=False, ended_=False, sequential_update=False, targets='Linear', sequential_targets=None, block_size=128, quantize=True, dampening_frac=0.01, config_groups=None, ignore=['lm_head'], disable_quantization_observer_epoch=None, num_calibration_steps=None, scheme='W8A8', model=None, layer_compressors_=None, compressible_layers_=None, quantization_modifier_=None)],
recipe_args=None,
remove_unused_columns=True,
report_to=['tensorboard'],
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
run_name=./output,
run_stages=False,
save_compressed=True,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=500,
save_strategy=IntervalStrategy.STEPS,
save_total_limit=None,
seed=42,
skip_memory_metrics=True,
split_batches=None,
tf32=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_cpu=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_mps_device=False,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
)
2024-10-07T17:13:06.468673+0800 | _check_create_state | INFO - State created for compression lifecycle
2024-10-07T17:13:06.469173+0800 | pre_initialize_structure | INFO - Compression lifecycle structure pre-initialized for 0 modifiers
2024-10-07T17:13:06.469284+0800 | pre_initialize_structure | INFO - Compression lifecycle structure pre-initialized for 0 modifiers
2024-10-07T17:13:06.521594+0800 | one_shot | INFO - *** One Shot ***
2024-10-07T17:13:06.524774+0800 | from_modifiers | INFO - Creating recipe from modifiers
2024-10-07T17:13:06.525471+0800 | create_instance | WARNING - Could not process input as a file path or zoo stub, attempting to process it as a string.
2024-10-07T17:13:06.552090+0800 | _check_compile_recipe | INFO - Recipe compiled and 1 modifiers created
2024-10-07T17:13:10.069509+0800 | _calibrate | INFO - Running SmoothQuantModifier calibration with 2048 samples...
  0%|                                                                                                                                                                                                                                                 | 0/2048 [00:00<?, ?it/s]Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2048/2048 [25:33<00:00,  1.34it/s]
2024-10-07T17:38:43.087949+0800 | _apply_smoothing | INFO - Smoothing activation scales...
2024-10-07T17:38:43.369880+0800 | on_initialize_structure | WARNING - GPTQ quantization is set to True without an active quantization modifier.
2024-10-07T17:38:43.370003+0800 | _build_quant_modifier | INFO - Building quantization modifier with args: {'targets': 'Linear', 'scheme': 'W8A8', 'ignore': ['lm_head']}
2024-10-07T17:38:43.418608+0800 | _check_calibration_data | INFO - Skipping QuantizationModifier calibration, it is not required for the provided quantization config.
2024-10-07T17:38:43.634312+0800 | initialize_compression | INFO - Preparing model.layers.0 for compression
2024-10-07T17:38:43.638369+0800 | initialize_compression | INFO - Preparing model.layers.1 for compression
2024-10-07T17:38:43.642619+0800 | initialize_compression | INFO - Preparing model.layers.2 for compression
2024-10-07T17:38:43.646822+0800 | initialize_compression | INFO - Preparing model.layers.3 for compression
Traceback (most recent call last):
  File "/root/autodl-tmp/quant.py", line 48, in <module>
    oneshot(
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 76, in oneshot
    main(model_args, data_args, training_args)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 359, in main
    stage_runner.one_shot()
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/transformers/finetune/runner.py", line 171, in one_shot
    self.trainer.one_shot(calibration_data=calib_data, stage=stage)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/transformers/finetune/session_mixin.py", line 401, in one_shot
    apply(
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/core/session_functions.py", line 184, in apply
    return active_session().apply(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/core/session.py", line 210, in apply
    self.initialize(**kwargs)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/core/session.py", line 156, in initialize
    mod_data = self._lifecycle.initialize(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/core/lifecycle.py", line 126, in initialize
    data = mod.initialize(state=self.state, **extras)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/stage.py", line 124, in initialize
    modifier.initialize(state, **kwargs)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/modifier.py", line 118, in initialize
    initialized = self.on_initialize(state=state, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 187, in on_initialize
    self.initialize_compression(modifiable_model, calibration_dataloader)
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/quantization/gptq/base.py", line 246, in initialize_compression
    compressor.pre_compress()
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/utils/layer_compressor.py", line 79, in pre_compress
    wrapper = self.module_compressor_class(full_name, layer)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/llmcompressor/modifiers/quantization/gptq/utils/gptq_wrapper.py", line 45, in __init__
    "H", torch.zeros((self.columns, self.columns), device=self.dev)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.85 GiB. GPU 0 has a total capacity of 23.64 GiB of which 2.58 GiB is free. Process 872142 has 21.06 GiB memory in use. Of the allocated memory 20.52 GiB is allocated by PyTorch, and 126.84 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)