Error when using AWQ model with 'use_gpu_id=False' #1768
Comments
bw-Deejee
changed the title
bw-Deejee
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
bw-Deejee
changed the title
bw-Deejee
changed the title
Using the newest version of the repo.
Following command works:
python generate.py --base_model=casperhansen/llama-3-70b-instruct-awq --load_awq=modelfollowing command doesn't:
python generate.py --base_model=casperhansen/llama-3-70b-instruct-awq --load_awq=model --use_gpu_id=Falseerror message:
... TypeError: functools.partial(<bound method AutoAWQForCausalLM.from_quantized of <class 'awq.models.auto.AutoAWQForCausalLM'>>, fuse_layers=True) got multiple values for keyword argument 'use_safetensors'Setting
--use_safetensors=Truedoesnt help.Is there any way to solve this?
The text was updated successfully, but these errors were encountered: