This change adds support for Intel Gaudi HPUs.

[emascarenhas]

Several configuration files are provided in the examples directory for use with Gaudi.

LLaMA-Factory features and optimizations including inferencing, training (sft, dpo, etc.), LoRA fine-tuning, distributed training with DeepSpeed and DDP are working. Please see README for details. Co-authored-by: Yaser Afshar [email protected] Co-authored-by: Edward Mascarenhas [email protected] Co-authored-by: Jianhong-Zhang [email protected] Co-authored-by: Wenbin Chen [email protected]
Co-authored-by: Voas, Tanner [email protected]

Before submitting

• [y] Did you read the contributor guideline?
• [y] Did you write any new necessary tests?

[hiyouga]

Thanks for your contribution, please view the comments

[ehartford]

Hello, I was wondering about the status of this? Is it safe to run a training job with this branch on Gaudi cluster?

[emascarenhas]

Hello, I was wondering about the status of this? Is it safe to run a training job with this branch on Gaudi cluster?

Yes you can. Please use the requirements-gaudi.txt to load the requirements in addition to the branch/PR. There are also yaml files in the examples/train_lora directory with _gaudi.yaml suffixes which should work out of the box.

[ehartford]

I try to run a job in the docker image:

cd docker/docker-hpu/
docker compose up -d
docker compose exec llamafactory bash

root@271f5f9d6340:/app# llamafactory-cli train train-qwen25.yaml
Traceback (most recent call last):
  File "/usr/local/bin/llamafactory-cli", line 5, in <module>
    from llamafactory.cli import main
  File "/app/src/llamafactory/__init__.py", line 43, in <module>
    from .extras.env import VERSION
  File "/app/src/llamafactory/extras/env.py", line 28, in <module>
    from .misc import is_torch_hpu_available
  File "/app/src/llamafactory/extras/misc.py", line 40, in <module>
    @lru_cache(maxsize=None)
NameError: name 'lru_cache' is not defined

[emascarenhas]

@ehartford ,
Sorry about that. The last two merge commits have a couple of changes which are causing this issue and another to do with transformers v 4.49.0.
This specific issue is due to the deletion of "from functools import lru_cache" line in src/llamafactory/extras/misc.py

Could you use the code at this commit in this branch for now?
https://github.com/hiyouga/LLaMA-Factory/tree/a16e3d47a29fb6d3d1b4afad9afca1d91f5b97c3

I will be pushing a commit to fix this after doing some more testing.

This set of instructions should work. Feel free to email me directly at [email protected] and we could also resolve other issues you may encounter.

cd to LLaMA Factory directory
git checkout a16e3d4
load docker as you normally would
pip install -e .
pip install -r requirements-gaudi.txt
llamafactory-cli train examples/train_lora/qwen2vl_lora_sft_gaudi.yaml