Skip to content

Commit 3a940da

Browse files
authored
[doc] Fix tokenizer related documentation (#10000)
`extension.llm.tokenizer.tokenizer` -> `pytorch_tokenizers.tools.llama2c.convert`
1 parent 9b7a878 commit 3a940da

File tree

4 files changed

+4
-4
lines changed

4 files changed

+4
-4
lines changed

examples/demo-apps/android/LlamaDemo/docs/delegates/qualcomm_README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,7 @@ You may also wonder what the "--metadata" flag is doing. This flag helps export
135135

136136
Convert tokenizer for Llama 2
137137
```
138-
python -m extension.llm.tokenizer.tokenizer -t tokenizer.model -o tokenizer.bin
138+
python -m pytorch_tokenizers.tools.llama2c.convert -t tokenizer.model -o tokenizer.bin
139139
```
140140
Rename tokenizer for Llama 3 with command: `mv tokenizer.model tokenizer.bin`. We are updating the demo app to support tokenizer in original format directly.
141141

examples/models/llama2/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ You can export and run the original Llama 2 7B model.
4141
```
4242
4. Create tokenizer.bin.
4343
```
44-
python -m extension.llm.tokenizer.tokenizer -t <tokenizer.model> -o tokenizer.bin
44+
python -m pytorch_tokenizers.tools.llama2c.convert -t <tokenizer.model> -o tokenizer.bin
4545
```
4646
4747
Pass the converted `tokenizer.bin` file instead of `tokenizer.model` for subsequent steps.

examples/models/phi-3-mini/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ pip uninstall -y transformers ; pip install transformers==4.44.2
1313
```
1414
cd executorch
1515
wget -O tokenizer.model "https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/resolve/main/tokenizer.model?download=true"
16-
python -m extension.llm.tokenizer.tokenizer -t tokenizer.model -o tokenizer.bin
16+
python -m pytorch_tokenizers.tools.llama2c.convert -t tokenizer.model -o tokenizer.bin
1717
```
1818
2. Export the model. This step will take a few minutes to finish.
1919
```

examples/qualcomm/oss_scripts/llama/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ wget "https://huggingface.co/karpathy/tinyllamas/resolve/main/stories110M.pt"
4141
wget "https://raw.githubusercontent.com/karpathy/llama2.c/master/tokenizer.model"
4242

4343
# tokenizer.bin:
44-
python -m extension.llm.tokenizer.tokenizer -t tokenizer.model -o tokenizer.bin
44+
python -m pytorch_tokenizers.tools.llama2c.convert -t tokenizer.model -o tokenizer.bin
4545

4646
# params.json:
4747
echo '{"dim": 768, "multiple_of": 32, "n_heads": 12, "n_layers": 12, "norm_eps": 1e-05, "vocab_size": 32000}' > params.json

0 commit comments

Comments
 (0)