galaxy/llama
1
0
Fork 0
mirror of https://github.com/ggerganov/llama.cpp.git synced 2025-05-14 09:36:56 +08:00
Code Issues Actions 85 Packages Projects Releases Wiki Activity
llama/examples/perplexity
History
Kawrakow 463173a6c0
llama : speedup tokenization (#2831) 
* Speedup tokenization

On current master it takes ~3.2 seconds to tokenize
Wikitext. With this change it becomes ~525 ms.

* Fixit: it was missing the piece after the last found occurence

---------

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
2023-08-27 16:50:33 +03:00
..
CMakeLists.txt
cmake : install targets (#2256)
2023-07-19 10:01:11 +03:00
perplexity.cpp
llama : speedup tokenization (#2831)
2023-08-27 16:50:33 +03:00
README.md
Fix whitespace, add .editorconfig, add GitHub workflow (#883)
2023-04-11 19:45:44 +00:00

README.md

perplexity

TODO