files-to-prompt . -e ts | ttok
Error: Encountered text corresponding to disallowed special token '<|endoftext|>'.
Use --allow-special to allow special tokens
then
files-to-prompt . -e ts | ttok --allow-special
Error: Cannot use --allow-special without --encode or --tokens
Then with --encode or --tokens I cannot count it. I just want to see how many tokens would it use to query the current directory.
then
Then with --encode or --tokens I cannot count it. I just want to see how many tokens would it use to query the current directory.