BPE Tokenizer Demo
1. Train Tokenizer
Training Text (paste text or fetch from URL):
Or Fetch Text from URL (ensure CORS is enabled on the server):
Fetch
Number of Merges (BPE Hyperparameter):
Train Tokenizer
Learned Merges (ordered):
Final Vocabulary (sample):
2. Test Tokenizer
Text to Tokenize:
Tokenize
Tokenized Output: