ojob.io/ai/tokenizer

Usage

ojob ojob.io/ai/tokenizer text="Hello, world!" model=o200k_base detail=true

Description

Encode text using tiktoken

Expects

Name Description
text Input text or file to encode
model Model name
detail Whether to output detailed token bytes