A token in an LLM (large language model) is the smallest unit that the model processes when handling text. In English, a token is typically a word or symbol. In Japanese, it is a unit that is split through methods such as morphological analysis (which may vary depending on the LLM). Tokenization makes it easier for the model to understand text.
The number of tokens affects the computational cost. The more tokens processed, the higher the computational cost.
