📐 GPT token estimation and context size utilities without a full tokenizer
-
Updated
Nov 29, 2024 - TypeScript
#
token-counter
Here are 3 public repositories matching this topic...
tokdu (Token Disk Usage) is a terminal-based utility that helps you analyze and visualize token usage in your codebase. Similar to the classic du (disk usage) command, tokdu shows you how many tokens your files and directories consume, which is essential when working with Large Language Models (LLMs) that have token limits.
python cli directory tokenizer scan openai curses token llm chatgpt tiktoken token-counter file-token-counter
-
Updated
Apr 7, 2025 - Python
Tiktoken-counter as standalone API
-
Updated
Aug 26, 2024 - Python
Improve this page
Add a description, image, and links to the token-counter topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the token-counter topic, visit your repo's landing page and select "manage topics."