Skip to main content

Introduction

JTokkit is a fast and efficient tokenizer designed for use in natural language processing tasks using the OpenAI models. It provides an easy-to-use interface for tokenizing input text, for example for counting required tokens in preparation of requests to the gpt-3.5-turbo model. This library resulted out of the need to have similar capacities in the JVM ecosystem as the library tiktoken provides for Python.

Features​

✅ Implements encoding and decoding via r50k_base, p50k_base, p50k_edit and cl100k_base

✅ Easy-to-use API

✅ Easy extensibility for custom encoding algorithms

✅ Zero Dependencies

✅ Supports Java 8 and above

✅ Fast and efficient performance

Performance​

JTokkit reaches 2-3 times the throughput of a comparable tokenizer. Take a look in the benchmarks for more details.

Installation​

You can install JTokkit by adding the following dependency to your Maven project:

<dependency>
    <groupId>com.knuddels</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.0</version>
</dependency>

Or alternatively using Gradle:

dependencies {
    implementation 'com.knuddels:jtokkit:1.1.0'
}