Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/tame-turkeys-compete.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"js-tiktoken": patch
---

Update guidance on lite
31 changes: 29 additions & 2 deletions js/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,36 @@ Install the library from NPM:
npm install js-tiktoken
```

## Usage
## Lite

Basic usage follows, which includes all the OpenAI encoders and ranks:
You can only load the ranks you need, which will significantly reduce the bundle size:

```typescript
import { Tiktoken } from "js-tiktoken/lite";
import o200k_base from "js-tiktoken/ranks/o200k_base";

const enc = new Tiktoken(o200k_base);
assert(enc.decode(enc.encode("hello world")) === "hello world");
```

Alternatively, encodings can be loaded dynamically from our CDN hosted on Cloudflare Pages.

```typescript
import { Tiktoken } from "js-tiktoken/lite";

const res = await fetch(`https://tiktoken.pages.dev/js/o200k_base.json`);
const o200k_base = await res.json();

const enc = new Tiktoken(o200k_base);
assert(enc.decode(enc.encode("hello world")) === "hello world");
```

## Full usage

If you need all the OpenAI tokenizers, you can import the entire library:

> [!CAUTION]
> This will include all the OpenAI tokenizers, which may significantly increase the bundle size. See

```typescript
import assert from "node:assert";
Expand Down
Loading