Introduction

The tokenize API endpoint allows you to easily convert some text into a list of tokens for a specific model.

Prerequisites

In order to use the embedding endpoint, here are the prerequisites:

• Having a Paradigm API key: if you do not have one, go to your Paradigm profile and generate a new API key.
• Having the desired LLM available in Paradigm: If you want to use a new model, you must add it to Paradigm from the admin interface.

Usage methods

There are several ways to call the endpoint:

1. With the python requests package (recommended)
2. Through a curl request: good to do a quick check or a first test

Python requests package

You can also avoid using the OpenAI python class and directly send request to the API endpoint through the requests package.

import requests
import os

# Get API key from environment
api_key = os.getenv("PARADIGM_API_KEY")

response = requests.request(
    method="POST",
    url="https://paradigm.lighton.ai/api/v2/tokenize",
    headers={
        'accept': "application/json",
        'Authorization': "Bearer <YOUR_API_KEY>"
    },
    json={
        "model": "alfred-40b-1123",
        "prompt": "This a test string"
    }
)

print(response.json())

You would then get a JSON answer as a dictionnary:

{
    'id': '8c0d73b9-b18a-4893-a38e-a4338a7d4e0e', 
    'tokens': [
        {'This': 1182}, 
        {'Ġa': 241}, 
        {'Ġtest': 1318}, 
        {'Ġstring': 3821}
    ], 
    'text': 'This a test string', 
    'n_tokens': 4, 
    'model': 'alfred-40b-1123'
}

cURL request

If you would prefer sending a request to Paradigm with a simple cURL command, here is an example:

curl --request POST \
  --url https://paradigm.lighton.ai/api/v2/tokenize \
  --header 'Authorization: Bearer <YOUR_API_KEY>' \
  --header 'content-type: application/json' \
  --data '{
  "model": "alfred-40b-1123",
  "prompt": "This a test string"
}'