OpenAI Launches ChatGPT API

OpenAI has finally launched the ChatGPT API

For some time, companies have claimed to use ChatGPT in their technology stack, something we've highlighted in the past. In reality, they were using GPT-3.5, also known as the davince-003 model.

While this model was an improvement from GPT-3, it lacked the fine-tuning conversational capabilities of ChatGPT that many people have been looking for or at least thought was available.

Emphasis on shopping

A number of big vendors have actually already had the ChatGPT API; Snapchat’s My AI feature, Shopify, Instacart, and Quizlet are some examples.

OpenAI's blog release post shows an obvious emphasis on ecommerce with short demos in the OpenAI blog post for Snapchat, Snapchat, Instacart, Quizlet, Shopify, and Speak.

It's now called gpt-3.5-turbo

The ChatGPT model is now called gpt-3.5-turbo.

OpenAI will be releasing updated models regularly which may affect the format of API calls but it's good to know developers will have access to the latest models.

It's much cheaper

The cost has reduced by 90%. Yes, it's really 10 times cheaper than GPT-3! This is really important, as I've lately had some misgivings about the rising cost of the most expensive Davinci model when creating chat applications and crunching through tokens.

It's the new go-to model

It looks like OpenAI is pushing gpt-3.5-turbo as the model to use for many of the tasks people are using Davinci for, not just chat applications:

• Compose emails or other written pieces
• Code in Python
• Provide answers to document-based queries
• Design conversational AI agents
• Give your software a human-like interface
• Offer tutelage in various subjects
• Translate seamlessly between languages
• Bring video game characters to life and more!"

It looks like Davinci prompts could well be transferable after a little rewriting.

Interesting API

There is an important change in the API. ChatGPT Consumes messages with metadata, not just unstructured text. The input, rendered as "token" sequences, is presented in a new format called chat markup language (ChatML).

So the main input is an array of message objects. You can see a request/response here:

Why ChatML?

The main security vulnerability for calls to the OpenAI LLMs has always been prompt injection attacks.

The prompt itself is of course just text. Messages to the chat completion are now segregated into 3 types:

• system
• user
• assistant

So the creation of ChatML is significant as it will help shore up these security vulnerabilities and allow for future development of the language e.g different datatypes to be sent as the payload.

Prompt Chaining

One of the challenges of working with LLMs to accomplish more complex tasks is managing successive prompts to send to the model which is all sent as unstructured text.

Frameworks like Langchain help with this but having a structured standard to sequence prompt nodes in a chain is a huge step forward.

Instructions & token management

The docs encourage experimentation to get the required results.

It looks like best practices may change depending on model versions...which might be interesting down the line for app integrations. There are still plenty of options to fine-tune responses e.g temperature

There is an emphasis on token management. Let's face it, setting a token limit and watching how many tokens have been used in an API call will be important... even though it's a lot cheaper.

No fine-tuning

Yes, as far as the API docs state...no fine-tuning of the model is available as of Mar 1st, 2023.

It remains to be seen whether this will be coming.

Policy changes

API usage data is no longer used for model training unless you opt-in.

Whisper API

Whisper, the speech-to-text model that was open-sourced in September 2022 has also just been released as an API.

Whisper is a multi-language speech recognition model so being able to provide a voice front-end interface with ChatGPT responses is now a nice option.