Working with AI APIs

What exactly is an API token? Let's try to understand. In a simple way. When you send a message to an AI API, you type in words.

But the machine underneath does not measure your text in words. It measures it in tokens. So what is a token? It is just a small piece of text.

Often a whole common word, and sometimes only a part of a longer or rarer one. Before the model reads anything, the API breaks your text into these pieces. And it does the same to the reply. Prompt and answer are both strings of tokens.

How big is one token? As a rough guide, about four characters, or three quarters of a word. So a hundred words lands near a hundred and thirty tokens. Now, why should you care?

Because tokens are the unit for everything. A model's context window, its short term memory, is measured in tokens. Not words, not characters. The bill works the same way.

An API charges you per token, for the ones you send and the ones it sends back. Tokens are the meter that is always running. This is why token count is not word count. Rare names, code, and other languages break into many small pieces.

The same text can cost more tokens than you would guess. So an API token is the unit of text an API works in. It is how the model reads your prompt, how it remembers, and how you get charged. Quick check now.

One question is coming up. Let's see if it clicked.

How does an API count your prompt tokens? Let's try to understand. In a simple way. You send a message to the API.

And along with the reply, it hands back a number. Your prompt tokens. But what is it actually counting? Before the model sees anything, a tokenizer runs on your text.

It splits your words into small pieces, the tokens. Then it counts them. But here is the twist. The API does not send only your words.

It wraps them. It adds a hidden system message, and labels marking who is speaking. And every one of those hidden pieces is a token too. The system message, the role labels, the boundary markers.

They all get counted, right alongside your text. Notice when this happens. The count is done before the model generates a single word. It is pure text splitting.

So the number is exact, and it never changes for the same input. Here is a catch. Each model brings its own tokenizer and its own wrapping. So the very same message can count as a different number of tokens on a different provider.

Where do you read the real number? In the response, a usage field lists prompt tokens. That is measured by the provider, not a guess. Trust it over any estimate.

So, prompt tokens are every token in the assembled input. Your words, plus the scaffolding the API adds around them. Count the whole request, not just what you typed. Quick check now.

One question is coming up. Let's see if it clicked.

How does your first API call work? Let's try to understand. In a simple way. Here is the first surprise.

Your code does not run the model. It lives on a giant computer far away. Your program just sends it a message, and waits for an answer. So what goes into that request?

Three things. An address, so it reaches the right service. Your secret key, so they know it is you. And a payload, the real content of your ask.

Let's open that payload. It is just structured text, in a format called JSON. Inside, two things. The name of the model you want, and your message to it.

Now you hit send. Your request travels across the internet to the provider. Their server runs the model on your message, and sends a response straight back. That response is JSON too.

And the reply you want is tucked deep inside it. So your code reaches in, past a few labels, and pulls out the text the model wrote. One more thing about a basic call. Your program pauses right here.

It waits until the whole reply has arrived, before moving to the next line. Step back and see the shape. You never downloaded the model. You are renting it, one call at a time.

The same request works from any language, on any device. So, an API call is one round trip. You send a request, an address, a key, and your message. The server runs the model, and mails the answer right back.

Quick check now. One question is coming up. Let's see if it clicked.

How does the temperature setting work? Let's try to understand. In a simple way. At every step, the model does not just blurt out a word.

It scores a whole list of possible next words. Each one gets a probability. How likely it is to come next. The simplest choice is to always grab the single highest scoring word.

Do that every time, and the same prompt gives you the exact same answer. Safe, but a little robotic. Temperature is one dial that changes this. It runs from zero upward.

Before the model picks, temperature reshapes those scores. It decides how much the long shots matter. Turn the temperature low, and the scores sharpen. The top word towers even higher, and the rest almost vanish.

The model plays it safe, focused and repeatable. Turn it high, and the scores flatten out. The gaps between words close, so unlikely words get a real chance to win. Now the output is surprising, varied, more creative.

What is really going on? Every score is divided by the temperature before the choice. A small number, like a half, stretches the gaps wide. A big number, like two, squeezes them together.

So you match the dial to the job. For facts, code, and pulling out data, keep it low. For brainstorming, stories, and fresh ideas, turn it up. So, temperature is one knob that reshapes the odds before the model picks.

Low means safe and repeatable. High means wild and varied. Same model, very different mood. Quick check now.

One question is coming up. Let's see if it clicked.

How does streaming work? Let's try to understand. In a simple way. First, picture asking without streaming.

The server writes the whole answer, then sends it back in one piece. You stare at a blank screen and wait. But here is the thing. The model never writes the whole answer at once.

It produces one token, then the next, then the next. So streaming is simple. Forward each token the instant it is made, instead of holding them all back until the end. To do that, the connection stays open.

Instead of one reply and goodbye, the server keeps pushing new pieces down the same pipe. Under the hood, each piece arrives as a tiny event. This is called server sent events. Every event carries a small chunk of text.

Your code reads each chunk as it lands, and adds it to the screen. That steady adding is the typewriter effect you see. The real win is the first word. It shows up almost right away, so you start reading while the rest is still being written.

One catch. Streaming does not make the model any faster. The total time is the same. You simply see the work early, not late.

Finally, a done signal marks the end. When your client sees it, the stream closes, and the answer is complete. So, streaming sends each token the moment it is ready, over one open connection. Same total time, but you read as it types.

Quick check now. One question is coming up. Let's see if it clicked.

How does forcing valid JSON work? Let's try to understand. In a simple way. Often you don't want a chatty paragraph.

You want data your program can read: a clean object, with named fields your code can pull out. But a language model only predicts text. It might wrap the answer in prose, add a code fence, or slip in a trailing comma. Your parser crashes.

The first fix is a setting called JSON mode. One flag in your request. It promises whatever comes back is valid JSON, something your parser will accept. But valid is not the same as correct.

JSON mode guarantees the braces close. It does not promise the fields are the ones you wanted. So you go further. You hand the API a schema: a blueprint of the exact fields and their types.

Now the reply must match that shape. How is this forced? At each step the model scores every possible next piece. Normally it may pick any.

Here the API checks the rules first. Any piece that would break the JSON is blocked before the pick. So only characters that keep it valid stay on the table. With a schema the allowed pieces get tighter still.

After a field name it must place a colon, then a value of the right type. One catch. The constraint fixes the form, not the meaning. So still describe the fields in your prompt, or you get neat but wrong data.

So, JSON mode locks the syntax. A schema locks the shape. The API does it by blocking illegal pieces as the model writes. Quick check now.

One question is coming up. Let's see if it clicked.

How does function calling work? Let's try to understand. In a simple way. On its own, a model has a blind spot.

It cannot check today's weather. It cannot run your code. It cannot read your database. It only writes text, from frozen knowledge.

So we hand it tools. You describe a function to the model. You give it a name, a short note on what it does, and the inputs it needs. Now, when a question needs that tool, the model does not reply in prose.

It pauses, and hands back a structured request instead. That request names one function, and fills in its arguments. Neat fields, packaged as clean data. Here is the key part.

The model does not run the function itself. It only asks for it. Your own code runs the real thing. So your code takes those arguments, calls the real function, and gets a result.

Then it hands that result back to the model. Now the model reads that fresh result, and writes the final answer in plain words for the user. Why is this reliable? Because the arguments come back as fields that match the shape you defined.

Not loose text you have to parse and hope. So function calling is a round trip. You describe the tool. The model picks it, and fills the arguments.

Your code runs it. The model answers. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is the input output price gap? Let's try to understand. In a simple way. Open any API price list and you see two numbers.

One price for the input tokens you send in. A higher price for the output tokens the model writes back. But a token is a token. Input or output, it is the same piece.

So why the price gap? Not the token, but the work behind it. Start with your input. The model already has every word.

So it reads them all together, in a single pass, side by side. This first read is called prefill. Ten tokens or a thousand, they sweep through the chip in one parallel batch. So each input token stays cheap.

Now the output. Here the model has nothing yet. It must invent the words, one at a time. Each new word leans on the word right before it.

And to make just one output word, the model runs the whole network, front to back. Ten words means ten separate runs, one after another. It cannot rush ahead. Word five needs word four to exist first.

So writing stays serial, never parallel. There is no shortcut. So the cost follows the effort. Input is one parallel read.

Output is many single steps. That is why output often costs several times more. So, input tokens are read together, so they stay cheap. Output tokens are written one by one, each its own pass, so they cost more.

The gap is the compute, not the token. Quick check now. One question is coming up. Let's see if it clicked.

How does a hard token limit work? Let's try to understand. In a simple way. Every API call takes one small setting, called max tokens.

It caps how many tokens the model is allowed to write back. Think of it as a ceiling on the answer. The model can write less. But it can never write more.

Once it hits the number, the reply stops. One thing to be clear on. This limit counts only the output, the words the model generates. Your prompt going in is not part of it.

Here is the catch. The model does not plan to fit. When it reaches the ceiling, the text stops. Often in the middle of a sentence.

So the reply comes with a stop reason. If it says length, the limit cut you off. If it says stop, the model finished on its own. A common mistake.

A low limit does not make the model concise. It does not shorten its thinking. It only chops the tail off a long answer. So why set it at all?

It is a safety valve. It caps your cost, since output tokens are billed. And it stops a runaway reply going on forever. Want a short and complete answer?

Ask for it in the prompt. Say, in one sentence. The limit is a guardrail, not a style instruction. So, max tokens is a hard ceiling on the output.

Reach it, and the reply is cut off, stop reason length. Use it to cap cost, not to shape the writing. Quick check now. One question is coming up.

Let's see if it clicked.

How does API cost estimation work? Let's try to understand. In a simple way. Every API bill comes from one simple idea.

You count the tokens, then multiply by the price. Cost equals tokens times price. But there is a twist. Input tokens and output tokens carry their own separate price.

So you estimate the two halves apart, then add them up. First, the input. You can count these tokens exactly, before you ever send, using the model's own tokenizer. Roughly, one token is three quarters of a word.

Now the output. This reply does not exist yet, so you cannot count it. You estimate a length instead. Setting a maximum caps it, giving you a safe worst case.

One more catch. Prices are not quoted per single token. They are quoted per million tokens. So divide your token count by a million, then multiply by the rate.

Now assemble it. Input tokens times the input rate, plus output tokens times the output rate. Add those two, and you have the cost of one request. But one request is tiny fractions of a cent.

Multiply by how many calls you make. A thousand calls a day turns those fractions into real money. Finally, check your work. Every response reports the exact tokens it used, in a usage field.

Compare that to your estimate, and tighten your numbers next time. So, count input, estimate output, price each at its own rate, and multiply by your call volume. That is how you estimate an API bill before it arrives. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is an API endpoint URL? Let's try to understand. In a simple way. When your code calls an AI model, it does not reach out into thin air.

It sends the request to one exact address on the internet. That address is the endpoint URL. Think of your request as a letter. The endpoint URL is the address written on the envelope.

It tells the request which server to travel to, and knock on. So let's open one up. A URL is built from parts, read left to right. The method, then the host, then the path.

It starts with h t t p s. That is the delivery method. The s means your request is encrypted on the way, so nobody can read it in transit. Next comes the host.

The domain name of the provider's server. This is whose machines you are actually talking to. Then the path, the part after the slash. This names which service you want.

Chat, embeddings, or images. One host, but the path picks the door. You will also spot a version, like v one, in the path. It pins the exact contract, so a future update cannot silently change what your code expects.

So one provider hands you many endpoints. One for chat, one for embeddings, one for audio. Different jobs, different URLs. You choose the one that fits your task.

So an endpoint URL is simply the full address of one service. Method, host, and path. Point your code there, and the request knows exactly where to go. Quick check now.

One question is coming up. Let's see if it clicked.

What exactly is a context window? Let's try to understand. In a simple way. A model does not remember anything between requests.

Everything it uses for one request must sit in a single place: the context window. Picture a desk. Only what fits on it can be used. That window is measured in tokens, not words or pages.

Remember, a token is a small piece of text, roughly three quarters of a word. It holds everything at once: your instructions, any pasted documents, the whole chat so far, and the reply being written. All of it shares one budget. This is where models differ.

One might hold eight thousand tokens. Another, a hundred and twenty eight thousand. A few now reach a million. A bigger window lets you paste more at once, a few pages versus a whole book.

Long chats stay in view instead of falling off the edge. Go over the limit, and the oldest text drops off, or the request is refused. The model cannot see what does not fit. But bigger is not free.

Every token is read on every step, so cost and delay both climb as you fill the window. And size is not the whole story. A model can hold a million tokens, yet lose a fact buried in the middle. So compare two things.

The window size, the ceiling on what fits. And how well the model uses the middle, not just the start. So the context window is the working memory, counted in tokens. Bigger holds more, costs more, and does not promise recall.

Quick check now. One question is coming up. Let's see if it clicked.

How does sending an image and text in one request work? Let's try to understand. In a simple way. Most models only read text.

But a vision model can also see. So you can hand it a picture and a question at the same time. Normally your message is just text. To add an image, the content turns into a list of parts.

One part holds your words. Another holds the image. The image travels one of two ways. You can paste the raw bytes, encoded as base sixty four text.

Or you give a URL the model fetches. The nice part is you can interleave them. Text, then an image, then more text. The model reads the parts in order, like one message.

But how does it read a picture? Inside, the model chops the image into small patches. Each patch becomes a token, like a word. So a picture is a block of image tokens.

Because of that, images are billed in tokens too. A big, high resolution image is many tokens. A small thumbnail, far fewer. But there is a catch.

A text only model cannot take an image. Send one, and the call errors. You need a model trained to see. Now you can point at a chart and ask what it shows.

Read a receipt. Describe a photo. One request, image and words together. So, one request can carry both.

The content is a list of parts, text and image. The image becomes tokens, and only a vision model can read it. Quick check now. One question is coming up.

Let's see if it clicked.

What exactly is an API key? Let's try to understand. In a simple way. An API key is a long, secret string of characters.

You send it with every request, tucked inside a header. The provider looks up that key, and sees whose account it belongs to. So the key is how a request proves it is really you. And that is the whole point.

To the API, the key is you. Whoever holds it can send requests, and every one is billed to you. So a leaked key is dangerous. The classic mistake is pasting it into your code, then pushing that code to a public repo.

Bots scan new repos within minutes. Another trap is the browser. Any code you ship to the front end can be read by every visitor. So never put the key there.

Call the API from your own server. The safe home is outside your code. Keep the key in an environment variable, or a secrets manager. And add that file to git ignore, so it is never committed.

If a key ever leaks, you rotate it. Revoke the old one, and generate a fresh one. The moment you do, the leaked key stops working. You can also limit the damage in advance.

Use a separate key for each app. Set a spending cap. Then one leak cannot drain everything. So, your API key is a password that spends your money.

Keep it off the front end, out of your code, and rotate it the instant it leaks. Quick check now. One question is coming up. Let's see if it clicked.

How does switching providers with one line of code work? Let's try to understand. In a simple way. Say your app calls one AI provider.

Now you want another. Maybe it is cheaper, faster, or just smarter. Do you rewrite everything? Here is the trick.

Most providers copied the same request shape. One company set the standard, and the others made themselves compatible with it. So your code builds the same request either way. A list of messages, each with a role and some content.

That part never changes. What does change? Just three settings. The address the request is sent to, the secret key, and the name of the model.

That address is the one line. Your library normally aims at one provider. Override it, and every call now flies to a different company. It works because the reply matches too.

The answer returns in the same shape, the same fields. So the code reading it never notices the swap. One catch. Compatible does not mean identical.

Some features, like certain tool calls, differ. So switch, then test that the edges still behave. You can go further. A gateway library gives one doorway to many providers.

You pass a model name, and it routes the call for you. Why bother? Price, speed, and safety. Chase a cheaper token, a faster reply, or keep a backup ready if your main one fails.

So, one shared request shape is the whole secret. Swap the address, the key, and the model. The rest of your code stays put. Quick check now.

One question is coming up. Let's see if it clicked.