Mastering Temperature and Top_p in ChatGPT API

In AI text generation models like ChatGPT, temperature and top_p are two parameters that control randomness and diversity in responses. They affect how tokens (words or phrases) are selected from the model's probability distribution. Here’s the difference:

1. Temperature (𝛕)

• Controls randomness by scaling the probability distribution.
• A higher temperature (e.g., 1.0 - 1.5) makes responses more diverse and creative, but also more random and unpredictable.
• A lower temperature (e.g., 0.1 - 0.3) makes responses more focused, deterministic, and repetitive.
• Example:
  • High Temperature (1.2) → "The sky is a canvas of endless hues, shifting from pink to deep indigo."
  • Low Temperature (0.2) → "The sky is blue."

2. Top-P (Nucleus Sampling)

• Controls randomness by restricting token selection to a subset of the most likely tokens.
• Instead of considering all possible words, top_p ensures only the most probable ones (whose cumulative probability adds up to p) are considered.
• Top-p = 1.0 → No restriction (like greedy sampling).
• Lower top-p (e.g., 0.3 - 0.5) → Limits the choice to fewer words, making output more focused and deterministic.
• Example:
  • High Top-P (0.9 - 1.0) → "The concert was an electrifying spectacle, filled with vibrant energy."
  • Low Top-P (0.3) → "The concert was enjoyable."

When to Use What?

• For highly creative outputs → Use higher temperature (e.g., 0.8 - 1.2) and/or higher top_p.
• For factual, precise responses → Use lower temperature (e.g., 0.2 - 0.5) and/or lower top_p.

💡 Tip: You can tweak either temperature or top_p, but not both aggressively at the same time, as they both influence randomness.

Here's a TypeScript example using OpenAI's API to demonstrate the difference between temperature and top_p in text generation.

Setup:

1. Install openai package:

   npm install openai dotenv

1. Create a .env file and add your API key:

   OPENAI_API_KEY=your_api_key_here

1. Use the following TypeScript code:

TypeScript Code: Temperature vs. Top-P

import { OpenAI } from 'openai';
import dotenv from "dotenv";

dotenv.config();

const openai = new OpenAI (
  {
    apiKey: process.env.OPENAI_API_KEY,
  }
);

async function generateText(
  prompt: string,
  temperature: number = 1.0,
  top_p: number = 1.0
): Promise<void> {
  try {
    const response = await this.openai.chat.completions.create({
      model: "gpt-4",
      messages: [{ role: "user", content: prompt }],
      temperature: temperature, // Controls randomness
      top_p: top_p, // Controls diversity
      max_tokens: 100,
    });

    console.log(`\n🔥 Temperature: ${temperature}, 🎯 Top-P: ${top_p}`);
    console.log(response.data.choices[0].message?.content);
  } catch (error) {
    console.error("Error:", error);
  }
}

const prompt = "Describe a futuristic city in 2050.";

// High temperature → More creative, unpredictable
generateText(prompt, 1.2, 1.0);

// Low temperature → More deterministic, factual
generateText(prompt, 0.2, 1.0);

// High top_p → More diverse words considered
generateText(prompt, 1.0, 0.9);

// Low top_p → Very focused choices, likely repetitive
generateText(prompt, 1.0, 0.3);

Expected Results:

High Temperature (1.2)

• "The city of 2050 is a dazzling wonderland of neon-lit floating districts, sky gardens that recycle air, and AI-powered robots running entire industries."

Low Temperature (0.2)

• "In 2050, cities have more electric vehicles, AI automation, and smart infrastructure for sustainability."

High Top-P (0.9)

• "By 2050, cities are vibrant with self-repairing roads, quantum-powered networks, and holographic entertainment hubs."

Low Top-P (0.3)

• "Cities in 2050 are automated, green, and efficient."

Key Takeaways:

• Higher Temperature → More creativity, but less consistency.
• Lower Temperature → More deterministic and factual.
• Higher Top-P → More diverse wording.
• Lower Top-P → More predictable and repetitive responses.