Forem: Kathryn Grayson Nanz

AI Crash Course: Prompt Engineering

Kathryn Grayson Nanz — Tue, 14 Apr 2026 17:32:39 +0000

Read the previous blog in this series: Hallucinations.

Prompting is the primary way we provide input to AI models: when we need a model to generate something for us, we begin the process by prompting it with a plain-language request to complete the task. When we’re having simple, back-and-forth personal exchanges with conversational models, our prompts can be extremely casual – sometimes, even identical to the way we’d talk to another human being! However, when we’re considering generative AI in the context of a feature we’re adding to our applications, casual prompts will no longer suffice.

To leverage AI in production applications, the results need to be (1) relatively consistent and (2) formatted in an expected, parseable structure. If a user asks the same question twice, the output should be similar both times. If our frontend was built to map over a JSON object, we don’t want the model to output markdown.

Because AI models are non-deterministic by nature, even the most perfect, precise prompt cannot prevent the occasional hallucination or error. However, prompt engineering is one of our most effective and accessible tools for getting a model to behave in the way we would like.

Why Engineer Prompts?

In the 2020 paper *Language Models are Few-Shot Learners,* Brown, et al. demonstrate that GPT-3 is able to successfully leverage in-prompt demonstrations (or “shots”) to perform tasks it was not explicitly trained to complete. This approach is known as “in-context learning”. The team tested the model with “zero-shot” (only the instruction), “one-shot” (one demonstration), and “few-shot” prompts (in the case of this study, 10-100 demonstrations, depending on available context – but worth noting that the term “few-shot” prompting today generally only refers to the inclusion of 2-10 demonstrations). As we can see in the chart below, they prove that “While zero-shot performance improves steadily with model size, few-shot performance increases more rapidly, demonstrating that larger models are more proficient at in-context learning.”

The important takeaway from this study for us is that, by using this approach, they were able to achieve results with non-finetuned models that were on par or exceeding the results from finetuned models. Models have, of course, advanced since the writing of this paper – it’s no longer entirely accurate to say that prompt engineering alone will offer equivalent results to finetuning (or other newer techniques, like RAG). However, it is still fair to say that if we engineer our prompts, we will achieve noticeably high-quality results. Depending on our project goal and budget, that may be good enough.

The Structure of a Well-Engineered Prompt

As of the time of this writing, there’s a common structure that has emerged as a best practice approach for prompt engineering: role + task + format + example. Prompts that include at least one of each of these pieces tend to perform better than prompts that do not – although there has been some documented difference between the models as to which order is preferable. For example, one model might do better when the task is the first thing listed, another when the task is last. Experimenting with your model of choice to see what gets the best results is still a required aspect of the prompt engineering process.

To look at a sample prompt using this structure, this is the system prompt from a playlist generating app I built with my fellow Progress DevRel (and friend) Alyssa Nicoll:

const SYSTEM_PROMPT = `You are a music recommendation assistant.
Given a user's prompt (mood, activity, genre, era, etc.), return a JSON object with a "tracks" array that includes at least 10 tracks.
Each track must have: title (string), artist (string), album (string), release year (number), confidence ranking for how well this song fits the prompt (number 0-100), one sentence explaining why this song fits the prompt (string). Return an error for unusable prompts: offensive content, gibberish, or explicitly impossible requests. Otherwise always try to generate recommendations. Return ONLY valid JSON, no markdown or explanation.
Example success: {"tracks":[{"title":"Song Name","artist":"Artist Name","album":"Album Name","year":2020,"confidence":92,"reason":"The driving rhythm and anthemic chorus perfectly match the high-energy workout vibe."}]};

If you’re totally new to prompt engineering and would like a deeper dive into that structural method, you might enjoy Understanding the Architecture of a Prompt, which explains in greater detail how to most effectively write each part.

As we do in that example, you may also find it helpful to think about the “system prompt” and the “user prompt” separately; in fact, many model APIs offer these as distinct inputs. The system prompt is intended to specify the context and description for the task, while the user prompt contains the task itself. In the playlist generator example, you’ll note that the end user’s input (the actual genre, mood, etc. that the model is meant to generate the playlist around) is not included. In this case, the system and user prompts get passed in via the openAI API under separate system and user roles.

javascript const completion = await openai.chat.completions.create({ model: "gpt-4o-mini", messages: [ { role: "system", content: SYSTEM_PROMPT }, { role: "user", content: prompt.trim() }, ], temperature: 0.7, });

Why Does Prompt Engineering Work?

While it’s helpful to know best practices, as developers it often serves us better to know the “why” behind the methodology: in this case, why does a prompt that’s engineered like the one above perform better than a prompt that just says “Recommend 10 songs based on the user’s input and format it as a JSON object”? Mostly, it has to do with how models work at a high level; prompts tend to be more successful when they mirror the types of data that the model was trained on.

For now, what’s most important to remember is that AI models are – at their core – document completion machines. Early AI models (before the chat interface became popular) didn’t “respond” to a prompt, they just continued them. So if my prompt was “Mary had a little”, the LLM would likely return “lamb” to complete the line. If you haven’t read the Tokens, Predictions and Temperature article from earlier in this series, I’d encourage you to go give it a quick skim – it explains that prediction process in more detail.

Even the chat models we’re so familiar with today are still doing exactly this – they’ve just been trained on transcripts of conversations, so they can “understand” the patterns of a back-and-forth exchange. When we interact with a conversational AI (like ChatGPT or Claude), the API is just wrapping our inputs (and its own outputs) in the structure of a transcript, so that the LLM can process them together in a structured context – like a document. This allows it to leverage its prediction mechanisms in order to “complete” that document. To the user, it feels like having a real conversation – to the LLM, it’s more like we submitted a half-finished play script and asked it to generate the rest for us.

There is, admittedly, very little functional difference between thinking that a model is conversing with us vs. understanding that it’s completing our conversational transcript. However, making that mental shift can help us shape our prompts more effectively. Often, when we think about NLP, we assume it means that because a computer can understand our “natural” language, we can instruct it in the same way we would another human. While it’s true that we can use natural language – as in words and sentences instead of code – the LLM is not going to process our request like a human would. Instead, it’s going to insert our prompt as the most recent line in its script, and attempt to continue the script to the best of its ability, based on all the other sample scripts it was trained on. By phrasing our requests like the samples in the training data and giving it as much context as possible directly in the prompt, we can enable it to make better, more accurate completions.

AI Crash Course: Hallucinations

Kathryn Grayson Nanz — Wed, 25 Mar 2026 13:10:15 +0000

In the last article, we talked about probability, token prediction, and how temperature can change the types of responses we get from AI models. However, it would be irresponsible of us to talk about generative AI without also addressing the elephant in the room: hallucination.

Hallucination is when generative AI models return a response that isn’t grounded in facts or based on their training data. You’ve almost certainly experienced it for yourself if you’ve chatted with an LLM for even a little while. As of the writing of this article, there is no known way to prevent AI models from occasionally generating hallucinated content. There are several things we can do to reduce hallucinations, but nothing will eliminate them completely.

This makes hallucinations one of the most prominent issues facing us today, as developers building with AI. Generative AI is incredibly impressive from a purely technical perspective, but if the output can’t be consistently trusted then our ability to build real-world apps leveraging it is limited. Of course, this varies based on field and use-case: as we discussed in the previous article, there will be situations where a higher rate of error is okay and situations where mistakes are unacceptable. Part of our role as ethical AI developers is to assess the situations where our application will be used and determine whether the rate of hallucination minimization that we can realistically achieve is within the bounds of acceptable risk – or whether the use of AI should be avoided entirely.

What causes hallucinations?

One of the most interesting (and challenging) aspects of AI hallucination is that we still don’t fully understand why it occurs or which underlying mechanisms are primarily responsible. We’ve identified a few things that make hallucinations worse and / or amplify them, but nothing can be traced back directly as the root cause.

A logical assumption to make would be that errors in the training data are responsible for hallucinated content. After all, as they say: “garbage in, garbage out”, right? However, Kalai et al. show in their 2025 paper Why Language Models Hallucinate that "…even if the training data were error-free, the objectives optimized during language model training would lead to errors being generated.” They theorize that hallucination happens because most AI model training rewards producing an answer rather than declining, which “encourages” models to generate plausible responses even when uncertain.

Technically, guessing an answer that has a slim chance of being correct (even if that chance is just 1 out of 10000) is more likely to be right than giving no answer (which has a 0% chance of being the right answer). That means that guessing is the “better” mathematical option – assuming that “better” is being evaluated based on “highest chances of being right” (which is how most AI training benchmarks are currently structured).

Let Model B be similar to Model A except that it never indicates uncertainty and always “guesses” when unsure. Model B will outperform A under 0-1 scoring, the basis of most current benchmarks. This creates an “epidemic” of penalizing uncertainty and abstention…

Another popular theory comes from the paper Shaking the foundations: delusions in sequence models for interaction and control by Ortega, et al. in 2021. They argue that hallucinations are caused by the model integrating its own responses as reference data and further extrapolating from there. This pollutes the data and creates a kind of compounding ouroboros of “self-delusion” as the model consumes its own (incorrect) output and builds upon it. While Ortega et al.'s theory was related to training sequential decision-making models, the underlying pattern – a model building on its own outputs to create compounding errors – shows up in generative models as well.

…the model update triggered by the collected data differs depending upon whether the data was generated by the model itself (i.e. actions) or outside of it (i.e. observations), and mixing them up leads to wrong inferences. These take the form of self-delusions where the model takes its own actions as evidence about the world…due to the presence of confounding variables.

Reducing hallucinations

With the disclaimer that we can’t (at this point) eliminate hallucinations entirely – there are still things we can do to reduce the chances of hallucinations showing up in our model’s responses. Whether you implement some or all of these will depend on what you’re building and what kinds of hallucinations your model is most often generating.

Reference Data

Hallucinations can be more likely when a model doesn’t have access to the correct information. This aligns with the “guessing” theory from earlier: if the training data was minimal or not well-aligned with the use of the model, then it’s more likely that the model will be asked questions that it’s unequipped to answer – and by extension, the model is more likely to make an unsubstantiated guess.

If you’re building and training your own models then you have the ability to adjust the training data – but many application developers are using pre-trained foundation models. That doesn’t mean we’re completely out of luck, though: we can still finetune the model we’re using, or use an approach like RAG to provide additional reference data to the model. The more current, accurate, relevant information we can provide a model with, the lower our chances of hallucination will be.

Context

Hallucinations are also common in longer exchanges with models; the longer the conversation gets, the more likely the model is to “get lost” and provide responses that may be technically correct but don’t make sense in the context of the current task.

The amount of history from the current conversation that an LLM can see at a given time is known as its context window. As a conversation gets longer, older messages may eventually fall outside the model’s context window and are no longer visible to the model. A model also has limited attention that it can distribute among all the tokens in a given conversation, in order to help it keep track of inter-conversational references. This is what helps a model read things like “Kathryn is a software engineer. They mostly write React.” and understand that “They” and “Kathryn” refer to the same person. As a model’s attention is stretched and more things fall out of its context window, it “forgets” information and becomes more likely to hallucinate.

The likelihood of this can be reduced by prompt engineering (”Remember when we discussed X? Make a plan based on X and adapt it to Y.”) or by providing additional context reference via Skills or reference documents. It can also be helpful to intentionally limit the length of a user’s exchange with the model – if you notice that the model is more likely to hallucinate during long exchanges, see if you can identify natural and non-disruptive places to reset and allow it to “start fresh”.

Prompting

Occasionally, hallucinations can be caused (at least in part) by the user’s input. Anh-Hoang, Tran and Nguyen define in Survey and analysis of hallucinations in large language models: attribution to prompting strategies or model behavior two main categories of hallucination: prompting-induced hallucinations and model-intrinsic hallucinations. Prompting-induced hallucinations occur when “prompts are vague, underspecified, or structurally misleading, pushing the model into speculative generation”. By being intentional and careful about the shaping of our own prompts – or by providing prompt guidance / instructions and shaping our end users’ prompts – we can reduce the chances of hallucination.

We might also help reduce hallucinations by including instructions in our prompt for the model to “say ‘I don’t know’ if you’re unsure of the answer”, asking it to cite sources, or similar. By telling the model that we would prefer no answer over the wrong answer, we might override that internal calculation about which option would be considered “better” based on the training benchmarks – we can attempt to change the grading scale, if you will.

Looking for more information on prompting? Read the next article in this series: Prompt Engineering.

AI Crash Course: Tokens, Prediction and Temperature

Kathryn Grayson Nanz — Wed, 04 Mar 2026 15:45:35 +0000

Read the first blog in this series: AI, ML, LLM, and More

We often describe AI models as “thinking,” but what is actually happening when an AI model “thinks”? When it’s drafting a response to us, how does it know what to say?

One of the most tempting (and common) misunderstandings related to AI models is the perception that they “think”—or have awareness of any kind, for that matter.

This is primarily a language problem: we (meaning humans) like to use the words and experiences that we’re most familiar with as a shorthand to communicate complex ideas. After all, how many times have you seen a webpage slowly loading and heard someone say “hang on, it’s thinking about it”? We “wake” computers up from being in “sleep” mode, we initiate network “handshakes,” we get annoyed with memory-”hungry” programs.

In the same way, we often describe AI models as “thinking,” sometimes even including the directive to “take as much time as you need to think about this” when prompting them! But what is actually happening when an AI model “thinks”? When it’s drafting a response to us, how does it know what to say?

The short answer is that AI models (especially text-focused LLMs, which we’ll use as the example for the rest of this article) are highly advanced token prediction machines. They use neural networks (a type of machine learning algorithm) to identify patterns across large contexts. Based on decades of research about how sentences are structured in a given language (like the prevalence of various words, and the statistical likelihood that one specific word will follow another), modern AI models are able to combine tokens into words, and then words into sentences.

Predictive Language Models

For the long answer … we actually have to start all the way back in the 1940s. Cryptography and cypher-breaking technology was developing at a breakneck pace in an attempt to intercept and decrypt enemy communications during WWII. If you could recognize and crack even one or two letters in an enciphered communication, these new predictive methods could be used to help determine what the other letters were likely to be.

For example, in English “E” is the most commonly used letter, and “T” and “H” are often used together. If we know that one letter in a word is “T,” we can calculate the likeliness that the next letter will be “H” (spoiler alert: it’s pretty high). This same probability calculation can be extended from letters to words, from words to phrases, and from phrases to sentences. If you’re interested in the true deep dive, you can still read the 1950 paper published about these learnings: “Prediction and Entropy of Printed English” (which, by the way, is where those earlier facts about “E,” “T” and “H” come from). If you want the overview, watch The Imitation Game (actually, just watch The Imitation Game anyway; it’s a great movie).

Fast-forward to today: computers have offered us ways to analyze huge amounts of language data in ways that were simply not available in the 1950s. Our knowledge on this topic and our ability to predict content has only gotten better over the last 70+ years.

When we’re training large language models (LLMs), most of what we’re doing is giving them these huge samples of language—which, in turn, allows them to leverage these predictive models to more accurately identify and generate specific word, phrase and sentence combinations. You can think of it like the predictive text on your smartphone, but with the dial turned up to 1000 because it’s not just looking at samples of how you text, it’s looking at millions of samples demonstrating various ways that humans have communicated in a given language over hundreds of years.

Tokens

However, it would be a bit of a misrepresentation to say that LLMs are “thinking” in words. In fact, LLMs process language via tokens which can be (but aren’t always) entire words. Tokens are the smallest units that a given language can be broken down into by a model.

If you’re familiar with design systems, you might have heard of design tokens. Design tokens are the smallest values in a design system: hex colors, font sizes, opacity percentages and so on. In the same way, language tokens can be thought of as the smallest pieces that words can be broken down into. This is commonly aligned with prefixes, suffixes, root words, possessives, contractions, etc., but can also include units that aren’t necessarily based on human language structure.

This is done for both flexibility and efficiency: for example, if you can train an English-based model to recognize “draw” and “ing,” then you don’t have to explicitly teach it “drawing.” The same idea can be extended to things like “has” or “should” + “n’t” and “make” or “teach” + “er.” This can also help it make “educated guesses” at user input words that weren’t included in its training material. So if a user says they’re “regoogling” something, the LLM can identify the prefix “re-”, the name “Google” and the suffix “-ing” and cobble together something reasonably close to a working definition.

Because of the intrinsic role they play in AI functionality, tokens have become one of the primary ways we measure various AI models. Tokens are used to measure the data that models are trained on (total tokens seen during training), how much a model can process at a given time (known as the context window), and—as you already know if you’re a developer building apps that integrate with popular foundation models—API usage (both input and output) for the purposes of monetization.

Temperature

Adjusting these predictive computations that determine which tokens are most likely to follow other tokens is also part of how we can shape the model’s responses. The temperature of an AI model refers to how often the model will choose tokens that are less statistically likely.

A model with a low temperature is more conservative; when selecting the next word in its predictive text chain, it will choose options that have a higher percentage of occurrence. For instance, a low temperature model would be far more likely to say “My favorite food is pizza” than “My favorite food is tteokbokki,” assuming it was trained on data where “pizza” followed the words “My favorite food is” 70% of the time and “tteokbokki” only followed 15% of the time. Increasing the temperature of the model increases the percentage of times the model will choose the less-popular token by flattening the probability distribution; lowering the temperature sharpens the distribution, making less-common responses less likely.

To be clear, these are made up statistics for the purpose of illustration—if we aren’t training a model ourselves, we cannot know what the actual percentage of occurrence is for these kinds of things (unless the people doing the training offer to share that information, which is rare).

A model with a low temperature is more predictable, whereas a model with a high temperature will be more novel—but also more prone to mistakes. As IBM says: “A high temperature value can make model outputs seem more creative but it's more accurate to think of them as being less determined by the training data.”

Ultimately, the temperature of the model should be determined based on its purpose and acceptable room for error. If you’re using an AI model in a professional application to answer questions about a company’s products, you probably want a very low temperature; the tolerance for error in that situation is low, and you don’t want the AI to offer less-common results. However, if you’re using a model personally to help you brainstorm D&D campaign ideas, a higher temperature could offer you less common suggestions (plus, you’re probably less bothered in this situation by results that don’t make sense).

Regardless of temperature, however, it’s important to acknowledge that if content is included in the training data, there’s some chance (no matter how low) that it will be selected for inclusion in a model’s response. Even with a very low temperature model, there’s still a non-zero chance that it will choose the less popular answer. Why not just always set models at the most conservative temperature? Mostly because, at that point, we could just program a set of dedicated responses—most users of LLMs (and generative AI models, in general) want the “intelligence” that comes with not getting exactly the same answer every time. After all, LLMs aren’t retrieving sentences from training data via a lookup-table; their primary benefit is in their ability to generate new sequences token-by-token based on what they’ve “learned.”

Bias

Finally, it’s worth noting that this also plays into how bias occurs in AI systems. To return to the food example we used when discussing temperature: it’s entirely possible for us to curate a dataset in which “tteokbokki” occurs more often than “pizza” and then train a model on that. In that case, if we were to ask the model about the food most people like the best, it would be more likely to say “tteokbokki” even though that’s (probably) not reflective of the general population.

Obviously, this is less of a concerning issue if we’re just talking about food—but more concerning for issues related to sex, gender, race, disability and more. If a model is trained on data where doctors are more often referred to with he/him pronouns, it will in turn be more likely to return content identifying doctors as male. If slurs or hate speech are included in significant percentages, that content will be returned by the model at a level reflective of its training data (unless actively mitigated, as described below). This can be further reinforced by feedback and responses from users that are referenced by the model as context or in post-training.

As you might imagine, this is a common issue for models trained on information scraped from the internet: from chat logs, message boards, forums and more. It is possible to counteract this by excluding harmful content from the training data or by including data that intentionally balances occurrences of specific content (i.e., including the phrases “She is a doctor.” and “They are a doctor.” at equal percentages to “He is a doctor.”). It can also (sometimes) be filtered on the output side, by building in checks for specific words and prompting the model to re-create the response if it includes forbidden content. However, this must be an intentional choice implemented by those responsible for creating the training data and maintaining the model.

Read the next blog in this series: Hallucinations.

AI Crash Course: AI, ML, LLM and More

Kathryn Grayson Nanz — Wed, 04 Mar 2026 15:42:26 +0000

Hello! Welcome to the beginning of a new series: AI Crash Course.

This is something I’ve been really excited to write because, while AI is quickly becoming a part of many peoples’ everyday lives, it can often feel like a bit of a black box. How does it work? Why does it work—or (perhaps more importantly), why doesn’t it work? What can it do? What tools can we use to work with it?

For many folks, our understanding of AI can be fairly surface-level and focused on our experience with it as an end user. This series aims to be an introductory course for anyone interested in learning more about the technical aspects of how AI models work, but feeling (perhaps) a bit intimidated and unsure where to start.

If you are a developer who has already been working extensively with building AI agents and skills, this will likely be too low-level for you (but hey, never hurts to refresh on the basics!). However, if you (like many) feel that you might have “missed the on-ramp” or if you’ve been tentatively working with AI in your applications without truly understanding what’s happening behind the scenes: you’re in the right place!

To start off, we’re going to make sure we’re all on the same page in terms of terminology. It’s common—especially outside of tech spaces—to see a handful of terms used almost interchangeably: AI, GenAI, ML, LLM, GPT, etc. Let’s take a moment to define each of these, so we can use them intentionally moving forward.

AI: Artificial Intelligence

IBM defines artificial intelligence (AI) as “technology that enables computers and machines to simulate human learning, comprehension, problem solving, decision making, creativity and autonomy.”

(Fun fact: IBM is also responsible for the famous 1979 slide reading, “A computer can never be held accountable, therefore a computer must never make a management decision.” So … things change, I suppose.)

A printed slide with all-caps text reading

AI is a high-level, general term that encompasses many more specific terms—in the same way that “exercise” can refer to many more specific movements (running, dancing, lifting and so on). Generally speaking, modern AI techniques involve training a computer on a dataset in order to do something it wasn’t explicitly programmed to do.

ML: Machine Learning

Machine learning (ML) is an approach for training AI systems. It’s called “learning” because the system is able to recognize patterns in the content and draw related conclusions, even if that conclusion wasn’t directly programmed into the system.

One common example of this is image recognition: if an AI model is trained on a dataset that includes many photos of dogs, it can learn to identify when a photo shows a dog even if that exact dog photo wasn’t included in the dataset it trained on.

Model

A model is any specific AI system that’s been trained in a particular way. Models can be small, locally hosted and trained on specific, proprietary data, or they can be larger systems trained on broad, general data.

Foundation Models

The larger, broadly trained models are known as foundation models. These are probably the ones you’ve used most often, such as GPT, Claude, Gemini, etc. They’ve been generally trained to be OK at many things, but not fantastic at any one thing.

Foundation models are meant to be built upon and augmented with additional layers and adjustments to help them get better at specific tasks. This can be done through approaches such as Retrieval-Augmented Generation (RAG) or prompt engineering (these terms are defined later in this article, if you’re not familiar with them).

The important part is that most adjustments to foundation models happen after they’re trained. While some foundation models allow developers to fine-tune (or further train a pretrained model on a smaller, specialized dataset), they don’t generally have access to change the original pretraining data of the model and can only refine the output.

GenAI: Generative Artificial Intelligence

GenAI refers specifically to the use of AI to create “original” content, typically by predicting content one piece at a time based on learned patterns. “Original” is in quotes in that previous sentence, because anything an AI creates is merely an inference from or remixing of the data it has been given access to.

ChatGPT and DALL-E are both examples of GenAI technologies—capable of generating content in response to a prompt (or directions) given by a user. GenAI can refer to text-based content, but it also includes video, images, audio and more. The main differentiator is that GenAI is creating content, rather than completing a task such as classifying, identifying or similar.

LLM: Large Language Model

LLMs are a specific type of GenAI model created with a focus on understanding and replying to human-generated text. They’re called “large language” models because their training data includes huge amounts of text—often thousands upon thousands of books, millions of documents, writing samples scraped from across the internet and synthetic data (AI-generated content). This makes them especially good at conversations and writing-related tasks such as drafting emails, writing articles, matching tone of voice and more.

Prompt

A prompt is the input we give to an AI model in order to return a response from it. Prompts can be as simple as plain-language questions (like “What are the best restaurants in Toronto?”), or they can be complex, multistep instructions including examples and additional context.

Prompt Engineering

The art of writing prompts in a way that enables the model to complete complex and specific tasks (without changing the model’s training) is known as prompt engineering. As Chip Huyen says in AI Engineering, “If you teach a model what to do via the context input into the model, you’re doing prompt engineering.”

A helpful way to think of it can be that a basic prompt tells the model what to do, while prompt engineering gives the model the context and tools to complete the task as well. This often (but doesn’t have to) includes:

Writing highly detailed instructions, sometimes including a persona (“Imagine you are a professor of history …”) or specific output formats (“Return the response in JSON matching the following example …)
Providing additional information or tools, such as a reference document (“Based on the attached grading scale, review the following essay …”)
Breaking down the request into smaller, chained tasks (“First, review the email for typos. Next, identify any additional steps …” rather than “Correct the following email.”)

Agent

Agents use an AI model as a reasoning engine and enable it to interact with tools or external environments to complete multistep tasks. By default, AI models don’t have live access to external systems or updated data, but an agent can wrap around the model and interact with specific environments (like the internet). This vastly extends the capabilities of a model and can be especially helpful for improving the responses of a model for a specific task.

For example, RAG (Retrieval-Augmented Generation) systems are often implemented with agent architectures, allowing the model to search and retrieve text or write and execute SQL queries within the environment of the new documents provided in the RAG database.

Skill

Skills are the specific “tools” that agents can make use to extend the capabilities of the AI model. For example, Vercel offers and maintains a skill related to “performance optimization for React and Next.js applications,” which is intended to offer agents the specific domain knowledge related to the Next.js framework that’s necessary to write React apps using their technology.

RAG: Retrieval-Augmented Generation

RAG, or Retrieval-Augmented Generation, is a technique that can improve the accuracy of a model’s responses by allowing it to query and retrieve information from a specified external database. Rather than adding content directly to the training data, RAG systems (often with the help of an agent) retrieve additional information from a separate source. This source is usually an intentionally curated collection of files such as past chat logs, software documentation, internal policy files or similar.

RAG tends to be an especially good fit for hyper-specific knowledge, allowing an AI model to answer questions involving information that isn’t generally available (such as “Does Progress Software give their employees the day off for International Women’s Day?”).

What’s Next?

Now that we have a shared vocabulary, we can start to dig a little deeper. In the other articles of this series, we'll be digging deeper into the specifics of how agents and skills work, how to effectively engineer prompts, what hallucinations are (and why they happen), plus much more.

Read the next blog in this series: Tokens, Prediction and Temperature

Progress Team Update: KendoReact Challenge and Nuclia Trial Issues

Kathryn Grayson Nanz — Mon, 22 Sep 2025 17:59:32 +0000

Hey folks! We've gotten some feedback about issues related to the Nuclia trials for the Dev Challenge. First off, I want to apologize – we thought we had adjustments to the trial process cleared and ready in advance of the hackathon, but turns out...not so much. 🙃 Totally a mistake on our end, and we're sorry for the inconvenience and confusion.

That being said, we do have some workarounds ready so you can still participate in the "RAGs to Riches" category!

Some folks have mentioned being unable to start a Nuclia free trial without a corporate email. To fix this, please reach out to me at nanz @ progress.com (or on the Progress Labs Discord) with the email you'd like to use, and I'll manually override the block and start your trial account with whatever email you'd prefer.
There's been some concern about the 14-day free trial timing out before the judging period. There are two ways to solve this: either reach out to me (over email or Discord, as above) and I can manually extend your trial OR just take a short video of your app and include it in the submission template.

Thanks so much, and again – very sorry for any trouble! We're super excited to see what you build, and here to help support however you need.

[Boost]

Kathryn Grayson Nanz — Thu, 11 Sep 2025 11:02:00 +0000

Jess Lee for The DEV Team

Sep 10 '25

Join the latest KendoReact Free Components Challenge: $3,000 in Prizes!

#devchallenge #kendoreactchallenge #webdev #react

Comments 27

4 min read

How I’m Using AI (as an AI Skeptic)

Kathryn Grayson Nanz — Mon, 04 Aug 2025 15:11:19 +0000

As my friends, family, coworkers, and probably several people on the internet already know: I am an AI skeptic – and I’m not particularly shy about it, either. I’m not really sold on the productivity improvement angle, and I’m especially cautious (and critical) when it comes to using it to create content. I’m routinely underwhelmed by emails, blogs, illustrations, logos, and more that were clearly created without human involvement. As an artist and writer myself, those aren’t things that I, personally, am seeking to replace or automate out in my own life.

That said, I also work in tech where AI is (for the time being) borderline inescapable. I also think it’s decidedly lame to write something off without giving it a genuine try. And, if I’m completely honest, I was also just plain curious: what was everyone else seeing in this that I wasn’t? Was there some trick or approach to using it that I just hadn’t mastered yet? I created a ChatGPT account, started using the CoPilot account work had provided for us, installed our Kendo UI AI Coding Assistant, and committed to giving “this AI thing” a real, honest shot for an extended period of time. I’ve been using it for the last couple months, and…well, my opinions are mixed.

There are some places where AI tooling excels and was genuinely helpful. There were many places where it wasn’t suited to what I was asking it to do, and I spent more time fighting with the machine than getting anything actually done. Ultimately, I’m not sold on AI as the do-everything tool it’s often marketed as – however, there are a handful of things it was great at that I have folded into my regular routine. With all that being said, here are the places and ways where I’m using AI (as a certified AI skeptic).

Where I Use AI

Pair Programming

I found (pretty darn quickly) that I do not like when an AI tool writes the code for me: by which I mean literally populating or auto-completing lines of code in my IDE. Vibe coding and I simply do not get along; I found it challenging to follow what was being written, I didn’t remember how I had structured or named things (because I hadn’t actually structured or named them), and it ultimately slowed me down significantly.

What I did find helpful was using the AI like a pair programming partner. My coworker Alyssa wrote about this a while ago, and it was part of what informed my approach and helped me find a middle ground that worked for me. In the past, if I was implementing something new, I’d try and find examples in the docs or a tutorial blog that walked me through it – and I’d almost always have to make some adjustments for it to work in the context of my own project. Now, it’s handy to ask the AI to generate my own step-by-step implementation tutorials, all customized to my exact tech stack and needs.

I also really like using it to replace looking stuff up in the docs. The Kendo UI AI Coding Assistant is particularly useful here – believe it or not, even though I’m the KendoReact Developer Advocate, I don’t actually have every possible prop for all 120+ components memorized yet (I know, I know, I’m working on it). Being able to throw a quick syntax question into the chat sidebar in VS Code is super handy. I’m not a fan of having it write the whole app for me – although it can do that – but it does have a much better “memory” than I do for whether that prop is called theme or themeColor (spoiler: it’s themeColor).

Of course, I’d also be remiss if I didn’t mention using AI for troubleshooting errors in my code – it’s saved me more than a few times, now. However, (like all troubleshooting) the trick here is to not fall down the rabbit hole. It’s shockingly easy to just ask it question after question, following the natural chain it creates and letting it suggest wilder and wilder approaches. By the time you’ve tried 4-5 things it’s suggested with no solution, you’re on the AI version of the 10th page of Google: the answer just isn’t going to be here. In the pre-AI days, I would always suggest to junior devs to set a time limit on their troubleshooting: if you’ve tried for 1-2 hours (max) and made no headway, it’s time to stop Stack Overflow-ing and call in another person. Although the tech is different now, I think the rule still stands; time box your AI query-ing and learn to identify when you’ve hit the point of diminishing returns.

To-do lists and Scheduling

One of the places where I have actually found AI to live up to the productivity hype is to do a braindump of all my tasks and goals at the beginning of each week and let it make a little schedule for me.

I’ll tell it my pre-existing obligations on each day (calls, appointments, etc.), personal to-dos (workouts, chores, social engagements), school assignments (when I was still finishing up my grad school work), and work tasks and ask it to make a structured to-do list for each day of the week. It’s good at grouping mentally similar tasks, reducing the amount of “code switching” you need to do, and it’s also helpful to me to see where I have time set aside for each thing. I was never a time-blocking kind of person; it was just too specific and I always felt like I wasted more time setting up the schedule vs actually completing the work. Outsourcing that work to the AI has been beneficial, and my Monday mornings now usually start with a schedule dump into ChatGPT.

Body Doubling (kind of)

This one is (admittedly) a little embarrassing, but for the sake of honesty I’m going to include it: I like to tell the AI chatbot when I start and finish tasks. I think of it kind of like body doubling, but without having to bother another actual person or actually sync up work schedules. I know we’ve moved into full-on placebo effect here, but something about knowing I have to “report back” when I’ve finished helps keep me on task. Brains are weird.

Summarizing My Own Writing

As I mentioned above, I don’t like to have the AI create content for me – that includes writing, which I do a lot of in my role. Conference talks, videos, blogs, ebooks: I spend a lot of time click-clacking away on my little keyboard, and I’m not terribly keen to outsource that work. Where I have found AI to be helpful in my writing process is to have it read my work and then ask it questions. What was the primary message the author was communicating in this piece? What were the main steps of this tutorial? What was the author’s tone?

Rather than having it check my work for accuracy (it’s decidedly not good at that) or rewrite my words, I like to have it summarize my work back to me so I can make sure I hit all the points I intended to hit and emphasized the right stuff. After all, someone else is probably going to be doing that – even if it’s not as direct as I’m doing here, they’ll be reading the Google AI summary of it or asking a question that some AI will reference my work to answer. It’s a helpful way to confirm that my most important messages are being effectively communicated when I write.

Where I Avoid AI

Writing

Yes, I know I just talked about this a little bit. But even beyond doing literal content creation, I also avoid AI generating outlines or overviews, emails, conference talk descriptions, DMs, etc. It’s just too opinionated, and it has a distinct tone of voice that doesn’t match my own. Plus, I know how I feel when I get an email or read a piece of work that wasn’t written by a human – and it feels bad. If it wasn’t important enough for you to write, then it’s not important enough for me to read.

Image Generation

Look guys, it’s just not good. Maybe it will be good in the future, but that future is not today. It’s always riddled with little mistakes, images that are supposed to look “real” have that kind of soft-edged glazed-over look, and just forget trying to generate anything with text in it. Websites like Unsplash offer high-quality, royalty-free images – just use those.

Research

Until AI can say “I can’t find that”, then I’ll never trust it for research – it will just hallucinate an answer and present it to you like the truth. That goes for everything from actual academic paper research to “what time does this restaurant open?”. I’m simply not interested in spending my time fact-checking a machine. I even switched away from Google because the (inaccurate) AI summaries at the top drove me crazy. Until AI chatbots are capable of admitting that they can’t do something, I’ll be DuckDuckGo-ing my questions (even though that doesn’t exactly roll off the tongue the way Google-ing did).

Vibe Coding

As detailed above. I could maybe see doing it for a small, one-off side project – but if it’s anything you’re going to have to work with again at literally any point in the future, it’s just not worth it. And even for that small, one-off thing, you’re more likely to learn and retain what you’ve worked with better if you write the code yourself.

How are you using AI?

Where have these tools fit into your life? Are you calling on them every day, or just a couple times a week? We’re all still finding our balance as AI tooling works its way into…well, just about everything. I’m not a believer in throwing the baby out with the bathwater, so I’ll keep playing with the new stuff as it comes out. After all, this isn’t an all-or-nothing game: take what works and leave the rest!

Getting Specific About CSS Specificity

Kathryn Grayson Nanz — Mon, 14 Apr 2025 21:01:36 +0000

You likely already knew that some methods of styling will override others when you’re writing CSS...but do you know why?

For example, a style assigned to a class will be preferred over one applied to a base element. Here, the header with the class name will be red, even though we said h2s should all be blue. That’s because CSS weighs classes as more specific than elements, and therefore prefers the red style defined to the class.

<style>
h2 { color: blue; } 
.my-header { color: red; } 
</style> 

<h2 class="my-header">Hello World!</h2>

But there’s more to this than some mysterious rock-paper-scissors game happening somewhere in the browser. In fact, CSS specificity is calculated using points and written using a 4-number notation in which each identifier in a style definition is is tallied. In the 4-number notation, there's a column for inline styles, IDs, classes, and elements.

Let's look at an example:

#nav .menu li a { font-weight: bold; }

Inline Style	ID	Class	Element
0	1	1	2

We would read this specificity level as [0, 1, 1, 2]. That's because there are no [0] inline styles in a CSS file, one [1] ID (the #nav), one [1] class (the .menu), and two [2] elements (the list and the anchor).

When two different styles are applied to the same thing, their specificity levels are compared and the higher one wins. So, in this case, if we wanted to override that example style, we could create another style that added another class [0, 1, 2, 2] – or, we could add an inline style in the HTML [1, 1, 1, 2]. And – for better or worse – !importants will trump everything, no contest.

You may be asking why we don’t just read these as numbers – in that example, one hundred and twelve versus one hundred twenty two or one thousand, one hundred and twelve. After all, if we think about this as a kind of base ten points system, we should just be able to add it all up and read the number from left to right…right?

Unfortunately, it’s not quite that easy. We use this particular notation because sometimes we’ll go over ten in any individual column – like if you were to put twelve classes on something. I don’t particularly encourage writing something that uses that many classes, but but hey – some of you are probably Tailwind users, right? 😉

If we had twelve classes, our specificity might look like [0, 0, 12, 1]. Just reading from left to right, we might be tempted to read that as one hundred and twenty one...but the specificity of something with twelve classes and an element would be different than something that had one ID, two classes, and one element: [0, 1, 2, 1] – which could also, arguably, be read from left to right as one hundred and twenty one. So, the commas are actually in place to discourage us from doing just that.

Specificity with `:is()` and `:where()`

Okay, with that background let's look at a fun real-world example with some new(ish) CSS: :is() and :where(). These are two ways for us to write compound selectors with more clarity and less hassle. In CSS, “compound selectors” is just a fancy way for us to say “styles that apply to more than one element.” This is useful for times when we want to do things like apply the same color to all our headers or zero out the defaults on several different base elements.

For a long time, the way we did this was using comma-separated lists, like this:

h1, h2, h3 { font-weight: bold; }

Now, we can use :is() or :where() to do the same thing, with (actually) a very similar syntax:

is:(h1, h2, h3) { font-weight: bold; }

where:(h1, h2, h3) { font-weight: bold;}

What does this have to do with specificity? Well, when we use :is(), the specificity level is determined by the most specific thing included in the list, but when we use :where() the specificity level is always zero.

That means that: :is(h1, h2, h3) has a specificity of [0, 0, 0, 1] – the specificity level defaults to the highest level included in the :is() list. If we wrote :is( #main, h1, h2, h3), the specificity would be [0, 1, 0, 0] because we included an ID.

If we were to use :where() in those same situations – both with and without the ID – the specificity would be [0, 0, 0, 0] in both cases because :where()s specificity level is always zero. That means that, if we’re feeling particularly fancy, we can use :where() to our advantage to manipulate specificity rules in our code – no matter what we put into our :where() list, it won't be counted towards the specificity level of that style rule.

That's your CSS trivia for today! Go forth and wow your coworkers with your knowledge of web dev minutia.

For anyone participating in the KendoReact Free challenge, I highly encourage you to take advantage of this awesome offer! Weizhi is a great resource and super knowledgable.

Kathryn Grayson Nanz — Fri, 14 Mar 2025 07:52:37 +0000

Offering Free Mentoring to Participants of the Upcoming KendoReact Free Components Challenge (Limited Time Offer)

Weizhi Chen ・ Mar 14

#kendoreactchallenge #kendoreact #telerik #react

[Boost]

Kathryn Grayson Nanz — Thu, 13 Mar 2025 09:04:15 +0000

Join the KendoReact Free Components Challenge: $5,000 in Prizes!

dev.to staff for The DEV Team ・ Mar 12

#kendoreactchallenge #react #devchallenge #webdev

Designer-Developer Collaboration: 2024 Survey Results

Kathryn Grayson Nanz — Wed, 30 Oct 2024 14:27:45 +0000

Perhaps you remember a post I made a few months ago, discussing some of the ways that Progress was talking to designers and developers about the handoff and how they collaborate. That included some in-person conversations, events at conferences, and – last but certainly not least – a survey.

Well, the results from the survey are in and I’m super excited to share them! If you’ve ever been curious about designer / developer communication, pain points, design system usage, and so much more, then you’ll definitely want to check this out.

Of course, I can’t say all that without offering up a little teaser, right? So here are my personal top 3 favorite tidbits from the survey results report:

3. Designers AND developers both think collaboration is smoother when they understand technical aspects of each other’s job.

We asked participants what they thought made for a smooth relationship between designers and developers, and the top three answers were:

Developers are involved in the process earlier.
Designers understood development technical constraints.
Developers knew more about design principles.

We already knew, from those earlier conversations, that devs wanted to be looped in earlier – but it was super interesting to see that both designers and developers thought they could stand to brush up on their knowledge of “the other side” a little bit. So, if you’re a designer, maybe it’s time to read up on some CSS syntax…and if you’re a dev, maybe it’s time to dig into a little color theory. Hey, it couldn’t hurt, right?

2. Half of all respondents have a design system.

32% have an established and maintained design system, while 20% have one that they’ve created recently and are working on building up. Additionally, another 10% have plans to build one in the future, but haven’t gotten started yet. If you were waiting to see if this whole “design system” thing would really catch on before investing the time…I think it’s safe to say that it has.

1. Only 26% are truly happy with the way design gets implemented in the final product.

Not gonna lie – this one kinda hurt to read. When asked if folks were happy with the way a design was ultimately translated into a functional website / application, only about 1 in 4 were. The remainder were split evenly between “Eh, it’s passable” and “Nope, not even close”. All things considered…not a glowing review.

See the rest of the results

You can see the answers to all the questions on our results webpage, but make sure you download the full report to get all the extra data analysis and interesting cross-sections. We think there’s a lot to learn, and it’s packed with helpful tips for improving the collaborative process at your workplace.

Also, if you took the survey: thank you so much! We really enjoyed going through all the responses and seeing what everyone had to say. Keep an eye out for next year’s survey, and – until then – keep creating awesome stuff.

Closing the Designer-Developer Gap

Kathryn Grayson Nanz — Wed, 21 Aug 2024 16:00:44 +0000

This year, we’ve been trying a little something new at conferences with the Progress booth – we’ve been collecting feedback from developers about their experience with the design to dev handoff. With a jumbo pack of sticky notes, a whiteboard, and a dream, our goal was to capture three different aspects of the design to development workflow: what tools people were using, what they liked about their process, and what they wished was different.

The response was incredible; hundreds of developers shared their opinions and experiences with us. We commiserated over common pain points, celebrated wins, and brainstormed what an ideal dev / design workflow could look like. As you might imagine, developers and designers directly working together was an often-recurring topic across these thousands of sticky notes – it’s almost impossible to talk about the handoff without talking about cross-team collaboration. So let’s take a look at the main things developers wish designers knew when it comes to building websites and applications.

1. Developers want to be involved in the design process

Developers we spoke to expressed a strong desire to be included in all phases of the project creation process. There were a great many responses related to cross-team collaboration and developer involvement – even (and especially) in work that isn’t traditionally considered development work. Developers are eager to participate earlier in the process and work together during the planning and design phases. They want to be able to help catch potential issues early and provide deep technical knowledge that can help shape the product or feature set long before any code is written. That might look like being included in scope definition calls, UX brainstorming, or other product planning meetings. In a wide variety of ways and across all different points in the process, devs want to be (as they said in the sticky notes) “involved”, “included”, and “collaborating”.

2. Developers want to share “the same language” with designers

That first wish for developers be more involved did come with a request: that the people they collaborate with also invest the time to understand technical requirements and “speak the same language”. Many responses mentioned the value of non-developers who understood the basics of the development process, as well as the challenges of designs and feature requests that were created without understanding the underlying technical requirements. For those developers already lucky enough to be working with designers, there were several mentions of wishing that those designers understood the development and implementation process more deeply – especially relating to the availability and limitations of components in a component library (when one is being used).

3. Developers want to reduce mid-project changes as much as possible

Lots of teams struggle to define project requirements accurately. There were tons of responses related to the need for clearly defined requirements at the beginning of the project (this included both sticky notes that mentioned liking clearly defined requirements and disliking poorly defined requirements). But it’s not just the kickoff phase of the project that’s challenging; it won’t surprise anyone to hear that scope creep came up quite a bit in the “dislike” category. In fact, 17 responses specifically mention the difficulties of scope / requirements changing mid-project. Whether those changes come from the product team, the design team, the client, or the developers themselves, they inevitably cause significant slowdown and frustration for everyone involved.

4. Developers value designers and want to have more designers on their teams

Many developers are working on under-resourced teams and (rather than focusing on software or other tooling) are wishing primarily for more designers – or any designers at all! Several sticky notes specifically mentioned hoping for some kind of design specialist to be hired (those wished-for roles included UI designers, as well as UX designers and researchers).

In our opinion, that involvement and cooperation between the teams is what’s really going to move the needle, more than any hot new library or fancy new process. The end product can truly shine when there’s a shared space for designers and developers to create together. That also means that both parties have put in the work to learn and respect the other's expertise, process, and workflow.

Building a culture of respect and shared responsibility won’t happen overnight – especially if the design and development departments at a company have historically been isolated. It takes a lot to change the status quo from throwing a design file over a metaphorical wall and hoping for the best, to true synchronicity and collaboration at all stages of the process.

One of the best ways we can start to shift that is by learning more and gaining a more thorough understanding of what all parties are looking for. By identifying those communication gaps, knowledge gaps, and expectation gaps, we can begin to bridge them together.

Want to help with that process and learn more about the current state of the industry? Share your experience and take the State of Designer-Developer Collaboration 2024 Survey. It’s a global survey that aims to shed light on the design handoff process, and the role design systems play in addressing the inherent challenges—and we need your input!

Forem: Kathryn Grayson Nanz

AI Crash Course: Prompt Engineering

Why Engineer Prompts?

The Structure of a Well-Engineered Prompt

Why Does Prompt Engineering Work?

AI Crash Course: Hallucinations

What causes hallucinations?

Reducing hallucinations

Reference Data

Context

Prompting

AI Crash Course: Tokens, Prediction and Temperature

Predictive Language Models

Tokens

Temperature

Bias

AI Crash Course: AI, ML, LLM and More

AI: Artificial Intelligence

ML: Machine Learning

Model

Foundation Models

GenAI: Generative Artificial Intelligence

LLM: Large Language Model

Prompt

Prompt Engineering

Agent

Skill

RAG: Retrieval-Augmented Generation

What’s Next?

Progress Team Update: KendoReact Challenge and Nuclia Trial Issues

[Boost]

Join the latest KendoReact Free Components Challenge: $3,000 in Prizes!

How I’m Using AI (as an AI Skeptic)

Where I Use AI

Pair Programming

To-do lists and Scheduling

Body Doubling (kind of)

Summarizing My Own Writing

Where I Avoid AI

Writing

Image Generation

Research

Vibe Coding

How are you using AI?

Getting Specific About CSS Specificity

Specificity with :is() and :where()

For anyone participating in the KendoReact Free challenge, I highly encourage you to take advantage of this awesome offer! Weizhi is a great resource and super knowledgable.

Offering Free Mentoring to Participants of the Upcoming KendoReact Free Components Challenge (Limited Time Offer)

Weizhi Chen ・ Mar 14

[Boost]

Join the KendoReact Free Components Challenge: $5,000 in Prizes!

dev.to staff for The DEV Team ・ Mar 12

Designer-Developer Collaboration: 2024 Survey Results

3. Designers AND developers both think collaboration is smoother when they understand technical aspects of each other’s job.

2. Half of all respondents have a design system.

1. Only 26% are truly happy with the way design gets implemented in the final product.

See the rest of the results

Closing the Designer-Developer Gap

1. Developers want to be involved in the design process

2. Developers want to share “the same language” with designers

3. Developers want to reduce mid-project changes as much as possible

4. Developers value designers and want to have more designers on their teams

Specificity with `:is()` and `:where()`