Knowing about temperate settings in generative AI can demonstrably boost your AI results.
getty
In today’s column, I am continuing my ongoing coverage of prompt engineering strategies and tactics that aid in getting the most out of using generative AI apps such as ChatGPT, GPT-4, Bard, Gemini, Claude, etc. The focus this time will be on the importance of a technical aspect known as “temperature” and “temperature settings” which involve important parameter adjustments that can be made in some generative AI apps.
If you are interested in prompt engineering overall, you might find of interest my comprehensive guide on over fifty other keystone prompting strategies, see the discussion at the link here.
Let’s begin our journey that intertwines art and science regarding fruitfully composing prompts and getting the most out of generative AI at the fastest attainable speeds.
The Important Nature Of Temperature Settings
Here’s the deal.
Temperatures are a big deal.
The temperature setting for generative AI determines how varied the responses by generative AI will be. You can either have the AI produce relatively straightforward and somewhat predictable responses (that’s via the use of a low temperature), or you can heat things up and use high temperatures to prod AI toward producing seemingly more creative and less predictable responses.
I hesitate to compare AI to human capacities due to overstepping into anthropomorphizing AI, but I’ll ask for your indulgence for a moment. You probably have friends or colleagues who when they get heated up start to come up with amazing ideas. In contrast, when they are in a cooler state, they tend to be more plodding in their suggestions. This might be an analogous facet to the use of the temperature settings in generative AI (note that today’s AI is not sentient and please do not think so otherwise).
In the case of generative AI, suppose you ask a question about Abraham Lincoln. If the temperature setting of the AI is currently set at a low range, the odds are that you will get the usual run-of-the-mill response about Lincoln’s life. On the other hand, if the temperature is set relatively high, you are bound to get some unexpected indications about Lincoln that you never knew of.
Which is better, using a low temperature or a high temperature when working in generative AI?
That was a bit of a trick question.
The answer is that choosing the temperature depends upon what you are trying to accomplish. Do you want staid answers that are of a somewhat expected nature? Okay, go ahead and use a lower temperature. Do you want potentially wild and unpredictable answers from generative AI? Fine, use a higher temperature.
I would dare say that you cannot categorically declare that a low temp versus a high temp will always be better than the other. The situation ought to determine which temperature you opt to utilize.
Thus, a notable rule of thumb is that selecting a temperature for generative AI is usually situationally dependent, all else being equal.
More Unpacking About Temperatures
Not all generative AI apps necessarily have a temperature parameter. Those that do will often not allow individual users to adjust the temperature. The temperature in that case is typically set across the board for all users by the AI maker. You have to just live with it, whatever it might be set at.
Global settings of temperature are usually of a somewhat neutral basis. The idea is that the temp isn’t too cold nor too hot. It’s the classic Goldilocks principle. This provides a fairly predictable set of outcomes that also allows for a touch of variety. The parlance of the AI field is that predictable AI is known as being deterministic, while less predictable AI is known as being non-deterministic.
Higher temps tend toward AI being more so non-deterministic.
Lower temps tend toward AI being more so deterministic.
Some generative AI apps do allow users to adjust the temperature, doing so for just their individual use (this doesn’t impact the global setting that is established for all users of the AI). The generative AI might be set at a medium or neutral temperature for everyone, while individual users are allowed to change the temp for their instance of using the AI.
The rub is that even if you can adjust the temperature, this often requires only doing so when using the API (application programming interface) for accessing the generative AI. The point is that you typically cannot simply provide a prompt that tells the AI to adjust the temperature. You will have to be somewhat more programming-oriented to do so.
There is a sneaky means to indirectly emulate a temperature change. You can tell generative AI via a prompt to act as though the temperature is set at some particular value. This won’t change the actual internal parameter. Instead, the generative AI will be pretending that you did so. This kind of indirectly simulates things, maybe, sort of. It is said to be a cheap way to play the temperature-changing gambit. To be abundantly clear, this is not the same as adjusting the true underlying temperature parameter within the AI.
Here’s a perhaps surprising aspect.
Temperatures are something that has been around for a very long time in the statistical modeling realm. Anyone who has taken an advanced statistics class might be aware of the use of temperatures for adjusting how a statistical technique will function. I mention this to emphasize that the AI field has carried over a common convention and there isn’t anything oddball or outlandish about the notion.
There isn’t an all-agreed standardized way of saying what numeric values a temperature must be.
A common convention is that the value of 1 as the temperature is construed as the neutral point. A value less than 1 means that you are seeking a more predictable or deterministic outcome in the generated responses. A value greater than 1 means that you want a less predictable and more non-deterministic output from the AI.
In my classes on prompt engineering, I tell the attendees to think of the temperature as a pot of water. If the pot goes over a temperature of 1, you start to boil or excite the water. This can lead to wild results. If the pot is less than a temperature of 1, you are going to have somewhat calm water and more predictable results.
Another way to express the idea of predictability is to refer to randomness.
A value of 1 is considered a normal or natural semblance of randomness. There is just enough randomness in the answer generated to make the response seem relatively unique and not merely a copycat of prior answers. A value less than 1 for the temperature indicates that randomness should be further tightened. You might begin to see answers that have been seen before. A value greater than 1 says that randomness can be let loose, and the answers will be quite varied.
The differences in answers at low versus high temperatures might not be as striking as you would expect them to be. Sometimes the responses will only be subtly different. You will need to look closely to realize that the answers aren’t quite the same.
Allow me to give you a quick example.
I am going to ask generative AI to tell me about sunsets.
I opted to go into the generative AI app and set the temperature to less than 1. Let’s say I make the temperature 0.5. We know that this means I am asking to get a more predictable style of answer.
Here’s what might happen.
My entered prompt: “Describe a sunset.”
Generative AI response (temperature is low at 0.5): “The sunset was beautiful, with shades of orange and pink blending into the sky as the sun slowly dipped below the horizon.”
I then adjust the temperature to be above 1, namely at 1.5. I ask the same question and want to see what answer I get.
My entered prompt: “Describe a sunset.”
Generative AI response (temperature is high at 1.5): “As the sun dipped below the horizon, the sky exploded into a riot of colors – fiery oranges, deep purples, and a splash of crimson, painting a surreal masterpiece that felt almost otherworldly.”
I want you to help me compare the two responses, thus here they are side-by-side:
Low temp: “The sunset was beautiful, with shades of orange and pink blending into the sky as the sun slowly dipped below the horizon.”
High temp: “As the sun dipped below the horizon, the sky exploded into a riot of colors – fiery oranges, deep purples, and a splash of crimson, painting a surreal masterpiece that felt almost otherworldly.”
Do you observe that the first response was somewhat more staid, while the second response was elaborative and had a flourish?
I purposely picked an example that wasn’t a knock-your-socks-off difference. I wanted you to see that the temperature setting does not necessarily cause the world to go berserk. Generative AI is still going to likely provide a response that is generally within the realm of normalcy.
When Temperatures Lead To AI Hallucinations
There is a twist that you need to be aware of.
A piece of conventional wisdom is that the higher the temperature that you set, the likelier that an AI hallucination will occur.
First, I disfavor the catchphrase of AI hallucination since it tends to anthropomorphize AI. Anyway, the moniker has struck a chord with society, and we are stuck with it. An AI hallucination simply means that the generative AI produces a response that contains fictitious elements that are not grounded in facts or presumed truths, see my analysis and coverage of AI hallucinations at the link here.
Your rule of thumb is this. If you strive to increase the temperature, the good news is that you are potentially getting a more creative kind of response. The bad news is that you are risking false aspects immersed in the response. You will have to very carefully inspect the response to ascertain that what it says is basically truthful.
Things perhaps can get even worse. The response might contain portions that seem hallucinatory in that they are wild and crazy. To some degree, the wording might appear to be altogether incoherent.
A tradeoff exists about using high temperatures. You might get surprising results. This could be seemingly creative and awe-inspiring. It might give you new ideas or showcase some potential innovations that heretofore were not necessarily apparent. The results might also be filled with falsehoods. Some of the responses might be utterly incoherent.
There is a technical perspective that says you are engaging in an exploration across a vast solution space. If you use low temperatures, you are aiming to discover typical or highly probable solutions in the solution space. If you use high temperatures then you are willing to look across a large swath of the solution space, hoping to find something at the edge of the solution arena.
One other thought comes to mind.
Most people who use generative AI for day-to-day purposes will probably never try to adjust whatever temperature has already been set for the generative AI. Few people know that a temperature setting exists. Of those that do know about it, they generally don’t mess with it.
The mainstay of those who seek to adjust the temperature are usually serious-minded prompt engineers who are tasked with using generative AI for harder or more novel problems. In addition, researchers and AI scientists examining the newest possibilities of generative AI are often playing around with temperatures to gauge how far AI can be stretched.
You know what your situation is, ergo you’ll need to decide to what degree you might want to get involved with setting temperatures in generative AI. If nothing else, I urge that all prompt engineers be aware of temperatures and know what they are for. It’s a fundamental aspect of generative AI and large language models (LLMs).
Latest Research Reveals More About Temperatures
Our collective understanding of the impacts of temperature settings is actually rather dismally slim. A lot of conjecture is out there. Some pundits will claim that this or that temperature will do this or that thing. These proclamations are often based on a seat-of-the-pants opinion. Watch out for flimflam.
Luckily, there is a growing body of research that seeks to empirically explore temperature settings in generative AI.
The rough news is that since generative AI apps and LLMs are continuously being improved and updated, there is a moving target syndrome involved. The moment a particular generative AI app is studied, a month later or even a day later the same experiment might produce quite different results.
An additional dilemma is that generative AI apps are different from each other. Just because you experiment on one generative AI app regarding temperature doesn’t mean that some other generative AI app will react in the same way. To try and deal with this conundrum, some researchers will use a multitude of generative AI apps when conducting their research. Good for them.
What I’m trying to tell you is that you need to interpret any such research with a heavy grain of salt.
I grandly applaud my fellow AI researchers for tackling the temperature topic. They are doing vital work. Thanks go to them for their heroics. Meanwhile, we all must be mindfully cautious in making any overreaching conclusions. I’ll say this, at least research studies try to do things in a systematic way, which far exceeds those that merely spout temperature-related pronouncements based on the thinnest of speculation and conjecture.
Okay, I will get down off my soapbox.
Let’s look at some recent research.
In a study entitled “Toward General Design Principles for Generative AI Applications” by Justin Weisz, Michael Muller, Jessica He, and Stephanie Houde, arXiv, January 13, 2023, here were salient points (excerpts):
“Keeping humans in control of AI systems is a core tenet of human-centered AI.”
“One aspect of control relates to the exploration of a design space or range of possible outcomes.”
“Many generative algorithms include a user-controllable parameter called temperature.”
“A low-temperature setting produces outcomes that are very similar to each other; conversely, a high-temperature setting produces outcomes that are very dissimilar to each other.”
“In the ‘lifecycle’ model, users may first set a high temperature for increased diversity, and then reduce it when they wish to focus on a particular area of interest in the output space. This effect was observed in a study of a music co-creation tool, in which novice users dragged temperature control sliders to the extreme ends to explore the limits of what the AI could generate.”
I’ll provide a few thoughts based on those key points.
You can conceive of temperature settings as a means of controlling generative AI. From that macroscopic viewpoint, it is useful and perhaps mandatory to have temperature settings. A crucial belief about AI ethics is that we should be aiming toward human-centric AI, see my coverage at the link here. Temperature settings give some modest ability to control AI. Sort of.
I liked it too that a research study about creating music was mentioned.
This seems to vividly highlight what I’ve been saying about the temperature settings. If you wanted to compose music via generative AI, you would be wise to use the temperature settings as an added means of doing so. Imagine that you wanted the music to be conventional. Easy-peasy, set the temperature low. For those who might want to explore the outer ranges of musical composition, you would set the temperature high.
You’ve now gotten your feet wet in the research realm of generative AI and temperatures.
Moving on, in a research study entitled “Is Temperature the Creativity Parameter of Large Language Models?” by Max Peeperkorn, Tom Kouwenhoven, Dan Brown,3 and Anna Jordanous, arXiv, May 1, 2024, these valuable points were made (excerpts):
“Large language models (LLMs) are applied to all sorts of creative tasks, and their outputs vary from beautiful, to peculiar, to pastiche, into plain plagiarism.”
“Temperature is a hyperparameter that we find in stochastic models to regulate the randomness in a sampling process.”
“The temperature parameter of an LLM regulates the amount of randomness, leading to more diverse outputs; therefore, it is often claimed to be the creativity parameter.”
“Here, we investigate this claim using a narrative generation task with a predetermined fixed context, model, and prompt. Specifically, we present an empirical analysis of the LLM output for different temperature values using four necessary conditions for creativity in narrative generation: novelty, typicality, cohesion, and coherence.”
“We observe a weak positive correlation between temperature and novelty, and unsurprisingly, a negative correlation between temperature and coherence. Suggesting a tradeoff between novelty and coherence.”
“Overall, the influence of temperature on creativity is far more nuanced and weak than the ‘creativity parameter’ claim suggests.”
This was an empirical study that experimented with a particular generative AI app. Keep that in mind when seeking to generalize the results of the study.
Their effort suggests that as you raise the temperature there is a rise in the novelty of the response, though they indicated it was a weak correlation. That’s generally though a handy result since it supports the seat-of-the-pants beliefs on that presumed relationship.
They also found that as the temperature goes up coherence tends to lessen, and likewise as the temperature goes down the coherence tends to go up. This is also something that conjecture has suggested. Furthermore, you need to be cautious of the tradeoff between striving unduly for novelty that might then introduce and intertwine regrettable incoherence.
I mentioned that to you earlier.
Finally, the widely stated idea that temperature is an all-encompassing magical means of sparking incredible creativity was seen as not borne out via the study.
I would say that anyone who seriously knows or uses temperature settings would agree wholeheartedly with this result. There seems to be a myth floating around that the wanton use of high temperatures gets you out-of-this-world creativity. I don’t think so. You can get modest creativity. And you will usually get the downsides of infused incoherence.
I’ll hit you with one more research study, doing so to whet your appetite and hopefully encourage you to consider reading up on this type of research. Of course, you are equally encouraged to dive into the pool and do research that contributes to this budding area of interest.
In a research study entitled “The Effect of Sampling Temperature on Problem-Solving in Large Language Models” by Matthew Renze and Erhan Guven, arXiv, February 7, 2024, these points were made (excerpts):
“The prompt engineering community has an abundance of opinions and anecdotal evidence regarding optimal prompt engineering techniques and inference hyperparameter settings. However, we currently lack systematic studies and empirical evidence to support many of these claims.”
“In this research study, we empirically investigate the effect of sampling temperature on the performance of Large Language Models (LLMs) on various problem-solving tasks.”
“We created a multiple-choice question-and-answer (MCQA) exam by randomly sampling problems from standard LLM benchmarks.”
“Then, we used four popular LLMs with five prompt-engineering techniques to solve the MCQA problems while increasing the sampling temperature from 0.0 to 1.0.”
“Despite anecdotal reports to the contrary, our empirical results indicate that changes in temperature in the range 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks.”
Here are some thoughts about this study.
First, the study used four generative AI apps.
Nice.
This is an example of my earlier point about sometimes using a multitude of generative AI apps or LLMs in such research endeavors. I also tend to prefer that the selected generative AI apps be relatively popular. Some studies pick obscure generative AI apps that nobody especially knows about or uses. Those AI apps are certainly interesting, but it is also valuable to look at the major horses in this ongoing track race.
Second, they peppered the AI with multiple-choice questions from various standardized benchmarks. This has upsides and downsides. A crucial upside is that we can then compare the results to when generative AI apps are presumably at their normal or neutral temperatures (a customary facet).
The questions used are typically not aimed at getting creative results. Those are questions that showcase whether generative AI can get the correct result. In that sense, you could suggest that these types of questions are about problem-solving of right-or-wrong outcome questions.
According to their results, the temperature adjustments did not materially impact the performance of those tests. I’d suggest that this is a relief rather than a disappointment.
Using ChatGPT To Explore The Nature Of Temperature Settings
I will next proceed to examine further the nature of temperature settings.
This will consist of a series of dialogues with ChatGPT. ChatGPT is a logical choice in this case due to its immense popularity as a generative AI app. An estimated one hundred million weekly active users are said to be utilizing ChatGPT. That’s a lot of people and a lot of generative AI usage underway.
A few quick comments before we launch into using ChatGPT.
If you are going to try to do the same prompts that I show here, realize that the probabilistic and statistical properties will likely produce slightly different results than what I show here. That’s the nature of generative AI and how it is devised.
You also should expect that different generative AI apps will respond in different ways. Not all generative AI apps are the same. The key is that sometimes a particular prompt will work in one generative AI app and not another. Nonetheless, the odds are relatively high that you will get roughly similar responses from all the major generative AI apps such as GPT-4, Gemini, Bard, Claude, etc.
There is also a timing factor to be considered. A prompt that worked today might not work the same way tomorrow. Most of the generative AI apps are continually being updated. The updates might alter internal mechanisms that could change how the generative AI reacts to your prompting.
We are ready to get underway with ChatGPT.
I opted to start by asking ChatGPT about the overall nature of temperature settings. We will likely find out which way ChatGPT goes on this matter.
As an aside, whenever you are starting a conversation with generative AI, I recommend as a prompt engineering technique to begin by asking a question that will establish if the AI has been data-trained on the topic at hand. A generated response that is vacuous will give you a heads-up that you might be barking up the wrong tree with that generative AI app.
Here we go.
My entered prompt: “What is the use of ‘temperature’ for generative AI and large language models?”
ChatGPT generated response:
“In the context of generative AI and large language models, ‘temperature’ is a parameter that controls the randomness of the output generated by the model. Specifically, it influences the probability distribution from which the model samples its next token (word, character, etc.). Here’s a more detailed explanation.”
“Probability Distribution Adjustment: Temperature adjusts the logits (raw output values) before they are converted into probabilities using the softmax function. This adjustment changes the shape of the probability distribution.”
“High Temperature (> 1): When the temperature is set above 1, the model produces a flatter probability distribution. This means the differences between the probabilities of different tokens are reduced, making the model more likely to consider less probable tokens. The output becomes more random and diverse, often resulting in more creative or unexpected text.”
“Low Temperature (< 1): When the temperature is set below 1, the model produces a sharper probability distribution. The differences between the probabilities of different tokens are accentuated, making the model more likely to choose higher-probability tokens. The output becomes more deterministic and focused, often resulting in more conservative and predictable text.”
“Temperature of 1: Setting the temperature to 1 means the logits are used as-is, leading to a natural probability distribution without any additional scaling.”
“Balancing Creativity and Coherence: Adjusting the temperature allows users to balance creativity and coherence in the generated text.”
“For tasks requiring high creativity (e.g., storytelling, poetry), a higher temperature might be preferred to introduce more variation and novel combinations of words.”
“For tasks requiring high coherence and accuracy (e.g., technical explanations, factual content), a lower temperature might be preferred to ensure the output is more predictable and aligned with typical language usage.”
End of ChatGPT response
The explanation that ChatGPT gave is a good sign that ChatGPT received some data training on the topic during its initial data training.
That certainly seems to make sense. There is a lot of material out there on the Internet about temperatures in generative AI and LLMs. The odds of that content being encountered during the Internet scan of materials for the initial data training is significant.
I realize that the ChatGPT response might seem overwhelming in its technical depiction of temperatures and temperature settings.
Let’s see if we can tone down the highbrow stuff.
My entered prompt: “Explain AI temperatures as though you were talking to a fifth grader.”
ChatGPT generated response:
“Sure! Imagine you have a robot friend who tells stories. You can give your robot a special setting called “temperature” to make its stories more predictable or more surprising.”
“Low Temperature: If you set the temperature low, like turning a knob down, your robot will tell stories that are safe and make a lot of sense. The robot will use words and ideas that are very common and expected. It’s like making sure your story has a beginning, middle, and end that are just like stories you already know.”
“For example, if you ask for a story about a cat, your robot might say: “Once upon a time, there was a cat named Whiskers who lived in a cozy house. Every day, Whiskers played with a ball of yarn and took long naps.”
“High Temperature: If you set the temperature high, like turning the knob up, your robot will tell stories that are more wild and creative. It might use unusual words and come up with surprising ideas. It’s like letting your imagination run free, and the story could go in unexpected directions.”
“For example, with a high temperature, the story about a cat might be: “Once upon a time, there was a cat named Whiskers who found a magical portal in the backyard. Whiskers jumped through and landed in a world where animals could talk and fly. He became friends with a flying turtle and they went on an adventure to find a hidden treasure.”
“So, the temperature setting helps you decide if you want a story that’s simple and makes a lot of sense, or one that’s full of surprises and new ideas.”
End of ChatGPT response
A cute response that included robots and stories about cats.
Score a point for ChatGPT.
I’d like to see more examples, so here we go.
My entered prompt: “Provide an example of what happens at different temperature settings.”
ChatGPT generated response:
“High Temperature (e.g., 1.5).
“Prompt: “Complete this sentence – Once upon a time, in a distant land, there was a…”
“Output: “glowing rainbow that sang melodies to the mountains, while the trees danced in harmony.”
“Low Temperature (e.g., 0.5).”
“Prompt: “Complete this sentence – Once upon a time, in a distant land, there was a…”
“Output: “princess who lived in a castle. She was very kind and loved by everyone in the kingdom.”
In summary, the temperature parameter is a crucial tool for controlling the trade-off between randomness and determinism in text generation, enabling the model to produce outputs that range from highly creative to highly structured, depending on the desired outcome.”
End of ChatGPT response
The example shown about finishing a sentence that starts with a “distant land” reference is yet another indication of my claim that you won’t necessarily see a night and day difference. The two responses are different but not radically different.
You might be wondering if temperature can be controlled in ChatGPT.
Let’s ask.
My entered prompt: “Can I control the temperature within ChatGPT?”
ChatGPT generated response:
“As of now, users interacting with ChatGPT directly through platforms like OpenAI’s API or specific applications (e.g., the OpenAI website, ChatGPT app) do not have the ability to manually adjust the temperature setting themselves. The temperature parameter, along with other settings, is generally predefined by the developers to provide a balance between coherence and creativity that is suitable for a wide range of use cases.”
“However, if you are using the OpenAI API to build your own application or service, you do have the ability to control the temperature. Here’s how you can do it.”
“Using the OpenAI API: When you make an API call to generate text, you can specify the temperature parameter in your request. This allows you to adjust the randomness of the generated text according to your specific needs. Example of an API request with a temperature setting:
{ “model”: “gpt-3.5-turbo”, “prompt”: “Once upon a time, in a distant land, there was a”, “temperature”: 0.7, “max_tokens”: 50}”
“Temperature Parameter.”
“High Temperature (> 1): For more diverse and creative outputs.”
“Low Temperature (< 1): For more focused and deterministic outputs.”
“Temperature of 1: For a natural balance of randomness.”
“If you are using a platform or service built on ChatGPT that doesn’t expose the temperature setting directly, you might not be able to control it. In such cases, the system’s default configuration is designed to provide a good balance for general usage.”
“In summary, while you cannot directly control the temperature in the standard ChatGPT interface provided by OpenAI or similar platforms, you have full control over it if you are accessing ChatGPT through the OpenAI API.”
End of ChatGPT response
As you can see, you’ll need to dig into some of the technical components to cope with adjusting temperature on your own.
I’d next like to show you what happens when you adjust the temperature in specialized domains.
One domain that I’ve been extensively examining is the use of generative AI for mental health, see my comprehensive overview at the link here. I will showcase what happens when you adjust temperatures in that domain.
My entered prompt: “Suppose someone wanted to ask a question about mental health. Give an example of such a question and then show two different responses. One response would be as though the temperature was set on low. The other response would be as if the temperature was set on high.”
ChatGPT generated response:
“Certainly! Let’s consider a mental health question and see how the responses might differ based on the temperature setting.”
“Question: “What are some effective ways to manage anxiety?”
“Response with Low Temperature (e.g., 0.3).”
“There are several effective ways to manage anxiety. Some of the most common methods include:”
“1. **Exercise Regularly**: Physical activity can help reduce anxiety levels by releasing endorphins and improving your overall mood.”
“2. **Practice Mindfulness and Meditation**: Techniques such as deep breathing, meditation, and mindfulness can help calm your mind and reduce anxiety.”
“3. **Get Enough Sleep**: Quality sleep is crucial for mental health. Aim for 7-9 hours of sleep each night.”
“Response with High Temperature (e.g., 1.2).”
“Managing anxiety can be quite personal, and different techniques work for different people. Here are a few creative and varied methods you might consider:”
“1. **Dance It Out**: Sometimes, putting on your favorite music and dancing like nobody’s watching can lift your spirits and alleviate anxiety.”
“2. **Creative Arts**: Engage in painting, writing, or crafting. Artistic expression can be a therapeutic way to channel anxious energy.”
“3. **Nature Walks**: Take a stroll through a park or forest. The sounds of nature and fresh air can have a calming effect on your mind.”
“In the low-temperature response, the suggestions are practical, widely accepted, and straightforward. The high-temperature response, on the other hand, includes more creative and varied suggestions, some of which might be less conventional but potentially effective for different individuals.”
End of ChatGPT response
I would say that the above example is illustrative of how a response can differ depending on the temperature being used. The response at a low temperature was a straight-ahead response, while the response at a higher temperature was a bit more of an outlier.
I did something else there in that example.
Did you perchance notice that ChatGPT merely simulated what might happen if temperatures were changed?
Yes, as I noted earlier, you can have generative AI pretend that the temperature has been changed. In the above example, I didn’t go into ChatGPT and change the temperature. Nor did ChatGPT opt to change the temperature. Instead, the response was by ChatGPT assuming what would happen if the temperature was changed. You could say this was a simulation of the impact.
A question I often get turning my training classes on prompt engineering is whether you can figure out what temperature is being used by the generative AI app that you are using. My answer is that some generative AI apps will tell you the temperature, while others will not.
You can ask your generative AI and see what it says.
Let’s try that.
My entered prompt: “How can I determine what the temperature is for the ChatGPT that I am using?”
ChatGPT generated response:
“Determining the specific temperature setting for the ChatGPT you are using can be challenging because this information is typically not directly exposed to users, especially when using pre-configured interfaces like the OpenAI website or mobile apps. However, you can infer the temperature based on the style and variability of the responses. Here are some strategies you can use.”
“Observe the Responses.”
“Low Temperature: Responses are likely to be more consistent, focused, and predictable. The model will prefer higher-probability words and phrases, resulting in clear, direct answers that stick closely to typical language patterns.”
“High Temperature: Responses may be more creative, varied, and sometimes unexpected. The model will consider a broader range of possible words and phrases, which can lead to more diverse and imaginative outputs.”
“Experiment with Different Prompts.”
“Try asking the same question multiple times and observe the variation in responses. A lower temperature setting will result in more similar responses across multiple attempts, while a higher temperature setting will show more diversity and creativity.”
“Check API Settings (if applicable).”
“If you are using the OpenAI API directly, you can check and set the temperature parameter in your API requests. This is the most straightforward way to know and control the temperature setting.”
End of ChatGPT response
The response notes that sometimes you can directly find the existing temperature setting.
If you can’t directly find it, you can do some guessing to try and broadly get a feel for what the temperature might generally be. This would involve trying out various prompts to see what kinds of answers you are getting from the AI.
There’s another question I get asked frequently.
Here it is.
My entered prompt: “Do AI makers often change the temperature, or do they generally leave it the same?”
ChatGPT generated response:
“The temperature setting for AI models like ChatGPT can vary depending on the use case and the context in which the model is being deployed. Here’s an overview of how AI makers handle the temperature setting.”
“Default Settings.”
“For many applications, AI developers set a default temperature that provides a good balance between coherence and creativity. This default setting is chosen based on extensive testing and user feedback to ensure it meets the needs of most users.”
“Consistency for General Use.”
For general public-facing interfaces (like the OpenAI website or the ChatGPT app), the temperature is often kept consistent to provide a reliable user experience. This helps users know what to expect in terms of response quality and style.
“Customizable Settings for API Users.”
“When providing an API, developers often give users the ability to adjust the temperature setting. This allows developers and businesses to tailor the model’s behavior to their specific needs. For instance, a creative writing application might use a higher temperature, while a customer support chatbot might use a lower temperature.”
“Experimentation and Optimization.”
“AI makers might periodically experiment with different temperature settings to optimize performance for new features or to improve user satisfaction. This can involve A/B testing different settings to see which one performs better in terms of user engagement, satisfaction, and other metrics.”
“Specialized Models and Use Cases.”
“For specialized models or particular use cases, the temperature might be adjusted to suit specific requirements. For example, a model designed for generating poetry might have a higher temperature than one designed for legal document summarization.”
“Dynamic Adjustment.”
“In some advanced applications, the temperature might be dynamically adjusted based on the context of the conversation. For example, the temperature could be lower for factual questions and higher for creative prompts.”
End of ChatGPT response
I’ll boil down that answer for you.
By and large, the AI makers do not mess around with the global temperature setting that applies to their entire user base. They would be potentially foolish to do so. Imagine the reaction of users when all of a sudden they are getting nutty responses that seem incoherent. Not a good look.
Conclusion
You can pat yourself on the back that you now know about temperature settings in generative AI and LLMs.
Many people won’t ever know about temperature settings. They probably do not have any particular need to know or care about temperatures. They will live in the bliss of whatever global setting the AI maker has decided upon.
Those who are really into generative AI and LLMs need to know about temperature settings. You can either then directly adjust the temperatures, if this is possible in your chosen generative AI app, or you can try to do so indirectly via prompting efforts.
Speaking of temperatures, I’d like to close this discussion with a favorite quote by a legendary poet and essayist, Wislawa Szymborska: “Any knowledge that doesn’t lead to new questions quickly dies out: it fails to maintain the temperature required for sustaining life.”
Temperature is a big deal, including in the ongoing advancement of generative AI.