Article complete

OpenAI Introduces Predicted Outputs for Improving Latency in LLMs

This predictive tool is deemed to be ideal for scenarios where developers know the general structure of the required output.

A man in a suit with digital glasses sits against a green background with binary code, symbolizing AI technology.

Key takeaways

  • OpenAI launches Predicted Outputs to enhance efficiency and reduce latency for GPT-4o models.
  • Utilize 'prediction strings' to expedite response times during repetitive tasks and minor edits.
  • Predicting known outputs can significantly decrease token generation, cutting user latency by approximately 50%.
  • Ideal for developers familiar with expected output structures, streamlining tasks like blog updates and code rewrites.

Takeaways drafted with AI from the article text.

This feature allows users to input a ‘prediction string’, an anticipated segment of the output which significantly reduces response times during repetitive tasks or minor document edits. 

OpenAI has said that since most of the output of an LLM is known before generation, predicting them in advance means generating fewer tokens. It is almost always the highest latency step when using an LLM: as a general heuristic, cutting 50% of your output tokens may cut ~50% user latency. 

According to users who tested it, the typical applications of this feature include updating existing text or making small changes to code, such as renaming variables or rephrasing specific content, where the AI response can closely match the provided input.

Testing of this feature has shown it performs best when there’s a close match between the prediction and the model’s response. If the prediction string diverges from the model’s output, the tool can become less efficient, leading to slower responses and higher costs. 

This predictive tool is deemed to be ideal for scenarios where developers know the general structure of the required output. 

For example, if a user wants the model to reword a document with minor changes or adjust specific lines of code, the predictive input helps streamline responses by letting the model reuse parts of the pre existing text. 

By contrast, the feature isn’t as beneficial for creating unique, original content, where responses cannot easily be anticipated in advance.

OpenAI encourages developers to experiment with the predictive text feature in controlled, predictable tasks to maximize efficiency, particularly in contexts that require frequent minor adjustments.

Published

Spotted an error? Tell our editors. Read our editorial standards and corrections.