How to cut LLM latency with prompt caching
This article explains how prompt caching reduces latency in large language models.
Reducing Latency with Prompt Caching
When running a large language model (LLM) on your own hardware, latency can be a significant concern. One effective way to reduce latency is by utilising prompt caching, a feature available in the llamay inference engine. This article explores how prompt caching works in llamay and how it can be used to improve the performance of your LLM.
To understand how prompt caching works, it is essential to know that a model call in llamay has two phases: PREFILL and DECODE. The PREFILL phase reads the prompt in one pass, while the DECODE phase writes the answer one token at a time. Prompt caching reuses the key-value state computed during the PREFILL phase, allowing repeated prefixes to be skipped. This means that if a prompt has a repeated prefix, the model will not re-read the entire prompt, resulting in reduced latency.
The following code block demonstrates how to run a model with llamay and observe the effects of prompt caching:
# Pull a model from Hugging Face
llamay pull hf:distilbert-base-uncased/distilbert-base-uncased.gguf
# Run the model with a prompt
llamay run -m distilbert-base-uncased -p "This is a test prompt"This example executes the model with the specified prompt. The output will include information on the number of cached tokens, which can be used to measure the effectiveness of prompt caching.
Understanding the Mechanics of Prompt Caching
Prompt caching in llamay is based on a radix prefix tree over token sequences. This data structure allows for efficient reuse of computed key-value states. Additionally, llamay provides explicit control over contexts through the /v1/contexts endpoint. This endpoint allows you to create a new context, which can be forked and reused across multiple requests.
The following code block demonstrates how to create a new context and fork it:
# Create a new context
curl -X POST http://localhost:11435/v1/contexts
# Fork the context
curl -X POST http://localhost:11435/v1/contexts/<id>/forkThis example creates a new context and forks it, allowing the reused prefix to be skipped in subsequent requests.
Managing Contexts and Forks
Managing contexts and forks is crucial to effectively utilising prompt caching. A fork is a page-table copy, not a memory copy, which means that it does not incur significant memory overhead. However, it is essential to note that anything that varies at the top of a prompt, such as a timestamp, can invalidate the entire cache.
The following table compares the different methods of managing contexts and forks:
| Method | Description | Memory Overhead |
|---|---|---|
| Create a new context | Create a new context from scratch | High |
| Fork a context | Create a page-table copy of an existing context | Low |
| Snapshot a context | Create a file that can be used to restore a context | Medium |
In general, forking a context is the most efficient method, as it incurs minimal memory overhead.
Precision and Accuracy Tradeoffs
When using prompt caching, it is essential to consider the precision and accuracy tradeoffs. The -kv flag in llamay sets the numeric precision of the cache, which can be either f32, q8, or vq8. This flag controls the memory and accuracy tradeoff, with higher precision resulting in increased memory usage.
The following code block demonstrates how to set the numeric precision of the cache:
# Run the model with a prompt and set the numeric precision to q8
llamay run -m distilbert-base-uncased -p "This is a test prompt" -kv q8This example sets the numeric precision of the cache to q8, which can result in a balance between memory usage and accuracy.
Monitoring and Optimising Prompt Caching
To monitor and optimise prompt caching, llamay provides the usage.prompt_tokens_details.cached_tokens metric, which reports the number of cached tokens. This metric can be used to measure the effectiveness of prompt caching and identify areas for optimisation.
The following code block demonstrates how to monitor the number of cached tokens:
# Run the model with a prompt and monitor the number of cached tokens
llamay run -m distilbert-base-uncased -p "This is a test prompt"This example includes information on the number of cached tokens in the output, which can be used to monitor and optimise prompt caching.
Common Pitfalls and Limitations
While prompt caching can significantly reduce latency, there are common pitfalls and limitations to be aware of. One of the most significant limitations is that anything that varies at the top of a prompt can invalidate the entire cache. This means that if a prompt includes a timestamp or other dynamic content, the cache may not be effective.
Additionally, prompt caching is not a replacement for response caching, and it is essential to understand the differences between the two. Prompt caching reuses the key-value state computed during the PREFILL phase, while response caching stores the entire response. llamay does not support response caching, and it is essential to use prompt caching effectively to reduce latency.
By understanding the mechanics of prompt caching and managing contexts and forks effectively, you can significantly reduce latency and improve the performance of your LLM. However, it is essential to be aware of the common pitfalls and limitations and to monitor and optimise prompt caching to achieve the best results.