The rise of AI-based applications is accelerating around the world and shows no signs of slowing down. According to IBM data, 42% of companies with more than 1,000 employees are actively using AI in their business, and an additional 40% are testing and experimenting with it.
As AI adoption accelerates, with platforms such as OpenAI’s GPT-4o and Google’s Gemini setting new performance benchmarks, organizations are discovering new applications for these technologies that can provide better results. Faced with new challenges linked to the deployment of technology on a large scale. More and more business workflows are incorporating calls to these AI models, significantly increasing their use. Do the use cases justify the increased spending on the latest models?
Embracing AI also means embracing the use of AI models and paying AI inference costs, at a time when many organizations are in cost-cutting mode. Faced with continued economic uncertainty, rising operational costs and increasing stakeholder pressure to achieve a return on investment, businesses are looking for ways to optimize their budgets and reduce unnecessary spending. The growing costs of AI infrastructure can be a source of tension, as organizations want to remain competitive and leverage the power of AI, while balancing these investments with financial prudence .
To complicate matters further, AI agents, which McKinsey says are the next frontier of GenAI and are largely expected to form the next wave of applications, will significantly increase the use of AI models as they rely on them for ongoing thinking and planning stages. Instead of single API calls to underlying models like those in OpenAI, agentic architectures can make many calls, increasing these costs. How can businesses address rising data costs while still powering the AI applications they need?
Vice President of AI and Search Products at Redis.
Understanding the Cost of AI at Scale
Rapid deployment of AI is driving costs on multiple fronts. Above all, organizations are investing in AI inference, which involves using a trained model to make predictions or decisions based on the inputs provided. Often they rely on APIs from major providers like OpenAI, Anthropic or cloud service providers like AWS or Google and pay based on usage. Alternatively, some organizations run their own inference and purchase or lease GPUs on which they deploy open source models such as Meta’s Llama.
Second, in many cases, organizations want to customize their AI models by “tuning” them. This can sometimes be an expensive process involving data preparation by creating training datasets and requiring computational resources for training.
Finally, building AI applications requires additional components, such as vector databases, which help increase inference by helping to retrieve relevant content from designated knowledge bases and thus improve accuracy and the relevance of AI model responses.
By examining the root causes and drivers of their AI costs, such as inference, training or fine-tuning of AI, as well as additional components such as databases, companies can minimize costs. storage costs and improve the performance of their AI applications.
Maximizing Efficiency with Semantic Caching
Semantic caching is a highly effective technique that organizations deploy to manage the cost of AI inference and to increase the speed and responsiveness of their applications. This involves storing and reusing the results of previous calculations according to their semantic meaning.
In other words, instead of relying on new AI calculations for new queries, a semantic cache can search a database for queries with similar meanings to those that have been asked before, thereby reducing costs. This approach helps reduce redundant calculations and improve the efficiency of applications such as inference or search.
In one particular study, researchers showed that up to 31% of queries to AI applications can be repetitive. Every unnecessary AI inference call incurs avoidable costs, but by implementing a semantic cache, organizations can significantly reduce these calls, reducing them by 30-80%. This method is crucial for creating scalable and responsive generative AI applications or chatbots. This approach not only optimizes costs, but also speeds up response times, helping businesses achieve more with less investment.
Balancing performance and costs
Organizations must optimize their technology stack and operational strategies to be able to deploy cutting-edge AI applications without incurring unsustainable infrastructure costs. This can help them find that crucial balance between performance and cost. Techniques such as semantic caching can play a vital role in this.
For companies struggling to scale their AI applications efficiently and cost-effectively, learning how to effectively manage this would become a key differentiator in the market. The key to enabling businesses to address the spiraling costs of generative AI applications and maximize their value could lie in the AI inference strategy. As generative AI systems become more and more complex, every LLM call must be as efficient as possible. By doing so, customers can access the information they need more quickly and businesses can minimize their financial footprint.
We have featured the best AI website builder.
This article was produced as part of TechRadarPro’s Expert Insights channel, where we feature the best and brightest minds in today’s technology industry. The views expressed here are those of the author and are not necessarily those of TechRadarPro or Future plc. If you would like to contribute, find out more here:
