LLMs today suffer from large-scale inaccuracies, but that doesn’t mean you should give up competitive ground while waiting to adopt generative AI.
Every business technology has a purpose, otherwise it wouldn’t exist. The corporate goal of Generative AI is to produce human-actionable results from technical, business, and linguistic data quickly and at scale to drive productivity, efficiency, and business gains. But this primary function of generative AI – providing a witty response – is also the source of the biggest obstacle to enterprise adoption of large language models (LLMs): so-called “hallucinations”. “.
Why do hallucinations occur? Because, at their core, LLMs are complex statistical matching systems. They analyze billions of data points in an effort to determine patterns and predict the most likely response to a given prompt. But while these models may impress us with the usefulness, depth, and creativity of their answers, tempting us to trust them every time, they are far from reliable. New research from Vectara discovered that chatbots can “invent” new information up to 27% of the time. In a corporate environment where the complexity of questions can vary widely, this figure increases even more. A recent reference from data.world’s AI Lab, using real business data, found that when deployed as a standalone solution, LLMs return accurate answers to most basic business queries. only 25.5% of the time. When it comes to mid-level or expert queries, which stay well within the bounds of typical data-driven enterprise queries, accuracy dropped to ZERO percent!
The tendency to hallucinate may be inconsequential for individuals playing with ChatGPT for small or new use cases. But when it comes to enterprise deployment, hallucinations pose a systemic risk. The consequences range from annoying (a service chatbot sharing irrelevant information during a customer interaction) to catastrophic, such as entering the wrong number on an SEC filing.
As things stand, generative AI remains a gamble for the company. However, it is also a necessary question. As we learned at the first OpenAI Developer Conference, 92% of Fortune 500 companies use OpenAI APIs. The potential of this technology in the enterprise is so transformative that the path forward is extremely clear: start adopting generative AI, knowing that the rewards come with serious risks. The alternative is to insulate yourself from risks and quickly fall behind the competition. THE inevitable increase in productivity is so obvious today that not taking advantage of it could be vital to the survival of a business. So, faced with this illusion of choice, how can organizations integrate generative AI into their workflows, while mitigating risk?
First, you need to prioritize your data foundation. Like any modern business technology, generative AI solutions are only as effective as the data they’re built on – and according to Cisco’s recent report AI Readiness Index, intent outstrips capability, especially on the data front. Cisco found that while 84% of companies worldwide believe AI will have a significant impact on their business, 81% lack the data centralization needed to fully leverage AI tools, and only 21% say that their network has “optimal” latency to support. demanding AI workloads. The same goes for data governance; only three in ten respondents currently have comprehensive AI policies and protocols in place, while only four in ten have systematic processes to address bias and fairness in AI.
As the benchmarking demonstrates, LLMs have a hard enough time finding factual answers reliably. Combine this with poor data quality, a lack of data centralization/management capabilities, limited governance policies and the risk of hallucinations – and the consequences that follow – skyrockets. Simply put, companies with strong data architecture have better, more accurate information and, by extension, their AI solutions are equipped to make better decisions. Working with a data catalog or evaluating internal governance and data entry processes may not be the most exciting part of adopting generative AI. But it is these considerations – governance, traceability and data quality – that could make or break the success of a generative AI initiative. This not only allows organizations to deploy enterprise AI solutions faster and more responsibly, but also to keep pace with the market as technology evolves.
Second, you need to build an AI-trained workforce. The research highlights the fact that techniques like advanced prompt engineering may be helpful in identifying and alleviating hallucinations. Other methods, such as fine-tuning, have been shown to significantly improve the accuracy of LLM, even to the point of outperforming larger, more advanced general-purpose models. However, employees can only deploy these tactics if they have the latest training and knowledge needed to do so. And let’s be honest: most employees aren’t. We’re just over a year since ChatGPT launched on November 30, 2022!
When a major vendor like Databricks or Snowflake releases new features, organizations flock to webinars, conferences, and workshops to ensure they can take advantage of the latest features. Generative AI should be no different. Create a culture in 2024 where training your team in AI best practices is your priority; for example, providing stipends for AI-specific training and development programs or bringing in an external training consultant, such as the work we have done at data.world with Rachel Woods, who serves on our advisory board and founded and runs The AI Exchange. We also promoted Brandon Gadoci, our first data.world employee outside of me and my co-founders, to be our VP of AI Operations. The staggering increase we’ve already seen in our internal productivity is nothing short of inspiring (I wrote about it in this three-part series.)Brandon I just reported it yesterday that we saw an astonishing 25% increase in our team productivity through the use of our in-house AI tools across all roles in 2023! Adopting this type of culture will go a long way in ensuring that your organization is equipped to understand, recognize and mitigate the threat of hallucinations.
Third, you need to stay on top of the thriving AI ecosystem. Like any revolutionary new technology, AI is surrounded by a proliferation of emerging practices, software, and processes aimed at minimizing risk and maximizing value. As transformative as LLMs may become, the wonderful truth is that we are only at the beginning of the long arc of AI evolution.
Technologies that were once foreign to your organization can become mission-critical. The above reference we published Saw LLMs supported by a knowledge graph – a decades-old architecture for contextualizing data in three dimensions (mapping and relating data much like a human brain works) — can improve accuracy by 300%! Similarly, technologies such as vector databases and retrieval augmented generation (RAG) have also gained importance given their ability to help solve the problem of hallucinations with LLMs. Long term, AI ambitions extend well beyond the major LLM vendor APIs available today, so stay curious and agile in your company’s AI investments.
Like any new technology, generative AI solutions are not perfect, and their tendency to hallucinate poses a very real threat to their current viability for large-scale enterprise deployment. However, these hallucinations should not stop organizations from experimenting and integrating these models into their workflows. Quite the contrary, in fact, as so eloquently stated declared by Ethan Mollick, AI pioneer and professor of entrepreneurship at Wharton: “…understanding comes from experimentation. » Imposed risk hallucinations should instead force corporate decision-makers to recognize the issues, take steps to mitigate that risk accordingly, and reap the early benefits of LLMs in the process. 2024 is the year your business should take the plunge.