TechTarget Details the Next Wave of Generative AI Technologies Beyond Large Language Models

TechTarget has published an industry overview outlining the emerging generative artificial intelligence technologies designed to complement or succeed current large language models. The report, authored by tech journalist and IT professor Chris Tozzi, identifies logical reasoning systems, real-time learning models, liquid learning networks, and small language models as the primary paradigms positioned to address the limitations of current systems, particularly their lack of real-time learning, reasoning constraints, and high computational demands.

Limitations of Current Large Language Models

Current generative AI relies heavily on static large language models (LLMs), which typically utilize the transformer architecture. While the attention mechanism in transformers allows models to analyze multiple text tokens simultaneously, standard LLMs face structural limitations that prevent them from performing more complex enterprise tasks:

• No capacity for continuous learning: LLMs cannot learn new information continuously. They are limited to generating content based on their original training data. While some conversational services can consider previous prompts during a session, they do so using capabilities external to the underlying model itself. Currently, there is no reliable way to train an LLM in real time to keep its knowledge base continuously up to date.
• Inability to reason: LLMs lack genuine reasoning capabilities and cannot understand cause-and-effect relationships.
• Extensive demand for compute resources: Training and running massive parameter models requires significant computational power and energy, limiting scalable deployment.
• Propensity to hallucinate: LLMs are prone to generating inaccurate information, a challenge that complicates their use in critical enterprise tasks.

Logical Reasoning Systems

Although LLMs cannot perform genuine reasoning, other computer systems can process data based on logic. Logical reasoning is one of the oldest paradigms in artificial intelligence, dating back to the 1950s when it was used to program checkers-playing computers, and supported by specialized programming languages like Prolog.

The primary drawback of logical reasoning systems is their reliance on developers to explicitly define logic rules, making it impossible to anticipate every scenario an AI might encounter. Because of this limitation, logical reasoning will likely never power generative AI on its own. However, these systems can be combined with LLMs, using logic rules to evaluate and refine the text generated by the model to mitigate errors.

Real-Time Learning Models

While static LLMs dominate current headlines, researchers are developing alternative model architectures capable of continuously learning new data.

One prominent example is AIGO, a system designed with an integrated neurosymbolic architecture (INSA). Unlike traditional LLMs, models built on architectures like INSA are capable of continuously adding to their active knowledge base without requiring resource-intensive retraining cycles.

Liquid Learning Networks

Liquid learning networks (LLNs) represent another continuous learning alternative to traditional LLMs. The primary mechanics of LLNs include:

• Real-time parameter modification: Unlike static LLMs, LLNs can dynamically adjust their parameters in real time based on incoming data.
• Continuous adaptation: This architecture allows the model to continuously adapt to new information as it is received.

Historically, LLNs have been used primarily for processing time-series data rather than interpreting open-ended, natural-language queries or generating novel content. However, researchers are exploring how the underlying architecture can be adapted for generative AI use cases to resolve the static knowledge limitations of traditional models.

Small Language Models

To counter the high infrastructure costs and computational demands of LLMs, organizations are increasingly adopting small language models (SLMs) [1.1]. An SLM functions similarly to an LLM but is trained on a smaller, highly curated dataset.

While larger training datasets are typically preferred to support a broad, general-purpose range of use cases, SLMs offer distinct advantages for targeted enterprise deployment:

• Reduced computational requirements: Due to their smaller size and fewer parameters, SLMs require significantly less computing power to train and run.
• Lower rates of hallucination: LLMs often hallucinate when their training datasets are so vast that the system fails to identify the correct connections between input data and training data. SLMs trained on narrower datasets are less prone to these errors.
• Suitability for narrow use cases: SLMs are increasingly positioned as direct LLM replacements for highly specialized tasks, such as answering customer queries about a specific business’s proprietary products or services.