Site icon

Beneath the Apps: Understanding the AI Stack

Alana Coopersmith

There is no denying that the widespread use and integration of Artificial Intelligence (AI) is ramping up. Businesses are increasingly incorporating Large Language Models (LLMs) and non-LLMs into their daily functions and products, whether it is an AI image generator, an AI customer service representative, an AI advertising strategy algorithm, or an AI-driven data collection capability, just to name a few functional examples. While AI’s growing prominence is no secret, the technology itself is less obvious. Beyond the public-facing tools sits a complex technological ecosystem that makes AI possible. This ecosystem is understood as the “AI Stack.” Within the stack, there are somewhat well-delineated layers that each meaningfully contribute to the user end product, whether that be a customer service bot, your Spotify Wrapped report, or a synthesized summary of an internet search query. To really understand how AI works, and how to effectively govern it, we must examine the AI Stack.

What is the AI Stack?

AI is not merely a website that provides a virtual conversationalist with (sometimes flawed) answers to your questions – it is a complex technological system, built upon a multifaceted and sophisticated stack of tools and technologies consisting of hardware, algorithms, and data sets. These numerous and complicated tools and technologies live and work together to develop, deploy and maintain robust AI systems. Despite some overlap, these tools and technologies are mostly categorized into neat “layers.” These layers, put together, form the “AI Stack.”

There’s no single, universally agreed-upon way to slice the AI Stack, but at a high level, there are typically at least  four major layers: the Compute Layer, the Model Layer, the Data Layer, and the Application Layer. Each of these layers contains its own dense ecosystem of components, stakeholders, and concerns – “minilayers” within the larger ones. Consider the Compute Layer, where the physical and digital infrastructure necessary to build, train, and operate AI systems live. One significant component of the Compute Layer is Advanced Microchips. While chips belong within the Compute Layer as a key component of the necessary infrastructure within the Stack, they themselves sit atop their own intricate supply chain, complete with mining, manufacturing, and global distribution processes.The same goes for data centers. Getting into the weeds of these extraordinarily important technologies is crucial, but is not the focus of this paper. For now, we explain the four macro-layers and identify the core components and players within each.

For policymakers, legal practitioners and national security professionals, understanding this layered view of AI isn’t just academic; it’s essential. Each layer of the stack involves different technologies, industries, and players, each performing specific roles. Governments seeking to advance human safety, individual rights, economic growth, or national security objectives have different parts of the stack and different players that fall within their regulatory reach. Understanding what each layer of the stack does, delineating the various components, and identifying some key players within each layer is essential for governments around the world to properly govern the newest, rapidly developing, and largely unknown technological frontier. 

Compute Layer

The Compute Layer of the AI Stack is the foundational level of AI development. Here, we find the infrastructure needed to actually build, train, and operate AI capabilities at scale, like hardware, physical infrastructure, digital infrastructure, and energy.

The Compute Layer includes components such as:

These parts form the backbone of all AI development. Computing technologies, like CPUs and GPUs, are the processors that enable training and inference tasks that are vital for model development and deployment. Storage solutions, such as solid-state drives and data storage systems, house large datasets and model artifacts, guaranteeing quick access and retrieval capabilities for seamless data and model management. High-speed networking infrastructure paves the way for efficient data flow and coordination among disparate components, fostering a cohesive and responsive AI ecosystem. Housing all of these tools and technologies are data centers – specialized facilities dedicated to the storage, processing, and dissemination of data. To build data centers, there must be enough land for the building itself, water for its cooling processes, and energy for power generation. 

Within the Compute Layer, we see some major players. For example, Nvidia and the Taiwan Semiconductor Manufacturing Company dominate the market for Advanced Microchips and processing units. Amazon Web Services (AWS), Microsoft, and Alphabet (Google) are the largest cloud infrastructure suppliers. 

From the ground-level land, water, and energy sources to the high-level cloud computing and network infrastructure, the Computing Layer is dynamic and essential. Without the computing layer, nothing else in the stack can operate. This layer is not only the technical cornerstone, but it is also a  growing focus of economic and national security policy as we face supply chain dependence for critical minerals and rare earth elements needed for computing, as well as energy demand vulnerabilities in the United States.

Algorithm Layer

Sitting on top of the compute layer is the algorithm layer. This includes AI models themselves, as well as the mathematical and engineering tools that enable those models to learn, adapt, and perform. 

Here, you find: 

Within the algorithmic layer, we see multiple types of models, most commonly categorized into LLMs and non-LLMs.

An LLM is a type of AI model that is trained on massive amounts of data so that the program can understand, generate, and reason about language. They’re usually built using deep neural networks, and can answer questions, write and summarize text, translate languages, generate code, and even hold conversations. Commonly known examples of LLMs include Anthropic’s Claude, Google’s Gemini, and OpenAI’s ChatGPT. While the most common LLMs are General LLMs like those three examples, there are also Code LLMs, e.g. Open AI’s Codex; Math LLMs, e.g. DeepSeek-Math; and Multimodal Models, e.g. Microsoft’s Kosmos-2.

Conversely, non-LLMs are just that – AI models that are not large language models. LLMs are called LLMs because their model sizes are massive, composed of billions, or even trillions, of parameters. LLMs’ training sets are also massive, taking in information from a possibly unlimited number of sources. LLMs can operate flexibly with natural language inputs. Non-LLMs, however, are usually smaller, more structured, and much more specific. They’re usually task or domain-specific, trained for narrow purposes, and operate with structured inputs to generate structured outputs. While LLM’s are largely known for being “black boxes” in terms of how they process information, non-LLMs are usually easier to interpret due to their narrow tasks and specific scopes.

You’ll typically see non-LLMs in specific cases, like traditional machine learning models deployed for fraud detection or credit scoring, classical natural language processing models used for keyword searches and simple text clarifications, computer vision models used for image recognition or facial recognition. 

This layer determines how a model learns from data, how it generalizes using data, and how advanced or capable it can become. Consequently, this layer garners significant discussion as policymakers, industry leaders, and the public wrestle with balancing public safety with algorithmic innovation.

Data Layer

Data stands as the touchstone of AI development. It serves as the raw material from which models gain insights, discern patterns, and make predictions. In this layer, we see the processes of data collection, preprocessing, and augmentation. The Data Layer determines not only what the model knows, but also how well it performs, how it behaves, and whether it inherits bias or blind spots.

The types of data used in AI models include:

These varying types of data are used by AI models in various ways. Largely, there are three distinct data categories: pre-training data, fine-tuning data, and post-deployment data. 

Pre-training data is also known as foundational training data, and it is massive, broad, and mostly general-purpose. This vast swath of information is used to train a model from its bare bones to learn language structure, general knowledge, and general reasoning patterns. Pre-training data can include all structured data, unstructured data, semi-structured data, temporal data, and geospatial data, as much of the information is sourced from web pages, books, code repositories, academia, forums, FAQs, and manuals. 

Fine-tuning data, otherwise known as task- or domain-specific training sets, are made of smaller, higher-quality datasets than those massive and diverse datasets used for pre-training. The fine-tuning datasets are used after pre-training to improve performance, align the models’ behaviors with its function, and adapt to its specific industry or function. Here you’ll see data that helps pair input with desired outputs, helps provide the model with guidance for how to prefer or rank its outputs, helps teach the model instructions for reasoning patterns, and industry specific examples, like medical notes or certain contract templates.

Pre-training data and fine-tuning data are essentially training data types. Post-deployment data, however, is not automatically data used for training. Post-deployment data is the data that the model processes once deployed, and includes enterprise inputs, individual user prompts, and tool outputs. Here, the data is used to generate responses in real time; provide context through memory and tool use; and logging for debugging, abuse detection, and product improvement. Whether post-deployment data becomes training data really depends largely on user consent, contractual terms, and regulatory constraints.

Not only does the Data Layer consist of the actual data, but also the players that compile, refine, process, and provide the data to the actual AI companies. For example, large-scale dataset curators provide massive, general-purpose information for pre-deployment purposes to AI companies. Some of these curators come from the nonprofit world, like Common Crawl and EleutherAI. Other dataset curators are commercial, largely invisible and servicing non-LLM AI models like LexisNexis or Westlaw for legal-focused AI models, Bloomberg for financial-focused AI models, or Springer for scientific-focused AI models. Much of the fine-tuning data comes from specialized data companies that don’t just own data, but also produce or refine it. Here, we have players like Scale AI, Appen, and TELUS International AI.  

In the Data Layer is where we see policy, legislative, and advocacy concerns pertaining to a broad range of issues including data privacy, data quality (garbage in garbage out), and monetization of personal information. 

Application Layer

Once the infrastructure is built and the models are both designed and trained, AI is ready to interface with the world. The Application Layer is where that happens – where AI models are embedded into software applications and utilized through user interfaces, application programming interfaces, or automated workflows. Sitting atop the Infrastructure, Algorithm, and Data Layers, the Application Layer delivers the value, and risks, of AI to the public. Because here is where AI most directly affects people, policy concerns are most visible and immediate at the Application Layer, even if the issue is borne within a different layer. 

The types of Applications you see powered by AI can include:

Typical Application functions include:

Each layer has its own bundle of diverse players, but the Application Layer’s key players are especially competitive and fragmented. Here, we see players ranging from big tech giants to the tech startups just getting their initial investments, as well as enterprise software vendors. 

Horizontal AI Application providers are those companies and platforms that work across varying industries. These are your big names, like OpenAI, Microsoft, Google, and Anthropic. Vertical AI Application providers are those that focus on single domains, like PathAI and Aidoc in healthcare, Harvey and Casetext in law, Kensho and Zest AI in Finance, HireVue and Pymetrics in HR and recruiting, and Gong and Jasper in sales and marketing. In addition to application providers, enterprise software companies that are adding AI features to their products also operate within this space. For example, Salesforce has embedded Einstein AI and Adobe has integrated Firefly in Creative Cloud. 

The Application Layer, as the most visible and interactive layer of the AI Stack, is where the most heated policy debates are focused on. From accountability questions, like who should be responsible for AI-driven outcomes and how much human oversight should be required; to safety and risk management concerns, such as how to prevent, detect, or mitigate attacks or breaches; to issues of informed consent, biases, and hallucinations, to name a few, AI’s risks and regulatory uncertainty are front and center once the technology reaches the user.

Conclusion

Put together, these four layers form the backbone of AI. Computing gives the foundation, algorithms give you the function, data gives you the context, and application gives you the finished product. Understanding the whole AI Stack is integral to policymaking, especially as recent policy conversations increasingly focus on chips and foundation models in both the economic and national security contexts. Effective policymaking requires an understanding of the whole stack, including the market participants and dynamics within it, and a recognition that each layer raises different questions, presents different concerns, and benefits from different policy approaches. A common understanding of each layer will help guarantee that policymakers, industry, and consumers are aligned and can work from a shared foundation to drive tailored, resilient, and forward-facing AI policy.

Exit mobile version