The goal of AI search is simple but profound: to break down knowledge silos, synthesize content from multiple sources, and deliver comprehensive answers in seconds. This promise of AI search — instant, insightful answers from across the enterprise — is now a reality thanks to Retrieval-Augmented Generation (RAG). Over the past couple of years, RAG-based systems emerged as a transformative approach for building AI that delivers accurate, contextually relevant responses.
It has proven to be immensely powerful with results that feel magical: ask a question in plain language and get a tailored answer, often with the exact piece of the document you need highlighted as evidence.
A recap: What RAG got right
Retrieval-Augmented Generation (RAG) is the engine under the hood of many AI search solutions. The system first retrieves relevant information, then augments a large language model (LLM) with that information to generate an answer.
RAG systems offered something groundbreaking: instead of a wall of links, it delivered synthesized answers, complete with citations and context, often within milliseconds. This was transformative.
Traditionally, RAG systems relied on building a vector index — a centralized, pre-processed database of enterprise content. Every document, support ticket, or message would be chunked, embedded, and stored. When a user asks a question, the system retrieves the most relevant snippets from this index and feeds them into a LLM to craft an answer grounded in actual enterprise content.
However, indexing isn’t what makes RAG groundbreaking — that’s only one possible source of content and also happened to be the most practical implementation available at the time. The real innovation is in getting relevant context into the LLM, whether that information comes from a local index or a live API call.
The pitfalls and hidden costs of indexing everything
To power a RAG system, enterprise search vendors historically pulled data from every possible system into a central index. There’s been a building tension between the underlying platform providers (where enterprise data resides), and third party search vendors (who extract all that valuable source data).
This “index everything” approach poses real challenges for several reasons:
1. Data replication and custody conflicts
Indexing means duplicating data from underlying platforms like Slack, Jira, and Salesforce. Even though customers own the data, the platform provider is still responsible for securing it.
For example, in a policy update from Slack, Salesforce cited “critical considerations around how customer data is handled” and the need to be “thoughtful and transparent” about data usage as reasons for tightening API access. This forbids enterprise search vendors from the bulk download and long-term external storage of data.
The loss of control over customer data is making platform vendors uneasy, and they’re starting to enforce stricter data guidelines. Indexing is critical to effective search, but is also increasingly viewed as a data control liability.
2. Strain on source systems
To keep indexes fresh, search tools frequently crawl entire datasets in bulk and put heavy loads on APIs not built for that.
These high-volume API calls or export jobs can put strain on the platform as Atlassian shared earlier this year, potentially degrading performance for its primary users. Atlassian has warned that high-volume crawls can degrade performance for actual users, and consequently, will start to enforce rate limits later this year.
3. Out-of-sync permissions
Indexes are snapshots of another platform’s data. This means it also captures the platform’s permissions for a moment in time, so there’s always some lag between when something changes in the source system and what’s reflected in the search index.
Permissions are changing constantly — employees could leave or get promoted, or files switch from public to private. Even a delay of a few hours could lead to oversharing sensitive information. With dozens of integrated sources, keeping all permissions perfectly in sync at all times is a daunting challenge — and any gap can be a security risk.
4. Data models are not uniform (loss of fidelity)
Indexing disparate data into a single store often means flattening it to a universal format (usually a generic “document” with fields like title, body, author, date, etc.). Data is complex — threaded Slack conversations, Salesforce leads, or Jira tickets — and each have their own unique structure and metadata.
When forced into a uniform index, much of that rich structure is either lost or hard to exploit. Relevance suffers, as the search engine can’t fully represent the nuances of each system. Context can get lost in translation when everything is reduced to the lowest common denominator of a text index.
5. Misaligned value for platform providers
Underlying platform providers have spent years building applications for customers to create all this data. Indexing lets third-party search vendors build a new value layer that monetizes the extracted data, without compensating the platforms.
Now these platforms want their portion of the new value chain and we’re seeing it play out: major platforms are placing stringent limits on third-party indexing or charging for it.
Slack and Atlassian are just two examples mentioned from earlier — Notion, a workspace app often asked to integrate with search tools — is reportedly reevaluating the data access it gives to outside firms and likely considering similar restrictions. The corporate data battles are heating up.
Indexing by third parties has started to look like a threat to platform providers. It’s not that enterprise search is unwelcome, but the way data is copied and exported is.
The industry is at an inflection point: Is there a way to deliver AI-powered search without these downsides?
Yes, and it is likely the new way forward.
The new search stack: Reasoning + live search
The demand for AI-powered enterprise search is skyrocketing and the technology stack is evolving profoundly. A new approach is emerging: Live Search with agentic reasoning.
Instead of indexing all data up front, an intelligent agent fetches data on the fly. It swaps the “big index” for a “live search” strategy powered by real-time queries and reasoning. An AI reasoning model (an LLM-based agent) interprets the query and dynamically decides which sources to query and how, bringing intelligence and context-awareness to every search.
This approach is sometimes described as “agentic search” or federated live search. Here’s what the new stack is, and how it works:
- Reasoning as the orchestrator: The core is a smart orchestrator (typically a LLM) that can understand the user’s request and decide what systems to search. If you ask about the updated PTO policy, it may search in Workday, Confluence, and HR Slack channels in parallel, each with the right query format. If the initial results are insufficient, it can reformulate queries or drill down further (autonomously) until it finds a good answer.
- Live search connectors/APIs: The reasoning model doesn’t have access to any database directly — it uses each platform’s search API, just like a user would via the native search bar. Instead, AI is doing the search behind the scenes, and using the employee’s own authentication token. This means real-time, source-enforced permissions, so users are only allowed to see what they have access to. The data stays in the underlying platform at all times, so permissions stay fresh, secure, and compliant at all times.
- Dynamic schema understanding: Instead of forcing all content into a flat index, the reasoning model can understand the structure of each system and tailor its queries accordingly. It knows that Jira has tickets, Confluence has pages, and Salesforce has contacts and opportunities. This enables more precise, context-aware retrieval where the AI can adapt on-the-fly to each system’s schema — which static indexes struggle to do.
- Synthesis and response generation: After retrieving the top results from each system, the AI aggregates and analyzes them, then feeds the most relevant snippets into a generative model to produce a synthesized answer. Because this happens in real-time, responses reflect the most current data — not yesterday’s index. It also tracks where each snippet came from and includes citations. The key is that the heavy work (searching each system) is done live, and the final answer is compiled on the spot.
What makes live search so powerful is that it helps to eliminate the need for bulk data ingestion or long-term storage. Instead of building massive indexes, just connect to source systems via APIs and orchestrate searches on demand. This eliminates many of the earlier concerns: no massive duplication of data (improving security posture), no continuous crawling (reducing load on source systems), and no stale permission issues (because each query checks current permissions).
It’s also far more scalable. Rather than building and maintaining massive indexes for every system, search tools can simply tap into hundreds of apps through their APIs. This opens the door to much broader coverage — even for platforms that don’t support full data export. It also realigns incentives: API usage can be measured and monetized, giving platforms a way to participate in the value being created.
Major players are validating this approach. OpenAI’s ChatGPT now connects to services like Google Drive, HubSpot, and SharePoint in real time — retrieving and synthesizing content on the fly without storing it. Microsoft’s Copilot does the same across its ecosystem.
This marks a shift from hoarding data in static indexes to bringing AI to the data dynamically. While challenges like API rate limits and inconsistent API quality remain, the direction is clear: real-time retrieval plus LLM reasoning is the new enterprise search standard.
This new stack doesn’t replace indexing — it augments it. Challenges like API rate limits, inconsistent search quality, and latency still exist. Instead of every AI vendor hoarding data in their own silo, allow data to stay where it is and bring the AI to the data, dynamically. But the shift is underway. It’s a fundamental architectural change — and a necessary one.
To index or not – how about both?
With all the momentum around live search, it’s fair to ask: is indexing going away? The answer is no — and it shouldn’t. The most effective enterprise search systems combine both indexed and live retrieval — it’s not a matter of either/or, but both. An AI agent can dynamically choose the best path per query.
There are still many scenarios where a pre-built index is beneficial, or even necessary:
- Speed and latency: A local vector index returns results in milliseconds — crucial for interactive use cases like chatbots or real-time UX, where API latency simply can’t keep up.
- Offline or complex querying: Not all platforms expose rich search APIs. Indexes give teams control over ranking, semantic search, filtering, and even custom ML-powered retrieval strategies.
- Data resilience: APIs can go down, get rate-limited, or change. An index serves as a stable fallback or cache, ensuring continuity for common or critical queries.
- Training and experimentation: Indexed data is invaluable for tuning embeddings, testing prompts, and iterating on relevance models. It gives teams a consistent sandbox for improving performance.
Indexing isn’t obsolete — it’s just no longer the only path to content. The best systems use both, intelligently.
From an enterprise IT leader’s perspective, a hybrid search strategy provides optionality and future-proofing. You can leverage indexing where it’s allowed and beneficial, and pivot to live retrieval where platforms demand it or where it offers an edge.
Smarter search, same mission
Enterprise search is undergoing a profound transformation. The shift from static indexing to agentic, real-time systems isn’t just technical — it’s architectural.
AI is no longer just surfacing documents; it’s reasoning across systems, interpreting intent, and assembling precise, contextual answers. But the mission remains the same: get the right information to the right person, at the right time, with as little friction as possible.
The best solutions now blend the speed of indexing with the freshness and flexibility of live API retrieval, orchestrated by intelligent agents that choose the best path per query. This hybrid model enables employees to benefit from both fast response times and always-up-to-date insights — without compromising on security, scalability, or platform compatibility. It’s not an either/or decision. It’s the strategic use of both.
For IT leaders, this is a breakthrough moment. Employees and organizations can finally treat AI assistants as trusted guides to company knowledge — capable of navigating every system they have access to, without replicating data or introducing risk.
With agentic reasoning and a hybrid retrieval stack, enterprise search becomes seamless, secure, and smart by design — a win for users, a win for IT, and a win for the platforms that power your business.
Moveworks Enterprise Search, powered by agentic RAG
Moveworks Enterprise Search blends index-based speed with the freshness of live API retrieval. Live Search allows an AI agent to query systems in real time — Google Calendar’s API, Salesforce’s API, and Slack’s API in parallel – each query crafted appropriately for that system — and then combine the results to answer the user. All of this happens at query time, no pre-indexing required.
It uses the employee’s own access token to ensure permissions are always enforced. This avoids the need to duplicate data or maintain a massive index, while still delivering fast, context-rich answers.
For content that doesn’t change often — like internal knowledge bases — Moveworks still uses vector indexing for low-latency responses.The platform can fall back to its index for sources where that makes sense, or switch to live queries for sources that support it well.
This hybrid model truly offers the best of both worlds. It’s a pragmatic, future-ready architecture that proves the real power of AI search lies in using both approaches where they make sense.
If you’re interested in learning more about Moveworks Enterprise Search, check it out in the Product Release Spotlight or request a demo today.
Table of contents
