Software Engineering Blog
Engineering Design

Inside Browser-Native AI Agents for Mapping

By: Dhrumil Shah

Inside Browser-Native AI Agents for Mapping

In Part 1, we explored why AI orchestration belongs in the browser, close to the map and its live application state. Here, we look inside the agents: how they find relevant data, choose tools, and interact with the map—and the engineering decisions that shaped their design.

Building on the earlier nursing homes example, we’ll use a more specific request:

“Show nonprofit nursing homes in Texas and tell me what percentage of all Texas nursing homes they represent.”

That sounds simple, but a web map can contain dozens of layers and hundreds of fields whose names and values don’t directly match the user’s wording. In this map, the agent must identify the United States Nursing Homes layer and discover that states are stored as two-letter codes in USER_State. Nonprofit facilities aren’t simply labeled “nonprofit”—they span multiple values in USER_Ownership_Type beginning with Non profit -. Answering the request requires discovering the relevant schema and values, calculating the percentage, and highlighting the matching facilities on the map.

Map highlighting nonprofit nursing homes in Texas alongside the agent’s count and percentage

Agent tools access the live map through the arcgis-map component, reading its layers, extent, and other view state when they execute. For a request such as “What’s in my current view?”, a query tool can use the current extent even if the user has just panned the map.

The ArcGIS Maps SDK for JavaScript supplies the underlying operations, including querying features, applying filters, computing geometry, and controlling the viewport. Orchestration runs in the browser; individual operations may use client-side data or call services as needed.

These agents are available through the SDK’s AI components. The arcgis-assistant component provides the conversational interface and orchestrates requests across registered agents.

From predefined workflows to skills

In early 2025, we began designing these agents with a straightforward approach: put the map’s layer and field metadata into the prompt and let the model decide what to use. This worked for smaller maps, but became less reliable as the number of layers and fields grew. The model sometimes selected the wrong fields or generated queries referencing fields that did not exist.

The first major improvement came from combining embeddings with workflows built in LangGraph. Embeddings helped us retrieve the layers and fields relevant to a request, while the graph organized the steps for retrieving context, generating queries, and calling tools. Providing focused context and an explicit execution path made the agents more reliable.

As we expanded the workflows, however, we found that our predefined paths could also limit recovery. If an agent selected the wrong layer or discovered that it needed more context, the workflow did not always give it a straightforward way to revisit an earlier decision.

Moving to skill-guided tool use

We introduced skills into this architecture: instructions that describe how to carry out a task using the available tools. Instead of encoding each decision as a graph branch, we let the agent choose tools, inspect their results, and determine what to do next. Combined with newer models, this has allowed us to simplify the workflows while giving agents more room to adjust when their first approach does not work.

Moving task-specific decision logic from predefined graph branches into skill instructions also substantially reduced the amount of workflow code we needed to maintain.

A predefined workflow follows fixed retrieval and query steps. A skill-guided agent chooses tools, inspects results, and repeats as needed before responding.
Simplified representations of a predefined workflow and a skill-guided tool loop.

Loading skills on demand

An agent can use multiple skills, each focused on a particular task. Through progressive disclosure, the model initially sees skill names and descriptions. When a skill is loaded, its detailed instructions are added to the context, and only the tools listed in its allowed-tools property are made available to the model. This keeps both instructions and tool availability focused on the task at hand.

The following excerpt from the query skill shows how its instructions guide the agent to find relevant layers and fields, query the data, and answer the user’s question.

---
name: query
description: Use to find, query, and summarize feature-layer data.
allowed-tools: >-
 listSharedResources findRelevantLayers findRelevantFields
 getAttribute getStatistics getTopFeatures queryFeatures
---

# Query

<!-- Simplified excerpt: instructions guide tool use, not a fixed graph.
The full skill also covers spatial queries, statistics, rankings,
and sharing query context for map updates. -->

- Find relevant layers with `findRelevantLayers`.
- Discover fields and values with `findRelevantFields`;
 never invent them.
- Use `queryFeatures` to count or retrieve matching features.
- Answer only from tool results; report failures directly.

In our testing, the skill-guided approach improved accuracy and flexibility, but required more model calls and tokens, increasing execution cost. Predefined workflows can be more efficient for predictable tasks, while skill-guided tool use is useful when the agent needs to adapt based on intermediate results. Both approaches remain supported and can be combined.

Vector search in the browser

We introduced embedding-based vector search to retrieve relevant layer and field metadata before query generation. Feature layers are like database tables, and their fields are like columns: the agent needs the relevant schema, not every available schema.

Embeddings represent text as numerical vectors, allowing us to search for layers and fields by meaning rather than exact wording.

Embedding generation is a separate web map preparation step. Embeddings are generated from layer titles and field metadata and stored as a resource on the web map item. When the application loads the map, it downloads those embeddings for browser-side retrieval.

In the skill-guided agent loop, the model decides when to retrieve context and what search query to use. The retrieval tool embeds that query, searches the stored layer and field embeddings, and returns relevant candidates. The agent can inspect those results and search again when more context is needed. In the nursing homes example, this helps identify the relevant layer and the state and ownership fields before constructing a query.

Embedding-based retrieval within a skill-guided agent loop.

Improving relevance and responsiveness

Our initial search ranked candidates by cosine similarity to the user’s request. This worked well in many cases, but maps with many similar fields—such as population1990, population1991, and so on—could produce a shortlist dominated by closely related fields, leaving out other useful context.

We added maximal marginal relevance (MMR) to balance relevance to the request with diversity among the selected results. This reduced redundancy in the context passed to the model. The additional comparisons increased the retrieval workload, so we moved the search into a web worker to keep it off the main thread.

Evolving embedding generation

Our early prototypes generated embeddings in the browser using all-MiniLM-L6-v2. We later experimented with server-hosted embedding models such as text-embedding-ada-002, which improved retrieval quality. At runtime, the retrieval tool calls the server to embed the search query, while vector comparisons and MMR ranking run locally in a browser web worker.

The agents we ship today use embedding-based retrieval, which requires generating, storing, and updating embeddings as map content and metadata change. We are also exploring alternatives such as Jev and OpenAI’s Decisions API to select relevant layers and fields without maintaining that embedding pipeline.

The Data Exploration Agent

The Data Exploration Agent answers questions about data in map layers, including counts, summary statistics, feature rankings, and spatial queries.

Tracing the nursing homes request

In one recorded run, the agent handled the request introduced earlier: “Show nonprofit nursing homes in Texas and tell me what percentage of all Texas nursing homes they represent.”

  1. Route the request: The orchestrator selected the Data Exploration Agent.
  2. Load the query skill: load_skill("query") loaded instructions for discovering and querying relevant data and made the skill’s tools available to the model.
  3. Find the layer: findRelevantLayers returned the United States Nursing Homes layer.
  4. Discover fields and values: findRelevantFields returned the state and ownership fields, with unique-value statistics showing TX and nonprofit ownership values beginning with Non profit -.
  5. Query twice: The agent called queryFeatures for nonprofit nursing homes in Texas and for all Texas nursing homes. The calls returned counts of 115 and 1,177, respectively.
  6. Load the filter skill: load_skill("filter") loaded instructions for updating the map using the existing query context and made the skill’s tools available to the model.
  7. Emphasize matching facilities: setFeatureEffect reused the nonprofit selection criteria, applying a drop shadow to matches while rendering nonmatches in grayscale with reduced opacity and blur.
  8. Return the result: The agent returned a status, summary, and output message reporting 115 of 1,177 facilities, or 9.8%. The orchestrator completed the request without calling another agent.

The nonprofit query used this where clause:

USER_State = 'TX'
AND USER_Ownership_Type LIKE 'Non profit - %'

The filter skill reused the layer ID and selection criteria established during querying, without repeating layer or field discovery. This kept the statistical answer and map update aligned within the same agent run. For purely statistical questions, the agent can return an answer without changing the map.

The tools determine whether to perform each operation client-side or through the backing feature service. When all the data needed is available in the browser, they operate on it locally. Otherwise, they use the service. This keeps execution close to the data without requiring the model to choose between client-side and server-side execution.

Data Exploration is one of several built-in agents we ship. Another is the Navigation Agent, which handles zooming, panning, and locating layers, features, places, and addresses using the same skill-guided tool-use pattern.

Custom agents

Developers can build custom agents and use them alongside the agents we ship, or configure an assistant entirely around custom agents. Each agent advertises a name and a short description of its capabilities. The orchestrator acts primarily as a router, using those descriptions to hand control to an agent suited to the request.

An orchestrator uses registered names and descriptions to route requests to built-in or custom agents. Optional human input passes through the application UI to pause and resume execution.
Orchestrator routing requests to selected built-in and custom agents.

For requests that require multiple agents, the orchestrator coordinates those handoffs as well. Each agent returns a status, a summary, and an output message for the user. These results help the orchestrator determine whether the request is complete or another agent needs to contribute.

This applies progressive disclosure at the agent level. The orchestrator does not need every agent’s internal instructions, skills, or tool definitions to make a routing decision—just enough information to know when to call it. The selected agent handles the details of completing its part of the task.

We support building agents with LangGraph directly, as well as utility functions that reduce the LangGraph code for common agent patterns. Both approaches let developers integrate domain-specific workflows into the same orchestration layer.

Several Esri applications, including those without maps, are already using this architecture to build AI-assisted experiences with custom agents tailored to their workflows.

Human-in-the-loop interactions

The orchestration layer passes the agent’s request for input to the application UI and routes the user’s response back to resume the agent. This gives built-in and custom agents a consistent pause-and-resume mechanism without requiring each agent to implement its own UI communication. For example, a custom agent can pause to ask for confirmation before writing a feature to a feature layer or database, then proceed only if the user approves.

Closing

Live map state, focused context retrieval, and skill-guided tool use give agents a practical foundation for working with the map. Tools can read the application’s current state, agents can gather additional context and revisit decisions, and human-in-the-loop support lets workflows pause when user input is needed.

In Part 1, the key question was where orchestration should run. In this post, we explored what that architecture enables in the browser: agents that find relevant data, answer questions, and coordinate map interactions, alongside custom agents built for an application’s own workflows.

Although the examples here focus on mapping, custom agents are not limited to map interactions. Developers can use the same orchestration layer to build workflows that do not interact with a map at all.

As models and retrieval approaches evolve, the separation between orchestration, agents, tools and UI components lets us improve those pieces independently. The goal remains the same: help users explore data and carry out tasks through natural language, with the application’s data and state guiding the work.

Related Articles

Next Article

Designing Meaningful API Load Tests with k6

Read this article