Case Study 02: 324-service enterprise multi-repo audit (ce vs ripgrep).
Two identical AI agents executed a 4-part cross-service architectural audit on a distributed multi-repo microservices workspace (324 independent microservice repositories, 93 shared library repositories, 33 workers, 13,782 source files) under symmetric system prompts with zero schema hints: one using ripgrep + Python scripts + view_file, and one using CodeExplorer (ce taxonomy + ce query + view_file).
Shorter Time to Answer
4m 16s (256s) with CodeExplorer (including ce taxonomy discovery) vs 29m 52s (1,792s) with ripgrep + Python scripts.
18 turns vs 285 turns (~16×)
Fewer Input Tokens
~686K cumulative input tokens (0 compactions) vs ~17.2M post-compaction tokens.
$0.10 vs $1.15 (~11×)
Manual File Reads
Without schema hints in the prompt, the ce agent discovered the 5-layer ontology via ce taxonomy and needed 16 file reads vs 114 file reads and 14 scripts.
27 tool calls vs 284 tool calls
Blast-Radius Recall
Both agents tied at 100% on Tasks 1 & 4. On Task 2, ripgrep missed 9 RTB microservices wired via environment host constants.
83/83 callers vs 9 services missed
1. CodeExplorer (
ce): v1.77.0 (.NET 10, SQLite + OpenCypher, schema v6: 209,271 nodes, 454,394 edges)2. Lexical CLI Baseline:
ripgrep (rg v14.1.1) + Python 3.10 + view_file3. LLM Model:
Gemini 3.8 Flash (symmetric system prompts with zero schema spoilers)4. Target Workspace: Enterprise Multi-Repo Microservices System (450 independent service/library repositories, 324 services, 13,782 files; service names anonymized 1-to-1)
1. Execution telemetry scorecard.
Extracted from transcript_full.jsonl (conversations 3e1e9d36 for ce and ede04347 for ripgrep) and committed in benchmarks/02-enterprise-microservices-deep-audit/results/metrics.json. Neither agent received any schema hints or pre-baked Cypher templates in its system prompt.
| Metric | ripgrep + Python (142 CLI runs + 114 file reads) |
CodeExplorer (10 CLI runs + 16 file reads) |
Delta |
|---|---|---|---|
Time to Answer (send_message) |
1,792.0 s (29 min 52 sec) | 256.0 s (4 min 16 sec) | 7.00× faster (-25m 36s) |
| Total Session Wall-Clock | 1,962.0 s (32 min 42 sec) | 332.0 s (5 min 32 sec) | 5.91× faster (-27m 10s) |
| LLM Reasoning Turns | 285 turns (584 steps) | 18 turns (46 steps) | 15.83× fewer turns (-93.7%) |
| Total Tool Calls | 284 (142 run_command, 114 view_file, 14 write_to_file, 13 manage_task, 1 send_message) |
27 (10 run_command, 16 view_file, 0 scripts written, 1 send_message) |
10.52× fewer tool calls (7.13× fewer file reads) |
| Raw Tool Output Ingested | 694,891 chars (~173,722 tokens) | 262,638 chars (~65,659 tokens) | 2.65× smaller tool payload |
| Context Window Compactions | 1 compaction (at Turn 162, 160.3K tokens) | 0 compactions | 0 context resets |
| Cumulative Input Tokens (post-compaction) | 68,776,567 chars (~17,194,141 tokens) | 2,742,953 chars (~685,738 est. / 1,314,308 API tokens) | 25.07× fewer input tokens (13.08× vs API) |
| API Cost w/ 90% Cache Discount (Flash / Pro) | $1.15 / $4.47 | $0.10 / $0.38 ($0.17 / $0.66 API) | 11.50× cheaper (Flash) |
| Task-by-Task Recall | 100% on Tasks 1 & 4; ~81% on Tasks 2 & 3 (missed 9 RTB services & 15+ callers) | 100% on all 4 tasks (all 9 RTB services & callers verified) | 0 missed services |
2. Breakdown of the 4 audit tasks (where ripgrep tied vs where it failed).
| Task | ripgrep + Python |
CodeExplorer |
Explanation |
|---|---|---|---|
| Task 1: Shared DB Tables (≥3 ORM Entities) | 100% (8/8 tables) | 100% (8/8 tables) | Tie on accuracy: ripgrep agent wrote Python scripts to scan @Entity / @Table decorators; ce discovered (:Table)-[:IMPLEMENTED_BY]->(:Type) via ce taxonomy (Turns 1 to 2). |
| Task 2: Outbound Service Call Fan-In (5 target services) | ~81% (missed 9 partner RTB services & 15+ callers) | 100% (all 9 RTB services & wrapper callers found) | ripgrep missed WorkerCoreClientService.updateEndpoint across 9 RTB services because the host URL was injected via an environment constant (WORKER_CORE_API). ce resolved all 9 via (:Service)-[:CALLED_BY]->(:Function). |
| Task 3: Polyglot Messaging Topology (TS + Go) | Found core TS & Go handlers; missed shared strategy publishers | 100% (all TS & Go publishers, subscribers, and wrapper callers) | ce queried Topic edges (PUBLISHED_BY / SUBSCRIBED_BY) and verified constant-aliased topics via surgical view_file reads. |
| Task 4: Multi-Hop Inheritance & Strategy Realization | 100% (13 + 5 + 8 = 26 classes) | 100% (13 + 5 + 8 = 26 classes) | Tie on accuracy: Solved on Turn 3 via (:Type)-[:INHERITS_FROM|IMPLEMENTS*1..5]->(:Type). |
3. Reproducibility: symmetric prompts & agent system instructions.
Click any drawer below to expand verbatim prompt textTask Prompt prompts/task-prompt.md: Shared 4-Part Cross-Service Audit Task Given Identically to Both Agents
Perform a comprehensive, cross-service architectural audit on the enterprise multi-repo microservices workspace (`src/` contains 320+ independent microservice repositories and 90+ shared library repositories across TypeScript/NestJS and Go). Answer all four architectural questions below with 100% completeness (do not guess or omit items; provide exact symbol/class/function names and relative file paths using forward slashes `/`): 1. **Cross-Service Database Table Persistence Coupling (Shared ORM Tables)**: Identify every database table (`Table`) in the workspace that has **3 or more distinct ORM entity classes (`@Entity` / model classes)** mapped/persisted to it across different services or libraries. For each such table, provide: - The exact table name - The total count of distinct entity classes mapped to it - Every mapped entity class name and its relative file path 2. **Cross-Service Synchronous Outbound Call Fan-In (`SERVICE_CALL`)**: Identify every caller service, function/method name, and relative file path across the workspace that makes an outbound synchronous service call (`linkGenerator` / HTTP client service call) to any of these 5 internal target services: - `svc-worker-core` - `svc-alert-notifier` - `svc-network-telemetry` - `svc-exchange-hub` - `svc-pricing-engine` Group your results by target service. 3. **Polyglot Asynchronous Messaging & Queue Topology (TypeScript & Go)**: For each of the following 5 messaging topics / queue channels across the TypeScript and Go services: - `topic-routing-rule-updated` - `PUB_SUB_PARTNER_NETWORK_TOPIC` - `REALTIME_BID_CACHE_QUEUE` - `TelemetryImpressionTopic` (`ImpressionTopic`) - `TelemetryConversionTopic` (`ConversionTopic`) Identify every function/method name and its declared relative file path that: (a) **Publishes** messages/events to the topic, and (b) **Subscribes / listens** to messages/events from the topic. 4. **Multi-Hop Inheritance & Strategy Realization Across Microservices**: List every class/type and its relative file path across the workspace that: (a) Inherits (directly or transitively) from `BaseEventController` (b) Implements the `ICloudPubSubService` interface (c) Implements the `IOptimizationCampaignStrategy` interface > **Confidentiality Note**: All proprietary service, table, topic, class, and interface identifiers in this benchmark artifact have been systematically anonymized to preserve commercial confidentiality while maintaining a 1-to-1 structural correspondence with the underlying enterprise multi-repo workspace.
System Prompt (ce) prompts/graph-agent.md: Symmetric System Instructions for CodeExplorer Agent (Zero Schema Hints)
# Agent System Instructions You are an expert software architect and researcher analyzing a large enterprise polyglot multi-repo microservices workspace (`324` independent service repositories, `93` shared library repositories, `13,782` source files across TypeScript/NestJS, Go, and SQL). You must answer the user's architectural questions with 100% accuracy and completeness based on the actual codebase, providing exact symbol names and relative file paths. CRITICAL BENCHMARK RULES: 1. You have access to the CodeExplorer CLI (`ce`) via `run_command` (which queries the pre-indexed `.codeexplorer/graph.db` semantic graph) and `view_file` to inspect source files when needed. 2. To discover available `ce` commands or inspect the graph schema (node labels, properties, and relationship types), use `ce --help`, `ce taxonomy`, or `ce status`. 3. Do NOT modify or create any files inside the repository.
System Prompt (ripgrep) prompts/baseline-agent.md: Symmetric System Instructions for ripgrep Baseline Agent
# Agent System Instructions You are an expert software architect and researcher analyzing a large enterprise polyglot multi-repo microservices workspace (`324` independent service repositories, `93` shared library repositories, `13,782` source files across TypeScript/NestJS, Go, and SQL). You must answer the user's architectural questions with 100% accuracy and completeness based on the actual codebase, providing exact symbol names and relative file paths. CRITICAL BENCHMARK RULES: 1. You have access to standard CLI search commands (`rg` / `ripgrep`, `git grep`, `Get-ChildItem`, `Select-String`, or custom shell/Python scripts) via `run_command`, and `view_file` to inspect source files. 2. You are STRICTLY FORBIDDEN from running `ce`, `ce.exe`, `codeexplorer`, or accessing `.codeexplorer/graph.db` (always pass `-g "!.codeexplorer/**"` if needed). 3. Do NOT modify or create any files inside the repository.