Benchmarks Catalog / Case Study 02: 324-Service Enterprise Multi-Repo Audit
450 modules (324 services, 13,782 files, TS + Go + SQL)

Case Study 02: 324-service enterprise multi-repo audit (ce vs ripgrep).

Two identical AI agents executed a 4-part cross-service architectural audit on a distributed multi-repo microservices workspace (324 independent microservice repositories, 93 shared library repositories, 33 workers, 13,782 source files) under symmetric system prompts with zero schema hints: one using ripgrep + Python scripts + view_file, and one using CodeExplorer (ce taxonomy + ce query + view_file).

~7.0×

Shorter Time to Answer

4m 16s (256s) with CodeExplorer (including ce taxonomy discovery) vs 29m 52s (1,792s) with ripgrep + Python scripts.

18 turns vs 285 turns (~16×)
~25×

Fewer Input Tokens

~686K cumulative input tokens (0 compactions) vs ~17.2M post-compaction tokens.

$0.10 vs $1.15 (~11×)
16 vs 114

Manual File Reads

Without schema hints in the prompt, the ce agent discovered the 5-layer ontology via ce taxonomy and needed 16 file reads vs 114 file reads and 14 scripts.

27 tool calls vs 284 tool calls
100% vs ~82%

Blast-Radius Recall

Both agents tied at 100% on Tasks 1 & 4. On Task 2, ripgrep missed 9 RTB microservices wired via environment host constants.

83/83 callers vs 9 services missed
Tested participants:
1. CodeExplorer (ce): v1.77.0 (.NET 10, SQLite + OpenCypher, schema v6: 209,271 nodes, 454,394 edges)
2. Lexical CLI Baseline: ripgrep (rg v14.1.1) + Python 3.10 + view_file
3. LLM Model: Gemini 3.8 Flash (symmetric system prompts with zero schema spoilers)
4. Target Workspace: Enterprise Multi-Repo Microservices System (450 independent service/library repositories, 324 services, 13,782 files; service names anonymized 1-to-1)

1. Execution telemetry scorecard.

Extracted from transcript_full.jsonl (conversations 3e1e9d36 for ce and ede04347 for ripgrep) and committed in benchmarks/02-enterprise-microservices-deep-audit/results/metrics.json. Neither agent received any schema hints or pre-baked Cypher templates in its system prompt.

Paired A/B Agent Telemetry Across 324 Microservices (13,782 files, Zero Schema Hints) Symmetric Prompts
Metric ripgrep + Python (142 CLI runs + 114 file reads) CodeExplorer (10 CLI runs + 16 file reads) Delta
Time to Answer (send_message) 1,792.0 s (29 min 52 sec) 256.0 s (4 min 16 sec) 7.00× faster (-25m 36s)
Total Session Wall-Clock 1,962.0 s (32 min 42 sec) 332.0 s (5 min 32 sec) 5.91× faster (-27m 10s)
LLM Reasoning Turns 285 turns (584 steps) 18 turns (46 steps) 15.83× fewer turns (-93.7%)
Total Tool Calls 284 (142 run_command, 114 view_file, 14 write_to_file, 13 manage_task, 1 send_message) 27 (10 run_command, 16 view_file, 0 scripts written, 1 send_message) 10.52× fewer tool calls (7.13× fewer file reads)
Raw Tool Output Ingested 694,891 chars (~173,722 tokens) 262,638 chars (~65,659 tokens) 2.65× smaller tool payload
Context Window Compactions 1 compaction (at Turn 162, 160.3K tokens) 0 compactions 0 context resets
Cumulative Input Tokens (post-compaction) 68,776,567 chars (~17,194,141 tokens) 2,742,953 chars (~685,738 est. / 1,314,308 API tokens) 25.07× fewer input tokens (13.08× vs API)
API Cost w/ 90% Cache Discount (Flash / Pro) $1.15 / $4.47 $0.10 / $0.38 ($0.17 / $0.66 API) 11.50× cheaper (Flash)
Task-by-Task Recall 100% on Tasks 1 & 4; ~81% on Tasks 2 & 3 (missed 9 RTB services & 15+ callers) 100% on all 4 tasks (all 9 RTB services & callers verified) 0 missed services

2. Breakdown of the 4 audit tasks (where ripgrep tied vs where it failed).

Task-Level Comparison Across 450 Modules 209,271 Nodes, 454,394 Edges in .codeexplorer/graph.db
Task ripgrep + Python CodeExplorer Explanation
Task 1: Shared DB Tables (≥3 ORM Entities) 100% (8/8 tables) 100% (8/8 tables) Tie on accuracy: ripgrep agent wrote Python scripts to scan @Entity / @Table decorators; ce discovered (:Table)-[:IMPLEMENTED_BY]->(:Type) via ce taxonomy (Turns 1 to 2).
Task 2: Outbound Service Call Fan-In (5 target services) ~81% (missed 9 partner RTB services & 15+ callers) 100% (all 9 RTB services & wrapper callers found) ripgrep missed WorkerCoreClientService.updateEndpoint across 9 RTB services because the host URL was injected via an environment constant (WORKER_CORE_API). ce resolved all 9 via (:Service)-[:CALLED_BY]->(:Function).
Task 3: Polyglot Messaging Topology (TS + Go) Found core TS & Go handlers; missed shared strategy publishers 100% (all TS & Go publishers, subscribers, and wrapper callers) ce queried Topic edges (PUBLISHED_BY / SUBSCRIBED_BY) and verified constant-aliased topics via surgical view_file reads.
Task 4: Multi-Hop Inheritance & Strategy Realization 100% (13 + 5 + 8 = 26 classes) 100% (13 + 5 + 8 = 26 classes) Tie on accuracy: Solved on Turn 3 via (:Type)-[:INHERITS_FROM|IMPLEMENTS*1..5]->(:Type).

3. Reproducibility: symmetric prompts & agent system instructions.

Click any drawer below to expand verbatim prompt text
Task Prompt prompts/task-prompt.md: Shared 4-Part Cross-Service Audit Task Given Identically to Both Agents
Perform a comprehensive, cross-service architectural audit on the enterprise multi-repo microservices workspace (`src/` contains 320+ independent microservice repositories and 90+ shared library repositories across TypeScript/NestJS and Go). Answer all four architectural questions below with 100% completeness (do not guess or omit items; provide exact symbol/class/function names and relative file paths using forward slashes `/`):

1. **Cross-Service Database Table Persistence Coupling (Shared ORM Tables)**:
   Identify every database table (`Table`) in the workspace that has **3 or more distinct ORM entity classes (`@Entity` / model classes)** mapped/persisted to it across different services or libraries.
   For each such table, provide:
   - The exact table name
   - The total count of distinct entity classes mapped to it
   - Every mapped entity class name and its relative file path

2. **Cross-Service Synchronous Outbound Call Fan-In (`SERVICE_CALL`)**:
   Identify every caller service, function/method name, and relative file path across the workspace that makes an outbound synchronous service call (`linkGenerator` / HTTP client service call) to any of these 5 internal target services:
   - `svc-worker-core`
   - `svc-alert-notifier`
   - `svc-network-telemetry`
   - `svc-exchange-hub`
   - `svc-pricing-engine`
   Group your results by target service.

3. **Polyglot Asynchronous Messaging & Queue Topology (TypeScript & Go)**:
   For each of the following 5 messaging topics / queue channels across the TypeScript and Go services:
   - `topic-routing-rule-updated`
   - `PUB_SUB_PARTNER_NETWORK_TOPIC`
   - `REALTIME_BID_CACHE_QUEUE`
   - `TelemetryImpressionTopic` (`ImpressionTopic`)
   - `TelemetryConversionTopic` (`ConversionTopic`)
   Identify every function/method name and its declared relative file path that:
   (a) **Publishes** messages/events to the topic, and
   (b) **Subscribes / listens** to messages/events from the topic.

4. **Multi-Hop Inheritance & Strategy Realization Across Microservices**:
   List every class/type and its relative file path across the workspace that:
   (a) Inherits (directly or transitively) from `BaseEventController`
   (b) Implements the `ICloudPubSubService` interface
   (c) Implements the `IOptimizationCampaignStrategy` interface

> **Confidentiality Note**: All proprietary service, table, topic, class, and interface identifiers in this benchmark artifact have been systematically anonymized to preserve commercial confidentiality while maintaining a 1-to-1 structural correspondence with the underlying enterprise multi-repo workspace.
System Prompt (ce) prompts/graph-agent.md: Symmetric System Instructions for CodeExplorer Agent (Zero Schema Hints)
# Agent System Instructions

You are an expert software architect and researcher analyzing a large enterprise polyglot multi-repo microservices workspace (`324` independent service repositories, `93` shared library repositories, `13,782` source files across TypeScript/NestJS, Go, and SQL).
You must answer the user's architectural questions with 100% accuracy and completeness based on the actual codebase, providing exact symbol names and relative file paths.

CRITICAL BENCHMARK RULES:
1. You have access to the CodeExplorer CLI (`ce`) via `run_command` (which queries the pre-indexed `.codeexplorer/graph.db` semantic graph) and `view_file` to inspect source files when needed.
2. To discover available `ce` commands or inspect the graph schema (node labels, properties, and relationship types), use `ce --help`, `ce taxonomy`, or `ce status`.
3. Do NOT modify or create any files inside the repository.
System Prompt (ripgrep) prompts/baseline-agent.md: Symmetric System Instructions for ripgrep Baseline Agent
# Agent System Instructions

You are an expert software architect and researcher analyzing a large enterprise polyglot multi-repo microservices workspace (`324` independent service repositories, `93` shared library repositories, `13,782` source files across TypeScript/NestJS, Go, and SQL).
You must answer the user's architectural questions with 100% accuracy and completeness based on the actual codebase, providing exact symbol names and relative file paths.

CRITICAL BENCHMARK RULES:
1. You have access to standard CLI search commands (`rg` / `ripgrep`, `git grep`, `Get-ChildItem`, `Select-String`, or custom shell/Python scripts) via `run_command`, and `view_file` to inspect source files.
2. You are STRICTLY FORBIDDEN from running `ce`, `ce.exe`, `codeexplorer`, or accessing `.codeexplorer/graph.db` (always pass `-g "!.codeexplorer/**"` if needed).
3. Do NOT modify or create any files inside the repository.