Case Study 05: OrchardCore modular .NET monolith audit (ce vs ripgrep).

Paired A/B agent benchmark on OrchardCMS/OrchardCore (open-source modular .NET CMS & framework with 334 projects and 7,837 files). Measures how an AI agent resolves AppSec authorization invariants, YesSql.ISession boundary violations, cross-module PublishedAsync pipelines, and IContentManager blast radius.

Inspect Outputs & metrics.json Inspect Tested Prompts

~2.5×

Shorter Time to Answer

4m 25s with CodeExplorer vs 11m 04s with ripgrep + view_file across 334 projects.

92 turns / 184 steps vs 193 turns / 389 steps

~4.1×

Fewer Input Tokens

~3.9M cumulative input tokens with CodeExplorer vs ~16.1M with ripgrep (~2.9× lower API cost).

$0.65 vs $1.91 (~1.9× less tool output)

~3.3×

Fewer File Reads

36 view_file reads (31 C# files + 5 schema/spill reads) vs 117 C# reads by ripgrep.

287 KB vs 560 KB tool output

100% vs ~93%

Symbol Recall & Context

CodeExplorer matched all 331/331 items; ripgrep guessed or missed 24 Task 4a caller methods from raw grep lines.

331/331 vs 307/331 verified items

Tested participants:
1. CodeExplorer (ce): v1.78.1 (.NET 10, SQLite + OpenCypher MCP server, 134,067 nodes, 288,527 edges)
2. Lexical CLI Baseline: ripgrep (rg v14.1.1) + view_file (pure lexical search + file reading; custom AST scripts disallowed, complementing Case 02)
3. LLM Model: Gemini 3.8 Flash (matched subagent prompts without write_to_file; see methodological note below)
4. Target Repository: OrchardCMS/OrchardCore (commit 05ad2d9a, 334 projects, 7,837 files)

1. What the agents had to solve on OrchardCore.

In a modular .NET monolith like OrchardCore, architectural rules span 334 projects: verifying security invariants, layer boundaries, reactive event pipelines, and breaking-change blast radius requires joining attributes, inheritance hierarchies, member access, and multi-hop call chains.

AppSec AuthorizeAsync Audit

Task 1

Audit all 88 non-anonymous [HttpPost], [HttpPut], and [HttpDelete] actions across 5 security modules to find which actions call AuthorizeAsync within 1 to 2 hops vs which do not.

67 calling vs 21 missing AuthorizeAsync

YesSql.ISession Boundary Audit

Task 2

Identify every Controller and Display Driver across src/OrchardCore.Modules/* that bypasses domain services to invoke operations directly on an injected YesSql.ISession field.

25 classes across 13 modules (ACCESSED_BY)

PublishedAsync Event Fan-Out

Task 3

Locate all PublishingAsync and PublishedAsync content handlers across the monorepo and trace their 1 to 2 hop downstream calls into DB/index stores, tag caches, and workflows.

16 concrete + 6 base handler methods

IContentManager Blast Radius

Task 4

Find all projects calling GetAsync, CreateAsync, or PublishAsync (including via DI service locators) and isolate over-coupled DEPENDS_ON projects with 0 calls.

19 caller projects + 38 over-coupled projects

2. Execution telemetry scorecard.

Measured from paired A/B agent runs on the same OrchardCMS/OrchardCore commit with matched subagent system prompts.

Paired A/B Agent Telemetry on OrchardCMS/OrchardCore (334 Projects, 7,837 Files) Source: benchmarks/05-polyglot-architecture-and-blast-radius/results/metrics.json

Metric	`ripgrep` + File Reads	`CodeExplorer`	Delta
Time to Answer (`send_message`)	664 s (11 min 04 sec) / 701 s wall	265 s (4 min 25 sec) / 299 s wall	2.51× faster (2.34× wall)
LLM Turns, Steps & Compactions	193 turns / 389 steps (0 compactions)	92 turns / 184 steps (0 compactions)	2.10× fewer turns (2.11× fewer steps)
Total Tool Calls & File Reads	192 (74 `rg` + 117 `view_file`, 1 `send_message`)	91 (54 `mcp` + 36 `view_file`, 1 `send_message`)	2.11× fewer calls (3.25× fewer reads)
Raw Tool Output Ingested	573,355 chars (~143,339 tokens)	294,355 chars (~73,589 tokens)	1.95× less tool payload
Cumulative Input Tokens	16,098,025 est. (25,878,074 API tokens)	3,910,897 est. (6,000,146 API tokens)	4.12× fewer input tokens (4.31× API tokens)
API Cost w/ 90% Cache Discount (Gemini Flash)	$1.91 Flash ($7.51 Pro)	$0.65 Flash ($2.52 Pro)	2.94× cheaper
Symbol Recall & Context Health (Tasks 1 to 4)	92.7% (307 / 331; 24 Task 4a caller errors)	100% (331 / 331 verified items)	+24 caller edges resolved via AST

Methodological & impartiality notes:
1. Controlled baseline scope (pure rg + view_file): Unlike Case Studies 01 and 02 (where baseline agents had write_to_file access and wrote custom TypeScript/Python AST parsers), neither subagent here had write_to_file, and baseline-agent.md explicitly forbade writing custom AST/graph-walking scripts. This isolates the interactive rg + view_file workflow most coding agents run out of the box.
2. Why CodeExplorer finished in 92 turns (184 steps) vs ripgrep's 193 turns (389 steps): The CodeExplorer MCP agent executed 54 MCP calls (51 execute_custom_read_cypher queries, 2 get_taxonomy calls, and 1 get_workspace_status call) plus 36 targeted view_file calls (31 C# source files, 1 MCP schema, and 4 spilled outputs). By contrast, ripgrep ran 74 rg searches and 117 C# view_file reads, burning 4.12× more cumulative input tokens (16.10M vs 3.91M).
3. Source-verified output audit (331 ground-truth items): CodeExplorer matched 331 / 331 items (100%). ripgrep matched 307 / 331 (92.7%): while it reached 100% on Tasks 1, 2, 3, and 4b, on Task 4a (94 / 118) raw rg call-site lines lacked enclosing method headers, leading the agent to guess non-existent method names (AddCreateContentEndpoint instead of HandleAsync on CreateEndpoint, ExecuteAsync instead of SaveAsync on SaveContentItemsStep, BuildMenuShapeAsync instead of DiscoverAsync on MenuShapes, and MetaWeblogGetUserBlogsAsync in place of 5 distinct MetaWeblog* methods on MetaWeblogHandler).

3. Reproducibility: matched prompts & task specification.

Click any drawer below to expand verbatim prompt text

Task Prompt prompts/task-prompt.md: Shared 4-Part OrchardCore Audit Task Given Identically to Both Agents

Perform a comprehensive, 4-part architectural governance, security, and blast-radius audit on the `OrchardCMS/OrchardCore` modular .NET monolith located at `./repos/OrchardCore` (`src/OrchardCore/OrchardCore.*` and `src/OrchardCore.Modules/OrchardCore.*`, excluding `test/` projects, `src/Templates/`, and `*Tests*` files). Answer all four engineering questions below with 100% completeness (do not guess, sample, or truncate items). To keep both query outputs and your final report concise, **do not include file path columns** and **use compact grouped bullet lists** rather than wide multi-line Markdown tables:

1. **Task 1 — AppSec Authorization Audit: State-Mutating Controller Actions Missing `AuthorizeAsync` (`Negative Graph Query / NOT EXISTS`)**:
   Across the following 5 security-sensitive modules:
   - `OrchardCore.Users`
   - `OrchardCore.Roles`
   - `OrchardCore.Tenants`
   - `OrchardCore.ContentTypes`
   - `OrchardCore.Workflows`
   Identify every public action method (`ControllerClass.ActionMethod`) on an MVC/API controller class (inheriting from `Controller` or `ControllerBase`, or decorated with `[Controller]` / `[ApiController]`) that is decorated with `[HttpPost]`, `[HttpPut]`, or `[HttpDelete]` (excluding `[AllowAnonymous]` actions) and:
   - **(1a)** Calls `IAuthorizationService.AuthorizeAsync` directly (`1` hop) or via a helper method within `2` hops (`1..2` hops), vs
   - **(1b)** **Does NOT call `AuthorizeAsync` within `1..2` hops** (indicating whether class-level attributes like `[Admin]` or `[Authorize]` are present on the controller).
   Group by module and report the exact counts and compact list of `ControllerClass.ActionMethod` for both **(1a)** and **(1b)**.

2. **Task 2 — Architectural Boundary Violations: Web & UI Layer Bypassing Services to Access `YesSql.ISession` Directly (`Semantic Invariant Audit`)**:
   In a clean modular architecture, MVC controllers and UI display drivers should interact with domain services (`IContentManager`, `ISiteService`, stores) rather than querying or mutating the low-level `YesSql.ISession` document session directly.
   Across all modules under `src/OrchardCore.Modules/*`, identify every non-test **Controller** class (inheriting from `Controller` or `ControllerBase`, or decorated with `[Controller]` / `[ApiController]`) and **Display Driver** class (inheriting from or implementing any `*DisplayDriver*` contract such as `DisplayDriver`, `ContentPartDisplayDriver`, `ContentFieldDisplayDriver`, `SectionDisplayDriver`, or `IDisplayDriver`) whose methods **directly invoke methods on an injected `YesSql.ISession` member** (such as `Query`, `QueryIndex`, `SaveAsync`, `GetAsync`, `CancelAsync`, `CreateConnectionAsync`).
   Group by module (`.csproj` project name), report the exact count of violating classes and method-to-session operation pairs per module, and list `ClassName.MethodName -> ISession.Operation`.

3. **Task 3 — Event Lifecycle Side-Effect Fan-Out: What Happens Across Modules When Content Is Published? (`Cross-Module Reactive Pipeline`)**:
   When content is published in `OrchardCore`, the pipeline broadcasts `PublishingAsync` and `PublishedAsync` across content handlers (`IContentHandler`, `ContentHandlerBase`, `ContentPartHandler`, `ContentFieldHandler`, or `IScopedIndexProvider` implementations).
   Identify every non-test class across `src/OrchardCore/` and `src/OrchardCore.Modules/` that implements or overrides `PublishingAsync` or `PublishedAsync`, and trace its **downstream calls (`1 to 2` hops)** to classify what side-effects each handler triggers:
   - **Database / Session / Index / Content / Store mutation** (`ISession`, `*Entries`, `*IndexProvider`, `*TaskManager`, `*Store`, `IContentManager`, `IContentDefinitionManager`, `ISiteService`, `IAuditTrailManager`, `ISitemapUpdateHandler`)
   - **Cache invalidation** (`ITagCache`, `RemoveTagAsync`, `IMemoryCache`, `IDistributedCache`)
   - **Workflow / Deferred Task / Template trigger** (`IWorkflowManager`, `TriggerEventAsync`, `AddDeferredTask`, `ILiquidTemplateManager`)
   Report the total count of handler methods (`ClassName.PublishingAsync` / `ClassName.PublishedAsync`) and list `Module -> HandlerClass.MethodName -> [Triggered Downstream Services/Methods]`.

4. **Task 4 — Breaking-Change Blast Radius & Unused Project Dependencies (`Layer 1–3 Cross-Layer Coupling`)**:
   Suppose we introduce a breaking signature change to these 3 core contract methods on `IContentManager` in `OrchardCore.ContentManagement.Abstractions`:
   - `IContentManager.GetAsync`
   - `IContentManager.CreateAsync`
   - `IContentManager.PublishAsync`
   Determine across all non-test projects (`!is_test`, excluding `test/` and `src/Templates/`):
   - **(4a) Direct Call Blast Radius**: Which projects contain classes/methods that invoke at least one of these 3 `IContentManager` methods directly (`1` hop), and how many distinct caller methods (`CallerClass.CallerMethod`) invoke each of `GetAsync`, `CreateAsync`, and `PublishAsync` per project?
   - **(4b) Over-Coupled Projects (`DEPENDS_ON` Without Call Usage)**: Which non-test projects declare a direct project dependency (`ProjectReference` / `DEPENDS_ON`) on `OrchardCore.ContentManagement.Abstractions`, yet contain **zero calls (`0` call sites)** to any of `GetAsync`, `CreateAsync`, or `PublishAsync`?
   Report exact project counts and compact lists for both **(4a)** and **(4b)**.

System Prompt (ce) prompts/graph-agent.md: Symmetric System Instructions for CodeExplorer Agent

# Agent System Instructions

You are a senior software engineer analyzing the `OrchardCMS/OrchardCore` repository (`334` projects, `7,837` files).
Answer all architectural questions with 100% accuracy and completeness based on the repository.

CRITICAL BENCHMARK RULES:
1. Use the **CodeExplorer MCP server (`codeexplorer`)** (passing `workspacePath: "./repos/OrchardCore"` to query the pre-indexed `.codeexplorer/graph.db` semantic graph) and `read_file` to inspect source files when needed. Do not invoke `ce` or `ce.exe` via `run_command`; call the `codeexplorer` MCP tools directly.
2. To inspect the graph schema (node labels, properties, and relationship types) and query the graph, use the `codeexplorer` MCP tools (`get_taxonomy`, `execute_custom_read_cypher`, `find_symbol`, `get_call_chain`, `get_project_dependencies`, etc.).
3. You are STRICTLY FORBIDDEN from reading or referencing anything inside `benchmarks/`.
4. Do NOT modify or create any files inside the repository.

System Prompt (ripgrep) prompts/baseline-agent.md: Symmetric System Instructions for ripgrep Baseline Agent

# Agent System Instructions

You are a senior software engineer analyzing the `OrchardCMS/OrchardCore` repository (`334` projects, `7,837` files).
Answer all architectural questions with 100% accuracy and completeness based on the repository.

CRITICAL BENCHMARK RULES:
1. You have access to the `ripgrep` CLI (`rg`) via `run_command` and `read_file` to inspect source files when needed. Do not write custom AST/graph-walking scripts; use `rg` searches and `read_file`.
2. You are STRICTLY FORBIDDEN from running `ce`, `ce.exe`, `codeexplorer`, or accessing `.codeexplorer/graph.db` (always pass `-g "!.codeexplorer/**"` if needed).
3. You are STRICTLY FORBIDDEN from reading or referencing anything inside `benchmarks/`.
4. Do NOT modify or create any files inside the repository.