Benchmarks Catalog / Case Study 01: Agent Audit on twentyhq/twenty
46 packages, 24,527 files, 2.49M LOC

Case Study 01: Autonomous agent audit on twentyhq/twenty (ce vs CBM).

Two identical AI agents were given the same 4-part architectural audit on the public twentyhq/twenty monorepo with 100% word-for-word symmetric system prompts: one equipped with CodeExplorer and one equipped with codebase-memory-mcp (CBM). Both graphs were pre-indexed before the run.

~6.0×

Shorter Time to Answer

CodeExplorer answered in 1m 33s (2m 14s wall); CBM took 9m 15s (10m 34s wall).

7 turns vs 28 turns
~7.0×

Fewer Input Tokens

~242K cumulative input tokens vs ~1.7M tokens across the session.

$0.06 vs $0.23 (~3.9×)
100% vs 3.4%

Native Graph Recall

ce resolved 381 / 381 items directly from graph.db. CBM returned 13 / 381 (only self-calls).

0 source files read by ce
0 vs 3

Fallback AST Scripts

After 8 CBM queries returned 0 cross-file edges, the CBM agent wrote 3 custom TS AST scripts & read 10 files.

0 scripts vs 1,037 LOC of JS
Tested participants:
1. CodeExplorer (ce): v1.77.2 (.NET 10, SQLite + OpenCypher)
2. codebase-memory-mcp (CBM): v0.12.1 (Go + SQLite)
3. LLM Model: Gemini 3.8 Flash (100% symmetric system prompts)
4. Target Repository: twentyhq/twenty (commit b9ef7107, 46 packages, 24,527 files, 2.49M LOC)

1. Execution telemetry scorecard.

All metrics are extracted directly from the runtime logs (transcript_full.jsonl) of conversations 4a49e7f1 (ce) and 9a849d22 (CBM). Both agents ran under 100% word-for-word symmetric system prompts with zero pre-baked query examples.

End-to-End Agent Execution Telemetry on twentyhq/twenty (24,527 files, 100% Symmetric Prompts) Simultaneous Paired A/B Run
Metric CBM (cbm + rg + 3 AST Scripts) CodeExplorer (ce Graph Only) Delta
Time to Answer (send_message) 555 s (9m 15s) 93 s (1m 33s) 6.0× faster (-7m 42s)
Total Session Wall-Clock 634 s (10m 34s) 134 s (2m 14s) 4.7× faster (-8m 20s)
LLM Reasoning Turns 28 turns (67 steps) 7 turns (23 steps) 4.0× fewer turns
Total Tool Calls 37 (8 cbm, 15 rg/Node, 10 view_file, 3 scripts, 1 send_message) 15 (1 ce taxonomy, 13 ce query, 0 file reads, 1 send_message) 2.5× fewer calls (0 file reads)
Raw Tool Output Ingested 243 KB (~60.8K tokens) 82.1 KB (~20.5K tokens) 3.0× smaller payload
Cumulative Input Tokens 1,678,913 tokens (129K uncached + 1.55M cached) 241,528 tokens (52.5K uncached + 189.0K cached) 7.0× fewer input tokens
Generated Output Tokens 29,139 tokens 7,741 tokens 3.8× fewer output tokens
API Cost w/ 90% Cache Discount (Flash / Pro) $0.23 / $0.86 $0.06 / $0.22 3.9× cheaper
Direct Graph Recall (No Fallback) 3.4% (13 / 381 items) 100% (381 / 381 items) 29× more items from graph

2. Why the CBM agent abandoned codebase-memory-mcp after 8 queries.

In Turns 1 to 5, cbm_twenty_audit_agent ran get_graph_schema and 7 query_graph calls across all four tasks. Because CBM returned zero cross-file relationships, the agent fell back to writing 3 custom TypeScript Compiler AST scripts:

Native Graph Recall Across Tasks 1 to 4 381 Ground-Truth Items
Audit Task GT Items CBM Graph Only CBM + 3 AST Scripts CodeExplorer Graph Only Root Cause of CBM Graph Gap
Task 1: Cross-Module DI Call Fan-In
BillingSubscriptionService, ThrottlerService, AccessTokenService
93 13 / 93 (14%)
(0 / 80 DI calls)
93 / 93 (100%) 93 / 93 (100%) Tracks intra-class this.method() calls (13) but does not resolve constructor-injected services across files (0/80).
Task 2: Inheritance & Interfaces
SyncableEntity, WorkspaceRelatedEntity, WorkspaceEntityMigrationBuilderService, WorkspaceResolverBuilderFactoryInterface
167 0 / 167 (0%) 167 / 167 (100%) 167 / 167 (100%) Does not link INHERITS / IMPLEMENTS across TypeScript module imports.
Task 3: BullMQ Queue Topology
Producers (39) & Consumers (37) across 5 queues
76 0 / 76 (0%) 76 / 76 (100%) 76 / 76 (100%) Does not model NestJS @InjectMessageQueue or @Processor decorators.
Task 4: API-to-DB Write Lineage
GraphQL @Mutation / REST → Service (0..3 hops) → TypeORM Table
45 0 / 45 (0%) 45 / 45 (100%) 45 / 45 (100%) Does not model TypeORM @Entity tables, repository write methods, or GraphQL @Mutation endpoints.
Total Across All 4 Tasks 381 13 / 381 (3.4%) 381 / 381 (100%) 381 / 381 (100%) ce solved the audit in 1m 33s from graph.db; CBM required 3 custom AST scripts + 10 file reads.

Verbatim Thinking Traces from CBM Agent (Steps 7 & 16)

// Step 7 (14:21:13 UTC):
"`codebase-memory-mcp`'s output for Task 1 is limited to intra-class method calls, neglecting inter-class dependency
injection calls. This suggests a potential flaw in how it identifies and retrieves method call information across class boundaries."

// Step 16 (14:22:19 UTC):
"`codebase-memory-mcp` failed to extract critical relationships from the TypeScript codebase, specifically inheritance,
implementation details, dependency injection calls, and interactions with queuing systems and database ORMs. A new script,
either in Python or Node.js with the TypeScript compiler, is needed to accurately extract this information."

3. Turn-by-turn context accumulation curve.

Every LLM turn re-sends the active conversation history. Because the CodeExplorer agent finished in 7 turns while the CBM agent iterated through 28 turns of AST script writing and file reading, cumulative billed input tokens differed by ~7.0× (~242K vs ~1.7M).

Active Context & Cumulative Billed Input Tokens Across All 28 Turns Source: results/metrics.json
Turn / Stage CBM Agent Context
Cumulative Input
CodeExplorer Agent Context
Cumulative Input
Agent Activity
Turn 1 13,697 tokens
13,697 total
18,091 tokens
18,091 total
Both agents inspect their graph schema (get_graph_schema vs ce taxonomy).
Turn 2 14,644 tokens
28,341 total
19,921 tokens
38,012 total
Both agents dispatch parallel Cypher queries (3 in CBM, 5 in ce).
Turn 3 19,412 tokens
47,753 total
30,559 tokens
68,571 total
ce queries Task 1 and Task 3 without file paths and inspects Task 4 tables/endpoints; CBM gets 0 rows on INHERITS|IMPLEMENTS.
Turn 4 20,468 tokens
68,221 total
39,397 tokens
107,968 total
ce runs 0 to 3 hop Endpoint → Function → WRITES_TO → Table lineage queries for Task 4; CBM retries INHERITS|IMPLEMENTS (0 rows).
Turn 5 20,986 tokens
89,207 total
41,397 tokens
149,365 total
ce verifies Query write paths and max inheritance depth; CBM runs its 8th empty query and switches to rg.
Turn 6 (ce Answers @ 1m 33s) 26,845 tokens
116,052 total
43,424 tokens
192,789 total
ce agent sends its complete 4-part audit report (381 / 381 items, 100% recall) via send_message at 93.0 s.
Turn 7 (ce Done @ 2m 14s) 34,652 tokens
150,704 total
48,739 tokens
241,528 total
ce session finishes at 134.0 s; CBM agent locates the local typescript compiler package.
Turn 8 35,270 tokens
185,974 total
− CBM agent inspects TypeORM entities and BullMQ queue decorators via rg before writing custom AST scripts.
Turns 9 to 10 40,330 tokens
261,810 total
− Writes and executes audit_part123.cjs (342 LOC of custom TS Compiler AST code) for Tasks 1 to 3.
Turns 11 to 16 66,022 tokens
591,310 total
− Writes and executes audit_part4.cjs (345 LOC), inspects truncated tool outputs, and runs follow-up rg checks.
Turns 17 to 19 78,629 tokens
817,158 total
− Writes and executes verify_part3_and_4.cjs (350 LOC) to refine TypeORM repository and entityManager write detection.
Turns 20 to 26 99,628 tokens
1,469,621 total
− Runs rg and reads 6 source files (GraphQL resolvers, services, and ApiPath.ts) via view_file to verify routes.
Turn 27 (CBM Answers @ 9m 15s) 100,346 tokens
1,569,967 total
− CBM agent sends its final audit report via send_message at 555.0 s.
Turn 28 (CBM Done @ 10m 34s) 108,946 tokens
1,678,913 total
− CBM session finishes at 634.0 s total wall-clock.

4. Reproducibility: tested prompts & 100% symmetric agent instructions.

Click any drawer below to expand verbatim prompt text
Task Prompt prompts/task-prompt.md: Shared 4-Part Audit Task Given Identically to Both Agents
Perform a comprehensive, 4-part architectural audit on the `twenty` monorepo located at `./repos/twenty` (`24,527` tracked files across `46` packages). Answer all four architectural questions below with 100% completeness (do not guess, truncate, or omit items; exclude `.spec.ts` / `.test.ts` / `__tests__` test files). To keep both query outputs and your final report concise, **do not include file path columns** and **use compact grouped bullet lists** rather than wide multi-line Markdown tables:

1. **Task 1 — Service Method Call Graph & DI Fan-In (`Layer 3`)**:
   Identify every non-test caller function/method (`CallerClass.callerMethod` and target `calleeMethod`) across the monorepo that invokes any method on these 3 core services:
   - `BillingSubscriptionService`
   - `ThrottlerService`
   - `AccessTokenService`
   Group by target service, report the total call pair count (`totalCallPairs`) for each service, and list `callerMethod -> calleeMethod` pairs grouped compactly by `CallerClass`.

2. **Task 2 — Multi-Hop Class Inheritance & Interface Realization (`Layer 3`)**:
   List every non-test class/type name (`className`) across the monorepo that:
   (a) Inherits **directly or transitively (1 to 3 hops)** from `SyncableEntity`
   (b) Inherits **directly or transitively (1 to 3 hops)** from `WorkspaceRelatedEntity` (include both direct 1-hop subclasses and 2-hop transitive subclasses)
   (c) Inherits (directly or transitively) from `WorkspaceEntityMigrationBuilderService`
   (d) Implements the `WorkspaceResolverBuilderFactoryInterface` interface
   For each of (a), (b), (c), and (d), report the exact total count and a compact comma-separated list of `className` values (grouped by hop distance where applicable).

3. **Task 3 — Asynchronous BullMQ Queue Producers & Consumers (`Layer 4`)**:
   For each of the following 5 BullMQ queues in `twenty`:
   - `webhook-queue`
   - `workspace-queue`
   - `campaign-queue`
   - `ai-queue`
   - `delete-cascade-queue`
   Identify every non-test `ClassName.methodName` that:
   (a) **Publishes / enqueues** jobs to the queue, and
   (b) **Subscribes / processes** jobs from the queue.
   Report exact counts and a compact comma-separated list of `ClassName.methodName` per queue and role.

4. **Task 4 — End-to-End API-to-Database Write Lineage (`Layer 4–5`)**:
   For each of the following 4 database tables:
   - `billingSubscription`
   - `billingCustomer`
   - `connectedAccount`
   - `messageChannel`
   Identify every incoming GraphQL (`@Mutation` / `@Query`) or REST (`@Controller`) endpoint and its handler/resolver method (`endpointName`, `httpMethod`/operation, and `HandlerClass.handlerMethodName`) that **writes (`WRITES_TO` — inserts, updates, saves, upserts, or deletes)** to that table either directly (`0` hops) or within `1 to 3` function call hops (`Endpoint -> Handler/Resolver -> Service -> Repository/Query -> Table`). Provide the distinct count and compact list of `endpointName (httpMethod) -> HandlerClass.handlerMethodName` per table.
System Prompt (ce) prompts/ce-agent.md: System Instructions for CodeExplorer Agent (ce_twenty_audit_agent)
# Agent System Instructions (`ce_twenty_audit_agent`)

You are an expert software architect and researcher equipped with the **CodeExplorer CLI (`ce`)** plus standard filesystem/search tools (`rg`, `read_file`, Python/PowerShell scripts).
You are analyzing the `twenty` monorepo at `./repos/twenty` (`24,527` tracked files, `46` packages), which is pre-indexed in `./repos/twenty/.codeexplorer/graph.db`.

CRITICAL BENCHMARK RULES:
1. Use your assigned graph CLI (`ce`) as your primary source of truth. First inspect the graph schema to understand available node labels and relationships. If and only if the graph does not contain the relationships needed to answer a question completely, you may fall back to `rg`, `read_file`, or scripts.
2. Available graph CLI commands (run via `run_command`):
   - Inspect schema & ontology:
     `ce taxonomy -d "./repos/twenty"` (or `ce --help`)
   - Query graph (`Cypher`):
     `ce query "<CYPHER>" -d "./repos/twenty" --no-truncate`
   - Search symbols:
     `ce search "<QUERY>" -d "./repos/twenty"`
   - Trace callers/callees:
     `ce callers "<NAME>" -d "./repos/twenty"`
3. You are STRICTLY FORBIDDEN from running `codebase-memory-mcp` or accessing any CBM cache files. Always pass `-g "!.codeexplorer/**"` when running `rg`.
4. Do NOT modify or create any files inside `./repos/twenty`.
System Prompt (CBM) prompts/cbm-agent.md: System Instructions for codebase-memory-mcp Agent (cbm_twenty_audit_agent)
# Agent System Instructions (`cbm_twenty_audit_agent`)

You are an expert software architect and researcher equipped with the **`codebase-memory-mcp` CLI (`codebase-memory-mcp`)** plus standard filesystem/search tools (`rg`, `read_file`, Python/PowerShell scripts).
You are analyzing the `twenty` monorepo at `./repos/twenty` (`24,527` tracked files, `46` packages), which is pre-indexed in `codebase-memory-mcp` under project name `twenty`.

CRITICAL BENCHMARK RULES:
1. Use your assigned graph CLI (`codebase-memory-mcp`) as your primary source of truth. First inspect the graph schema to understand available node labels and relationships. If and only if the graph does not contain the relationships needed to answer a question completely, you may fall back to `rg`, `read_file`, or scripts.
2. Available graph CLI commands (run via `run_command`):
   - Inspect schema & ontology:
     `codebase-memory-mcp cli get_graph_schema --project twenty --format json` (or `codebase-memory-mcp --help`)
   - Query graph (`Cypher`):
     `codebase-memory-mcp cli query_graph --project twenty --query "<CYPHER>" --max-rows 5000 --max-output-tokens 100000 --format json`
   - Search symbols:
     `codebase-memory-mcp cli search_graph --project twenty --name-pattern "<REGEX>" --format json`
   - Trace callers/callees:
     `codebase-memory-mcp cli trace_path --project twenty --function-name "<NAME>" --direction inbound --format json`
3. You are STRICTLY FORBIDDEN from running `ce`, `codeexplorer`, or accessing any `.codeexplorer` / `graph.db` files. Always pass `-g "!.codeexplorer/**"` when running `rg`.
4. Do NOT modify or create any files inside `./repos/twenty`.