Case Study 01: Autonomous agent audit on twentyhq/twenty (ce vs CBM).
Two identical AI agents were given the same 4-part architectural audit on the public twentyhq/twenty monorepo with 100% word-for-word symmetric system prompts: one equipped with CodeExplorer and one equipped with codebase-memory-mcp (CBM). Both graphs were pre-indexed before the run.
Shorter Time to Answer
CodeExplorer answered in 1m 33s (2m 14s wall); CBM took 9m 15s (10m 34s wall).
7 turns vs 28 turns
Fewer Input Tokens
~242K cumulative input tokens vs ~1.7M tokens across the session.
$0.06 vs $0.23 (~3.9×)
Native Graph Recall
ce resolved 381 / 381 items directly from graph.db. CBM returned 13 / 381 (only self-calls).
0 source files read by ce
Fallback AST Scripts
After 8 CBM queries returned 0 cross-file edges, the CBM agent wrote 3 custom TS AST scripts & read 10 files.
0 scripts vs 1,037 LOC of JS
1. CodeExplorer (
ce): v1.77.2 (.NET 10, SQLite + OpenCypher)2. codebase-memory-mcp (
CBM): v0.12.1 (Go + SQLite)3. LLM Model:
Gemini 3.8 Flash (100% symmetric system prompts)4. Target Repository:
twentyhq/twenty (commit b9ef7107, 46 packages, 24,527 files, 2.49M LOC)1. Execution telemetry scorecard.
All metrics are extracted directly from the runtime logs (transcript_full.jsonl) of conversations 4a49e7f1 (ce) and 9a849d22 (CBM). Both agents ran under 100% word-for-word symmetric system prompts with zero pre-baked query examples.
| Metric | CBM (cbm + rg + 3 AST Scripts) |
CodeExplorer (ce Graph Only) |
Delta |
|---|---|---|---|
Time to Answer (send_message) |
555 s (9m 15s) | 93 s (1m 33s) | 6.0× faster (-7m 42s) |
| Total Session Wall-Clock | 634 s (10m 34s) | 134 s (2m 14s) | 4.7× faster (-8m 20s) |
| LLM Reasoning Turns | 28 turns (67 steps) | 7 turns (23 steps) | 4.0× fewer turns |
| Total Tool Calls | 37 (8 cbm, 15 rg/Node, 10 view_file, 3 scripts, 1 send_message) |
15 (1 ce taxonomy, 13 ce query, 0 file reads, 1 send_message) |
2.5× fewer calls (0 file reads) |
| Raw Tool Output Ingested | 243 KB (~60.8K tokens) | 82.1 KB (~20.5K tokens) | 3.0× smaller payload |
| Cumulative Input Tokens | 1,678,913 tokens (129K uncached + 1.55M cached) | 241,528 tokens (52.5K uncached + 189.0K cached) | 7.0× fewer input tokens |
| Generated Output Tokens | 29,139 tokens | 7,741 tokens | 3.8× fewer output tokens |
| API Cost w/ 90% Cache Discount (Flash / Pro) | $0.23 / $0.86 | $0.06 / $0.22 | 3.9× cheaper |
| Direct Graph Recall (No Fallback) | 3.4% (13 / 381 items) | 100% (381 / 381 items) | 29× more items from graph |
2. Why the CBM agent abandoned codebase-memory-mcp after 8 queries.
In Turns 1 to 5, cbm_twenty_audit_agent ran get_graph_schema and 7 query_graph calls across all four tasks. Because CBM returned zero cross-file relationships, the agent fell back to writing 3 custom TypeScript Compiler AST scripts:
| Audit Task | GT Items | CBM Graph Only |
CBM + 3 AST Scripts |
CodeExplorer Graph Only |
Root Cause of CBM Graph Gap |
|---|---|---|---|---|---|
Task 1: Cross-Module DI Call Fan-InBillingSubscriptionService, ThrottlerService, AccessTokenService |
93 | 13 / 93 (14%) (0 / 80 DI calls) |
93 / 93 (100%) | 93 / 93 (100%) | Tracks intra-class this.method() calls (13) but does not resolve constructor-injected services across files (0/80). |
Task 2: Inheritance & InterfacesSyncableEntity, WorkspaceRelatedEntity, WorkspaceEntityMigrationBuilderService, WorkspaceResolverBuilderFactoryInterface |
167 | 0 / 167 (0%) | 167 / 167 (100%) | 167 / 167 (100%) | Does not link INHERITS / IMPLEMENTS across TypeScript module imports. |
| Task 3: BullMQ Queue Topology Producers (39) & Consumers (37) across 5 queues |
76 | 0 / 76 (0%) | 76 / 76 (100%) | 76 / 76 (100%) | Does not model NestJS @InjectMessageQueue or @Processor decorators. |
| Task 4: API-to-DB Write Lineage GraphQL @Mutation / REST → Service (0..3 hops) → TypeORM Table |
45 | 0 / 45 (0%) | 45 / 45 (100%) | 45 / 45 (100%) | Does not model TypeORM @Entity tables, repository write methods, or GraphQL @Mutation endpoints. |
| Total Across All 4 Tasks | 381 | 13 / 381 (3.4%) | 381 / 381 (100%) | 381 / 381 (100%) | ce solved the audit in 1m 33s from graph.db; CBM required 3 custom AST scripts + 10 file reads. |
Verbatim Thinking Traces from CBM Agent (Steps 7 & 16)
// Step 7 (14:21:13 UTC): "`codebase-memory-mcp`'s output for Task 1 is limited to intra-class method calls, neglecting inter-class dependency injection calls. This suggests a potential flaw in how it identifies and retrieves method call information across class boundaries." // Step 16 (14:22:19 UTC): "`codebase-memory-mcp` failed to extract critical relationships from the TypeScript codebase, specifically inheritance, implementation details, dependency injection calls, and interactions with queuing systems and database ORMs. A new script, either in Python or Node.js with the TypeScript compiler, is needed to accurately extract this information."
3. Turn-by-turn context accumulation curve.
Every LLM turn re-sends the active conversation history. Because the CodeExplorer agent finished in 7 turns while the CBM agent iterated through 28 turns of AST script writing and file reading, cumulative billed input tokens differed by ~7.0× (~242K vs ~1.7M).
| Turn / Stage | CBM Agent ContextCumulative Input |
CodeExplorer Agent ContextCumulative Input |
Agent Activity |
|---|---|---|---|
| Turn 1 | 13,697 tokens 13,697 total |
18,091 tokens 18,091 total |
Both agents inspect their graph schema (get_graph_schema vs ce taxonomy). |
| Turn 2 | 14,644 tokens 28,341 total |
19,921 tokens 38,012 total |
Both agents dispatch parallel Cypher queries (3 in CBM, 5 in ce). |
| Turn 3 | 19,412 tokens 47,753 total |
30,559 tokens 68,571 total |
ce queries Task 1 and Task 3 without file paths and inspects Task 4 tables/endpoints; CBM gets 0 rows on INHERITS|IMPLEMENTS. |
| Turn 4 | 20,468 tokens 68,221 total |
39,397 tokens 107,968 total |
ce runs 0 to 3 hop Endpoint → Function → WRITES_TO → Table lineage queries for Task 4; CBM retries INHERITS|IMPLEMENTS (0 rows). |
| Turn 5 | 20,986 tokens 89,207 total |
41,397 tokens 149,365 total |
ce verifies Query write paths and max inheritance depth; CBM runs its 8th empty query and switches to rg. |
Turn 6 (ce Answers @ 1m 33s) |
26,845 tokens 116,052 total |
43,424 tokens 192,789 total |
ce agent sends its complete 4-part audit report (381 / 381 items, 100% recall) via send_message at 93.0 s. |
Turn 7 (ce Done @ 2m 14s) |
34,652 tokens 150,704 total |
48,739 tokens 241,528 total |
ce session finishes at 134.0 s; CBM agent locates the local typescript compiler package. |
| Turn 8 | 35,270 tokens 185,974 total |
− | CBM agent inspects TypeORM entities and BullMQ queue decorators via rg before writing custom AST scripts. |
| Turns 9 to 10 | 40,330 tokens 261,810 total |
− | Writes and executes audit_part123.cjs (342 LOC of custom TS Compiler AST code) for Tasks 1 to 3. |
| Turns 11 to 16 | 66,022 tokens 591,310 total |
− | Writes and executes audit_part4.cjs (345 LOC), inspects truncated tool outputs, and runs follow-up rg checks. |
| Turns 17 to 19 | 78,629 tokens 817,158 total |
− | Writes and executes verify_part3_and_4.cjs (350 LOC) to refine TypeORM repository and entityManager write detection. |
| Turns 20 to 26 | 99,628 tokens 1,469,621 total |
− | Runs rg and reads 6 source files (GraphQL resolvers, services, and ApiPath.ts) via view_file to verify routes. |
Turn 27 (CBM Answers @ 9m 15s) |
100,346 tokens 1,569,967 total |
− | CBM agent sends its final audit report via send_message at 555.0 s. |
Turn 28 (CBM Done @ 10m 34s) |
108,946 tokens 1,678,913 total |
− | CBM session finishes at 634.0 s total wall-clock. |
4. Reproducibility: tested prompts & 100% symmetric agent instructions.
Click any drawer below to expand verbatim prompt textTask Prompt prompts/task-prompt.md: Shared 4-Part Audit Task Given Identically to Both Agents
Perform a comprehensive, 4-part architectural audit on the `twenty` monorepo located at `./repos/twenty` (`24,527` tracked files across `46` packages). Answer all four architectural questions below with 100% completeness (do not guess, truncate, or omit items; exclude `.spec.ts` / `.test.ts` / `__tests__` test files). To keep both query outputs and your final report concise, **do not include file path columns** and **use compact grouped bullet lists** rather than wide multi-line Markdown tables: 1. **Task 1 — Service Method Call Graph & DI Fan-In (`Layer 3`)**: Identify every non-test caller function/method (`CallerClass.callerMethod` and target `calleeMethod`) across the monorepo that invokes any method on these 3 core services: - `BillingSubscriptionService` - `ThrottlerService` - `AccessTokenService` Group by target service, report the total call pair count (`totalCallPairs`) for each service, and list `callerMethod -> calleeMethod` pairs grouped compactly by `CallerClass`. 2. **Task 2 — Multi-Hop Class Inheritance & Interface Realization (`Layer 3`)**: List every non-test class/type name (`className`) across the monorepo that: (a) Inherits **directly or transitively (1 to 3 hops)** from `SyncableEntity` (b) Inherits **directly or transitively (1 to 3 hops)** from `WorkspaceRelatedEntity` (include both direct 1-hop subclasses and 2-hop transitive subclasses) (c) Inherits (directly or transitively) from `WorkspaceEntityMigrationBuilderService` (d) Implements the `WorkspaceResolverBuilderFactoryInterface` interface For each of (a), (b), (c), and (d), report the exact total count and a compact comma-separated list of `className` values (grouped by hop distance where applicable). 3. **Task 3 — Asynchronous BullMQ Queue Producers & Consumers (`Layer 4`)**: For each of the following 5 BullMQ queues in `twenty`: - `webhook-queue` - `workspace-queue` - `campaign-queue` - `ai-queue` - `delete-cascade-queue` Identify every non-test `ClassName.methodName` that: (a) **Publishes / enqueues** jobs to the queue, and (b) **Subscribes / processes** jobs from the queue. Report exact counts and a compact comma-separated list of `ClassName.methodName` per queue and role. 4. **Task 4 — End-to-End API-to-Database Write Lineage (`Layer 4–5`)**: For each of the following 4 database tables: - `billingSubscription` - `billingCustomer` - `connectedAccount` - `messageChannel` Identify every incoming GraphQL (`@Mutation` / `@Query`) or REST (`@Controller`) endpoint and its handler/resolver method (`endpointName`, `httpMethod`/operation, and `HandlerClass.handlerMethodName`) that **writes (`WRITES_TO` — inserts, updates, saves, upserts, or deletes)** to that table either directly (`0` hops) or within `1 to 3` function call hops (`Endpoint -> Handler/Resolver -> Service -> Repository/Query -> Table`). Provide the distinct count and compact list of `endpointName (httpMethod) -> HandlerClass.handlerMethodName` per table.
System Prompt (ce) prompts/ce-agent.md: System Instructions for CodeExplorer Agent (ce_twenty_audit_agent)
# Agent System Instructions (`ce_twenty_audit_agent`)
You are an expert software architect and researcher equipped with the **CodeExplorer CLI (`ce`)** plus standard filesystem/search tools (`rg`, `read_file`, Python/PowerShell scripts).
You are analyzing the `twenty` monorepo at `./repos/twenty` (`24,527` tracked files, `46` packages), which is pre-indexed in `./repos/twenty/.codeexplorer/graph.db`.
CRITICAL BENCHMARK RULES:
1. Use your assigned graph CLI (`ce`) as your primary source of truth. First inspect the graph schema to understand available node labels and relationships. If and only if the graph does not contain the relationships needed to answer a question completely, you may fall back to `rg`, `read_file`, or scripts.
2. Available graph CLI commands (run via `run_command`):
- Inspect schema & ontology:
`ce taxonomy -d "./repos/twenty"` (or `ce --help`)
- Query graph (`Cypher`):
`ce query "<CYPHER>" -d "./repos/twenty" --no-truncate`
- Search symbols:
`ce search "<QUERY>" -d "./repos/twenty"`
- Trace callers/callees:
`ce callers "<NAME>" -d "./repos/twenty"`
3. You are STRICTLY FORBIDDEN from running `codebase-memory-mcp` or accessing any CBM cache files. Always pass `-g "!.codeexplorer/**"` when running `rg`.
4. Do NOT modify or create any files inside `./repos/twenty`.
System Prompt (CBM) prompts/cbm-agent.md: System Instructions for codebase-memory-mcp Agent (cbm_twenty_audit_agent)
# Agent System Instructions (`cbm_twenty_audit_agent`)
You are an expert software architect and researcher equipped with the **`codebase-memory-mcp` CLI (`codebase-memory-mcp`)** plus standard filesystem/search tools (`rg`, `read_file`, Python/PowerShell scripts).
You are analyzing the `twenty` monorepo at `./repos/twenty` (`24,527` tracked files, `46` packages), which is pre-indexed in `codebase-memory-mcp` under project name `twenty`.
CRITICAL BENCHMARK RULES:
1. Use your assigned graph CLI (`codebase-memory-mcp`) as your primary source of truth. First inspect the graph schema to understand available node labels and relationships. If and only if the graph does not contain the relationships needed to answer a question completely, you may fall back to `rg`, `read_file`, or scripts.
2. Available graph CLI commands (run via `run_command`):
- Inspect schema & ontology:
`codebase-memory-mcp cli get_graph_schema --project twenty --format json` (or `codebase-memory-mcp --help`)
- Query graph (`Cypher`):
`codebase-memory-mcp cli query_graph --project twenty --query "<CYPHER>" --max-rows 5000 --max-output-tokens 100000 --format json`
- Search symbols:
`codebase-memory-mcp cli search_graph --project twenty --name-pattern "<REGEX>" --format json`
- Trace callers/callees:
`codebase-memory-mcp cli trace_path --project twenty --function-name "<NAME>" --direction inbound --format json`
3. You are STRICTLY FORBIDDEN from running `ce`, `codeexplorer`, or accessing any `.codeexplorer` / `graph.db` files. Always pass `-g "!.codeexplorer/**"` when running `rg`.
4. Do NOT modify or create any files inside `./repos/twenty`.