Benchmarks Catalog / Case Study 03: CodeExplorer vs codebase-memory-mcp
twenty (2.49M LOC) & immich (461k LOC)

Case Study 03: CodeExplorer vs codebase-memory-mcp (CBM).

Head-to-head indexer and MCP server comparison on twenty and immich: how accurately each engine extracts routes, function calls, inheritance, queues, and database tables across 371 source-verified items.

78% vs 38%

Basic Code & Call Graph

Across 130 basic items supported by both tools (routes, HTTP calls, DI method calls, class inheritance): ce scored 102/130 vs CBM 49/130 (7 wins, 1 tie, 0 losses).

102 / 130 vs 49 / 130
~98%

Queues, DB Tables & SQL

Across 241 message queue, ORM table, SQL read/write lineage, and sub-project items: ce matched 235/241. CBM does not index these.

235 / 241 verified items
~4.3 to 4.7ms

OpenCypher Latency

Single-shot warm OpenCypher round-trip in ce (4.3 to 4.7 ms) vs 38.1 to 71.2 ms in CBM over persistent JSON-RPC stdio.

~8.9× to 15× faster Cypher query
97s vs 147s

Where CBM Won

CBM indexed twenty ~1.5× faster with ~2.2× lower peak RAM (2.1 GB vs 4.7 GB), while ce was faster on immich with a 5.1× smaller DB.

CBM faster on 24.5k-file cold scan
Tested participants:
1. CodeExplorer (ce): v1.75.0 (.NET 10, SQLite + OpenCypher, 36 MCP tools)
2. codebase-memory-mcp (CBM): v0.12.1 (win32-x64 Go + SQLite binary, 17 MCP tools)
3. Target Repositories: twentyhq/twenty (commit 12e8fae4, 24,527 files, 2.49M LOC) & immich-app/immich (commit 2153d22f, 3,205 files, 461k LOC)

1. Graph completeness: basic code vs full architecture.

CBM indexes files, classes, routes, and function calls, whereas CodeExplorer also extracts message queues, ORM database tables, and SQL read/write queries. To keep the comparison fair, we separate the 371 verified items into Basic Code (130 items supported by both tools) and Architecture, DB & Queues (241 items supported only by CodeExplorer).

Group A: Basic Code & Call Graph (130 Verified Items Supported by Both Tools, seed=42) ce: 7 Wins, 1 Tie, 0 Losses vs CBM
What Was Tested Repo Sample Size CodeExplorer CBM Why the Scores Differ
Incoming HTTP & GraphQL Endpoints (C1) twenty 20 100% (20/20) 0% (0/20) ce indexes NestJS GraphQL @Resolver operations and template-literal controller routes.
Incoming HTTP & FastAPI Endpoints (C1) immich 20 100% (20/20) 90% (18/20) ce resolves TypeScript enum route prefixes (@Controller(RouteKey.Asset)).
Outbound HTTP Client Calls (C2) twenty 12 50% (6/12) 25% (3/12) ce extracts fetch, axios, and wrapper HTTP client calls.
Outbound HTTP Client Calls (C2) immich 6 83% (5/6) 0% (0/6) ce resolves variable-URL and template-literal HTTP requests.
Function & DI Service Call Graph (C6) twenty 12 100% (12/12) 75% (9/12) CBM misses calls on constructor-injected TypeScript service fields across files.
Function & DI Service Call Graph (C6) immich 20 100% (20/20) 60% (12/20) ce links NestJS constructor DI properties to target class methods.
Class Inheritance & implements (C7) twenty 20 80% (16/20) 20% (4/20) ce links cross-package base classes and generic TypeScript interfaces.
Class Inheritance & Mixins (C7) immich 20 15% (3/20) 15% (3/20) Tie at 15%: 17 of 20 DTOs use functional mixin inheritance (extends createZodDto(...)).
Group A Total: Basic Code & Call Graph Both 130 78% (102/130) 38% (49/130) On immich alone (66 GT): ce 73% (48/66) vs CBM 50% (33/66).
Group B Total: Queues, DB Tables & SQL Lineage Both 241 97.5% (235/241) Not modeled (0/241) BullMQ producers/consumers, TypeORM/Kysely tables & columns, SQL read/write lineage, sub-projects.

2. Cold indexing speed, disk size, and memory.

CodeExplorer was faster and produced a 5.1× smaller SQLite database on immich (3,205 files). On twenty (24,527 files), CBM completed cold indexing ~1.5× faster with lower peak RAM because it extracts only basic AST and call edges, whereas ce also runs 250 framework, ORM, and queue analyzers.

Cold Indexing Performance on twenty (24,527 files) & immich (3,205 files) Identical Windows Workstation
Metric Repo CodeExplorer CBM Winner & Notes
Cold Index Wall Time immich (3.2k files) 17.60 s 20.60 s ce (1.17× faster)
Cold Index Wall Time twenty (24.5k files) 147.33 s (Layer 1 to 5 pipeline) 96.90 s (Layer 1 to 3 pipeline) CBM (1.52× faster)
SQLite Database Size on Disk immich / twenty 55.1 MB / 784.0 MB 282.0 MB / 842.5 MB ce (5.1× smaller on immich; 7% smaller on twenty)
Peak Working Set (RAM) immich / twenty 780.5 MB / 4,660.6 MB 595.0 MB / 2,076.0 MB CBM (1.3× to 2.2× lower peak RAM)
Parse Warnings immich / twenty 0 warnings / 0 warnings 4 partial / 209 partial + 1 unusable ce (0 parse failures)

3. MCP server latency & tool trade-offs.

Single-shot JSON-RPC round-trip measurements over persistent stdio sessions (results/mcp-benchmark.json).

Single-Shot Warm JSON-RPC Latency (immich / twenty) ce (36 tools) vs CBM (17 tools)
MCP Operation CodeExplorer (immich / twenty) CBM (immich / twenty) Winner & Notes
Architecture Overview (get_architecture_map / get_architecture) 267.7 ms (4.4 KB) / 371.7 ms (11.0 KB) 117.5 ms (3.4 KB) / 286.9 ms (6.1 KB) CBM faster; ce lists all 21 / 46 sub-projects and DBs.
Graph Schema / Taxonomy (1st / Cached) 22.3 ms (1.9 ms cached) / 23.2 ms (2.7 ms cached) 35.6 ms (40.9 ms 2nd) / 72.0 ms (75.3 ms 2nd) ce 1.6× to 3.1× faster (21× to 28× cached) with full property schema (51.5 to 54.7 KB vs 1.0 KB).
Read-Only Cypher Query (LIMIT 10) 4.3 ms (1.0 KB) / 4.7 ms (1.1 KB) 38.1 ms (0.3 KB) / 71.2 ms (0.3 KB) ce 8.9× to 15× faster on single-shot Cypher execution.
Inbound Callers by Method Name (validate) 76.5 ms (1.5 KB) / 530.1 ms (2.5 KB) 7.5 ms (18.6 KB ambiguous) / 6.5 ms (45.6 KB ambiguous) ce resolves callers in 1 turn; CBM returns 62 / 67 ambiguous candidates requiring a 2nd call.
Symbol Lookup by Name (AuthService) 80.8 ms (0.4 KB) / 367.3 ms (1.1 KB) 55.6 ms (1.1 KB) / 97.5 ms (0.8 KB) CBM 1.5× to 3.8× faster on symbol lookup by name.
File Outline (auth.service.ts / auth.resolver.ts) 13.4 ms (0.4 KB) / 92.7 ms (0.9 KB) 17.3 ms (4.8 KB) / 40.5 ms (7.8 KB) ce 1.3× faster on immich; CBM 2.3× faster on twenty (ce payload 8× to 12× smaller).
Unique Capabilities VS Code extension, 32+ CI gates (ce gate), data lineage, test impact Snapshot diffing (compare_graphs), OpenTelemetry overlay (ingest_traces), ADRs (manage_adr) Complementary tool surfaces beyond core graph queries.

4. Reproducibility: tested queries & ground-truth specification.

Click any drawer below to expand verbatim queries
Group A Queries (C1, C2, C6, C7) scripts/evaluate_ce_vs_cbm.py: Exact OpenCypher Queries Executed on Both Tools (130 Basic Code Items)
# C1: Incoming HTTP / GraphQL / FastAPI Endpoints
# CodeExplorer (ce):
MATCH (e:Endpoint)-[:EXPOSED_BY]->(f:Function)
RETURN f.file_path, f.start_line, f.name, e.http_method, e.route_template

# codebase-memory-mcp (CBM):
MATCH (m)-[:HANDLES]->(r:Route)
RETURN m.file_path, m.start_line, m.name, r.name

# ------------------------------------------------------------------------------
# C2: Outbound Inter-Service & External HTTP Calls
# CodeExplorer (ce):
MATCH (ext:ExternalService)-[r:CALLED_BY]->(f:Function)
WHERE COALESCE(ext.protocol, 'http') <> 'graphql'
RETURN f.file_path, COALESCE(r.line, f.start_line), f.name, COALESCE(ext.url, ext.name);

MATCH (f:Function)-[r:CALLS_ENDPOINT]->(ep:Endpoint)
WHERE COALESCE(ep.protocol, 'REST') <> 'GraphQL'
RETURN f.file_path, COALESCE(r.line, f.start_line), f.name, COALESCE(ep.route_template, ep.name);

# codebase-memory-mcp (CBM):
MATCH (a)-[:HTTP_CALLS]->(b)
RETURN a.file_path, a.start_line, a.name, b.name

# ------------------------------------------------------------------------------
# C6: Call Graph (Direct & DI Service Method Callers)
# CodeExplorer (ce):
MATCH (caller:Function)-[:CALLS]->(callee:Function)
WHERE callee.name IN [<target_methods>]
OPTIONAL MATCH (enclosing:Function)-[:DECLARES]->(caller)
RETURN caller.file_path, COALESCE(enclosing.start_line, caller.start_line), COALESCE(enclosing.name, caller.name), callee.name

# codebase-memory-mcp (CBM):
MATCH (caller)-[:CALLS]->(callee)
WHERE callee.name = '<target_method>'
RETURN caller.file_path, caller.start_line, caller.name, callee.name

# ------------------------------------------------------------------------------
# C7: Class Inheritance & Interface Implementation
# CodeExplorer (ce):
MATCH (c:Type)-[r:INHERITS_FROM|IMPLEMENTS]->(p)
RETURN c.file_path, c.start_line, c.name, type(r), p.name

# codebase-memory-mcp (CBM):
MATCH (c)-[r:INHERITS|IMPLEMENTS]->(p)
RETURN c.file_path, c.start_line, c.name, type(r), p.name
Group B Queries (C3 to C5, C8 to C11) scripts/evaluate_ce_vs_cbm.py: Queue, ORM Table, Data Lineage & 6 Architectural Queries (241 Items)
# C3 & C8: Message Queue Producers & Consumers (BullMQ / @OnJob)
MATCH (t:Topic)-[:PUBLISHED_BY]->(f:Function)
RETURN f.file_path, f.start_line, f.name, t.name;

MATCH (t:Topic)-[:SUBSCRIBED_BY]->(m:Function)
RETURN m.file_path, m.start_line, m.name, t.name;

# C4 & C10: Database Tables, Entities & Schema Members
MATCH (t:Table) OPTIONAL MATCH (t)-[:IMPLEMENTED_BY]->(ent:Type)
RETURN COALESCE(ent.file_path, t.path), ent.start_line, COALESCE(ent.name, t.name), t.name;

MATCH (t:Table)-[:IMPLEMENTED_BY]->(ent:Type)-[:HAS_MEMBER]->(m:Member)
RETURN ent.file_path, m.start_line, m.name, t.name;

# C5: Code-to-Table Read/Write Lineage (TypeORM & Kysely)
MATCH (q:Query)-[r:WRITES_TO|READS_FROM]->(t:Table)
OPTIONAL MATCH (q)-[:ACCESSED_BY]->(f:Function)
RETURN COALESCE(f.file_path, q.file_path), q.start_line, COALESCE(f.name, q.name), type(r), t.name;

# C9: Monorepo Services & Sub-Projects
MATCH (p:Project) RETURN p.path, labels(p), p.name;

# C11: 6 End-to-End Architectural Queries (Single-Query Expressibility)
# Q1 — Topics with producers but no consumers (or vice versa):
MATCH (t:Topic) WHERE NOT (t)-[:SUBSCRIBED_BY]->() OR NOT (t)-[:PUBLISHED_BY]->() RETURN t.name
# Q2 — Pairs of projects sharing a database:
MATCH (p1:Project)-[:USES_DB]->(d:Database)<-[:USES_DB]-(p2:Project) WHERE p1.name < p2.name RETURN p1.name, p2.name, d.name
# Q3 — Top written database tables:
MATCH (q:Query)-[:WRITES_TO]->(t:Table) RETURN t.name, count(q) AS writes ORDER BY writes DESC LIMIT 10
# Q4 — Endpoints transitively writing to a database table within 1..4 call hops:
MATCH (e:Endpoint)-[:EXPOSED_BY]->(f:Function)-[:CALLS*1..4]->(w:Function)<-[:DECLARED_IN]-(q:Query)-[:WRITES_TO]->(t:Table) RETURN e.name, t.name LIMIT 20
# Q5 — Per-project endpoint and published topic counts:
MATCH (p:Project) OPTIONAL MATCH (p)-[:EXPOSES]->(e:Endpoint) OPTIONAL MATCH (p)-[:PUBLISHES_TO]->(tp:Topic) RETURN p.name, count(DISTINCT e), count(DISTINCT tp)
# Q6 — End-to-end chain: Endpoint -> Call Chain (0..3 hops) -> Queue Publish -> Queue Subscriber:
MATCH (e:Endpoint)-[:EXPOSED_BY]->(f:Function)-[:CALLS*0..3]->(pub:Function)<-[:PUBLISHED_BY]-(t:Topic)-[:SUBSCRIBED_BY]->(sub) RETURN e.name, t.name, sub.name LIMIT 20
Ground-Truth Spec groundtruth/patterns.md: Deterministic Ground-Truth Sampling Rules (seed = 42, 371 Items)
All patterns were verified against real source files in `twentyhq/twenty` (`12e8fae471865e9dab60c0b0fd59f318505c804e`) and `immich-app/immich` (`2153d22f34349de690cd98890ee4c635808da638`). Test files (`*.spec.ts`, `*.test.ts`, `__tests__`, `test/`, `e2e/`, `fixtures/`, `test_*.py`) are excluded across all categories. Stratified random sampling uses `random.Random(42)`.

- C1 (Incoming HTTP / GraphQL / FastAPI Endpoints): NestJS `@Controller` + `@(Get|Post|Put|Patch|Delete)`, NestJS GraphQL `@Resolver` + `@(Query|Mutation|Subscription)`, and Python FastAPI `@(app|router).(get|post|put|delete|patch)`.
- C2 (Outbound Inter-Service / External HTTP Calls): `fetch(...)`, `axios.(get|post|put|delete|patch|request)`, `this.httpService.(get|post|...)` (excluding IMAP `imapClient.fetch`).
- C3 & C8 (Message Queues & Async Jobs): `twenty` `enum MessageQueue` (27 queues), `@InjectMessageQueue` producers, `@Processor` + `@Process` consumers; `immich` `enum QueueName` (19 queues) & `enum JobName`, `jobRepository.queue` producers, `@OnJob` consumers.
- C4 & C10 (Database Tables & Schema): `twenty` TypeORM `@Entity` + `@Column`/`@ManyToOne`/`@OneToMany`; `immich` `@Table` + `@Column`/`@ForeignKeyColumn`.
- C5 (Code-to-Table Lineage): `twenty` `@InjectRepository` / `@InjectWorkspaceScopedRepository` read (`find*`, `count`, `createQueryBuilder`) & write (`save`, `insert`, `update`, `delete`, `softDelete`, `upsert`); `immich` Kysely `selectFrom` (read) & `insertInto`/`updateTable`/`deleteFrom` (write).
- C6 (Call Graph): Deterministic sample of 10 service methods across `*.service.ts`, tracing non-test callers invoking `this.<serviceField>.<methodName>(` or `this.<methodName>(`.
- C7 (Inheritance & Implementation): Top 5 base classes/interfaces with ≥3 non-test classes matching `export (abstract )?class <Child> (extends|implements) <Parent>`.