Benchmarks Catalog / Case Study 04: CodeExplorer vs Graphify
twenty (2.49M LOC) & immich (461k LOC)

Case Study 04: CodeExplorer vs Graphify.

Head-to-head indexer and MCP server comparison on immich and twenty: cold indexing scalability, MCP startup latency, and graph completeness across 371 verified items.

~6.1× / >18×

Faster Cold Indexing

~18s vs ~108s on immich (3.2k files), and ~147s vs DNF (>45m timeout) on twenty (24.5k files).

SQLite vs NetworkX JSON
78% vs 18%

Basic Code & Call Graph

Across 130 basic items: ce scored 102/130 vs Graphify 23/130 (on immich alone: ce 73% vs Graphify 35%).

6 wins, 2 ties, 0 losses
~98%

Queues, DB Tables & SQL

ce matched 235/241 queue, ORM table, data-lineage, and sub-project items. Graphify does not index these.

235 / 241 verified items
100% Tie

Where Graphify Tied

On immich Call Graph & DI (C6), Graphify tied ce at 100% (20/20), resolving TS constructor DI accurately.

20/20 on immich C6
Tested participants:
1. CodeExplorer (ce): v1.75.0 (.NET 10, SQLite + OpenCypher, 36 MCP tools)
2. Graphify (graphifyy): v0.9.104 (Python 3.10, Tree-sitter + NetworkX --code-only, 10 MCP tools)
3. Target Repositories: twentyhq/twenty (commit 12e8fae4, 24,527 files, 2.49M LOC) & immich-app/immich (commit 2153d22f, 3,205 files, 461k LOC)

1. Cold indexing & scalability across repository sizes.

On immich (3,205 files), both tools completed cleanly, with ce finishing ~6.1× faster. On twenty (24,527 files), Graphify’s Python AST & NetworkX pipeline stalled on a single CPU core in build_from_json and did not finish within the 45-minute timeout.

Cold Indexing Performance on immich (3,205 files) & twenty (24,527 files) Deterministic --code-only mode
Metric Repo CodeExplorer Graphify Outcome
Cold Index Wall Time immich (3.2k files) 17.60 s (55.1 MB SQLite DB) 107.60 s (32.0 MB JSON / 52.3 MB dir) ce 6.11× faster
Cold Index Wall Time twenty (24.5k files) 147.33 s (784.0 MB SQLite DB) DNF (>2,740 s / >45.6 min timeout) ce >18.6× faster (Graphify timed out)
Peak Working Set (RAM) immich / twenty 780.5 MB / 4,660.6 MB 956.9 MB / 2,140.0 MB (stalled) ce 1.23× lower RAM on immich
Parse Warnings / Errors immich 0 warnings 9 syntax errors + 171 empty files ce 0 parse failures
Methodological note on --code-only vs LLM synthesis: Graphify also offers an optional LLM pass for extracting semantic / rationale nodes from Markdown docs, PDFs, and images. This benchmark evaluates deterministic static code extraction (--code-only) so that every result measures the indexer itself rather than an external LLM’s token budget or non-deterministic sampling.

2. Graph completeness: basic code vs full architecture.

Because Graphify timed out on twenty, we report both immich-only recall (66 basic items where both tools completed indexing) and combined recall across all 371 verified items.

Group A: Basic Code (130 Items) & Group B: Queues, DB Tables & SQL (241 Items) ce: 6 Wins, 2 Ties, 0 Losses vs Graphify
What Was Tested Repo Sample Size CodeExplorer Graphify Why the Scores Differ
Incoming HTTP & FastAPI Endpoints (C1) immich 20 100% (20/20) 0% (0/20 strict) Graphify emits decorator references without extracting route URLs or HTTP verbs.
Outbound HTTP Client Calls (C2) immich 6 83% (5/6) 0% (0/6) Graphify does not extract outbound HTTP client calls.
Function & DI Service Call Graph (C6) immich 20 100% (20/20) 100% (20/20) Tie at 100%: Graphify’s TS extractor resolves constructor property types (this.svc.method()) on immich.
Class Inheritance & Mixins (C7) immich 20 15% (3/20) 15% (3/20) Tie at 15%: 17 of 20 DTOs use functional mixin inheritance (extends createZodDto(...)).
All Basic Code Categories (C1, C2, C6, C7) twenty 64 84% (54/64) 0% (DNF) Graphify timed out during cold indexing on twenty.
Group A Total on immich Only (Both Tools Completed) immich 66 73% (48/66) 35% (23/66) Direct comparison on the repository where both indexers completed.
Group A Combined Total (immich + twenty) Both 130 78% (102/130) 18% (23/130) 6 wins, 2 ties, 0 losses.
Group B Total: Queues, DB Tables & SQL Lineage Both 241 97.5% (235/241) Not modeled (0/241) Graphify parses raw .sql DDL only and does not model Kysely/TypeORM tables, BullMQ queues, or SQL lineage.

3. MCP server latency & tool trade-offs on immich.

Because Graphify loads graph.json into an in-memory Python NetworkX graph on startup, cold MCP initialization takes ~3.7 seconds vs ~0.6 seconds for ce’s SQLite engine.

MCP Server Startup & Query Latency on immich ce (36 tools) vs Graphify (10 tools + 6 resources)
MCP Operation CodeExplorer Graphify Notes
Cold MCP Init (initialize) 599.0 ms (63.9 MB RSS) 3,682.9 ms (162.8 MB RSS) ce 6.15× faster startup with 2.55× lower memory.
Architecture / Hub Overview (get_architecture_map / god_nodes) 239.2 ms (4.5 KB, 21 sub-projects & DBs) 21.8 ms (0.7 KB, top 20 high-degree nodes) Graphify faster from in-memory NetworkX graph; ce maps sub-projects, DBs, ingress & egress.
Graph Schema / Stats (1st / Cached) 23.3 ms (1.7 ms cached, 51.2 KB) 76.1 ms (73.0 ms 2nd call, 0.2 KB) ce 3.3× faster (43× cached); Graphify’s graph_stats spawns git rev-parse on every call.
Symbol Lookup by Name / ID (AuthService) 36.1 ms by name (0.4 KB table) 3.7 ms (Ambiguous: 5 nodes) / 9.6 ms by ID ce resolves in 1 turn; Graphify is fast (9.6 ms) once given the exact slug ID.
Multi-Token Search (authentication token validate) 41.7 ms (3.0 KB, FTS5 search) 333.8 ms (3.6 KB, depth-2 BFS) ce 8.0× faster via SQLite FTS5 index.
Inbound Callers by Name / ID (validate) 37.0 ms by name / 19.0 ms by ID (1.5 KB, 8 callers) 3.8 ms (Ambiguous: 6 nodes) / 2.4 ms by ID ce resolves all 8 callers in 1 turn; Graphify is fast (2.4 ms) once disambiguated to 1 node ID.
Unique Capabilities OpenCypher queries (3.1 ms), DB & queue lineage, 32+ CI gates, VS Code extension Leiden community clustering, Obsidian vault export, D3 HTML export, triage_prs Different primary design goals (architecture & CI governance vs knowledge-graph clustering & visualization).

4. Reproducibility: tested queries, MCP payloads & ground-truth spec.

Click any drawer below to expand verbatim queries
Queries & Matchers scripts/eval_ce_vs_graphify.py: Exact OpenCypher Queries (ce) & NetworkX Graph Extraction Rules (Graphify)
# CodeExplorer (ce v1.75.0) OpenCypher Queries:
# C1 (Endpoints):
MATCH (e:Endpoint)-[:EXPOSED_BY]->(f:Function) RETURN f.file_path, f.start_line, f.name, e.http_method, e.route_template
# C2 (Outbound HTTP):
MATCH (ext:ExternalService)-[r:CALLED_BY]->(f:Function) WHERE COALESCE(ext.protocol, 'http') <> 'graphql' RETURN f.file_path, COALESCE(r.line, f.start_line), f.name, COALESCE(ext.url, ext.name)
# C3 & C8 (Queues / Topics):
MATCH (t:Topic)-[:PUBLISHED_BY]->(f:Function) RETURN f.file_path, f.start_line, f.name, t.name;
MATCH (t:Topic)-[:SUBSCRIBED_BY]->(m:Function) RETURN m.file_path, m.start_line, m.name, t.name
# C4 & C10 (ORM Tables & Schema):
MATCH (t:Table) OPTIONAL MATCH (t)-[:IMPLEMENTED_BY]->(ent:Type) RETURN COALESCE(ent.file_path, t.path), ent.start_line, COALESCE(ent.name, t.name), t.name
# C5 (Read/Write Lineage):
MATCH (q:Query)-[r:WRITES_TO|READS_FROM]->(t:Table) OPTIONAL MATCH (q)-[:ACCESSED_BY]->(f:Function) RETURN COALESCE(f.file_path, q.file_path), q.start_line, COALESCE(f.name, q.name), type(r), t.name
# C6 (Call Graph & DI Callers):
MATCH (caller:Function)-[:CALLS]->(callee:Function) WHERE callee.name IN [<target_methods>] OPTIONAL MATCH (enclosing:Function)-[:DECLARES]->(caller) RETURN caller.file_path, COALESCE(enclosing.start_line, caller.start_line), COALESCE(enclosing.name, caller.name), callee.name
# C7 (Inheritance & implements):
MATCH (c:Type)-[r:INHERITS_FROM|IMPLEMENTS]->(p) RETURN c.file_path, c.start_line, c.name, type(r), p.name

# ------------------------------------------------------------------------------
# Graphify (v0.9.104) graph.json Node & Link Extraction (eval_ce_vs_graphify.py):
# - Cold indexing command: `graphify index <repo_path> --code-only`
# - C6 (Call Graph): Links in `graph.json["links"]` where `relation == "calls"`, matching source function `(source_file, source_line, label)` and target method `.label.split(".")[-1]`.
# - C7 (Inheritance & implements): Links in `graph.json["links"]` where `relation in ("inherits", "implements")`, matching child class `(source_file, source_line, label)` and parent class/interface label.
MCP JSON-RPC Suite scripts/bench_mcp_graphify.py: Exact MCP Tool Invocations Benchmarked Over Persistent stdio
# 1. Graph Schema / Stats
# CodeExplorer (`ce mcp --root <repo>`):
{"name": "get_taxonomy", "arguments": {}}
# Graphify (`python -m graphify.serve graphify-out/graph.json`):
{"name": "graph_stats", "arguments": {}}

# 2. Multi-Token Semantic / BFS Search
# CodeExplorer:
{"name": "search_identifiers", "arguments": {"query": "authentication token validate", "limit": 15}}
# Graphify:
{"name": "query_graph", "arguments": {"question": "authentication token validate", "depth": 2}}

# 3. Direct Symbol Lookup
# CodeExplorer:
{"name": "find_symbol", "arguments": {"name": "AuthService"}}
# Graphify:
{"name": "get_node", "arguments": {"label": "AuthService"}}

# 4. Inbound Callers by Method Name
# CodeExplorer:
{"name": "get_callers", "arguments": {"target": "validate"}}
# Graphify:
{"name": "get_neighbors", "arguments": {"label": "validate", "relation_filter": "calls"}}

# 5. Call Chain / Shortest Path + OpenCypher Query
# CodeExplorer:
{"name": "get_call_chain", "arguments": {"startFunction": "login", "endFunction": "validate"}}
{"name": "execute_custom_read_cypher", "arguments": {"query": "MATCH (n:Function) RETURN n.name LIMIT 10"}}
# Graphify:
{"name": "shortest_path", "arguments": {"source": "<login_node_id>", "target": "<validate_node_id>", "undirected": true}}
Ground-Truth Spec groundtruth/patterns.md: Deterministic Ground-Truth Sampling Rules (seed = 42, 371 Items)
Identical 371 ground-truth items (`seed = 42`) used in Case Study 03 across `twentyhq/twenty` (`12e8fae471865e9dab60c0b0fd59f318505c804e`) and `immich-app/immich` (`2153d22f34349de690cd98890ee4c635808da638`):
- Group A (130 Basic Code Items across L1 to L3): C1 Incoming Endpoints (40), C2 Outbound HTTP Calls (18), C6 Call Graph & DI Callers (32), C7 Class Inheritance & Interface Implementation (40).
- Group B (241 Architectural Items across L4 to L5): C3 Queue Producers & Consumers (70), C4 ORM Tables (40), C5 Code-to-Table Read/Write Lineage (40), C8 Topic Resolution (37), C9 Services & Topology Edges (14), C10 Table Columns & Relations (28), C11 End-to-End Architectural Queries (12).