Case Study 04: CodeExplorer vs Graphify.
Head-to-head indexer and MCP server comparison on immich and twenty: cold indexing scalability, MCP startup latency, and graph completeness across 371 verified items.
Faster Cold Indexing
~18s vs ~108s on immich (3.2k files), and ~147s vs DNF (>45m timeout) on twenty (24.5k files).
SQLite vs NetworkX JSON
Basic Code & Call Graph
Across 130 basic items: ce scored 102/130 vs Graphify 23/130 (on immich alone: ce 73% vs Graphify 35%).
6 wins, 2 ties, 0 losses
Queues, DB Tables & SQL
ce matched 235/241 queue, ORM table, data-lineage, and sub-project items. Graphify does not index these.
235 / 241 verified items
Where Graphify Tied
On immich Call Graph & DI (C6), Graphify tied ce at 100% (20/20), resolving TS constructor DI accurately.
20/20 on immich C6
1. CodeExplorer (
ce): v1.75.0 (.NET 10, SQLite + OpenCypher, 36 MCP tools)2. Graphify (
graphifyy): v0.9.104 (Python 3.10, Tree-sitter + NetworkX --code-only, 10 MCP tools)3. Target Repositories:
twentyhq/twenty (commit 12e8fae4, 24,527 files, 2.49M LOC) & immich-app/immich (commit 2153d22f, 3,205 files, 461k LOC)1. Cold indexing & scalability across repository sizes.
On immich (3,205 files), both tools completed cleanly, with ce finishing ~6.1× faster. On twenty (24,527 files), Graphify’s Python AST & NetworkX pipeline stalled on a single CPU core in build_from_json and did not finish within the 45-minute timeout.
| Metric | Repo | CodeExplorer |
Graphify |
Outcome |
|---|---|---|---|---|
| Cold Index Wall Time | immich (3.2k files) |
17.60 s (55.1 MB SQLite DB) | 107.60 s (32.0 MB JSON / 52.3 MB dir) | ce 6.11× faster |
| Cold Index Wall Time | twenty (24.5k files) |
147.33 s (784.0 MB SQLite DB) | DNF (>2,740 s / >45.6 min timeout) | ce >18.6× faster (Graphify timed out) |
| Peak Working Set (RAM) | immich / twenty |
780.5 MB / 4,660.6 MB | 956.9 MB / 2,140.0 MB (stalled) | ce 1.23× lower RAM on immich |
| Parse Warnings / Errors | immich |
0 warnings | 9 syntax errors + 171 empty files | ce 0 parse failures |
--code-only vs LLM synthesis: Graphify also offers an optional LLM pass for extracting semantic / rationale nodes from Markdown docs, PDFs, and images. This benchmark evaluates deterministic static code extraction (--code-only) so that every result measures the indexer itself rather than an external LLM’s token budget or non-deterministic sampling.
2. Graph completeness: basic code vs full architecture.
Because Graphify timed out on twenty, we report both immich-only recall (66 basic items where both tools completed indexing) and combined recall across all 371 verified items.
| What Was Tested | Repo | Sample Size | CodeExplorer |
Graphify |
Why the Scores Differ |
|---|---|---|---|---|---|
Incoming HTTP & FastAPI Endpoints (C1) |
immich |
20 | 100% (20/20) | 0% (0/20 strict) | Graphify emits decorator references without extracting route URLs or HTTP verbs. |
Outbound HTTP Client Calls (C2) |
immich |
6 | 83% (5/6) | 0% (0/6) | Graphify does not extract outbound HTTP client calls. |
Function & DI Service Call Graph (C6) |
immich |
20 | 100% (20/20) | 100% (20/20) | Tie at 100%: Graphify’s TS extractor resolves constructor property types (this.svc.method()) on immich. |
Class Inheritance & Mixins (C7) |
immich |
20 | 15% (3/20) | 15% (3/20) | Tie at 15%: 17 of 20 DTOs use functional mixin inheritance (extends createZodDto(...)). |
All Basic Code Categories (C1, C2, C6, C7) |
twenty |
64 | 84% (54/64) | 0% (DNF) | Graphify timed out during cold indexing on twenty. |
Group A Total on immich Only (Both Tools Completed) |
immich |
66 | 73% (48/66) | 35% (23/66) | Direct comparison on the repository where both indexers completed. |
Group A Combined Total (immich + twenty) |
Both | 130 | 78% (102/130) | 18% (23/130) | 6 wins, 2 ties, 0 losses. |
| Group B Total: Queues, DB Tables & SQL Lineage | Both | 241 | 97.5% (235/241) | Not modeled (0/241) | Graphify parses raw .sql DDL only and does not model Kysely/TypeORM tables, BullMQ queues, or SQL lineage. |
3. MCP server latency & tool trade-offs on immich.
Because Graphify loads graph.json into an in-memory Python NetworkX graph on startup, cold MCP initialization takes ~3.7 seconds vs ~0.6 seconds for ce’s SQLite engine.
| MCP Operation | CodeExplorer |
Graphify |
Notes |
|---|---|---|---|
Cold MCP Init (initialize) |
599.0 ms (63.9 MB RSS) | 3,682.9 ms (162.8 MB RSS) | ce 6.15× faster startup with 2.55× lower memory. |
Architecture / Hub Overview (get_architecture_map / god_nodes) |
239.2 ms (4.5 KB, 21 sub-projects & DBs) | 21.8 ms (0.7 KB, top 20 high-degree nodes) | Graphify faster from in-memory NetworkX graph; ce maps sub-projects, DBs, ingress & egress. |
| Graph Schema / Stats (1st / Cached) | 23.3 ms (1.7 ms cached, 51.2 KB) | 76.1 ms (73.0 ms 2nd call, 0.2 KB) | ce 3.3× faster (43× cached); Graphify’s graph_stats spawns git rev-parse on every call. |
Symbol Lookup by Name / ID (AuthService) |
36.1 ms by name (0.4 KB table) | 3.7 ms (Ambiguous: 5 nodes) / 9.6 ms by ID |
ce resolves in 1 turn; Graphify is fast (9.6 ms) once given the exact slug ID. |
Multi-Token Search (authentication token validate) |
41.7 ms (3.0 KB, FTS5 search) | 333.8 ms (3.6 KB, depth-2 BFS) | ce 8.0× faster via SQLite FTS5 index. |
Inbound Callers by Name / ID (validate) |
37.0 ms by name / 19.0 ms by ID (1.5 KB, 8 callers) | 3.8 ms (Ambiguous: 6 nodes) / 2.4 ms by ID |
ce resolves all 8 callers in 1 turn; Graphify is fast (2.4 ms) once disambiguated to 1 node ID. |
| Unique Capabilities | OpenCypher queries (3.1 ms), DB & queue lineage, 32+ CI gates, VS Code extension | Leiden community clustering, Obsidian vault export, D3 HTML export, triage_prs |
Different primary design goals (architecture & CI governance vs knowledge-graph clustering & visualization). |
4. Reproducibility: tested queries, MCP payloads & ground-truth spec.
Click any drawer below to expand verbatim queriesQueries & Matchers scripts/eval_ce_vs_graphify.py: Exact OpenCypher Queries (ce) & NetworkX Graph Extraction Rules (Graphify)
# CodeExplorer (ce v1.75.0) OpenCypher Queries:
# C1 (Endpoints):
MATCH (e:Endpoint)-[:EXPOSED_BY]->(f:Function) RETURN f.file_path, f.start_line, f.name, e.http_method, e.route_template
# C2 (Outbound HTTP):
MATCH (ext:ExternalService)-[r:CALLED_BY]->(f:Function) WHERE COALESCE(ext.protocol, 'http') <> 'graphql' RETURN f.file_path, COALESCE(r.line, f.start_line), f.name, COALESCE(ext.url, ext.name)
# C3 & C8 (Queues / Topics):
MATCH (t:Topic)-[:PUBLISHED_BY]->(f:Function) RETURN f.file_path, f.start_line, f.name, t.name;
MATCH (t:Topic)-[:SUBSCRIBED_BY]->(m:Function) RETURN m.file_path, m.start_line, m.name, t.name
# C4 & C10 (ORM Tables & Schema):
MATCH (t:Table) OPTIONAL MATCH (t)-[:IMPLEMENTED_BY]->(ent:Type) RETURN COALESCE(ent.file_path, t.path), ent.start_line, COALESCE(ent.name, t.name), t.name
# C5 (Read/Write Lineage):
MATCH (q:Query)-[r:WRITES_TO|READS_FROM]->(t:Table) OPTIONAL MATCH (q)-[:ACCESSED_BY]->(f:Function) RETURN COALESCE(f.file_path, q.file_path), q.start_line, COALESCE(f.name, q.name), type(r), t.name
# C6 (Call Graph & DI Callers):
MATCH (caller:Function)-[:CALLS]->(callee:Function) WHERE callee.name IN [<target_methods>] OPTIONAL MATCH (enclosing:Function)-[:DECLARES]->(caller) RETURN caller.file_path, COALESCE(enclosing.start_line, caller.start_line), COALESCE(enclosing.name, caller.name), callee.name
# C7 (Inheritance & implements):
MATCH (c:Type)-[r:INHERITS_FROM|IMPLEMENTS]->(p) RETURN c.file_path, c.start_line, c.name, type(r), p.name
# ------------------------------------------------------------------------------
# Graphify (v0.9.104) graph.json Node & Link Extraction (eval_ce_vs_graphify.py):
# - Cold indexing command: `graphify index <repo_path> --code-only`
# - C6 (Call Graph): Links in `graph.json["links"]` where `relation == "calls"`, matching source function `(source_file, source_line, label)` and target method `.label.split(".")[-1]`.
# - C7 (Inheritance & implements): Links in `graph.json["links"]` where `relation in ("inherits", "implements")`, matching child class `(source_file, source_line, label)` and parent class/interface label.
MCP JSON-RPC Suite scripts/bench_mcp_graphify.py: Exact MCP Tool Invocations Benchmarked Over Persistent stdio
# 1. Graph Schema / Stats
# CodeExplorer (`ce mcp --root <repo>`):
{"name": "get_taxonomy", "arguments": {}}
# Graphify (`python -m graphify.serve graphify-out/graph.json`):
{"name": "graph_stats", "arguments": {}}
# 2. Multi-Token Semantic / BFS Search
# CodeExplorer:
{"name": "search_identifiers", "arguments": {"query": "authentication token validate", "limit": 15}}
# Graphify:
{"name": "query_graph", "arguments": {"question": "authentication token validate", "depth": 2}}
# 3. Direct Symbol Lookup
# CodeExplorer:
{"name": "find_symbol", "arguments": {"name": "AuthService"}}
# Graphify:
{"name": "get_node", "arguments": {"label": "AuthService"}}
# 4. Inbound Callers by Method Name
# CodeExplorer:
{"name": "get_callers", "arguments": {"target": "validate"}}
# Graphify:
{"name": "get_neighbors", "arguments": {"label": "validate", "relation_filter": "calls"}}
# 5. Call Chain / Shortest Path + OpenCypher Query
# CodeExplorer:
{"name": "get_call_chain", "arguments": {"startFunction": "login", "endFunction": "validate"}}
{"name": "execute_custom_read_cypher", "arguments": {"query": "MATCH (n:Function) RETURN n.name LIMIT 10"}}
# Graphify:
{"name": "shortest_path", "arguments": {"source": "<login_node_id>", "target": "<validate_node_id>", "undirected": true}}
Ground-Truth Spec groundtruth/patterns.md: Deterministic Ground-Truth Sampling Rules (seed = 42, 371 Items)
Identical 371 ground-truth items (`seed = 42`) used in Case Study 03 across `twentyhq/twenty` (`12e8fae471865e9dab60c0b0fd59f318505c804e`) and `immich-app/immich` (`2153d22f34349de690cd98890ee4c635808da638`): - Group A (130 Basic Code Items across L1 to L3): C1 Incoming Endpoints (40), C2 Outbound HTTP Calls (18), C6 Call Graph & DI Callers (32), C7 Class Inheritance & Interface Implementation (40). - Group B (241 Architectural Items across L4 to L5): C3 Queue Producers & Consumers (70), C4 ORM Tables (40), C5 Code-to-Table Read/Write Lineage (40), C8 Topic Resolution (37), C9 Services & Topology Edges (14), C10 Table Columns & Relations (28), C11 End-to-End Architectural Queries (12).