ClickBench Disqualifies pgrust Engine Over Embedded Query Kernels and Result Caching
ClickHouse maintainer Alexey Milovidov merged Pull Request #2165 removing pgrust from the ClickBench benchmark after binary and source analysis revealed dedicated query kernels and non-disableable hot-run result caching. The analysis demonstrated that pgrust functioned as a static harness for ClickBench's 43 queries rather than a general-purpose query engine.
Embedded Physical Plans and Hardcoded Query Kernels Inspection of the `pgrust` v0.4-preview binary and public v0.3 source code confirmed that the engine bypassed general query planning by embedding static physical plans tailored exclusively to ClickBench. The compiled `aarch64` release binary contains direct source references ranging from `crates/gravity/backend/irprobe/src/q00.rs` through `q36.rs`, along with `tpch_q01.rs`. These files act as hardcoded IR probe kernels for specific benchmark queries.
In the v0.3 source, the benchmark SQL file `queries-clickbench.sql` is compiled into the executor crate via `include_str!`, referencing hand-lowered physical plans stored in `clickbench.ron`. Instead of evaluating queries through a general execution path, plan matchers inspect incoming SQL against fixed ClickBench patterns, emitting explicit error messages named after benchmark queries when structural deviations occur.
- Queries Q0, Q1, Q2, Q3, Q6, Q19, and Q29 execute as `MetadataAnswer` phases, bypassing table scans entirely.
- Query Q19 reconstructs result lines directly from predicate constants.
- Query Q29 uses an algebraic rewrite (`SUM(x+k) = SUM(x) + k*COUNT`) resolved via metadata statistics.
- Query Q18 requires a persisted derived column with baked bit widths of `26/6/20`.
- Query Q28 uses a single-entry lookup table (`HOST_EXTRACT_PATTERN`) to evaluate regular expressions.
Hot-Run Verdict Replay and Un-Disableable Caching ClickBench rules mandate that query result caches and near-end pipeline caches must be disabled, and entries must be marked as untuned (`tuned: no`). `pgrust` violated these rules by caching filter verdicts and replaying previous run answers during warm execution phases.
Plan notes for Q20, Q21, and Q23 state that warm runs perform a pure fold over cached verdicts with zero columns scanned. On a 96-core Graviton (Neoverse-V2) test machine, `EXPLAIN` output on the v0.4-preview binary reported `SQE: engine (family: windowreplay)` for Q20, while Q1 executed under `family: metaanswer` with the underlying scan marked as never executed.
- Query Q20 (`URL LIKE '%google%'`) required 10.8 seconds on a cold start, dropping to 11 ms on the second run and 5 ms on the third run.
- Altering the predicate constant (`'%rambler%'`) yielded a 283 ms cold start, followed by 12 ms and 4 ms warm runs.
- Disabling the configuration flags `pgrust.gravity_entry = off` and `pgrust.condition_cache = off` failed to disable the cache, with repeated `LIKE` operations continuing to return in 3 to 4 ms.
Absence of General Columnar Execution The engine lacks a general execution path for columnar tables backed by `pgrcolumnar2`, which does not support write operations (`INSERT`, `UPDATE`, or `DELETE`). When executing SQL queries that fall outside the 43 predefined ClickBench shapes, `pgrust` refuses execution rather than falling back to a slower general scan.
During validation, the binary rejected 20 out of 67 near-benchmark probes and 3 out of 49 perturbed query variants. Unserved shapes fail with `ERROR: sqe: statement over columnar relation "hits" is not served`. Rejections included basic SQL operations such as `SELECT MIN(URL) FROM hits`, `SELECT AVG(UserID + 1) FROM hits`, subqueries, and non-standard `HAVING` expressions. While the 43 benchmark queries returned mathematically correct output compared to `clickhouse local`, the implementation measured a specialized benchmark lookup table rather than a usable database system.