We recently upgraded one of our production ClickHouse database clusters from m8g.48xlarge instances to the newer m9g.48xlarge generation. To ensure a fair "apples-to-apples" comparison, we kept the core count, memory, and storage configuration identical.
While the new servers cost 9% more per hour, the upgrade was a clear win. The new hardware handles the same workload using roughly 25% less CPU power, and our data read speeds improved significantly. Below is a high-level summary of the results.
The screenshot below shows the time when we upgraded the cluster.
Why this was worth measuring properly
ClickHouse is the single largest cost on our cloud bill. Because our database is our biggest operational expense, even small efficiency gains here lead to massive savings at our scale - a +20% gain in CPU efficiency is often worth more than all other code changes we ship in a quarter combined.
When Graviton 5 instances became available in our region (eu-central-1), we saw an opportunity to optimize our upcoming capacity planning. While AWS marketing claims regarding the new m9g instances sounded promising, we needed to verify the real-world impact ourselves. Based on our positive history with past Graviton upgrades, we wanted a clear answer to two questions:
- On our specific query mix, how much CPU power does this new hardware actually save?
- Does the 9% higher hourly cost pay for itself through those performance gains?
What changed, and what stayed the same
To make sure our results were accurate, we performed a "same-size swap." This means we kept the core count, memory, storage configuration, ClickHouse version, and data structure exactly the same. We also left the "Keeper ensemble" - the underlying management layer that coordinates the cluster - completely untouched.
By keeping these variables identical, we ensured that any performance differences were strictly due to the new hardware, rather than external configuration changes. The table below details the specific hardware differences between the old and new instances:
| m8g.48xlarge (Graviton 4) | m9g.48xlarge (Graviton 5) | Delta | |
|---|---|---|---|
| vCPU | 192 | 192 | same |
| Memory | 768 GiB | 768 GiB | same |
| Clock Speed | 2.7 GHz | 3.3 GHz | +22% |
| Network | 50 Gbps | 100 Gbps | +100% |
| EBS Throughput | 40,000 Mbps | 72,000 Mbps | +80% |
| EBS IOPS | 240,000 | 288,000 | +20% |
How we measured it
Testing this on a live production cluster is tricky. A standard "A/B test" (where we run half the cluster on new hardware and half on old) wouldn't work: if one half is slower, it slows down every distributed query, which would corrupt the results for both sides.
Instead, we used a "before-and-after" comparison. Because workloads can change daily, we took two steps to ensure our comparison remained fair:
- Comparable Days: We selected two ordinary weekdays - Tuesday, September 1st (on the old hardware) and Friday, September 4th (on the new hardware) - to ensure the traffic patterns were as consistent as possible.
- Standardized Query Weighting: Rather than comparing raw daily totals, we calculated the CPU efficiency for every individual query type. We then applied those efficiency gains to the traffic mix of our baseline day.
Think of this as creating a standardized "basket" of queries. We calculated exactly how much CPU power those specific queries required on the old hardware versus the new. This ensures that even if our query volume shifted slightly between days, we are comparing the efficiency of the same work, rather than just daily totals.
To keep our data accurate, we ignored very infrequent queries (those processing less than 1 GiB of data per day). For these low-volume queries, efficiency metrics are often skewed by startup costs rather than true performance.
The two days were close enough that our reweighting process barely had to adjust the data:
| Sep 1 (m8g) | Sep 4 (m9g) | Delta | |
|---|---|---|---|
| Rows ingested | 197.72 B | 188.27 B | -4.8% |
| Bytes ingested | 19.15 TiB | 17.67 TiB | -7.7% |
| New parts | 58.96 M | 57.68 M | -2.2% |
| Queries executed (one node) | 25.67 M | 26.10 M | +1.7% |
| Bytes read + written (one node) | 348.90 TiB | 352.82 TiB | +1.1% |
| Initial SELECTs (one node) | 819,016 | 957,019 | +16.9% |
CPU efficiency: A deeper look
Cluster-level averages can sometimes be misleading if traffic patterns shift between days. To avoid this, we analyzed the efficiency of individual query types. By comparing each specific query shape against itself on both hardware generations - and then reweighting the results to match our baseline traffic mix - we ensured an accurate, "apples-to-apples" comparison.
| Query kind | Shapes | m8g CPU-ms/MB | m9g CPU-ms/MB | Efficiency Gain |
|---|---|---|---|---|
| Select | 2,035 | 5.967 | 4.419 | 21% |
| Insert | 18 | 4.304 | 3.053 | 28% |
| AsyncInsertFlush | 1 | 6.594 | 4.405 | 33% |
The data confirms that the new hardware is significantly more efficient:
- Reads: Require 21% less CPU power per byte processed.
- Writes: Require 28% less CPU power per byte written.
- Background tasks (AsyncInsertFlush): Cost 33% less CPU power.
The write path saw the largest gains, which aligns with Graviton 5’s architectural strengths. It excels at tasks like data compression and checksum calculations - tight, CPU-intensive operations that stay within the processor's cache. These are exactly the types of tasks that dominate the "insert flush" process, explaining why we saw such a notable performance jump there.
Per query shape: Where the biggest gains happened
Aggregate data can sometimes hide the truth, making it look like everything improved equally when reality might be different. To verify, we looked at the ten most resource-heavy query types (by CPU usage on the old hardware) and compared them directly against their performance on the new m9g instances.
| Query shape | Calls m8g | Calls m9g | p50 m8g | p50 m9g | p95 m8g | p95 m9g | CPU-ms/MB m8g | CPU-ms/MB m9g |
|---|---|---|---|---|---|---|---|---|
| Metric metadata lookup | 161,546 | 190,350 | 61 ms | 29 ms | 262 ms | 73 ms | 3.41 | 2.29 |
| Metric metadata, variant | 65,430 | 82,930 | 66 ms | 33 ms | 591 ms | 113 ms | 4.21 | 2.92 |
| Filter bar, metadata + resources | 8,555 | 14,555 | 170 ms | 94 ms | 714 ms | 294 ms | 1.47 | 1.08 |
| Exponential histogram read | 103,334 | 125,714 | 36 ms | 20 ms | 63 ms | 31 ms | 9.54 | 5.17 |
| Metric metadata, variant | 41,497 | 42,521 | 70 ms | 36 ms | 662 ms | 117 ms | 5.01 | 3.54 |
| Gauge read with temp tables | 47,035 | 46,969 | 62 ms | 34 ms | 132 ms | 75 ms | 7.17 | 4.25 |
| Filter bar, metadata + resources | 15,625 | 16,802 | 171 ms | 95 ms | 639 ms | 231 ms | 2.41 | 1.73 |
| Log search with resources | 1,234 | 1,461 | 204 ms | 66 ms | 760 ms | 115 ms | 0.46 | 0.42 |
| Filter bar, heavy variant | 1,729 | 2,093 | 420 ms | 195 ms | 1,958 ms | 725 ms | 1.14 | 0.93 |
| Histogram read with temp tables | 28,002 | 32,166 | 70 ms | 38 ms | 119 ms | 58 ms | 7.57 | 4.01 |
All ten query types improved across the board: every one of them became faster and more CPU-efficient, even while handling higher query volumes than before.
The most dramatic improvements were in "tail latency" (the p95 column, representing the slowest 5% of queries). On our busiest query shape, the time for these slow queries dropped from 262 ms to 73 ms, and for log searches, it dropped from 760 ms to 115 ms. This matches our expectations: because the new hardware offers higher per-core performance and faster storage access, the queries that were previously held back by disk or processing bottlenecks saw the biggest performance leaps.
Overall read latency
| Metric | Sep 1 (m8g) | Sep 4 (m9g) | Improvement |
|---|---|---|---|
| Total SELECT queries | 819,016 | 957,019 | +16.9% |
| p50 Latency (Median) | 60 ms | 31 ms | 48% faster |
| p95 Latency (Tail) | 402 ms | 157 ms | 61% faster |
| p99 Latency (Slowest) | 1,130 ms | 415 ms | 63% faster |
| p99.9 Latency | 3,487 ms | 1,418 ms | 59% faster |
While individual query analysis is the most accurate way to measure performance, looking at overall read latency confirms the trend: the new hardware handles significantly more traffic while delivering much faster responses. Even though we processed 17% more queries, every response time metric improved by nearly 50% or more. This confirms that the efficiency gains we saw in individual query shapes translate directly into a much faster, more responsive experience for our users.
Background work: merges
"Merges" in ClickHouse are automatic, background processes that combine smaller data parts into larger, more efficient ones. While users don't see these tasks, they are crucial for keeping the database performant, and they rely heavily on CPU and I/O power. We were surprised by how much more efficient these background tasks became on the new hardware:
| Merge Metric | Sep 1 (m8g) | Sep 4 (m9g) | Improvement |
|---|---|---|---|
| Merges executed | 1,127,869 | 1,037,903 | -8.0% |
| Data merged | 8.17 TiB | 8.50 TiB | +4.0% |
| Throughput per second | 21.7 MiB/s | 30.6 MiB/s | +41% |
| p50 duration | 82 ms | 81 ms | -1% |
| p95 duration | 1,013 ms | 721 ms | -29% |
The key takeaway is that the new hardware handles fewer, larger merge operations while processing more total data. We achieved 41% higher throughput per second. Similar to our query results, the biggest gains were seen in the "tail" performance (p95), where the slowest merges became 29% faster.
What the 9% costs, and what it buys
Graviton 5 is not free. In the eu-central-1 region, the m9g.48xlarge instance costs exactly 9% more than the older m8g.48xlarge.
| Cost comparison (eu-central-1) | m8g.48xlarge | m9g.48xlarge | Delta |
|---|---|---|---|
| Price per node/hour | $10.32 | $11.25 | +9% |
While the hourly rate is higher, the "cost per unit of work" tells a different story. When we factor in the efficiency gains - the fact that the new hardware handles significantly more data for that extra 9% - the actual cost to run our database drops:
| Effective cost savings | Efficiency Gain | Price Premium | Net Savings |
|---|---|---|---|
| SELECTs (per byte) | 21% | 9% | 13.9% cheaper |
| INSERTs (per byte) | 28% | 9% | 21.3% cheaper |
| Background Tasks | 33% | 9% | 27.2% cheaper |
| Cluster CPU (Avg) | 22.7% | 9% | 15.8% cheaper |
Even though we pay more per hour, we are getting far more "work" done per dollar. By almost every metric, the new hardware is significantly more cost-effective.
Availability: The limiting factor
The main reason we haven't rolled this upgrade out everywhere yet is supply. As of early September 2026, the Graviton 5 series (m9g, m9gd, c9g, and c9gd) is generally available in eu-central-1, us-west-2, and us-east-2. However, other regions - such as eu-west-1 - do not yet offer these instances.
Additionally, memory-optimized node types (the r9g series) do not exist yet, so our clusters that rely on those will remain on the r8g generation for the time being.
We are very happy with these performance gains and plan to expand this upgrade to our remaining AWS regions as soon as the hardware becomes available.
