ARCHITECTURE

Performance Architecture: Roofline and Memory Budget

Turn workload counts and data movement into a bounded performance target before choosing pipelines or memory structures. This is generic guidance, not a description of any commercial chip’s undisclosed implementation.

Inputs

Representative workloads and service-level targets
Operation mix, tensor/packet shapes, and reuse opportunities
Candidate compute rate and bandwidth/latency budgets per memory level

Method

Count useful operations and bytes at each boundary; define arithmetic intensity as ops/bytes for each workload phase.
Set an optimistic ceiling min(compute peak, intensity × sustained bandwidth), then record assumptions separately from measured values.
Trace data lifetime from external memory through shared buffers to registers; budget capacity, ports, and refill time for every reuse claim.
For bandwidth-bound phases, improve locality, layout, batching, or compression before adding arithmetic units; reassess the byte count.
For compute-bound phases, inspect utilization losses from dependencies, imbalance, precision, and control overhead; prototype the highest-risk assumption.

Deliverables

Workload-to-roofline table with stated ceilings
Memory traffic and reuse ledger by hierarchy level
Architecture decision log with sensitivity ranges

Pitfalls

Treating peak bandwidth as sustained bandwidth or counting cache hits as free.
Optimizing a kernel-level ceiling while ignoring queueing, transfers, or end-to-end latency.

Sources

Accelerating HPC Applications with NVIDIA Nsight Compute Roofline Analysis ↗
NVIDIA Developer · Introducing hierarchical roofline analysis