Why reference is faster than pointer, How ISA improve this

Background

2 years ago, I submitted a Verilator PR #5254 to replace existing huge struct pointer dereference into reference, and measured performance +11% with LLVM, and measured -9% sampled branch instruction at runtime.

I have many insights to share regarding why using references can lead to better performance compared to pointer dereferences in certain scenarios, and how new ISA extensions like x86-APX can further optimize such access patterns.

Let's dig into the reasons behind this performance difference and explore how references can be more efficient than pointers in certain contexts.

How references make difference in LLVM IR?

A reference and a pointer are the same thing in LLVM IR: an opaque ptr. To see where they differ, compile two functions with an identical body, one taking S* and one taking S&, and diff the IR of the two functions.

// ref.cpp
struct S { int a, b, c, d; };

void by_ptr(S* s) { s->b = s->a ? (s->c & 0xff) : (s->d >> 3); }
void by_ref(S& s) { s.b  = s.a  ? (s.c  & 0xff) : (s.d  >> 3); }

Then we try compile using latest LLVM master branch commit 8e84c4a901a7fe6ba80b3c305b200c3e9bbc4606.

clang++ -fno-discard-value-names -emit-llvm -S ref.cpp -o ref.ll
diff <(sed -n '/^define.*by_ptr/,/^}/p' ref.ll) <(sed -n '/^define.*by_ref/,/^}/p' ref.ll)

1c1
< define dso_local void @_Z6by_ptrP1S(ptr noundef %s) #0 {
---
> define dso_local void @_Z6by_refR1S(ptr noundef nonnull align 4 dereferenceable(16) %s) #0 {
5c5
<   %0 = load ptr, ptr %s.addr, align 8
---
>   %0 = load ptr, ptr %s.addr, align 8, !nonnull !5, !align !6
12c12
<   %2 = load ptr, ptr %s.addr, align 8
---
>   %2 = load ptr, ptr %s.addr, align 8, !nonnull !5, !align !6
19c19
<   %4 = load ptr, ptr %s.addr, align 8
---
>   %4 = load ptr, ptr %s.addr, align 8, !nonnull !5, !align !6
27c27
<   %6 = load ptr, ptr %s.addr, align 8
---
>   %6 = load ptr, ptr %s.addr, align 8, !nonnull !5, !align !6

A reference in C++ carries additional guarantees compared to a pointer, which the compiler encodes in the LLVM IR as the nonnull, align, and dereferenceable attributes.

• nonnull: a reference is always bound to an object.
• align 4: that object is a real S, so the address has the alignment of an S.
• dereferenceable(16): the 16 bytes starting at %s, i.e. the whole S, can be loaded without trapping.

C++ hands the compiler these facts for free. A reference must refer to a valid object while it is used; a pointer may be null, one past the end of an array, or anything else.

Now run the optimizer on the same file and diff the two functions again.

sed 's/ optnone//' ref.ll | opt -O3 -S -o ref.opt.ll
diff -u <(sed -n '/^define.*by_ptr/,/^}/p' ref.opt.ll) <(sed -n '/^define.*by_ref/,/^}/p' ref.opt.ll)

@@ -1,23 +1,14 @@
-define dso_local void @_Z6by_ptrP1S(ptr nofree noundef captures(none) initializes((4, 8)) %s) local_unnamed_addr #0 {
+define dso_local void @_Z6by_refR1S(ptr nofree noundef nonnull align 4 captures(none) dereferenceable(16) initializes((4, 8)) %s) local_unnamed_addr #0 {
 entry:
   %0 = load i32, ptr %s, align 4
   %tobool.not = icmp eq i32 %0, 0
-  br i1 %tobool.not, label %cond.false, label %cond.true
-
-cond.true:                                        ; preds = %entry
   %c = getelementptr inbounds nuw i8, ptr %s, i64 8
   %1 = load i32, ptr %c, align 4
   %and = and i32 %1, 255
-  br label %cond.end
-
-cond.false:                                       ; preds = %entry
   %d = getelementptr inbounds nuw i8, ptr %s, i64 12
   %2 = load i32, ptr %d, align 4
   %shr = ashr i32 %2, 3
-  br label %cond.end
-
-cond.end:                                         ; preds = %cond.false, %cond.true
-  %cond = phi i32 [ %and, %cond.true ], [ %shr, %cond.false ]
+  %cond = select i1 %tobool.not, i32 %shr, i32 %and
   %b = getelementptr inbounds nuw i8, ptr %s, i64 4
   store i32 %cond, ptr %b, align 4
   ret void

As we can see, with dereferenceable attributes, the compiler has additional knowledge about the memory pointed to by the reference. Thus, there is no worry about executing loads that would otherwise be unsafe if the pointer could be null or point to invalid memory to fault (SIGSEGV), then the compiler can safely hoist those loads above branches and optimize the code more aggressively.

Sometimes, branch can be better if branch is easy to predict, as it allows the CPU to speculatively execute instructions that reduce the unused load instructions and improve overall performance.

However, for Verilator with large RTL design, the branch density is very high, and the loop latency of the two same branch is very very far apart. For example, when Verilator evaluates a large RTL design like XiangShan, there will be about 100M instruction need to execute for each simulation cycle, and ~10% of them are branch instructions. Even worse, when BTB (Branch Target Buffer) inside CPU cannot hold such a large number of branches, all the branch will just like predicted to not taken. In XiangShan on Verilator, there will be about 50% of branches are likely to taken, leading to frequent mispredictions and significant performance penalties.

Thus for such scenarios, minimizing branches and relying on straight-line code with select instructions, as seen in the optimized LLVM IR for references, can lead to better overall performance by reducing the likelihood of branch mispredictions.

For these reasons, using references with appropriate attributes in C++ can help the compiler generate more efficient LLVM IR, ultimately leading to better runtime performance in scenarios with high branch density.

Is dereferenceable attribute actually safe?

dereferenceable(16) is a promise that the 16 bytes behind s can be loaded at any time without trapping.

However, C++ promises something weaker: a reference is bound to a valid object when the function is entered. Nothing in the language stops a callee from destroying that object. delete &s is legal if s was allocated with new, and a push_back on a std::vector invalidates every reference to its elements.

So, is the compiler allowed to speculate a load through a reference after a call? Here is a program that is well-defined C++ and tests exactly that:

// f.cpp
struct S { int a, b, c, d; };
void g();
int f(S& s, bool use) {
    g();                              // for all the compiler knows, g() may free the object behind s
    return use ? (s.a & 0xff) : 0;    // s is only touched when use is true
}

// main.cpp
#include <sys/mman.h>
#include <cstdio>
struct S { int a, b, c, d; };
int f(S& s, bool use);
static S* obj;
void g() { munmap(obj, 4096); }       // the object's page is gone after this
int main() {
    obj = static_cast<S*>(mmap(nullptr, 4096, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0));
    int r = f(*obj, /*use=*/false);   // well-defined: s is never read after g()
    printf("f returned %d\n", r);
    return 0;
}

f is called with use == false, so after g() unmaps the page the program never reads s again. The C++ abstract machine is happy with this. Now compile f.cpp twice with the same trunk clang: once as-is, and once with the hidden flag that restores the way LLVM up to 22 interpreted the attribute.

clang++ -O3 -c f.cpp -o f.o
clang++ -O3 -mllvm -use-dereferenceable-at-point-semantics=0 -c f.cpp -o f_old.o
clang++ main.cpp f.o     -o demo     && ./demo;     echo "exit $?"
clang++ main.cpp f_old.o -o demo_old && ./demo_old; echo "exit $?"

f returned 0
exit 0
exit 139

Exit code 139 is SIGSEGV. The old interpretation hoisted the load above the use test, and the load read the unmapped page:

; f_old.o: how LLVM <= 22 compiled f
        call    _Z1gv@PLT
        movzx   eax, byte ptr [r14]       ; speculative load, runs even when use == false
        test    ebx, ebx
        cmove   eax, ebx

; f.o: LLVM 23.1 and later
        call    _Z1gv@PLT
        xor     eax, eax
        test    ebp, ebp
        je      .LBB0_2
        movzx   eax, byte ptr [rbx]       ; only when use == true
.LBB0_2:

So the honest answer is: the attribute was not safe the way LLVM used it for years, and LLVM fixed that in July 2026 with PR #204795 "Enable dereferenceable-at-point semantics", which shipped in LLVM 23.1. The LangRef now says:

"The dereferenceable attribute only implies dereferenceability at the point of the attribute (i.e. on function entry for arguments or at the point of the call for return values). The underlying object may still get freed after that point. Other attributes such as nofree can be used to exclude frees."

The motivation given in the PR is exactly the C++ case: "C++ does not actually make any guarantees that the underlying memory does not get freed during the execution of the function". The same hole existed for malloc return values, which LLVM also marks dereferenceable.

What "at the point" means in practice (see isDereferenceableAndAlignedPointer in llvm/lib/Analysis/Loads.cpp and willNotFreeBetween in ValueTracking.cpp): when LLVM wants to speculate a load through a dereferenceable argument, it first asks whether the pointer can be freed at all. It cannot if the whole function is nofree, if the argument is nofree noalias, or (new in LLVM 24) if it is nofreeobj. Otherwise LLVM walks backwards from the load to the function entry, and every call on the way must be nofree, nothing on the way may synchronize, and the walk gives up after 32 instructions (a hard-coded MaxInstrsToCheckForFree). One opaque call, or one long enough straight-line block, and the fact is gone.

In the example above, g() is exactly such a call. If g were visible and inferred nofree, or if f called nothing at all, the cmove would still be there.

This has a direct consequence for my Verilator PR. The generated code does not take a reference parameter. It does

auto& vlSelfRef = std::ref(*vlSelf).get();

which works because std::reference_wrapper<T>::get() returns T&, so the call to get() carries dereferenceable(sizeof(T)) on its result, and the SimplifyCFG run that sits right before the inliner uses it. After inlining, the call is gone and so is the attribute, so that one pass is the only chance. Under the at-point semantics a call result is something that "can be freed", so every later load needs the 32-instruction, nofree-only walk back to get(). A Verilator eval function is hundreds of statements long and calls trigger helpers before the first mux, so the walk never succeeds. Counting conditional jumps in the code Verilator generates for a small design with four muxes and a register bank, once as emitted (reference) and once with vlSelfRef. rewritten to vlSelfRef-> (pointer):

With LLVM 23.1 or newer, the std::ref trick buys nothing on this design. I have submitted Verilator PR #8339 to replace all the existing pointer parameters with reference parameters, thus fixing the performance issue.

How x86-APX further improves performance

Everything so far comes down to one problem: a load / store that the program did not ask for may fault, so the compiler needs a proof that it cannot. References supply that proof, and the previous section showed how brittle the proof is. Intel APX (Advanced Performance Extensions) attacks the problem from the hardware side with a new instruction family, CFCMOVcc, and it will be appear in Intel Nova Lake and Diamond Rapids. The CF prefix means conditionally faulting: it is cmov, but when the condition is false and the operand is in memory, the memory access is not performed and any fault it would have raised is suppressed. The forms, with the names LLVM uses in X86InstrCMovSetCC.td:

Only 16, 32 and 64-bit integer operands exist. With this instruction the compiler no longer needs to know anything about the pointer. The hardware simply does not touch memory when the condition is false, and since it is not a branch, nothing is predicted and nothing can be mispredicted.

Take the pointer version of the mux from the beginning, s->b = s->a ? s->c : 0, and add -mapx-features=cf:

// mux.cpp
struct S { int a, b, c, d; };
void by_ptr(S* s) { s->b = s->a ? s->c : 0; }
void by_ref(S& s) { s.b  = s.a  ? s.c  : 0; }

clang++ -O3 -mapx-features=cf -S mux.cpp

; -O3
_Z6by_ptrP1S:
    cmpl    $0, (%rdi)
    je  .LBB0_1
    movl    8(%rdi), %eax
    movl    %eax, 4(%rdi)
    retq
.LBB0_1:
    xorl    %eax, %eax
    movl    %eax, 4(%rdi)
    retq

; -O3 -mapx-features=cf
_Z6by_ptrP1S:
    cmpl    $0, (%rdi)
    cfcmovnel   8(%rdi), %eax
    movl    %eax, 4(%rdi)
    retq

As we can see, the pointer version now uses the cfcmov instruction, which conditionally moves the value without a branch, improving performance if the branch is hard to predict.

However, you may mentioned that I'm using a new cpp file called mux.cpp for this demonstration, which is separate from the previous examples ref.cpp. Because the ref.cpp cannot being optimized with the cfcmov instruction now since it has two memory accesses involved in the conditional assignment, while most of time branch prediction would handle it efficiently.

Thus, we may leave the future work to make the compiler further optimize such patterns using the cfcmov instruction.

Can we make the performance better with Verilator's case even without x86-APX?

Even with replacing pointer types with references, we only reduce the branch by 8.75%, and gets 10.19% improvement in overall performance. However, this risky optimization stops 2 month ago. Luckily, we always have a optimization that is PGO (Profile-Guided Optimization)!

Profile-guided optimization further helps in improving performance by providing the compiler with runtime information about branch behavior, thus compiler would be able to determine the branch is likely to be taken or not taken more accurately. My findings is that:

• With LLVM's -fprofile-instr-generate (Frontend PGO), it will make the Verilator XiangShan get ~1.5x speedup.
• With LLVM's -fprofile-generate (Backend PGO) or BOLT's -reorder-blocks=ext-tsp, it will make the Verilator XiangShan get ~2x speedup with only ~0.2% Branch MPKI, but BOLT is much faster with hardware branch sampling (Intel LBR or AMD LbrExtV2) since it does not require recompilation, as long as we have binaries compiled with -Wl,--emit-relocs.

For these reasons, profile-guided optimization can significantly enhance the performance benefits of using references and other code optimizations by providing the compiler with accurate runtime information about branch behavior.