Is TSO a historical burden for modern CPU?

Preface

x86-TSO (Total Store Order) is the memory consistency model used by x86 and x86-64 architectures. It provides a balance between performance and ease of programming by allowing certain types of instruction reordering while maintaining a relatively strong consistency model. However, as modern CPUs have evolved, there has been ongoing debate about whether TSO remains the optimal choice for contemporary computing needs. Newer architectures, such as aarch64 and RISC-V, have adopted relaxed memory models that allow more reordering of memory operations between different threads, which can potentially lead to better performance in multi-core systems.

I’m interested in exploring whether TSO is becoming a historical burden for modern CPUs, especially in the context of multi-core and high-performance computing. Luckily, Apple Silicon from M1 have an custom implementation of TSO on aarch64 architecture, and can be toggled on and off via a MSR register. This provides a unique opportunity to compare the performance of TSO and relaxed memory models on the same hardware platform.

And I have developed tools to evaluate it on Apple M1 with Asahi Linux by m1tso-linux and m1-pmu-gen. Make it possible to measure the performance counter events under different memory models.

I assume the reader have basic knowledge of memory consistency models, you should at least know what is the difference between TSO and relaxed memory model. You can refer to Memory Ordering for a quick introduction. (I’m sorry that post has only Chinese version for now.)

Intuition

The reordering type is the ordering of memory operations that a CPU is allowed to perform out of program order. The following table summarizes the differences in reordering types allowed by TSO and relaxed memory models:

As we can see from the table, TSO allows store to load reordering, which allows store buffers to exist for better performance. However, TSO disallows load to load and store to store reordering, and also load to store reordering. In contrast, relaxed memory models allow all types of reordering, which can lead to better performance in multi-core systems.

Thus, how can we implement a hardware TSO on a real OoO (Out-of-Order) CPU in addition to Relaxed memory model with store buffer already present? While the store buffer ensures that store to store reordering is prevented, also store to load reordering is allowed. Then, the only thing we need to do is to prevent load to load reordering.

One possible solution is to implement a mechanism that tracks cache snooping for completed but not yet retired loads. If a load is found to have been snooped by another core, the CPU can replay the loads in program order to ensure that the load to load reordering is prevented. This approach can be implemented with minimal performance overhead, as it only requires additional tracking for loads that have been snooped by other cores.

That’s the simplest case. In practice, complex OoO CPUs may have other optimizations that make use of relaxed memory models, Apple M1 will tell us the real performance impact of TSO.

A simple TSO test

Motivated by testtso, I wrote a similar test program to evaluate the performance impact of TSO on Apple M1.

#include <atomic>
#include <thread>
#include <cassert>

constexpr size_t BUFFER_SIZE = 1 << 12;
std::atomic<int> buffer[BUFFER_SIZE];

void writer() {
    int local_counter = 0;
    while (true) {
        // 1. Sequentially write local_counter to the entire buffer
        for (size_t i = 0; i < BUFFER_SIZE; ++i) {
            buffer[i].store(local_counter, std::memory_order_relaxed);
        }
        local_counter ++;
    }
}

void reader() {
    while (true) {
        for (size_t i = 0; i < BUFFER_SIZE - 1; ++i) {
            // 2. Check that each adjacent pair in the buffer is non-decreasing
            int val1 = buffer[i+1].load(std::memory_order_relaxed);
            int val2 = buffer[i  ].load(std::memory_order_relaxed);
#ifndef NO_ASSERT
            /* In TSO, writes are seen in order, loads are not reordered
               with other loads.  Thus, buffer[i+1] should never be less
               than buffer[i] in global view in TSO.  But in relaxed memory
               model, it's possible to observe val1 > val2 due to store to
               store can be reordered, load to load can also be reordered. */
            assert(val1 <= val2);
#endif
        }
    }
}

int main() {
    std::thread writer_thread(writer);
    std::thread reader_thread(reader);

    writer_thread.join();
    reader_thread.join();

    return 0;
}

Then compile on GCC 15 with -O3 and run it on different systems.

On typical x86-64 systems, this test should pass without any assertion failures (unless you observed integer overflow after a long run). However, I have observed that on Apple M1 (Firestorm cores) and Raspberry Pi 4 (Cortex-A72), the test fails frequently when TSO is disabled, indicating that the relaxed memory model allows reordering that violates the expected behavior.

Thus, here comes a question: what is the performance impact of TSO on this test? How x86-64 systems perform on this test compared to Apple M1 with TSO enabled and disabled?

We first look into the IPC (Instructions Per Cycle) metric on Apple M1 with TSO enabled and disabled, using perf stat --per-thread -e instructions,cycles and modified the test by comment out the assertion line to skip the failure issue. All the experiments have pinned the threads to only one Firestorm cluster on Apple M1 by numactl --physcpubind=4-5.

The reader thread shows a significant performance drop when TSO is enabled, while the writer thread shows a substantial performance improvement.

Here comes a question: What happens to the writer thread when TSO is enabled?

Let’s dive into the performance counter events. Since there is no so many performance counter can be configured at once on Apple M1, I sample 2 events every one second to iterate over all events on Apple M1, and filter out the events that occurred 0.0001 per instruction or more in RMO mode. Since our program is doing the exact same work in both modes no matter what the other thread is doing, we can compare the per-instruction event counts between RMO and TSO modes to see what changed.

For the reader thread:

As we can see, there is more LD_UNIT_UOP and L1D_CACHE_MISS_LD in TSO mode, indicating that loads are taking longer to complete due to the additional constraints imposed by TSO, perhaps due to the load flushes/replays when loads are snooped by other cores. This leads to more stalls in the scheduling and mapping stages, resulting in lower IPC.

For the writer thread:

Here we can see a significant reduction in ST_NT_UOP (Store Non-Temporal UOP) in TSO mode. While nearly all stores in RMO mode are non-temporal stores, which bypass the cache and go directly to memory, in TSO mode, there is no non-temporal stores observed. This suggests that TSO mode restricts the use of non-temporal stores in Apple M1, likely due to the need to maintain store order for TSO compliance. As a result, stores may have to go through the cache hierarchy, leading to increased cache writebacks and potentially higher latency for store operations. This change in store behavior could explain the improved performance of the writer thread in TSO mode, as it may benefit from better cache utilization and reduced memory access latency compared to RMO mode.

From the above analysis, we can conclude that TSO imposes additional constraints on memory operations that can lead to performance degradation in certain scenarios, particularly for load-heavy workloads. However, when it meets the wrong non-temporal store usage, Apple’s TSO implementation on M1 can actually improve performance by enforcing a more cache-friendly store behavior. That’s quite interesting!

Real world applications

To further evaluate the performance impact of TSO on real-world applications, I ran SPECrate 2006 and SPECrate 2017 benchmarks with only 1 copy (Single Thread) on Apple M1 with TSO enabled and disabled. All the benchmarks were compiled with GCC 15 with -O3 -march=armv8.5-a. Note that I didn’t enabled LTO like most SPECrate submissions do to reduce the complexity of the build process. The results are as follows:

SPECrate 2006:

SPECrate 2017:

From the benchmark results, we can observe that several benchmarks experience significant performance degradation when TSO is enabled. Notably, benchmarks such as 464.h264ref, 483.xalancbmk, 410.bwaves, 434.zeusmp, 436.cactusADM, 437.leslie3d, 453.povray, 459.GemsFDTD, 465.tonto, and 470.lbm in SPECrate 2006 show performance drops ranging from approximately 11% to 64%. Similarly, in SPECrate 2017, benchmarks like 525.x264_r, 503.bwaves_r, 507.cactuBSSN_r, 511.povray_r, 521.wrf_r, 527.cam4_r, 549.fotonik3d_r, and 554.roms_r exhibit performance reductions between roughly 17% to 38%.

How will x86 systems outcome on these benchmarks? It’s hard to say without the same hardware platform to compare. But based on the previous analysis of the simple TSO test, we can speculate that workloads with high memory access patterns, especially those involving frequent loads and stores, are more likely to be negatively impacted by TSO due to the additional constraints it imposes on memory operation ordering.

Conclusion

From the experiments and analysis conducted on Apple M1 with TSO enabled and disabled, we can conclude that TSO can indeed be a performance burden for modern CPUs in certain scenarios. The simple TSO test demonstrated that TSO imposes additional constraints on memory operations, leading to increased load latencies and stalls in load-heavy workloads. However, not all optimizations to overcome TSO’s limitations are effective. For instance, Apple’s non-temporal store optimization is disabled under TSO, which can lead to performance improvements in specific workloads when TSO is enabled.

It’s better to implement such microarchitectural features on gem5 and characterize them more thoroughly. Hope to do it in the future.

Source code of the simple TSO test is available at github.com/cyyself/tsotest.