<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://devsh0.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://devsh0.github.io/" rel="alternate" type="text/html" /><updated>2026-11-17T17:28:33+00:00</updated><id>https://devsh0.github.io/feed.xml</id><title type="html">Shannonigans</title><subtitle>Blog about random computer stuff.</subtitle><entry><title type="html">Reversing the Next-Page Prefetcher in Skylake-X</title><link href="https://devsh0.github.io/microarchitecture,/intel,/computer-architecture/2026/11/17/reversing-the-next-page-prefetcher-in-skylake-x.html" rel="alternate" type="text/html" title="Reversing the Next-Page Prefetcher in Skylake-X" /><published>2026-11-17T16:00:25+00:00</published><updated>2026-11-17T16:00:25+00:00</updated><id>https://devsh0.github.io/microarchitecture,/intel,/computer-architecture/2026/11/17/reversing-the-next-page-prefetcher-in-skylake-x</id><content type="html" xml:base="https://devsh0.github.io/microarchitecture,/intel,/computer-architecture/2026/11/17/reversing-the-next-page-prefetcher-in-skylake-x.html"><![CDATA[<h2 id="introduction">Introduction</h2>

<p>Intel publishes very little about next-page prefetchers in their P-cores. Back in the Ivy Bridge days, it was known that
IVB included a next-page prefetcher thanks to this ancient excerpt from an old copy of the Intel Architectures Optimization
Reference Manual:</p>

<blockquote>
  <p>Hardware prefetch enhancement: A next-page prefetcher (NPP) is added in Intel microarchitecture code name Ivy Bridge.
The NPP is triggered by sequential accesses to cache lines approaching the page boundary, either upwards or downwards.</p>
</blockquote>

<p>A recent issue of the optimization manual states that Redwood Cove includes a Last Level Cache (LLC) Page Prefetcher
which is like NPP but with a slightly more sophisticated prefetch algorithm:</p>

<blockquote>
  <p>The LLC Page Prefetcher extends the Next-Page-Prefetcher algorithm, which detects a sequence of accesses that are about
to reach the end of a page (either going up towards the next page in memory or down towards the previous page in memory)
and triggers a linear prefetch to the start of the next predicted page. NPP intends to prefetch the translation of the
next page and initiate prefetching of the first few cache lines from that predicted page. The LLC Page Prefetch enhances
Next-Page Prefetcher in two significant ways:</p>

  <ul>
    <li>It issues prefetches two pages ahead in linear address space.</li>
    <li>The prefetch attempts to bring the entire 4Kb page to the last level cache using opportunistic IDI request slots
without the DCU or MLC buffers.</li>
  </ul>
</blockquote>

<p>Redwood Cove is a relatively recent microarchitecture which shipped in Meteor Lake P-cores starting late 2023. Ivy Bridge
on the other hand is very old; It launched back in 2012 and was discontinued by 2015 right before Skylake launched. That
raises the question, what happened in-between? Did every microarchitecture after IVB include a next-page prefetcher?</p>

<p>It’s unlikely that the details remain unchanged from one microarchitecture to the next but as it turns out, I recently
got unlimited access to a Skylake-X machine that I didn’t have to pay a single dime for. Given the setup, it’s only natural
I got nerd-sniped into reversing the next-page prefetcher…and that’s what this post is about. I hope you enjoy it!</p>

<h2 id="so-whats-a-prefetcher">So What’s a Prefetcher</h2>

<p>Hardware component that preloads data that the program might need soon. The goal is to eliminate memory stalls when the
CPU actually demands the piece of data that the prefetcher could have loaded earlier. All kinds of prefetchers exist in
modern CPUs. Next-Line prefetchers prefetch the immediate next cache line when the N’th cache line is accessed. Stride
prefetchers (usually IP-based) track the stride of consecutive memory accesses that originate from the same instruction and
if the access pattern turns out to be regular enough, it eagerly initiates the load from address that it predicted. Next-Page
Prefetchers preload page table entries into the Translation Lookaside Buffer (TLB) provided software traverses memory pages
in predictable manner. These by no means are the only three kind; a high-performance CPU may implement many other types
of prefetchers, some of which are documented in <a href="https://intel.github.io/optimization-zone/hardware/HWPrefetchTuning/#123-next-page-prefetcher-l1-npp">this</a> Intel whitepaper.</p>

<p>Prefetchers that are required to act fast are logically simple and tend not to cross page boundaries because that may complicate
things with, e.g. page faults and page walks on TLB misses. For example, the next-line prefetcher is supposed to fetch
the next cache line if the accesses are sequential, but if the next cache line sits in a different page it may decide not
to prefetch at all. Next-page prefetchers are slightly different in that regard; they prefetch the next page table entry
(PTE) into the TLB. They have to work across page boundaries by design.</p>

<p>In this post I’ll try to answer the following questions:</p>

<ol>
  <li>Does Skylake-X have a next-page prefetcher?</li>
  <li>If yes, what triggers the next-page prefetch?</li>
  <li>How many “future” PTEs are prefetched?</li>
  <li>Where does the NPP load future PTEs?</li>
  <li>What’s the direction of the prefetch? Forward-only? Backward-only (that’d be weird!)? Both?</li>
</ol>

<p>But before we start, let’s prepare the stage for our experiments.</p>

<h2 id="setting-up-linux-for-experiments">Setting-up Linux for Experiments</h2>

<p>If you don’t intend to run the experiments described in this post, you can skip this part.</p>

<p>Documenting the many details of performance monitoring on various platforms deserves a post of its own, and a fairly long
one at that. For simplicity I’m only going to focus on Linux. Linux allows configuring performance monitoring events via
the <code class="language-plaintext highlighter-rouge">perf_event_open</code> syscall. That’s what I’ll be using in my experiments too but generally speaking, you don’t need special
system APIs to access PMU counters. At least on Intel, you only need some way of writing a set of performance monitoring
MSRs (usually only the <code class="language-plaintext highlighter-rouge">IA32_PERFEVTSELx</code> MSRs) and read the counters back from a set of corresponding <code class="language-plaintext highlighter-rouge">IA32_PMCx</code> MSRs.
<a href="https://intel.github.io/SDM/instruction/WRMSR.html"><code class="language-plaintext highlighter-rouge">WRMSR</code></a> and <a href="https://intel.github.io/SDM/instruction/RDMSR.html"><code class="language-plaintext highlighter-rouge">RDMSR</code></a> require Ring0 access so if you’re on Windows, you’d have to
<a href="https://learn.microsoft.com/en-us/windows-hardware/drivers/gettingstarted/writing-a-very-small-kmdf--driver">write a kernel-mode driver</a> and use <a href="https://learn.microsoft.com/en-us/cpp/intrinsics/readmsr"><code class="language-plaintext highlighter-rouge">__readmsr</code></a>/<a href="https://learn.microsoft.com/en-us/cpp/intrinsics/writemsr"><code class="language-plaintext highlighter-rouge">__writemsr</code></a> intrinsics to read/write
the relevant MSRs. That said, the hard part of making custom drivers for Windows isn’t writing the driver itself, rather
convincing the kernel to load your driver. More on that in a future blog post.</p>

<p>Performance monitoring tools that allow reading PMU counters (e.g. perf, VTune) may also work. I’d stick to <code class="language-plaintext highlighter-rouge">perf_event_open</code>
because it allows better control over the exact segment of code I need to measure. The <a href="https://man7.org/linux/man-pages/man2/perf_event_open.2.html"><code class="language-plaintext highlighter-rouge">manpage</code></a> includes an
example of how to use <code class="language-plaintext highlighter-rouge">perf_event_open</code>. For our experimenting we need three things in order:</p>

<ol>
  <li>Configuring the events we want to count. Initially all counters will be disabled.</li>
  <li>Enabling the counters and running the workload. Here’s where the <code class="language-plaintext highlighter-rouge">IA32_PMCx</code> counters start ticking.</li>
  <li>Once the workload completes execution, stopping the counters and reading their values.</li>
</ol>

<p>We’ll write a helper for each of these steps: <code class="language-plaintext highlighter-rouge">configure_event_group</code> to configure the events, <code class="language-plaintext highlighter-rouge">start_counters</code> to enable
the counters, and <code class="language-plaintext highlighter-rouge">stop_counters</code> to stop and read the counters. With that aside, here’s <code class="language-plaintext highlighter-rouge">configure_event_group</code>:</p>

<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="c1">// Configures a set of events (i.e. an "event group") for counting.</span>
<span class="kt">void</span> <span class="nf">configure_event_group</span><span class="p">(</span><span class="n">PerfContext</span><span class="o">&amp;</span> <span class="n">ctx</span><span class="p">)</span> <span class="p">{</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">ctx</span><span class="p">.</span><span class="n">nevents</span> <span class="o">&lt;</span> <span class="mi">1</span> <span class="o">||</span> <span class="n">ctx</span><span class="p">.</span><span class="n">nevents</span> <span class="o">&gt;</span> <span class="mi">4</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">printf</span><span class="p">(</span><span class="s">"Number of events must be between 1..4!</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
        <span class="n">exit</span><span class="p">(</span><span class="n">EXIT_FAILURE</span><span class="p">);</span>
    <span class="p">}</span>

    <span class="kt">int</span> <span class="n">pid</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>                <span class="c1">// Count events for the calling process.</span>
    <span class="kt">int</span> <span class="n">cpu</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>               <span class="c1">// Count events for the calling thread. Counters don't reset when thread migrates CPU.</span>
    <span class="kt">int</span> <span class="n">group_fd</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span><span class="p">;</span>          <span class="c1">// The group leader, always created with fd = -1.</span>
    <span class="kt">int</span> <span class="n">flags</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>              <span class="c1">// Use no special flags.</span>

    <span class="n">perf_event_attr</span> <span class="n">attr</span><span class="p">;</span>
    <span class="n">memset</span><span class="p">(</span><span class="o">&amp;</span><span class="n">attr</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">attr</span><span class="p">));</span>

    <span class="n">attr</span><span class="p">.</span><span class="n">type</span> <span class="o">=</span> <span class="n">PERF_TYPE_RAW</span><span class="p">;</span>              <span class="c1">// Allow configuring events with raw event codes.</span>
    <span class="n">attr</span><span class="p">.</span><span class="n">size</span> <span class="o">=</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">attr</span><span class="p">);</span>
    <span class="n">attr</span><span class="p">.</span><span class="n">config</span> <span class="o">=</span> <span class="n">ctx</span><span class="p">.</span><span class="n">events</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>            <span class="c1">// Configure the first event in the event group.</span>
    <span class="n">attr</span><span class="p">.</span><span class="n">disabled</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>                      <span class="c1">// Start out with counters disabled.</span>
    <span class="n">attr</span><span class="p">.</span><span class="n">exclude_kernel</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>                <span class="c1">// Stop counting when control transfers to kernel space.</span>
    <span class="n">attr</span><span class="p">.</span><span class="n">exclude_hv</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>                    <span class="c1">// Stop counting when control transfers to hypervisor.</span>
    <span class="n">attr</span><span class="p">.</span><span class="n">read_format</span> <span class="o">=</span> <span class="n">PERF_FORMAT_GROUP</span><span class="p">;</span>   <span class="c1">// Read all counters in the event group with one `read` call.</span>

    <span class="n">group_fd</span> <span class="o">=</span> <span class="n">syscall</span><span class="p">(</span><span class="n">SYS_perf_event_open</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">attr</span><span class="p">,</span> <span class="n">pid</span><span class="p">,</span> <span class="n">cpu</span><span class="p">,</span> <span class="n">group_fd</span><span class="p">,</span> <span class="n">flags</span><span class="p">);</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">group_fd</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">perror</span><span class="p">(</span><span class="s">"perf_event_open"</span><span class="p">);</span>
        <span class="n">exit</span><span class="p">(</span><span class="n">EXIT_FAILURE</span><span class="p">);</span>
    <span class="p">}</span>
    <span class="n">ctx</span><span class="p">.</span><span class="n">open_fds</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">group_fd</span><span class="p">;</span>

    <span class="c1">// Add the rest of the events in the event group.</span>
    <span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">ctx</span><span class="p">.</span><span class="n">nevents</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">attr</span><span class="p">.</span><span class="n">config</span> <span class="o">=</span> <span class="n">ctx</span><span class="p">.</span><span class="n">events</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
        <span class="kt">int</span> <span class="n">fd</span> <span class="o">=</span> <span class="n">syscall</span><span class="p">(</span><span class="n">SYS_perf_event_open</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">attr</span><span class="p">,</span> <span class="n">pid</span><span class="p">,</span> <span class="n">cpu</span><span class="p">,</span> <span class="n">group_fd</span><span class="p">,</span> <span class="n">flags</span><span class="p">);</span>
        <span class="k">if</span> <span class="p">(</span><span class="n">fd</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
            <span class="n">perror</span><span class="p">(</span><span class="s">"perf_event_open"</span><span class="p">);</span>
            <span class="n">exit</span><span class="p">(</span><span class="n">EXIT_FAILURE</span><span class="p">);</span>
        <span class="p">}</span>
        <span class="n">ctx</span><span class="p">.</span><span class="n">open_fds</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">fd</span><span class="p">;</span>
    <span class="p">}</span>
<span class="p">}</span></code></pre></figure>

<p>Full source code of every experiment, data structure, and helpers used in this article can be found on <a href="https://gist.github.com/devsh0/66fea1019aee8eda54925ee75b73852a">github</a>.
The <code class="language-plaintext highlighter-rouge">group_fd</code> identifies the event group that you configure via <code class="language-plaintext highlighter-rouge">perf_event_open</code>. You can use that <code class="language-plaintext highlighter-rouge">group_fd</code> to perform
certain operations on the event group, like enabling or disabling the counters. That’s what we’re going to do next: enable
the counters with <code class="language-plaintext highlighter-rouge">start_counters</code> which is defined like this:</p>

<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="c1">// Sets off the counters for events in the event group. ctx.open_fds[0] is the group_fd.</span>
<span class="kt">void</span> <span class="nf">start_counters</span><span class="p">(</span><span class="n">PerfContext</span><span class="o">&amp;</span> <span class="n">ctx</span><span class="p">)</span> <span class="p">{</span>
    <span class="kt">int</span> <span class="n">ret</span> <span class="o">=</span> <span class="n">ioctl</span><span class="p">(</span><span class="n">ctx</span><span class="p">.</span><span class="n">open_fds</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">PERF_EVENT_IOC_ENABLE</span><span class="p">,</span> <span class="n">PERF_IOC_FLAG_GROUP</span><span class="p">);</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">ret</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">perror</span><span class="p">(</span><span class="s">"ioctl"</span><span class="p">);</span>
        <span class="n">exit</span><span class="p">(</span><span class="n">EXIT_FAILURE</span><span class="p">);</span>
    <span class="p">}</span>
<span class="p">}</span></code></pre></figure>

<p>Calling <code class="language-plaintext highlighter-rouge">start_counters</code> sets off the counters for every event configured in the event group identified by <code class="language-plaintext highlighter-rouge">group_fd</code>.
At this point we’ll run our workload, then call <code class="language-plaintext highlighter-rouge">stop_counters</code> to stop and read the counters:</p>

<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="c1">// Stops and reads the counters in the event group. ctx.open_fds[0] is the group_fd.</span>
<span class="kt">void</span> <span class="nf">stop_counters</span><span class="p">(</span><span class="n">PerfContext</span><span class="o">&amp;</span> <span class="n">ctx</span><span class="p">)</span> <span class="p">{</span>
    <span class="c1">// Stop the counters.</span>
    <span class="kt">int</span> <span class="n">group_fd</span> <span class="o">=</span> <span class="n">ctx</span><span class="p">.</span><span class="n">open_fds</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
    <span class="kt">int</span> <span class="n">ret</span> <span class="o">=</span> <span class="n">ioctl</span><span class="p">(</span><span class="n">group_fd</span><span class="p">,</span> <span class="n">PERF_EVENT_IOC_DISABLE</span><span class="p">,</span> <span class="n">PERF_IOC_FLAG_GROUP</span><span class="p">);</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">ret</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">perror</span><span class="p">(</span><span class="s">"ioctl"</span><span class="p">);</span>
        <span class="n">exit</span><span class="p">(</span><span class="n">EXIT_FAILURE</span><span class="p">);</span>
    <span class="p">}</span>
    <span class="c1">// Read the counters.</span>
    <span class="n">u64</span> <span class="n">counters_with_header</span><span class="p">[</span><span class="mi">5</span><span class="p">];</span>
    <span class="kt">int</span> <span class="n">len</span> <span class="o">=</span> <span class="p">(</span><span class="n">ctx</span><span class="p">.</span><span class="n">nevents</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="o">*</span> <span class="nf">sizeof</span><span class="p">(</span><span class="n">u64</span><span class="p">);</span>
    <span class="n">ret</span> <span class="o">=</span> <span class="n">read</span><span class="p">(</span><span class="n">group_fd</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">counters_with_header</span><span class="p">,</span> <span class="n">len</span><span class="p">);</span>
    <span class="k">if</span> <span class="p">(</span><span class="n">ret</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">perror</span><span class="p">(</span><span class="s">"read"</span><span class="p">);</span>
        <span class="n">exit</span><span class="p">(</span><span class="n">EXIT_FAILURE</span><span class="p">);</span>
    <span class="p">}</span>
    <span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">ctx</span><span class="p">.</span><span class="n">nevents</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">ctx</span><span class="p">.</span><span class="n">counters</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">counters_with_header</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="mi">1</span><span class="p">];</span>
    <span class="p">}</span>
    <span class="c1">// Close all fds in the event group.</span>
    <span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">ctx</span><span class="p">.</span><span class="n">nevents</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
        <span class="kt">int</span> <span class="n">ret</span> <span class="o">=</span> <span class="n">close</span><span class="p">(</span><span class="n">ctx</span><span class="p">.</span><span class="n">open_fds</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span>
        <span class="k">if</span> <span class="p">(</span><span class="n">ret</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
            <span class="n">perror</span><span class="p">(</span><span class="s">"close"</span><span class="p">);</span>
            <span class="n">exit</span><span class="p">(</span><span class="n">EXIT_FAILURE</span><span class="p">);</span>
        <span class="p">}</span>
    <span class="p">}</span>
<span class="p">}</span></code></pre></figure>

<p>Before we can read PMU counters using the functions defined above, we need to tweak some settings on Linux. Depending on
your distro, certain restrictions may apply by default on hardware PMU access due to security reasons. You can overcome
them by setting <a href="https://www.kernel.org/doc/html/latest/admin-guide/perf-security.html#unprivileged-users"><code class="language-plaintext highlighter-rouge">perf_event_paranoid</code></a> to <code class="language-plaintext highlighter-rouge">0</code> or <code class="language-plaintext highlighter-rouge">-1</code>:</p>

<figure class="highlight"><pre><code class="language-shell" data-lang="shell"><span class="nb">echo</span> <span class="s2">"-1"</span> | <span class="nb">sudo tee</span> /proc/sys/kernel/perf_event_paranoid</code></pre></figure>

<p>You may also need to <a href="https://access.redhat.com/solutions/46111">disable THP</a> and pin the calling thread to a core if there’s too much going on in your
machine.</p>

<p>Now we have all the pieces required to start monitoring hardware events. Throughout this post we’d work with the following events:</p>

<table>
  <thead>
    <tr>
      <th>Event</th>
      <th>Config</th>
      <th>Description</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><a href="https://perfmon-events.intel.com/platforms/skylakex/core-events/core/?h=skylak#event-MEM_INST_RETIRED.STLB_MISS_LOADS">MEM_INST_RETIRED.STLB_MISS_LOADS</a></td>
      <td>0x11D0</td>
      <td>Retired load instructions that start a miss in the STLB.</td>
    </tr>
    <tr>
      <td><a href="https://perfmon-events.intel.com/platforms/skylakex/core-events/core/?h=skylak#event-MEM_INST_RETIRED.STLB_MISS_STORES">MEM_INST_RETIRED.STLB_MISS_STORES</a></td>
      <td>0x12D0</td>
      <td>Retired store instructions that start a miss in the STLB.</td>
    </tr>
    <tr>
      <td><a href="https://perfmon-events.intel.com/platforms/skylakex/core-events/core/?h=skylak#event-DTLB_LOAD_MISSES.STLB_HIT">DTLB_LOAD_MISSES.STLB_HIT</a></td>
      <td>0x2008</td>
      <td>Loads that miss the data TLB and hit the unified STLB.</td>
    </tr>
  </tbody>
</table>

<p>It’s note hard to observe that the event configs are of the form 0x&lt;Umask&gt;&lt;EventSel&gt;, where Umask and EventSel are coming
straight from the “PROGRAMMING INFO” column of the corresponding event in the <a href="https://perfmon-events.intel.com">perfmon website</a> (the ‘H’
suffix is Intel’s way of saying the number is given in hexadecimal). Not all event configs are derived this way, but it’s
enough for the simple ones we’re gonna use here.</p>

<h2 id="tlb-hierarchy-in-skylake-x">TLB Hierarchy in Skylake-X</h2>

<p>SKX (codename for Skylake-X) has two TLB levels: L1 and L2. Each SKX core gets a physically separate L1 data TLB (dTLB)
and L1 instruction TLB (iTLB). The L2 TLB (also called Second Level TLB or STLB) is unified, i.e. both instruction and
data PTEs go to the same STLB.</p>

<p>Both L1 and L2 TLBs are partitioned based on the page size. L1 dTLB has a total of 100 entries: 64 reserved for 4k pages,
32 for 2M or 4M pages, and 4 for 1G pages. The dTLB is 4-way set associative so each set can hold 4 page-table entries.
The iTLB is also partitioned: has 128 entries for 4k pages, 16 entries for 2M or 4M pages, and none for 1G pages. The
STLB has 1536 entries shared between 4k and 2M pages and an additional 16 entries for 1G huge pages. STLB is not inclusive
of the L1 TLBs. That means at any given instance, a page-table entry may exist in one of the L1 TLBs, the STLB, or both.</p>

<p>For the purposes of this post, we don’t need to dig any further into the geometry of SKX TLBs. It’s time to run some
experiments!</p>

<h2 id="does-it-exist">Does it Exist</h2>

<p>The very first question that we should ask is whether the next-page prefetcher exists in SKX at all. NPPs seem like a
good idea in general; given that Intel had the silicon budget to include one in Ivy Bridge, we expect the answer to be
“Yes” in SKX too. To test this, we’ll design a workload that walks a mapped range of 1000 4K pages sequentially and
measure how many TLB misses we get. I cannot imagine a more favourable access pattern for the NPP to kick in if it truly
exists. If the workload below doesn’t trigger the prefetcher, we can safely conclude that SKX doesn’t have one.</p>

<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="p">[[</span><span class="n">gnu</span><span class="o">::</span><span class="n">always_inline</span><span class="p">]]</span>
<span class="kr">inline</span> <span class="kt">void</span> <span class="nf">touch_all_bytes</span><span class="p">(</span><span class="kt">int</span> <span class="n">zero</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">u8</span><span class="o">*</span> <span class="n">map_base</span> <span class="o">=</span> <span class="p">(</span><span class="n">u8</span><span class="o">*</span><span class="p">)</span><span class="n">MAP_BASE</span><span class="p">;</span>
    <span class="n">u8</span> <span class="n">dummy</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
    <span class="k">for</span> <span class="p">(</span><span class="n">u64</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">MAP_SIZE</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">dummy</span> <span class="o">+=</span> <span class="n">map_base</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="n">zero</span><span class="p">];</span>
        <span class="n">zero</span> <span class="o">=</span> <span class="n">zero</span> <span class="o">&amp;</span> <span class="n">dummy</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="k">asm</span> <span class="nf">volatile</span><span class="p">(</span><span class="s">""</span> <span class="o">::</span> <span class="s">"r"</span><span class="p">(</span><span class="n">dummy</span><span class="p">));</span>
<span class="p">}</span></code></pre></figure>

<p>Not much going on here. We have a function that touches every byte in the mapped range starting at <code class="language-plaintext highlighter-rouge">MAP_BASE</code>. The seemingly
useless <code class="language-plaintext highlighter-rouge">zero</code> helps ensure that loads from the mapped range execute serially. Take a closer look at how we’re using
<code class="language-plaintext highlighter-rouge">zero</code> in the loop. Loading the next byte from the mapped range at iteration <code class="language-plaintext highlighter-rouge">i</code> depends on the value of <code class="language-plaintext highlighter-rouge">zero</code> computed
at iteration <code class="language-plaintext highlighter-rouge">i-1</code>. Computing <code class="language-plaintext highlighter-rouge">zero</code> at iteration <code class="language-plaintext highlighter-rouge">i-1</code> in turn depends on computing <code class="language-plaintext highlighter-rouge">dummy</code> at iteration <code class="language-plaintext highlighter-rouge">i-1</code>. We have
a cyclic loop-carried dependency chain here: the load in <code class="language-plaintext highlighter-rouge">map_base[i + zero]</code> feeds <code class="language-plaintext highlighter-rouge">add</code> in <code class="language-plaintext highlighter-rouge">dummy += ...</code>. The result
of that <code class="language-plaintext highlighter-rouge">add</code> then feeds the <code class="language-plaintext highlighter-rouge">and</code> in <code class="language-plaintext highlighter-rouge">zero = zero &amp; ...</code>. Result of the <code class="language-plaintext highlighter-rouge">and</code> feeds the next load at iteration <code class="language-plaintext highlighter-rouge">i+1</code>.
In layman terms, the loads at iteration <code class="language-plaintext highlighter-rouge">i</code> and iteration <code class="language-plaintext highlighter-rouge">i+1</code> cannot execute together or out-of-order. This is intentional:
alloweing out-of-order execution of the two loads could significantly perturb our results. We’ll see why in a moment.</p>

<p>The last line is a common trick to avoid the compiler throw away our computation; it’s falsely advertising that the inline
asm uses the value of <code class="language-plaintext highlighter-rouge">dummy</code> computed in the loop and forcing the compiler to materialize <code class="language-plaintext highlighter-rouge">dummy</code> into a register.</p>

<p>With <code class="language-plaintext highlighter-rouge">g++ -O2</code>, the loop compiles to:</p>

<figure class="highlight"><pre><code class="language-assembly" data-lang="assembly">.loop:
    movsxd rdx, ebp                     ; tmp = zero
    add    rdx, rax                     ; tmp += i
    add    rax, 0x1                     ; i++
    add    bl, BYTE PTR [rcx+rdx]       ; dummy += map_base[tmp]
    and    ebp, ebx                     ; zero = zero &amp; dummy
    cmp    rax, 4096000                 ; if (i &lt; MAP_SIZE)
    jne    .loop                        ; goto .loop</code></pre></figure>

<p>Nice. A few instructions here and there but we got the loop compiled exactly how we wanted. Note that the dependency chain
we just discussed translates exactly to assembly too. Running the workload gives the following count for the two events:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DTLB_LOAD_MISSES.STLB_HIT           = 70
MEM_INST_RETIRED.STLB_MISS_LOADS    = 1
</code></pre></div></div>
<p>Over a few runs <code class="language-plaintext highlighter-rouge">DTLB_LOAD_MISSES.STLB_HIT</code> fluctuates between the 50..75 band, but never exceeds 75. That’s already
interesting: we walked through 1000 pages in the loop but <code class="language-plaintext highlighter-rouge">DTLB_LOAD_MISSES.STLB_HIT</code> is nowhere near 1000.
<code class="language-plaintext highlighter-rouge">DTLB_LOAD_MISSES.STLB_HIT = 70</code> means only 70 out of the 1000 PTE requests missed in dTLB and hit STLB. The obvious next
question is what happened to rest of the 1000 - 70 = 930 fresh PTE requests that the loop might have initiated? There are
two possible answers:</p>

<ol>
  <li>They miss in the STLB too.</li>
  <li>They didn’t miss dTLB.</li>
</ol>

<p>We can quickly rule out (1) because <code class="language-plaintext highlighter-rouge">MEM_INST_RETIRED.STLB_MISS_LOADS = 1</code>. So that leaves us with (2): about 930 new PTE
requests hit in the dTLB when the demand arrived. But that must mean that SKX has a prefetcher that’s prefilling the dTLB
with future PTEs. Yep! It’s the next-page prefetcher in action. Well, strictly speaking this could also be a non-NPP prefetcher
that is allowed prefetch beyond page boundaries but as they say, if it quacks like a duck, it’s the next-page prefetcher!</p>

<blockquote class="note">
  <p>Subtle point that is easy to miss: the prefetcher is loading entries into th dTLB. This doesn’t automatically
prove or disprove that it’s also prefilling STLB. In SKX the STLB acts as a victim cache, i.e. entries that are
evicted from L1 TLBs are stored in the STLB. Thinking along those lines, it doesn’t make much sense for the prefetcher
to prefill the STLB too, but I’d refrain from such speculations. Perhaps a carefully designed experiment can unveil that
secret, but that’s not my focus in this post. We’d also not worry about the iTLB for now.</p>
</blockquote>

<p>Microarchitecture explorations typically require careful use of statistics to extract signal buried in noise. Luckily in
this case the signal is so strong that we can get away with sloppiness.</p>

<p>Alright, we have established that SKX has a next-page prefetcher that prefills the dTLB. On to the next question!</p>

<h2 id="what-triggers-the-next-page-prefetcher">What Triggers the Next-Page Prefetcher</h2>

<p>Something in the loop in <code class="language-plaintext highlighter-rouge">touch_all_bytes</code> must have triggered the next-page prefetch. The most obvious guess is a load
from the previous page. Question is, which load? Is it just a single load from a special offset? A set of loads? There
are so many questions that could be asked here…we need an experiment that narrows down the space of possibilities. We can
imagine a workload that touches only the left half of each page (bytes in the range <code class="language-plaintext highlighter-rouge">page[0..2047]</code>, inclusive) and count the
TLB misses. If we still see TLB miss numbers like in our first experiment, that’d mean accesses from the left half of the
page triggers the prefetch. If we instead see TLB misses that add up to 1000 (or more), we can repeat the same experiment
but this time touching the right half of the page (bytes in the range <code class="language-plaintext highlighter-rouge">page[2048..4095]</code>, inclusive). A generic workload
that accepts an <code class="language-plaintext highlighter-rouge">offset</code> and a <code class="language-plaintext highlighter-rouge">slice_length</code> can handle both cases:</p>

<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="p">[[</span><span class="n">gnu</span><span class="o">::</span><span class="n">always_inline</span><span class="p">]]</span>
<span class="kr">inline</span> <span class="kt">void</span> <span class="nf">touch_page_slice</span><span class="p">(</span><span class="n">u64</span> <span class="n">offset</span><span class="p">,</span> <span class="n">u64</span> <span class="n">slice_length</span><span class="p">,</span> <span class="kt">int</span> <span class="n">zero</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">u8</span> <span class="n">dummy</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
    <span class="k">for</span> <span class="p">(</span><span class="n">u64</span> <span class="n">p</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">p</span> <span class="o">&lt;</span> <span class="n">N_MAPPED_PAGES</span><span class="p">;</span> <span class="n">p</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">u8</span><span class="o">*</span> <span class="n">page_base</span> <span class="o">=</span> <span class="p">(</span><span class="n">u8</span><span class="o">*</span><span class="p">)(</span><span class="n">MAP_BASE</span> <span class="o">+</span> <span class="n">p</span> <span class="o">*</span> <span class="n">PAGE_SIZE</span><span class="p">);</span>
        <span class="k">for</span> <span class="p">(</span><span class="n">u64</span> <span class="n">i</span> <span class="o">=</span> <span class="n">offset</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">offset</span><span class="o">+</span><span class="n">slice_length</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
            <span class="n">dummy</span> <span class="o">+=</span> <span class="n">page_base</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="n">zero</span><span class="p">];</span>
            <span class="n">zero</span> <span class="o">&amp;=</span> <span class="n">dummy</span><span class="p">;</span>
        <span class="p">}</span>
    <span class="p">}</span>
    <span class="k">asm</span> <span class="nf">volatile</span><span class="p">(</span><span class="s">""</span> <span class="o">::</span> <span class="s">"r"</span><span class="p">(</span><span class="n">dummy</span><span class="p">));</span>
<span class="p">}</span></code></pre></figure>

<p>When <code class="language-plaintext highlighter-rouge">touch_page_slice</code> is called with <code class="language-plaintext highlighter-rouge">offset = 0, slice_length = 2048</code>, it walks left half of each of the 1000 mapped
pages. Walking the right half is just as simple, just call <code class="language-plaintext highlighter-rouge">touch_page_slice</code> with <code class="language-plaintext highlighter-rouge">offset = 2048</code> and <code class="language-plaintext highlighter-rouge">slice_length = 2048</code>.
Results from walking the left half:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Left half:
DTLB_LOAD_MISSES.STLB_HIT               = 33
MEM_INST_RETIRED.STLB_MISS_LOADS        = 1001
</code></pre></div></div>

<p>Walking the left half generated <code class="language-plaintext highlighter-rouge">MEM_INST_RETIRED.STLB_MISS_LOADS = 1001</code>. That’s suspiciously close to the total number
of pages we walked! The most sensible explanation is that the NPP did not trigger while walking the left half, thus every
new PTE request missed in the dTLB. Those misses then made it to STLB and then missed there too.</p>

<p>Alright, what about the right half? Here’s the result from waking the right half:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Right half:
DTLB_LOAD_MISSES.STLB_HIT           = 32
MEM_INST_RETIRED.STLB_MISS_LOADS    = 2
</code></pre></div></div>

<p>Comparatively, walking the right half is not bad at all. In fact this time the prefetcher clearly did trigger. We can keep
splitting the slice that triggers prefetch binary-search style and further narrow the access range. The table below shows
the results:</p>

<table>
  <thead>
    <tr>
      <th>Left half</th>
      <th>Right half</th>
      <th>Slice length</th>
      <th>Pretech Triggered By</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>[0..2047]</td>
      <td>[2048..4095]</td>
      <td>2048</td>
      <td>Right Half</td>
    </tr>
    <tr>
      <td>[2048..3071]</td>
      <td>[3072..4095]</td>
      <td>1024</td>
      <td>Right Half</td>
    </tr>
    <tr>
      <td>[3072..3583]</td>
      <td>[3584..4095]</td>
      <td>512</td>
      <td>Left Half</td>
    </tr>
    <tr>
      <td>[3072..3327]</td>
      <td>[3328..3583]</td>
      <td>256</td>
      <td>Right Half</td>
    </tr>
    <tr>
      <td>[3328..3455]</td>
      <td>[3456..3583]</td>
      <td>128</td>
      <td>Left Half</td>
    </tr>
    <tr>
      <td>[3328..3391]</td>
      <td>[3392..3455]</td>
      <td>64</td>
      <td>Neither Half</td>
    </tr>
  </tbody>
</table>

<p>Hmm, neither left nor right slice of the last row triggers the prefetcher. But we do know that walking <code class="language-plaintext highlighter-rouge">page[3328..3455]</code>
does work. So here’s the new plan: we start our walk at offset 3328 but not stop at 3391, rather keep pushing the right
edge one byte at a time until we cause the prefetcher to trigger. For example, we’d walk <code class="language-plaintext highlighter-rouge">page[3328..3392]</code> and if that
didn’t trigger NPP, we’d walk <code class="language-plaintext highlighter-rouge">page[3328..3393]</code> and so forth. We expect that the prefetcher would trigger before we reach
<code class="language-plaintext highlighter-rouge">page[3328..3455]</code>. We can reuse <code class="language-plaintext highlighter-rouge">touch_page_slice</code> here too; just call it with <code class="language-plaintext highlighter-rouge">offset = 3328</code> and <code class="language-plaintext highlighter-rouge">slice_length = 65,66,...</code>.</p>

<p>It doesn’t take long to identify the slice that triggers prefetch: the sneaky <code class="language-plaintext highlighter-rouge">page[3328..3392]</code>. Calling <code class="language-plaintext highlighter-rouge">touch_page_slice</code>
with <code class="language-plaintext highlighter-rouge">offset = 3328</code> and <code class="language-plaintext highlighter-rouge">slice_length = 65</code> immediately triggers the prefetch. If you’re into this kind of shannonigans,
it shouldn’t take too long to notice that 3392 is the first byte into the 53rd cache line (counting zero-indexed) within
a page. Ok, does that mean accessing the 53rd cache line triggers the prefetch? But that can’t be true because we accessed
every single byte in the 53rd line and verified no prefetches (see “Right half” of the last row in the table). Moreover,
offset 3328 starts the 52nd cache line (counting zero-indexed) within a page, so we also know that accessing every byte
in the 52nd cache line also doesn’t trigger the prefetch (see the “Left half” of the last row in the table). Could it be
that for the prefetcher to trigger, subsequent loads must cross the 52nd cache line into the 53rd cache line? If the
prefetch condition truly is as simple as “two subsequent loads, one from cache line 52, the next from cache line 53”,
then the following must hold:</p>

<ol>
  <li>Touching every byte in a page except any from cache line 52 will not trigger prefetch.</li>
  <li>Touching every byte in a page except any from cache line 53 will not trigger prefetch.</li>
  <li>Two back-to-back loads, the first loading any byte in cache line 52 and the second loading any byte in cache line 53
will be sufficient to trigger prefetch.</li>
</ol>

<p>To test these hypotheses, we need a workload that touches every byte in a page except a few. This should work:</p>

<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="p">[[</span><span class="n">gnu</span><span class="o">::</span><span class="n">always_inline</span><span class="p">]]</span>
<span class="kr">inline</span> <span class="kt">void</span> <span class="nf">touch_page_with_hole</span><span class="p">(</span><span class="n">u64</span> <span class="n">hole_offset</span><span class="p">,</span> <span class="n">u64</span> <span class="n">hole_length</span><span class="p">,</span> <span class="kt">int</span> <span class="n">zero</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">u8</span> <span class="n">dummy</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
    <span class="n">u64</span> <span class="n">repeat_idx</span> <span class="o">=</span> <span class="n">hole_offset</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">?</span> <span class="mi">0</span> <span class="o">:</span> <span class="n">hole_offset</span> <span class="o">-</span> <span class="mi">1</span><span class="p">;</span>
    <span class="k">for</span> <span class="p">(</span><span class="n">u64</span> <span class="n">p</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">p</span> <span class="o">&lt;</span> <span class="n">N_MAPPED_PAGES</span><span class="p">;</span> <span class="n">p</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">u8</span><span class="o">*</span> <span class="n">page_base</span> <span class="o">=</span> <span class="p">(</span><span class="n">u8</span><span class="o">*</span><span class="p">)(</span><span class="n">MAP_BASE</span> <span class="o">+</span> <span class="n">p</span> <span class="o">*</span> <span class="n">PAGE_SIZE</span><span class="p">);</span>
        <span class="k">for</span> <span class="p">(</span><span class="n">u64</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">PAGE_SIZE</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
            <span class="n">u64</span> <span class="n">outside_hole</span> <span class="o">=</span> <span class="p">(</span><span class="n">i</span> <span class="o">-</span> <span class="n">hole_offset</span><span class="p">)</span> <span class="o">&gt;=</span> <span class="n">hole_length</span><span class="p">;</span>
            <span class="n">u64</span> <span class="n">idx</span> <span class="o">=</span> <span class="p">(</span><span class="n">i</span> <span class="o">&amp;</span> <span class="o">-</span><span class="n">outside_hole</span><span class="p">)</span> <span class="o">|</span> <span class="p">(</span><span class="n">repeat_idx</span> <span class="o">&amp;</span> <span class="p">(</span><span class="n">outside_hole</span> <span class="o">-</span> <span class="mi">1</span><span class="p">));</span>
            <span class="n">dummy</span> <span class="o">+=</span> <span class="n">page_base</span><span class="p">[</span><span class="n">idx</span> <span class="o">+</span> <span class="n">zero</span><span class="p">];</span>
            <span class="n">zero</span> <span class="o">&amp;=</span> <span class="n">dummy</span><span class="p">;</span>
        <span class="p">}</span>
    <span class="p">}</span>
    <span class="k">asm</span> <span class="nf">volatile</span><span class="p">(</span><span class="s">""</span> <span class="o">::</span> <span class="s">"r"</span><span class="p">(</span><span class="n">dummy</span><span class="p">));</span>
<span class="p">}</span></code></pre></figure>

<p>The strange looking trick to compute <code class="language-plaintext highlighter-rouge">idx</code> in the inner loop helps avoid a branch in the probe path. Yeah, we are also
banking on the fact that the branch predictor would do just fine when we’re done walking a page in the inner loop and
jump back to the outer loop. Strictly speaking, a branch mispredict shouldn’t mess with TLBs in a way that affects our
results but in general, eliminating noise sources to the best of your ability is a good idea with these kind of experiments.
Aside from that there’s not much going on here. We are skipping the range <code class="language-plaintext highlighter-rouge">page[hole_offset, hole_offset+hole_length]</code>
and touching every other byte in the page. When <code class="language-plaintext highlighter-rouge">i</code> falls within the skipped range, we simply redirect the load to
<code class="language-plaintext highlighter-rouge">hole_offset - 1</code>.</p>

<p>Ok so what are the results if we skip cache line 52 and 53, i.e. range <code class="language-plaintext highlighter-rouge">page[3328, 3391]</code> and <code class="language-plaintext highlighter-rouge">page[3392, 3456]</code>:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Skipping 52nd cache line (touch_page_with_hole(3328, 64, zero)):
DTLB_LOAD_MISSES.STLB_HIT           = 63
MEM_INST_RETIRED.STLB_MISS_LOADS    = 1000

Skipping 53rd cache line (touch_page_with_hole(3392, 64, zero)):
DTLB_LOAD_MISSES.STLB_HIT           = 66
MEM_INST_RETIRED.STLB_MISS_LOADS    = 1000
</code></pre></div></div>

<p><code class="language-plaintext highlighter-rouge">MEM_INST_RETIRED.STLB_MISS_LOADS = 1000</code> for both cases. This verifies (1) and (2) from our list of hypotheses. We can
test (3) by calling <code class="language-plaintext highlighter-rouge">touch_page_slice</code> with <code class="language-plaintext highlighter-rouge">offset = 3391</code> and <code class="language-plaintext highlighter-rouge">slice_length = 2</code>. Here are the results:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Load from cache line 52 and 53:
DTLB_LOAD_MISSES.STLB_HIT           = 4
MEM_INST_RETIRED.STLB_MISS_LOADS    = 3
</code></pre></div></div>

<p>As expected, merely touching a single byte in each of 52nd and 53rd cache lines engages the prefetcher. Now admittedly
the experiment doesn’t prove the “any byte in either cache line” part; we’ve only shown that loading last byte in cache
line 52 followed by the load of first byte in cache line 53 triggers the prefetch, but you can easily verify that by
tweaking the parameters of <code class="language-plaintext highlighter-rouge">touch_page_with_hole</code>. For example, <code class="language-plaintext highlighter-rouge">hole_offset = 3329</code> and <code class="language-plaintext highlighter-rouge">hole_length = 126</code> touches first
byte of line 52 and last byte of line 53, skipping everything in-between. This still triggers prefetch. To limit the size
of this post, I’ll leave out certain details like that from here on after. I’d be happy to fill those gaps in the comment
section if you have questions.</p>

<p>Note that for the prefetcher to trigger, load from 52nd cache line must execute before the load from 53rd cache line
(unless the prefetcher is bidirectional, but we’ll get to that in a minute). This is why it was crucial to introduce
the dependency chain that throttles out-of-order execution of loads from the page.</p>

<p>If you’re wondering why would Intel design the prefetcher with such a thin trigger condition, you’re not alone. I had the
same exact question. I mean, triggering prefetch when loads get closer and closer to the next page makes sense. But why
put all your bets on crossing just the 52nd cache line?! It’d make more sense if any couple of accesses beyond a threshold
triggered the prefetch. Peter Cordes (yeah, <a href="https://stackoverflow.com/users/peter-cordes">that Peter Cordes</a>) had a <a href="https://stackoverflow.com/a/80011111/14180973">brilliant answer</a>
on why this makes sense. Go read it if you’re interested.</p>

<h2 id="what-about-stores">What About Stores?</h2>

<p>Good question. Our experiments so far have only used loads. A sane person would think stores shouldn’t behave any different.
But to my surprise, stores don’t seem to trigger prefetches. If you’re browsing the <a href="https://gist.github.com/devsh0/66fea1019aee8eda54925ee75b73852a">full source</a> on GitHub you’d
notice that we are faulting the mapped range with loads. That choice is deliberate: we’re intentionally mapping every
virtual page in the mapped range to the physical zero page. This reduces cache footprint, and potentially restricts one
more source of noise.</p>

<p>We cannot keep doing this for this next experiment, though; we don’t want page faults in the probe path. So we switch to
faulting with stores before we start the experiment. That out of the way, here’s the kernel we’ll use to verify no prefetch
on stores:</p>

<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="p">[[</span><span class="n">gnu</span><span class="o">::</span><span class="n">always_inline</span><span class="p">]]</span>
<span class="kr">inline</span> <span class="kt">void</span> <span class="nf">touch_with_stores</span><span class="p">(</span><span class="kt">int</span> <span class="n">zero</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">u8</span><span class="o">*</span> <span class="n">map_base</span> <span class="o">=</span> <span class="p">(</span><span class="n">u8</span><span class="o">*</span><span class="p">)</span><span class="n">MAP_BASE</span><span class="p">;</span>
    <span class="k">for</span> <span class="p">(</span><span class="n">u64</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">MAP_SIZE</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">map_base</span><span class="p">[</span><span class="n">i</span> <span class="o">+</span> <span class="n">zero</span><span class="p">]</span> <span class="o">=</span> <span class="n">i</span><span class="p">;</span>
        <span class="n">zero</span> <span class="o">*=</span> <span class="n">zero</span><span class="p">;</span>
    <span class="p">}</span>
<span class="p">}</span></code></pre></figure>

<p>Running this produces <code class="language-plaintext highlighter-rouge">MEM_INST_RETIRED.STLB_MISS_STORES = 1000</code>. Note that this time we are measuring STLB misses on
stores. You can verify that each of these misses start at least one page-walk by counting <code class="language-plaintext highlighter-rouge">DTLB_STORE_MISSES.WALK_COMPLETED</code>
along with <code class="language-plaintext highlighter-rouge">MEM_INST_RETIRED.STLB_MISS_STORES</code>. That rules out the possibility of dTLB hits and a last-level TLB (other than
STLB) catching all the misses from STLB.</p>

<h2 id="how-many-ptes-are-prefetched">How Many PTEs are Prefetched</h2>

<p>We have established that the next-page prefetcher prefetches at least one future PTE. Let’s figure out how many future
PTEs are prefetched <em>exactly</em>. We’ll use the following workload for this:</p>

<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="kt">void</span> <span class="nf">how_many_pages_prefetched</span><span class="p">(</span><span class="kt">int</span> <span class="n">future_limit</span><span class="p">,</span> <span class="kt">int</span> <span class="n">zero</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">u8</span> <span class="n">dummy</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
    <span class="kt">int</span> <span class="n">poke_limit</span> <span class="o">=</span> <span class="mi">8</span><span class="p">;</span>
    <span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">p</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">p</span> <span class="o">&lt;</span> <span class="n">poke_limit</span><span class="p">;</span> <span class="n">p</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">u8</span><span class="o">*</span> <span class="n">page_base</span> <span class="o">=</span> <span class="p">(</span><span class="n">u8</span><span class="o">*</span><span class="p">)(</span><span class="n">MAP_BASE</span> <span class="o">+</span> <span class="n">p</span> <span class="o">*</span> <span class="n">PAGE_SIZE</span><span class="p">);</span>
        <span class="n">dummy</span> <span class="o">+=</span> <span class="n">page_base</span><span class="p">[</span><span class="mi">3391</span> <span class="o">+</span> <span class="n">zero</span><span class="p">];</span>
        <span class="n">zero</span> <span class="o">&amp;=</span> <span class="n">dummy</span><span class="p">;</span>
        <span class="n">dummy</span> <span class="o">+=</span> <span class="n">page_base</span><span class="p">[</span><span class="mi">3392</span> <span class="o">+</span> <span class="n">zero</span><span class="p">];</span>
        <span class="n">zero</span> <span class="o">&amp;=</span> <span class="n">dummy</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="n">future_limit</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">u8</span><span class="o">*</span> <span class="n">page_base</span> <span class="o">=</span> <span class="p">(</span><span class="n">u8</span><span class="o">*</span><span class="p">)(</span><span class="n">MAP_BASE</span> <span class="o">+</span> <span class="p">(</span><span class="n">i</span> <span class="o">+</span> <span class="n">poke_limit</span><span class="p">)</span> <span class="o">*</span> <span class="n">PAGE_SIZE</span><span class="p">);</span>
        <span class="n">dummy</span> <span class="o">+=</span> <span class="n">page_base</span><span class="p">[</span><span class="n">zero</span><span class="p">];</span>
        <span class="n">zero</span> <span class="o">&amp;=</span> <span class="n">dummy</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="k">asm</span> <span class="nf">volatile</span><span class="p">(</span><span class="s">""</span> <span class="o">::</span> <span class="s">"r"</span><span class="p">(</span><span class="n">dummy</span><span class="p">));</span>
<span class="p">}</span></code></pre></figure>

<p>The idea is to “heat up” the prefetcher by crossing 52nd cache line in the first <code class="language-plaintext highlighter-rouge">poke_limit</code> pages — that’s what the first
loop does — then load first byte of <code class="language-plaintext highlighter-rouge">future_limit</code> pages beyond <code class="language-plaintext highlighter-rouge">poke_limit</code> in the second loop. We’ll call this function
repeatedly, each time increasing <code class="language-plaintext highlighter-rouge">future_limit</code> by 1, and count the TLB misses. We can get a baseline with <code class="language-plaintext highlighter-rouge">future_limit = 0</code>
first:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Run 1:
DTLB_LOAD_MISSES.STLB_HIT           = 1
MEM_INST_RETIRED.STLB_MISS_LOADS    = 1

Run 2:
DTLB_LOAD_MISSES.STLB_HIT           = 2
MEM_INST_RETIRED.STLB_MISS_LOADS    = 1

Run 3:
DTLB_LOAD_MISSES.STLB_HIT           = 0
MEM_INST_RETIRED.STLB_MISS_LOADS    = 3

...
</code></pre></div></div>

<p>Over many runs, STLB miss baseline fluctuates between 1 and 3 but never seems to exceed 3. Alright, we can work with that.
Now let’s try <code class="language-plaintext highlighter-rouge">future_limit = 1</code>:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Run 1:
DTLB_LOAD_MISSES.STLB_HIT           = 0
MEM_INST_RETIRED.STLB_MISS_LOADS    = 3

Run 2:
DTLB_LOAD_MISSES.STLB_HIT           = 1
MEM_INST_RETIRED.STLB_MISS_LOADS    = 2

Run 3:
DTLB_LOAD_MISSES.STLB_HIT           = 1
MEM_INST_RETIRED.STLB_MISS_LOADS    = 2

...
</code></pre></div></div>

<p>Yep, expected! The last load in the “heater” must’ve triggered the prefetch for page at <code class="language-plaintext highlighter-rouge">poke_limit</code>, so <code class="language-plaintext highlighter-rouge">future_limit = 1</code>
shouldn’t increase the number of observed misses. What about <code class="language-plaintext highlighter-rouge">future_limit = 2</code>?</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Run 1:
DTLB_LOAD_MISSES.STLB_HIT           = 0
MEM_INST_RETIRED.STLB_MISS_LOADS    = 4

Run 2:
DTLB_LOAD_MISSES.STLB_HIT           = 0
MEM_INST_RETIRED.STLB_MISS_LOADS    = 5

Run 3:
DTLB_LOAD_MISSES.STLB_HIT           = 1
MEM_INST_RETIRED.STLB_MISS_LOADS    = 4

...
</code></pre></div></div>

<p>Looks like the NPP prefetches only one page ahead. Cool, no complaints there!</p>

<h2 id="prefetch-direction">Prefetch Direction</h2>

<p>Does the prefetcher work in backward direction too? This is the last piece of the puzzle and fortunately very easy to test:
just walk the mapped range backwards. If you get close to 1000 misses the answer is no, otherwise yes. Here’s the workload:</p>

<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="p">[[</span><span class="n">gnu</span><span class="o">::</span><span class="n">always_inline</span><span class="p">]]</span>
<span class="kr">inline</span> <span class="kt">void</span> <span class="nf">touch_bytes_backward</span><span class="p">(</span><span class="kt">int</span> <span class="n">zero</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">u8</span><span class="o">*</span> <span class="n">map_base</span> <span class="o">=</span> <span class="p">(</span><span class="n">u8</span><span class="o">*</span><span class="p">)</span><span class="n">MAP_BASE</span><span class="p">;</span>
    <span class="n">u8</span> <span class="n">dummy</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
    <span class="k">for</span> <span class="p">(</span><span class="n">u64</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="n">i</span> <span class="o">&lt;=</span> <span class="n">MAP_SIZE</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
        <span class="n">dummy</span> <span class="o">+=</span> <span class="n">map_base</span><span class="p">[</span><span class="n">MAP_SIZE</span> <span class="o">-</span> <span class="n">i</span> <span class="o">+</span> <span class="n">zero</span><span class="p">];</span>
        <span class="n">zero</span> <span class="o">&amp;=</span> <span class="n">dummy</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="k">asm</span> <span class="nf">volatile</span><span class="p">(</span><span class="s">""</span> <span class="o">::</span> <span class="s">"r"</span><span class="p">(</span><span class="n">dummy</span><span class="p">));</span>
<span class="p">}</span></code></pre></figure>

<p>…and the results:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DTLB_LOAD_MISSES.STLB_HIT               = 69
MEM_INST_RETIRED.STLB_MISS_LOADS        = 1000
</code></pre></div></div>

<p>This is a bit shocking. If you recall from the “Introduction” section, Intel manuals seem to suggest the next-page prefetcher
triggers while walking “upwards” (what I call forward) or “downward” (what I call backward). This behavior is documented
for both Ivy Bridge and Redwood Cove…but that’s not what we see for SKX. I don’t have any theories explaining this disconnect
between SKX and the other uarchs, if you do feel free to share in the comment section.</p>

<h2 id="so-where-can-i-use-this-knowledge">So Where Can I Use This Knowledge</h2>

<p>If you’re a software developer looking for a cool new way of optimizing program performance, probably nowhere. Optimizing
programs for a single microarchitecture from a single CPU vendor is almost always a bad idea for anything with half-life
beyond next Friday. My goal here is to share how I attempt reversing undocumented machine behavior and invite fellow nerds
to kick-start discussions that hopefully help me or somebody else learn something new.</p>

<p>That said, if you’re one of those weirdos who can manage to care only about one type of machine (Skylake-X to be precise),
you may find the details interesting from a practical PoV. Either way, I’m tired. This is too much writing for a first
post. 👋</p>]]></content><author><name></name></author><category term="microarchitecture," /><category term="intel," /><category term="computer-architecture" /><summary type="html"><![CDATA[Introduction]]></summary></entry></feed>