Skip to content

Tweak the number of instructions on the rseq fast path. - #578

Draft
copybara-service[bot] wants to merge 1 commit into
masterfrom
test_967818468
Draft

Tweak the number of instructions on the rseq fast path.#578
copybara-service[bot] wants to merge 1 commit into
masterfrom
test_967818468

Conversation

@copybara-service

Copy link
Copy Markdown

Tweak the number of instructions on the rseq fast path.

Instead of storing an end index for a given sizeclass freelist,
the header now stores how many further elements there is room
for to push. This is equivalent (obviously,
current + remaining_elements == end), but it allows us to
compare (remaining_elements == 0) instead of (current == end),
which means we can use the CPU's carry flag to get the compare
for free.

What we pay for this is that we need to update remaining_elements
both on push and pop, whereas earlier, we'd only need to update
current. For push, this costs us nothing; we can just load
the two values together in a 32-bit load (saving the separate
load of end for the compare) and do current++, remaining_elements--;
in the one arithmetic operation. (We still need to extract the
low 16 bits of this word into a separate register for indexing
purposes.)

However, for pop, we pay by having to use a 32-bit rmw addition
where we earlier just had a decrement (of the current index)
and a 16-bit store. (The alternative, keeping the 32-bit value
in a separate register, caused spills on some of the slower paths
and benchmarked really poorly on Milan for whatever reason.
Skylake was fine, as was Zen 2.) Overall this seems still be a win.
Microbenchmarks on Milan:

name cpu/op cpu/op vs base
BM_new_delete/1 5.077n ± 0% 4.016n ± 0% -20.91% (n=50)
BM_new_delete/8 5.077n ± 0% 4.022n ± 0% -20.78% (n=50)
BM_new_delete/64 4.990n ± 0% 4.052n ± 0% -18.79% (n=50)
BM_new_sized_delete/1 4.889n ± 0% 4.261n ± 6% -12.85% (n=50)
BM_new_sized_delete/8 4.892n ± 0% 4.263n ± 6% -12.86% (n=50)
BM_new_sized_delete/64 4.797n ± 0% 4.280n ± 6% -10.78% (n=50)

We update Aarch64 assembly accordingly, but it is not the main
focus of this patch.

Instead of storing an end index for a given sizeclass freelist,
the header now stores how many further elements there is room
for to push. This is equivalent (obviously,
current + remaining_elements == end), but it allows us to
compare (remaining_elements == 0) instead of (current == end),
which means we can use the CPU's carry flag to get the compare
for free.

What we pay for this is that we need to update remaining_elements
both on push and pop, whereas earlier, we'd only need to update
current. For push, this costs us nothing; we can just load
the two values together in a 32-bit load (saving the separate
load of end for the compare) and do current++, remaining_elements--;
in the one arithmetic operation. (We still need to extract the
low 16 bits of this word into a separate register for indexing
purposes.)

However, for pop, we pay by having to use a 32-bit rmw addition
where we earlier just had a decrement (of the current index)
and a 16-bit store. (The alternative, keeping the 32-bit value
in a separate register, caused spills on some of the slower paths
and benchmarked really poorly on Milan for whatever reason.
Skylake was fine, as was Zen 2.) Overall this seems still be a win.
Microbenchmarks on Milan:

  name                           cpu/op         cpu/op      vs base
  BM_new_delete/1                5.077n ±  0%    4.016n ±   0%   -20.91% (n=50)
  BM_new_delete/8                5.077n ±  0%    4.022n ±   0%   -20.78% (n=50)
  BM_new_delete/64               4.990n ±  0%    4.052n ±   0%   -18.79% (n=50)
  BM_new_sized_delete/1          4.889n ±  0%    4.261n ±   6%   -12.85% (n=50)
  BM_new_sized_delete/8          4.892n ±  0%    4.263n ±   6%   -12.86% (n=50)
  BM_new_sized_delete/64         4.797n ±  0%    4.280n ±   6%   -10.78% (n=50)

We update Aarch64 assembly accordingly, but it is not the main
focus of this patch.

PiperOrigin-RevId: 967818468
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant