Stowage/go
2
0
Fork 0
mirror of https://github.com/golang/go.git synced 2025-12-08 06:10:04 +00:00
Code Issues Projects Releases Packages Wiki Activity
go/test/codegen/copy.go

125 lines
2.6 KiB
Go
Raw Normal View History

cmd/compile/internal/ssa: inline small memmove for arm64 This patch enables the optimization for arm64 target. Performance results on Amberwing for strconv benchmark: name old time/op new time/op delta Quote 721ns ± 0% 617ns ± 0% -14.40% (p=0.016 n=5+4) QuoteRune 118ns ± 0% 117ns ± 0% -0.85% (p=0.008 n=5+5) AppendQuote 436ns ± 2% 321ns ± 0% -26.31% (p=0.008 n=5+5) AppendQuoteRune 34.7ns ± 0% 28.4ns ± 0% -18.16% (p=0.000 n=5+4) [Geo mean] 189ns 160ns -15.41% Change-Id: I5714c474e7483d07ca338fbaf49beb4bbcc11c44 Reviewed-on: https://go-review.googlesource.com/98735 Run-TryBot: Cherry Zhang <cherryyz@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Giovanni Bajo <rasky@develer.com> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2018-03-05 15:51:54 -05:00
// asmcheck
// Copyright 2018 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package codegen
cmd/compile: convert memmove call into Move when arguments are disjoint Move ops can be faster than memmove calls because the number of bytes to be moved is fixed and they don't incur the overhead of a call. This change allows memmove to be converted into a Move op when the arguments are disjoint. The optimization is only enabled on s390x at the moment, however other architectures may also benefit from it in the future. The memmove inlining rule triggers an extra 12 times when compiling the standard library. It will most likely make more of a difference as the disjoint function is improved over time (to recognize fresh heap allocations for example). Change-Id: I9af570dcfff28257b8e59e0ff584a46d8e248310 Reviewed-on: https://go-review.googlesource.com/110064 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ilya Tocar <ilya.tocar@intel.com>
2018-04-29 15:12:50 +01:00
import "runtime"
cmd/compile: generate constants for NeqPtr, EqPtr and IsNonNil ops If both inputs are constant offsets from the same pointer then we can evaluate NeqPtr and EqPtr at compile time. Triggers a few times during all.bash. Removes a conditional branch in the following code: copy(x[1:], x[:]) This branch was recently added as an optimization in CL 94596. We now skip the memmove if the pointers are equal. However, in the above code we know at compile time that they are never equal. Also, when the offset is variable, check if the offset is zero rather than if the pointers are equal. For example: copy(x[a:], x[:]) This would now skip the copy if a == 0, rather than if x + a == x. Finally I've also added a rule to make IsNonNil true for pointers to values on the stack. The nil check elimination pass will catch these anyway, but eliminating them here might eliminate branches earlier. Change-Id: If72f436fef0a96ad0f4e296d3a1f8b6c3e712085 Reviewed-on: https://go-review.googlesource.com/106635 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2018-04-12 15:24:08 +01:00
// Check small copies are replaced with moves.
test/codegen: port all small memmove tests to codegen This change ports all the remaining tests checking that small memmoves are replaced with MOVs to the new codegen test harness, and deletes them from the asm_test file. Change-Id: I01c94b441e27a5d61518035af62d62779dafeb56 Reviewed-on: https://go-review.googlesource.com/100476 Run-TryBot: Alberto Donizetti <alb.donizetti@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2018-03-14 11:07:06 +01:00
func movesmall4() {
x := [...]byte{1, 2, 3, 4}
// 386:-".*memmove"
// amd64:-".*memmove"
// arm:-".*memmove"
// arm64:-".*memmove"
test/codegen: enable more tests for ppc64/ppc64le Adding cases for ppc64,ppc64le to the codegen tests where appropriate. Change-Id: Idf8cbe88a4ab4406a4ef1ea777bd15a58b68f3ed Reviewed-on: https://go-review.googlesource.com/c/142557 Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-15 12:53:07 -04:00
// ppc64:-".*memmove"
// ppc64le:-".*memmove"
test/codegen: port all small memmove tests to codegen This change ports all the remaining tests checking that small memmoves are replaced with MOVs to the new codegen test harness, and deletes them from the asm_test file. Change-Id: I01c94b441e27a5d61518035af62d62779dafeb56 Reviewed-on: https://go-review.googlesource.com/100476 Run-TryBot: Alberto Donizetti <alb.donizetti@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2018-03-14 11:07:06 +01:00
copy(x[1:], x[:])
}
func movesmall7() {
cmd/compile/internal/ssa: inline small memmove for arm64 This patch enables the optimization for arm64 target. Performance results on Amberwing for strconv benchmark: name old time/op new time/op delta Quote 721ns ± 0% 617ns ± 0% -14.40% (p=0.016 n=5+4) QuoteRune 118ns ± 0% 117ns ± 0% -0.85% (p=0.008 n=5+5) AppendQuote 436ns ± 2% 321ns ± 0% -26.31% (p=0.008 n=5+5) AppendQuoteRune 34.7ns ± 0% 28.4ns ± 0% -18.16% (p=0.000 n=5+4) [Geo mean] 189ns 160ns -15.41% Change-Id: I5714c474e7483d07ca338fbaf49beb4bbcc11c44 Reviewed-on: https://go-review.googlesource.com/98735 Run-TryBot: Cherry Zhang <cherryyz@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Giovanni Bajo <rasky@develer.com> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2018-03-05 15:51:54 -05:00
x := [...]byte{1, 2, 3, 4, 5, 6, 7}
test/codegen: port all small memmove tests to codegen This change ports all the remaining tests checking that small memmoves are replaced with MOVs to the new codegen test harness, and deletes them from the asm_test file. Change-Id: I01c94b441e27a5d61518035af62d62779dafeb56 Reviewed-on: https://go-review.googlesource.com/100476 Run-TryBot: Alberto Donizetti <alb.donizetti@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2018-03-14 11:07:06 +01:00
// 386:-".*memmove"
// amd64:-".*memmove"
// arm64:-".*memmove"
test/codegen: enable more tests for ppc64/ppc64le Adding cases for ppc64,ppc64le to the codegen tests where appropriate. Change-Id: Idf8cbe88a4ab4406a4ef1ea777bd15a58b68f3ed Reviewed-on: https://go-review.googlesource.com/c/142557 Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-15 12:53:07 -04:00
// ppc64:-".*memmove"
// ppc64le:-".*memmove"
test/codegen: port all small memmove tests to codegen This change ports all the remaining tests checking that small memmoves are replaced with MOVs to the new codegen test harness, and deletes them from the asm_test file. Change-Id: I01c94b441e27a5d61518035af62d62779dafeb56 Reviewed-on: https://go-review.googlesource.com/100476 Run-TryBot: Alberto Donizetti <alb.donizetti@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2018-03-14 11:07:06 +01:00
copy(x[1:], x[:])
}
func movesmall16() {
x := [...]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16}
// amd64:-".*memmove"
cmd/compile: improve lowered moves and zeros for ppc64le This change includes the following: - Generate LXV/STXV sequences instead of LXVD2X/STXVD2X on power9. These instructions do not require an index register, which allows more loads and stores within a loop without initializing multiple index registers. The LoweredQuadXXX generate LXV/STXV. - Create LoweredMoveXXXShort and LoweredZeroXXXShort for short moves that don't generate loops, and therefore don't clobber the address registers or flags. - Use registers other than R3 and R4 to avoid conflicting with registers that have already been allocated to avoid unnecessary register moves. - Eliminate the use of R14 as scratch register and use R31 instead. - Add PCALIGN when the LoweredMoveXXX or LoweredZeroXXX generates a loop with more than 3 iterations. This performance opportunity was noticed in github.com/golang/snappy benchmarks. Results on power9: WordsDecode1e1 54.1ns ± 0% 53.8ns ± 0% -0.51% (p=0.029 n=4+4) WordsDecode1e2 287ns ± 0% 282ns ± 1% -1.83% (p=0.029 n=4+4) WordsDecode1e3 3.98µs ± 0% 3.64µs ± 0% -8.52% (p=0.029 n=4+4) WordsDecode1e4 66.9µs ± 0% 67.0µs ± 0% +0.20% (p=0.029 n=4+4) WordsDecode1e5 723µs ± 0% 723µs ± 0% -0.01% (p=0.200 n=4+4) WordsDecode1e6 7.21ms ± 0% 7.21ms ± 0% -0.02% (p=1.000 n=4+4) WordsEncode1e1 29.9ns ± 0% 29.4ns ± 0% -1.51% (p=0.029 n=4+4) WordsEncode1e2 2.12µs ± 0% 1.75µs ± 0% -17.70% (p=0.029 n=4+4) WordsEncode1e3 11.7µs ± 0% 11.2µs ± 0% -4.61% (p=0.029 n=4+4) WordsEncode1e4 119µs ± 0% 120µs ± 0% +0.36% (p=0.029 n=4+4) WordsEncode1e5 1.21ms ± 0% 1.22ms ± 0% +0.41% (p=0.029 n=4+4) WordsEncode1e6 12.0ms ± 0% 12.0ms ± 0% +0.57% (p=0.029 n=4+4) RandomEncode 286µs ± 0% 203µs ± 0% -28.82% (p=0.029 n=4+4) ExtendMatch 47.4µs ± 0% 47.0µs ± 0% -0.85% (p=0.029 n=4+4) Change-Id: Iecad3a39ae55280286e42760a5c9d5c1168f5858 Reviewed-on: https://go-review.googlesource.com/c/go/+/226539 Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-03-30 15:23:19 -04:00
// ppc64:".*memmove"
// ppc64le:".*memmove"
test/codegen: port all small memmove tests to codegen This change ports all the remaining tests checking that small memmoves are replaced with MOVs to the new codegen test harness, and deletes them from the asm_test file. Change-Id: I01c94b441e27a5d61518035af62d62779dafeb56 Reviewed-on: https://go-review.googlesource.com/100476 Run-TryBot: Alberto Donizetti <alb.donizetti@gmail.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2018-03-14 11:07:06 +01:00
copy(x[1:], x[:])
cmd/compile/internal/ssa: inline small memmove for arm64 This patch enables the optimization for arm64 target. Performance results on Amberwing for strconv benchmark: name old time/op new time/op delta Quote 721ns ± 0% 617ns ± 0% -14.40% (p=0.016 n=5+4) QuoteRune 118ns ± 0% 117ns ± 0% -0.85% (p=0.008 n=5+5) AppendQuote 436ns ± 2% 321ns ± 0% -26.31% (p=0.008 n=5+5) AppendQuoteRune 34.7ns ± 0% 28.4ns ± 0% -18.16% (p=0.000 n=5+4) [Geo mean] 189ns 160ns -15.41% Change-Id: I5714c474e7483d07ca338fbaf49beb4bbcc11c44 Reviewed-on: https://go-review.googlesource.com/98735 Run-TryBot: Cherry Zhang <cherryyz@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Giovanni Bajo <rasky@develer.com> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2018-03-05 15:51:54 -05:00
}
cmd/compile: generate constants for NeqPtr, EqPtr and IsNonNil ops If both inputs are constant offsets from the same pointer then we can evaluate NeqPtr and EqPtr at compile time. Triggers a few times during all.bash. Removes a conditional branch in the following code: copy(x[1:], x[:]) This branch was recently added as an optimization in CL 94596. We now skip the memmove if the pointers are equal. However, in the above code we know at compile time that they are never equal. Also, when the offset is variable, check if the offset is zero rather than if the pointers are equal. For example: copy(x[a:], x[:]) This would now skip the copy if a == 0, rather than if x + a == x. Finally I've also added a rule to make IsNonNil true for pointers to values on the stack. The nil check elimination pass will catch these anyway, but eliminating them here might eliminate branches earlier. Change-Id: If72f436fef0a96ad0f4e296d3a1f8b6c3e712085 Reviewed-on: https://go-review.googlesource.com/106635 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2018-04-12 15:24:08 +01:00
var x [256]byte
cmd/compile: convert memmove call into Move when arguments are disjoint Move ops can be faster than memmove calls because the number of bytes to be moved is fixed and they don't incur the overhead of a call. This change allows memmove to be converted into a Move op when the arguments are disjoint. The optimization is only enabled on s390x at the moment, however other architectures may also benefit from it in the future. The memmove inlining rule triggers an extra 12 times when compiling the standard library. It will most likely make more of a difference as the disjoint function is improved over time (to recognize fresh heap allocations for example). Change-Id: I9af570dcfff28257b8e59e0ff584a46d8e248310 Reviewed-on: https://go-review.googlesource.com/110064 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ilya Tocar <ilya.tocar@intel.com>
2018-04-29 15:12:50 +01:00
// Check that large disjoint copies are replaced with moves.
cmd/compile: improve lowered moves and zeros for ppc64le This change includes the following: - Generate LXV/STXV sequences instead of LXVD2X/STXVD2X on power9. These instructions do not require an index register, which allows more loads and stores within a loop without initializing multiple index registers. The LoweredQuadXXX generate LXV/STXV. - Create LoweredMoveXXXShort and LoweredZeroXXXShort for short moves that don't generate loops, and therefore don't clobber the address registers or flags. - Use registers other than R3 and R4 to avoid conflicting with registers that have already been allocated to avoid unnecessary register moves. - Eliminate the use of R14 as scratch register and use R31 instead. - Add PCALIGN when the LoweredMoveXXX or LoweredZeroXXX generates a loop with more than 3 iterations. This performance opportunity was noticed in github.com/golang/snappy benchmarks. Results on power9: WordsDecode1e1 54.1ns ± 0% 53.8ns ± 0% -0.51% (p=0.029 n=4+4) WordsDecode1e2 287ns ± 0% 282ns ± 1% -1.83% (p=0.029 n=4+4) WordsDecode1e3 3.98µs ± 0% 3.64µs ± 0% -8.52% (p=0.029 n=4+4) WordsDecode1e4 66.9µs ± 0% 67.0µs ± 0% +0.20% (p=0.029 n=4+4) WordsDecode1e5 723µs ± 0% 723µs ± 0% -0.01% (p=0.200 n=4+4) WordsDecode1e6 7.21ms ± 0% 7.21ms ± 0% -0.02% (p=1.000 n=4+4) WordsEncode1e1 29.9ns ± 0% 29.4ns ± 0% -1.51% (p=0.029 n=4+4) WordsEncode1e2 2.12µs ± 0% 1.75µs ± 0% -17.70% (p=0.029 n=4+4) WordsEncode1e3 11.7µs ± 0% 11.2µs ± 0% -4.61% (p=0.029 n=4+4) WordsEncode1e4 119µs ± 0% 120µs ± 0% +0.36% (p=0.029 n=4+4) WordsEncode1e5 1.21ms ± 0% 1.22ms ± 0% +0.41% (p=0.029 n=4+4) WordsEncode1e6 12.0ms ± 0% 12.0ms ± 0% +0.57% (p=0.029 n=4+4) RandomEncode 286µs ± 0% 203µs ± 0% -28.82% (p=0.029 n=4+4) ExtendMatch 47.4µs ± 0% 47.0µs ± 0% -0.85% (p=0.029 n=4+4) Change-Id: Iecad3a39ae55280286e42760a5c9d5c1168f5858 Reviewed-on: https://go-review.googlesource.com/c/go/+/226539 Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-03-30 15:23:19 -04:00
func moveDisjointStack32() {
var s [32]byte
// ppc64:-".*memmove"
// ppc64le:-".*memmove"
// ppc64le/power8:"LXVD2X",-"ADD",-"BC"
// ppc64le/power9:"LXV",-"LXVD2X",-"ADD",-"BC"
copy(s[:], x[:32])
runtime.KeepAlive(&s)
}
func moveDisjointStack64() {
var s [96]byte
// ppc64:-".*memmove"
// ppc64le:-".*memmove"
// ppc64le/power8:"LXVD2X","ADD","BC"
// ppc64le/power9:"LXV",-"LXVD2X",-"ADD",-"BC"
copy(s[:], x[:96])
runtime.KeepAlive(&s)
}
cmd/compile: convert memmove call into Move when arguments are disjoint Move ops can be faster than memmove calls because the number of bytes to be moved is fixed and they don't incur the overhead of a call. This change allows memmove to be converted into a Move op when the arguments are disjoint. The optimization is only enabled on s390x at the moment, however other architectures may also benefit from it in the future. The memmove inlining rule triggers an extra 12 times when compiling the standard library. It will most likely make more of a difference as the disjoint function is improved over time (to recognize fresh heap allocations for example). Change-Id: I9af570dcfff28257b8e59e0ff584a46d8e248310 Reviewed-on: https://go-review.googlesource.com/110064 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ilya Tocar <ilya.tocar@intel.com>
2018-04-29 15:12:50 +01:00
func moveDisjointStack() {
var s [256]byte
// s390x:-".*memmove"
cmd/compile: enable disjoint memmove inlining on amd64 Memmove can use AVX/prefetches/other optional instructions, so only do it for small sizes, when call overhead dominates. Change-Id: Ice5e93deb11462217f7fb5fc350b703109bb4090 Reviewed-on: https://go-review.googlesource.com/112517 Run-TryBot: Ilya Tocar <ilya.tocar@intel.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Michael Munday <mike.munday@ibm.com>
2018-05-09 15:49:22 -05:00
// amd64:-".*memmove"
cmd/compile: improve lowered moves and zeros for ppc64le This change includes the following: - Generate LXV/STXV sequences instead of LXVD2X/STXVD2X on power9. These instructions do not require an index register, which allows more loads and stores within a loop without initializing multiple index registers. The LoweredQuadXXX generate LXV/STXV. - Create LoweredMoveXXXShort and LoweredZeroXXXShort for short moves that don't generate loops, and therefore don't clobber the address registers or flags. - Use registers other than R3 and R4 to avoid conflicting with registers that have already been allocated to avoid unnecessary register moves. - Eliminate the use of R14 as scratch register and use R31 instead. - Add PCALIGN when the LoweredMoveXXX or LoweredZeroXXX generates a loop with more than 3 iterations. This performance opportunity was noticed in github.com/golang/snappy benchmarks. Results on power9: WordsDecode1e1 54.1ns ± 0% 53.8ns ± 0% -0.51% (p=0.029 n=4+4) WordsDecode1e2 287ns ± 0% 282ns ± 1% -1.83% (p=0.029 n=4+4) WordsDecode1e3 3.98µs ± 0% 3.64µs ± 0% -8.52% (p=0.029 n=4+4) WordsDecode1e4 66.9µs ± 0% 67.0µs ± 0% +0.20% (p=0.029 n=4+4) WordsDecode1e5 723µs ± 0% 723µs ± 0% -0.01% (p=0.200 n=4+4) WordsDecode1e6 7.21ms ± 0% 7.21ms ± 0% -0.02% (p=1.000 n=4+4) WordsEncode1e1 29.9ns ± 0% 29.4ns ± 0% -1.51% (p=0.029 n=4+4) WordsEncode1e2 2.12µs ± 0% 1.75µs ± 0% -17.70% (p=0.029 n=4+4) WordsEncode1e3 11.7µs ± 0% 11.2µs ± 0% -4.61% (p=0.029 n=4+4) WordsEncode1e4 119µs ± 0% 120µs ± 0% +0.36% (p=0.029 n=4+4) WordsEncode1e5 1.21ms ± 0% 1.22ms ± 0% +0.41% (p=0.029 n=4+4) WordsEncode1e6 12.0ms ± 0% 12.0ms ± 0% +0.57% (p=0.029 n=4+4) RandomEncode 286µs ± 0% 203µs ± 0% -28.82% (p=0.029 n=4+4) ExtendMatch 47.4µs ± 0% 47.0µs ± 0% -0.85% (p=0.029 n=4+4) Change-Id: Iecad3a39ae55280286e42760a5c9d5c1168f5858 Reviewed-on: https://go-review.googlesource.com/c/go/+/226539 Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-03-30 15:23:19 -04:00
// ppc64:-".*memmove"
// ppc64le:-".*memmove"
// ppc64le/power8:"LXVD2X"
// ppc64le/power9:"LXV",-"LXVD2X"
cmd/compile: convert memmove call into Move when arguments are disjoint Move ops can be faster than memmove calls because the number of bytes to be moved is fixed and they don't incur the overhead of a call. This change allows memmove to be converted into a Move op when the arguments are disjoint. The optimization is only enabled on s390x at the moment, however other architectures may also benefit from it in the future. The memmove inlining rule triggers an extra 12 times when compiling the standard library. It will most likely make more of a difference as the disjoint function is improved over time (to recognize fresh heap allocations for example). Change-Id: I9af570dcfff28257b8e59e0ff584a46d8e248310 Reviewed-on: https://go-review.googlesource.com/110064 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ilya Tocar <ilya.tocar@intel.com>
2018-04-29 15:12:50 +01:00
copy(s[:], x[:])
runtime.KeepAlive(&s)
}
cmd/compile: enable disjoint memmove inlining on amd64 Memmove can use AVX/prefetches/other optional instructions, so only do it for small sizes, when call overhead dominates. Change-Id: Ice5e93deb11462217f7fb5fc350b703109bb4090 Reviewed-on: https://go-review.googlesource.com/112517 Run-TryBot: Ilya Tocar <ilya.tocar@intel.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Michael Munday <mike.munday@ibm.com>
2018-05-09 15:49:22 -05:00
func moveDisjointArg(b *[256]byte) {
cmd/compile: convert memmove call into Move when arguments are disjoint Move ops can be faster than memmove calls because the number of bytes to be moved is fixed and they don't incur the overhead of a call. This change allows memmove to be converted into a Move op when the arguments are disjoint. The optimization is only enabled on s390x at the moment, however other architectures may also benefit from it in the future. The memmove inlining rule triggers an extra 12 times when compiling the standard library. It will most likely make more of a difference as the disjoint function is improved over time (to recognize fresh heap allocations for example). Change-Id: I9af570dcfff28257b8e59e0ff584a46d8e248310 Reviewed-on: https://go-review.googlesource.com/110064 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ilya Tocar <ilya.tocar@intel.com>
2018-04-29 15:12:50 +01:00
var s [256]byte
// s390x:-".*memmove"
cmd/compile: enable disjoint memmove inlining on amd64 Memmove can use AVX/prefetches/other optional instructions, so only do it for small sizes, when call overhead dominates. Change-Id: Ice5e93deb11462217f7fb5fc350b703109bb4090 Reviewed-on: https://go-review.googlesource.com/112517 Run-TryBot: Ilya Tocar <ilya.tocar@intel.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Michael Munday <mike.munday@ibm.com>
2018-05-09 15:49:22 -05:00
// amd64:-".*memmove"
cmd/compile: improve lowered moves and zeros for ppc64le This change includes the following: - Generate LXV/STXV sequences instead of LXVD2X/STXVD2X on power9. These instructions do not require an index register, which allows more loads and stores within a loop without initializing multiple index registers. The LoweredQuadXXX generate LXV/STXV. - Create LoweredMoveXXXShort and LoweredZeroXXXShort for short moves that don't generate loops, and therefore don't clobber the address registers or flags. - Use registers other than R3 and R4 to avoid conflicting with registers that have already been allocated to avoid unnecessary register moves. - Eliminate the use of R14 as scratch register and use R31 instead. - Add PCALIGN when the LoweredMoveXXX or LoweredZeroXXX generates a loop with more than 3 iterations. This performance opportunity was noticed in github.com/golang/snappy benchmarks. Results on power9: WordsDecode1e1 54.1ns ± 0% 53.8ns ± 0% -0.51% (p=0.029 n=4+4) WordsDecode1e2 287ns ± 0% 282ns ± 1% -1.83% (p=0.029 n=4+4) WordsDecode1e3 3.98µs ± 0% 3.64µs ± 0% -8.52% (p=0.029 n=4+4) WordsDecode1e4 66.9µs ± 0% 67.0µs ± 0% +0.20% (p=0.029 n=4+4) WordsDecode1e5 723µs ± 0% 723µs ± 0% -0.01% (p=0.200 n=4+4) WordsDecode1e6 7.21ms ± 0% 7.21ms ± 0% -0.02% (p=1.000 n=4+4) WordsEncode1e1 29.9ns ± 0% 29.4ns ± 0% -1.51% (p=0.029 n=4+4) WordsEncode1e2 2.12µs ± 0% 1.75µs ± 0% -17.70% (p=0.029 n=4+4) WordsEncode1e3 11.7µs ± 0% 11.2µs ± 0% -4.61% (p=0.029 n=4+4) WordsEncode1e4 119µs ± 0% 120µs ± 0% +0.36% (p=0.029 n=4+4) WordsEncode1e5 1.21ms ± 0% 1.22ms ± 0% +0.41% (p=0.029 n=4+4) WordsEncode1e6 12.0ms ± 0% 12.0ms ± 0% +0.57% (p=0.029 n=4+4) RandomEncode 286µs ± 0% 203µs ± 0% -28.82% (p=0.029 n=4+4) ExtendMatch 47.4µs ± 0% 47.0µs ± 0% -0.85% (p=0.029 n=4+4) Change-Id: Iecad3a39ae55280286e42760a5c9d5c1168f5858 Reviewed-on: https://go-review.googlesource.com/c/go/+/226539 Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-03-30 15:23:19 -04:00
// ppc64:-".*memmove"
// ppc64le:-".*memmove"
// ppc64le/power8:"LXVD2X"
// ppc64le/power9:"LXV",-"LXVD2X"
cmd/compile: convert memmove call into Move when arguments are disjoint Move ops can be faster than memmove calls because the number of bytes to be moved is fixed and they don't incur the overhead of a call. This change allows memmove to be converted into a Move op when the arguments are disjoint. The optimization is only enabled on s390x at the moment, however other architectures may also benefit from it in the future. The memmove inlining rule triggers an extra 12 times when compiling the standard library. It will most likely make more of a difference as the disjoint function is improved over time (to recognize fresh heap allocations for example). Change-Id: I9af570dcfff28257b8e59e0ff584a46d8e248310 Reviewed-on: https://go-review.googlesource.com/110064 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ilya Tocar <ilya.tocar@intel.com>
2018-04-29 15:12:50 +01:00
copy(s[:], b[:])
runtime.KeepAlive(&s)
}
func moveDisjointNoOverlap(a *[256]byte) {
// s390x:-".*memmove"
cmd/compile: enable disjoint memmove inlining on amd64 Memmove can use AVX/prefetches/other optional instructions, so only do it for small sizes, when call overhead dominates. Change-Id: Ice5e93deb11462217f7fb5fc350b703109bb4090 Reviewed-on: https://go-review.googlesource.com/112517 Run-TryBot: Ilya Tocar <ilya.tocar@intel.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Michael Munday <mike.munday@ibm.com>
2018-05-09 15:49:22 -05:00
// amd64:-".*memmove"
cmd/compile: improve lowered moves and zeros for ppc64le This change includes the following: - Generate LXV/STXV sequences instead of LXVD2X/STXVD2X on power9. These instructions do not require an index register, which allows more loads and stores within a loop without initializing multiple index registers. The LoweredQuadXXX generate LXV/STXV. - Create LoweredMoveXXXShort and LoweredZeroXXXShort for short moves that don't generate loops, and therefore don't clobber the address registers or flags. - Use registers other than R3 and R4 to avoid conflicting with registers that have already been allocated to avoid unnecessary register moves. - Eliminate the use of R14 as scratch register and use R31 instead. - Add PCALIGN when the LoweredMoveXXX or LoweredZeroXXX generates a loop with more than 3 iterations. This performance opportunity was noticed in github.com/golang/snappy benchmarks. Results on power9: WordsDecode1e1 54.1ns ± 0% 53.8ns ± 0% -0.51% (p=0.029 n=4+4) WordsDecode1e2 287ns ± 0% 282ns ± 1% -1.83% (p=0.029 n=4+4) WordsDecode1e3 3.98µs ± 0% 3.64µs ± 0% -8.52% (p=0.029 n=4+4) WordsDecode1e4 66.9µs ± 0% 67.0µs ± 0% +0.20% (p=0.029 n=4+4) WordsDecode1e5 723µs ± 0% 723µs ± 0% -0.01% (p=0.200 n=4+4) WordsDecode1e6 7.21ms ± 0% 7.21ms ± 0% -0.02% (p=1.000 n=4+4) WordsEncode1e1 29.9ns ± 0% 29.4ns ± 0% -1.51% (p=0.029 n=4+4) WordsEncode1e2 2.12µs ± 0% 1.75µs ± 0% -17.70% (p=0.029 n=4+4) WordsEncode1e3 11.7µs ± 0% 11.2µs ± 0% -4.61% (p=0.029 n=4+4) WordsEncode1e4 119µs ± 0% 120µs ± 0% +0.36% (p=0.029 n=4+4) WordsEncode1e5 1.21ms ± 0% 1.22ms ± 0% +0.41% (p=0.029 n=4+4) WordsEncode1e6 12.0ms ± 0% 12.0ms ± 0% +0.57% (p=0.029 n=4+4) RandomEncode 286µs ± 0% 203µs ± 0% -28.82% (p=0.029 n=4+4) ExtendMatch 47.4µs ± 0% 47.0µs ± 0% -0.85% (p=0.029 n=4+4) Change-Id: Iecad3a39ae55280286e42760a5c9d5c1168f5858 Reviewed-on: https://go-review.googlesource.com/c/go/+/226539 Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-03-30 15:23:19 -04:00
// ppc64:-".*memmove"
// ppc64le:-".*memmove"
// ppc64le/power8:"LXVD2X"
// ppc64le/power9:"LXV",-"LXVD2X"
cmd/compile: convert memmove call into Move when arguments are disjoint Move ops can be faster than memmove calls because the number of bytes to be moved is fixed and they don't incur the overhead of a call. This change allows memmove to be converted into a Move op when the arguments are disjoint. The optimization is only enabled on s390x at the moment, however other architectures may also benefit from it in the future. The memmove inlining rule triggers an extra 12 times when compiling the standard library. It will most likely make more of a difference as the disjoint function is improved over time (to recognize fresh heap allocations for example). Change-Id: I9af570dcfff28257b8e59e0ff584a46d8e248310 Reviewed-on: https://go-review.googlesource.com/110064 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Ilya Tocar <ilya.tocar@intel.com>
2018-04-29 15:12:50 +01:00
copy(a[:], a[128:])
}
// Check that no branches are generated when the pointers are [not] equal.
cmd/compile: generate constants for NeqPtr, EqPtr and IsNonNil ops If both inputs are constant offsets from the same pointer then we can evaluate NeqPtr and EqPtr at compile time. Triggers a few times during all.bash. Removes a conditional branch in the following code: copy(x[1:], x[:]) This branch was recently added as an optimization in CL 94596. We now skip the memmove if the pointers are equal. However, in the above code we know at compile time that they are never equal. Also, when the offset is variable, check if the offset is zero rather than if the pointers are equal. For example: copy(x[a:], x[:]) This would now skip the copy if a == 0, rather than if x + a == x. Finally I've also added a rule to make IsNonNil true for pointers to values on the stack. The nil check elimination pass will catch these anyway, but eliminating them here might eliminate branches earlier. Change-Id: If72f436fef0a96ad0f4e296d3a1f8b6c3e712085 Reviewed-on: https://go-review.googlesource.com/106635 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2018-04-12 15:24:08 +01:00
func ptrEqual() {
// amd64:-"JEQ",-"JNE"
test/codegen: enable more tests for ppc64/ppc64le Adding cases for ppc64,ppc64le to the codegen tests where appropriate. Change-Id: Idf8cbe88a4ab4406a4ef1ea777bd15a58b68f3ed Reviewed-on: https://go-review.googlesource.com/c/142557 Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-15 12:53:07 -04:00
// ppc64:-"BEQ",-"BNE"
cmd/compile: generate constants for NeqPtr, EqPtr and IsNonNil ops If both inputs are constant offsets from the same pointer then we can evaluate NeqPtr and EqPtr at compile time. Triggers a few times during all.bash. Removes a conditional branch in the following code: copy(x[1:], x[:]) This branch was recently added as an optimization in CL 94596. We now skip the memmove if the pointers are equal. However, in the above code we know at compile time that they are never equal. Also, when the offset is variable, check if the offset is zero rather than if the pointers are equal. For example: copy(x[a:], x[:]) This would now skip the copy if a == 0, rather than if x + a == x. Finally I've also added a rule to make IsNonNil true for pointers to values on the stack. The nil check elimination pass will catch these anyway, but eliminating them here might eliminate branches earlier. Change-Id: If72f436fef0a96ad0f4e296d3a1f8b6c3e712085 Reviewed-on: https://go-review.googlesource.com/106635 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2018-04-12 15:24:08 +01:00
// ppc64le:-"BEQ",-"BNE"
// s390x:-"BEQ",-"BNE"
copy(x[:], x[:])
}
func ptrOneOffset() {
// amd64:-"JEQ",-"JNE"
test/codegen: enable more tests for ppc64/ppc64le Adding cases for ppc64,ppc64le to the codegen tests where appropriate. Change-Id: Idf8cbe88a4ab4406a4ef1ea777bd15a58b68f3ed Reviewed-on: https://go-review.googlesource.com/c/142557 Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-15 12:53:07 -04:00
// ppc64:-"BEQ",-"BNE"
cmd/compile: generate constants for NeqPtr, EqPtr and IsNonNil ops If both inputs are constant offsets from the same pointer then we can evaluate NeqPtr and EqPtr at compile time. Triggers a few times during all.bash. Removes a conditional branch in the following code: copy(x[1:], x[:]) This branch was recently added as an optimization in CL 94596. We now skip the memmove if the pointers are equal. However, in the above code we know at compile time that they are never equal. Also, when the offset is variable, check if the offset is zero rather than if the pointers are equal. For example: copy(x[a:], x[:]) This would now skip the copy if a == 0, rather than if x + a == x. Finally I've also added a rule to make IsNonNil true for pointers to values on the stack. The nil check elimination pass will catch these anyway, but eliminating them here might eliminate branches earlier. Change-Id: If72f436fef0a96ad0f4e296d3a1f8b6c3e712085 Reviewed-on: https://go-review.googlesource.com/106635 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2018-04-12 15:24:08 +01:00
// ppc64le:-"BEQ",-"BNE"
// s390x:-"BEQ",-"BNE"
copy(x[1:], x[:])
}
func ptrBothOffset() {
// amd64:-"JEQ",-"JNE"
test/codegen: enable more tests for ppc64/ppc64le Adding cases for ppc64,ppc64le to the codegen tests where appropriate. Change-Id: Idf8cbe88a4ab4406a4ef1ea777bd15a58b68f3ed Reviewed-on: https://go-review.googlesource.com/c/142557 Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
2018-10-15 12:53:07 -04:00
// ppc64:-"BEQ",-"BNE"
cmd/compile: generate constants for NeqPtr, EqPtr and IsNonNil ops If both inputs are constant offsets from the same pointer then we can evaluate NeqPtr and EqPtr at compile time. Triggers a few times during all.bash. Removes a conditional branch in the following code: copy(x[1:], x[:]) This branch was recently added as an optimization in CL 94596. We now skip the memmove if the pointers are equal. However, in the above code we know at compile time that they are never equal. Also, when the offset is variable, check if the offset is zero rather than if the pointers are equal. For example: copy(x[a:], x[:]) This would now skip the copy if a == 0, rather than if x + a == x. Finally I've also added a rule to make IsNonNil true for pointers to values on the stack. The nil check elimination pass will catch these anyway, but eliminating them here might eliminate branches earlier. Change-Id: If72f436fef0a96ad0f4e296d3a1f8b6c3e712085 Reviewed-on: https://go-review.googlesource.com/106635 Run-TryBot: Michael Munday <mike.munday@ibm.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org>
2018-04-12 15:24:08 +01:00
// ppc64le:-"BEQ",-"BNE"
// s390x:-"BEQ",-"BNE"
copy(x[1:], x[2:])
}
Reference in a new issue Copy permalink