On some architectures, for async preemption the injected call
needs to clobber a register (usually REGTMP) in order to return
to the preempted function. As a consequence, the PC ranges where
REGTMP is live are not preemptible.
The uses of REGTMP are usually generated by the assembler, where
it needs to load or materialize a large constant or offset that
doesn't fit into the instruction. In those cases, REGTMP is not
live at the start of the instruction sequence. Instead of giving
up preemption in those cases, we could preempt it and restart the
sequence when resuming the execution. Basically, this is like
reissuing an interrupted instruction, except that here the
"instruction" is a Prog that consists of multiple machine
instructions. For this to work, we need to generate PC data to
mark the start of the Prog.
Currently this is only done for ARM64.
TODO: the split-stack function prologue is currently not async
preemptible. We could use this mechanism, preempt it and restart
at the function entry.
Change-Id: I37cb282f8e606e7ab6f67b3edfdc6063097b4bd1
Reviewed-on: https://go-review.googlesource.com/c/go/+/208126
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
This commit adds a new option to the x86 assembler. If the
GOAMD64 environment variable is set to alignedjumps (the
default) and we're doing a 64 bit build, the assembler will
make sure that neither stand alone nor macro-fused jumps will
end on or cross 32 byte boundaries. To achieve this, functions
are aligned on 32 byte boundaries, rather than 16 bytes, and
jump instructions are padded to ensure that they do not
cross or end on 32 byte boundaries. Jumps are padded
by adding a NOP instruction of the appropriate length before
the jump.
The commit is likely to result in larger binary sizes when
GOAMD64=alignedjumps. On the binaries tested so far, an
increase of between 1.4% and 1.5% has been observed.
Updates #35881
Co-authored-by: David Chase <drchase@google.com>
Change-Id: Ief0722300bc3f987098e4fd92b22b14ad6281d91
Reviewed-on: https://go-review.googlesource.com/c/go/+/219357
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This commit extends the -spectre flag to cmd/asm and adds
a new Spectre mitigation mode "ret", which enables the use
of retpolines.
Retpolines prevent speculation about the target of an indirect
jump or call and are described in more detail here:
https://support.google.com/faqs/answer/7625886
Change-Id: I4f2cb982fa94e44d91e49bd98974fd125619c93a
Reviewed-on: https://go-review.googlesource.com/c/go/+/222661
Reviewed-by: Keith Randall <khr@golang.org>
The 2-instruction TLS access sequence
MOVQ TLS, BX
MOVQ 0(BX)(TLS*1), BX
is not async preemptible, as if it is preempted and resumed on a
different thread, the TLS address may become invalid.
May fix#35349. (This is a rare failure and I haven't been able
to reproduce it.)
Change-Id: Ie1a366fd0d7d73627dc62ee2de01c0aa09365f2b
Reviewed-on: https://go-review.googlesource.com/c/go/+/206903
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Austin Clements <austin@google.com>
When compiling for GOARCH=386 GOOS=android, the compiler was attaching
R_TLS_LE relocations inappropriately -- as of Go 1.13 the TLS access
recipe for Android refers to a runtime symbol and no longer needs this
type of relocation (which was causing a crash when the linker tried to
process it).
Updates #29674.
Fixes#34788.
Change-Id: Ida01875011b524586597b1f7e273aa14e11815d6
Reviewed-on: https://go-review.googlesource.com/c/go/+/200337
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Elias Naur <mail@eliasnaur.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
This is part two if the nacl removal. Part 1 was CL 199499.
This CL removes amd64p32 support, which might be useful in the future
if we implement the x32 ABI. It also removes the nacl bits in the
toolchain, and some remaining nacl bits.
Updates #30439
Change-Id: I2475d5bb066d1b474e00e40d95b520e7c2e286e1
Reviewed-on: https://go-review.googlesource.com/c/go/+/200077
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Use the following (suboptimal) script to obtain a list of possible
typos:
#!/usr/bin/env sh
set -x
git ls-files |\
grep -e '\.\(c\|cc\|go\)$' |\
xargs -n 1\
awk\
'/\/\// { gsub(/.*\/\//, ""); print; } /\/\*/, /\*\// { gsub(/.*\/\*/, ""); gsub(/\*\/.*/, ""); }' |\
hunspell -d en_US -l |\
grep '^[[:upper:]]\{0,1\}[[:lower:]]\{1,\}$' |\
grep -v -e '^.\{1,4\}$' -e '^.\{16,\}$' |\
sort -f |\
uniq -c |\
awk '$1 == 1 { print $2; }'
Then, go through the results manually and fix the most obvious typos in
the non-vendored code.
Change-Id: I3cb5830a176850e1a0584b8a40b47bde7b260eae
Reviewed-on: https://go-review.googlesource.com/c/go/+/193848
Reviewed-by: Robert Griesemer <gri@golang.org>
Those print statements are not a good debug helpers
and only clutter the code.
Change-Id: Ifbf450a04e6fa538af68e6352c016728edb4119a
Reviewed-on: https://go-review.googlesource.com/c/go/+/160537
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
Since BSWAP operation on 16-bit registers is undefined,
forbid the usage of BSWAPW. Users should rely on XCHGB instead.
This behavior is consistent with what GAS does.
Fixes#29167
Change-Id: I3b31e3dd2acfd039f7564a1c17e6068617bcde8d
Reviewed-on: https://go-review.googlesource.com/c/go/+/174312
Run-TryBot: Iskander Sharipov <quasilyte@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
An instruction that references TLS, e.g.
MOVQ 0(TLS), AX
on some platforms (e.g. Android), or in shared mode, may be
translated to (assuming TLS offset already loaded to CX)
MOVQ 0(CX)(TLS*1), AX
which in turns translates to
movq %fs:(%rcx), %rax
We have rejected non-zero offset for TLS reference, like 16(TLS).
Actually, the instruction can take offset, i.e. it is a valid
instruction for, e.g.,
movq %fs:16(%rcx),%rcx
So, allow offset in TLS reference.
Change-Id: Iaf1996bad7fe874e0c298ea441af5acb136a4028
Reviewed-on: https://go-review.googlesource.com/c/go/+/171151
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
AVX-512 instructions that use RIP-relative addressing and require the
R bit of the EVEX prefix to be zero, i.e., instructions that use Z8-Z15 or
Z24-Z31, are incorrectly encoded by the assembler. The reason is that
the location of the offset at which the relative address is to be written
is incorrectly computed when the R bit is clear.
For example,
VMOVUPS bInitX<>+0(SB), Z0
encodes correctly to
62 f1 7c 48 10 05 66 e9 02 00
whereas
VMOVUPS bInitX<>+0(SB), Z8
encodes incorrectly to
62 71 7c 48 10 05 00 56 e9 02 00
Note the extra zero byte between the ModR/M byte (05) and the relative
address starting with 56. This error results in the first byte of the
following instruction being overwritten and typically, a program crash.
This commit fixes the issue in the same way that is fixed for VEX encoded
instructions, by simply not incrementing the offset for EVEX instructions.
Existing test code created for a similar VEX encoding issue (19518) has
been modified to also test for the issue addressed by this commit.
Fixes#31001
Change-Id: If84719ac22ebb5fb3c42ff96cd32b611ad497414
Reviewed-on: https://go-review.googlesource.com/c/go/+/168562
Run-TryBot: Ilya Tocar <ilya.tocar@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ilya Tocar <ilya.tocar@intel.com>
Currently, obj.Ctxt's symbol table does not distinguish between ABI0
and ABIInternal symbols. This is *almost* okay, since a given symbol
name in the final object file is only going to belong to one ABI or
the other, but it requires that the compiler mark a Sym as being a
function symbol before it retrieves its LSym. If it retrieves the LSym
first, that LSym will be created as ABI0, and later marking the Sym as
a function symbol won't change the LSym's ABI.
Marking a Sym as a function symbol before looking up its LSym sounds
easy, except Syms have a dual purpose: they are used just as interned
strings (every function, variable, parameter, etc with the same
textual name shares a Sym), and *also* to store state for whatever
package global has that name. As a result, it's easy to slip up and
look up an LSym when a Sym is serving as the name of a local variable,
and then later mark it as a function when it's serving as the global
with the name.
In general, we were careful to avoid this, but #29610 demonstrates one
case where we messed up. Because of on-demand importing from indexed
export data, it's possible to compile a method wrapper for a type
imported from another package before importing an init function from
that package. If the argument of the method is named "init", the
"init" LSym will be created as a data symbol when compiling the
wrapper, before it gets marked as a function symbol.
To fix this, we separate obj.Ctxt's symbol tables for ABI0 and
ABIInternal symbols. This way, the compiler will simply get a
different LSym once the Sym takes on its package-global meaning as a
function.
This fixes the above ordering issue, and means we no longer need to go
out of our way to create the "init" function early and mark it as a
function symbol.
Fixes#29610.
Updates #27539.
Change-Id: Id9458b40017893d46ef9e4a3f9b47fc49e1ce8df
Reviewed-on: https://go-review.googlesource.com/c/157017
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
This implements compiler and linker support for separating the
function calling ABI into two ABIs: a stable and an internal ABI. At
the moment, the two ABIs are identical, but we'll be able to evolve
the internal ABI without breaking existing assembly code that depends
on the stable ABI for calling to and from Go.
The Go compiler generates internal ABI symbols for all Go functions.
It uses the symabis information produced by the assembler to create
ABI wrappers whenever it encounters a body-less Go function that's
defined in assembly or a Go function that's referenced from assembly.
Since the two ABIs are currently identical, for the moment this is
implemented using "ABI alias" symbols, which are just forwarding
references to the native ABI symbol for a function. This way there's
no actual code involved in the ABI wrapper, which is good because
we're not deriving any benefit from it right now. Once the ABIs
diverge, we can eliminate ABI aliases.
The linker represents these different ABIs internally as different
versions of the same symbol. This way, the linker keeps us honest,
since every symbol definition and reference also specifies its
version. The linker is responsible for resolving ABI aliases.
Fixes#27539.
Change-Id: I197c52ec9f8fc435db8f7a4259029b20f6d65e95
Reviewed-on: https://go-review.googlesource.com/c/147160
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David Chase <drchase@google.com>
There are still some references to the bare Syscall functions
in the stdlib. I will root those out in a following CL.
(This CL is big enough as it is.)
Most are in vendor directories:
cmd/vendor/golang.org/x/sys/unix/
vendor/golang_org/x/net/route/syscall.go
syscall/bpf_bsd.go
syscall/exec_unix.go
syscall/flock.go
Update #17490
Change-Id: I69ab707811530c26b652b291cadee92f5bf5c1a4
Reviewed-on: https://go-review.googlesource.com/c/141639
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Elias Naur <elias.naur@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
- Uncomment tests for AVX512 encoder
- Permit instruction suffixes for x86
- Permit limited reg list [reg-reg] syntax for x86 for multi-source ops
- EVEX encoding support in obj/x86 (Z-cases, asmevex, etc.)
- optabs and ytabs generated by x86avxgen (https://golang.org/cl/107216)
Note: suffix formatting implemented with updated CConv function.
Now arch asm backend should register formatting function by
calling RegisterOpSuffix.
Updates #22779
Change-Id: I076a167ee49582700e058c56ad74e6696710c8c8
Reviewed-on: https://go-review.googlesource.com/113315
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Use 0-terminated opbyte sequences for Zlit-like movtabs instead of E=0xff.
movCodeFullPtr is unused (load full ptr is unsupported), but it should
be removed in a separate CL (if removed at all).
Passes toolstash-check.
Change-Id: I28436718d93b017153de0e50e3bcec344ea4ee05
Reviewed-on: https://go-review.googlesource.com/107076
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Memory arguments for debug/control register moves are a
minefield for programmer: not useful, but can lead to errors.
See referenced issue for detailed explanation.
Fixes#24981
Change-Id: I918e81cd4a8b1dfcfc9023cdfc3de45abe29e749
Reviewed-on: https://go-review.googlesource.com/107075
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reject to compile I386/AMD64 asm code that contains
(Register)(PseudoReg*scale) forms of memory operands.
Example of such program: "CALL (AX)(PC*2)".
PseudoReg is one of the PC, FP, SB (but not SP).
When pseudo-register is used in register indirect as
scaled index base, x86 backend will panic because
its register file misses SB/FP/PC registers.
Fixes#12657.
Change-Id: I30fca797b537cbc86ab47583ae96c6a0c59acaa1
Reviewed-on: https://go-review.googlesource.com/107835
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Special cases described in TODO comments are
fixed in https://golang.org/cl/6901.
One of those blocks was needed until old 6a assembler was removed.
This happened in https://golang.org/cl/12784.
Fixes#24734 (Also contains more details and reasoning)
Change-Id: If1f2f155d36ab236b16ae6f309a0716e00aa6371
Reviewed-on: https://go-review.googlesource.com/105156
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
VEX constants were used when instructions were added by hand.
Now all VEX-encoded instructions are auto-generated by x86avxgen,
so there is no need for those anymore.
Change-Id: Ida63e5e23a8b819b15f61ac98980dec45a21617c
Reviewed-on: https://go-review.googlesource.com/104775
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
Fixes golint receiver name complaints.
We can't go with "a" name as it sometimes is used for obj.Addr args.
Change-Id: I66556f4e3dc42cfaaa4db3ed7772fa6756ea9a9b
Reviewed-on: https://go-review.googlesource.com/104796
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>
We already replaced most loops with PutOpBytesLit where possible,
do this in a last few places.
Change-Id: I8c90de017810145a12394fa6b887755e9111b22a
Reviewed-on: https://go-review.googlesource.com/102276
Run-TryBot: Ilya Tocar <ilya.tocar@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Fixes#14327
Much of the code is based on the linux/amd64 code that implements these
build modes, and code is shared where possible.
Change-Id: Ia510f2023768c0edbc863aebc585929ec593b332
Reviewed-on: https://go-review.googlesource.com/93875
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
The imm8 argument consists of 4 2-bit indices, so it can take values up
to $255. However, the assembler was treating it as Yi8, which reads
"fits in int8". Add a Yu8 variant, to also keep backwards compatibility
with negative values possible with Yi8.
Fixes#24378.
Change-Id: I24ddb19c219b54d039a6c1bcdb903717d1c7c3b8
Reviewed-on: https://go-review.googlesource.com/100475
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Replace some ints with bool and use arrays istead of slices where
possible.
Change-Id: I510bdaec48f9c437685e72c4a3291cffeb7ef5fc
Reviewed-on: https://go-review.googlesource.com/83859
Run-TryBot: Ilya Tocar <ilya.tocar@intel.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This reverts commit 08f19bbde1.
Reason for revert:
The changed transformation takes effect on a larger set
of code snippets than expected.
For example, this:
func foo() {
// Comment
bar()
}
becomes:
func foo() {
// Comment
bar()
}
This is an unintended consequence.
Change-Id: Ifca88d6267dab8a8170791f7205124712bf8ace8
Reviewed-on: https://go-review.googlesource.com/81335
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Joe Tsai <joetsai@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Fixes VBLENDVP{D/S}, VPBLENDVB encoding for /is4 imm8[7:4]
encoded register operand.
Explanation:
`reg[r]+regrex[r]+1` will yield correct values for 8..15 reg indexes,
but for 0..7 it gives `index+1` results.
There was no test that used lower 8 register with /is4 encoding,
so the bug passed the tests.
The proper solution is to get 4th bit from regrex with a proper shift:
`reg[r]|(regrex[r]<<1)`.
Instead of inlining `reg[r]|(regrex[r]<<1)` expr,
using new `regIndex(r)` function.
Test that reproduces this issue is added to
amd64enc_extra.s test suite.
Bug came from https://golang.org/cl/70650.
Change-Id: I846a25e88d5e6df88df9d9c3f5fe94ec55416a33
Reviewed-on: https://go-review.googlesource.com/78815
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ilya Tocar <ilya.tocar@intel.com>
Enables AVX2 gather instructions and VSIB support,
which makes vm32{x,y} vm64{x,y} operands encodable.
AXXX constants placed with respect to sorting order.
New VEX optabs inserted near non-VEX entries to simplify
potential transition to auto-generated VSIB optabs.
Tests go into new AMD64 encoder test file (amd64enc_extra.s)
to avoid unnecessary interactions with auto-generated "amd64enc.s".
Side note: x86avxgen did not produced these instructions
because x86.v0.2.csv misses them.
This also explains why x86 test suite have no AVX2 gather
instructions tests.
List of new instructions:
VGATHERPDP
VGATHERDPS
VGATHERQPD
VGATHERQPS
VPGATHERDD
VPGATHERDQ
VPGATHERQD
VPGATHERQQ
Change-Id: Iac852f3c5016523670bd99de6bec6a48f66fb4f6
Reviewed-on: https://go-review.googlesource.com/77970
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ilya Tocar <ilya.tocar@intel.com>
To improve readability when exported fields are removed,
forbid the printer from emitting an empty line before the first comment
in a const, var, or type block.
Also, when printing the "Has filtered or unexported fields." message,
add an empty line before it to separate the message from the struct
or interfact contents.
Before the change:
<<<
type NamedArg struct {
// Name is the name of the parameter placeholder.
//
// If empty, the ordinal position in the argument list will be
// used.
//
// Name must omit any symbol prefix.
Name string
// Value is the value of the parameter.
// It may be assigned the same value types as the query
// arguments.
Value interface{}
// contains filtered or unexported fields
}
>>>
After the change:
<<<
type NamedArg struct {
// Name is the name of the parameter placeholder.
//
// If empty, the ordinal position in the argument list will be
// used.
//
// Name must omit any symbol prefix.
Name string
// Value is the value of the parameter.
// It may be assigned the same value types as the query
// arguments.
Value interface{}
// contains filtered or unexported fields
}
>>>
Fixes#18264
Change-Id: I9fe17ca39cf92fcdfea55064bd2eaa784ce48c88
Reviewed-on: https://go-review.googlesource.com/71990
Run-TryBot: Joe Tsai <thebrokentoaster@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
Current AllowedOpCodes is 1024, which is not enough for modern x86.
Changed limit to 2048 (though AVX512 will exceed this).
Additional Z-cases and ytab tables are added to make it possible
to handle missing AVX1 and AVX2 instructions.
This CL is required by x86avxgen to work properly:
https://go-review.googlesource.com/c/arch/+/66972
Change-Id: I290214bbda554d2cba53349f50dcd34014fe4cee
Reviewed-on: https://go-review.googlesource.com/70650
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ilya Tocar <ilya.tocar@intel.com>
Add support for ADX cpuid bit detection and all instructions,
implied by that bit (ADOX/ADCX). They are useful for rsa and math/big in
general.
Change-Id: Idaa93303ead48fd18b9b3da09b3e79de2f7e2193
Reviewed-on: https://go-review.googlesource.com/74850
Run-TryBot: Ilya Tocar <ilya.tocar@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
1. Move AXXX constants (A-enumeration) from "a.out.go" to "aenum.go"
2. Move VEX-encoded optabs from "asm6.go" to "vex_optabs.go"
Also run "go generate" over aenum.go. This explains diff in "anames.go".
Initialization of opindex is split into 2 loops:
one for `vexOptab`, second for `optab`.
Rationale:
when VEX instructions are generated with current structure,
asm6.go is modified, which can lead to merge conflicts and
larger diffs than desired. Same for a.out.go.
This change makes x86avxgen usage possible:
https://go-review.googlesource.com/c/arch/+/66972
Change-Id: Id9eefcf5ccf0a89440e5d01bcb80926a8163b41d
Reviewed-on: https://go-review.googlesource.com/70630
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
