Change-Id: I84ec73d3ddef913a87cb9b48147c44ac3e7c8a8d
Reviewed-on: https://go-review.googlesource.com/c/go/+/651957
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Currently, at a cgo callback where there is already a Go frame on
the stack (i.e. C->Go->C->Go), we require that at the inner Go
callback the SP is within the g0's stack bounds set by a previous
callback. This is to prevent that the C code switches stack while
having a Go frame on the stack, which we don't really support. But
this could also happen when we cannot get accurate stack bounds,
e.g. when pthread_getattr_np is not available. Since the stack
bounds are just estimates based on the current SP, if there are
multiple C->Go callbacks with various stack depth, it is possible
that the SP of a later callback falls out of a previous call's
estimate. This leads to runtime throw in a seemingly reasonable
program.
This CL changes it to save the old g0 stack bounds at cgocallback,
update the bounds, and restore the old bounds at return. So each
callback will get its own stack bounds based on the current SP,
and when it returns, the outer callback has the its old stack
bounds restored.
Also, at a cgo callback when there is no Go frame on the stack,
we currently always get new stack bounds. We do this because if
we can only get estimated bounds based on the SP, and the stack
depth varies a lot between two C->Go calls, the previous
estimates may be off and we fall out or nearly fall out of the
previous bounds. But this causes a performance problem: the
pthread API to get accurate stack bounds (pthread_getattr_np) is
very slow when called on the main thread. Getting the stack bounds
every time significantly slows down repeated C->Go calls on the
main thread.
This CL fixes it by "caching" the stack bounds if they are
accurate. I.e. at the second time Go calls into C, if the previous
stack bounds are accurate, and the current SP is in bounds, we can
be sure it is the same stack and we don't need to update the bounds.
This avoids the repeated calls to pthread_getattr_np. If we cannot
get the accurate bounds, we continue to update the stack bounds
based on the SP, and that operation is very cheap.
On a Linux/AMD64 machine with glibc:
name old time/op new time/op delta
CgoCallbackMainThread-8 96.4µs ± 3% 0.1µs ± 2% -99.92% (p=0.000 n=10+9)
Fixes#68285.
Fixes#68587.
Change-Id: I3422badd5ad8ff63e1a733152d05fb7a44d5d435
Reviewed-on: https://go-review.googlesource.com/c/go/+/600296
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
The 1-minute test timeout seems a bit arbitrary. Use
internal/testenv.Command, which applies t.Deadline to subcommand.
For #67566.
Change-Id: If84c96e353bdfaf02c9b123758198e031305ae32
Reviewed-on: https://go-review.googlesource.com/c/go/+/599056
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Currently, set_crosscall2 takes the address of crosscall2 without
using the GOT, which, on some architectures, results in a
PC-relative relocation (e.g. R_AARCH64_ADR_PREL_PG_HI21 on ARM64)
to the crosscall2 symbol. But crosscall2 is dynamically exported,
so the C linker thinks it may bind to a symbol from a different
DSO. Some C linker may not like a PC-relative relocation to such a
symbol. Using a local trampoline to avoid taking the address of a
dynamically exported symbol.
It may be possible to not dynamically export crosscall2. But this
CL is safer for backport. Later we may remove the trampolines
after unexport crosscall2, if they are not needed.
Fixes#62556.
Change-Id: Id28457f65ef121d3f87d8189803abc65ed453283
Reviewed-on: https://go-review.googlesource.com/c/go/+/533535
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Many cgo integration tests do a lot of common setup in TestMain, and
that means they require a lot from the test environment to even get
off the ground. If something is missing, right now they print a "SKIP"
message to stderr and exit without running any tests.
Make these behave more like normal tests by instead setting a global
skip function if some precondition isn't satisfied, and having every
test call that. This way we run the tests and see them skip.
I would prefer something much more structured. For example, if we
replaced the global state set up by TestMain in these tests by instead
calling a function that returned that state (after setting it up on
the first call), that function could do the appropriate skips and
there would be no way to accidentally access this state without
checking the preconditions. But that's substantially more work and may
be much easier after we do further cleanup of these tests.
Change-Id: I92de569fd27596798c5e478402449cd735ec53a4
Reviewed-on: https://go-review.googlesource.com/c/go/+/497096
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
There are many copies of overlaydir_test.go between the cgo tests
from when these couldn't share code. Now that they can, merge these
copies into a cmd/cgo/internal/cgotest package.
Change-Id: I203217f5d08e6306cb049a13718652cf7c447b80
Reviewed-on: https://go-review.googlesource.com/c/go/+/497078
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
CL 495918 enabled testcarchive much more widely and added many dynamic
test skips. CL 495855 added TestDeepStack before these dynamic skips
were in. Unfortunately, the two CLs don't logically commute, so when
CL 495918 landed, it broke at least nocgo builders and platforms that
don't support c-archive builds. Fix this by adding the necessary skips
to TestDeepStack.
Change-Id: I3d352f731fe67a01c7b96871fde772db8eb21b5d
Reviewed-on: https://go-review.googlesource.com/c/go/+/496376
Auto-Submit: Austin Clements <austin@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Robert Griesemer <gri@google.com>
This CL moves many cgo test conditions out of dist and into the tests
themselves, now that they can use the testenv.Must* helpers.
This refines a lot of the conditions, which happens to have the effect
of enabling many tests on Android and iOS that are disabled by
too-coarse GOOS checks in dist today.
Fixes#15919.
Change-Id: I2947526b08928d2f7f89f107b5b2403b32092ed8
Reviewed-on: https://go-review.googlesource.com/c/go/+/495918
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
This reapplies CL 485500, with a fix drafted in CL 492987 incorporated.
CL 485500 is reverted due to #60004 and #60007. #60004 is fixed in
CL 492743. #60007 is fixed in CL 492987 (incorporated in this CL).
[Original CL 485500 description]
This reapplies CL 481061, with the followup fixes in CL 482975, CL 485315, and
CL 485316 incorporated.
CL 481061, by doujiang24 <doujiang24@gmail.com>, speed up C to Go
calls by binding the M to the C thread. See below for its
description.
CL 482975 is a followup fix to a C declaration in testprogcgo.
CL 485315 is a followup fix for x_cgo_getstackbound on Illumos.
CL 485316 is a followup cleanup for ppc64 assembly.
CL 479915 passed the G to _cgo_getstackbound for direct updates to
gp.stack.lo. A G can be reused on a new thread after the previous thread
exited. This could trigger the C TSAN race detector because it couldn't
see the synchronization in Go (lockextra) preventing the same G from
being used on multiple threads at the same time.
We work around this by passing the address of a stack variable to
_cgo_getstackbound rather than the G. The stack is generally unique per
thread, so TSAN won't see the same address from multiple threads. Even
if stacks are reused across threads by pthread, C TSAN should see the
synchonization in the stack allocator.
A regression test is added to misc/cgo/testsanitizer.
[Original CL 481061 description]
This reapplies CL 392854, with the followup fixes in CL 479255,
CL 479915, and CL 481057 incorporated.
CL 392854, by doujiang24 <doujiang24@gmail.com>, speed up C to Go
calls by binding the M to the C thread. See below for its
description.
CL 479255 is a followup fix for a small bug in ARM assembly code.
CL 479915 is another followup fix to address C to Go calls after
the C code uses some stack, but that CL is also buggy.
CL 481057, by Michael Knyszek, is a followup fix for a memory leak
bug of CL 479915.
[Original CL 392854 description]
In a C thread, it's necessary to acquire an extra M by using needm while invoking a Go function from C. But, needm and dropm are heavy costs due to the signal-related syscalls.
So, we change to not dropm while returning back to C, which means binding the extra M to the C thread until it exits, to avoid needm and dropm on each C to Go call.
Instead, we only dropm while the C thread exits, so the extra M won't leak.
When invoking a Go function from C:
Allocate a pthread variable using pthread_key_create, only once per shared object, and register a thread-exit-time destructor.
And store the g0 of the current m into the thread-specified value of the pthread key, only once per C thread, so that the destructor will put the extra M back onto the extra M list while the C thread exits.
When returning back to C:
Skip dropm in cgocallback, when the pthread variable has been created, so that the extra M will be reused the next time invoke a Go function from C.
This is purely a performance optimization. The old version, in which needm & dropm happen on each cgo call, is still correct too, and we have to keep the old version on systems with cgo but without pthreads, like Windows.
This optimization is significant, and the specific value depends on the OS system and CPU, but in general, it can be considered as 10x faster, for a simple Go function call from a C thread.
For the newly added BenchmarkCGoInCThread, some benchmark results:
1. it's 28x faster, from 3395 ns/op to 121 ns/op, in darwin OS & Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
2. it's 6.5x faster, from 1495 ns/op to 230 ns/op, in Linux OS & Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
[CL 479915 description]
Currently, when C calls into Go the first time, we grab an M
using needm, which sets m.g0's stack bounds using the SP. We don't
know how big the stack is, so we simply assume 32K. Previously,
when the Go function returns to C, we drop the M, and the next
time C calls into Go, we put a new stack bound on the g0 based on
the current SP. After CL 392854, we don't drop the M, and the next
time C calls into Go, we reuse the same g0, without recomputing
the stack bounds. If the C code uses quite a bit of stack space
before calling into Go, the SP may be well below the 32K stack
bound we assumed, so the runtime thinks the g0 stack overflows.
This CL makes needm get a more accurate stack bound from
pthread. (In some platforms this may still be a guess as we don't
know exactly where we are in the C stack), but it is probably
better than simply assuming 32K.
[CL 492987 description]
On the first call into Go from a C thread, currently we set the g0
stack's high bound imprecisely based on the SP. With CL 485500, we
keep the M and don't recompute the stack bounds when it calls into
Go again. If the first call is made when the C thread uses some
deep stack, but a subsequent call is made with a shallower stack,
the SP may be above g0.stack.hi.
This is usually okay as we don't check usually stack.hi. One place
where we do check for stack.hi is in the signal handler, in
adjustSignalStack. In particular, C TSAN delivers signals on the
g0 stack (instead of the usual signal stack). If the SP is above
g0.stack.hi, we don't see it is on the g0 stack, and throws.
This CL makes it get an accurate stack upper bound with the
pthread API (on the platforms where it is available).
Also add some debug print for the "handler not on signal stack"
throw.
Fixes#51676.
Fixes#59294.
Fixes#59678.
Fixes#60007.
Change-Id: Ie51c8e81ade34ec81d69fd7bce1fe0039a470776
Reviewed-on: https://go-review.googlesource.com/c/go/+/495855
Run-TryBot: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
This test package uses syscall.SIGSEGV and syscall.SIGPIPE, which are
defined on most, but not all platforms. Normally this test runs as
part of dist test, which only registers this test on platforms that
support c-archive build mode, which includes all platforms that define
these signals. But this doesn't help if you're just trying to type
check everything in cmd.
Add build constraints so that this package type checks on all
platforms.
Fixes#60164.
Updates #37486.
Change-Id: Id3f9ad4cc9f80146de16aedcf85d108a77215ae6
Reviewed-on: https://go-review.googlesource.com/c/go/+/494659
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Austin Clements <austin@google.com>
Auto-Submit: Austin Clements <austin@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Host tests are used for emulated builders that use cross-compilation.
Today, this is the android-{386,amd64}-emu builders and all wasm
builders. These builders run all.bash on a linux/amd64 host to build
all packages and most tests for the emulated guest, and then run the
resulting test binaries inside the emulated guest. A small number of
test packages are “host tests”: these run on the host rather than the
guest because they invoke the Go toolchain themselves (which only
lives on the host) and run the resulting binaries in the guest.
However, this host test mechanism is barely used today, despite being
quite complex. This complexity is also causing significant friction to
implementing structured all.bash output.
As of this CL, the whole host test mechanism runs a total of 10 test
cases on a total of two builders (android-{386,amd64}-emu). There are
clearly several tests that are incorrectly being skipped, so we could
expand it to cover more test cases, but it would still apply to only
two builders. Furthermore, the two other Android builders
(android-{arm,arm64}-corellium) build the Go toolchain directly inside
Android and also have access to a C toolchain, so they are able to get
significantly better test coverage without the use of host tests. This
suggests that the android-*-emu builders could do the same. All of
these tests are cgo-related, so they don't run on the wasm hosts
anyway.
Given the incredibly low value of host tests today, they are not worth
their implementation complexity and the friction they cause. Hence,
this CL drops support for host tests. (This was also the last use of
rtSequential, so we drop support for sequential tests, too.)
Fixes#59999.
Change-Id: I3eaca853a8907abc8247709f15a0d19a872dd22d
Reviewed-on: https://go-review.googlesource.com/c/go/+/492986
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
This moves most misc/cgo tests to cmd/cgo/internal. This is mostly a
trivial rename and updating dist/test.go for the new paths, plus
excluding these packages from regular dist test registration. A few
tests were sensitive to what path they ran in, so we update those.
This will let these tests access facilities in internal/testenv.
For #37486.
Change-Id: I3ed417c7c22d9b667f2767c0cb1f59118fcd4af6
Reviewed-on: https://go-review.googlesource.com/c/go/+/492720
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2023-05-12 11:59:56 +00:00
Renamed from misc/cgo/testcarchive/carchive_test.go (Browse further)
