runtime: Remove write barriers during STW.

The GC assumes that there will be no asynchronous write barriers when
the world is stopped. This keeps the synchronization between write
barriers and the GC simple. However, currently, there are a few places
in runtime code where this assumption does not hold.
The GC stops the world by collecting all Ps, which stops all user Go
code, but small parts of the runtime can run without a P. For example,
the code that releases a P must still deschedule its G onto a runnable
queue before stopping. Similarly, when a G returns from a long-running
syscall, it must run code to reacquire a P.
Currently, this code can contain write barriers. This can lead to the
GC collecting reachable objects if something like the following
sequence of events happens:
1. GC stops the world by collecting all Ps.
2. G #1 returns from a syscall (for example), tries to install a
pointer to object X, and calls greyobject on X.
3. greyobject on G #1 marks X, but does not yet add it to a write
buffer. At this point, X is effectively black, not grey, even though
it may point to white objects.
4. GC reaches X through some other path and calls greyobject on X, but
greyobject does nothing because X is already marked.
5. GC completes.
6. greyobject on G #1 adds X to a work buffer, but it's too late.
7. Objects that were reachable only through X are incorrectly collected.
To fix this, we check the invariant that no asynchronous write
barriers happen when the world is stopped by checking that write
barriers always have a P, and modify all currently known sources of
these writes to disable the write barrier. In all modified cases this
is safe because the object in question will always be reachable via
some other path.

Some of the trace code was turned off, in particular the
code that traces returning from a syscall. The GC assumes
that as far as the heap is concerned the thread is stopped
when it is in a syscall. Upon returning the trace code
must not do any heap writes for the same reasons discussed
above.

Fixes #10098
Fixes #9953
Fixes #9951
Fixes #9884

May relate to #9610 #9771

Change-Id: Ic2e70b7caffa053e56156838eb8d89503e3c0c8a
Reviewed-on: https://go-review.googlesource.com/7504
Reviewed-by: Austin Clements <austin@google.com>

src/runtime/runtime2.go

@@ -117,6 +117,38 @@ func (gp guintptr) ptr() *g {
 	return (*g)(unsafe.Pointer(gp))
 }
 
+// ps, ms, gs, and mcache are structures that must be manipulated at a level
+// lower than that of the normal Go language. For example the routine that
+// stops the world removes the p from the m structure informing the GC that
+// this P is stopped and then it moves the g to the global runnable queue.
+// If write barriers were allowed to happen at this point not only does
+// the GC think the thread is stopped but the underlying structures
+// like a p or m are not in a state that is not coherent enough to
+// support the write barrier actions.
+// This is particularly painful since a partially executed write barrier
+// may mark the object but be delinquent in informing the GC that the
+// object needs to be scanned.
+
+// setGNoWriteBarriers does *gdst = gval without a write barrier.
+func setGNoWriteBarrier(gdst **g, gval *g) {
+	*(*uintptr)(unsafe.Pointer(gdst)) = uintptr(unsafe.Pointer(gval))
+}
+
+// setMNoWriteBarriers does *mdst = mval without a write barrier.
+func setMNoWriteBarrier(mdst **m, mval *m) {
+	*(*uintptr)(unsafe.Pointer(mdst)) = uintptr(unsafe.Pointer(mval))
+}
+
+// setPNoWriteBarriers does *pdst = pval without a write barrier.
+func setPNoWriteBarrier(pdst **p, pval *p) {
+	*(*uintptr)(unsafe.Pointer(pdst)) = uintptr(unsafe.Pointer(pval))
+}
+
+// setMcacheNoWriteBarriers does *mcachedst = mcacheval without a write barrier.
+func setMcacheNoWriteBarrier(mcachedst **mcache, mcacheval *mcache) {
+	*(*uintptr)(unsafe.Pointer(mcachedst)) = uintptr(unsafe.Pointer(mcacheval))
+}
+
 type gobuf struct {
 	// The offsets of sp, pc, and g are known to (hard-coded in) libmach.
 	sp   uintptr
@@ -233,13 +265,13 @@ type m struct {
 	morebuf gobuf // gobuf arg to morestack
 
 	// Fields not known to debuggers.
-	procid        uint64         // for debuggers, but offset not hard-coded
-	gsignal       *g             // signal-handling g
-	tls           [4]uintptr     // thread-local storage (for x86 extern register)
-	mstartfn      unsafe.Pointer // todo go func()
-	curg          *g             // current running goroutine
-	caughtsig     *g             // goroutine running during fatal signal
-	p             *p             // attached p for executing go code (nil if not executing go code)
+	procid        uint64     // for debuggers, but offset not hard-coded
+	gsignal       *g         // signal-handling g
+	tls           [4]uintptr // thread-local storage (for x86 extern register)
+	mstartfn      uintptr    // TODO: type as func(); note: this is a non-heap allocated func()
+	curg          *g         // current running goroutine
+	caughtsig     *g         // goroutine running during fatal signal
+	p             *p         // attached p for executing go code (nil if not executing go code)
 	nextp         *p
 	id            int32
 	mallocing     int32