runtime: add ragged global barrier function

This adds forEachP, which performs a general-purpose ragged global
barrier. forEachP takes a callback and invokes it for every P at a GC
safe point.

Ps that are idle or in a syscall are considered to be at a continuous
safe point. forEachP ensures that these Ps do not change state by
forcing all syscall Ps into idle and holding the sched.lock.

To ensure that Ps do not enter syscall or idle without running the
safe-point function, this adds checks for a pending callback every
place there is currently a gcwaiting check.

We'll use forEachP to replace the STW around enabling the write
barrier and to replace the current asynchronous per-M wbuf cache with
a cooperatively managed per-P gcWork cache.

Change-Id: Ie944f8ce1fead7c79bf271d2f42fcd61a41bb3cc
Reviewed-on: https://go-review.googlesource.com/8206
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-by: Rick Hudson <rlh@golang.org>

src/runtime/runtime2.go

@@ -38,8 +38,8 @@ const (
 
 const (
 	// P status
-	_Pidle = iota
-	_Prunning
+	_Pidle    = iota
+	_Prunning // Only this P is allowed to change from _Prunning.
 	_Psyscall
 	_Pgcstop
 	_Pdead
@@ -388,6 +388,8 @@ type p struct {
 	// disposed on certain GC state transitions.
 	gcw gcWork
 
+	runSafePointFn uint32 // if 1, run sched.safePointFn at next safe point
+
 	pad [64]byte
 }
 
@@ -437,6 +439,10 @@ type schedt struct {
 	sysmonnote note
 	lastpoll   uint64
 
+	// safepointFn should be called on each P at the next GC
+	// safepoint if p.runSafePointFn is set.
+	safePointFn func(*p)
+
 	profilehz int32 // cpu profiling rate
 
 	procresizetime int64 // nanotime() of last change to gomaxprocs