cmd/go: run cgo and cgo compiles in their own actions

This change splits package builds further into even more actions.

Between the cache action (and cover action, if present) and the actual
build action we insert three layers of actions:

              Check Cache Action
              |      |
              |      V
              | Run Cover Action (if cover requested)
              |      |
              |      V
 New Layer 1  | Run Cgo Action (if package is built with cgo)
              |      | | | (multiple arrows representing fan-out to
              |      V V V  multiple actions)
 New Layer 2  | gcc Compile Action for each C/C++/S/F/M file (if cgo)
              |      |/    (arrow represeting fan-in to single action)
              |      V
 New Layer 3  | Cgo collect action (if cgo)
              |      |
              \      V
                Build action

The first run cgo action takes the input source files and runs cgo on
them to produce the cgo-processed go files, which are given to the
compiler, and to produce additional C files to compile and headers to
use in the following compilations. The action also takes care of running
SWIG before running cgo if there are swig files. This will produce
additional cgo sources that are inputs to cgo.

The run cgo action action fans out to multiple actions to do each of the
C/C++/Obj-C/assembly/Fortran compilations for the non-Go files in the
package, as well as those produced as outputs by the cgo command
invocation.

These actions then join into a single noop "collect" action which
primarily exists so that we don't pollute the build action's
dependencies with a bunch of non-go compile actions. (The build action
expects its dependencies to mostly be other build actions).

All of this work in the new actions was previously being done in the
build action itself. There's still a remnant of the original cgo logic
left in the build action to run the cgo command with -dynimport to
produce a go file to be built with the rest of the package, and do some
checks.

Most of this CL consists of moving code around. Just like the previous
CL breaking out the coverage logic into a separate action, we don't
cache the outputs of the cgo actions, and just treat all the actions
used to build a single package as one cacheable unit. This makes things
a bit simpler. If we decide in a future CL to cache the outputs
separately, we could remove the dependency on the cover action on the
check cache action (which in turn depends on all the package's
dependencies) and could start non-go compilation pretty much as early as
we want in the build.

The 'cgoAction' function in action.go takes care of creating the layers
of cgo action dependencies, which are inserted as dependencies of the
build action. It's mostly straightforward, except for the fact that we
need to tell each non-go compile action which non-go file to compile, so
we need to compute the names of the generated files. (Alternatively we
could give each action a number and have it build the nth file produced
by the run cgo action, but that seems even more complicated). The actors
produced to run the action logic are pretty light wrappers around the
execution logic in exec.go.

In the 'build' function in exec.go, most of the new code mainly checks
for the information from the cgo actions to use instead of running it,
and then passes it to the processCgoOutputs function. The only other
weird thing in the build functian is that we we call the logic to
compute the nonGoOverlay separately just for the C files that are being
built with gccgo. We compute the overlay for the non-go files used in a
cgo build in the run cgo action.

The 'cgo' function that previously ran the cgo logic for the build has
now been split into three: the first half, which runs cgo is now in the
runCgo function, the center part, which compiles the non-go files is now
partly in creating the invididual non-go compile actions, as well as the
cgoCompileActor's Act function. And the final part, which runs
cgo -dynimport is now in processCgoOutputs. These parts communicate with
each other through the providers that are set on the cgo actions.

One further improvement we can make to this change in the future is to
compile the dynimport file separately from the build action: its output
is only needed by the linker. This would remove any dependencies from
dependent packages' build actions on the cgo compile actions, allowing
more flexibility for scheduling actions.

Fixes #9887

Change-Id: Ie3c70bbf985148ba73094cddfc78c39dc6faad6e
Reviewed-on: https://go-review.googlesource.com/c/go/+/694475
Reviewed-by: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Matloob <matloob@google.com>

src/cmd/go/internal/work/action.go

@@ -19,6 +19,7 @@ import (
 	"internal/platform"
 	"os"
 	"path/filepath"
+	"slices"
 	"strings"
 	"sync"
 	"time"
@@ -578,6 +579,47 @@ func (ca *coverActor) Act(b *Builder, ctx context.Context, a *Action) error {
 	return nil
 }
 
+// runCgoActor implements the Actor interface for running the cgo command for the package.
+type runCgoActor struct {
+}
+
+func (c runCgoActor) Act(b *Builder, ctx context.Context, a *Action) error {
+	var cacheProvider *checkCacheProvider
+	for _, a1 := range a.Deps {
+		if pr, ok := a1.Provider.(*checkCacheProvider); ok {
+			cacheProvider = pr
+			break
+		}
+	}
+	need := cacheProvider.need
+	need &^= needCovMetaFile // handled by cover action
+	if need == 0 {
+		return nil
+	}
+	return b.runCgo(ctx, a)
+}
+
+type cgoCompileActor struct {
+	file string
+
+	compileFunc  func(*Action, string, string, []string, string) error
+	getFlagsFunc func(*runCgoProvider) []string
+
+	flags *[]string
+}
+
+func (c cgoCompileActor) Act(b *Builder, ctx context.Context, a *Action) error {
+	pr, ok := a.Deps[0].Provider.(*runCgoProvider)
+	if !ok {
+		return nil // cgo was not needed. do nothing.
+	}
+	a.nonGoOverlay = pr.nonGoOverlay
+	buildAction := a.triggers[0].triggers[0] // cgo compile -> cgo collect -> build
+
+	a.actionID = cache.Subkey(buildAction.actionID, "cgo compile "+c.file) // buildAction's action id was computed by the check cache action.
+	return c.compileFunc(a, a.Objdir, a.Target, c.getFlagsFunc(pr), c.file)
+}
+
 // CompileAction returns the action for compiling and possibly installing
 // (according to mode) the given package. The resulting action is only
 // for building packages (archives), never for linking executables.
@@ -677,6 +719,17 @@ func (b *Builder) CompileAction(mode, depMode BuildMode, p *load.Package) *Actio
 			a.Deps = append(a.Deps, coverAction)
 		}
 
+		// Create actions to run swig and cgo if needed. These actions always run in the
+		// same go build invocation as the build action and their actions are not cached
+		// separately, so they can use the same objdir.
+		if p.UsesCgo() || p.UsesSwig() {
+			deps := []*Action{cacheAction}
+			if coverAction != nil {
+				deps = append(deps, coverAction)
+			}
+			a.Deps = append(a.Deps, b.cgoAction(p, a.Objdir, deps, coverAction != nil))
+		}
+
 		return a
 	})
 
@@ -701,6 +754,114 @@ func (b *Builder) CompileAction(mode, depMode BuildMode, p *load.Package) *Actio
 	return a
 }
 
+func (b *Builder) cgoAction(p *load.Package, objdir string, deps []*Action, hasCover bool) *Action {
+	cgoCollectAction := b.cacheAction("cgo collect", p, func() *Action {
+		// Run cgo
+		runCgo := b.cacheAction("cgo run", p, func() *Action {
+			return &Action{
+				Package: p,
+				Mode:    "cgo run",
+				Actor:   &runCgoActor{},
+				Objdir:  objdir,
+				Deps:    deps,
+			}
+		})
+
+		// Determine which files swig will produce in the cgo run action. We'll need to create
+		// actions to compile the C and C++ files produced by swig, as well as the C file
+		// produced by cgo processing swig's Go file outputs.
+		swigGo, swigC, swigCXX := b.swigOutputs(p, objdir)
+
+		oseq := 0
+		nextOfile := func() string {
+			oseq++
+			return objdir + fmt.Sprintf("_x%03d.o", oseq)
+		}
+		compileAction := func(file string, getFlagsFunc func(*runCgoProvider) []string, compileFunc func(*Action, string, string, []string, string) error) *Action {
+			mode := "cgo compile " + file
+			return b.cacheAction(mode, p, func() *Action {
+				return &Action{
+					Package: p,
+					Mode:    mode,
+					Actor:   &cgoCompileActor{file: file, getFlagsFunc: getFlagsFunc, compileFunc: compileFunc},
+					Deps:    []*Action{runCgo},
+					Objdir:  objdir,
+					Target:  nextOfile(),
+				}
+			})
+		}
+
+		var collectDeps []*Action
+
+		// Add compile actions for C files generated by cgo.
+		cgoFiles := p.CgoFiles
+		if hasCover {
+			cgoFiles = slices.Clone(cgoFiles)
+			for i := range cgoFiles {
+				cgoFiles[i] = strings.TrimSuffix(cgoFiles[i], ".go") + ".cover.go"
+			}
+		}
+		cfiles := []string{"_cgo_export.c"}
+		for _, fn := range slices.Concat(cgoFiles, swigGo) {
+			cfiles = append(cfiles, strings.TrimSuffix(filepath.Base(fn), ".go")+".cgo2.c")
+		}
+		for _, f := range cfiles {
+			collectDeps = append(collectDeps, compileAction(objdir+f, (*runCgoProvider).cflags, b.gcc))
+		}
+
+		// Add compile actions for S files.
+		var sfiles []string
+		// In a package using cgo, cgo compiles the C, C++ and assembly files with gcc.
+		// There is one exception: runtime/cgo's job is to bridge the
+		// cgo and non-cgo worlds, so it necessarily has files in both.
+		// In that case gcc only gets the gcc_* files.
+		if p.Standard && p.ImportPath == "runtime/cgo" {
+			for _, f := range p.SFiles {
+				if strings.HasPrefix(f, "gcc_") {
+					sfiles = append(sfiles, f)
+				}
+			}
+		} else {
+			sfiles = p.SFiles
+		}
+		for _, f := range sfiles {
+			collectDeps = append(collectDeps, compileAction(f, (*runCgoProvider).cflags, b.gas))
+		}
+
+		// Add compile actions for C files in the package, M files, and those generated by swig.
+		for _, f := range slices.Concat(p.CFiles, p.MFiles, swigC) {
+			collectDeps = append(collectDeps, compileAction(f, (*runCgoProvider).cflags, b.gcc))
+		}
+
+		// Add compile actions for C++ files in the package, and those generated by swig.
+		for _, f := range slices.Concat(p.CXXFiles, swigCXX) {
+			collectDeps = append(collectDeps, compileAction(f, (*runCgoProvider).cxxflags, b.gxx))
+		}
+
+		// Add compile actions for Fortran files in the package.
+		for _, f := range p.FFiles {
+			collectDeps = append(collectDeps, compileAction(f, (*runCgoProvider).fflags, b.gfortran))
+		}
+
+		// Add a single convenience action that does nothing to join the previous action,
+		// and better separate the cgo action dependencies of the build action from the
+		// build actions for its package dependencies.
+		return &Action{
+			Mode: "collect cgo",
+			Actor: ActorFunc(func(b *Builder, ctx context.Context, a *Action) error {
+				// Use the cgo run action's provider as our provider output,
+				// so it can be easily accessed by the build action.
+				a.Provider = a.Deps[0].Deps[0].Provider
+				return nil
+			}),
+			Deps:   collectDeps,
+			Objdir: objdir,
+		}
+	})
+
+	return cgoCollectAction
+}
+
 // VetAction returns the action for running go vet on package p.
 // It depends on the action for compiling p.
 // If the caller may be causing p to be installed, it is up to the caller

src/cmd/go/internal/work/exec.go

@@ -464,6 +464,38 @@ func allowedVersion(v string) bool {
 	return gover.Compare(gover.Local(), v) >= 0
 }
 
+func (b *Builder) computeNonGoOverlay(a *Action, p *load.Package, sh *Shell, objdir string, nonGoFileLists [][]string) error {
+OverlayLoop:
+	for _, fs := range nonGoFileLists {
+		for _, f := range fs {
+			if fsys.Replaced(mkAbs(p.Dir, f)) {
+				a.nonGoOverlay = make(map[string]string)
+				break OverlayLoop
+			}
+		}
+	}
+	if a.nonGoOverlay != nil {
+		for _, fs := range nonGoFileLists {
+			for i := range fs {
+				from := mkAbs(p.Dir, fs[i])
+				dst := objdir + filepath.Base(fs[i])
+				if err := sh.CopyFile(dst, fsys.Actual(from), 0666, false); err != nil {
+					return err
+				}
+				a.nonGoOverlay[from] = dst
+			}
+		}
+	}
+
+	return nil
+}
+
+// needsBuild reports whether the Action (which must be mode "build") needs
+// to produce the built output.
+func (b *Builder) needsBuild(a *Action) bool {
+	return !b.IsCmdList && a.needBuild || b.NeedExport
+}
+
 const (
 	needBuild uint32 = 1 << iota
 	needCgoHdr
@@ -688,12 +720,15 @@ func (b *Builder) build(ctx context.Context, a *Action) (err error) {
 
 	var cacheProvider *checkCacheProvider
 	var coverPr *coverProvider
+	var runCgoPr *runCgoProvider
 	for _, dep := range a.Deps {
 		switch pr := dep.Provider.(type) {
 		case *coverProvider:
 			coverPr = pr
 		case *checkCacheProvider:
 			cacheProvider = pr
+		case *runCgoProvider:
+			runCgoPr = pr
 		}
 	}
 	if cacheProvider == nil {
@@ -702,6 +737,7 @@ func (b *Builder) build(ctx context.Context, a *Action) (err error) {
 
 	need := cacheProvider.need
 	need &^= needCovMetaFile // handled by cover action
+	need &^= needCgoHdr      // handled by run cgo action // TODO: accumulate "negative" need bits from actions
 
 	if need == 0 {
 		return
@@ -736,107 +772,40 @@ func (b *Builder) build(ctx context.Context, a *Action) (err error) {
 	}
 
 	gofiles := str.StringList(p.GoFiles)
-	cgofiles := str.StringList(p.CgoFiles)
 	cfiles := str.StringList(p.CFiles)
 	sfiles := str.StringList(p.SFiles)
-	cxxfiles := str.StringList(p.CXXFiles)
+	var objects, cgoObjects []string
-	var objects, cgoObjects, pcCFLAGS, pcLDFLAGS []string
-
-	if p.UsesCgo() || p.UsesSwig() {
-		if pcCFLAGS, pcLDFLAGS, err = b.getPkgConfigFlags(a); err != nil {
-			return
-		}
-	}
-
-	// Compute overlays for .c/.cc/.h/etc. and if there are any overlays
-	// put correct contents of all those files in the objdir, to ensure
-	// the correct headers are included. nonGoOverlay is the overlay that
-	// points from nongo files to the copied files in objdir.
-	nonGoFileLists := [][]string{p.CFiles, p.SFiles, p.CXXFiles, p.HFiles, p.FFiles}
-OverlayLoop:
-	for _, fs := range nonGoFileLists {
-		for _, f := range fs {
-			if fsys.Replaced(mkAbs(p.Dir, f)) {
-				a.nonGoOverlay = make(map[string]string)
-				break OverlayLoop
-			}
-		}
-	}
-	if a.nonGoOverlay != nil {
-		for _, fs := range nonGoFileLists {
-			for i := range fs {
-				from := mkAbs(p.Dir, fs[i])
-				dst := objdir + filepath.Base(fs[i])
-				if err := sh.CopyFile(dst, fsys.Actual(from), 0666, false); err != nil {
-					return err
-				}
-				a.nonGoOverlay[from] = dst
-			}
-		}
-	}
 
 	// If we're doing coverage, preprocess the .go files and put them in the work directory
 	if p.Internal.Cover.Mode != "" {
 		gofiles = coverPr.goSources
-		cgofiles = coverPr.cgoSources
 	}
 
-	// Run SWIG on each .swig and .swigcxx file.
-	// Each run will generate two files, a .go file and a .c or .cxx file.
-	// The .go file will use import "C" and is to be processed by cgo.
-	// For -cover test or build runs, this needs to happen after the cover
-	// tool is run; we don't want to instrument swig-generated Go files,
-	// see issue #64661.
-	if p.UsesSwig() {
-		outGo, outC, outCXX, err := b.swig(a, objdir, pcCFLAGS)
-		if err != nil {
-			return err
-		}
-		cgofiles = append(cgofiles, outGo...)
-		cfiles = append(cfiles, outC...)
-		cxxfiles = append(cxxfiles, outCXX...)
-	}
-
-	// Run cgo.
 	if p.UsesCgo() || p.UsesSwig() {
+		if runCgoPr == nil {
+			base.Fatalf("internal error: could not find runCgoProvider")
+		}
+
 		// In a package using cgo, cgo compiles the C, C++ and assembly files with gcc.
 		// There is one exception: runtime/cgo's job is to bridge the
 		// cgo and non-cgo worlds, so it necessarily has files in both.
 		// In that case gcc only gets the gcc_* files.
-		var gccfiles []string
-		gccfiles = append(gccfiles, cfiles...)
 		cfiles = nil
 		if p.Standard && p.ImportPath == "runtime/cgo" {
-			filter := func(files, nongcc, gcc []string) ([]string, []string) {
+			// filter to the non-gcc files.
-				for _, f := range files {
+			i := 0
-					if strings.HasPrefix(f, "gcc_") {
+			for _, f := range sfiles {
-						gcc = append(gcc, f)
+				if !strings.HasPrefix(f, "gcc_") {
+					sfiles[i] = f
+					i++
+				}
+			}
+			sfiles = sfiles[:i]
 		} else {
-						nongcc = append(nongcc, f)
-					}
-				}
-				return nongcc, gcc
-			}
-			sfiles, gccfiles = filter(sfiles, sfiles[:0], gccfiles)
-		} else {
-			for _, sfile := range sfiles {
-				data, err := os.ReadFile(filepath.Join(p.Dir, sfile))
-				if err == nil {
-					if bytes.HasPrefix(data, []byte("TEXT")) || bytes.Contains(data, []byte("\nTEXT")) ||
-						bytes.HasPrefix(data, []byte("DATA")) || bytes.Contains(data, []byte("\nDATA")) ||
-						bytes.HasPrefix(data, []byte("GLOBL")) || bytes.Contains(data, []byte("\nGLOBL")) {
-						return fmt.Errorf("package using cgo has Go assembly file %s", sfile)
-					}
-				}
-			}
-			gccfiles = append(gccfiles, sfiles...)
 			sfiles = nil
 		}
 
-		outGo, outObj, err := b.cgo(a, base.Tool("cgo"), objdir, pcCFLAGS, pcLDFLAGS, mkAbsFiles(p.Dir, cgofiles), gccfiles, cxxfiles, p.MFiles, p.FFiles)
+		outGo, outObj, err := b.processCgoOutputs(a, runCgoPr, base.Tool("cgo"), objdir)
-
-		// The files in cxxfiles have now been handled by b.cgo.
-		cxxfiles = nil
 
 		if err != nil {
 			return err
@@ -857,12 +826,8 @@ OverlayLoop:
 	srcfiles = append(srcfiles, gofiles...)
 	srcfiles = append(srcfiles, sfiles...)
 	srcfiles = append(srcfiles, cfiles...)
-	srcfiles = append(srcfiles, cxxfiles...)
 	b.cacheSrcFiles(a, srcfiles)
 
-	// Running cgo generated the cgo header.
-	need &^= needCgoHdr
-
 	// Sanity check only, since Package.load already checked as well.
 	if len(gofiles) == 0 {
 		return &load.NoGoError{Package: p}
@@ -991,6 +956,12 @@ OverlayLoop:
 		}
 	}
 
+	if err := b.computeNonGoOverlay(a, p, sh, objdir, [][]string{cfiles}); err != nil {
+		return err
+	}
+
+	// Compile C files in a package being built with gccgo. We disallow
+	// C files when compiling with gc unless swig or cgo is used.
 	for _, file := range cfiles {
 		out := file[:len(file)-len(".c")] + ".o"
 		if err := BuildToolchain.cc(b, a, objdir+out, file); err != nil {
@@ -1694,8 +1665,7 @@ func splitPkgConfigOutput(out []byte) ([]string, error) {
 }
 
 // Calls pkg-config if needed and returns the cflags/ldflags needed to build a's package.
-func (b *Builder) getPkgConfigFlags(a *Action) (cflags, ldflags []string, err error) {
+func (b *Builder) getPkgConfigFlags(a *Action, p *load.Package) (cflags, ldflags []string, err error) {
-	p := a.Package
 	sh := b.Shell(a)
 	if pcargs := p.CgoPkgConfig; len(pcargs) > 0 {
 		// pkg-config permits arguments to appear anywhere in
@@ -2172,6 +2142,20 @@ func (b *Builder) gcc(a *Action, workdir, out string, flags []string, cfile stri
 	return b.ccompile(a, out, flags, cfile, b.GccCmd(p.Dir, workdir))
 }
 
+// gas runs the gcc c compiler to create an object file from a single C assembly file.
+func (b *Builder) gas(a *Action, workdir, out string, flags []string, sfile string) error {
+	p := a.Package
+	data, err := os.ReadFile(filepath.Join(p.Dir, sfile))
+	if err == nil {
+		if bytes.HasPrefix(data, []byte("TEXT")) || bytes.Contains(data, []byte("\nTEXT")) ||
+			bytes.HasPrefix(data, []byte("DATA")) || bytes.Contains(data, []byte("\nDATA")) ||
+			bytes.HasPrefix(data, []byte("GLOBL")) || bytes.Contains(data, []byte("\nGLOBL")) {
+			return fmt.Errorf("package using cgo has Go assembly file %s", sfile)
+		}
+	}
+	return b.ccompile(a, out, flags, sfile, b.GccCmd(p.Dir, workdir))
+}
+
 // gxx runs the g++ C++ compiler to create an object from a single C++ file.
 func (b *Builder) gxx(a *Action, workdir, out string, flags []string, cxxfile string) error {
 	p := a.Package
@@ -2191,6 +2175,8 @@ func (b *Builder) ccompile(a *Action, outfile string, flags []string, file strin
 	file = mkAbs(p.Dir, file)
 	outfile = mkAbs(p.Dir, outfile)
 
+	flags = slices.Clip(flags) // If we append to flags, write to a new slice that we own.
+
 	// Elide source directory paths if -trimpath is set.
 	// This is needed for source files (e.g., a .c file in a package directory).
 	// TODO(golang.org/issue/36072): cgo also generates files with #line
@@ -2741,26 +2727,97 @@ func buildFlags(name, defaults string, fromPackage []string, check func(string,
 
 var cgoRe = lazyregexp.New(`[/\\:]`)
 
-func (b *Builder) cgo(a *Action, cgoExe, objdir string, pcCFLAGS, pcLDFLAGS, cgofiles, gccfiles, gxxfiles, mfiles, ffiles []string) (outGo, outObj []string, err error) {
+type runCgoProvider struct {
+	CFLAGS, CXXFLAGS, FFLAGS, LDFLAGS []string
+	notCompatibleForInternalLinking   bool
+	nonGoOverlay                      map[string]string
+	goFiles                           []string // processed cgo files for the compiler
+}
+
+func (pr *runCgoProvider) cflags() []string {
+	return pr.CFLAGS
+}
+
+func (pr *runCgoProvider) cxxflags() []string {
+	return pr.CXXFLAGS
+}
+
+func (pr *runCgoProvider) fflags() []string {
+	return pr.CXXFLAGS
+}
+
+func (pr *runCgoProvider) ldflags() []string {
+	return pr.LDFLAGS
+}
+
+func mustGetCoverInfo(a *Action) *coverProvider {
+	for _, dep := range a.Deps {
+		if dep.Mode == "cover" {
+			return dep.Provider.(*coverProvider)
+		}
+	}
+	base.Fatalf("internal error: cover provider not found")
+	panic("unreachable")
+}
+
+func (b *Builder) runCgo(ctx context.Context, a *Action) error {
 	p := a.Package
 	sh := b.Shell(a)
+	objdir := a.Objdir
+
+	if err := sh.Mkdir(objdir); err != nil {
+		return err
+	}
+
+	nonGoFileLists := [][]string{p.CFiles, p.SFiles, p.CXXFiles, p.HFiles, p.FFiles}
+	if err := b.computeNonGoOverlay(a, p, sh, objdir, nonGoFileLists); err != nil {
+		return err
+	}
+
+	cgofiles := slices.Clip(p.CgoFiles)
+	if a.Package.Internal.Cover.Mode != "" {
+		cp := mustGetCoverInfo(a)
+		cgofiles = cp.cgoSources
+	}
+
+	pcCFLAGS, pcLDFLAGS, err := b.getPkgConfigFlags(a, p)
+	if err != nil {
+		return err
+	}
+
+	// Run SWIG on each .swig and .swigcxx file.
+	// Each run will generate two files, a .go file and a .c or .cxx file.
+	// The .go file will use import "C" and is to be processed by cgo.
+	// For -cover test or build runs, this needs to happen after the cover
+	// tool is run; we don't want to instrument swig-generated Go files,
+	// see issue #64661.
+	if p.UsesSwig() {
+		if err := b.swig(a, objdir, pcCFLAGS); err != nil {
+			return err
+		}
+		outGo, _, _ := b.swigOutputs(p, objdir)
+		cgofiles = append(cgofiles, outGo...)
+	}
+
+	cgoExe := base.Tool("cgo")
+	cgofiles = mkAbsFiles(p.Dir, cgofiles)
 
 	cgoCPPFLAGS, cgoCFLAGS, cgoCXXFLAGS, cgoFFLAGS, cgoLDFLAGS, err := b.CFlags(p)
 	if err != nil {
-		return nil, nil, err
+		return err
 	}
 
 	cgoCPPFLAGS = append(cgoCPPFLAGS, pcCFLAGS...)
 	cgoLDFLAGS = append(cgoLDFLAGS, pcLDFLAGS...)
 	// If we are compiling Objective-C code, then we need to link against libobjc
-	if len(mfiles) > 0 {
+	if len(p.MFiles) > 0 {
 		cgoLDFLAGS = append(cgoLDFLAGS, "-lobjc")
 	}
 
 	// Likewise for Fortran, except there are many Fortran compilers.
 	// Support gfortran out of the box and let others pass the correct link options
 	// via CGO_LDFLAGS
-	if len(ffiles) > 0 {
+	if len(p.FFiles) > 0 {
 		fc := cfg.Getenv("FC")
 		if fc == "" {
 			fc = "gfortran"
@@ -2787,13 +2844,7 @@ func (b *Builder) cgo(a *Action, cgoExe, objdir string, pcCFLAGS, pcLDFLAGS, cgo
 	// #58620, and #58848.
 	flagSources := []string{"CGO_CFLAGS", "CGO_CXXFLAGS", "CGO_FFLAGS"}
 	flagLists := [][]string{cgoCFLAGS, cgoCXXFLAGS, cgoFFLAGS}
-	if flagsNotCompatibleWithInternalLinking(flagSources, flagLists) {
+	notCompatibleWithInternalLinking := flagsNotCompatibleWithInternalLinking(flagSources, flagLists)
-		tokenFile := objdir + "preferlinkext"
-		if err := sh.writeFile(tokenFile, nil); err != nil {
-			return nil, nil, err
-		}
-		outObj = append(outObj, tokenFile)
-	}
 
 	if cfg.BuildMSan {
 		cgoCFLAGS = append([]string{"-fsanitize=memory"}, cgoCFLAGS...)
@@ -2811,11 +2862,11 @@ func (b *Builder) cgo(a *Action, cgoExe, objdir string, pcCFLAGS, pcLDFLAGS, cgo
 	// cgo
 	// TODO: CGO_FLAGS?
 	gofiles := []string{objdir + "_cgo_gotypes.go"}
-	cfiles := []string{"_cgo_export.c"}
+	cfiles := []string{objdir + "_cgo_export.c"}
 	for _, fn := range cgofiles {
 		f := strings.TrimSuffix(filepath.Base(fn), ".go")
 		gofiles = append(gofiles, objdir+f+".cgo1.go")
-		cfiles = append(cfiles, f+".cgo2.c")
+		cfiles = append(cfiles, objdir+f+".cgo2.c")
 	}
 
 	// TODO: make cgo not depend on $GOARCH?
@@ -2889,70 +2940,61 @@ func (b *Builder) cgo(a *Action, cgoExe, objdir string, pcCFLAGS, pcLDFLAGS, cgo
 	}
 
 	if err := sh.run(p.Dir, p.ImportPath, cgoenv, cfg.BuildToolexec, cgoExe, "-objdir", objdir, "-importpath", p.ImportPath, cgoflags, ldflagsOption, "--", cgoCPPFLAGS, cgoCFLAGS, cgofiles); err != nil {
-		return nil, nil, err
+		return err
-	}
-	outGo = append(outGo, gofiles...)
-
-	// Use sequential object file names to keep them distinct
-	// and short enough to fit in the .a header file name slots.
-	// We no longer collect them all into _all.o, and we'd like
-	// tools to see both the .o suffix and unique names, so
-	// we need to make them short enough not to be truncated
-	// in the final archive.
-	oseq := 0
-	nextOfile := func() string {
-		oseq++
-		return objdir + fmt.Sprintf("_x%03d.o", oseq)
 	}
 
-	// gcc
+	a.Provider = &runCgoProvider{
-	cflags := str.StringList(cgoCPPFLAGS, cgoCFLAGS)
+		CFLAGS:                          str.StringList(cgoCPPFLAGS, cgoCFLAGS),
-	for _, cfile := range cfiles {
+		CXXFLAGS:                        str.StringList(cgoCPPFLAGS, cgoCXXFLAGS),
-		ofile := nextOfile()
+		FFLAGS:                          str.StringList(cgoCPPFLAGS, cgoFFLAGS),
-		if err := b.gcc(a, a.Objdir, ofile, cflags, objdir+cfile); err != nil {
+		LDFLAGS:                         cgoLDFLAGS,
-			return nil, nil, err
+		notCompatibleForInternalLinking: notCompatibleWithInternalLinking,
-		}
+		nonGoOverlay:                    a.nonGoOverlay,
-		outObj = append(outObj, ofile)
+		goFiles:                         gofiles,
 	}
 
-	for _, file := range gccfiles {
+	return nil
-		ofile := nextOfile()
-		if err := b.gcc(a, a.Objdir, ofile, cflags, file); err != nil {
-			return nil, nil, err
-		}
-		outObj = append(outObj, ofile)
 }
 
-	cxxflags := str.StringList(cgoCPPFLAGS, cgoCXXFLAGS)
+func (b *Builder) processCgoOutputs(a *Action, runCgoProvider *runCgoProvider, cgoExe, objdir string) (outGo, outObj []string, err error) {
-	for _, file := range gxxfiles {
+	outGo = slices.Clip(runCgoProvider.goFiles)
-		ofile := nextOfile()
+
-		if err := b.gxx(a, a.Objdir, ofile, cxxflags, file); err != nil {
+	// TODO(matloob): Pretty much the only thing this function is doing is
+	// producing the dynimport go files. But we should be able to compile
+	// those separately from the package itself: we just need to get the
+	// compiled output to the linker. That means that we can remove the
+	// dependency of this build action on the outputs of the cgo compile actions
+	// (though we'd still need to depend on the runCgo action of course).
+
+	sh := b.Shell(a)
+
+	// Output the preferlinkext file if the run cgo action determined this package
+	// was not compatible for internal linking based on CFLAGS, CXXFLAGS, or FFLAGS.
+	if runCgoProvider.notCompatibleForInternalLinking {
+		tokenFile := objdir + "preferlinkext"
+		if err := sh.writeFile(tokenFile, nil); err != nil {
 			return nil, nil, err
 		}
-		outObj = append(outObj, ofile)
+		outObj = append(outObj, tokenFile)
 	}
 
-	for _, file := range mfiles {
+	var collectAction *Action
-		ofile := nextOfile()
+	for _, dep := range a.Deps {
-		if err := b.gcc(a, a.Objdir, ofile, cflags, file); err != nil {
+		if dep.Mode == "collect cgo" {
-			return nil, nil, err
+			collectAction = dep
 		}
-		outObj = append(outObj, ofile)
 	}
-
+	if collectAction == nil {
-	fflags := str.StringList(cgoCPPFLAGS, cgoFFLAGS)
+		base.Fatalf("internal error: no cgo collect action")
-	for _, file := range ffiles {
-		ofile := nextOfile()
-		if err := b.gfortran(a, a.Objdir, ofile, fflags, file); err != nil {
-			return nil, nil, err
 	}
-		outObj = append(outObj, ofile)
+	for _, dep := range collectAction.Deps {
+		outObj = append(outObj, dep.Target)
 	}
 
 	switch cfg.BuildToolchainName {
 	case "gc":
 		importGo := objdir + "_cgo_import.go"
-		dynOutGo, dynOutObj, err := b.dynimport(a, objdir, importGo, cgoExe, cflags, cgoLDFLAGS, outObj)
+		dynOutGo, dynOutObj, err := b.dynimport(a, objdir, importGo, cgoExe, runCgoProvider.CFLAGS, runCgoProvider.LDFLAGS, outObj)
 		if err != nil {
 			return nil, nil, err
 		}
@@ -3031,16 +3073,16 @@ func (b *Builder) cgo(a *Action, cgoExe, objdir string, pcCFLAGS, pcLDFLAGS, cgo
 			}
 		}
 
-		// We expect to find the contents of cgoLDFLAGS in flags.
+		// We expect to find the contents of cgoLDFLAGS used when running the CGO action in flags.
-		if len(cgoLDFLAGS) > 0 {
+		if len(runCgoProvider.LDFLAGS) > 0 {
 		outer:
 			for i := range flags {
-				for j, f := range cgoLDFLAGS {
+				for j, f := range runCgoProvider.LDFLAGS {
 					if f != flags[i+j] {
 						continue outer
 					}
 				}
-				flags = append(flags[:i], flags[i+len(cgoLDFLAGS):]...)
+				flags = append(flags[:i], flags[i+len(runCgoProvider.LDFLAGS):]...)
 				break
 			}
 		}
@@ -3156,43 +3198,43 @@ func (b *Builder) dynimport(a *Action, objdir, importGo, cgoExe string, cflags,
 // Run SWIG on all SWIG input files.
 // TODO: Don't build a shared library, once SWIG emits the necessary
 // pragmas for external linking.
-func (b *Builder) swig(a *Action, objdir string, pcCFLAGS []string) (outGo, outC, outCXX []string, err error) {
+func (b *Builder) swig(a *Action, objdir string, pcCFLAGS []string) error {
 	p := a.Package
 
 	if err := b.swigVersionCheck(); err != nil {
-		return nil, nil, nil, err
+		return err
 	}
 
 	intgosize, err := b.swigIntSize(objdir)
 	if err != nil {
-		return nil, nil, nil, err
+		return err
 	}
 
 	for _, f := range p.SwigFiles {
-		goFile, cFile, err := b.swigOne(a, f, objdir, pcCFLAGS, false, intgosize)
+		if err := b.swigOne(a, f, objdir, pcCFLAGS, false, intgosize); err != nil {
-		if err != nil {
+			return err
-			return nil, nil, nil, err
-		}
-		if goFile != "" {
-			outGo = append(outGo, goFile)
-		}
-		if cFile != "" {
-			outC = append(outC, cFile)
 		}
 	}
 	for _, f := range p.SwigCXXFiles {
-		goFile, cxxFile, err := b.swigOne(a, f, objdir, pcCFLAGS, true, intgosize)
+		if b.swigOne(a, f, objdir, pcCFLAGS, true, intgosize); err != nil {
-		if err != nil {
+			return err
-			return nil, nil, nil, err
 		}
-		if goFile != "" {
+	}
+	return nil
+}
+
+func (b *Builder) swigOutputs(p *load.Package, objdir string) (outGo, outC, outCXX []string) {
+	for _, f := range p.SwigFiles {
+		goFile, cFile := swigOneOutputs(f, objdir, false)
 		outGo = append(outGo, goFile)
+		outC = append(outC, cFile)
 	}
-		if cxxFile != "" {
+	for _, f := range p.SwigCXXFiles {
+		goFile, cxxFile := swigOneOutputs(f, objdir, true)
+		outGo = append(outGo, goFile)
 		outCXX = append(outCXX, cxxFile)
 	}
-	}
+	return outGo, outC, outCXX
-	return outGo, outC, outCXX, nil
 }
 
 // Make sure SWIG is new enough.
@@ -3305,13 +3347,13 @@ func (b *Builder) swigIntSize(objdir string) (intsize string, err error) {
 }
 
 // Run SWIG on one SWIG input file.
-func (b *Builder) swigOne(a *Action, file, objdir string, pcCFLAGS []string, cxx bool, intgosize string) (outGo, outC string, err error) {
+func (b *Builder) swigOne(a *Action, file, objdir string, pcCFLAGS []string, cxx bool, intgosize string) error {
 	p := a.Package
 	sh := b.Shell(a)
 
 	cgoCPPFLAGS, cgoCFLAGS, cgoCXXFLAGS, _, _, err := b.CFlags(p)
 	if err != nil {
-		return "", "", err
+		return err
 	}
 
 	var cflags []string
@@ -3321,17 +3363,8 @@ func (b *Builder) swigOne(a *Action, file, objdir string, pcCFLAGS []string, cxx
 		cflags = str.StringList(cgoCPPFLAGS, pcCFLAGS, cgoCFLAGS)
 	}
 
-	n := 5 // length of ".swig"
+	base := swigBase(file, cxx)
-	if cxx {
+	newGoFile, outC := swigOneOutputs(file, objdir, cxx)
-		n = 8 // length of ".swigcxx"
-	}
-	base := file[:len(file)-n]
-	goFile := base + ".go"
-	gccBase := base + "_wrap."
-	gccExt := "c"
-	if cxx {
-		gccExt = "cxx"
-	}
 
 	gccgo := cfg.BuildToolchainName == "gccgo"
 
@@ -3341,7 +3374,7 @@ func (b *Builder) swigOne(a *Action, file, objdir string, pcCFLAGS []string, cxx
 		"-cgo",
 		"-intgosize", intgosize,
 		"-module", base,
-		"-o", objdir + gccBase + gccExt,
+		"-o", outC,
 		"-outdir", objdir,
 	}
 
@@ -3363,31 +3396,52 @@ func (b *Builder) swigOne(a *Action, file, objdir string, pcCFLAGS []string, cxx
 
 	out, err := sh.runOut(p.Dir, nil, "swig", args, file)
 	if err != nil && (bytes.Contains(out, []byte("-intgosize")) || bytes.Contains(out, []byte("-cgo"))) {
-		return "", "", errors.New("must have SWIG version >= 3.0.6")
+		return errors.New("must have SWIG version >= 3.0.6")
 	}
 	if err := sh.reportCmd("", "", out, err); err != nil {
-		return "", "", err
+		return err
 	}
 
 	// If the input was x.swig, the output is x.go in the objdir.
 	// But there might be an x.go in the original dir too, and if it
 	// uses cgo as well, cgo will be processing both and will
 	// translate both into x.cgo1.go in the objdir, overwriting one.
-	// Rename x.go to _x_swig.go to avoid this problem.
+	// Rename x.go to _x_swig.go (newGoFile) to avoid this problem.
 	// We ignore files in the original dir that begin with underscore
 	// so _x_swig.go cannot conflict with an original file we were
 	// going to compile.
-	goFile = objdir + goFile
+	goFile := objdir + base + ".go"
-	newGoFile := objdir + "_" + base + "_swig.go"
 	if cfg.BuildX || cfg.BuildN {
 		sh.ShowCmd("", "mv %s %s", goFile, newGoFile)
 	}
 	if !cfg.BuildN {
 		if err := os.Rename(goFile, newGoFile); err != nil {
-			return "", "", err
+			return err
 		}
 	}
-	return newGoFile, objdir + gccBase + gccExt, nil
+
+	return nil
+}
+
+func swigBase(file string, cxx bool) string {
+	n := 5 // length of ".swig"
+	if cxx {
+		n = 8 // length of ".swigcxx"
+	}
+	return file[:len(file)-n]
+}
+
+func swigOneOutputs(file, objdir string, cxx bool) (outGo, outC string) {
+	base := swigBase(file, cxx)
+	gccBase := base + "_wrap."
+	gccExt := "c"
+	if cxx {
+		gccExt = "cxx"
+	}
+
+	newGoFile := objdir + "_" + base + "_swig.go"
+	cFile := objdir + gccBase + gccExt
+	return newGoFile, cFile
 }
 
 // disableBuildID adjusts a linker command line to avoid creating a