cmd/dist: make toolchain build reproducible

- Build cmd with CGO_ENABLED=0. Doing so removes the C compiler
  toolchain from the reproducibility perimeter and also results in
  cmd/go and cmd/pprof binaries that are statically linked,
  so that they will run on a wider variety of systems.
  In particular the Linux versions will run on Alpine and NixOS
  without needing a simulation of libc.so.6.

  The potential downside of disabling cgo is that cmd/go and cmd/pprof
  use the pure Go network resolver instead of the host resolver on
  Unix systems. This means they will not be able to use non-DNS
  resolver mechanisms that may be specified in /etc/resolv.conf,
  such as mDNS. Neither program seems likely to need non-DNS names
  like those, however.

  macOS and Windows systems still use the host resolver, which they
  access without cgo.

- Build cmd with -trimpath when building a release.
  Doing so removes $GOPATH from the file name prefixes stored in the
  binary, so that the build directory does not leak into the final artifacts.

- When CC and CXX are empty, do not pick values to hard-code into
  the source tree and binaries. Instead, emit code that makes the
  right decision at runtime. In addition to reproducibility, this
  makes cross-compiled toolchains work better. A macOS toolchain
  cross-compiled on Linux will now correctly look for clang,
  instead of looking for gcc because it was built on Linux.

- Convert \ to / in file names stored in .a files.
  These are converted to / in the final binaries, but the hashes of
  the .a files affect the final build ID of the binaries. Without this
  change, builds of a Windows toolchain on Windows and non-Windows
  machines produce identical binaries except for the input hash part
  of the build ID.

- Due to the conversion of \ to / in .a files, convert back when
  reading inline bodies on Windows to preserve output file names
  in error messages.

Combined, these four changes (along with Go 1.20's removal of
installed pkg/**.a files and conversion of macOS net away from cgo)
make the output of make.bash fully reproducible, even when
cross-compiling: a released macOS toolchain built on Linux or Windows
will contain exactly the same bits as a released macOS toolchain
built on macOS.

The word "released" in the previous sentence is important.
For the build IDs in the binaries to work out the same on
both systems, a VERSION file must exist to provide a consistent
compiler build ID (instead of using a content hash of the binary).

For #24904.
Fixes #57007.

Change-Id: I665e1ef4ff207d6ff469452347dca5bfc81050e6
Reviewed-on: https://go-review.googlesource.com/c/go/+/454836
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Russ Cox <rsc@golang.org>
Auto-Submit: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
This commit is contained in:
Russ Cox 2022-12-02 10:39:30 -05:00 committed by Gopher Robot
parent dbe327a640
commit 8a27154bcd
9 changed files with 165 additions and 67 deletions

View file

@ -9,6 +9,7 @@ import (
"go/constant"
"internal/buildcfg"
"internal/pkgbits"
"path/filepath"
"strings"
"cmd/compile/internal/base"
@ -268,13 +269,14 @@ func (pr *pkgReader) posBaseIdx(idx pkgbits.Index) *src.PosBase {
// "$GOROOT" to buildcfg.GOROOT is a close-enough approximation to
// satisfy this.
//
// TODO(mdempsky): De-duplicate this logic with similar logic in
// cmd/link/internal/ld's expandGoroot. However, this will probably
// require being more consistent about when we use native vs UNIX
// file paths.
// The export data format only ever uses slash paths
// (for cross-operating-system reproducible builds),
// but error messages need to use native paths (backslash on Windows)
// as if they had been specified on the command line.
// (The go command always passes native paths to the compiler.)
const dollarGOROOT = "$GOROOT"
if buildcfg.GOROOT != "" && strings.HasPrefix(filename, dollarGOROOT) {
filename = buildcfg.GOROOT + filename[len(dollarGOROOT):]
filename = filepath.FromSlash(buildcfg.GOROOT + filename[len(dollarGOROOT):])
}
if r.Bool() {