Commit graph

976 commits

Author SHA1 Message Date
Jeremy Faller
664accc7be [dev.link] cmd/link: remove sym.Symbols from loader
Pushing the wavefront forward requires a single source of truth for
symbols. This CL removes sym.Symbols from the loader for host object
loaders, allowing us to have the single source of truth for symbols be
the loader, not some strange combination of sym.Symbols and the loader.

Change-Id: Ib8ff0117ebe6040fade346047535ff342d781081
Reviewed-on: https://go-review.googlesource.com/c/go/+/209217
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-12-12 20:57:54 +00:00
Than McIntosh
29f886e427 [dev.link] cmd/link: avoid allsyms loop in initarray setup
In the linker's symtab() function, avoid looping over the context's
Syms.Allsyms array to locate the entry symbol when setting up the init
array section; do an explicit ABI0 symbol lookup instead. This is a
minor efficiency tweak / code cleanup.

Fixes #20205.

Change-Id: I2ebc17a3cb2cd63e9f5052bc80f1b0ac72c960e3
Reviewed-on: https://go-review.googlesource.com/c/go/+/209838
Run-TryBot: Than McIntosh <thanm@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-12-04 18:39:51 +00:00
Cherry Zhang
f7672d39ca [dev.link] all: merge branch 'master' into dev.link
Bring in Than's fix of #35779.

The only merge conflict is cmd/link/internal/loadelf/ldelf.go,
with a modification-deletion conflict.

Change-Id: Id2fcfd2094a31120966a6ea9c462b4ec76646b10
2019-12-03 10:38:43 -05:00
Than McIntosh
e4c01f088b cmd/link: additional fixes for -newobj and "ld -r" ELF host objects
The previous fix for this issue (CL 208479) was not general enough;
this patch revises it to handle more cases.

The problem with the original fix was that once a sym.Symbol is
created for a given static symbol and given a bogus anonymous version
of -1, we hit problems if some other non-anonymous symbol (created by
host object loading) had relocations targeting the static symbol.

In this patch instead of assigning a fixed anonymous version of -1 to
such symbols, each time loader.Create is invoked we create a new
(unique) anonymous version for the sym.Symbol, then enter the result
into the loader's extStaticSyms map, permitting it to be found in
lookups when processing relocation targets.

NB: this code will hopefully get a lot simpler once we can move host
object loading away from early sym.Symbol creation.

Updates #35779.

Change-Id: I450ff577e17549025565d355d6707a2d28a5a617
Reviewed-on: https://go-review.googlesource.com/c/go/+/208778
Run-TryBot: Than McIntosh <thanm@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-12-02 19:05:01 +00:00
Than McIntosh
f29e53d664 cmd/link: fix bug with -newobj and "ld -r" ELF host objects
When the ELF host object loader encounters a static/hidden symbol, it
creates a sym.Symbol for it but does not enter it into the sym.Symbols
lookup table. Under -newobj mode, this was not happening correctly; we
were adding the sym via loader.LookupOrCreate, which resulted in
collisions when it encountered symbols with the same name + version +
section (this can happen for "ld -r" objects).

Fixes #35779.

Change-Id: I36d40fc1efc03fc1cd8ae6b76cb6a0d2a957389c
Reviewed-on: https://go-review.googlesource.com/c/go/+/208479
Run-TryBot: Than McIntosh <thanm@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-23 00:06:41 +00:00
Joel Sing
1046a9cb1b cmd/link,cmd/internal/obj/riscv: implement archreloc for riscv64
Based on the riscv-go port.

Updates #27532

Change-Id: I478254306441c253d3a2c09c10932ad1ac0be3c6
Reviewed-on: https://go-review.googlesource.com/c/go/+/204625
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-11-19 06:08:56 +00:00
Cherry Zhang
27c0aeee3f [dev.link] all: merge branch 'master' into dev.link
Clean merge.

Change-Id: I4a73cd4bac2f4d35d2c07c39f2bc01c6075fadab
2019-11-15 16:33:22 -05:00
Ville Skyttä
440f7d6404 all: fix a bunch of misspellings
Change-Id: I5b909df0fd048cd66c5a27fca1b06466d3bcaac7
GitHub-Last-Rev: 778c5d2131
GitHub-Pull-Request: golang/go#35624
Reviewed-on: https://go-review.googlesource.com/c/go/+/207421
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-11-15 21:04:43 +00:00
Elias Naur
d856e05d64 cmd/link/internal/ld: set MachO platform to PLATFORM_IOS on iOS
CL 205340 changed the linker to skip certain flags when linking
for iOS. However, for host linking on iOS (such as on the Corellium
darwin/arm64 builders) the MachO platform defaults to PLATFORM_MACOS,
and the flags are not skipped.

Avoids warnings such as

    ld: warning: -no_pie ignored for arm64

Updates #32963

Change-Id: Ib6b4c2375fd14cf89410bf5ff1537b692b7a1c15
Reviewed-on: https://go-review.googlesource.com/c/go/+/206337
Run-TryBot: Elias Naur <mail@eliasnaur.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-11-15 21:02:07 +00:00
Elias Naur
d3c2b1f176 cmd/link/internal/ld,misc/cgo/testcshared: don't -fuse-ld=gold on Android
The NDK is switching to ldd, and will stop including the gold linker.

Change-Id: If74168017c9874134b34010906ab1d94001528b6
Reviewed-on: https://go-review.googlesource.com/c/go/+/206840
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-11-14 17:43:48 +00:00
Joel Sing
bf5f64107a cmd/internal/objabi,cmd/link: initial linker support for riscv64
Provide initial linker support for riscv64.

Based on riscv-go port.

Updates #27532

Change-Id: I8a881ce41cd49efef0358bad9171d4d18aaf7ab2
Reviewed-on: https://go-review.googlesource.com/c/go/+/204624
Run-TryBot: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-13 08:07:51 +00:00
Cherry Zhang
62dfb27827 [dev.link] cmd/link: delete old object file format support
There are more cleanups to do, but I want to keep this CL mostly
a pure deletion.

Change-Id: I30f4891a2ea54545fd6b83041746ab65895537e1
Reviewed-on: https://go-review.googlesource.com/c/go/+/206558
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2019-11-12 19:59:04 +00:00
Jeremy Faller
aacffb6576 [dev.link] cmd/link: remove old objs from host obj loaders
Change-Id: I9ee853e37090ea015aba817d9aeb2f4cc9ff5a1b
Reviewed-on: https://go-review.googlesource.com/c/go/+/206539
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-11 20:19:37 +00:00
Than McIntosh
795e8c2317 cmd/go: address DWARF linker issues with -buildmode=plugin on Darwin
Assorted fixups in the linker needed to enable turning back on
DWARF generation when building plugins for Darwin. Includes:

 - don't suppress import of runtime/cgo in the linker for
   Darwin if we are linking in plugin mode

 - in calcCompUnitRanges handle the case where we encounter
   linker-generated functions that have no associated Unit (and
   also have no DWARF)

 - generalize a guard in relocsym() include so as to avoid
   triggering a spurious error on go.info symbols in plugin mode

Updates #21647.
Updates #27502.

Change-Id: I317fea97bef2f3461e31498e63f9fd6d8b8f4b23
Reviewed-on: https://go-review.googlesource.com/c/go/+/182959
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
2019-11-11 17:37:48 +00:00
Cherry Zhang
8cdb769b08 [dev.link] cmd: reenable newobj mode by default
Change-Id: I6e820a77c516363f350c2aad5bc17a2d7e7501e3
Reviewed-on: https://go-review.googlesource.com/c/go/+/206457
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-11-11 16:39:35 +00:00
Cherry Zhang
7cd858cbcb [dev.link] all: merge branch 'master' into dev.link
Clean merge.

Change-Id: Id03e6f00790ad52e0202111bbbc35f5c05567427
2019-11-11 10:56:15 -05:00
Gerrit Code Review
bababde766 Merge "cmd: merge branch 'dev.link' into master" 2019-11-08 20:24:43 +00:00
Than McIntosh
17722c21e7 [dev.link] cmd/link/internal/loader: reduce ABI alias postprocessing overhead
In loadObjFull when populating the sym.Reloc vector for live symbols,
avoid using the loader.SymType() method to determine if a relocation
targets an ABI alias; since invoking loader.SymType requires a
global-to-local index translation and a read from the object file.
Instead just look at the target symbol itself, which has already been
created at this point.

Hyperkube performance numbers for this change:

name                         old time/op  new time/op  delta
RelinkHyperkube               29.9s ± 2%   29.2s ± 3%  -2.42%  (p=0.000 n=20+20)
RelinkWithoutDebugHyperkube   22.0s ± 3%   21.4s ± 3%  -2.58%  (p=0.000 n=20+20)

Change-Id: Ib7696d8760dd0485240246d6d640668fbf451d71
Reviewed-on: https://go-review.googlesource.com/c/go/+/205257
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2019-11-08 16:35:06 +00:00
Jeremy Faller
b4245ea544 [dev.link] cmd/link: use mmapped data for macho host objs
Move all the reads from macho host objects to mmapped data.

Change-Id: I9904f148feab6ef972d814a93964bcad04207b13
Reviewed-on: https://go-review.googlesource.com/c/go/+/205841
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
2019-11-08 16:32:35 +00:00
Joel Sing
a2b1dc863f cmd/link,cmd/internal/objabi: factor out direct call identification
Factor out the direct CALL identification code from objabi.IsDirectJump and
use this in two places that have separately maintained lists of reloc types.
Provide an objabi.IsDirectCallOrJump function that implements the original
behaviour of objabi.IsDirectJump.

Change-Id: I48131bae92b2938fd7822110d53df0b4ffb35766
Reviewed-on: https://go-review.googlesource.com/c/go/+/196577
Run-TryBot: Joel Sing <joel@sing.id.au>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-06 17:03:51 +00:00
Elias Naur
1bd974eeb7 cmd/link/internal/ld: omit bitcode-incompatible flags on iOS simulator
The -Wl,-headerpad, -Wl,-no_pie, -Wl,-pagezero_size flags are
incompatible with the bitcode-related flags used for iOS.

We already omitted the flags on darwin/arm and darwin/arm64; this change
omits the flags on all platforms != macOS so that building for the iOS
simulator works.

Updates #32963

Change-Id: Ic9af0daf01608f5ae0f70858e3045e399de7e95b
Reviewed-on: https://go-review.googlesource.com/c/go/+/205340
Run-TryBot: Elias Naur <mail@eliasnaur.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-11-06 14:56:38 +00:00
Clément Chigot
6dc250f456 cmd/link: fix the size of typerel.* with c-archive buildmode
With buildmode=c-archive, "runtime.types" type isn't STYPE but
STYPERELRO.
On AIX, this symbol is present in the symbol table and not under
typerel.* outersymbol. Therefore, the size of typerel.* must be adapted.

Fixes #35342

Change-Id: Ib982c6557d9b41bc3d8775e4825650897f9e0ee6
Reviewed-on: https://go-review.googlesource.com/c/go/+/205338
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-11-06 13:55:04 +00:00
Cherry Zhang
bbae923d20 cmd: merge branch 'dev.link' into master
In the dev.link branch we implemented the new object file format
and (part of) the linker improvements described in
https://golang.org/s/better-linker

The new object file is index-based and provides random access.
The linker maps the object files into read-only memory, and
access symbols on-demand using indices, as opposed to reading
all object files sequentially into the heap with the old format.

The linker carries symbol informations using indices (as opposed
to Symbol data structure). Symbols are created after the
reachability analysis, and only created for reachable symbols.
This reduces the linker's memory usage.

Linking cmd/compile, it creates ~25% fewer Symbols, and reduces
memory usage (inuse_space) by ~15%. (More results from Than.)

Currently, both the old and new object file formats are supported.
The old format is used by default. The new format can be turned
on by using the compiler/assembler/linker's -newobj flag. Note
that the flag needs to be specified consistently to all
compilations, i.e.

go build -gcflags=all=-newobj -asmflags=all=-newobj -ldflags=-newobj

Change-Id: Ia0e35306b5b9b5b19fdc7fa7c602d4ce36fa6abd
2019-11-05 14:57:48 -05:00
Cherry Zhang
9cf6c65ca3 [dev.link] cmd: default to old object file format
Flip back to the old object files for Go 1.14.

Change-Id: I4ad499460fb7156b63fc63e9c6ea4f7099e20af2
Reviewed-on: https://go-review.googlesource.com/c/go/+/204098
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-11-05 19:51:52 +00:00
Matthew Dempsky
e341e93c51 cmd/compile, cmd/link: add coverage instrumentation for libfuzzer
This CL adds experimental coverage instrumentation similar to what
github.com/dvyukov/go-fuzz produces in its -libfuzzer mode. The
coverage can be enabled by compiling with -d=libfuzzer. It's intended
to be used in conjunction with -buildmode=c-archive to produce an ELF
archive (.a) file that can be linked with libFuzzer. See #14565 for
example usage.

The coverage generates a unique 8-bit counter for each basic block in
the original source code, and emits an increment operation. These
counters are then collected into the __libfuzzer_extra_counters ELF
section for use by libFuzzer.

Updates #14565.

Change-Id: I239758cc0ceb9ca1220f2d9d3d23b9e761db9bf1
Reviewed-on: https://go-review.googlesource.com/c/go/+/202117
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2019-11-05 00:00:36 +00:00
Cherry Zhang
d9ee4b2859 cmd/link/internal/ld: fix TestArchiveBuildInvokeWithExec
TestArchiveBuildInvokeWithExec is failing on darwin due to
duplicated symbols, because the C definition (int fortytwo;) is
copied to two generated cgo sources. In fact, this test is about
building c-archive, but doesn't need to import "C". Removed the
"C" import.

Change-Id: I3a17546e01272a7ae37e6417791ab949fb44597e
Reviewed-on: https://go-review.googlesource.com/c/go/+/205278
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
2019-11-04 21:38:36 +00:00
Than McIntosh
bf7e55b618 cmd/link: directly exec archive command if external tmpdir
When linking a Go archive, if the archiver invocation is the very last
thing that needs to happen in the link (no "atexit" cleanups required
remove the locally created tmpdir) then call syscall.Exec to invoke
the archiver command instead of the usual exec.Command. This has the
effect of reducing peak memory use for the linker overall, since we
don't be holding onto all of the linker's live memory while the
archiver is running.

Change-Id: Ibbe22d8d67a70cc2a4f91c68aab56d19fb77c393
Reviewed-on: https://go-review.googlesource.com/c/go/+/203821
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-04 18:51:13 +00:00
Cherry Zhang
d3660e8ef8 cmd/link: enable PIE on darwin/arm
We used to pass -no_pie to external linker on darwin/arm, which
is incompatible with -fembed-bitcode. CL 201358 attempted to
remove the -no_pie flag, but it resulted the darwin linker to
complain about absolute addressing in TEXT segment.

On darwin/arm, we already get away from absolute addressing in
the TEXT section. The complained absolute addressing is in
RODATA, which was embedded in the TEXT segment. This CL moves
RODATA to the DATA segment, like what we already did on ARM64
and on AMD64 in c-archive/c-shared buildmodes for the same reason.
So there is no absolute addressing in the TEXT segment, which
allows us to remove -no_pie flag.

Fixes #35252.
Updates #32963.

Change-Id: Id6e3a594cb066d257d4f58fadb4a3ee4672529f7
Reviewed-on: https://go-review.googlesource.com/c/go/+/205060
Reviewed-by: Elias Naur <mail@eliasnaur.com>
2019-11-04 15:18:34 +00:00
Cherry Zhang
db75205a9b [dev.link] cmd/link: restore -strictdups flag in newobj mode
Change-Id: I93ad769595fa343400afa342af12e1445abff084
Reviewed-on: https://go-review.googlesource.com/c/go/+/204918
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
2019-11-03 20:19:00 +00:00
Cherry Zhang
ab4a71fca7 [dev.link] cmd/link: use the start of compilation unit for R_ADDRCUOFF
R_ADDRCUOFF refers to the offset from the start of the
compilation unit. Now that we could have multiple compilation
units per package, we should use the start of the compilation
unit, instead of the start of the package.

Change-Id: I3d6a3c5a5c47c9ccf88091ed05413ced826f9934
Reviewed-on: https://go-review.googlesource.com/c/go/+/204828
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
2019-11-03 19:43:59 +00:00
Cherry Zhang
c6621d9241 [dev.link] cmd/link: restore -dumpdep in new deadcode pass
Change-Id: I2e52206a95c9463df5661664726a8b3bbf3ad1fc
Reviewed-on: https://go-review.googlesource.com/c/go/+/204825
Reviewed-by: Than McIntosh <thanm@google.com>
2019-11-03 04:57:50 +00:00
Cherry Zhang
36dbdbe9bd [dev.link] cmd/link: use min-heap for work queue for better locality
In the deadcode pass, we use a work queue for the flood algorithm.
Currently this is a simple LIFO queue. In this order, there is
poor locality in accessing object files.

Since the global indices are assigned in package DAG order, edges
are mostly either within a package or from a smaller index to a
larger one. (With named symbols, there can be backward edges, but
shouldn't be too many.) Using a min-heap for the work queue, we
access all symbols in one object, then move to next one. It
rarely needs to revisit an object that is already visted. This
should result in better locality.

Benchmark result from Than (thanks!):

name                      old time/op       new time/op       delta
LinkCompiler                    1.74s ±11%        1.61s ± 9%  -7.80%  (p=0.000 n=20+19)
LinkWithoutDebugCompiler        1.27s ±11%        1.15s ± 9%  -9.02%  (p=0.000 n=20+20)

Currently this uses the container/heap package, which uses
interface elements. If this allocates too much, we may consider
to hand-code the min heap.

Change-Id: I216d5291c432fe1f40b0b8f4f1b9d388807bf6c5
Reviewed-on: https://go-review.googlesource.com/c/go/+/204438
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2019-11-02 19:22:12 +00:00
Cherry Zhang
24ea07d592 [dev.link] cmd/link: add internal packages at the end
Currently in the linker we load internal packges first, then the
main package, and then load imported packages following the
dependency graph. As a result, packages are loaded mostly in the
dependency order, except the internal packages. The global symbol
indices are assigned the same way.

By loading the internal packages at the end, the packages are
loaded in the dependency order, so are the global indices. This
way, a relocation edge is mostly either within a packge or a
forward edge from a smaller index to a larger one. This allows
us to use a min-heap work queue in the deadcode pass, to achieve
better spatial locality (in the next CL).

Change-Id: I01fa9b3cf0c9e9e66006040f6378a51fd78f0f39
Reviewed-on: https://go-review.googlesource.com/c/go/+/204437
Reviewed-by: Jeremy Faller <jeremy@golang.org>
2019-11-02 19:21:37 +00:00
Jeremy Faller
496e4273e4 [dev.link] cmd/link: fix loadpe to work with new obj file
Change-Id: I0fe88df182f13e7f04c8de0b82e111db441a26e2
Reviewed-on: https://go-review.googlesource.com/c/go/+/204341
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
2019-11-02 19:21:08 +00:00
Cherry Zhang
48a0b97902 [dev.link] cmd/link: set cgo attributes early when internal cgo linking
In newobj mode, cgo attributes are typically set later, as we
create sym.Symbols later. But when internal cgo linking, the
host object loaders still work with sym.Symbols, and the cgo
attributes need to be set for them to work properly. Therefore,
set them early. This will cause creating some Symbols eagerly,
but they are mostly host object symbols and will need to be
created anyway.

Now all cgo internal linking tests pass on ELF systems.

Change-Id: I023a4df4429acc8ebf5e185f62e6809198497a78
Reviewed-on: https://go-review.googlesource.com/c/go/+/204857
Reviewed-by: Than McIntosh <thanm@google.com>
2019-11-02 00:36:59 +00:00
Than McIntosh
5aea597905 [dev.link] cmd/link/internal/loader: do more bulk allocation
Change the loader to do more bulk allocation when making slices of
small objects (sym.Reloc, etc) as part of creating and populating
sym.Symbols in loader.LoadFull(). This replaces a large number of
small allocations with a smaller number of large allocations,
improving performace. Compilebench numbers (linker portion) for this
change:

name                      old time/op       new time/op       delta
LinkCompiler                    1.71s ±11%        1.57s ± 9%   -8.35%  (p=0.000 n=19+20)
LinkWithoutDebugCompiler        1.19s ±14%        1.10s ±13%   -7.93%  (p=0.000 n=20+19)

name                      old user-time/op  new user-time/op  delta
LinkCompiler                    1.86s ±15%        1.34s ±10%  -28.02%  (p=0.000 n=20+20)
LinkWithoutDebugCompiler        1.05s ±14%        0.95s ± 9%   -9.17%  (p=0.000 n=19+20)

Hyperkube from kubernetes doesn't show any significant benefit (which
seems a little surprising).

Change-Id: Ide97f78532fb60b08bb6e4cfa097e9058f7ea8ab
Reviewed-on: https://go-review.googlesource.com/c/go/+/203457
Run-TryBot: Than McIntosh <thanm@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
2019-11-01 18:49:50 +00:00
Jeremy Faller
e961b26c27 [dev.link] cmd/link: fix xcoff loader for new obj format
config.go needs to be removed from this CL.

Change-Id: I04a267feeae1551bb18f6a03a725adc9db593fdb
Reviewed-on: https://go-review.googlesource.com/c/go/+/204099
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-01 15:36:44 +00:00
Than McIntosh
c0555a2a7a [dev.link] all: merge branch 'master' into dev.link
Fixed a couple of minor conflicts in lib.go and deadcode.go
relating to debug logging.

Change-Id: I58335fc42ab1f1f3409fd8354da4f26419e8fb22
2019-11-01 10:45:24 -04:00
Than McIntosh
219922e95b [dev.link] cmd/link: add support to new deadcode for field tracking
Fix up the new dead code pass to include support for populating the
ctxt "Reachparent" map, which is needed to support field tracking.
Since we don't have sym.Symbols created at the point where new dead
code runs, keep track of reachability using global symbol indices, and
then once loader.LoadFull is complete we can translate the index
mappings into symbol mappings.

The fieldtracking output is unfortunately different relative to
master, due to differences in the order in which symbols are
encountered in deadcode, but I have eyeballed the results to make sure
they look reasonable.

Change-Id: I48c7a4597f05c00f15af3bfd37fc15ab4d0017c2
Reviewed-on: https://go-review.googlesource.com/c/go/+/204342
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-01 13:54:58 +00:00
Cherry Zhang
63815923fc [dev.link] cmd/link: resolve ABI aliases for external symbols
ABI alias references in Go symbols are resolved during
loadObjFull. But for external symbols they are not resolved. If
there is a reference from an external symbol to a Go ABIInternal
symbol, this reference will be invalid as it is not resolved.

The old code resolve ABI aliases in the deadcode pass. But the
new deadcode pass doesn't do it, as it works with indices instead
of Symbols. We do this in LoadFull.

This makes all internal cgo linking tests pass on Mach-O.

Change-Id: Iac6c084c03f5ddbcc9455527800ce7ed7313f9a7
Reviewed-on: https://go-review.googlesource.com/c/go/+/204698
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
2019-11-01 13:51:36 +00:00
Cherry Zhang
4a43a50791 [dev.link] cmd/link: process cgo_import_static before host object loading
In internal linking mode, we need to process cgo_import_static
directives before loading host objects, because the directive is
to tell the host object loader how to deal with imported symbols.

This should fix linking with old object files. I think there
needs some similar logic for new object files, but I'll leave
that for later.

Change-Id: Icaa286de626ea1876086dbdd015047084c92caf9
Reviewed-on: https://go-review.googlesource.com/c/go/+/204697
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
2019-11-01 03:40:20 +00:00
Jeremy Faller
396a9b98ba [dev.link] cmd/link: fix macho host objects
Small updates to macho loader, fixing some misunderstandings I had about
using the new object file format.

Change-Id: I9224b01ca327e3a087ebfa36800bd6eef6abcc80
Reviewed-on: https://go-review.googlesource.com/c/go/+/204097
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-10-31 15:22:28 +00:00
Jeremy Faller
931845aee3 [dev.link] cmd/link: fix merge error
Change-Id: Ief8384a74ac9cf303a959656f807f34a0ff9873b
Reviewed-on: https://go-review.googlesource.com/c/go/+/204518
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-10-31 14:12:22 +00:00
Jeremy Faller
0e76e2f4e5 [dev.link] cmd/link: elf host obj support w/ new obj files
Add support for elf host objects with new object file format.

Change-Id: Ic5be1953359b9b6b78d9a0b715af69763aefd227
Reviewed-on: https://go-review.googlesource.com/c/go/+/201728
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
2019-10-31 13:11:51 +00:00
Cherry Zhang
5a210b5858 [dev.link] cmd/link: keep DWARF constant DIE symbols live
DWARF constant DIE symbols are not referenced by any other symbol,
but are needed by the DWARF pass, where they get linked to the
compilation unit.

Reenable gdb constant test.

Change-Id: If77a0d379d9a6f1591939345bc31b027c2567f22
Reviewed-on: https://go-review.googlesource.com/c/go/+/204397
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
2019-10-31 01:09:58 +00:00
Cherry Zhang
43d25a61c1 [dev.link] cmd/link: ensure deterministic order in dynexp
dynexp is used for generating the dynamic symbol table. It is
created from a map. Sort it to ensure deterministic order.

Should fix solaris build.

Change-Id: I561b9da3a4136a7ea41139073f76c98fb069d4fa
Reviewed-on: https://go-review.googlesource.com/c/go/+/204378
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
2019-10-31 01:09:34 +00:00
Than McIntosh
851b1f40b4 [dev.link] cmd/link: add new slice interface for querying aux symbols
Add a new loader.Loader.ReadAuxSyms method that returns a slice
containing the ids of the aux symbols for a specified global symbol.
This is similar to the new interface recently added that allows you to
get back a slice of relocations (as opposed to making calls into the
loader for each one). This was idea suggested by Cherry. Compilebench
numbers:

name                      old time/op       new time/op       delta
LinkCompiler                    1.63s ± 9%        1.57s ± 7%  -3.84%  (p=0.006 n=20+20)
LinkWithoutDebugCompiler        1.15s ±11%        1.11s ±11%    ~     (p=0.108 n=20+20)

name                      old user-time/op  new user-time/op  delta
LinkCompiler                    1.99s ± 8%        2.00s ±12%    ~     (p=0.751 n=19+19)
LinkWithoutDebugCompiler        1.14s ±11%        1.19s ±21%    ~     (p=0.183 n=20+20)

Change-Id: Iab6cbe18419aaa61d9cadb3f626a4515c71f2686
Reviewed-on: https://go-review.googlesource.com/c/go/+/203501
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-10-31 01:09:14 +00:00
Than McIntosh
e90e6e75f8 [dev.link] cmd/link/internal/loader: add 1-element object cache for use in toLocal
To speed up the loader.Loader.toLocal() method, cache the index of the
most recently accessed object file and check that object's sym range
in toLocal() before doing a full binary search over all object symbol
ranges. This speeds up relink of kubernetes/hyperkube by about 2%, and
improves compilebench (relative to the dev.link branch) by about 5%:

name                      old time/op       new time/op       delta
LinkCompiler                    1.62s ± 8%        1.50s ± 9%  -7.21%  (p=0.000 n=20+19)
LinkWithoutDebugCompiler        1.13s ± 8%        1.09s ±12%    ~     (p=0.052 n=20+20)

name                      old user-time/op  new user-time/op  delta
LinkCompiler                    1.94s ±18%        1.97s ±16%    ~     (p=0.813 n=19+20)
LinkWithoutDebugCompiler        1.15s ±16%        1.13s ±12%    ~     (p=0.547 n=20+20)

Change-Id: Id5a8a847b533858373c0462f03972d436eda6748
Reviewed-on: https://go-review.googlesource.com/c/go/+/204337
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-10-31 01:04:59 +00:00
Elias Naur
f4e32aeed1 cmd/link/internal/ld: remove flags incompatible with -fembed-bitcode
The flags -headerpad, -Wl,-no_pie and -pagezero_size are incompatible with
the -fembed-bitcode flag used by `gomobile build`. Than McIntosh
suggested we might not need the offending flags; this change removes
the flags on darwin/arm64 and -headerpad, -pagezero_size on darwin/arm.

The -Wl,-no_pie flag is left for darwin/arm because linking fails
without it:

ld: warning: PIE disabled. Absolute addressing (perhaps -mdynamic-no-pic) not allowed in code signed PIE, but used in _runtime.rodata from /var/folders/qq/qxn86k813bn9fjxydm095rxw0000gp/T/workdir-host-darwin-amd64-zenly-ios/tmp/go-link-225285265/go.o. To fix this warning, don't compile with -mdynamic-no-pic or link with -Wl,-no_pie

Discussion: https://groups.google.com/d/msg/golang-dev/U1jK3xmmGAk/j0_ty46EDAAJ

I've verified the CL on the builders, built the "flappy" example from
gomobile with `gomobile build`, and verified that flappy runs on an
iPhone 5S.

Updates #32963

Change-Id: I783abc93ccf3c1d2b7ca00144b7164ba223d3529
Reviewed-on: https://go-review.googlesource.com/c/go/+/201358
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-10-30 08:17:29 +00:00
John Papandriopoulos
59a6847039 cmd/link: pass-through undefined call targets in external link mode
Allows Go asm calls referencing a function in a .syso file to be
passed through to the external linker, that would have otherwise
raised a "relocation target X not defined" error in cmd/link.

Fixes #33139

Change-Id: I2a8eb6063ebcd05fac96f141acf7652cf9189766
Reviewed-on: https://go-review.googlesource.com/c/go/+/198798
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2019-10-28 22:10:06 +00:00