They tend to be things like ".shape.int" which are noisy, if not
otherwise confusing.
It would be nice to somehow print the real instantiations here, but that
requires keeping track of the dictionary argument so the instantiating
types could be found. One day, maybe, but not today.
Fixes#48578
Change-Id: I0968d24e110b6d47c9468c45372a6979575a8d29
Reviewed-on: https://go-review.googlesource.com/c/go/+/352118
Trust: Keith Randall <khr@golang.org>
Trust: Dan Scales <danscales@google.com>
Run-TryBot: Keith Randall <khr@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dan Scales <danscales@google.com>
Currently, relocation type is stored as uint8 in object files, as
Go relocations do not exceed 255. In the linker, however, it is
used as a 16-bit type, because external relocations can exceed
255. The linker has to store the extra byte in a side table. This
complicates many things.
Just store it as uint16 in object files. This simplifies things,
with a small cost of increasing the object file sizes.
before after
hello.o 1672 1678
runtime.a 7927784 8056194
Change-Id: I313cf44ad0b8b3b76e35055ae55d911ff35e3158
Reviewed-on: https://go-review.googlesource.com/c/go/+/268477
Trust: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
This CL adds support of PIE internal linking on darwin/amd64.
This is also preparation for supporting internal linking on
darwin/arm64 (macOS), which requires PIE for everything.
Updates #38485.
Change-Id: I2ed58583dcc102f5e0521982491fc7ba6f2754ed
Reviewed-on: https://go-review.googlesource.com/c/go/+/261642
Trust: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
Move the pctables out of pclntab_old. Creates a new generator symbol,
runtime.pctab, which holds all the deduplicated pctables. Also, tightens
up some of the types in runtime.
Darwin, cmd/compile statistics:
alloc/op
Pclntab_GC 26.4MB ± 0% 13.8MB ± 0%
allocs/op
Pclntab_GC 89.9k ± 0% 86.4k ± 0%
liveB
Pclntab_GC 25.5M ± 0% 24.2M ± 0%
No significant change in binary size.
Change-Id: I1560fd4421f8a210f8d4b508fbc54e1780e338f9
Reviewed-on: https://go-review.googlesource.com/c/go/+/248332
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Move the function names out of runtime.pclntab_old, creating
runtime.funcnametab. There is an unfortunate artifact in this change in
that calculating the funcID still requires loading the name. Future work
will likely pull this out and put it into the object file Funcs.
ls -l cmd/compile (darwin):
before: 18524016
after: 18519952
The difference in size can be attributed to alignment in pclntab_old.
Change-Id: Ibcbb230d4632178f8fcd0667165f5335786381f8
Reviewed-on: https://go-review.googlesource.com/c/go/+/243223
Reviewed-by: Austin Clements <austin@google.com>
As of July 2020, a fair amount of the new linker's live memory, and
runtime is spent generating pclntab. In an effort to streamline that
code, this change starts breaking up the generation of runtime.pclntab
into smaller chunks that can run later in a link. These changes are
described in an (as yet not widely distributed) document that lays out
an improved format. Largely the work consists of breaking up
runtime.pclntab into smaller pieces, stopping much of the data
rewriting, and getting runtime.pclntab into a form where we can reason
about its size and look to shrink it. This change is the first part of
that work -- just pulling out the header, and demonstrating where a
majority of that work will be.
Change-Id: I65618d0d0c780f7e5977c9df4abdbd1696fedfcb
Reviewed-on: https://go-review.googlesource.com/c/go/+/241598
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Austin Clements <austin@google.com>
Rename Reloc2 to Reloc, At2 to At, Aux2 to Aux.
Change-Id: Ic98d83c080e8cd80fbe1837c8f0aa134033508ce
Reviewed-on: https://go-review.googlesource.com/c/go/+/245578
Reviewed-by: Jeremy Faller <jeremy@golang.org>
We have Reloc and Reloc2. Reloc2 is the better approach and most
code uses Reloc2. There are still uses of Reloc. This CL migrates
them to Reloc2, and removes Reloc.
Change-Id: Id5f6a6019e1e044add682d05e70ebb1548ec58d9
Reviewed-on: https://go-review.googlesource.com/c/go/+/245577
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Introduce a new loader method "AddInteriorSym" to be used when
establishing container/containee symbol relationships for host object
sub-symbols and GOT/dynamic sub-symbols.
Interior symbols are employed in situations where you have a
"container" or "payload" symbol that has content, and then a series of
"interior" sub-symbols that point into a portion of the container
symbol's content. Each interior symbol will typically have a useful
name / size / value, but no content of its own. From a symbol table
perspective the container symbol is anonymous, but the interior
symbols are added to the output symbol table.
Change-Id: I919ed5dbbfe2ef2c9a76214f7ea9b384a1be6297
Reviewed-on: https://go-review.googlesource.com/c/go/+/240508
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
The loader's SymbolBuilder Add*/Set* methods include a call to mark
the underlying symbol as reachable (as a convenience, so that callers
would not have to set it explicitly). This code was carried over from
the corresponding sym.Symbol methods; back in the sym.Symbol world
unreachable symbols were never removed from the AllSyms slice, hence
setting and checking reachability was a good deal more important.
With the advent of the loader and the new deadcode implementation,
there is less of a need for this sort of fallback, and in addition the
implicit attr setting introduces data races in the the loader if there
are SymbolBuilder Add*/Set* method calls in parallel threads, as well
as adding overhead to the methods.
This patch gets rid of the implicit reachability setting, and instead
marks reachability in CreateSymForUpdate, as well as adding a few
explicit SetAttrReachable calls where needed.
Change-Id: I029a0c5a4a24237826a7831f9cbe5180d44cbc40
Reviewed-on: https://go-review.googlesource.com/c/go/+/237678
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
This deletes all sym.Symbol and sym.Reloc references. This is
certainly not complete, and there are more cleanups to do. But I
feel this makes a good first round.
Change-Id: I7621d016957f7ef114be5f0606fcb3ad6aee71c8
Reviewed-on: https://go-review.googlesource.com/c/go/+/234097
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
This patch begins the work of converting the linker's dodata phase to
work with loader APIs. Passes all.bash on linux/amd64, but hasn't been
tested on anything else (more arch-specific code needs to be written).
Use of the new dodata() phase is currently gated by a temporary
command line flag ("-newdodata"), and there is code in the linker's
main routine to insure that we only use the new version for the right
GOOS/GOARCH (currently restricted to ELF + AMD64).
Change-Id: Ied3966677d2a450bc3e0990e0f519b3fceaab806
Reviewed-on: https://go-review.googlesource.com/c/go/+/229706
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Clients may need to invoke the loader.Reloc2.SetType method to reset
the type of a relocation from external flavor to internal flavor,
meaning that the external type add-in needs to be zeroed (this is
needed when adding dynsym entries).
Add a new SymbolBuider method to support mutating the type of a reloc
for an external symbol, so that the external type can be changed as
well (Reloc2 doesn't have access to that). Also add similar methods
for updating target symbol and addend, so as to have a consistent
interface for ext reloc mutation.
Change-Id: I8e26cdae0a0f353019acba5f9c8a0506e3970266
Reviewed-on: https://go-review.googlesource.com/c/go/+/229604
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
This is more or less a direct translation, to get things going.
There are more things we can do to make it better, especially on
the handling of container symbols.
Change-Id: I11a0087e402be8d42b9d06869385ead531755272
Reviewed-on: https://go-review.googlesource.com/c/go/+/229125
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
If we are internal linking a static executable, and address assignment
has happened, then when emitting some parts of DWARF we can just emit
a function address directly instead of generating a relocation. For
external linking or other build modes, we are generating a relocatable
binary so we still need to emit relocations.
This CL inspired by Cherry's similar CL for pclntab at
https://go-review.googlesource.com/c/go/+/228478.
Change-Id: Ib03fbe2dd72d0ba746bf46015e0f2d6c3f3d53ab
Reviewed-on: https://go-review.googlesource.com/c/go/+/228537
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Introduce SetAddrPlus method for use with SymbolBuilder.
Change-Id: Ieb2986b2fc1675b7a3a71c84b7219b3d564ac122
Reviewed-on: https://go-review.googlesource.com/c/go/+/227757
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
This is in prepration of removing the old loader.Reloc. This also
introduces a way of adding a slice of relocations more
efficiently (will be used in the next CL).
Change-Id: I3eaee7fb3a3e102a8670990f4a31c40d0b17b8c2
Reviewed-on: https://go-review.googlesource.com/c/go/+/227761
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Implement text address assignment and trampoline generation using
the loader.
Note: the trampoline insertion part doesn't actually work. It
also needs to propagate Aux symbols for external symbols in
LoadFull. But it won't be needed after converting pclntab
generation, so I'll leave it out for now. This could break
linking large binaries on PPC64 and ARM.
Change-Id: Ie46a35b25d7c027983dd877207cfa8f67c32530b
Reviewed-on: https://go-review.googlesource.com/c/go/+/227482
Reviewed-by: Than McIntosh <thanm@google.com>
Add support to the loader for getting/setting the 'variant' property
of a symbol relocation. The variant property handles unusual or
infrequently used relocations that have both a type and a variant of
that type (this is needed for S390).
In the sym.Symbol world, a relocation variant is a field on the
'relocExt' extension that is part of sym.Reloc. In this new
implementation for the loader, reloc variants are stored in a side
table (a map) in the loader, and accessed via loader methods.
Change-Id: I62bf54ae7ff6d500c0ea8d2dbe759b2431087378
Reviewed-on: https://go-review.googlesource.com/c/go/+/227018
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Store external relocations in (almost) the same format as the Go
objects, so we can handle them more uniformly.
There is a small speedup:
(linking cmd/compile)
Deadcode 67.8ms ± 3% 61.1ms ± 3% -9.94% (p=0.008 n=5+5)
Dostkcheck 41.2ms ± 2% 38.8ms ± 3% -5.99% (p=0.008 n=5+5)
Change-Id: I8616e10b26235904201d6c9465f5ae32a49c9949
Reviewed-on: https://go-review.googlesource.com/c/go/+/226365
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This converts the second portion of DWARF-gen (dwarfGenerateDebugSyms
function and friends) to use the loader, as opposed to sym.Symbols.
Second phase is invoked as part of dodata(), which is fairly far along
in the linker, meaning that the "converted to loader.Symbol" wavefront
has not yet reached that point.
To account for this, the patch contains some shim code that runs
initially to copy back things from sym.Symbols into the loader
(notable symbol values), and then a second hook to be code after
dwarf-gen is complete to that copies the results back into the
appropriate sym.Symbols. This code is obviously temporary, but needed
for the time being.
Change-Id: I9b830b08b16480a1a5230cb52b592db1c550af18
Reviewed-on: https://go-review.googlesource.com/c/go/+/220987
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
With unique global indices, MakeSymbolUpdater will not change the
symbol's index. So no need to return a new index.
Change-Id: I5b4fd6a0167cc74476880bbf4382c524ecde7721
Reviewed-on: https://go-review.googlesource.com/c/go/+/219227
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Currently, when mapping symbol's local index to global index, for
duplicated and overwriting/overwritten symbols, each appearance
of the symbol gets a global index, with one being the "primary",
and others "redirect" to it through the overwrite map. Basically,
the local-global index mapping is one to one, with overwrite/
dedup happening in global index level.
This has a few drawbacks:
- All symbol accesses effectively need to query the overwrite
map. This may hurt performance.
- For multi-level overwrites, (Y overwrites X, Z overwrites Y),
this can get quite complicated, and we have to follow the
redirection recursively.
- Failed to follow or to update the overwrite map leads to bugs.
In this CL, we change the index mapping mechanism so that each
symbol get a unique global index. Multiple appearances of the
same symbol get the same index. Now the local-global index
mapping is N to one. Overwrite/dedup happens directly in the
local-global mapping.
We keep both mapping directions in arrays. Each object carries
an array for its local-global mapping. The loader carries an
array mapping global index to the "primary" local index, which is
the one we should load from. This way, we can get rid of the
overwrite map, and index conversions are simply array accesses.
TODO: we still make reservation of the index space upfront, and
leave holes for dup symbols. Maybe get rid of the reservation and
holes.
Change-Id: Ia251489d5f2ff16a0b3156a71d141a70cdf03a4e
Reviewed-on: https://go-review.googlesource.com/c/go/+/217064
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Currently, the symbol updater uses a pointer pointing to the
loader's payloads array. If the payloads slice grows (and moves),
the pointer may become stale and no longer point to the symbol's
actual payload. Specifically, consider
sb, sym := l.MakeSymbolUpdater(...)
// add a bunch of external symbols, which grows payload slice
sb.SetType(t)
l.SymType(sym) // may not return t
sb.SetType on line 3 may not have the desired effect, as
sb.extSymPayload may no longer point to the right payload. As a
result, the type we get on line 4 may be not the one we set.
Fix this by making the payload's address permanent. Once it is
allocated it will never move.
Change-Id: Iab190ea5aceb5c37f91d09ad4ffd458e881b03f4
Reviewed-on: https://go-review.googlesource.com/c/go/+/217063
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Revamp the way that symbol overwrites are handled to deal with symbols
that are overwritten more than once (such as "_cgo_mmap"). The
scenario here is that a symbol can be overwritten twice, once during
preload, and then again when host objects are read during internal
linking. This can result in a situation where we have two entries in
the overwrite map, from X -> Y and then from Y -> Z. Rather than
search the overwrite map when adding new entries, add a helper routine
for querying the map that catches this situation and fixes it up.
Also with this patch is a couple of tweaks to the loader.Dump method
to insure that it can dump the entire global index space without
crashing due to odd overwrites (as in the scenario above).
Change-Id: Ib6c8a0e03e92fc2b57318001711b501eeaf12249
Reviewed-on: https://go-review.googlesource.com/c/go/+/212098
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Add in the hooks to SymbolBuilder and to the loader to allow
the linker to make modifications to a non-external symbol (e.g.
a sym whose index is less than loader.extStart).
The basic idea is to manufacture a new external symbol with the same
name and version, then import the old symbol's content (type, data,
relocations, etc) into the payload struct for the new symbol, and
finally redirect the name lookup tables to target the new sym for the
specified name/version.
This change is needed in order to convert over the host object loaders
to avoid use of sym.Symbol.
Change-Id: I79cd42b23794e830bbdbcbcd2c500c35c351f01f
Reviewed-on: https://go-review.googlesource.com/c/go/+/211897
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Add SymbolBuilder helper type -- this type provides a set of methods
intended to make it easy to manipulate the content of a symbol (type,
relocations, data, etc).
Change-Id: I579bf8d04650e66d33a9780a6c2347a576c94c6f
Reviewed-on: https://go-review.googlesource.com/c/go/+/210178
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
