Stowage/go
2
0
Fork 0
mirror of https://github.com/golang/go.git synced 2025-12-08 06:10:04 +00:00
Code Issues Projects Releases Packages Wiki Activity
go/src/runtime/mpagealloc_test.go

1036 lines
32 KiB
Go
Raw Normal View History

runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
// Copyright 2019 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package runtime_test
import (
"fmt"
. "runtime"
"testing"
)
func checkPageAlloc(t *testing.T, want, got *PageAlloc) {
// Ensure start and end are correct.
wantStart, wantEnd := want.Bounds()
gotStart, gotEnd := got.Bounds()
if gotStart != wantStart {
t.Fatalf("start values not equal: got %d, want %d", gotStart, wantStart)
}
if gotEnd != wantEnd {
t.Fatalf("end values not equal: got %d, want %d", gotEnd, wantEnd)
}
for i := gotStart; i < gotEnd; i++ {
runtime: convert page allocator bitmap to sparse array Currently the page allocator bitmap is implemented as a single giant memory mapping which is reserved at init time and committed as needed. This causes problems on systems that don't handle large uncommitted mappings well, or institute low virtual address space defaults as a memory limiting mechanism. This change modifies the implementation of the page allocator bitmap away from a directly-mapped set of bytes to a sparse array in same vein as mheap.arenas. This will hurt performance a little but the biggest gains are from the lockless allocation possible with the page allocator, so the impact of this extra layer of indirection should be minimal. In fact, this is exactly what we see: https://perf.golang.org/search?q=upload:20191125.5 This reduces the amount of mapped (PROT_NONE) memory needed on systems with 48-bit address spaces to ~600 MiB down from almost 9 GiB. The bulk of this remaining memory is used by the summaries. Go processes with 32-bit address spaces now always commit to 128 KiB of memory for the bitmap. Previously it would only commit the pages in the bitmap which represented the range of addresses (lowest address to highest address, even if there are unused regions in that range) used by the heap. Updates #35568. Updates #35451. Change-Id: I0ff10380156568642b80c366001eefd0a4e6c762 Reviewed-on: https://go-review.googlesource.com/c/go/+/207497 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-14 23:58:50 +00:00
// Check the bitmaps. Note that we may have nil data.
runtime: add scavenging code for new page allocator This change adds a scavenger for the new page allocator along with tests. The scavenger walks over the heap backwards once per GC, looking for memory to scavenge. It walks across the heap without any lock held, searching optimistically. If it finds what appears to be a scavenging candidate it acquires the heap lock and attempts to verify it. Upon verification it then scavenges. Notably, unlike the old scavenger, it doesn't show any preference for huge pages and instead follows a more strict last-page-first policy. Updates #35112. Change-Id: I0621ef73c999a471843eab2d1307ae5679dd18d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/195697 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-21 00:24:25 +00:00
gb, wb := got.PallocData(i), want.PallocData(i)
runtime: convert page allocator bitmap to sparse array Currently the page allocator bitmap is implemented as a single giant memory mapping which is reserved at init time and committed as needed. This causes problems on systems that don't handle large uncommitted mappings well, or institute low virtual address space defaults as a memory limiting mechanism. This change modifies the implementation of the page allocator bitmap away from a directly-mapped set of bytes to a sparse array in same vein as mheap.arenas. This will hurt performance a little but the biggest gains are from the lockless allocation possible with the page allocator, so the impact of this extra layer of indirection should be minimal. In fact, this is exactly what we see: https://perf.golang.org/search?q=upload:20191125.5 This reduces the amount of mapped (PROT_NONE) memory needed on systems with 48-bit address spaces to ~600 MiB down from almost 9 GiB. The bulk of this remaining memory is used by the summaries. Go processes with 32-bit address spaces now always commit to 128 KiB of memory for the bitmap. Previously it would only commit the pages in the bitmap which represented the range of addresses (lowest address to highest address, even if there are unused regions in that range) used by the heap. Updates #35568. Updates #35451. Change-Id: I0ff10380156568642b80c366001eefd0a4e6c762 Reviewed-on: https://go-review.googlesource.com/c/go/+/207497 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-14 23:58:50 +00:00
if gb == nil && wb == nil {
continue
}
if (gb == nil && wb != nil) || (gb != nil && wb == nil) {
t.Errorf("chunk %d nilness mismatch", i)
}
runtime: add scavenging code for new page allocator This change adds a scavenger for the new page allocator along with tests. The scavenger walks over the heap backwards once per GC, looking for memory to scavenge. It walks across the heap without any lock held, searching optimistically. If it finds what appears to be a scavenging candidate it acquires the heap lock and attempts to verify it. Upon verification it then scavenges. Notably, unlike the old scavenger, it doesn't show any preference for huge pages and instead follows a more strict last-page-first policy. Updates #35112. Change-Id: I0621ef73c999a471843eab2d1307ae5679dd18d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/195697 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-21 00:24:25 +00:00
if !checkPallocBits(t, gb.PallocBits(), wb.PallocBits()) {
t.Logf("in chunk %d (mallocBits)", i)
}
if !checkPallocBits(t, gb.Scavenged(), wb.Scavenged()) {
t.Logf("in chunk %d (scavenged)", i)
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
}
}
// TODO(mknyszek): Verify summaries too?
}
runtime: track ranges of address space which are owned by the heap This change adds a new inUse field to the allocator which tracks ranges of addresses that are owned by the heap. It is updated on each heap growth. These ranges are tracked in an array which is kept sorted. In practice this array shouldn't exceed its initial allocation except in rare cases and thus should be small (ideally exactly 1 element in size). In a hypothetical worst-case scenario wherein we have a 1 TiB heap and 4 MiB arenas (note that the address ranges will never be at a smaller granularity than an arena, since arenas are always allocated contiguously), inUse would use at most 4 MiB of memory if the heap mappings were completely discontiguous (highly unlikely) with an additional 2 MiB leaked from previous allocations. Furthermore, the copies that are done to keep the inUse array sorted will copy at most 4 MiB of memory in such a scenario, which, assuming a conservative copying rate of 5 GiB/s, amounts to about 800µs. However, note that in practice: 1) Most 64-bit platforms have 64 MiB arenas. 2) The copies should incur little-to-no page faults, meaning a copy rate closer to 25-50 GiB/s is expected. 3) Go heaps are almost always mostly contiguous. Updates #35514. Change-Id: I3ad07f1c2b5b9340acf59ecc3b9ae09e884814fe Reviewed-on: https://go-review.googlesource.com/c/go/+/207757 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Reviewed-by: Austin Clements <austin@google.com>
2019-11-15 23:30:30 +00:00
func TestPageAllocGrow(t *testing.T) {
runtime: disable pageAlloc tests on OpenBSD in short mode This change disables pageAlloc tests on OpenBSD in short mode because pageAlloc holds relatively large virtual memory reservations and we make two during the pageAlloc tests. The runtime may also be carrying one such reservation making the virtual memory requirement for testing the Go runtime three times as much as just running a Go binary. This causes problems for folks who just want to build and test Go (all.bash) on OpenBSD but either don't have machines with at least 4ish GiB of RAM (per-process virtual memory limits are capped at some constant factor times the amount of physical memory) or their per-process virtual memory limits are low for other reasons. Fixes #36210. Change-Id: I8d89cfde448d4cd2fefff4ad6ffed90de63dd527 Reviewed-on: https://go-review.googlesource.com/c/go/+/212177 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-12-19 17:44:58 +00:00
if GOOS == "openbsd" && testing.Short() {
t.Skip("skipping because virtual memory is limited; see #36210")
}
runtime: use inUse ranges to map in summary memory only as needed Prior to this change, if the heap was very discontiguous (such as in TestArenaCollision) it's possible we could map a large amount of memory as R/W and commit it. We would use only the start and end to track what should be mapped, and we would extend that mapping as needed to accomodate a potentially fragmented address space. After this change, we only map exactly the part of the summary arrays that we need by using the inUse ranges from the previous change. This reduces the GCSys footprint of TestArenaCollision from 300 MiB to 18 MiB. Because summaries are no longer mapped contiguously, this means the scavenger can no longer iterate directly. This change also updates the scavenger to borrow ranges out of inUse and iterate over only the parts of the heap which are actually currently in use. This is both an optimization and necessary for correctness. Fixes #35514. Change-Id: I96bf0c73ed0d2d89a00202ece7b9d089a53bac90 Reviewed-on: https://go-review.googlesource.com/c/go/+/207758 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-11-18 19:23:39 +00:00
type test struct {
runtime: track ranges of address space which are owned by the heap This change adds a new inUse field to the allocator which tracks ranges of addresses that are owned by the heap. It is updated on each heap growth. These ranges are tracked in an array which is kept sorted. In practice this array shouldn't exceed its initial allocation except in rare cases and thus should be small (ideally exactly 1 element in size). In a hypothetical worst-case scenario wherein we have a 1 TiB heap and 4 MiB arenas (note that the address ranges will never be at a smaller granularity than an arena, since arenas are always allocated contiguously), inUse would use at most 4 MiB of memory if the heap mappings were completely discontiguous (highly unlikely) with an additional 2 MiB leaked from previous allocations. Furthermore, the copies that are done to keep the inUse array sorted will copy at most 4 MiB of memory in such a scenario, which, assuming a conservative copying rate of 5 GiB/s, amounts to about 800µs. However, note that in practice: 1) Most 64-bit platforms have 64 MiB arenas. 2) The copies should incur little-to-no page faults, meaning a copy rate closer to 25-50 GiB/s is expected. 3) Go heaps are almost always mostly contiguous. Updates #35514. Change-Id: I3ad07f1c2b5b9340acf59ecc3b9ae09e884814fe Reviewed-on: https://go-review.googlesource.com/c/go/+/207757 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Reviewed-by: Austin Clements <austin@google.com>
2019-11-15 23:30:30 +00:00
chunks []ChunkIdx
inUse []AddrRange
runtime: use inUse ranges to map in summary memory only as needed Prior to this change, if the heap was very discontiguous (such as in TestArenaCollision) it's possible we could map a large amount of memory as R/W and commit it. We would use only the start and end to track what should be mapped, and we would extend that mapping as needed to accomodate a potentially fragmented address space. After this change, we only map exactly the part of the summary arrays that we need by using the inUse ranges from the previous change. This reduces the GCSys footprint of TestArenaCollision from 300 MiB to 18 MiB. Because summaries are no longer mapped contiguously, this means the scavenger can no longer iterate directly. This change also updates the scavenger to borrow ranges out of inUse and iterate over only the parts of the heap which are actually currently in use. This is both an optimization and necessary for correctness. Fixes #35514. Change-Id: I96bf0c73ed0d2d89a00202ece7b9d089a53bac90 Reviewed-on: https://go-review.googlesource.com/c/go/+/207758 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-11-18 19:23:39 +00:00
}
tests := map[string]test{
runtime: track ranges of address space which are owned by the heap This change adds a new inUse field to the allocator which tracks ranges of addresses that are owned by the heap. It is updated on each heap growth. These ranges are tracked in an array which is kept sorted. In practice this array shouldn't exceed its initial allocation except in rare cases and thus should be small (ideally exactly 1 element in size). In a hypothetical worst-case scenario wherein we have a 1 TiB heap and 4 MiB arenas (note that the address ranges will never be at a smaller granularity than an arena, since arenas are always allocated contiguously), inUse would use at most 4 MiB of memory if the heap mappings were completely discontiguous (highly unlikely) with an additional 2 MiB leaked from previous allocations. Furthermore, the copies that are done to keep the inUse array sorted will copy at most 4 MiB of memory in such a scenario, which, assuming a conservative copying rate of 5 GiB/s, amounts to about 800µs. However, note that in practice: 1) Most 64-bit platforms have 64 MiB arenas. 2) The copies should incur little-to-no page faults, meaning a copy rate closer to 25-50 GiB/s is expected. 3) Go heaps are almost always mostly contiguous. Updates #35514. Change-Id: I3ad07f1c2b5b9340acf59ecc3b9ae09e884814fe Reviewed-on: https://go-review.googlesource.com/c/go/+/207757 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Reviewed-by: Austin Clements <austin@google.com>
2019-11-15 23:30:30 +00:00
"One": {
chunks: []ChunkIdx{
BaseChunkIdx,
},
inUse: []AddrRange{
{PageBase(BaseChunkIdx, 0), PageBase(BaseChunkIdx+1, 0)},
},
},
"Contiguous2": {
chunks: []ChunkIdx{
BaseChunkIdx,
BaseChunkIdx + 1,
},
inUse: []AddrRange{
{PageBase(BaseChunkIdx, 0), PageBase(BaseChunkIdx+2, 0)},
},
},
"Contiguous5": {
chunks: []ChunkIdx{
BaseChunkIdx,
BaseChunkIdx + 1,
BaseChunkIdx + 2,
BaseChunkIdx + 3,
BaseChunkIdx + 4,
},
inUse: []AddrRange{
{PageBase(BaseChunkIdx, 0), PageBase(BaseChunkIdx+5, 0)},
},
},
"Discontiguous": {
chunks: []ChunkIdx{
BaseChunkIdx,
BaseChunkIdx + 2,
BaseChunkIdx + 4,
},
inUse: []AddrRange{
{PageBase(BaseChunkIdx, 0), PageBase(BaseChunkIdx+1, 0)},
{PageBase(BaseChunkIdx+2, 0), PageBase(BaseChunkIdx+3, 0)},
{PageBase(BaseChunkIdx+4, 0), PageBase(BaseChunkIdx+5, 0)},
},
},
"Mixed": {
chunks: []ChunkIdx{
BaseChunkIdx,
BaseChunkIdx + 1,
BaseChunkIdx + 2,
BaseChunkIdx + 4,
},
inUse: []AddrRange{
{PageBase(BaseChunkIdx, 0), PageBase(BaseChunkIdx+3, 0)},
{PageBase(BaseChunkIdx+4, 0), PageBase(BaseChunkIdx+5, 0)},
},
},
"WildlyDiscontiguous": {
chunks: []ChunkIdx{
BaseChunkIdx,
BaseChunkIdx + 1,
BaseChunkIdx + 0x10,
BaseChunkIdx + 0x21,
},
inUse: []AddrRange{
{PageBase(BaseChunkIdx, 0), PageBase(BaseChunkIdx+2, 0)},
{PageBase(BaseChunkIdx+0x10, 0), PageBase(BaseChunkIdx+0x11, 0)},
{PageBase(BaseChunkIdx+0x21, 0), PageBase(BaseChunkIdx+0x22, 0)},
},
},
runtime: fix off-by-1 error on address ranges When growing the address ranges, the new length is the old length + 1. Fixes #36113. Change-Id: I1b425f78e473cfa3cbdfe6113e166663f41fc9f3 Reviewed-on: https://go-review.googlesource.com/c/go/+/211157 Run-TryBot: Cherry Zhang <cherryyz@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Michael Knyszek <mknyszek@google.com>
2019-12-12 16:41:24 -05:00
"ManyDiscontiguous": {
// The initial cap is 16. Test 33 ranges, to exercise the growth path (twice).
chunks: []ChunkIdx{
BaseChunkIdx, BaseChunkIdx + 2, BaseChunkIdx + 4, BaseChunkIdx + 6,
BaseChunkIdx + 8, BaseChunkIdx + 10, BaseChunkIdx + 12, BaseChunkIdx + 14,
BaseChunkIdx + 16, BaseChunkIdx + 18, BaseChunkIdx + 20, BaseChunkIdx + 22,
BaseChunkIdx + 24, BaseChunkIdx + 26, BaseChunkIdx + 28, BaseChunkIdx + 30,
BaseChunkIdx + 32, BaseChunkIdx + 34, BaseChunkIdx + 36, BaseChunkIdx + 38,
BaseChunkIdx + 40, BaseChunkIdx + 42, BaseChunkIdx + 44, BaseChunkIdx + 46,
BaseChunkIdx + 48, BaseChunkIdx + 50, BaseChunkIdx + 52, BaseChunkIdx + 54,
BaseChunkIdx + 56, BaseChunkIdx + 58, BaseChunkIdx + 60, BaseChunkIdx + 62,
BaseChunkIdx + 64,
},
inUse: []AddrRange{
{PageBase(BaseChunkIdx, 0), PageBase(BaseChunkIdx+1, 0)},
{PageBase(BaseChunkIdx+2, 0), PageBase(BaseChunkIdx+3, 0)},
{PageBase(BaseChunkIdx+4, 0), PageBase(BaseChunkIdx+5, 0)},
{PageBase(BaseChunkIdx+6, 0), PageBase(BaseChunkIdx+7, 0)},
{PageBase(BaseChunkIdx+8, 0), PageBase(BaseChunkIdx+9, 0)},
{PageBase(BaseChunkIdx+10, 0), PageBase(BaseChunkIdx+11, 0)},
{PageBase(BaseChunkIdx+12, 0), PageBase(BaseChunkIdx+13, 0)},
{PageBase(BaseChunkIdx+14, 0), PageBase(BaseChunkIdx+15, 0)},
{PageBase(BaseChunkIdx+16, 0), PageBase(BaseChunkIdx+17, 0)},
{PageBase(BaseChunkIdx+18, 0), PageBase(BaseChunkIdx+19, 0)},
{PageBase(BaseChunkIdx+20, 0), PageBase(BaseChunkIdx+21, 0)},
{PageBase(BaseChunkIdx+22, 0), PageBase(BaseChunkIdx+23, 0)},
{PageBase(BaseChunkIdx+24, 0), PageBase(BaseChunkIdx+25, 0)},
{PageBase(BaseChunkIdx+26, 0), PageBase(BaseChunkIdx+27, 0)},
{PageBase(BaseChunkIdx+28, 0), PageBase(BaseChunkIdx+29, 0)},
{PageBase(BaseChunkIdx+30, 0), PageBase(BaseChunkIdx+31, 0)},
{PageBase(BaseChunkIdx+32, 0), PageBase(BaseChunkIdx+33, 0)},
{PageBase(BaseChunkIdx+34, 0), PageBase(BaseChunkIdx+35, 0)},
{PageBase(BaseChunkIdx+36, 0), PageBase(BaseChunkIdx+37, 0)},
{PageBase(BaseChunkIdx+38, 0), PageBase(BaseChunkIdx+39, 0)},
{PageBase(BaseChunkIdx+40, 0), PageBase(BaseChunkIdx+41, 0)},
{PageBase(BaseChunkIdx+42, 0), PageBase(BaseChunkIdx+43, 0)},
{PageBase(BaseChunkIdx+44, 0), PageBase(BaseChunkIdx+45, 0)},
{PageBase(BaseChunkIdx+46, 0), PageBase(BaseChunkIdx+47, 0)},
{PageBase(BaseChunkIdx+48, 0), PageBase(BaseChunkIdx+49, 0)},
{PageBase(BaseChunkIdx+50, 0), PageBase(BaseChunkIdx+51, 0)},
{PageBase(BaseChunkIdx+52, 0), PageBase(BaseChunkIdx+53, 0)},
{PageBase(BaseChunkIdx+54, 0), PageBase(BaseChunkIdx+55, 0)},
{PageBase(BaseChunkIdx+56, 0), PageBase(BaseChunkIdx+57, 0)},
{PageBase(BaseChunkIdx+58, 0), PageBase(BaseChunkIdx+59, 0)},
{PageBase(BaseChunkIdx+60, 0), PageBase(BaseChunkIdx+61, 0)},
{PageBase(BaseChunkIdx+62, 0), PageBase(BaseChunkIdx+63, 0)},
{PageBase(BaseChunkIdx+64, 0), PageBase(BaseChunkIdx+65, 0)},
},
},
runtime: track ranges of address space which are owned by the heap This change adds a new inUse field to the allocator which tracks ranges of addresses that are owned by the heap. It is updated on each heap growth. These ranges are tracked in an array which is kept sorted. In practice this array shouldn't exceed its initial allocation except in rare cases and thus should be small (ideally exactly 1 element in size). In a hypothetical worst-case scenario wherein we have a 1 TiB heap and 4 MiB arenas (note that the address ranges will never be at a smaller granularity than an arena, since arenas are always allocated contiguously), inUse would use at most 4 MiB of memory if the heap mappings were completely discontiguous (highly unlikely) with an additional 2 MiB leaked from previous allocations. Furthermore, the copies that are done to keep the inUse array sorted will copy at most 4 MiB of memory in such a scenario, which, assuming a conservative copying rate of 5 GiB/s, amounts to about 800µs. However, note that in practice: 1) Most 64-bit platforms have 64 MiB arenas. 2) The copies should incur little-to-no page faults, meaning a copy rate closer to 25-50 GiB/s is expected. 3) Go heaps are almost always mostly contiguous. Updates #35514. Change-Id: I3ad07f1c2b5b9340acf59ecc3b9ae09e884814fe Reviewed-on: https://go-review.googlesource.com/c/go/+/207757 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Reviewed-by: Austin Clements <austin@google.com>
2019-11-15 23:30:30 +00:00
}
runtime: use inUse ranges to map in summary memory only as needed Prior to this change, if the heap was very discontiguous (such as in TestArenaCollision) it's possible we could map a large amount of memory as R/W and commit it. We would use only the start and end to track what should be mapped, and we would extend that mapping as needed to accomodate a potentially fragmented address space. After this change, we only map exactly the part of the summary arrays that we need by using the inUse ranges from the previous change. This reduces the GCSys footprint of TestArenaCollision from 300 MiB to 18 MiB. Because summaries are no longer mapped contiguously, this means the scavenger can no longer iterate directly. This change also updates the scavenger to borrow ranges out of inUse and iterate over only the parts of the heap which are actually currently in use. This is both an optimization and necessary for correctness. Fixes #35514. Change-Id: I96bf0c73ed0d2d89a00202ece7b9d089a53bac90 Reviewed-on: https://go-review.googlesource.com/c/go/+/207758 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-11-18 19:23:39 +00:00
if PageAlloc64Bit != 0 {
tests["ExtremelyDiscontiguous"] = test{
chunks: []ChunkIdx{
BaseChunkIdx,
BaseChunkIdx + 0x100000, // constant translates to O(TiB)
},
inUse: []AddrRange{
{PageBase(BaseChunkIdx, 0), PageBase(BaseChunkIdx+1, 0)},
{PageBase(BaseChunkIdx+0x100000, 0), PageBase(BaseChunkIdx+0x100001, 0)},
},
}
}
runtime: track ranges of address space which are owned by the heap This change adds a new inUse field to the allocator which tracks ranges of addresses that are owned by the heap. It is updated on each heap growth. These ranges are tracked in an array which is kept sorted. In practice this array shouldn't exceed its initial allocation except in rare cases and thus should be small (ideally exactly 1 element in size). In a hypothetical worst-case scenario wherein we have a 1 TiB heap and 4 MiB arenas (note that the address ranges will never be at a smaller granularity than an arena, since arenas are always allocated contiguously), inUse would use at most 4 MiB of memory if the heap mappings were completely discontiguous (highly unlikely) with an additional 2 MiB leaked from previous allocations. Furthermore, the copies that are done to keep the inUse array sorted will copy at most 4 MiB of memory in such a scenario, which, assuming a conservative copying rate of 5 GiB/s, amounts to about 800µs. However, note that in practice: 1) Most 64-bit platforms have 64 MiB arenas. 2) The copies should incur little-to-no page faults, meaning a copy rate closer to 25-50 GiB/s is expected. 3) Go heaps are almost always mostly contiguous. Updates #35514. Change-Id: I3ad07f1c2b5b9340acf59ecc3b9ae09e884814fe Reviewed-on: https://go-review.googlesource.com/c/go/+/207757 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com> Reviewed-by: Austin Clements <austin@google.com>
2019-11-15 23:30:30 +00:00
for name, v := range tests {
v := v
t.Run(name, func(t *testing.T) {
// By creating a new pageAlloc, we will
// grow it for each chunk defined in x.
x := make(map[ChunkIdx][]BitRange)
for _, c := range v.chunks {
x[c] = []BitRange{}
}
b := NewPageAlloc(x, nil)
defer FreePageAlloc(b)
got := b.InUse()
want := v.inUse
// Check for mismatches.
if len(got) != len(want) {
t.Fail()
} else {
for i := range want {
if want[i] != got[i] {
t.Fail()
break
}
}
}
if t.Failed() {
t.Logf("found inUse mismatch")
t.Logf("got:")
for i, r := range got {
t.Logf("\t#%d [0x%x, 0x%x)", i, r.Base, r.Limit)
}
t.Logf("want:")
for i, r := range want {
t.Logf("\t#%d [0x%x, 0x%x)", i, r.Base, r.Limit)
}
}
})
}
}
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
func TestPageAllocAlloc(t *testing.T) {
runtime: disable pageAlloc tests on OpenBSD in short mode This change disables pageAlloc tests on OpenBSD in short mode because pageAlloc holds relatively large virtual memory reservations and we make two during the pageAlloc tests. The runtime may also be carrying one such reservation making the virtual memory requirement for testing the Go runtime three times as much as just running a Go binary. This causes problems for folks who just want to build and test Go (all.bash) on OpenBSD but either don't have machines with at least 4ish GiB of RAM (per-process virtual memory limits are capped at some constant factor times the amount of physical memory) or their per-process virtual memory limits are low for other reasons. Fixes #36210. Change-Id: I8d89cfde448d4cd2fefff4ad6ffed90de63dd527 Reviewed-on: https://go-review.googlesource.com/c/go/+/212177 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-12-19 17:44:58 +00:00
if GOOS == "openbsd" && testing.Short() {
t.Skip("skipping because virtual memory is limited; see #36210")
}
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
type hit struct {
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
npages, base, scav uintptr
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
}
runtime: ensure that searchAddr always refers to inUse memory This change formalizes an assumption made by the page allocator, which is that (*pageAlloc).searchAddr should never refer to memory that is not represented by (*pageAlloc).inUse. The portion of address space covered by (*pageAlloc).inUse reflects the parts of the summary arrays which are guaranteed to mapped, and so looking at any summary which is not reflected there may cause a segfault. In fact, this can happen today. This change thus also removes a micro-optimization which is the only case which may cause (*pageAlloc).searchAddr to point outside of any region covered by (*pageAlloc).inUse, and adds a test verifying that the current segfault can no longer occur. Change-Id: I98b534f0ffba8656d3bd6d782f6fc22549ddf1c2 Reviewed-on: https://go-review.googlesource.com/c/go/+/216697 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-01-28 19:59:19 +00:00
type test struct {
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav map[ChunkIdx][]BitRange
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
before map[ChunkIdx][]BitRange
after map[ChunkIdx][]BitRange
hits []hit
runtime: ensure that searchAddr always refers to inUse memory This change formalizes an assumption made by the page allocator, which is that (*pageAlloc).searchAddr should never refer to memory that is not represented by (*pageAlloc).inUse. The portion of address space covered by (*pageAlloc).inUse reflects the parts of the summary arrays which are guaranteed to mapped, and so looking at any summary which is not reflected there may cause a segfault. In fact, this can happen today. This change thus also removes a micro-optimization which is the only case which may cause (*pageAlloc).searchAddr to point outside of any region covered by (*pageAlloc).inUse, and adds a test verifying that the current segfault can no longer occur. Change-Id: I98b534f0ffba8656d3bd6d782f6fc22549ddf1c2 Reviewed-on: https://go-review.googlesource.com/c/go/+/216697 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-01-28 19:59:19 +00:00
}
tests := map[string]test{
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
"AllFree1": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, 1}, {2, 2}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{1, PageBase(BaseChunkIdx, 0), PageSize},
{1, PageBase(BaseChunkIdx, 1), 0},
{1, PageBase(BaseChunkIdx, 2), PageSize},
{1, PageBase(BaseChunkIdx, 3), PageSize},
{1, PageBase(BaseChunkIdx, 4), 0},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, 5}},
},
},
"ManyArena1": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
BaseChunkIdx + 2: {{0, PallocChunkPages - 1}},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
BaseChunkIdx + 2: {{0, PallocChunkPages}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{1, PageBase(BaseChunkIdx+2, PallocChunkPages-1), PageSize},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
BaseChunkIdx + 2: {{0, PallocChunkPages}},
},
},
"NotContiguous1": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 0xff: {{0, 0}},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 0xff: {{0, PallocChunkPages}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{1, PageBase(BaseChunkIdx+0xff, 0), PageSize},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 0xff: {{0, 1}},
},
},
"AllFree2": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, 3}, {7, 1}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{2, PageBase(BaseChunkIdx, 0), 2 * PageSize},
{2, PageBase(BaseChunkIdx, 2), PageSize},
{2, PageBase(BaseChunkIdx, 4), 0},
{2, PageBase(BaseChunkIdx, 6), PageSize},
{2, PageBase(BaseChunkIdx, 8), 0},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, 10}},
},
},
"Straddle2": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages - 1}},
BaseChunkIdx + 1: {{1, PallocChunkPages - 1}},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{PallocChunkPages - 1, 1}},
BaseChunkIdx + 1: {},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{2, PageBase(BaseChunkIdx, PallocChunkPages-1), PageSize},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
},
},
"AllFree5": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, 8}, {9, 1}, {17, 5}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{5, PageBase(BaseChunkIdx, 0), 5 * PageSize},
{5, PageBase(BaseChunkIdx, 5), 4 * PageSize},
{5, PageBase(BaseChunkIdx, 10), 0},
{5, PageBase(BaseChunkIdx, 15), 3 * PageSize},
{5, PageBase(BaseChunkIdx, 20), 2 * PageSize},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, 25}},
},
},
"AllFree64": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{21, 1}, {63, 65}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{64, PageBase(BaseChunkIdx, 0), 2 * PageSize},
{64, PageBase(BaseChunkIdx, 64), 64 * PageSize},
{64, PageBase(BaseChunkIdx, 128), 0},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, 192}},
},
},
"AllFree65": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{129, 1}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{65, PageBase(BaseChunkIdx, 0), 0},
{65, PageBase(BaseChunkIdx, 65), PageSize},
{65, PageBase(BaseChunkIdx, 130), 0},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, 195}},
},
},
"ExhaustPallocChunkPages-3": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{10, 1}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{PallocChunkPages - 3, PageBase(BaseChunkIdx, 0), PageSize},
{PallocChunkPages - 3, 0, 0},
{1, PageBase(BaseChunkIdx, PallocChunkPages-3), 0},
{2, PageBase(BaseChunkIdx, PallocChunkPages-2), 0},
{1, 0, 0},
{PallocChunkPages - 3, 0, 0},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
},
},
"AllFreePallocChunkPages": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, 1}, {PallocChunkPages - 1, 1}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{PallocChunkPages, PageBase(BaseChunkIdx, 0), 2 * PageSize},
{PallocChunkPages, 0, 0},
{1, 0, 0},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
},
},
"StraddlePallocChunkPages": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages / 2}},
BaseChunkIdx + 1: {{PallocChunkPages / 2, PallocChunkPages / 2}},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
BaseChunkIdx + 1: {{3, 100}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{PallocChunkPages, PageBase(BaseChunkIdx, PallocChunkPages/2), 100 * PageSize},
{PallocChunkPages, 0, 0},
{1, 0, 0},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
},
},
"StraddlePallocChunkPages+1": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages / 2}},
BaseChunkIdx + 1: {},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{PallocChunkPages + 1, PageBase(BaseChunkIdx, PallocChunkPages/2), (PallocChunkPages + 1) * PageSize},
{PallocChunkPages, 0, 0},
{1, PageBase(BaseChunkIdx+1, PallocChunkPages/2+1), PageSize},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages/2 + 2}},
},
},
"AllFreePallocChunkPages*2": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
BaseChunkIdx + 1: {},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
BaseChunkIdx + 1: {},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{PallocChunkPages * 2, PageBase(BaseChunkIdx, 0), 0},
{PallocChunkPages * 2, 0, 0},
{1, 0, 0},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
},
},
"NotContiguousPallocChunkPages*2": {
before: map[ChunkIdx][]BitRange{
runtime: make the test addresses for pageAlloc smaller on 32-bit This change makes the test addresses start at 1 GiB instead of 2 GiB to support mips and mipsle, which only have 31-bit address spaces. It also changes some tests to use smaller offsets for the chunk index to avoid jumping too far ahead in the address space to support 31-bit address spaces. The tests don't require such large jumps for what they're testing anyway. Updates #35112. Fixes #35440. Change-Id: Ic68ff2b0a1f10ef37ac00d4bb5b910ddcdc76f2e Reviewed-on: https://go-review.googlesource.com/c/go/+/205938 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-07 22:42:38 +00:00
BaseChunkIdx: {},
BaseChunkIdx + 0x40: {},
BaseChunkIdx + 0x41: {},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
runtime: make the test addresses for pageAlloc smaller on 32-bit This change makes the test addresses start at 1 GiB instead of 2 GiB to support mips and mipsle, which only have 31-bit address spaces. It also changes some tests to use smaller offsets for the chunk index to avoid jumping too far ahead in the address space to support 31-bit address spaces. The tests don't require such large jumps for what they're testing anyway. Updates #35112. Fixes #35440. Change-Id: Ic68ff2b0a1f10ef37ac00d4bb5b910ddcdc76f2e Reviewed-on: https://go-review.googlesource.com/c/go/+/205938 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-07 22:42:38 +00:00
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 0x40: {},
BaseChunkIdx + 0x41: {},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: make the test addresses for pageAlloc smaller on 32-bit This change makes the test addresses start at 1 GiB instead of 2 GiB to support mips and mipsle, which only have 31-bit address spaces. It also changes some tests to use smaller offsets for the chunk index to avoid jumping too far ahead in the address space to support 31-bit address spaces. The tests don't require such large jumps for what they're testing anyway. Updates #35112. Fixes #35440. Change-Id: Ic68ff2b0a1f10ef37ac00d4bb5b910ddcdc76f2e Reviewed-on: https://go-review.googlesource.com/c/go/+/205938 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-07 22:42:38 +00:00
{PallocChunkPages * 2, PageBase(BaseChunkIdx+0x40, 0), 0},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{21, PageBase(BaseChunkIdx, 0), 21 * PageSize},
{1, PageBase(BaseChunkIdx, 21), PageSize},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
runtime: make the test addresses for pageAlloc smaller on 32-bit This change makes the test addresses start at 1 GiB instead of 2 GiB to support mips and mipsle, which only have 31-bit address spaces. It also changes some tests to use smaller offsets for the chunk index to avoid jumping too far ahead in the address space to support 31-bit address spaces. The tests don't require such large jumps for what they're testing anyway. Updates #35112. Fixes #35440. Change-Id: Ic68ff2b0a1f10ef37ac00d4bb5b910ddcdc76f2e Reviewed-on: https://go-review.googlesource.com/c/go/+/205938 Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2019-11-07 22:42:38 +00:00
BaseChunkIdx: {{0, 22}},
BaseChunkIdx + 0x40: {{0, PallocChunkPages}},
BaseChunkIdx + 0x41: {{0, PallocChunkPages}},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
},
"StraddlePallocChunkPages*2": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages / 2}},
BaseChunkIdx + 1: {},
BaseChunkIdx + 2: {{PallocChunkPages / 2, PallocChunkPages / 2}},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, 7}},
BaseChunkIdx + 1: {{3, 5}, {121, 10}},
BaseChunkIdx + 2: {{PallocChunkPages/2 + 12, 2}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{PallocChunkPages * 2, PageBase(BaseChunkIdx, PallocChunkPages/2), 15 * PageSize},
{PallocChunkPages * 2, 0, 0},
{1, 0, 0},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
BaseChunkIdx + 2: {{0, PallocChunkPages}},
},
},
"StraddlePallocChunkPages*5/4": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages * 3 / 4}},
BaseChunkIdx + 2: {{0, PallocChunkPages * 3 / 4}},
BaseChunkIdx + 3: {{0, 0}},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{PallocChunkPages / 2, PallocChunkPages/4 + 1}},
BaseChunkIdx + 2: {{PallocChunkPages / 3, 1}},
BaseChunkIdx + 3: {{PallocChunkPages * 2 / 3, 1}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{PallocChunkPages * 5 / 4, PageBase(BaseChunkIdx+2, PallocChunkPages*3/4), PageSize},
{PallocChunkPages * 5 / 4, 0, 0},
{1, PageBase(BaseChunkIdx+1, PallocChunkPages*3/4), PageSize},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages*3/4 + 1}},
BaseChunkIdx + 2: {{0, PallocChunkPages}},
BaseChunkIdx + 3: {{0, PallocChunkPages}},
},
},
"AllFreePallocChunkPages*7+5": {
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
BaseChunkIdx + 1: {},
BaseChunkIdx + 2: {},
BaseChunkIdx + 3: {},
BaseChunkIdx + 4: {},
BaseChunkIdx + 5: {},
BaseChunkIdx + 6: {},
BaseChunkIdx + 7: {},
},
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{50, 1}},
BaseChunkIdx + 1: {{31, 1}},
BaseChunkIdx + 2: {{7, 1}},
BaseChunkIdx + 3: {{200, 1}},
BaseChunkIdx + 4: {{3, 1}},
BaseChunkIdx + 5: {{51, 1}},
BaseChunkIdx + 6: {{20, 1}},
BaseChunkIdx + 7: {{1, 1}},
},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
hits: []hit{
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
{PallocChunkPages*7 + 5, PageBase(BaseChunkIdx, 0), 8 * PageSize},
{PallocChunkPages*7 + 5, 0, 0},
{1, PageBase(BaseChunkIdx+7, 5), 0},
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
BaseChunkIdx + 2: {{0, PallocChunkPages}},
BaseChunkIdx + 3: {{0, PallocChunkPages}},
BaseChunkIdx + 4: {{0, PallocChunkPages}},
BaseChunkIdx + 5: {{0, PallocChunkPages}},
BaseChunkIdx + 6: {{0, PallocChunkPages}},
BaseChunkIdx + 7: {{0, 6}},
},
},
}
runtime: ensure that searchAddr always refers to inUse memory This change formalizes an assumption made by the page allocator, which is that (*pageAlloc).searchAddr should never refer to memory that is not represented by (*pageAlloc).inUse. The portion of address space covered by (*pageAlloc).inUse reflects the parts of the summary arrays which are guaranteed to mapped, and so looking at any summary which is not reflected there may cause a segfault. In fact, this can happen today. This change thus also removes a micro-optimization which is the only case which may cause (*pageAlloc).searchAddr to point outside of any region covered by (*pageAlloc).inUse, and adds a test verifying that the current segfault can no longer occur. Change-Id: I98b534f0ffba8656d3bd6d782f6fc22549ddf1c2 Reviewed-on: https://go-review.googlesource.com/c/go/+/216697 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-01-28 19:59:19 +00:00
if PageAlloc64Bit != 0 {
const chunkIdxBigJump = 0x100000 // chunk index offset which translates to O(TiB)
// This test attempts to trigger a bug wherein we look at unmapped summary
// memory that isn't just in the case where we exhaust the heap.
//
// It achieves this by placing a chunk such that its summary will be
// at the very end of a physical page. It then also places another chunk
// much further up in the address space, such that any allocations into the
// first chunk do not exhaust the heap and the second chunk's summary is not in the
// page immediately adjacent to the first chunk's summary's page.
// Allocating into this first chunk to exhaustion and then into the second
// chunk may then trigger a check in the allocator which erroneously looks at
// unmapped summary memory and crashes.
// Figure out how many chunks are in a physical page, then align BaseChunkIdx
// to a physical page in the chunk summary array. Here we only assume that
// each summary array is aligned to some physical page.
sumsPerPhysPage := ChunkIdx(PhysPageSize / PallocSumBytes)
baseChunkIdx := BaseChunkIdx &^ (sumsPerPhysPage - 1)
tests["DiscontiguousMappedSumBoundary"] = test{
before: map[ChunkIdx][]BitRange{
baseChunkIdx + sumsPerPhysPage - 1: {},
baseChunkIdx + chunkIdxBigJump: {},
},
scav: map[ChunkIdx][]BitRange{
baseChunkIdx + sumsPerPhysPage - 1: {},
baseChunkIdx + chunkIdxBigJump: {},
},
hits: []hit{
{PallocChunkPages - 1, PageBase(baseChunkIdx+sumsPerPhysPage-1, 0), 0},
{1, PageBase(baseChunkIdx+sumsPerPhysPage-1, PallocChunkPages-1), 0},
{1, PageBase(baseChunkIdx+chunkIdxBigJump, 0), 0},
{PallocChunkPages - 1, PageBase(baseChunkIdx+chunkIdxBigJump, 1), 0},
{1, 0, 0},
},
after: map[ChunkIdx][]BitRange{
baseChunkIdx + sumsPerPhysPage - 1: {{0, PallocChunkPages}},
baseChunkIdx + chunkIdxBigJump: {{0, PallocChunkPages}},
},
}
runtime: validate candidate searchAddr in pageAlloc.find Currently pageAlloc.find attempts to find a better estimate for the first free page in the heap, even if the space its looking for isn't necessarily going to be the first free page in the heap (e.g. if npages >= 2). However, in doing so it has the potential to return a searchAddr candidate that doesn't actually correspond to mapped memory, but this candidate might still be adopted. As a result, pageAlloc.alloc's fast path may look at unmapped summary memory and segfault. This case is rare on most operating systems since the heap is kept fairly contiguous, so the chance that the candidate searchAddr discovered is unmapped is fairly low. Even so, this is totally possible and outside the user's control when it happens (in fact, it's likely to happen consistently for a given user on a given system). Fix this problem by ensuring that our candidate always points to mapped memory. We do this by looking at mheap's arenas structure first. If it turns out our candidate doesn't correspond to mapped memory, then we look at inUse to round up the searchAddr to the next mapped address. While we're here, clean up some documentation related to searchAddr. Fixes #40191. Change-Id: I759efec78987e4a8fde466ae45aabbaa3d9d4214 Reviewed-on: https://go-review.googlesource.com/c/go/+/242680 Run-TryBot: Michael Knyszek <mknyszek@google.com> Reviewed-by: Austin Clements <austin@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org>
2020-07-13 19:51:50 +00:00
// Test to check for issue #40191. Essentially, the candidate searchAddr
// discovered by find may not point to mapped memory, so we need to handle
// that explicitly.
//
// chunkIdxSmallOffset is an offset intended to be used within chunkIdxBigJump.
// It is far enough within chunkIdxBigJump that the summaries at the beginning
// of an address range the size of chunkIdxBigJump will not be mapped in.
const chunkIdxSmallOffset = 0x503
tests["DiscontiguousBadSearchAddr"] = test{
before: map[ChunkIdx][]BitRange{
// The mechanism for the bug involves three chunks, A, B, and C, which are
// far apart in the address space. In particular, B is chunkIdxBigJump +
// chunkIdxSmalloffset chunks away from B, and C is 2*chunkIdxBigJump chunks
// away from A. A has 1 page free, B has several (NOT at the end of B), and
// C is totally free.
// Note that B's free memory must not be at the end of B because the fast
// path in the page allocator will check if the searchAddr even gives us
// enough space to place the allocation in a chunk before accessing the
// summary.
BaseChunkIdx + chunkIdxBigJump*0: {{0, PallocChunkPages - 1}},
BaseChunkIdx + chunkIdxBigJump*1 + chunkIdxSmallOffset: {
{0, PallocChunkPages - 10},
{PallocChunkPages - 1, 1},
},
BaseChunkIdx + chunkIdxBigJump*2: {},
},
scav: map[ChunkIdx][]BitRange{
BaseChunkIdx + chunkIdxBigJump*0: {},
BaseChunkIdx + chunkIdxBigJump*1 + chunkIdxSmallOffset: {},
BaseChunkIdx + chunkIdxBigJump*2: {},
},
hits: []hit{
// We first allocate into A to set the page allocator's searchAddr to the
// end of that chunk. That is the only purpose A serves.
{1, PageBase(BaseChunkIdx, PallocChunkPages-1), 0},
// Then, we make a big allocation that doesn't fit into B, and so must be
// fulfilled by C.
//
// On the way to fulfilling the allocation into C, we estimate searchAddr
// using the summary structure, but that will give us a searchAddr of
// B's base address minus chunkIdxSmallOffset chunks. These chunks will
// not be mapped.
{100, PageBase(baseChunkIdx+chunkIdxBigJump*2, 0), 0},
// Now we try to make a smaller allocation that can be fulfilled by B.
// In an older implementation of the page allocator, this will segfault,
// because this last allocation will first try to access the summary
// for B's base address minus chunkIdxSmallOffset chunks in the fast path,
// and this will not be mapped.
{9, PageBase(baseChunkIdx+chunkIdxBigJump*1+chunkIdxSmallOffset, PallocChunkPages-10), 0},
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx + chunkIdxBigJump*0: {{0, PallocChunkPages}},
BaseChunkIdx + chunkIdxBigJump*1 + chunkIdxSmallOffset: {{0, PallocChunkPages}},
BaseChunkIdx + chunkIdxBigJump*2: {{0, 100}},
},
}
runtime: ensure that searchAddr always refers to inUse memory This change formalizes an assumption made by the page allocator, which is that (*pageAlloc).searchAddr should never refer to memory that is not represented by (*pageAlloc).inUse. The portion of address space covered by (*pageAlloc).inUse reflects the parts of the summary arrays which are guaranteed to mapped, and so looking at any summary which is not reflected there may cause a segfault. In fact, this can happen today. This change thus also removes a micro-optimization which is the only case which may cause (*pageAlloc).searchAddr to point outside of any region covered by (*pageAlloc).inUse, and adds a test verifying that the current segfault can no longer occur. Change-Id: I98b534f0ffba8656d3bd6d782f6fc22549ddf1c2 Reviewed-on: https://go-review.googlesource.com/c/go/+/216697 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-01-28 19:59:19 +00:00
}
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
for name, v := range tests {
v := v
t.Run(name, func(t *testing.T) {
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
b := NewPageAlloc(v.before, v.scav)
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
defer FreePageAlloc(b)
for iter, i := range v.hits {
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
a, s := b.Alloc(i.npages)
if a != i.base {
t.Fatalf("bad alloc #%d: want base 0x%x, got 0x%x", iter+1, i.base, a)
}
if s != i.scav {
t.Fatalf("bad alloc #%d: want scav %d, got %d", iter+1, i.scav, s)
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
}
}
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
want := NewPageAlloc(v.after, v.scav)
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
defer FreePageAlloc(want)
checkPageAlloc(t, want, b)
})
}
}
func TestPageAllocExhaust(t *testing.T) {
runtime: disable pageAlloc tests on OpenBSD in short mode This change disables pageAlloc tests on OpenBSD in short mode because pageAlloc holds relatively large virtual memory reservations and we make two during the pageAlloc tests. The runtime may also be carrying one such reservation making the virtual memory requirement for testing the Go runtime three times as much as just running a Go binary. This causes problems for folks who just want to build and test Go (all.bash) on OpenBSD but either don't have machines with at least 4ish GiB of RAM (per-process virtual memory limits are capped at some constant factor times the amount of physical memory) or their per-process virtual memory limits are low for other reasons. Fixes #36210. Change-Id: I8d89cfde448d4cd2fefff4ad6ffed90de63dd527 Reviewed-on: https://go-review.googlesource.com/c/go/+/212177 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-12-19 17:44:58 +00:00
if GOOS == "openbsd" && testing.Short() {
t.Skip("skipping because virtual memory is limited; see #36210")
}
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
for _, npages := range []uintptr{1, 2, 3, 4, 5, 8, 16, 64, 1024, 1025, 2048, 2049} {
npages := npages
t.Run(fmt.Sprintf("%d", npages), func(t *testing.T) {
// Construct b.
bDesc := make(map[ChunkIdx][]BitRange)
for i := ChunkIdx(0); i < 4; i++ {
bDesc[BaseChunkIdx+i] = []BitRange{}
}
runtime: add scavenging code for new page allocator This change adds a scavenger for the new page allocator along with tests. The scavenger walks over the heap backwards once per GC, looking for memory to scavenge. It walks across the heap without any lock held, searching optimistically. If it finds what appears to be a scavenging candidate it acquires the heap lock and attempts to verify it. Upon verification it then scavenges. Notably, unlike the old scavenger, it doesn't show any preference for huge pages and instead follows a more strict last-page-first policy. Updates #35112. Change-Id: I0621ef73c999a471843eab2d1307ae5679dd18d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/195697 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-21 00:24:25 +00:00
b := NewPageAlloc(bDesc, nil)
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
defer FreePageAlloc(b)
// Allocate into b with npages until we've exhausted the heap.
nAlloc := (PallocChunkPages * 4) / int(npages)
for i := 0; i < nAlloc; i++ {
addr := PageBase(BaseChunkIdx, uint(i)*uint(npages))
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
if a, _ := b.Alloc(npages); a != addr {
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
t.Fatalf("bad alloc #%d: want 0x%x, got 0x%x", i+1, addr, a)
}
}
// Check to make sure the next allocation fails.
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
if a, _ := b.Alloc(npages); a != 0 {
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
t.Fatalf("bad alloc #%d: want 0, got 0x%x", nAlloc, a)
}
// Construct what we want the heap to look like now.
allocPages := nAlloc * int(npages)
wantDesc := make(map[ChunkIdx][]BitRange)
for i := ChunkIdx(0); i < 4; i++ {
if allocPages >= PallocChunkPages {
wantDesc[BaseChunkIdx+i] = []BitRange{{0, PallocChunkPages}}
allocPages -= PallocChunkPages
} else if allocPages > 0 {
wantDesc[BaseChunkIdx+i] = []BitRange{{0, uint(allocPages)}}
allocPages = 0
} else {
wantDesc[BaseChunkIdx+i] = []BitRange{}
}
}
runtime: add scavenging code for new page allocator This change adds a scavenger for the new page allocator along with tests. The scavenger walks over the heap backwards once per GC, looking for memory to scavenge. It walks across the heap without any lock held, searching optimistically. If it finds what appears to be a scavenging candidate it acquires the heap lock and attempts to verify it. Upon verification it then scavenges. Notably, unlike the old scavenger, it doesn't show any preference for huge pages and instead follows a more strict last-page-first policy. Updates #35112. Change-Id: I0621ef73c999a471843eab2d1307ae5679dd18d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/195697 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-21 00:24:25 +00:00
want := NewPageAlloc(wantDesc, nil)
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
defer FreePageAlloc(want)
// Check to make sure the heap b matches what we want.
checkPageAlloc(t, want, b)
})
}
}
func TestPageAllocFree(t *testing.T) {
runtime: disable pageAlloc tests on OpenBSD in short mode This change disables pageAlloc tests on OpenBSD in short mode because pageAlloc holds relatively large virtual memory reservations and we make two during the pageAlloc tests. The runtime may also be carrying one such reservation making the virtual memory requirement for testing the Go runtime three times as much as just running a Go binary. This causes problems for folks who just want to build and test Go (all.bash) on OpenBSD but either don't have machines with at least 4ish GiB of RAM (per-process virtual memory limits are capped at some constant factor times the amount of physical memory) or their per-process virtual memory limits are low for other reasons. Fixes #36210. Change-Id: I8d89cfde448d4cd2fefff4ad6ffed90de63dd527 Reviewed-on: https://go-review.googlesource.com/c/go/+/212177 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-12-19 17:44:58 +00:00
if GOOS == "openbsd" && testing.Short() {
t.Skip("skipping because virtual memory is limited; see #36210")
}
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
tests := map[string]struct {
before map[ChunkIdx][]BitRange
after map[ChunkIdx][]BitRange
npages uintptr
frees []uintptr
}{
"Free1": {
npages: 1,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, 0),
PageBase(BaseChunkIdx, 1),
PageBase(BaseChunkIdx, 2),
PageBase(BaseChunkIdx, 3),
PageBase(BaseChunkIdx, 4),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{5, PallocChunkPages - 5}},
},
},
"ManyArena1": {
npages: 1,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
BaseChunkIdx + 2: {{0, PallocChunkPages}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, PallocChunkPages/2),
PageBase(BaseChunkIdx+1, 0),
PageBase(BaseChunkIdx+2, PallocChunkPages-1),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages / 2}, {PallocChunkPages/2 + 1, PallocChunkPages/2 - 1}},
BaseChunkIdx + 1: {{1, PallocChunkPages - 1}},
BaseChunkIdx + 2: {{0, PallocChunkPages - 1}},
},
},
"Free2": {
npages: 2,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, 0),
PageBase(BaseChunkIdx, 2),
PageBase(BaseChunkIdx, 4),
PageBase(BaseChunkIdx, 6),
PageBase(BaseChunkIdx, 8),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{10, PallocChunkPages - 10}},
},
},
"Straddle2": {
npages: 2,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{PallocChunkPages - 1, 1}},
BaseChunkIdx + 1: {{0, 1}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, PallocChunkPages-1),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
BaseChunkIdx + 1: {},
},
},
"Free5": {
npages: 5,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, 0),
PageBase(BaseChunkIdx, 5),
PageBase(BaseChunkIdx, 10),
PageBase(BaseChunkIdx, 15),
PageBase(BaseChunkIdx, 20),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{25, PallocChunkPages - 25}},
},
},
"Free64": {
npages: 64,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, 0),
PageBase(BaseChunkIdx, 64),
PageBase(BaseChunkIdx, 128),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{192, PallocChunkPages - 192}},
},
},
"Free65": {
npages: 65,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, 0),
PageBase(BaseChunkIdx, 65),
PageBase(BaseChunkIdx, 130),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{195, PallocChunkPages - 195}},
},
},
"FreePallocChunkPages": {
npages: PallocChunkPages,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, 0),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
},
},
"StraddlePallocChunkPages": {
npages: PallocChunkPages,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{PallocChunkPages / 2, PallocChunkPages / 2}},
BaseChunkIdx + 1: {{0, PallocChunkPages / 2}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, PallocChunkPages/2),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
BaseChunkIdx + 1: {},
},
},
"StraddlePallocChunkPages+1": {
npages: PallocChunkPages + 1,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, PallocChunkPages/2),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages / 2}},
BaseChunkIdx + 1: {{PallocChunkPages/2 + 1, PallocChunkPages/2 - 1}},
},
},
"FreePallocChunkPages*2": {
npages: PallocChunkPages * 2,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, 0),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
BaseChunkIdx + 1: {},
},
},
"StraddlePallocChunkPages*2": {
npages: PallocChunkPages * 2,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
BaseChunkIdx + 2: {{0, PallocChunkPages}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, PallocChunkPages/2),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages / 2}},
BaseChunkIdx + 1: {},
BaseChunkIdx + 2: {{PallocChunkPages / 2, PallocChunkPages / 2}},
},
},
"AllFreePallocChunkPages*7+5": {
npages: PallocChunkPages*7 + 5,
before: map[ChunkIdx][]BitRange{
BaseChunkIdx: {{0, PallocChunkPages}},
BaseChunkIdx + 1: {{0, PallocChunkPages}},
BaseChunkIdx + 2: {{0, PallocChunkPages}},
BaseChunkIdx + 3: {{0, PallocChunkPages}},
BaseChunkIdx + 4: {{0, PallocChunkPages}},
BaseChunkIdx + 5: {{0, PallocChunkPages}},
BaseChunkIdx + 6: {{0, PallocChunkPages}},
BaseChunkIdx + 7: {{0, PallocChunkPages}},
},
frees: []uintptr{
PageBase(BaseChunkIdx, 0),
},
after: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
BaseChunkIdx + 1: {},
BaseChunkIdx + 2: {},
BaseChunkIdx + 3: {},
BaseChunkIdx + 4: {},
BaseChunkIdx + 5: {},
BaseChunkIdx + 6: {},
BaseChunkIdx + 7: {{5, PallocChunkPages - 5}},
},
},
}
for name, v := range tests {
v := v
t.Run(name, func(t *testing.T) {
runtime: add scavenging code for new page allocator This change adds a scavenger for the new page allocator along with tests. The scavenger walks over the heap backwards once per GC, looking for memory to scavenge. It walks across the heap without any lock held, searching optimistically. If it finds what appears to be a scavenging candidate it acquires the heap lock and attempts to verify it. Upon verification it then scavenges. Notably, unlike the old scavenger, it doesn't show any preference for huge pages and instead follows a more strict last-page-first policy. Updates #35112. Change-Id: I0621ef73c999a471843eab2d1307ae5679dd18d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/195697 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-21 00:24:25 +00:00
b := NewPageAlloc(v.before, nil)
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
defer FreePageAlloc(b)
runtime: add scavenging code for new page allocator This change adds a scavenger for the new page allocator along with tests. The scavenger walks over the heap backwards once per GC, looking for memory to scavenge. It walks across the heap without any lock held, searching optimistically. If it finds what appears to be a scavenging candidate it acquires the heap lock and attempts to verify it. Upon verification it then scavenges. Notably, unlike the old scavenger, it doesn't show any preference for huge pages and instead follows a more strict last-page-first policy. Updates #35112. Change-Id: I0621ef73c999a471843eab2d1307ae5679dd18d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/195697 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-21 00:24:25 +00:00
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
for _, addr := range v.frees {
b.Free(addr, v.npages)
}
runtime: add scavenging code for new page allocator This change adds a scavenger for the new page allocator along with tests. The scavenger walks over the heap backwards once per GC, looking for memory to scavenge. It walks across the heap without any lock held, searching optimistically. If it finds what appears to be a scavenging candidate it acquires the heap lock and attempts to verify it. Upon verification it then scavenges. Notably, unlike the old scavenger, it doesn't show any preference for huge pages and instead follows a more strict last-page-first policy. Updates #35112. Change-Id: I0621ef73c999a471843eab2d1307ae5679dd18d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/195697 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-21 00:24:25 +00:00
want := NewPageAlloc(v.after, nil)
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
defer FreePageAlloc(want)
runtime: add scavenging code for new page allocator This change adds a scavenger for the new page allocator along with tests. The scavenger walks over the heap backwards once per GC, looking for memory to scavenge. It walks across the heap without any lock held, searching optimistically. If it finds what appears to be a scavenging candidate it acquires the heap lock and attempts to verify it. Upon verification it then scavenges. Notably, unlike the old scavenger, it doesn't show any preference for huge pages and instead follows a more strict last-page-first policy. Updates #35112. Change-Id: I0621ef73c999a471843eab2d1307ae5679dd18d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/195697 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-21 00:24:25 +00:00
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
checkPageAlloc(t, want, b)
})
}
}
func TestPageAllocAllocAndFree(t *testing.T) {
runtime: disable pageAlloc tests on OpenBSD in short mode This change disables pageAlloc tests on OpenBSD in short mode because pageAlloc holds relatively large virtual memory reservations and we make two during the pageAlloc tests. The runtime may also be carrying one such reservation making the virtual memory requirement for testing the Go runtime three times as much as just running a Go binary. This causes problems for folks who just want to build and test Go (all.bash) on OpenBSD but either don't have machines with at least 4ish GiB of RAM (per-process virtual memory limits are capped at some constant factor times the amount of physical memory) or their per-process virtual memory limits are low for other reasons. Fixes #36210. Change-Id: I8d89cfde448d4cd2fefff4ad6ffed90de63dd527 Reviewed-on: https://go-review.googlesource.com/c/go/+/212177 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com>
2019-12-19 17:44:58 +00:00
if GOOS == "openbsd" && testing.Short() {
t.Skip("skipping because virtual memory is limited; see #36210")
}
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
type hit struct {
alloc bool
npages uintptr
base uintptr
}
tests := map[string]struct {
init map[ChunkIdx][]BitRange
hits []hit
}{
// TODO(mknyszek): Write more tests here.
"Chunks8": {
init: map[ChunkIdx][]BitRange{
BaseChunkIdx: {},
BaseChunkIdx + 1: {},
BaseChunkIdx + 2: {},
BaseChunkIdx + 3: {},
BaseChunkIdx + 4: {},
BaseChunkIdx + 5: {},
BaseChunkIdx + 6: {},
BaseChunkIdx + 7: {},
},
hits: []hit{
{true, PallocChunkPages * 8, PageBase(BaseChunkIdx, 0)},
{false, PallocChunkPages * 8, PageBase(BaseChunkIdx, 0)},
{true, PallocChunkPages * 8, PageBase(BaseChunkIdx, 0)},
{false, PallocChunkPages * 8, PageBase(BaseChunkIdx, 0)},
{true, PallocChunkPages * 8, PageBase(BaseChunkIdx, 0)},
{false, PallocChunkPages * 8, PageBase(BaseChunkIdx, 0)},
{true, 1, PageBase(BaseChunkIdx, 0)},
{false, 1, PageBase(BaseChunkIdx, 0)},
{true, PallocChunkPages * 8, PageBase(BaseChunkIdx, 0)},
},
},
}
for name, v := range tests {
v := v
t.Run(name, func(t *testing.T) {
runtime: add scavenging code for new page allocator This change adds a scavenger for the new page allocator along with tests. The scavenger walks over the heap backwards once per GC, looking for memory to scavenge. It walks across the heap without any lock held, searching optimistically. If it finds what appears to be a scavenging candidate it acquires the heap lock and attempts to verify it. Upon verification it then scavenges. Notably, unlike the old scavenger, it doesn't show any preference for huge pages and instead follows a more strict last-page-first policy. Updates #35112. Change-Id: I0621ef73c999a471843eab2d1307ae5679dd18d6 Reviewed-on: https://go-review.googlesource.com/c/go/+/195697 Reviewed-by: Keith Randall <khr@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-21 00:24:25 +00:00
b := NewPageAlloc(v.init, nil)
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
defer FreePageAlloc(b)
for iter, i := range v.hits {
if i.alloc {
runtime: count scavenged bits for new allocation for new page allocator This change makes it so that the new page allocator returns the number of pages that are scavenged in a new allocation so that mheap can update memstats appropriately. The accounting could be embedded into pageAlloc, but that would make the new allocator more difficult to test. Updates #35112. Change-Id: I0f94f563d7af2458e6d534f589d2e7dd6af26d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/195698 Reviewed-by: Austin Clements <austin@google.com>
2019-09-10 18:53:51 +00:00
if a, _ := b.Alloc(i.npages); a != i.base {
runtime: add new page allocator core This change adds a new bitmap-based allocator to the runtime with tests. It does not yet integrate the page allocator into the runtime and thus this change is almost purely additive. Updates #35112. Change-Id: Ic3d024c28abee8be8797d3918116a80f901cc2bf Reviewed-on: https://go-review.googlesource.com/c/go/+/190622 Run-TryBot: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Austin Clements <austin@google.com>
2019-08-14 16:32:12 +00:00
t.Fatalf("bad alloc #%d: want 0x%x, got 0x%x", iter+1, i.base, a)
}
} else {
b.Free(i.base, i.npages)
}
}
})
}
}
Reference in a new issue Copy permalink