Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-12-08 06:09:58 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Kernel/VM/AnonymousVMObject.cpp

499 lines
18 KiB
C++
Raw Normal View History

Meta: Add license header to source files As suggested by Joshua, this commit adds the 2-clause BSD license as a comment block to the top of every source file. For the first pass, I've just added myself for simplicity. I encourage everyone to add themselves as copyright holders of any file they've added or modified in some significant way. If I've added myself in error somewhere, feel free to replace it with the appropriate copyright holder instead. Going forward, all new source files should include a license header.
2020-01-18 09:38:21 +01:00
/*
* Copyright (c) 2018-2020, Andreas Kling <kling@serenityos.org>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice, this
* list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
* SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
#include <Kernel/Process.h>
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
#include <Kernel/VM/AnonymousVMObject.h>
Kernel: Use a shared physical page for zero-filled pages until written This patch adds a globally shared zero-filled PhysicalPage that will be mapped into every slot of every zero-filled AnonymousVMObject until that page is written to, achieving CoW-like zero-filled pages. Initial testing show that this doesn't actually achieve any sharing yet but it seems like a good design regardless, since it may reduce the number of page faults taken by programs. If you look at the refcount of MM.shared_zero_page() it will have quite a high refcount, but that's just because everything maps it everywhere. If you want to see the "real" refcount, you can build with the MAP_SHARED_ZERO_PAGE_LAZILY flag, and we'll defer mapping of the shared zero page until the first NP read fault. I've left this behavior behind a flag for future testing of this code.
2020-02-15 13:12:02 +01:00
#include <Kernel/VM/MemoryManager.h>
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
#include <Kernel/VM/PhysicalPage.h>
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
//#define COMMIT_DEBUG
//#define PAGE_FAULT_DEBUG
Kernel: Move all code into the Kernel namespace
2020-02-16 01:27:42 +01:00
namespace Kernel {
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
RefPtr<VMObject> AnonymousVMObject::clone()
{
// We need to acquire our lock so we copy a sane state
ScopedSpinLock lock(m_lock);
// We're the parent. Since we're about to become COW we need to
// commit the number of pages that we need to potentially allocate
// so that the parent is still guaranteed to be able to have all
// non-volatile memory available.
size_t need_cow_pages = 0;
{
// We definitely need to commit non-volatile areas
for_each_nonvolatile_range([&](const VolatilePageRange& nonvolatile_range) {
need_cow_pages += nonvolatile_range.count;
return IterationDecision::Continue;
});
}
#ifdef COMMIT_DEBUG
klog() << "Cloning " << this << ", need " << need_cow_pages << " committed cow pages";
#endif
if (!MM.commit_user_physical_pages(need_cow_pages))
return {};
// Create or replace the committed cow pages. When cloning a previously
// cloned vmobject, we want to essentially "fork", leaving us and the
// new clone with one set of shared committed cow pages, and the original
// one would keep the one it still has. This ensures that the original
// one and this one, as well as the clone have sufficient resources
// to cow all pages as needed
m_shared_committed_cow_pages = adopt(*new CommittedCowPages(need_cow_pages));
// Both original and clone become COW. So create a COW map for ourselves
// or reset all pages to be copied again if we were previously cloned
ensure_or_reset_cow_map();
return adopt(*new AnonymousVMObject(*this));
}
RefPtr<AnonymousVMObject> AnonymousVMObject::create_with_size(size_t size, AllocationStrategy commit)
{
if (commit == AllocationStrategy::Reserve || commit == AllocationStrategy::AllocateNow) {
// We need to attempt to commit before actually creating the object
if (!MM.commit_user_physical_pages(ceil_div(size, PAGE_SIZE)))
return {};
}
return adopt(*new AnonymousVMObject(size, commit));
}
NonnullRefPtr<AnonymousVMObject> AnonymousVMObject::create_with_physical_page(PhysicalPage& page)
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
{
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
return adopt(*new AnonymousVMObject(page));
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
}
Kernel: AnonymousVMObject::create_for_physical_range() should fail more Previously it was not possible for this function to fail. You could exploit this by triggering the creation of a VMObject whose physical memory range would wrap around the 32-bit limit. It was quite easy to map kernel memory into userspace and read/write whatever you wanted in it. Test: Kernel/bxvga-mmap-kernel-into-userspace.cpp
2020-01-28 20:48:07 +01:00
RefPtr<AnonymousVMObject> AnonymousVMObject::create_for_physical_range(PhysicalAddress paddr, size_t size)
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
{
Kernel: AnonymousVMObject::create_for_physical_range() should fail more Previously it was not possible for this function to fail. You could exploit this by triggering the creation of a VMObject whose physical memory range would wrap around the 32-bit limit. It was quite easy to map kernel memory into userspace and read/write whatever you wanted in it. Test: Kernel/bxvga-mmap-kernel-into-userspace.cpp
2020-01-28 20:48:07 +01:00
if (paddr.offset(size) < paddr) {
dbg() << "Shenanigans! create_for_physical_range(" << paddr << ", " << size << ") would wrap around";
return nullptr;
}
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
return adopt(*new AnonymousVMObject(paddr, size));
}
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
AnonymousVMObject::AnonymousVMObject(size_t size, AllocationStrategy strategy)
Kernel: Use a FixedArray for VMObject::m_physical_pages This makes VMObject 8 bytes smaller since we can use the array size as the page count. The size() is now also computed from the page count instead of being a separate value. This makes sizes always be a multiple of PAGE_SIZE, which is sane.
2019-08-07 20:12:50 +02:00
: VMObject(size)
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
, m_volatile_ranges_cache({ 0, page_count() })
, m_unused_committed_pages(strategy == AllocationStrategy::Reserve ? page_count() : 0)
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
{
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (strategy == AllocationStrategy::AllocateNow) {
// Allocate all pages right now. We know we can get all because we committed the amount needed
Kernel: Implement lazy committed page allocation By designating a committed page pool we can guarantee to have physical pages available for lazy allocation in mappings. However, when forking we will overcommit. The assumption is that worst-case it's better for the fork to die due to insufficient physical memory on COW access than the parent that created the region. If a fork wants to ensure that all memory is available (trigger a commit) then it can use madvise. This also means that fork now can gracefully fail if we don't have enough physical pages available.
2020-09-04 21:12:25 -06:00
for (size_t i = 0; i < page_count(); ++i)
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
physical_pages()[i] = MM.allocate_committed_user_physical_page(MemoryManager::ShouldZeroFill::Yes);
} else {
auto& initial_page = (strategy == AllocationStrategy::Reserve) ? MM.lazy_committed_page() : MM.shared_zero_page();
for (size_t i = 0; i < page_count(); ++i)
physical_pages()[i] = initial_page;
Kernel: Implement lazy committed page allocation By designating a committed page pool we can guarantee to have physical pages available for lazy allocation in mappings. However, when forking we will overcommit. The assumption is that worst-case it's better for the fork to die due to insufficient physical memory on COW access than the parent that created the region. If a fork wants to ensure that all memory is available (trigger a commit) then it can use madvise. This also means that fork now can gracefully fail if we don't have enough physical pages available.
2020-09-04 21:12:25 -06:00
}
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
}
AnonymousVMObject::AnonymousVMObject(PhysicalAddress paddr, size_t size)
Kernel: Use a FixedArray for VMObject::m_physical_pages This makes VMObject 8 bytes smaller since we can use the array size as the page count. The size() is now also computed from the page count instead of being a separate value. This makes sizes always be a multiple of PAGE_SIZE, which is sane.
2019-08-07 20:12:50 +02:00
: VMObject(size)
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
, m_volatile_ranges_cache({ 0, page_count() })
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
{
Kernel: Remove some unnecessary casts to uintptr_t VirtualAddress is constructible from uintptr_t and const void*. PhysicalAddress is constructible from uintptr_t but not const void*.
2020-01-20 13:06:41 +01:00
ASSERT(paddr.page_base() == paddr);
Kernel: Use a FixedArray for VMObject::m_physical_pages This makes VMObject 8 bytes smaller since we can use the array size as the page count. The size() is now also computed from the page count instead of being a separate value. This makes sizes always be a multiple of PAGE_SIZE, which is sane.
2019-08-07 20:12:50 +02:00
for (size_t i = 0; i < page_count(); ++i)
physical_pages()[i] = PhysicalPage::create(paddr.offset(i * PAGE_SIZE), false, false);
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
}
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
AnonymousVMObject::AnonymousVMObject(PhysicalPage& page)
: VMObject(PAGE_SIZE)
, m_volatile_ranges_cache({ 0, page_count() })
{
physical_pages()[0] = page;
}
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
AnonymousVMObject::AnonymousVMObject(const AnonymousVMObject& other)
: VMObject(other)
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
, m_volatile_ranges_cache({ 0, page_count() }) // do *not* clone this
, m_volatile_ranges_cache_dirty(true) // do *not* clone this
, m_purgeable_ranges() // do *not* clone this
, m_unused_committed_pages(other.m_unused_committed_pages)
, m_cow_map() // do *not* clone this
, m_shared_committed_cow_pages(other.m_shared_committed_cow_pages) // share the pool
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
{
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
// We can't really "copy" a spinlock. But we're holding it. Clear in the clone
ASSERT(other.m_lock.is_locked());
m_lock.initialize();
// The clone also becomes COW
ensure_or_reset_cow_map();
if (m_unused_committed_pages > 0) {
// The original vmobject didn't use up all commited pages. When
// cloning (fork) we will overcommit. For this purpose we drop all
// lazy-commit references and replace them with shared zero pages.
for (size_t i = 0; i < page_count(); i++) {
auto& phys_page = m_physical_pages[i];
if (phys_page && phys_page->is_lazy_committed_page()) {
phys_page = MM.shared_zero_page();
if (--m_unused_committed_pages == 0)
break;
}
}
ASSERT(m_unused_committed_pages == 0);
}
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
}
AnonymousVMObject::~AnonymousVMObject()
{
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
// Return any unused committed pages
if (m_unused_committed_pages > 0)
MM.uncommit_user_physical_pages(m_unused_committed_pages);
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
}
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
int AnonymousVMObject::purge()
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
{
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
LOCKER(m_paging_lock);
return purge_impl();
}
int AnonymousVMObject::purge_with_interrupts_disabled(Badge<MemoryManager>)
{
ASSERT_INTERRUPTS_DISABLED();
if (m_paging_lock.is_locked())
return 0;
return purge_impl();
}
void AnonymousVMObject::set_was_purged(const VolatilePageRange& range)
{
ASSERT(m_lock.is_locked());
for (auto* purgeable_ranges : m_purgeable_ranges)
purgeable_ranges->set_was_purged(range);
}
int AnonymousVMObject::purge_impl()
{
int purged_page_count = 0;
ScopedSpinLock lock(m_lock);
for_each_volatile_range([&](const auto& range) {
int purged_in_range = 0;
auto range_end = range.base + range.count;
for (size_t i = range.base; i < range_end; i++) {
auto& phys_page = m_physical_pages[i];
if (phys_page && !phys_page->is_shared_zero_page()) {
ASSERT(!phys_page->is_lazy_committed_page());
++purged_in_range;
}
phys_page = MM.shared_zero_page();
}
if (purged_in_range > 0) {
purged_page_count += purged_in_range;
set_was_purged(range);
for_each_region([&](auto& region) {
if (&region.vmobject() == this) {
if (auto owner = region.get_owner()) {
// we need to hold a reference the process here (if there is one) as we may not own this region
klog() << "Purged " << purged_in_range << " pages from region " << region.name() << " owned by " << *owner << " at " << region.vaddr_from_page_index(range.base) << " - " << region.vaddr_from_page_index(range.base + range.count);
} else {
klog() << "Purged " << purged_in_range << " pages from region " << region.name() << " (no ownership) at " << region.vaddr_from_page_index(range.base) << " - " << region.vaddr_from_page_index(range.base + range.count);
}
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
region.remap_vmobject_page_range(range.base, range.count);
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
}
});
}
return IterationDecision::Continue;
});
return purged_page_count;
}
void AnonymousVMObject::register_purgeable_page_ranges(PurgeablePageRanges& purgeable_page_ranges)
{
ScopedSpinLock lock(m_lock);
purgeable_page_ranges.set_vmobject(this);
ASSERT(!m_purgeable_ranges.contains_slow(&purgeable_page_ranges));
m_purgeable_ranges.append(&purgeable_page_ranges);
}
void AnonymousVMObject::unregister_purgeable_page_ranges(PurgeablePageRanges& purgeable_page_ranges)
{
ScopedSpinLock lock(m_lock);
for (size_t i = 0; i < m_purgeable_ranges.size(); i++) {
if (m_purgeable_ranges[i] != &purgeable_page_ranges)
continue;
purgeable_page_ranges.set_vmobject(nullptr);
m_purgeable_ranges.remove(i);
return;
}
ASSERT_NOT_REACHED();
}
bool AnonymousVMObject::is_any_volatile() const
{
ScopedSpinLock lock(m_lock);
for (auto& volatile_ranges : m_purgeable_ranges) {
ScopedSpinLock lock(volatile_ranges->m_volatile_ranges_lock);
if (!volatile_ranges->is_empty())
return true;
}
return false;
}
size_t AnonymousVMObject::remove_lazy_commit_pages(const VolatilePageRange& range)
{
ASSERT(m_lock.is_locked());
size_t removed_count = 0;
auto range_end = range.base + range.count;
for (size_t i = range.base; i < range_end; i++) {
auto& phys_page = m_physical_pages[i];
if (phys_page && phys_page->is_lazy_committed_page()) {
phys_page = MM.shared_zero_page();
removed_count++;
ASSERT(m_unused_committed_pages > 0);
if (--m_unused_committed_pages == 0)
break;
}
}
return removed_count;
}
void AnonymousVMObject::update_volatile_cache()
{
ASSERT(m_lock.is_locked());
ASSERT(m_volatile_ranges_cache_dirty);
m_volatile_ranges_cache.clear();
for_each_nonvolatile_range([&](const VolatilePageRange& range) {
m_volatile_ranges_cache.add_unchecked(range);
return IterationDecision::Continue;
});
m_volatile_ranges_cache_dirty = false;
}
void AnonymousVMObject::range_made_volatile(const VolatilePageRange& range)
{
ASSERT(m_lock.is_locked());
if (m_unused_committed_pages == 0)
return;
// We need to check this range for any pages that are marked for
// lazy committed allocation and turn them into shared zero pages
// and also adjust the m_unused_committed_pages for each such page.
// Take into account all the other views as well.
size_t uncommit_page_count = 0;
for_each_volatile_range([&](const auto& r) {
auto intersected = range.intersected(r);
if (!intersected.is_empty()) {
uncommit_page_count += remove_lazy_commit_pages(intersected);
if (m_unused_committed_pages == 0)
return IterationDecision::Break;
}
return IterationDecision::Continue;
});
// Return those committed pages back to the system
if (uncommit_page_count > 0) {
#ifdef COMMIT_DEBUG
klog() << "Uncommit " << uncommit_page_count << " lazy-commit pages from " << this;
#endif
MM.uncommit_user_physical_pages(uncommit_page_count);
}
m_volatile_ranges_cache_dirty = true;
}
void AnonymousVMObject::range_made_nonvolatile(const VolatilePageRange&)
{
ASSERT(m_lock.is_locked());
m_volatile_ranges_cache_dirty = true;
}
size_t AnonymousVMObject::count_needed_commit_pages_for_nonvolatile_range(const VolatilePageRange& range)
{
ASSERT(m_lock.is_locked());
ASSERT(!range.is_empty());
size_t need_commit_pages = 0;
auto range_end = range.base + range.count;
for (size_t page_index = range.base; page_index < range_end; page_index++) {
// COW pages are accounted for in m_shared_committed_cow_pages
if (m_cow_map && m_cow_map->get(page_index))
continue;
auto& phys_page = m_physical_pages[page_index];
if (phys_page && phys_page->is_shared_zero_page())
need_commit_pages++;
}
return need_commit_pages;
}
size_t AnonymousVMObject::mark_committed_pages_for_nonvolatile_range(const VolatilePageRange& range, size_t mark_total)
{
ASSERT(m_lock.is_locked());
ASSERT(!range.is_empty());
ASSERT(mark_total > 0);
size_t pages_updated = 0;
auto range_end = range.base + range.count;
for (size_t page_index = range.base; page_index < range_end; page_index++) {
// COW pages are accounted for in m_shared_committed_cow_pages
if (m_cow_map && m_cow_map->get(page_index))
continue;
auto& phys_page = m_physical_pages[page_index];
if (phys_page && phys_page->is_shared_zero_page()) {
phys_page = MM.lazy_committed_page();
if (++pages_updated == mark_total)
break;
}
}
#ifdef COMMIT_DEBUG
klog() << "Added " << pages_updated << " lazy-commit pages to " << this;
#endif
m_unused_committed_pages += pages_updated;
return pages_updated;
}
RefPtr<PhysicalPage> AnonymousVMObject::allocate_committed_page(size_t page_index)
{
{
ScopedSpinLock lock(m_lock);
ASSERT(m_unused_committed_pages > 0);
// We should't have any committed page tags in volatile regions
ASSERT([&]() {
for (auto* purgeable_ranges : m_purgeable_ranges) {
if (purgeable_ranges->is_volatile(page_index))
return false;
}
return true;
}());
m_unused_committed_pages--;
}
return MM.allocate_committed_user_physical_page(MemoryManager::ShouldZeroFill::Yes);
}
Bitmap& AnonymousVMObject::ensure_cow_map()
{
if (!m_cow_map)
m_cow_map = make<Bitmap>(page_count(), true);
return *m_cow_map;
}
void AnonymousVMObject::ensure_or_reset_cow_map()
{
if (!m_cow_map)
m_cow_map = make<Bitmap>(page_count(), true);
else
m_cow_map->fill(true);
}
bool AnonymousVMObject::should_cow(size_t page_index, bool is_shared) const
{
auto& page = physical_pages()[page_index];
if (page && (page->is_shared_zero_page() || page->is_lazy_committed_page()))
return true;
if (is_shared)
return false;
return m_cow_map && m_cow_map->get(page_index);
}
void AnonymousVMObject::set_should_cow(size_t page_index, bool cow)
{
ensure_cow_map().set(page_index, cow);
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
}
Kernel: Move all code into the Kernel namespace
2020-02-16 01:27:42 +01:00
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
size_t AnonymousVMObject::cow_pages() const
Kernel: Implement lazy committed page allocation By designating a committed page pool we can guarantee to have physical pages available for lazy allocation in mappings. However, when forking we will overcommit. The assumption is that worst-case it's better for the fork to die due to insufficient physical memory on COW access than the parent that created the region. If a fork wants to ensure that all memory is available (trigger a commit) then it can use madvise. This also means that fork now can gracefully fail if we don't have enough physical pages available.
2020-09-04 21:12:25 -06:00
{
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (!m_cow_map)
return 0;
return m_cow_map->count_slow(true);
}
bool AnonymousVMObject::is_nonvolatile(size_t page_index)
{
if (m_volatile_ranges_cache_dirty)
update_volatile_cache();
return !m_volatile_ranges_cache.contains(page_index);
}
PageFaultResponse AnonymousVMObject::handle_cow_fault(size_t page_index, VirtualAddress vaddr)
{
ASSERT_INTERRUPTS_DISABLED();
ScopedSpinLock lock(m_lock);
auto& page_slot = physical_pages()[page_index];
bool have_committed = m_shared_committed_cow_pages && is_nonvolatile(page_index);
if (page_slot->ref_count() == 1) {
#ifdef PAGE_FAULT_DEBUG
dbg() << " >> It's a COW page but nobody is sharing it anymore. Remap r/w";
#endif
set_should_cow(page_index, false);
if (have_committed) {
if (m_shared_committed_cow_pages->return_one())
m_shared_committed_cow_pages = nullptr;
}
return PageFaultResponse::Continue;
}
RefPtr<PhysicalPage> page;
if (have_committed) {
#ifdef PAGE_FAULT_DEBUG
dbg() << " >> It's a committed COW page and it's time to COW!";
#endif
page = m_shared_committed_cow_pages->allocate_one();
} else {
#ifdef PAGE_FAULT_DEBUG
dbg() << " >> It's a COW page and it's time to COW!";
#endif
page = MM.allocate_user_physical_page(MemoryManager::ShouldZeroFill::No);
if (page.is_null()) {
klog() << "MM: handle_cow_fault was unable to allocate a physical page";
return PageFaultResponse::OutOfMemory;
}
}
u8* dest_ptr = MM.quickmap_page(*page);
#ifdef PAGE_FAULT_DEBUG
dbg() << " >> COW " << page->paddr() << " <- " << page_slot->paddr();
#endif
{
SmapDisabler disabler;
void* fault_at;
if (!safe_memcpy(dest_ptr, vaddr.as_ptr(), PAGE_SIZE, fault_at)) {
if ((u8*)fault_at >= dest_ptr && (u8*)fault_at <= dest_ptr + PAGE_SIZE)
dbg() << " >> COW: error copying page " << page_slot->paddr() << "/" << vaddr << " to " << page->paddr() << "/" << VirtualAddress(dest_ptr) << ": failed to write to page at " << VirtualAddress(fault_at);
else if ((u8*)fault_at >= vaddr.as_ptr() && (u8*)fault_at <= vaddr.as_ptr() + PAGE_SIZE)
dbg() << " >> COW: error copying page " << page_slot->paddr() << "/" << vaddr << " to " << page->paddr() << "/" << VirtualAddress(dest_ptr) << ": failed to read from page at " << VirtualAddress(fault_at);
else
ASSERT_NOT_REACHED();
}
}
page_slot = move(page);
MM.unquickmap_page();
set_should_cow(page_index, false);
return PageFaultResponse::Continue;
Kernel: Implement lazy committed page allocation By designating a committed page pool we can guarantee to have physical pages available for lazy allocation in mappings. However, when forking we will overcommit. The assumption is that worst-case it's better for the fork to die due to insufficient physical memory on COW access than the parent that created the region. If a fork wants to ensure that all memory is available (trigger a commit) then it can use madvise. This also means that fork now can gracefully fail if we don't have enough physical pages available.
2020-09-04 21:12:25 -06:00
}
Kernel: Move all code into the Kernel namespace
2020-02-16 01:27:42 +01:00
}
Reference in a new issue Copy permalink