Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-12-08 06:09:58 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Kernel/VM/Region.cpp

628 lines
23 KiB
C++
Raw Normal View History

Meta: Add license header to source files As suggested by Joshua, this commit adds the 2-clause BSD license as a comment block to the top of every source file. For the first pass, I've just added myself for simplicity. I encourage everyone to add themselves as copyright holders of any file they've added or modified in some significant way. If I've added myself in error somewhere, feel free to replace it with the appropriate copyright holder instead. Going forward, all new source files should include a license header.
2020-01-18 09:38:21 +01:00
/*
* Copyright (c) 2018-2020, Andreas Kling <kling@serenityos.org>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice, this
* list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
* SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
Kernel: Add missing #includes now that <AK/StdLibExtras.h> is smaller
2020-03-08 12:33:14 +01:00
#include <AK/Memory.h>
AK: Reduce header dependency graph of String.h String.h no longer pulls in StringView.h. We do this by moving a bunch of String functions out-of-line.
2020-03-23 13:45:10 +01:00
#include <AK/StringView.h>
Kernel: Track user accessibility per Region. Region now has is_user_accessible(), which informs the memory manager how to map these pages. Previously, we were just passing a "bool user_allowed" to various functions and I'm not at all sure that any of that was correct. All the Region constructors are now hidden, and you must go through one of these helpers to construct a region: - Region::create_user_accessible(...) - Region::create_kernel_only(...) That ensures that we don't accidentally create a Region without specifying user accessibility. :^)
2019-07-19 16:09:34 +02:00
#include <Kernel/FileSystem/Inode.h>
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
#include <Kernel/Process.h>
#include <Kernel/Thread.h>
Kernel: Split VMObject into two classes: Anonymous- and InodeVMObject InodeVMObject is a VMObject with an underlying Inode in the filesystem. AnonymousVMObject has no Inode. I'm happy that InodeVMObject::inode() can now return Inode& instead of VMObject::inode() return Inode*. :^)
2019-08-07 18:06:17 +02:00
#include <Kernel/VM/AnonymousVMObject.h>
Kernel: Run clang-format on everything.
2019-06-07 11:43:58 +02:00
#include <Kernel/VM/MemoryManager.h>
Kernel: Reduce header dependencies of MemoryManager and Region
2020-02-16 01:33:41 +01:00
#include <Kernel/VM/PageDirectory.h>
Kernel: Run clang-format on everything.
2019-06-07 11:43:58 +02:00
#include <Kernel/VM/Region.h>
Kernel: Simplify some dbg() logging We don't have to log the process name/PID/TID, dbg() automatically adds that as a prefix to every line. Also we don't have to do .characters() on Strings passed to dbg() :^)
2020-02-29 12:51:44 +01:00
#include <Kernel/VM/SharedInodeVMObject.h>
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
Kernel: Merge MemoryManager::map_region_at_address() into Region::map()
2019-11-04 00:05:57 +01:00
//#define MM_DEBUG
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
//#define PAGE_FAULT_DEBUG
Kernel: Merge MemoryManager::map_region_at_address() into Region::map()
2019-11-04 00:05:57 +01:00
Kernel: Move all code into the Kernel namespace
2020-02-16 01:27:42 +01:00
namespace Kernel {
Kernel: Pass "shared" flag to Region constructor Before this change, we would sometimes map a region into the address space with !is_shared(), and then moments later call set_shared(true). I found this very confusing while debugging, so this patch makes us pass the initial shared flag to the Region constructor, ensuring that it's in the correct state by the time we first map the region.
2021-01-02 16:38:05 +01:00
Region::Region(const Range& range, NonnullRefPtr<VMObject> vmobject, size_t offset_in_vmobject, const String& name, u8 access, bool cacheable, bool kernel, bool shared)
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
: PurgeablePageRanges(vmobject)
, m_range(range)
Kernel: Rename vmo => vmobject everywhere
2019-12-19 19:13:44 +01:00
, m_offset_in_vmobject(offset_in_vmobject)
, m_vmobject(move(vmobject))
Kernel: Tweak some String&& => const String&. String&& is just not very practical. Also return const String& when the returned string is a member variable. The call site is free to make a copy if he wants, but otherwise we can avoid the retain count churn.
2019-06-07 20:58:12 +02:00
, m_name(name)
Kernel: Make the Process allocate_region* API's understand "int prot". Instead of having to inspect 'prot' at every call site, make the Process API's take care of that so we can just pass it through.
2019-05-30 16:14:37 +02:00
, m_access(access)
Kernel: Pass "shared" flag to Region constructor Before this change, we would sometimes map a region into the address space with !is_shared(), and then moments later call set_shared(true). I found this very confusing while debugging, so this patch makes us pass the initial shared flag to the Region constructor, ensuring that it's in the correct state by the time we first map the region.
2021-01-02 16:38:05 +01:00
, m_shared(shared)
Kernel: Change Region allocation helpers We now can create a cacheable Region, so when map() is called, if a Region is cacheable then all the virtual memory space being allocated to it will be marked as not cache disabled. In addition to that, OS components can create a Region that will be mapped to a specific physical address by using the appropriate helper method.
2020-01-09 23:29:31 +02:00
, m_cacheable(cacheable)
Kernel: Add mechanism to identity map the lowest 2MB
2020-06-01 22:55:09 -06:00
, m_kernel(kernel)
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
{
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
m_vmobject->ref_region();
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
register_purgeable_page_ranges();
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
MM.register_region(*this);
}
Region::~Region()
{
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
m_vmobject->unref_region();
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
unregister_purgeable_page_ranges();
Kernel: Disable interrupts throughout ~Region() We don't want an interrupt handler to access the VM data structures while their internal consistency is broken.
2019-09-05 11:13:10 +02:00
// Make sure we disable interrupts so we don't get interrupted between unmapping and unregistering.
// Unmapping the region will give the VM back to the RangeAllocator, so an interrupt handler would
// find the address<->region mappings in an invalid state there.
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
ScopedSpinLock lock(s_mm_lock);
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
if (m_page_directory) {
Kernel: Move region map/unmap operations into the Region class The more Region can take care of itself, the better.
2019-11-03 20:37:03 +01:00
unmap(ShouldDeallocateVirtualMemoryRange::Yes);
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
ASSERT(!m_page_directory);
}
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
MM.unregister_region(*this);
}
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
void Region::register_purgeable_page_ranges()
{
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (m_vmobject->is_anonymous()) {
auto& vmobject = static_cast<AnonymousVMObject&>(*m_vmobject);
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
vmobject.register_purgeable_page_ranges(*this);
}
}
void Region::unregister_purgeable_page_ranges()
{
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (m_vmobject->is_anonymous()) {
auto& vmobject = static_cast<AnonymousVMObject&>(*m_vmobject);
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
vmobject.unregister_purgeable_page_ranges(*this);
}
}
Kernel: Pass new region owner to Region::clone
2021-01-01 00:02:40 -07:00
OwnPtr<Region> Region::clone(Process& new_owner)
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
{
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
ASSERT(Process::current());
Kernel: Track user accessibility per Region. Region now has is_user_accessible(), which informs the memory manager how to map these pages. Previously, we were just passing a "bool user_allowed" to various functions and I'm not at all sure that any of that was correct. All the Region constructors are now hidden, and you must go through one of these helpers to construct a region: - Region::create_user_accessible(...) - Region::create_kernel_only(...) That ensures that we don't accidentally create a Region without specifying user accessibility. :^)
2019-07-19 16:09:34 +02:00
Kernel: Aggregate TLB flush requests for Regions for SMP Rather than sending one TLB flush request for each page, aggregate them so that we're not spamming the other processors with FlushTLB IPIs.
2020-07-06 12:47:08 -06:00
ScopedSpinLock lock(s_mm_lock);
Kernel+LibC: Add minherit() and MAP_INHERIT_ZERO This patch adds the minherit() syscall originally invented by OpenBSD. Only the MAP_INHERIT_ZERO mode is supported for now. If set on an mmap region, that region will be zeroed out on fork().
2020-04-12 20:22:26 +02:00
if (m_inherit_mode == InheritMode::ZeroedOnFork) {
ASSERT(m_mmap);
ASSERT(!m_shared);
ASSERT(vmobject().is_anonymous());
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
auto new_vmobject = AnonymousVMObject::create_with_size(size(), AllocationStrategy::Reserve); // TODO: inherit committed non-volatile areas?
if (!new_vmobject)
return {};
Kernel: Pass new region owner to Region::clone
2021-01-01 00:02:40 -07:00
auto zeroed_region = Region::create_user_accessible(&new_owner, m_range, new_vmobject.release_nonnull(), 0, m_name, m_access);
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
zeroed_region->copy_purgeable_page_ranges(*this);
Kernel+LibC: Add minherit() and MAP_INHERIT_ZERO This patch adds the minherit() syscall originally invented by OpenBSD. Only the MAP_INHERIT_ZERO mode is supported for now. If set on an mmap region, that region will be zeroed out on fork().
2020-04-12 20:22:26 +02:00
zeroed_region->set_mmap(m_mmap);
zeroed_region->set_inherit_mode(m_inherit_mode);
return zeroed_region;
}
Kernel: CoW-clone private inode-backed memory regions on fork() When forking a process, we now turn all of the private inode-backed mmap() regions into copy-on-write regions in both the parent and child. This patch also removes an assertion that becomes irrelevant.
2020-03-01 10:54:18 +01:00
if (m_shared) {
Kernel: Implement some basic stack pointer validation VM regions can now be marked as stack regions, which is then validated on syscall, and on page fault. If a thread is caught with its stack pointer pointing into anything that's *not* a Region with its stack bit set, we'll crash the whole process with SIGSTKFLT. Userspace must now allocate custom stacks by using mmap() with the new MAP_STACK flag. This mechanism was first introduced in OpenBSD, and now we have it too, yay! :^)
2019-11-17 12:11:43 +01:00
ASSERT(!m_stack);
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
#ifdef MM_DEBUG
Kernel: Simplify some dbg() logging We don't have to log the process name/PID/TID, dbg() automatically adds that as a prefix to every line. Also we don't have to do .characters() on Strings passed to dbg() :^)
2020-02-29 12:51:44 +01:00
dbg() << "Region::clone(): Sharing " << name() << " (" << vaddr() << ")";
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
#endif
Kernel: Add some InodeVMObject type assertions in Region::clone() Let's make sure that we're never cloning shared inode-backed objects as if they were private, and vice versa.
2020-03-01 11:08:28 +01:00
if (vmobject().is_inode())
ASSERT(vmobject().is_shared_inode());
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
// Create a new region backed by the same VMObject.
Kernel: Pass new region owner to Region::clone
2021-01-01 00:02:40 -07:00
auto region = Region::create_user_accessible(&new_owner, m_range, m_vmobject, m_offset_in_vmobject, m_name, m_access);
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (m_vmobject->is_anonymous())
region->copy_purgeable_page_ranges(*this);
Kernel: Copy Region's "is_mmap" flag when cloning regions for fork() Otherwise child processes will not be allowed to munmap(), madvise(), etc. on the cloned regions!
2020-01-10 19:24:01 +01:00
region->set_mmap(m_mmap);
Kernel: Cloned shared regions should also be marked as shared
2020-02-08 02:39:46 +01:00
region->set_shared(m_shared);
Kernel: Copy Region's "is_mmap" flag when cloning regions for fork() Otherwise child processes will not be allowed to munmap(), madvise(), etc. on the cloned regions!
2020-01-10 19:24:01 +01:00
return region;
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
}
Kernel: Add some InodeVMObject type assertions in Region::clone() Let's make sure that we're never cloning shared inode-backed objects as if they were private, and vice versa.
2020-03-01 11:08:28 +01:00
if (vmobject().is_inode())
ASSERT(vmobject().is_private_inode());
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
auto vmobject_clone = vmobject().clone();
Kernel: Implement lazy committed page allocation By designating a committed page pool we can guarantee to have physical pages available for lazy allocation in mappings. However, when forking we will overcommit. The assumption is that worst-case it's better for the fork to die due to insufficient physical memory on COW access than the parent that created the region. If a fork wants to ensure that all memory is available (trigger a commit) then it can use madvise. This also means that fork now can gracefully fail if we don't have enough physical pages available.
2020-09-04 21:12:25 -06:00
if (!vmobject_clone)
return {};
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
#ifdef MM_DEBUG
Kernel: Simplify some dbg() logging We don't have to log the process name/PID/TID, dbg() automatically adds that as a prefix to every line. Also we don't have to do .characters() on Strings passed to dbg() :^)
2020-02-29 12:51:44 +01:00
dbg() << "Region::clone(): CoWing " << name() << " (" << vaddr() << ")";
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
#endif
Kernel: Simplify some dbg() logging We don't have to log the process name/PID/TID, dbg() automatically adds that as a prefix to every line. Also we don't have to do .characters() on Strings passed to dbg() :^)
2020-02-29 12:51:44 +01:00
// Set up a COW region. The parent (this) region becomes COW as well!
Kernel: Teach Region how to remap itself Now remapping (i.e flushing kernel metadata to the CPU page tables) is done by simply calling Region::remap().
2019-11-03 20:59:54 +01:00
remap();
Kernel: Pass new region owner to Region::clone
2021-01-01 00:02:40 -07:00
auto clone_region = Region::create_user_accessible(&new_owner, m_range, vmobject_clone.release_nonnull(), m_offset_in_vmobject, m_name, m_access);
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (m_vmobject->is_anonymous())
clone_region->copy_purgeable_page_ranges(*this);
Kernel: Implement some basic stack pointer validation VM regions can now be marked as stack regions, which is then validated on syscall, and on page fault. If a thread is caught with its stack pointer pointing into anything that's *not* a Region with its stack bit set, we'll crash the whole process with SIGSTKFLT. Userspace must now allocate custom stacks by using mmap() with the new MAP_STACK flag. This mechanism was first introduced in OpenBSD, and now we have it too, yay! :^)
2019-11-17 12:11:43 +01:00
if (m_stack) {
ASSERT(is_readable());
ASSERT(is_writable());
ASSERT(vmobject().is_anonymous());
clone_region->set_stack(true);
}
Kernel: Copy Region's "is_mmap" flag when cloning regions for fork() Otherwise child processes will not be allowed to munmap(), madvise(), etc. on the cloned regions!
2020-01-10 19:24:01 +01:00
clone_region->set_mmap(m_mmap);
Kernel: Defer creation of Region CoW bitmaps until they're needed Instead of allocating and populating a Copy-on-Write bitmap for each Region up front, wait until we actually clone the Region for sharing with another process. In most cases, we never need any CoW bits and we save ourselves a lot of kmalloc() memory and time.
2019-10-01 19:58:41 +02:00
return clone_region;
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
}
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
void Region::set_vmobject(NonnullRefPtr<VMObject>&& obj)
{
if (m_vmobject.ptr() == obj.ptr())
return;
unregister_purgeable_page_ranges();
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
m_vmobject->unref_region();
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
m_vmobject = move(obj);
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
m_vmobject->ref_region();
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
register_purgeable_page_ranges();
}
bool Region::is_volatile(VirtualAddress vaddr, size_t size) const
{
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (!m_vmobject->is_anonymous())
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
return false;
auto offset_in_vmobject = vaddr.get() - (this->vaddr().get() - m_offset_in_vmobject);
size_t first_page_index = PAGE_ROUND_DOWN(offset_in_vmobject) / PAGE_SIZE;
size_t last_page_index = PAGE_ROUND_UP(offset_in_vmobject + size) / PAGE_SIZE;
return is_volatile_range({ first_page_index, last_page_index - first_page_index });
}
auto Region::set_volatile(VirtualAddress vaddr, size_t size, bool is_volatile, bool& was_purged) -> SetVolatileError
{
was_purged = false;
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (!m_vmobject->is_anonymous())
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
return SetVolatileError::NotPurgeable;
auto offset_in_vmobject = vaddr.get() - (this->vaddr().get() - m_offset_in_vmobject);
if (is_volatile) {
// If marking pages as volatile, be prudent by not marking
// partial pages volatile to prevent potentially non-volatile
// data to be discarded. So rund up the first page and round
// down the last page.
size_t first_page_index = PAGE_ROUND_UP(offset_in_vmobject) / PAGE_SIZE;
size_t last_page_index = PAGE_ROUND_DOWN(offset_in_vmobject + size) / PAGE_SIZE;
if (first_page_index != last_page_index)
add_volatile_range({ first_page_index, last_page_index - first_page_index });
} else {
// If marking pages as non-volatile, round down the first page
// and round up the last page to make sure the beginning and
// end of the range doesn't inadvertedly get discarded.
size_t first_page_index = PAGE_ROUND_DOWN(offset_in_vmobject) / PAGE_SIZE;
size_t last_page_index = PAGE_ROUND_UP(offset_in_vmobject + size) / PAGE_SIZE;
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
switch (remove_volatile_range({ first_page_index, last_page_index - first_page_index }, was_purged)) {
case PurgeablePageRanges::RemoveVolatileError::Success:
case PurgeablePageRanges::RemoveVolatileError::SuccessNoChange:
break;
case PurgeablePageRanges::RemoveVolatileError::OutOfMemory:
return SetVolatileError::OutOfMemory;
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
}
}
return SetVolatileError::Success;
}
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
size_t Region::cow_pages() const
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
{
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (!vmobject().is_anonymous())
Kernel+SystemMonitor: Expose the number of set CoW bits in each Region This number tells us how many more pages in a given region will trigger a CoW fault if written to.
2019-12-15 16:53:00 +01:00
return 0;
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
return static_cast<const AnonymousVMObject&>(vmobject()).cow_pages();
Kernel+SystemMonitor: Expose the number of set CoW bits in each Region This number tells us how many more pages in a given region will trigger a CoW fault if written to.
2019-12-15 16:53:00 +01:00
}
Kernel+SystemMonitor: Expose amount of per-process dirty private memory Dirty private memory is all memory in non-inode-backed mappings that's process-private, meaning it's not shared with any other process. This patch exposes that number via SystemMonitor, giving us an idea of how much memory each process is responsible for all on its own.
2019-12-29 12:28:32 +01:00
size_t Region::amount_dirty() const
{
if (!vmobject().is_inode())
return amount_resident();
Kernel: Remove some more harmless InodeVMObject miscasts
2020-03-01 12:27:03 +01:00
return static_cast<const InodeVMObject&>(vmobject()).amount_dirty();
Kernel+SystemMonitor: Expose amount of per-process dirty private memory Dirty private memory is all memory in non-inode-backed mappings that's process-private, meaning it's not shared with any other process. This patch exposes that number via SystemMonitor, giving us an idea of how much memory each process is responsible for all on its own.
2019-12-29 12:28:32 +01:00
}
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
size_t Region::amount_resident() const
{
size_t bytes = 0;
for (size_t i = 0; i < page_count(); ++i) {
Kernel: Add Region helpers for accessing underlying physical pages Since a Region is basically a view into a potentially larger VMObject, it was always necessary to include the Region starting offset when accessing its underlying physical pages. Until now, you had to do that manually, but this patch adds a simple Region::physical_page() for read-only access and a physical_page_slot() when you want a mutable reference to the RefPtr<PhysicalPage> itself. A lot of code is simplified by making use of this.
2020-04-28 16:19:50 +02:00
auto* page = physical_page(i);
Kernel: Make Region::amount_shared() and amount_resident() lazy-aware Don't count the lazy-committed page towards shared/resident amounts.
2021-01-02 00:47:55 +01:00
if (page && !page->is_shared_zero_page() && !page->is_lazy_committed_page())
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
bytes += PAGE_SIZE;
}
return bytes;
}
size_t Region::amount_shared() const
{
size_t bytes = 0;
for (size_t i = 0; i < page_count(); ++i) {
Kernel: Add Region helpers for accessing underlying physical pages Since a Region is basically a view into a potentially larger VMObject, it was always necessary to include the Region starting offset when accessing its underlying physical pages. Until now, you had to do that manually, but this patch adds a simple Region::physical_page() for read-only access and a physical_page_slot() when you want a mutable reference to the RefPtr<PhysicalPage> itself. A lot of code is simplified by making use of this.
2020-04-28 16:19:50 +02:00
auto* page = physical_page(i);
Kernel: Make Region::amount_shared() and amount_resident() lazy-aware Don't count the lazy-committed page towards shared/resident amounts.
2021-01-02 00:47:55 +01:00
if (page && page->ref_count() > 1 && !page->is_shared_zero_page() && !page->is_lazy_committed_page())
Kernel: Move VM-related files into Kernel/VM/. Also break MemoryManager.{cpp,h} into one file per class.
2019-04-03 15:13:07 +02:00
bytes += PAGE_SIZE;
}
return bytes;
}
Kernel: Track user accessibility per Region. Region now has is_user_accessible(), which informs the memory manager how to map these pages. Previously, we were just passing a "bool user_allowed" to various functions and I'm not at all sure that any of that was correct. All the Region constructors are now hidden, and you must go through one of these helpers to construct a region: - Region::create_user_accessible(...) - Region::create_kernel_only(...) That ensures that we don't accidentally create a Region without specifying user accessibility. :^)
2019-07-19 16:09:34 +02:00
Kernel: Pass "shared" flag to Region constructor Before this change, we would sometimes map a region into the address space with !is_shared(), and then moments later call set_shared(true). I found this very confusing while debugging, so this patch makes us pass the initial shared flag to the Region constructor, ensuring that it's in the correct state by the time we first map the region.
2021-01-02 16:38:05 +01:00
NonnullOwnPtr<Region> Region::create_user_accessible(Process* owner, const Range& range, NonnullRefPtr<VMObject> vmobject, size_t offset_in_vmobject, const StringView& name, u8 access, bool cacheable, bool shared)
Kernel: Track user accessibility per Region. Region now has is_user_accessible(), which informs the memory manager how to map these pages. Previously, we were just passing a "bool user_allowed" to various functions and I'm not at all sure that any of that was correct. All the Region constructors are now hidden, and you must go through one of these helpers to construct a region: - Region::create_user_accessible(...) - Region::create_kernel_only(...) That ensures that we don't accidentally create a Region without specifying user accessibility. :^)
2019-07-19 16:09:34 +02:00
{
Kernel: Pass "shared" flag to Region constructor Before this change, we would sometimes map a region into the address space with !is_shared(), and then moments later call set_shared(true). I found this very confusing while debugging, so this patch makes us pass the initial shared flag to the Region constructor, ensuring that it's in the correct state by the time we first map the region.
2021-01-02 16:38:05 +01:00
auto region = make<Region>(range, move(vmobject), offset_in_vmobject, name, access, cacheable, false, shared);
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
if (owner)
region->m_owner = owner->make_weak_ptr();
Kernel: Track user accessibility per Region. Region now has is_user_accessible(), which informs the memory manager how to map these pages. Previously, we were just passing a "bool user_allowed" to various functions and I'm not at all sure that any of that was correct. All the Region constructors are now hidden, and you must go through one of these helpers to construct a region: - Region::create_user_accessible(...) - Region::create_kernel_only(...) That ensures that we don't accidentally create a Region without specifying user accessibility. :^)
2019-07-19 16:09:34 +02:00
region->m_user_accessible = true;
return region;
}
Kernel: Change Region allocation helpers We now can create a cacheable Region, so when map() is called, if a Region is cacheable then all the virtual memory space being allocated to it will be marked as not cache disabled. In addition to that, OS components can create a Region that will be mapped to a specific physical address by using the appropriate helper method.
2020-01-09 23:29:31 +02:00
NonnullOwnPtr<Region> Region::create_kernel_only(const Range& range, NonnullRefPtr<VMObject> vmobject, size_t offset_in_vmobject, const StringView& name, u8 access, bool cacheable)
Kernel: Track user accessibility per Region. Region now has is_user_accessible(), which informs the memory manager how to map these pages. Previously, we were just passing a "bool user_allowed" to various functions and I'm not at all sure that any of that was correct. All the Region constructors are now hidden, and you must go through one of these helpers to construct a region: - Region::create_user_accessible(...) - Region::create_kernel_only(...) That ensures that we don't accidentally create a Region without specifying user accessibility. :^)
2019-07-19 16:09:34 +02:00
{
Kernel: Pass "shared" flag to Region constructor Before this change, we would sometimes map a region into the address space with !is_shared(), and then moments later call set_shared(true). I found this very confusing while debugging, so this patch makes us pass the initial shared flag to the Region constructor, ensuring that it's in the correct state by the time we first map the region.
2021-01-02 16:38:05 +01:00
auto region = make<Region>(range, move(vmobject), offset_in_vmobject, name, access, cacheable, true, false);
Kernel: Track user accessibility per Region. Region now has is_user_accessible(), which informs the memory manager how to map these pages. Previously, we were just passing a "bool user_allowed" to various functions and I'm not at all sure that any of that was correct. All the Region constructors are now hidden, and you must go through one of these helpers to construct a region: - Region::create_user_accessible(...) - Region::create_kernel_only(...) That ensures that we don't accidentally create a Region without specifying user accessibility. :^)
2019-07-19 16:09:34 +02:00
region->m_user_accessible = false;
Kernel+LibELF: Enable SMAP protection during non-syscall exec() When loading a new executable, we now map the ELF image in kernel-only memory and parse it there. Then we use copy_to_user() when initializing writable regions with data from the executable. Note that the exec() syscall still disables SMAP protection and will require additional work. This patch only affects kernel-originated process spawns.
2020-01-10 06:57:18 +01:00
return region;
}
Kernel: Defer creation of Region CoW bitmaps until they're needed Instead of allocating and populating a Copy-on-Write bitmap for each Region up front, wait until we actually clone the Region for sharing with another process. In most cases, we never need any CoW bits and we save ourselves a lot of kmalloc() memory and time.
2019-10-01 19:58:41 +02:00
bool Region::should_cow(size_t page_index) const
{
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (!vmobject().is_anonymous())
Kernel: Defer creation of Region CoW bitmaps until they're needed Instead of allocating and populating a Copy-on-Write bitmap for each Region up front, wait until we actually clone the Region for sharing with another process. In most cases, we never need any CoW bits and we save ourselves a lot of kmalloc() memory and time.
2019-10-01 19:58:41 +02:00
return false;
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
return static_cast<const AnonymousVMObject&>(vmobject()).should_cow(first_page_index() + page_index, m_shared);
Kernel: Defer creation of Region CoW bitmaps until they're needed Instead of allocating and populating a Copy-on-Write bitmap for each Region up front, wait until we actually clone the Region for sharing with another process. In most cases, we never need any CoW bits and we save ourselves a lot of kmalloc() memory and time.
2019-10-01 19:58:41 +02:00
}
void Region::set_should_cow(size_t page_index, bool cow)
{
ASSERT(!m_shared);
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (vmobject().is_anonymous())
static_cast<AnonymousVMObject&>(vmobject()).set_should_cow(first_page_index() + page_index, cow);
Kernel: Defer creation of Region CoW bitmaps until they're needed Instead of allocating and populating a Copy-on-Write bitmap for each Region up front, wait until we actually clone the Region for sharing with another process. In most cases, we never need any CoW bits and we save ourselves a lot of kmalloc() memory and time.
2019-10-01 19:58:41 +02:00
}
Kernel: Move page remapping into Region::remap_page(index) Let Region deal with this, instead of everyone calling MemoryManager.
2019-11-03 15:32:11 +01:00
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
bool Region::map_individual_page_impl(size_t page_index)
Kernel: Move page remapping into Region::remap_page(index) Let Region deal with this, instead of everyone calling MemoryManager.
2019-11-03 15:32:11 +01:00
{
Kernel: Protect the PageDirectory from concurrent access
2020-10-31 17:19:18 -06:00
ASSERT(m_page_directory->get_lock().own_lock());
Kernel: Aggregate TLB flush requests for Regions for SMP Rather than sending one TLB flush request for each page, aggregate them so that we're not spamming the other processors with FlushTLB IPIs.
2020-07-06 12:47:08 -06:00
auto page_vaddr = vaddr_from_page_index(page_index);
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
auto* pte = MM.ensure_pte(*m_page_directory, page_vaddr);
if (!pte) {
#ifdef MM_DEBUG
dbg() << "MM: >> region map (PD=" << m_page_directory->cr3() << " " << name() << " cannot create PTE for " << page_vaddr;
#endif
return false;
}
Kernel: Add Region helpers for accessing underlying physical pages Since a Region is basically a view into a potentially larger VMObject, it was always necessary to include the Region starting offset when accessing its underlying physical pages. Until now, you had to do that manually, but this patch adds a simple Region::physical_page() for read-only access and a physical_page_slot() when you want a mutable reference to the RefPtr<PhysicalPage> itself. A lot of code is simplified by making use of this.
2020-04-28 16:19:50 +02:00
auto* page = physical_page(page_index);
if (!page || (!is_readable() && !is_writable())) {
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
pte->clear();
Kernel: Share code between Region::map() and Region::remap_page() These were doing mostly the same things, so let's just share the code.
2020-01-01 19:30:38 +01:00
} else {
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
pte->set_cache_disabled(!m_cacheable);
pte->set_physical_page_base(page->paddr().get());
pte->set_present(true);
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (page->is_shared_zero_page() || page->is_lazy_committed_page() || should_cow(page_index))
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
pte->set_writable(false);
Kernel: Share code between Region::map() and Region::remap_page() These were doing mostly the same things, so let's just share the code.
2020-01-01 19:30:38 +01:00
else
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
pte->set_writable(is_writable());
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (Processor::current().has_feature(CPUFeature::NX))
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
pte->set_execute_disabled(!is_executable());
pte->set_user_allowed(is_user_accessible());
Kernel: Move page remapping into Region::remap_page(index) Let Region deal with this, instead of everyone calling MemoryManager.
2019-11-03 15:32:11 +01:00
#ifdef MM_DEBUG
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
dbg() << "MM: >> region map (PD=" << m_page_directory->cr3() << ", PTE=" << (void*)pte->raw() << "{" << pte << "}) " << name() << " " << page_vaddr << " => " << page->paddr() << " (@" << page << ")";
Kernel: Move page remapping into Region::remap_page(index) Let Region deal with this, instead of everyone calling MemoryManager.
2019-11-03 15:32:11 +01:00
#endif
Kernel: Change Region allocation helpers We now can create a cacheable Region, so when map() is called, if a Region is cacheable then all the virtual memory space being allocated to it will be marked as not cache disabled. In addition to that, OS components can create a Region that will be mapped to a specific physical address by using the appropriate helper method.
2020-01-09 23:29:31 +02:00
}
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
return true;
Kernel: Move page remapping into Region::remap_page(index) Let Region deal with this, instead of everyone calling MemoryManager.
2019-11-03 15:32:11 +01:00
}
Kernel: Move region map/unmap operations into the Region class The more Region can take care of itself, the better.
2019-11-03 20:37:03 +01:00
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
bool Region::do_remap_vmobject_page_range(size_t page_index, size_t page_count)
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
{
bool success = true;
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
ASSERT(s_mm_lock.own_lock());
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
ASSERT(m_page_directory);
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
if (!translate_vmobject_page_range(page_index, page_count))
return success; // not an error, region doesn't map this page range
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
ScopedSpinLock page_lock(m_page_directory->get_lock());
size_t index = page_index;
while (index < page_index + page_count) {
if (!map_individual_page_impl(index)) {
success = false;
break;
}
index++;
}
if (index > page_index)
MM.flush_tlb(vaddr_from_page_index(page_index), index - page_index);
return success;
}
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
bool Region::remap_vmobject_page_range(size_t page_index, size_t page_count)
{
bool success = true;
ScopedSpinLock lock(s_mm_lock);
auto& vmobject = this->vmobject();
if (vmobject.is_shared_by_multiple_regions()) {
vmobject.for_each_region([&](auto& region) {
if (!region.do_remap_vmobject_page_range(page_index, page_count))
success = false;
});
} else {
if (!do_remap_vmobject_page_range(page_index, page_count))
success = false;
}
return success;
}
bool Region::do_remap_vmobject_page(size_t page_index, bool with_flush)
Kernel: Share code between Region::map() and Region::remap_page() These were doing mostly the same things, so let's just share the code.
2020-01-01 19:30:38 +01:00
{
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
ScopedSpinLock lock(s_mm_lock);
Kernel: Protect the PageDirectory from concurrent access
2020-10-31 17:19:18 -06:00
ASSERT(m_page_directory);
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
if (!translate_vmobject_page(page_index))
return true; // not an error, region doesn't map this page
Kernel: Protect the PageDirectory from concurrent access
2020-10-31 17:19:18 -06:00
ScopedSpinLock page_lock(m_page_directory->get_lock());
Kernel: Add Region helpers for accessing underlying physical pages Since a Region is basically a view into a potentially larger VMObject, it was always necessary to include the Region starting offset when accessing its underlying physical pages. Until now, you had to do that manually, but this patch adds a simple Region::physical_page() for read-only access and a physical_page_slot() when you want a mutable reference to the RefPtr<PhysicalPage> itself. A lot of code is simplified by making use of this.
2020-04-28 16:19:50 +02:00
ASSERT(physical_page(page_index));
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
bool success = map_individual_page_impl(page_index);
Kernel: Aggregate TLB flush requests for Regions for SMP Rather than sending one TLB flush request for each page, aggregate them so that we're not spamming the other processors with FlushTLB IPIs.
2020-07-06 12:47:08 -06:00
if (with_flush)
MM.flush_tlb(vaddr_from_page_index(page_index));
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
return success;
Kernel: Share code between Region::map() and Region::remap_page() These were doing mostly the same things, so let's just share the code.
2020-01-01 19:30:38 +01:00
}
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
bool Region::remap_vmobject_page(size_t page_index, bool with_flush)
{
bool success = true;
ScopedSpinLock lock(s_mm_lock);
auto& vmobject = this->vmobject();
if (vmobject.is_shared_by_multiple_regions()) {
vmobject.for_each_region([&](auto& region) {
if (!region.do_remap_vmobject_page(page_index, with_flush))
success = false;
});
} else {
if (!do_remap_vmobject_page(page_index, with_flush))
success = false;
}
return success;
}
Kernel: Move region map/unmap operations into the Region class The more Region can take care of itself, the better.
2019-11-03 20:37:03 +01:00
void Region::unmap(ShouldDeallocateVirtualMemoryRange deallocate_range)
{
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
ScopedSpinLock lock(s_mm_lock);
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (!m_page_directory)
return;
Kernel: Protect the PageDirectory from concurrent access
2020-10-31 17:19:18 -06:00
ScopedSpinLock page_lock(m_page_directory->get_lock());
Kernel: Release page tables when no longer needed When unmapping regions, check if page tables can be freed. This is a follow-up change for #3254.
2020-08-27 21:29:17 -06:00
size_t count = page_count();
for (size_t i = 0; i < count; ++i) {
Kernel: Aggregate TLB flush requests for Regions for SMP Rather than sending one TLB flush request for each page, aggregate them so that we're not spamming the other processors with FlushTLB IPIs.
2020-07-06 12:47:08 -06:00
auto vaddr = vaddr_from_page_index(i);
Kernel: Release page tables when no longer needed When unmapping regions, check if page tables can be freed. This is a follow-up change for #3254.
2020-08-27 21:29:17 -06:00
MM.release_pte(*m_page_directory, vaddr, i == count - 1);
Kernel: Move region map/unmap operations into the Region class The more Region can take care of itself, the better.
2019-11-03 20:37:03 +01:00
#ifdef MM_DEBUG
Kernel: Add Region helpers for accessing underlying physical pages Since a Region is basically a view into a potentially larger VMObject, it was always necessary to include the Region starting offset when accessing its underlying physical pages. Until now, you had to do that manually, but this patch adds a simple Region::physical_page() for read-only access and a physical_page_slot() when you want a mutable reference to the RefPtr<PhysicalPage> itself. A lot of code is simplified by making use of this.
2020-04-28 16:19:50 +02:00
auto* page = physical_page(i);
dbg() << "MM: >> Unmapped " << vaddr << " => P" << String::format("%p", page ? page->paddr().get() : 0) << " <<";
Kernel: Move region map/unmap operations into the Region class The more Region can take care of itself, the better.
2019-11-03 20:37:03 +01:00
#endif
}
Kernel: Aggregate TLB flush requests for Regions for SMP Rather than sending one TLB flush request for each page, aggregate them so that we're not spamming the other processors with FlushTLB IPIs.
2020-07-06 12:47:08 -06:00
MM.flush_tlb(vaddr(), page_count());
Kernel: Add mechanism to identity map the lowest 2MB
2020-06-01 22:55:09 -06:00
if (deallocate_range == ShouldDeallocateVirtualMemoryRange::Yes) {
if (m_page_directory->range_allocator().contains(range()))
m_page_directory->range_allocator().deallocate(range());
else
m_page_directory->identity_range_allocator().deallocate(range());
}
Kernel: Remove Region API's for setting/unsetting the page directory This is done implicitly by mapping or unmapping the region.
2019-11-04 00:23:31 +01:00
m_page_directory = nullptr;
Kernel: Move region map/unmap operations into the Region class The more Region can take care of itself, the better.
2019-11-03 20:37:03 +01:00
}
Kernel: Change Region allocation helpers We now can create a cacheable Region, so when map() is called, if a Region is cacheable then all the virtual memory space being allocated to it will be marked as not cache disabled. In addition to that, OS components can create a Region that will be mapped to a specific physical address by using the appropriate helper method.
2020-01-09 23:29:31 +02:00
void Region::set_page_directory(PageDirectory& page_directory)
Kernel: Move region map/unmap operations into the Region class The more Region can take care of itself, the better.
2019-11-03 20:37:03 +01:00
{
Kernel: Remove Region API's for setting/unsetting the page directory This is done implicitly by mapping or unmapping the region.
2019-11-04 00:23:31 +01:00
ASSERT(!m_page_directory || m_page_directory == &page_directory);
Kernel: Aggregate TLB flush requests for Regions for SMP Rather than sending one TLB flush request for each page, aggregate them so that we're not spamming the other processors with FlushTLB IPIs.
2020-07-06 12:47:08 -06:00
ASSERT(s_mm_lock.own_lock());
Kernel: Remove Region API's for setting/unsetting the page directory This is done implicitly by mapping or unmapping the region.
2019-11-04 00:23:31 +01:00
m_page_directory = page_directory;
Kernel: Change Region allocation helpers We now can create a cacheable Region, so when map() is called, if a Region is cacheable then all the virtual memory space being allocated to it will be marked as not cache disabled. In addition to that, OS components can create a Region that will be mapped to a specific physical address by using the appropriate helper method.
2020-01-09 23:29:31 +02:00
}
Kernel: Aggregate TLB flush requests for Regions for SMP Rather than sending one TLB flush request for each page, aggregate them so that we're not spamming the other processors with FlushTLB IPIs.
2020-07-06 12:47:08 -06:00
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
bool Region::map(PageDirectory& page_directory)
Kernel: Change Region allocation helpers We now can create a cacheable Region, so when map() is called, if a Region is cacheable then all the virtual memory space being allocated to it will be marked as not cache disabled. In addition to that, OS components can create a Region that will be mapped to a specific physical address by using the appropriate helper method.
2020-01-09 23:29:31 +02:00
{
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
ScopedSpinLock lock(s_mm_lock);
Kernel: Protect the PageDirectory from concurrent access
2020-10-31 17:19:18 -06:00
ScopedSpinLock page_lock(page_directory.get_lock());
Kernel: Aggregate TLB flush requests for Regions for SMP Rather than sending one TLB flush request for each page, aggregate them so that we're not spamming the other processors with FlushTLB IPIs.
2020-07-06 12:47:08 -06:00
set_page_directory(page_directory);
Kernel: Merge MemoryManager::map_region_at_address() into Region::map()
2019-11-04 00:05:57 +01:00
#ifdef MM_DEBUG
Region: Use dbg() instead of dbgprintf()
2020-02-24 16:40:42 +02:00
dbg() << "MM: Region::map() will map VMO pages " << first_page_index() << " - " << last_page_index() << " (VMO page count: " << vmobject().page_count() << ")";
Kernel: Merge MemoryManager::map_region_at_address() into Region::map()
2019-11-04 00:05:57 +01:00
#endif
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
size_t page_index = 0;
while (page_index < page_count()) {
if (!map_individual_page_impl(page_index))
break;
++page_index;
}
if (page_index > 0) {
MM.flush_tlb(vaddr(), page_index);
return page_index == page_count();
}
return false;
Kernel: Move region map/unmap operations into the Region class The more Region can take care of itself, the better.
2019-11-03 20:37:03 +01:00
}
Kernel: Teach Region how to remap itself Now remapping (i.e flushing kernel metadata to the CPU page tables) is done by simply calling Region::remap().
2019-11-03 20:59:54 +01:00
void Region::remap()
{
ASSERT(m_page_directory);
Kernel: Merge MemoryManager::map_region_at_address() into Region::map()
2019-11-04 00:05:57 +01:00
map(*m_page_directory);
Kernel: Teach Region how to remap itself Now remapping (i.e flushing kernel metadata to the CPU page tables) is done by simply calling Region::remap().
2019-11-03 20:59:54 +01:00
}
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
PageFaultResponse Region::handle_fault(const PageFault& fault)
{
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
ScopedSpinLock lock(s_mm_lock);
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
auto page_index_in_region = page_index_from_address(fault.vaddr());
if (fault.type() == PageFault::Type::PageNotPresent) {
Kernel: Write-only regions should still be mapped as present There is no real "read protection" on x86, so we have no choice but to map write-only pages simply as "present & read/write". If we get a read page fault in a non-readable region, that's still a correctness issue, so we crash the process. It's by no means a complete protection against invalid reads, since it's trivial to fool the kernel by first causing a write fault in the same region.
2020-01-20 13:06:55 +01:00
if (fault.is_read() && !is_readable()) {
Region: Use dbg() instead of dbgprintf()
2020-02-24 16:40:42 +02:00
dbg() << "NP(non-readable) fault in Region{" << this << "}[" << page_index_in_region << "]";
Kernel: Crash on memory access in non-readable regions This patch makes it possible to make memory regions non-readable. This is enforced using the "present" bit in the page tables. A process that hits an not-present page fault in a non-readable region will be crashed.
2019-12-02 19:14:16 +01:00
return PageFaultResponse::ShouldCrash;
}
Kernel: Unmap non-readable pages This was caught by running all crash tests with "crash -A". Basically, non-readable pages need to not be mapped *at all* so that a "page not present" exception is provoked on access. Unfortunately x86 does not support write-only mappings, so this is the best we can do. Fixes #1336.
2020-03-06 09:58:59 +01:00
if (fault.is_write() && !is_writable()) {
dbg() << "NP(non-writable) write fault in Region{" << this << "}[" << page_index_in_region << "] at " << fault.vaddr();
return PageFaultResponse::ShouldCrash;
}
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
if (vmobject().is_inode()) {
#ifdef PAGE_FAULT_DEBUG
Region: Use dbg() instead of dbgprintf()
2020-02-24 16:40:42 +02:00
dbg() << "NP(inode) fault in Region{" << this << "}[" << page_index_in_region << "]";
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
#endif
return handle_inode_fault(page_index_in_region);
}
Kernel: Implement lazy committed page allocation By designating a committed page pool we can guarantee to have physical pages available for lazy allocation in mappings. However, when forking we will overcommit. The assumption is that worst-case it's better for the fork to die due to insufficient physical memory on COW access than the parent that created the region. If a fork wants to ensure that all memory is available (trigger a commit) then it can use madvise. This also means that fork now can gracefully fail if we don't have enough physical pages available.
2020-09-04 21:12:25 -06:00
auto& page_slot = physical_page_slot(page_index_in_region);
if (page_slot->is_lazy_committed_page()) {
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
auto page_index_in_vmobject = translate_to_vmobject_page(page_index_in_region);
page_slot = static_cast<AnonymousVMObject&>(*m_vmobject).allocate_committed_page(page_index_in_vmobject);
remap_vmobject_page(page_index_in_vmobject);
Kernel: Implement lazy committed page allocation By designating a committed page pool we can guarantee to have physical pages available for lazy allocation in mappings. However, when forking we will overcommit. The assumption is that worst-case it's better for the fork to die due to insufficient physical memory on COW access than the parent that created the region. If a fork wants to ensure that all memory is available (trigger a commit) then it can use madvise. This also means that fork now can gracefully fail if we don't have enough physical pages available.
2020-09-04 21:12:25 -06:00
return PageFaultResponse::Continue;
}
Kernel: Use a shared physical page for zero-filled pages until written This patch adds a globally shared zero-filled PhysicalPage that will be mapped into every slot of every zero-filled AnonymousVMObject until that page is written to, achieving CoW-like zero-filled pages. Initial testing show that this doesn't actually achieve any sharing yet but it seems like a good design regardless, since it may reduce the number of page faults taken by programs. If you look at the refcount of MM.shared_zero_page() it will have quite a high refcount, but that's just because everything maps it everywhere. If you want to see the "real" refcount, you can build with the MAP_SHARED_ZERO_PAGE_LAZILY flag, and we'll defer mapping of the shared zero page until the first NP read fault. I've left this behavior behind a flag for future testing of this code.
2020-02-15 13:12:02 +01:00
#ifdef MAP_SHARED_ZERO_PAGE_LAZILY
if (fault.is_read()) {
Kernel: Implement lazy committed page allocation By designating a committed page pool we can guarantee to have physical pages available for lazy allocation in mappings. However, when forking we will overcommit. The assumption is that worst-case it's better for the fork to die due to insufficient physical memory on COW access than the parent that created the region. If a fork wants to ensure that all memory is available (trigger a commit) then it can use madvise. This also means that fork now can gracefully fail if we don't have enough physical pages available.
2020-09-04 21:12:25 -06:00
page_slot = MM.shared_zero_page();
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
remap_vmobject_page(translate_to_vmobject_page(page_index_in_region));
Kernel: Use a shared physical page for zero-filled pages until written This patch adds a globally shared zero-filled PhysicalPage that will be mapped into every slot of every zero-filled AnonymousVMObject until that page is written to, achieving CoW-like zero-filled pages. Initial testing show that this doesn't actually achieve any sharing yet but it seems like a good design regardless, since it may reduce the number of page faults taken by programs. If you look at the refcount of MM.shared_zero_page() it will have quite a high refcount, but that's just because everything maps it everywhere. If you want to see the "real" refcount, you can build with the MAP_SHARED_ZERO_PAGE_LAZILY flag, and we'll defer mapping of the shared zero page until the first NP read fault. I've left this behavior behind a flag for future testing of this code.
2020-02-15 13:12:02 +01:00
return PageFaultResponse::Continue;
}
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
return handle_zero_fault(page_index_in_region);
Kernel: Use a shared physical page for zero-filled pages until written This patch adds a globally shared zero-filled PhysicalPage that will be mapped into every slot of every zero-filled AnonymousVMObject until that page is written to, achieving CoW-like zero-filled pages. Initial testing show that this doesn't actually achieve any sharing yet but it seems like a good design regardless, since it may reduce the number of page faults taken by programs. If you look at the refcount of MM.shared_zero_page() it will have quite a high refcount, but that's just because everything maps it everywhere. If you want to see the "real" refcount, you can build with the MAP_SHARED_ZERO_PAGE_LAZILY flag, and we'll defer mapping of the shared zero page until the first NP read fault. I've left this behavior behind a flag for future testing of this code.
2020-02-15 13:12:02 +01:00
#else
LibWeb: Move everything into the Web namespace
2020-03-07 10:27:02 +01:00
dbg() << "BUG! Unexpected NP fault at " << fault.vaddr();
return PageFaultResponse::ShouldCrash;
Kernel: Use a shared physical page for zero-filled pages until written This patch adds a globally shared zero-filled PhysicalPage that will be mapped into every slot of every zero-filled AnonymousVMObject until that page is written to, achieving CoW-like zero-filled pages. Initial testing show that this doesn't actually achieve any sharing yet but it seems like a good design regardless, since it may reduce the number of page faults taken by programs. If you look at the refcount of MM.shared_zero_page() it will have quite a high refcount, but that's just because everything maps it everywhere. If you want to see the "real" refcount, you can build with the MAP_SHARED_ZERO_PAGE_LAZILY flag, and we'll defer mapping of the shared zero page until the first NP read fault. I've left this behavior behind a flag for future testing of this code.
2020-02-15 13:12:02 +01:00
#endif
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
}
ASSERT(fault.type() == PageFault::Type::ProtectionViolation);
Kernel: Don't CoW non-writable pages A page fault in a page marked for CoW should not trigger a CoW if the page is non-writable. I think this makes sense.
2019-12-02 19:20:09 +01:00
if (fault.access() == PageFault::Access::Write && is_writable() && should_cow(page_index_in_region)) {
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
#ifdef PAGE_FAULT_DEBUG
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
dbg() << "PV(cow) fault in Region{" << this << "}[" << page_index_in_region << "] at " << fault.vaddr();
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
#endif
Kernel: Implement lazy committed page allocation By designating a committed page pool we can guarantee to have physical pages available for lazy allocation in mappings. However, when forking we will overcommit. The assumption is that worst-case it's better for the fork to die due to insufficient physical memory on COW access than the parent that created the region. If a fork wants to ensure that all memory is available (trigger a commit) then it can use madvise. This also means that fork now can gracefully fail if we don't have enough physical pages available.
2020-09-04 21:12:25 -06:00
auto* phys_page = physical_page(page_index_in_region);
if (phys_page->is_shared_zero_page() || phys_page->is_lazy_committed_page()) {
Kernel: Use a shared physical page for zero-filled pages until written This patch adds a globally shared zero-filled PhysicalPage that will be mapped into every slot of every zero-filled AnonymousVMObject until that page is written to, achieving CoW-like zero-filled pages. Initial testing show that this doesn't actually achieve any sharing yet but it seems like a good design regardless, since it may reduce the number of page faults taken by programs. If you look at the refcount of MM.shared_zero_page() it will have quite a high refcount, but that's just because everything maps it everywhere. If you want to see the "real" refcount, you can build with the MAP_SHARED_ZERO_PAGE_LAZILY flag, and we'll defer mapping of the shared zero page until the first NP read fault. I've left this behavior behind a flag for future testing of this code.
2020-02-15 13:12:02 +01:00
#ifdef PAGE_FAULT_DEBUG
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
dbg() << "NP(zero) fault in Region{" << this << "}[" << page_index_in_region << "] at " << fault.vaddr();
Kernel: Use a shared physical page for zero-filled pages until written This patch adds a globally shared zero-filled PhysicalPage that will be mapped into every slot of every zero-filled AnonymousVMObject until that page is written to, achieving CoW-like zero-filled pages. Initial testing show that this doesn't actually achieve any sharing yet but it seems like a good design regardless, since it may reduce the number of page faults taken by programs. If you look at the refcount of MM.shared_zero_page() it will have quite a high refcount, but that's just because everything maps it everywhere. If you want to see the "real" refcount, you can build with the MAP_SHARED_ZERO_PAGE_LAZILY flag, and we'll defer mapping of the shared zero page until the first NP read fault. I've left this behavior behind a flag for future testing of this code.
2020-02-15 13:12:02 +01:00
#endif
return handle_zero_fault(page_index_in_region);
}
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
return handle_cow_fault(page_index_in_region);
}
Kernel: Use klog() instead of kprintf() Also, duplicate data in dbg() and klog() calls were removed. In addition, leakage of virtual address to kernel log is prevented. This is done by replacing kprintf() calls to dbg() calls with the leaked data instead. Also, other kprintf() calls were replaced with klog().
2020-03-01 21:45:39 +02:00
dbg() << "PV(error) fault in Region{" << this << "}[" << page_index_in_region << "] at " << fault.vaddr();
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
return PageFaultResponse::ShouldCrash;
}
PageFaultResponse Region::handle_zero_fault(size_t page_index_in_region)
{
ASSERT_INTERRUPTS_DISABLED();
ASSERT(vmobject().is_anonymous());
Kernel: Start implementing purgeable memory support It's now possible to get purgeable memory by using mmap(MAP_PURGEABLE). Purgeable memory has a "volatile" flag that can be set using madvise(): - madvise(..., MADV_SET_VOLATILE) - madvise(..., MADV_SET_NONVOLATILE) When in the "volatile" state, the kernel may take away the underlying physical memory pages at any time, without notifying the owner. This gives you a guilt discount when caching very large things. :^) Setting a purgeable region to non-volatile will return whether or not the memory has been taken away by the kernel while being volatile. Basically, if madvise(..., MADV_SET_NONVOLATILE) returns 1, that means the memory was purged while volatile, and whatever was in that piece of memory needs to be reconstructed before use.
2019-12-09 19:12:38 +01:00
LOCKER(vmobject().m_paging_lock);
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
Kernel: Add Region helpers for accessing underlying physical pages Since a Region is basically a view into a potentially larger VMObject, it was always necessary to include the Region starting offset when accessing its underlying physical pages. Until now, you had to do that manually, but this patch adds a simple Region::physical_page() for read-only access and a physical_page_slot() when you want a mutable reference to the RefPtr<PhysicalPage> itself. A lot of code is simplified by making use of this.
2020-04-28 16:19:50 +02:00
auto& page_slot = physical_page_slot(page_index_in_region);
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
auto page_index_in_vmobject = translate_to_vmobject_page(page_index_in_region);
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
Kernel: Implement lazy committed page allocation By designating a committed page pool we can guarantee to have physical pages available for lazy allocation in mappings. However, when forking we will overcommit. The assumption is that worst-case it's better for the fork to die due to insufficient physical memory on COW access than the parent that created the region. If a fork wants to ensure that all memory is available (trigger a commit) then it can use madvise. This also means that fork now can gracefully fail if we don't have enough physical pages available.
2020-09-04 21:12:25 -06:00
if (!page_slot.is_null() && !page_slot->is_shared_zero_page() && !page_slot->is_lazy_committed_page()) {
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
#ifdef PAGE_FAULT_DEBUG
Region: Use dbg() instead of dbgprintf()
2020-02-24 16:40:42 +02:00
dbg() << "MM: zero_page() but page already present. Fine with me!";
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
#endif
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
if (!remap_vmobject_page(page_index_in_vmobject))
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
return PageFaultResponse::OutOfMemory;
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
return PageFaultResponse::Continue;
}
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
auto current_thread = Thread::current();
if (current_thread != nullptr)
current_thread->did_zero_fault();
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
Kernel: Implement lazy committed page allocation By designating a committed page pool we can guarantee to have physical pages available for lazy allocation in mappings. However, when forking we will overcommit. The assumption is that worst-case it's better for the fork to die due to insufficient physical memory on COW access than the parent that created the region. If a fork wants to ensure that all memory is available (trigger a commit) then it can use madvise. This also means that fork now can gracefully fail if we don't have enough physical pages available.
2020-09-04 21:12:25 -06:00
if (page_slot->is_lazy_committed_page()) {
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
page_slot = static_cast<AnonymousVMObject&>(*m_vmobject).allocate_committed_page(page_index_in_vmobject);
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
#ifdef PAGE_FAULT_DEBUG
dbg() << " >> ALLOCATED COMMITTED " << page_slot->paddr();
#endif
Kernel: Implement lazy committed page allocation By designating a committed page pool we can guarantee to have physical pages available for lazy allocation in mappings. However, when forking we will overcommit. The assumption is that worst-case it's better for the fork to die due to insufficient physical memory on COW access than the parent that created the region. If a fork wants to ensure that all memory is available (trigger a commit) then it can use madvise. This also means that fork now can gracefully fail if we don't have enough physical pages available.
2020-09-04 21:12:25 -06:00
} else {
page_slot = MM.allocate_user_physical_page(MemoryManager::ShouldZeroFill::Yes);
if (page_slot.is_null()) {
klog() << "MM: handle_zero_fault was unable to allocate a physical page";
return PageFaultResponse::OutOfMemory;
}
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
#ifdef PAGE_FAULT_DEBUG
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
dbg() << " >> ALLOCATED " << page_slot->paddr();
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
#endif
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
}
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
if (!remap_vmobject_page(page_index_in_vmobject)) {
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
klog() << "MM: handle_zero_fault was unable to allocate a page table to map " << page_slot;
return PageFaultResponse::OutOfMemory;
}
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
return PageFaultResponse::Continue;
}
PageFaultResponse Region::handle_cow_fault(size_t page_index_in_region)
{
ASSERT_INTERRUPTS_DISABLED();
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
auto current_thread = Thread::current();
if (current_thread)
current_thread->did_cow_fault();
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
if (!vmobject().is_anonymous())
return PageFaultResponse::ShouldCrash;
Kernel: Make copy_to/from_user safe and remove unnecessary checks Since the CPU already does almost all necessary validation steps for us, we don't really need to attempt to do this. Doing it ourselves doesn't really work very reliably, because we'd have to account for other processors modifying virtual memory, and we'd have to account for e.g. pages not being able to be allocated due to insufficient resources. So change the copy_to/from_user (and associated helper functions) to use the new safe_memcpy, which will return whether it succeeded or not. The only manual validation step needed (which the CPU can't perform for us) is making sure the pointers provided by user mode aren't pointing to kernel mappings. To make it easier to read/write from/to either kernel or user mode data add the UserOrKernelBuffer helper class, which will internally either use copy_from/to_user or directly memcpy, or pass the data through directly using a temporary buffer on the stack. Last but not least we need to keep syscall params trivial as we need to copy them from/to user mode using copy_from/to_user.
2020-09-11 21:11:07 -06:00
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
auto page_index_in_vmobject = translate_to_vmobject_page(page_index_in_region);
auto response = reinterpret_cast<AnonymousVMObject&>(vmobject()).handle_cow_fault(page_index_in_vmobject, vaddr().offset(page_index_in_region * PAGE_SIZE));
if (!remap_vmobject_page(page_index_in_vmobject))
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
return PageFaultResponse::OutOfMemory;
Kernel: Merge PurgeableVMObject into AnonymousVMObject This implements memory commitments and lazy-allocation of committed memory.
2020-09-05 15:52:14 -06:00
return response;
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
}
PageFaultResponse Region::handle_inode_fault(size_t page_index_in_region)
{
ASSERT_INTERRUPTS_DISABLED();
ASSERT(vmobject().is_inode());
LOCKER(vmobject().m_paging_lock);
Kernel: Fix some problems with Thread::wait_on and Lock This changes the Thread::wait_on function to not enable interrupts upon leaving, which caused some problems with page fault handlers and in other situations. It may now be called from critical sections, with interrupts enabled or disabled, and returns to the same state. This also requires some fixes to Lock. To aid debugging, a new define LOCK_DEBUG is added that enables checking for Lock leaks upon finalization of a Thread.
2020-11-30 19:04:36 -07:00
ASSERT_INTERRUPTS_DISABLED();
Kernel: Split InodeVMObject into two subclasses We now have PrivateInodeVMObject and SharedInodeVMObject, corresponding to MAP_PRIVATE and MAP_SHARED respectively. Note that PrivateInodeVMObject is not used yet.
2020-02-28 20:20:35 +01:00
auto& inode_vmobject = static_cast<InodeVMObject&>(vmobject());
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
auto page_index_in_vmobject = translate_to_vmobject_page(page_index_in_region);
auto& vmobject_physical_page_entry = inode_vmobject.physical_pages()[page_index_in_vmobject];
Kernel: The inode fault handler should grab the VMObject lock earlier It doesn't look healthy to create raw references into an array before a temporary unlock. In fact, that temporary unlock looks generally unhealthy, but it's a different problem.
2020-02-08 12:54:06 +01:00
Kernel: Add a mode flag to sys$purge and allow purging clean inodes
2019-12-29 13:16:53 +01:00
#ifdef PAGE_FAULT_DEBUG
Kernel: Tidy up debug logging a little bit When using dbg() in the kernel, the output is automatically prefixed with [Process(PID:TID)]. This makes it a lot easier to understand which thread is generating the output. This patch also cleans up some common logging messages and removes the now-unnecessary "dbg() << *current << ..." pattern.
2020-01-21 16:14:39 +01:00
dbg() << "Inode fault in " << name() << " page index: " << page_index_in_region;
Kernel: Add a mode flag to sys$purge and allow purging clean inodes
2019-12-29 13:16:53 +01:00
#endif
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
if (!vmobject_physical_page_entry.is_null()) {
#ifdef PAGE_FAULT_DEBUG
Region: Use dbg() instead of dbgprintf()
2020-02-24 16:40:42 +02:00
dbg() << ("MM: page_in_from_inode() but page already present. Fine with me!");
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
#endif
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
if (!remap_vmobject_page(page_index_in_vmobject))
Kernel: Handle committing pages in regions more gracefully Sometimes a physical underlying page may be there, but we may be unable to allocate a page table that may be needed to map it. Bubble up such mapping errors so that they can be handled more appropriately.
2020-09-01 16:10:54 -06:00
return PageFaultResponse::OutOfMemory;
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
return PageFaultResponse::Continue;
}
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
auto current_thread = Thread::current();
if (current_thread)
current_thread->did_inode_fault();
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
#ifdef MM_DEBUG
Region: Use dbg() instead of dbgprintf()
2020-02-24 16:40:42 +02:00
dbg() << "MM: page_in_from_inode ready to read from inode";
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
#endif
Kernel: Fix some problems with Thread::wait_on and Lock This changes the Thread::wait_on function to not enable interrupts upon leaving, which caused some problems with page fault handlers and in other situations. It may now be called from critical sections, with interrupts enabled or disabled, and returns to the same state. This also requires some fixes to Lock. To aid debugging, a new define LOCK_DEBUG is added that enables checking for Lock leaks upon finalization of a Thread.
2020-11-30 19:04:36 -07:00
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
u8 page_buffer[PAGE_SIZE];
auto& inode = inode_vmobject.inode();
Kernel: Make copy_to/from_user safe and remove unnecessary checks Since the CPU already does almost all necessary validation steps for us, we don't really need to attempt to do this. Doing it ourselves doesn't really work very reliably, because we'd have to account for other processors modifying virtual memory, and we'd have to account for e.g. pages not being able to be allocated due to insufficient resources. So change the copy_to/from_user (and associated helper functions) to use the new safe_memcpy, which will return whether it succeeded or not. The only manual validation step needed (which the CPU can't perform for us) is making sure the pointers provided by user mode aren't pointing to kernel mappings. To make it easier to read/write from/to either kernel or user mode data add the UserOrKernelBuffer helper class, which will internally either use copy_from/to_user or directly memcpy, or pass the data through directly using a temporary buffer on the stack. Last but not least we need to keep syscall params trivial as we need to copy them from/to user mode using copy_from/to_user.
2020-09-11 21:11:07 -06:00
auto buffer = UserOrKernelBuffer::for_kernel_buffer(page_buffer);
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
auto nread = inode.read_bytes(page_index_in_vmobject * PAGE_SIZE, PAGE_SIZE, buffer, nullptr);
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
if (nread < 0) {
Kernel: Use klog() instead of kprintf() Also, duplicate data in dbg() and klog() calls were removed. In addition, leakage of virtual address to kernel log is prevented. This is done by replacing kprintf() calls to dbg() calls with the leaked data instead. Also, other kprintf() calls were replaced with klog().
2020-03-01 21:45:39 +02:00
klog() << "MM: handle_inode_fault had error (" << nread << ") while reading!";
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
return PageFaultResponse::ShouldCrash;
}
if (nread < PAGE_SIZE) {
// If we read less than a page, zero out the rest to avoid leaking uninitialized data.
memset(page_buffer + nread, 0, PAGE_SIZE - nread);
}
Kernel: Fix some problems with Thread::wait_on and Lock This changes the Thread::wait_on function to not enable interrupts upon leaving, which caused some problems with page fault handlers and in other situations. It may now be called from critical sections, with interrupts enabled or disabled, and returns to the same state. This also requires some fixes to Lock. To aid debugging, a new define LOCK_DEBUG is added that enables checking for Lock leaks upon finalization of a Thread.
2020-11-30 19:04:36 -07:00
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
vmobject_physical_page_entry = MM.allocate_user_physical_page(MemoryManager::ShouldZeroFill::No);
if (vmobject_physical_page_entry.is_null()) {
Kernel: Use klog() instead of kprintf() Also, duplicate data in dbg() and klog() calls were removed. In addition, leakage of virtual address to kernel log is prevented. This is done by replacing kprintf() calls to dbg() calls with the leaked data instead. Also, other kprintf() calls were replaced with klog().
2020-03-01 21:45:39 +02:00
klog() << "MM: handle_inode_fault was unable to allocate a physical page";
Kernel: Crash the current process on OOM (instead of panicking kernel) This patch adds PageFaultResponse::OutOfMemory which informs the fault handler that we were unable to allocate a necessary physical page and cannot continue. In response to this, the kernel will crash the current process. Because we are OOM, we can't symbolicate the crash like we normally would (since the ELF symbolication code needs to allocate), so we also communicate to Process::crash() that we're out of memory. Now we can survive "allocate 300 MB" (only the allocate process dies.) This is definitely not perfect and can easily end up killing a random innocent other process who happened to allocate one page at the wrong time, but it's a *lot* better than panicking on OOM. :^)
2020-05-06 21:11:38 +02:00
return PageFaultResponse::OutOfMemory;
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
}
Kernel: Enable the x86 WP bit to catch invalid memory writes in ring 0 Setting this bit will cause the CPU to generate a page fault when writing to read-only memory, even if we're executing in the kernel. Seemingly the only change needed to make this work was to have the inode-backed page fault handler use a temporary mapping for writing the read-from-disk data into the newly-allocated physical page.
2019-12-21 16:21:13 +01:00
u8* dest_ptr = MM.quickmap_page(*vmobject_physical_page_entry);
Kernel: Make copy_to/from_user safe and remove unnecessary checks Since the CPU already does almost all necessary validation steps for us, we don't really need to attempt to do this. Doing it ourselves doesn't really work very reliably, because we'd have to account for other processors modifying virtual memory, and we'd have to account for e.g. pages not being able to be allocated due to insufficient resources. So change the copy_to/from_user (and associated helper functions) to use the new safe_memcpy, which will return whether it succeeded or not. The only manual validation step needed (which the CPU can't perform for us) is making sure the pointers provided by user mode aren't pointing to kernel mappings. To make it easier to read/write from/to either kernel or user mode data add the UserOrKernelBuffer helper class, which will internally either use copy_from/to_user or directly memcpy, or pass the data through directly using a temporary buffer on the stack. Last but not least we need to keep syscall params trivial as we need to copy them from/to user mode using copy_from/to_user.
2020-09-11 21:11:07 -06:00
{
void* fault_at;
if (!safe_memcpy(dest_ptr, page_buffer, PAGE_SIZE, fault_at)) {
if ((u8*)fault_at >= dest_ptr && (u8*)fault_at <= dest_ptr + PAGE_SIZE)
dbg() << " >> inode fault: error copying data to " << vmobject_physical_page_entry->paddr() << "/" << VirtualAddress(dest_ptr) << ", failed at " << VirtualAddress(fault_at);
else
ASSERT_NOT_REACHED();
}
}
Kernel: Enable the x86 WP bit to catch invalid memory writes in ring 0 Setting this bit will cause the CPU to generate a page fault when writing to read-only memory, even if we're executing in the kernel. Seemingly the only change needed to make this work was to have the inode-backed page fault handler use a temporary mapping for writing the read-from-disk data into the newly-allocated physical page.
2019-12-21 16:21:13 +01:00
MM.unquickmap_page();
Kernel: If a VMObject is shared, broadcast page remappings If we remap pages (e.g. lazy allocation) inside a VMObject that is shared among more than one region, broadcast it to any other region that may be mapping the same page.
2021-01-02 12:03:14 -07:00
remap_vmobject_page(page_index_in_vmobject);
Kernel: Move page fault handling from MemoryManager to Region After the page fault handler has found the region in which the fault occurred, do the rest of the work in the region itself. This patch also makes all fault types consistently crash the process if a new page is needed but we're all out of pages.
2019-11-04 00:45:33 +01:00
return PageFaultResponse::Continue;
}
Kernel: Move all code into the Kernel namespace
2020-02-16 01:27:42 +01:00
Kernel: Memory purging improvements This adds the ability for a Region to define volatile/nonvolatile areas within mapped memory using madvise(). This also means that memory purging takes into account all views of the PurgeableVMObject and only purges memory that is not needed by all of them. When calling madvise() to change an area to nonvolatile memory, return whether memory from that area was purged. At that time also try to remap all memory that is requested to be nonvolatile, and if insufficient pages are available notify the caller of that fact.
2020-09-02 22:57:09 -06:00
RefPtr<Process> Region::get_owner()
{
return m_owner.strong_ref();
}
Kernel: Move all code into the Kernel namespace
2020-02-16 01:27:42 +01:00
}
Reference in a new issue Copy permalink