Stowage/ladybird
2
0
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2025-12-08 06:09:58 +00:00
Code Issues Projects Releases Packages Wiki Activity
ladybird/Kernel/Arch/i386/CPU.cpp

2284 lines
86 KiB
C++
Raw Normal View History

Meta: Add license header to source files As suggested by Joshua, this commit adds the 2-clause BSD license as a comment block to the top of every source file. For the first pass, I've just added myself for simplicity. I encourage everyone to add themselves as copyright holders of any file they've added or modified in some significant way. If I've added myself in error somewhere, feel free to replace it with the appropriate copyright holder instead. Going forward, all new source files should include a license header.
2020-01-18 09:38:21 +01:00
/*
* Copyright (c) 2018-2020, Andreas Kling <kling@serenityos.org>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* 1. Redistributions of source code must retain the above copyright notice, this
* list of conditions and the following disclaimer.
*
* 2. Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
* SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
* CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
#include <AK/Assertions.h>
Meta: Split debug defines into multiple headers. The following script was used to make these changes: #!/bin/bash set -e tmp=$(mktemp -d) echo "tmp=$tmp" find Kernel \( -name '*.cpp' -o -name '*.h' \) | sort > $tmp/Kernel.files find . \( -path ./Toolchain -prune -o -path ./Build -prune -o -path ./Kernel -prune \) -o \( -name '*.cpp' -o -name '*.h' \) -print | sort > $tmp/EverythingExceptKernel.files cat $tmp/Kernel.files | xargs grep -Eho '[A-Z0-9_]+_DEBUG' | sort | uniq > $tmp/Kernel.macros cat $tmp/EverythingExceptKernel.files | xargs grep -Eho '[A-Z0-9_]+_DEBUG' | sort | uniq > $tmp/EverythingExceptKernel.macros comm -23 $tmp/Kernel.macros $tmp/EverythingExceptKernel.macros > $tmp/Kernel.unique comm -1 $tmp/Kernel.macros $tmp/EverythingExceptKernel.macros > $tmp/EverythingExceptKernel.unique cat $tmp/Kernel.unique | awk '{ print "#cmakedefine01 "$1 }' > $tmp/Kernel.header cat $tmp/EverythingExceptKernel.unique | awk '{ print "#cmakedefine01 "$1 }' > $tmp/EverythingExceptKernel.header for macro in $(cat $tmp/Kernel.unique) do cat $tmp/Kernel.files | xargs grep -l $macro >> $tmp/Kernel.new-includes ||: done cat $tmp/Kernel.new-includes | sort > $tmp/Kernel.new-includes.sorted for macro in $(cat $tmp/EverythingExceptKernel.unique) do cat $tmp/Kernel.files | xargs grep -l $macro >> $tmp/Kernel.old-includes ||: done cat $tmp/Kernel.old-includes | sort > $tmp/Kernel.old-includes.sorted comm -23 $tmp/Kernel.new-includes.sorted $tmp/Kernel.old-includes.sorted > $tmp/Kernel.includes.new comm -13 $tmp/Kernel.new-includes.sorted $tmp/Kernel.old-includes.sorted > $tmp/Kernel.includes.old comm -12 $tmp/Kernel.new-includes.sorted $tmp/Kernel.old-includes.sorted > $tmp/Kernel.includes.mixed for file in $(cat $tmp/Kernel.includes.new) do sed -i -E 's/#include <AK\/Debug\.h>/#include <Kernel\/Debug\.h>/' $file done for file in $(cat $tmp/Kernel.includes.mixed) do echo "mixed include in $file, requires manual editing." done
2021-01-25 16:07:10 +01:00
#include <Kernel/Debug.h>
Kernel: Don't let signals unblock threads while handling a page fault It was possible to signal a process while it was paging in an inode backed VM object. This would cause the inode read to EINTR, and the page fault handler would assert. Solve this by simply not unblocking threads due to signals if they are currently busy handling a page fault. This is probably not the best way to solve this issue, so I've added a FIXME to that effect.
2021-01-21 00:06:19 +01:00
#include <AK/ScopeGuard.h>
Kernel: Use klog() instead of kprintf() Also, duplicate data in dbg() and klog() calls were removed. In addition, leakage of virtual address to kernel log is prevented. This is done by replacing kprintf() calls to dbg() calls with the leaked data instead. Also, other kprintf() calls were replaced with klog().
2020-03-01 21:45:39 +02:00
#include <AK/String.h>
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
#include <AK/StringBuilder.h>
Kernel: Run clang-format on everything.
2019-06-07 11:43:58 +02:00
#include <AK/Types.h>
Kernel: Move i386.{cpp,h} => Arch/i386/CPU.{cpp,h} There's a ton of work that would need to be done before we could spin up on another architecture, but let's at least try to separate things out a bit.
2019-06-07 20:02:01 +02:00
#include <Kernel/Arch/i386/CPU.h>
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
#include <Kernel/Arch/i386/ISRStubs.h>
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
#include <Kernel/Arch/i386/ProcessorInfo.h>
Kernel: Add safe atomic functions This allows us to perform atomic operations on potentially unsafe user space pointers.
2020-12-19 18:48:56 -07:00
#include <Kernel/Arch/i386/SafeMem.h>
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
#include <Kernel/IO.h>
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
#include <Kernel/Interrupts/APIC.h>
#include <Kernel/Interrupts/GenericInterruptHandler.h>
#include <Kernel/Interrupts/IRQHandler.h>
#include <Kernel/Interrupts/InterruptManagement.h>
#include <Kernel/Interrupts/SharedIRQHandler.h>
CPU: Simplify handle_interrupt() function
2020-02-28 18:25:19 +02:00
#include <Kernel/Interrupts/SpuriousInterruptHandler.h>
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
#include <Kernel/Interrupts/UnhandledInterruptHandler.h>
Kernel: Dump backtrace on user process page fault.
2019-04-28 22:05:13 +02:00
#include <Kernel/KSyms.h>
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
#include <Kernel/Process.h>
Kernel: Make use of interrupts as an entropy source Booting old computers without RDRAND/RDSEED and without a disk makes the system severely starved for entropy. Uses interrupts as a source to side-step that issue. Also warn whenever the system is starved of entropy, because that's a non-obvious failure mode.
2021-01-24 18:17:54 +01:00
#include <Kernel/Random.h>
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
#include <Kernel/SpinLock.h>
#include <Kernel/Thread.h>
Kernel: Run clang-format on everything.
2019-06-07 11:43:58 +02:00
#include <Kernel/VM/MemoryManager.h>
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
#include <Kernel/VM/PageDirectory.h>
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
#include <Kernel/VM/ProcessPagingScope.h>
Kernel+LibC: Make page fault crashes a bit more readable. We'll now try to detect crashes that were due to dereferencing nullptr, uninitialized malloc() memory, or recently free()'d memory. It's not perfect but I think it's pretty good. :^) Also added some color to the most important parts of the crash log, and added some more modes to /bin/crash for exercising this code. Fixes #243.
2019-06-19 20:52:12 +02:00
#include <LibC/mallocdefs.h>
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
Kernel: Move all code into the Kernel namespace
2020-02-16 01:27:42 +01:00
namespace Kernel {
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
static DescriptorTablePointer s_idtr;
Kernel: Put GDT and IDT in BSS rather than runtime-allocating them.
2019-05-17 18:25:50 +02:00
static Descriptor s_idt[256];
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
static GenericInterruptHandler* s_interrupt_handler[GENERIC_INTERRUPT_HANDLERS_COUNT];
Add IRQHandler class that can be subclasses to handle an IRQ. Also move Keyboard to a class implementation using this pattern.
2018-10-22 12:58:29 +02:00
Kernel: Make use of interrupts as an entropy source Booting old computers without RDRAND/RDSEED and without a disk makes the system severely starved for entropy. Uses interrupts as a source to side-step that issue. Also warn whenever the system is starved of entropy, because that's a non-obvious failure mode.
2021-01-24 18:17:54 +01:00
static EntropySource s_entropy_source_interrupts{EntropySource::Static::Interrupts};
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
// The compiler can't see the calls to these functions inside assembly.
// Declare them, to avoid dead code warnings.
extern "C" void enter_thread_context(Thread* from_thread, Thread* to_thread);
extern "C" void context_first_init(Thread* from_thread, Thread* to_thread, TrapFrame* trap);
extern "C" u32 do_init_context(Thread* thread, u32 flags);
Everywhere: void arguments to C functions Problem: - C functions with no arguments require a single `void` in the argument list. Solution: - Put the `void` in the argument list of functions in C header files.
2020-12-24 08:41:54 -07:00
extern "C" void exit_kernel_thread(void);
extern "C" void pre_init_finished(void);
extern "C" void post_init_finished(void);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
extern "C" void handle_interrupt(TrapFrame*);
Kernel: Run clang-format on everything.
2019-06-07 11:43:58 +02:00
Kernel: Rename RegisterDump => RegisterState
2020-02-16 00:15:37 +01:00
#define EH_ENTRY(ec, title) \
extern "C" void title##_asm_entry(); \
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
extern "C" void title##_handler(TrapFrame*); \
Kernel: Rename RegisterDump => RegisterState
2020-02-16 00:15:37 +01:00
asm( \
".globl " #title "_asm_entry\n" \
"" #title "_asm_entry: \n" \
" pusha\n" \
" pushl %ds\n" \
" pushl %es\n" \
" pushl %fs\n" \
" pushl %gs\n" \
" pushl %ss\n" \
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
" mov $" __STRINGIFY(GDT_SELECTOR_DATA0) ", %ax\n" \
Kernel: Rename RegisterDump => RegisterState
2020-02-16 00:15:37 +01:00
" mov %ax, %ds\n" \
" mov %ax, %es\n" \
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
" mov $" __STRINGIFY(GDT_SELECTOR_PROC) ", %ax\n" \
" mov %ax, %fs\n" \
" pushl %esp \n" /* set TrapFrame::regs */ \
" subl $" __STRINGIFY(TRAP_FRAME_SIZE - 4) ", %esp \n" \
" pushl %esp \n" \
Kernel: Rename RegisterDump => RegisterState
2020-02-16 00:15:37 +01:00
" cld\n" \
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
" call enter_trap_no_irq \n" \
Kernel: Rename RegisterDump => RegisterState
2020-02-16 00:15:37 +01:00
" call " #title "_handler\n" \
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
" jmp common_trap_exit \n");
Kernel: Run clang-format on everything.
2019-06-07 11:43:58 +02:00
Kernel: Rename RegisterDump => RegisterState
2020-02-16 00:15:37 +01:00
#define EH_ENTRY_NO_CODE(ec, title) \
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
extern "C" void title##_handler(TrapFrame*); \
Kernel: Rename RegisterDump => RegisterState
2020-02-16 00:15:37 +01:00
extern "C" void title##_asm_entry(); \
asm( \
".globl " #title "_asm_entry\n" \
"" #title "_asm_entry: \n" \
" pushl $0x0\n" \
" pusha\n" \
" pushl %ds\n" \
" pushl %es\n" \
" pushl %fs\n" \
" pushl %gs\n" \
" pushl %ss\n" \
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
" mov $" __STRINGIFY(GDT_SELECTOR_DATA0) ", %ax\n" \
Kernel: Rename RegisterDump => RegisterState
2020-02-16 00:15:37 +01:00
" mov %ax, %ds\n" \
" mov %ax, %es\n" \
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
" mov $" __STRINGIFY(GDT_SELECTOR_PROC) ", %ax\n" \
" mov %ax, %fs\n" \
" pushl %esp \n" /* set TrapFrame::regs */ \
" subl $" __STRINGIFY(TRAP_FRAME_SIZE - 4) ", %esp \n" \
" pushl %esp \n" \
Kernel: Rename RegisterDump => RegisterState
2020-02-16 00:15:37 +01:00
" cld\n" \
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
" call enter_trap_no_irq \n" \
Kernel: Rename RegisterDump => RegisterState
2020-02-16 00:15:37 +01:00
" call " #title "_handler\n" \
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
" jmp common_trap_exit \n");
Turn the syscall interrupt into a trap (by switching the gate type.) This leaves interrupts enabled while we're in the kernel, which is precisely what we want. This uncovered a horrendous problem with kernel tasks silently overflowing their stacks. For now I've simply increased the stack size but I need a more MMU-y solution for this eventually.
2018-10-19 11:28:43 +02:00
Kernel: Rename RegisterDump => RegisterState
2020-02-16 00:15:37 +01:00
static void dump(const RegisterState& regs)
Turn the syscall interrupt into a trap (by switching the gate type.) This leaves interrupts enabled while we're in the kernel, which is precisely what we want. This uncovered a horrendous problem with kernel tasks silently overflowing their stacks. For now I've simply increased the stack size but I need a more MMU-y solution for this eventually.
2018-10-19 11:28:43 +02:00
{
AK: Rename the common integer typedefs to make it obvious what they are. These types can be picked up by including <AK/Types.h>: * u8, u16, u32, u64 (unsigned) * i8, i16, i32, i64 (signed)
2019-07-03 21:17:35 +02:00
u16 ss;
u32 esp;
Kernel: Fix detecting in what ring a crash happened The ring is determined based on the CS register. This fixes crashes being handled as ring 3 crashes even though EIP/CS clearly showed that the crash happened in the kernel.
2020-09-10 09:29:09 -06:00
if (!(regs.cs & 3)) {
Kernel: Implement some basic stack pointer validation VM regions can now be marked as stack regions, which is then validated on syscall, and on page fault. If a thread is caught with its stack pointer pointing into anything that's *not* a Region with its stack bit set, we'll crash the whole process with SIGSTKFLT. Userspace must now allocate custom stacks by using mmap() with the new MAP_STACK flag. This mechanism was first introduced in OpenBSD, and now we have it too, yay! :^)
2019-11-17 12:11:43 +01:00
ss = regs.ss;
Turn the syscall interrupt into a trap (by switching the gate type.) This leaves interrupts enabled while we're in the kernel, which is precisely what we want. This uncovered a horrendous problem with kernel tasks silently overflowing their stacks. For now I've simply increased the stack size but I need a more MMU-y solution for this eventually.
2018-10-19 11:28:43 +02:00
esp = regs.esp;
} else {
Kernel: Rename {ss,esp}_if_crossRing to userspace_{ss,esp} These were always so awkwardly named.
2020-01-09 18:02:01 +01:00
ss = regs.userspace_ss;
esp = regs.userspace_esp;
Turn the syscall interrupt into a trap (by switching the gate type.) This leaves interrupts enabled while we're in the kernel, which is precisely what we want. This uncovered a horrendous problem with kernel tasks silently overflowing their stacks. For now I've simply increased the stack size but I need a more MMU-y solution for this eventually.
2018-10-19 11:28:43 +02:00
}
Kernel: Use klog() instead of kprintf() Also, duplicate data in dbg() and klog() calls were removed. In addition, leakage of virtual address to kernel log is prevented. This is done by replacing kprintf() calls to dbg() calls with the leaked data instead. Also, other kprintf() calls were replaced with klog().
2020-03-01 21:45:39 +02:00
klog() << "exception code: " << String::format("%04x", regs.exception_code) << " (isr: " << String::format("%04x", regs.isr_number);
klog() << " pc=" << String::format("%04x", (u16)regs.cs) << ":" << String::format("%08x", regs.eip) << " flags=" << String::format("%04x", (u16)regs.eflags);
klog() << " stk=" << String::format("%04x", ss) << ":" << String::format("%08x", esp);
klog() << " ds=" << String::format("%04x", (u16)regs.ds) << " es=" << String::format("%04x", (u16)regs.es) << " fs=" << String::format("%04x", (u16)regs.fs) << " gs=" << String::format("%04x", (u16)regs.gs);
klog() << "eax=" << String::format("%08x", regs.eax) << " ebx=" << String::format("%08x", regs.ebx) << " ecx=" << String::format("%08x", regs.ecx) << " edx=" << String::format("%08x", regs.edx);
klog() << "ebp=" << String::format("%08x", regs.ebp) << " esp=" << String::format("%08x", regs.esp) << " esi=" << String::format("%08x", regs.esi) << " edi=" << String::format("%08x", regs.edi);
Kernel: Move kernel above the 3GB virtual address mark The kernel and its static data structures are no longer identity-mapped in the bottom 8MB of the address space, but instead move above 3GB. The first 8MB above 3GB are pseudo-identity-mapped to the bottom 8MB of the physical address space. But things don't have to stay this way! Thanks to Jesse who made an earlier attempt at this, it was really easy to get device drivers working once the page tables were in place! :^) Fixes #734.
2020-01-17 19:59:20 +01:00
u32 cr0;
asm("movl %%cr0, %%eax"
: "=a"(cr0));
u32 cr2;
asm("movl %%cr2, %%eax"
: "=a"(cr2));
Kernel: Add getter and setter for the X86 CR3 register This gets rid of a bunch of inline assembly.
2020-02-10 20:00:32 +01:00
u32 cr3 = read_cr3();
Kernel: Move kernel above the 3GB virtual address mark The kernel and its static data structures are no longer identity-mapped in the bottom 8MB of the address space, but instead move above 3GB. The first 8MB above 3GB are pseudo-identity-mapped to the bottom 8MB of the physical address space. But things don't have to stay this way! Thanks to Jesse who made an earlier attempt at this, it was really easy to get device drivers working once the page tables were in place! :^) Fixes #734.
2020-01-17 19:59:20 +01:00
u32 cr4;
asm("movl %%cr4, %%eax"
: "=a"(cr4));
Kernel: Use klog() instead of kprintf() Also, duplicate data in dbg() and klog() calls were removed. In addition, leakage of virtual address to kernel log is prevented. This is done by replacing kprintf() calls to dbg() calls with the leaked data instead. Also, other kprintf() calls were replaced with klog().
2020-03-01 21:45:39 +02:00
klog() << "cr0=" << String::format("%08x", cr0) << " cr2=" << String::format("%08x", cr2) << " cr3=" << String::format("%08x", cr3) << " cr4=" << String::format("%08x", cr4);
Kernel: Move kernel above the 3GB virtual address mark The kernel and its static data structures are no longer identity-mapped in the bottom 8MB of the address space, but instead move above 3GB. The first 8MB above 3GB are pseudo-identity-mapped to the bottom 8MB of the physical address space. But things don't have to stay this way! Thanks to Jesse who made an earlier attempt at this, it was really easy to get device drivers working once the page tables were in place! :^) Fixes #734.
2020-01-17 19:59:20 +01:00
Kernel: Fix detecting in what ring a crash happened The ring is determined based on the CS register. This fixes crashes being handled as ring 3 crashes even though EIP/CS clearly showed that the crash happened in the kernel.
2020-09-10 09:29:09 -06:00
auto process = Process::current();
Kernel: Add safe_memcpy, safe_memset and safe_strnlen These special functions can be used to safely copy/set memory or determine the length of a string, e.g. provided by user mode. In the event of a page fault, safe_memcpy/safe_memset will return false and safe_strnlen will return -1.
2020-09-10 19:49:31 -06:00
u8 code[8];
void* fault_at;
if (process && safe_memcpy(code, (void*)regs.eip, 8, fault_at)) {
Kernel: Fix SMAP violation when doing a crash dump
2020-01-06 14:26:47 +01:00
SmapDisabler disabler;
Kernel: Add safe_memcpy, safe_memset and safe_strnlen These special functions can be used to safely copy/set memory or determine the length of a string, e.g. provided by user mode. In the event of a page fault, safe_memcpy/safe_memset will return false and safe_strnlen will return -1.
2020-09-10 19:49:31 -06:00
klog() << "code: " << String::format("%02x", code[0]) << " " << String::format("%02x", code[1]) << " " << String::format("%02x", code[2]) << " " << String::format("%02x", code[3]) << " " << String::format("%02x", code[4]) << " " << String::format("%02x", code[5]) << " " << String::format("%02x", code[6]) << " " << String::format("%02x", code[7]);
Kernel: Reduce code duplication in exception handlers.
2019-02-20 12:28:41 +01:00
}
}
Kernel: Crash the current process on OOM (instead of panicking kernel) This patch adds PageFaultResponse::OutOfMemory which informs the fault handler that we were unable to allocate a necessary physical page and cannot continue. In response to this, the kernel will crash the current process. Because we are OOM, we can't symbolicate the crash like we normally would (since the ELF symbolication code needs to allocate), so we also communicate to Process::crash() that we're out of memory. Now we can survive "allocate 300 MB" (only the allocate process dies.) This is definitely not perfect and can easily end up killing a random innocent other process who happened to allocate one page at the wrong time, but it's a *lot* better than panicking on OOM. :^)
2020-05-06 21:11:38 +02:00
void handle_crash(RegisterState& regs, const char* description, int signal, bool out_of_memory)
Kernel: Reduce code duplication in exception handlers.
2019-02-20 12:28:41 +01:00
{
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
auto process = Process::current();
if (!process) {
Kernel: Use klog() instead of kprintf() Also, duplicate data in dbg() and klog() calls were removed. In addition, leakage of virtual address to kernel log is prevented. This is done by replacing kprintf() calls to dbg() calls with the leaked data instead. Also, other kprintf() calls were replaced with klog().
2020-03-01 21:45:39 +02:00
klog() << description << " with !current";
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
Processor::halt();
Kernel: Introduce threads, and refactor everything in support of it. The scheduler now operates on threads, rather than on processes. Each process has a main thread, and can have any number of additional threads. The process exits when the main thread exits. This patch doesn't actually spawn any additional threads, it merely does all the plumbing needed to make it possible. :^)
2019-03-23 22:03:17 +01:00
}
Kernel: Reduce code duplication in exception handlers.
2019-02-20 12:28:41 +01:00
Kernel: Always switch to own page tables when crashing/asserting I noticed this while debugging a crash in backtrace generation. If a process would crash while temporarily inspecting another process's address space, the crashing thread would still use the other process's page tables while handling the crash, causing all kinds of confusion when trying to walk the stack of the crashing thread.
2020-01-19 09:54:58 +01:00
// If a process crashed while inspecting another process,
// make sure we switch back to the right page tables.
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
MM.enter_process_paging_scope(*process);
Kernel: Always switch to own page tables when crashing/asserting I noticed this while debugging a crash in backtrace generation. If a process would crash while temporarily inspecting another process's address space, the crashing thread would still use the other process's page tables while handling the crash, causing all kinds of confusion when trying to walk the stack of the crashing thread.
2020-01-19 09:54:58 +01:00
Kernel: Make Processor::id a static function This eliminates the window between calling Processor::current and the member function where a thread could be moved to another processor. This is generally not as big of a concern as with Processor::current_thread, but also slightly more light weight.
2021-01-26 20:44:01 -07:00
klog() << "CRASH: CPU #" << Processor::id() << " " << description << ". Ring " << (regs.cs & 3) << ".";
Kernel: Send more specific signals when crashing due to CPU exceptions. - For division by zero, send SIGFPE. - For illegal instruction, send SIGILL. - For the rest, default to SIGSEGV.
2019-05-26 02:08:51 +02:00
dump(regs);
Kernel: Dump backtrace on illegal opcode exception.
2019-05-22 13:22:27 +02:00
Kernel: Fix detecting in what ring a crash happened The ring is determined based on the CS register. This fixes crashes being handled as ring 3 crashes even though EIP/CS clearly showed that the crash happened in the kernel.
2020-09-10 09:29:09 -06:00
if (!(regs.cs & 3)) {
Meta: Scale back overly informal user-facing strings We were getting a little overly memey in some places, so let's scale things back to business-casual. Informal language is fine in comments, commits and debug logs, but let's keep the runtime nice and presentable. :^)
2020-06-17 18:20:28 +02:00
klog() << "Crash in ring 0 :(";
Kernel: Symbolicate the crash address too, not just the call stack. Also print it in shiny red to make it extra easy to spot. :^) Fixes #244.
2019-06-19 18:50:02 +02:00
dump_backtrace();
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
Processor::halt();
Turn the syscall interrupt into a trap (by switching the gate type.) This leaves interrupts enabled while we're in the kernel, which is precisely what we want. This uncovered a horrendous problem with kernel tasks silently overflowing their stacks. For now I've simply increased the stack size but I need a more MMU-y solution for this eventually.
2018-10-19 11:28:43 +02:00
}
Kernel: Implement some basic stack pointer validation VM regions can now be marked as stack regions, which is then validated on syscall, and on page fault. If a thread is caught with its stack pointer pointing into anything that's *not* a Region with its stack bit set, we'll crash the whole process with SIGSTKFLT. Userspace must now allocate custom stacks by using mmap() with the new MAP_STACK flag. This mechanism was first introduced in OpenBSD, and now we have it too, yay! :^)
2019-11-17 12:11:43 +01:00
cli();
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
process->crash(signal, regs.eip, out_of_memory);
Kernel: Share code between all the exceptions that cause process crash.
2019-06-25 05:55:18 +02:00
}
Kernel: Tidy up kernel entry points a little bit Now that we can see the kernel entry points all the time in profiles, let's tweak the names a little bit and switch to named exceptions.
2019-12-14 16:09:07 +01:00
EH_ENTRY_NO_CODE(6, illegal_instruction);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
void illegal_instruction_handler(TrapFrame* trap)
Kernel: Share code between all the exceptions that cause process crash.
2019-06-25 05:55:18 +02:00
{
Kernel: Enable SMAP protection on IRQ and exception entry It would be nice to do this in the assembly code, but we have to check if the feature is available before doing a CLAC, so I've put this in the C++ code for now.
2020-01-08 07:27:37 +01:00
clac();
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
handle_crash(*trap->regs, "Illegal instruction", SIGILL);
Kernel: Share code between all the exceptions that cause process crash.
2019-06-25 05:55:18 +02:00
}
Kernel: Tidy up kernel entry points a little bit Now that we can see the kernel entry points all the time in profiles, let's tweak the names a little bit and switch to named exceptions.
2019-12-14 16:09:07 +01:00
EH_ENTRY_NO_CODE(0, divide_error);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
void divide_error_handler(TrapFrame* trap)
Kernel: Share code between all the exceptions that cause process crash.
2019-06-25 05:55:18 +02:00
{
Kernel: Enable SMAP protection on IRQ and exception entry It would be nice to do this in the assembly code, but we have to check if the feature is available before doing a CLAC, so I've put this in the C++ code for now.
2020-01-08 07:27:37 +01:00
clac();
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
handle_crash(*trap->regs, "Divide error", SIGFPE);
Kernel: Share code between all the exceptions that cause process crash.
2019-06-25 05:55:18 +02:00
}
Kernel: Tidy up kernel entry points a little bit Now that we can see the kernel entry points all the time in profiles, let's tweak the names a little bit and switch to named exceptions.
2019-12-14 16:09:07 +01:00
EH_ENTRY(13, general_protection_fault);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
void general_protection_fault_handler(TrapFrame* trap)
Kernel: Share code between all the exceptions that cause process crash.
2019-06-25 05:55:18 +02:00
{
Kernel: Enable SMAP protection on IRQ and exception entry It would be nice to do this in the assembly code, but we have to check if the feature is available before doing a CLAC, so I've put this in the C++ code for now.
2020-01-08 07:27:37 +01:00
clac();
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
handle_crash(*trap->regs, "General protection fault", SIGSEGV);
Turn the syscall interrupt into a trap (by switching the gate type.) This leaves interrupts enabled while we're in the kernel, which is precisely what we want. This uncovered a horrendous problem with kernel tasks silently overflowing their stacks. For now I've simply increased the stack size but I need a more MMU-y solution for this eventually.
2018-10-19 11:28:43 +02:00
}
Kernel: Implement lazy FPU state restore.
2019-01-25 07:52:44 +01:00
// 7: FPU not available exception
Kernel: Tidy up kernel entry points a little bit Now that we can see the kernel entry points all the time in profiles, let's tweak the names a little bit and switch to named exceptions.
2019-12-14 16:09:07 +01:00
EH_ENTRY_NO_CODE(7, fpu_exception);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
void fpu_exception_handler(TrapFrame*)
Snazz up the windows with some title bar gradients. :^)
2019-01-25 05:01:27 +01:00
{
Kernel: Switch to eagerly restoring x86 FPU state on context switch Lazy FPU restore is well known to be vulnerable to timing attacks, and eager restore is a lot simpler anyway, so let's just do it eagerly.
2020-01-01 16:49:08 +01:00
// Just clear the TS flag. We've already restored the FPU state eagerly.
// FIXME: It would be nice if we didn't have to do this at all.
Kernel: Implement lazy FPU state restore.
2019-01-25 07:52:44 +01:00
asm volatile("clts");
Snazz up the windows with some title bar gradients. :^)
2019-01-25 05:01:27 +01:00
}
Kernel: Add safe_memcpy, safe_memset and safe_strnlen These special functions can be used to safely copy/set memory or determine the length of a string, e.g. provided by user mode. In the event of a page fault, safe_memcpy/safe_memset will return false and safe_strnlen will return -1.
2020-09-10 19:49:31 -06:00
More paging stuff. The test userspace process now runs at linear address 0x300000 which is mapped to a dynamically allocated page from the MemoryManager. Cool!
2018-10-18 13:05:00 +02:00
// 14: Page Fault
Kernel: Tidy up kernel entry points a little bit Now that we can see the kernel entry points all the time in profiles, let's tweak the names a little bit and switch to named exceptions.
2019-12-14 16:09:07 +01:00
EH_ENTRY(14, page_fault);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
void page_fault_handler(TrapFrame* trap)
More paging stuff. The test userspace process now runs at linear address 0x300000 which is mapped to a dynamically allocated page from the MemoryManager. Cool!
2018-10-18 13:05:00 +02:00
{
Kernel: Enable SMAP protection on IRQ and exception entry It would be nice to do this in the assembly code, but we have to check if the feature is available before doing a CLAC, so I've put this in the C++ code for now.
2020-01-08 07:27:37 +01:00
clac();
Virtual consoles kinda work! We now make three VirtualConsoles at boot: tty0, tty1, and tty2. We launch an instance of /bin/sh in each one. You switch between them with Alt+1/2/3 How very very cool :^)
2018-10-30 15:33:37 +01:00
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
auto& regs = *trap->regs;
AK: Rename the common integer typedefs to make it obvious what they are. These types can be picked up by including <AK/Types.h>: * u8, u16, u32, u64 (unsigned) * i8, i16, i32, i64 (signed)
2019-07-03 21:17:35 +02:00
u32 fault_address;
Kernel: Run clang-format on everything.
2019-06-07 11:43:58 +02:00
asm("movl %%cr2, %%eax"
Kernel+LibC: Make page fault crashes a bit more readable. We'll now try to detect crashes that were due to dereferencing nullptr, uninitialized malloc() memory, or recently free()'d memory. It's not perfect but I think it's pretty good. :^) Also added some color to the most important parts of the crash log, and added some more modes to /bin/crash for exercising this code. Fixes #243.
2019-06-19 20:52:12 +02:00
: "=a"(fault_address));
More paging stuff. The test userspace process now runs at linear address 0x300000 which is mapped to a dynamically allocated page from the MemoryManager. Cool!
2018-10-18 13:05:00 +02:00
Everywhere: Debug macros instead of constexpr. This was done with the following script: find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/dbgln<debug_([a-z_]+)>/dbgln<\U\1_DEBUG>/' {} \; find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/if constexpr \(debug_([a-z0-9_]+)/if constexpr \(\U\1_DEBUG/' {} \;
2021-01-23 23:59:27 +01:00
if constexpr (PAGE_FAULT_DEBUG) {
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-13 21:48:51 +01:00
u32 fault_page_directory = read_cr3();
dbgln("CPU #{} ring {} {} page fault in PD={:#x}, {}{} {}",
Kernel: Make Processor::id a static function This eliminates the window between calling Processor::current and the member function where a thread could be moved to another processor. This is generally not as big of a concern as with Processor::current_thread, but also slightly more light weight.
2021-01-26 20:44:01 -07:00
Processor::is_initialized() ? Processor::id() : 0,
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-13 21:48:51 +01:00
regs.cs & 3,
regs.exception_code & 1 ? "PV" : "NP",
fault_page_directory,
regs.exception_code & 8 ? "reserved-bit " : "",
regs.exception_code & 2 ? "write" : "read",
VirtualAddress(fault_address));
More paging stuff. The test userspace process now runs at linear address 0x300000 which is mapped to a dynamically allocated page from the MemoryManager. Cool!
2018-10-18 13:05:00 +02:00
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-13 21:48:51 +01:00
dump(regs);
}
Kernel: Dump registers and code on ring0 page fault.
2019-01-25 00:32:44 +01:00
Kernel: Handle safe_memcpy/safe_memset/safe_strnlen faults in irq handlers Fix gracefully failing these calls if used within IRQ handlers. If we're handling IRQs, we need to handle these failures first, because we can't really resolve page faults in a meaningful way. But if we know that it was one of these functions that failed, then we can gracefully handle the situation. This solves a crash where the Scheduler attempts to produce backtraces in the timer irq, some of which cause faults. Fixes #3492
2020-09-14 13:10:49 -06:00
bool faulted_in_kernel = !(regs.cs & 3);
if (faulted_in_kernel && Processor::current().in_irq()) {
// If we're faulting in an IRQ handler, first check if we failed
// due to safe_memcpy, safe_strnlen, or safe_memset. If we did,
// gracefully continue immediately. Because we're in an IRQ handler
// we can't really try to resolve the page fault in a meaningful
// way, so we need to do this before calling into
// MemoryManager::handle_page_fault, which would just bail and
// request a crash
if (handle_safe_access_fault(regs, fault_address))
return;
}
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
auto current_thread = Thread::current();
Kernel: Don't let signals unblock threads while handling a page fault It was possible to signal a process while it was paging in an inode backed VM object. This would cause the inode read to EINTR, and the page fault handler would assert. Solve this by simply not unblocking threads due to signals if they are currently busy handling a page fault. This is probably not the best way to solve this issue, so I've added a FIXME to that effect.
2021-01-21 00:06:19 +01:00
if (current_thread)
current_thread->set_handling_page_fault(true);
ScopeGuard guard = [current_thread] {
if (current_thread)
current_thread->set_handling_page_fault(false);
};
Kernel: Handle safe_memcpy/safe_memset/safe_strnlen faults in irq handlers Fix gracefully failing these calls if used within IRQ handlers. If we're handling IRQs, we need to handle these failures first, because we can't really resolve page faults in a meaningful way. But if we know that it was one of these functions that failed, then we can gracefully handle the situation. This solves a crash where the Scheduler attempts to produce backtraces in the timer irq, some of which cause faults. Fixes #3492
2020-09-14 13:10:49 -06:00
if (!faulted_in_kernel && !MM.validate_user_stack(current_thread->process(), VirtualAddress(regs.userspace_esp))) {
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything:
2021-01-10 14:24:59 +01:00
dbgln("Invalid stack pointer: {}", VirtualAddress(regs.userspace_esp));
Kernel: Implement some basic stack pointer validation VM regions can now be marked as stack regions, which is then validated on syscall, and on page fault. If a thread is caught with its stack pointer pointing into anything that's *not* a Region with its stack bit set, we'll crash the whole process with SIGSTKFLT. Userspace must now allocate custom stacks by using mmap() with the new MAP_STACK flag. This mechanism was first introduced in OpenBSD, and now we have it too, yay! :^)
2019-11-17 12:11:43 +01:00
handle_crash(regs, "Bad stack on page fault", SIGSTKFLT);
ASSERT_NOT_REACHED();
}
Kernel+LibC: Make page fault crashes a bit more readable. We'll now try to detect crashes that were due to dereferencing nullptr, uninitialized malloc() memory, or recently free()'d memory. It's not perfect but I think it's pretty good. :^) Also added some color to the most important parts of the crash log, and added some more modes to /bin/crash for exercising this code. Fixes #243.
2019-06-19 20:52:12 +02:00
auto response = MM.handle_page_fault(PageFault(regs.exception_code, VirtualAddress(fault_address)));
More paging stuff. The test userspace process now runs at linear address 0x300000 which is mapped to a dynamically allocated page from the MemoryManager. Cool!
2018-10-18 13:05:00 +02:00
Kernel: Crash the current process on OOM (instead of panicking kernel) This patch adds PageFaultResponse::OutOfMemory which informs the fault handler that we were unable to allocate a necessary physical page and cannot continue. In response to this, the kernel will crash the current process. Because we are OOM, we can't symbolicate the crash like we normally would (since the ELF symbolication code needs to allocate), so we also communicate to Process::crash() that we're out of memory. Now we can survive "allocate 300 MB" (only the allocate process dies.) This is definitely not perfect and can easily end up killing a random innocent other process who happened to allocate one page at the wrong time, but it's a *lot* better than panicking on OOM. :^)
2020-05-06 21:11:38 +02:00
if (response == PageFaultResponse::ShouldCrash || response == PageFaultResponse::OutOfMemory) {
Kernel: Handle safe_memcpy/safe_memset/safe_strnlen faults in irq handlers Fix gracefully failing these calls if used within IRQ handlers. If we're handling IRQs, we need to handle these failures first, because we can't really resolve page faults in a meaningful way. But if we know that it was one of these functions that failed, then we can gracefully handle the situation. This solves a crash where the Scheduler attempts to produce backtraces in the timer irq, some of which cause faults. Fixes #3492
2020-09-14 13:10:49 -06:00
if (faulted_in_kernel && handle_safe_access_fault(regs, fault_address)) {
Kernel: Add safe_memcpy, safe_memset and safe_strnlen These special functions can be used to safely copy/set memory or determine the length of a string, e.g. provided by user mode. In the event of a page fault, safe_memcpy/safe_memset will return false and safe_strnlen will return -1.
2020-09-10 19:49:31 -06:00
// If this would be a ring0 (kernel) fault and the fault was triggered by
// safe_memcpy, safe_strnlen, or safe_memset then we resume execution at
// the appropriate _fault label rather than crashing
return;
}
Kernel: Crash the current process on OOM (instead of panicking kernel) This patch adds PageFaultResponse::OutOfMemory which informs the fault handler that we were unable to allocate a necessary physical page and cannot continue. In response to this, the kernel will crash the current process. Because we are OOM, we can't symbolicate the crash like we normally would (since the ELF symbolication code needs to allocate), so we also communicate to Process::crash() that we're out of memory. Now we can survive "allocate 300 MB" (only the allocate process dies.) This is definitely not perfect and can easily end up killing a random innocent other process who happened to allocate one page at the wrong time, but it's a *lot* better than panicking on OOM. :^)
2020-05-06 21:11:38 +02:00
if (response != PageFaultResponse::OutOfMemory) {
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
if (current_thread->has_signal_handler(SIGSEGV)) {
current_thread->send_urgent_signal_to_self(SIGSEGV);
Kernel: Crash the current process on OOM (instead of panicking kernel) This patch adds PageFaultResponse::OutOfMemory which informs the fault handler that we were unable to allocate a necessary physical page and cannot continue. In response to this, the kernel will crash the current process. Because we are OOM, we can't symbolicate the crash like we normally would (since the ELF symbolication code needs to allocate), so we also communicate to Process::crash() that we're out of memory. Now we can survive "allocate 300 MB" (only the allocate process dies.) This is definitely not perfect and can easily end up killing a random innocent other process who happened to allocate one page at the wrong time, but it's a *lot* better than panicking on OOM. :^)
2020-05-06 21:11:38 +02:00
return;
}
Kernel: Don't build with -mregparm=3 It was really confusing to have different calling conventions in kernel and userspace. Also this has prevented us from linking with libgcc.
2019-11-06 13:03:45 +01:00
}
Kernel: Send SIGSEGV on seg-fault Now programs can catch the SIGSEGV signal when they segfault. This commit also introduced the send_urgent_signal_to_self method, which is needed to send signals to a thread when handling exceptions caused by the same thread.
2019-10-07 22:22:50 +13:00
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything:
2021-01-10 14:24:59 +01:00
dbgln("Unrecoverable page fault, {}{}{} address {}",
regs.exception_code & PageFaultFlags::ReservedBitViolation ? "reserved bit violation / " : "",
regs.exception_code & PageFaultFlags::InstructionFetch ? "instruction fetch / " : "",
regs.exception_code & PageFaultFlags::Write ? "write to" : "read from",
VirtualAddress(fault_address));
AK: Rename the common integer typedefs to make it obvious what they are. These types can be picked up by including <AK/Types.h>: * u8, u16, u32, u64 (unsigned) * i8, i16, i32, i64 (signed)
2019-07-03 21:17:35 +02:00
u32 malloc_scrub_pattern = explode_byte(MALLOC_SCRUB_BYTE);
u32 free_scrub_pattern = explode_byte(FREE_SCRUB_BYTE);
Kernel: Add crash logging heuristic for uninitialized kmalloc()/kfree() Since we scrub both kmalloc() and kfree() with predictable values, we can log a helpful message when hitting a crash that looks like it might be a dereference of such scrubbed data.
2020-02-01 10:26:05 +01:00
u32 kmalloc_scrub_pattern = explode_byte(KMALLOC_SCRUB_BYTE);
u32 kfree_scrub_pattern = explode_byte(KFREE_SCRUB_BYTE);
Kernel: Add memory scrubbing in slab_alloc() and slab_dealloc() These now scrub allocated and freed memory like kmalloc()/kfree() was already doing.
2020-02-01 10:36:25 +01:00
u32 slab_alloc_scrub_pattern = explode_byte(SLAB_ALLOC_SCRUB_BYTE);
u32 slab_dealloc_scrub_pattern = explode_byte(SLAB_DEALLOC_SCRUB_BYTE);
Kernel+LibC: Make page fault crashes a bit more readable. We'll now try to detect crashes that were due to dereferencing nullptr, uninitialized malloc() memory, or recently free()'d memory. It's not perfect but I think it's pretty good. :^) Also added some color to the most important parts of the crash log, and added some more modes to /bin/crash for exercising this code. Fixes #243.
2019-06-19 20:52:12 +02:00
if ((fault_address & 0xffff0000) == (malloc_scrub_pattern & 0xffff0000)) {
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything:
2021-01-10 14:24:59 +01:00
dbgln("Note: Address {} looks like it may be uninitialized malloc() memory", VirtualAddress(fault_address));
Kernel+LibC: Make page fault crashes a bit more readable. We'll now try to detect crashes that were due to dereferencing nullptr, uninitialized malloc() memory, or recently free()'d memory. It's not perfect but I think it's pretty good. :^) Also added some color to the most important parts of the crash log, and added some more modes to /bin/crash for exercising this code. Fixes #243.
2019-06-19 20:52:12 +02:00
} else if ((fault_address & 0xffff0000) == (free_scrub_pattern & 0xffff0000)) {
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything:
2021-01-10 14:24:59 +01:00
dbgln("Note: Address {} looks like it may be recently free()'d memory", VirtualAddress(fault_address));
Kernel: Add crash logging heuristic for uninitialized kmalloc()/kfree() Since we scrub both kmalloc() and kfree() with predictable values, we can log a helpful message when hitting a crash that looks like it might be a dereference of such scrubbed data.
2020-02-01 10:26:05 +01:00
} else if ((fault_address & 0xffff0000) == (kmalloc_scrub_pattern & 0xffff0000)) {
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything:
2021-01-10 14:24:59 +01:00
dbgln("Note: Address {} looks like it may be uninitialized kmalloc() memory", VirtualAddress(fault_address));
Kernel: Add crash logging heuristic for uninitialized kmalloc()/kfree() Since we scrub both kmalloc() and kfree() with predictable values, we can log a helpful message when hitting a crash that looks like it might be a dereference of such scrubbed data.
2020-02-01 10:26:05 +01:00
} else if ((fault_address & 0xffff0000) == (kfree_scrub_pattern & 0xffff0000)) {
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything:
2021-01-10 14:24:59 +01:00
dbgln("Note: Address {} looks like it may be recently kfree()'d memory", VirtualAddress(fault_address));
Kernel: Add memory scrubbing in slab_alloc() and slab_dealloc() These now scrub allocated and freed memory like kmalloc()/kfree() was already doing.
2020-02-01 10:36:25 +01:00
} else if ((fault_address & 0xffff0000) == (slab_alloc_scrub_pattern & 0xffff0000)) {
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything:
2021-01-10 14:24:59 +01:00
dbgln("Note: Address {} looks like it may be uninitialized slab_alloc() memory", VirtualAddress(fault_address));
Kernel: Add memory scrubbing in slab_alloc() and slab_dealloc() These now scrub allocated and freed memory like kmalloc()/kfree() was already doing.
2020-02-01 10:36:25 +01:00
} else if ((fault_address & 0xffff0000) == (slab_dealloc_scrub_pattern & 0xffff0000)) {
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything:
2021-01-10 14:24:59 +01:00
dbgln("Note: Address {} looks like it may be recently slab_dealloc()'d memory", VirtualAddress(fault_address));
Kernel+LibC: Make page fault crashes a bit more readable. We'll now try to detect crashes that were due to dereferencing nullptr, uninitialized malloc() memory, or recently free()'d memory. It's not perfect but I think it's pretty good. :^) Also added some color to the most important parts of the crash log, and added some more modes to /bin/crash for exercising this code. Fixes #243.
2019-06-19 20:52:12 +02:00
} else if (fault_address < 4096) {
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything:
2021-01-10 14:24:59 +01:00
dbgln("Note: Address {} looks like a possible nullptr dereference", VirtualAddress(fault_address));
Kernel+LibC: Make page fault crashes a bit more readable. We'll now try to detect crashes that were due to dereferencing nullptr, uninitialized malloc() memory, or recently free()'d memory. It's not perfect but I think it's pretty good. :^) Also added some color to the most important parts of the crash log, and added some more modes to /bin/crash for exercising this code. Fixes #243.
2019-06-19 20:52:12 +02:00
}
Kernel: Crash the current process on OOM (instead of panicking kernel) This patch adds PageFaultResponse::OutOfMemory which informs the fault handler that we were unable to allocate a necessary physical page and cannot continue. In response to this, the kernel will crash the current process. Because we are OOM, we can't symbolicate the crash like we normally would (since the ELF symbolication code needs to allocate), so we also communicate to Process::crash() that we're out of memory. Now we can survive "allocate 300 MB" (only the allocate process dies.) This is definitely not perfect and can easily end up killing a random innocent other process who happened to allocate one page at the wrong time, but it's a *lot* better than panicking on OOM. :^)
2020-05-06 21:11:38 +02:00
handle_crash(regs, "Page Fault", SIGSEGV, response == PageFaultResponse::OutOfMemory);
More paging stuff. The test userspace process now runs at linear address 0x300000 which is mapped to a dynamically allocated page from the MemoryManager. Cool!
2018-10-18 13:05:00 +02:00
} else if (response == PageFaultResponse::Continue) {
Everywhere: Use CMake to generate AK/Debug.h. This was done with the help of several scripts, I dump them here to easily find them later: awk '/#ifdef/ { print "#cmakedefine01 "$2 }' AK/Debug.h.in for debug_macro in $(awk '/#ifdef/ { print $2 }' AK/Debug.h.in) do find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/#ifdef '$debug_macro'/#if '$debug_macro'/' {} \; done # Remember to remove WRAPPER_GERNERATOR_DEBUG from the list. awk '/#cmake/ { print "set("$2" ON)" }' AK/Debug.h.in
2021-01-23 23:29:11 +01:00
#if PAGE_FAULT_DEBUG
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything: The modifications in this commit were automatically made using the following command: find . -name '*.cpp' -exec sed -i -E 's/dbg\(\) << ("[^"{]*");/dbgln\(\1\);/' {} \;
2021-01-09 18:51:44 +01:00
dbgln("Continuing after resolved page fault");
Implement COW pages! :^) sys$fork() now clones all writable regions with per-page COW bits. The pages are then mapped read-only and we handle a PF by COWing the pages. This is quite delightful. Obviously there's lots of work to do still, and it needs better data structures, but the general concept works.
2018-11-05 13:48:07 +01:00
#endif
More paging stuff. The test userspace process now runs at linear address 0x300000 which is mapped to a dynamically allocated page from the MemoryManager. Cool!
2018-10-18 13:05:00 +02:00
} else {
ASSERT_NOT_REACHED();
}
}
CPU: Handle Debug exception We currently only care about debug exceptions that are triggered by the single-step execution mode. The debug exception is translated to a SIGTRAP, which can be caught and handled by the tracing thread.
2020-04-13 16:37:47 +03:00
EH_ENTRY_NO_CODE(1, debug);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
void debug_handler(TrapFrame* trap)
CPU: Handle Debug exception We currently only care about debug exceptions that are triggered by the single-step execution mode. The debug exception is translated to a SIGTRAP, which can be caught and handled by the tracing thread.
2020-04-13 16:37:47 +03:00
{
clac();
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
auto& regs = *trap->regs;
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
auto current_thread = Thread::current();
Kernel: Change wait blocking to Process-only blocking This prevents zombies created by multi-threaded applications and brings our model back to closer to what other OSs do. This also means that SIGSTOP needs to halt all threads, and SIGCONT needs to resume those threads.
2020-12-08 21:18:45 -07:00
auto& process = current_thread->process();
if ((regs.cs & 3) == 0) {
CPU: Handle Debug exception We currently only care about debug exceptions that are triggered by the single-step execution mode. The debug exception is translated to a SIGTRAP, which can be caught and handled by the tracing thread.
2020-04-13 16:37:47 +03:00
klog() << "Debug Exception in Ring0";
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
Processor::halt();
CPU: Handle Debug exception We currently only care about debug exceptions that are triggered by the single-step execution mode. The debug exception is translated to a SIGTRAP, which can be caught and handled by the tracing thread.
2020-04-13 16:37:47 +03:00
return;
}
constexpr u8 REASON_SINGLESTEP = 14;
bool is_reason_singlestep = (read_dr6() & (1 << REASON_SINGLESTEP));
if (!is_reason_singlestep)
return;
Kernel: Change wait blocking to Process-only blocking This prevents zombies created by multi-threaded applications and brings our model back to closer to what other OSs do. This also means that SIGSTOP needs to halt all threads, and SIGCONT needs to resume those threads.
2020-12-08 21:18:45 -07:00
if (auto tracer = process.tracer()) {
tracer->set_regs(regs);
CPU: Handle Debug exception We currently only care about debug exceptions that are triggered by the single-step execution mode. The debug exception is translated to a SIGTRAP, which can be caught and handled by the tracing thread.
2020-04-13 16:37:47 +03:00
}
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
current_thread->send_urgent_signal_to_self(SIGTRAP);
CPU: Handle Debug exception We currently only care about debug exceptions that are triggered by the single-step execution mode. The debug exception is translated to a SIGTRAP, which can be caught and handled by the tracing thread.
2020-04-13 16:37:47 +03:00
}
CPU: Handle breakpoint trap Also, start working on the debugger app.
2020-04-03 14:50:17 +03:00
EH_ENTRY_NO_CODE(3, breakpoint);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
void breakpoint_handler(TrapFrame* trap)
CPU: Handle breakpoint trap Also, start working on the debugger app.
2020-04-03 14:50:17 +03:00
{
clac();
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
auto& regs = *trap->regs;
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
auto current_thread = Thread::current();
Kernel: Change wait blocking to Process-only blocking This prevents zombies created by multi-threaded applications and brings our model back to closer to what other OSs do. This also means that SIGSTOP needs to halt all threads, and SIGCONT needs to resume those threads.
2020-12-08 21:18:45 -07:00
auto& process = current_thread->process();
if ((regs.cs & 3) == 0) {
CPU: Handle breakpoint trap Also, start working on the debugger app.
2020-04-03 14:50:17 +03:00
klog() << "Breakpoint Trap in Ring0";
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
Processor::halt();
CPU: Handle breakpoint trap Also, start working on the debugger app.
2020-04-03 14:50:17 +03:00
return;
}
Kernel: Change wait blocking to Process-only blocking This prevents zombies created by multi-threaded applications and brings our model back to closer to what other OSs do. This also means that SIGSTOP needs to halt all threads, and SIGCONT needs to resume those threads.
2020-12-08 21:18:45 -07:00
if (auto tracer = process.tracer()) {
tracer->set_regs(regs);
CPU: Handle breakpoint trap Also, start working on the debugger app.
2020-04-03 14:50:17 +03:00
}
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
current_thread->send_urgent_signal_to_self(SIGTRAP);
CPU: Handle breakpoint trap Also, start working on the debugger app.
2020-04-03 14:50:17 +03:00
}
Kernel: Use klog() instead of kprintf() Also, duplicate data in dbg() and klog() calls were removed. In addition, leakage of virtual address to kernel log is prevented. This is done by replacing kprintf() calls to dbg() calls with the leaked data instead. Also, other kprintf() calls were replaced with klog().
2020-03-01 21:45:39 +02:00
#define EH(i, msg) \
static void _exception##i() \
{ \
klog() << msg; \
u32 cr0, cr2, cr3, cr4; \
asm("movl %%cr0, %%eax" \
: "=a"(cr0)); \
asm("movl %%cr2, %%eax" \
: "=a"(cr2)); \
asm("movl %%cr3, %%eax" \
: "=a"(cr3)); \
asm("movl %%cr4, %%eax" \
: "=a"(cr4)); \
klog() << "CR0=" << String::format("%x", cr0) << " CR2=" << String::format("%x", cr2) << " CR3=" << String::format("%x", cr3) << " CR4=" << String::format("%x", cr4); \
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
Processor::halt(); \
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
}
EH(2, "Unknown error")
EH(4, "Overflow")
EH(5, "Bounds check")
EH(8, "Double fault")
EH(9, "Coprocessor segment overrun")
EH(10, "Invalid TSS")
EH(11, "Segment not present")
EH(12, "Stack exception")
EH(15, "Unknown error")
EH(16, "Coprocessor error")
Kernel: Detect APs and boot them into protected mode This isn't fully working, the APs pretend like they're fully initialized and are just halted permanently for now.
2020-06-04 09:10:16 -06:00
const DescriptorTablePointer& get_idtr()
{
return s_idtr;
}
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
static void unimp_trap()
{
Kernel: Use klog() instead of kprintf() Also, duplicate data in dbg() and klog() calls were removed. In addition, leakage of virtual address to kernel log is prevented. This is done by replacing kprintf() calls to dbg() calls with the leaked data instead. Also, other kprintf() calls were replaced with klog().
2020-03-01 21:45:39 +02:00
klog() << "Unhandled IRQ.";
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
Processor::Processor::halt();
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
}
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
GenericInterruptHandler& get_interrupt_handler(u8 interrupt_number)
Add IRQHandler class that can be subclasses to handle an IRQ. Also move Keyboard to a class implementation using this pattern.
2018-10-22 12:58:29 +02:00
{
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
ASSERT(s_interrupt_handler[interrupt_number] != nullptr);
return *s_interrupt_handler[interrupt_number];
Add IRQHandler class that can be subclasses to handle an IRQ. Also move Keyboard to a class implementation using this pattern.
2018-10-22 12:58:29 +02:00
}
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
static void revert_to_unused_handler(u8 interrupt_number)
Add IRQHandler class that can be subclasses to handle an IRQ. Also move Keyboard to a class implementation using this pattern.
2018-10-22 12:58:29 +02:00
{
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
new UnhandledInterruptHandler(interrupt_number);
}
void register_generic_interrupt_handler(u8 interrupt_number, GenericInterruptHandler& handler)
{
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
ASSERT(interrupt_number < GENERIC_INTERRUPT_HANDLERS_COUNT);
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
if (s_interrupt_handler[interrupt_number] != nullptr) {
Kernel: Change HandlerPurpose to HandlerType Also, GenericInterruptHandler class requires to implement two new methods.
2020-03-05 19:13:55 +02:00
if (s_interrupt_handler[interrupt_number]->type() == HandlerType::UnhandledInterruptHandler) {
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
s_interrupt_handler[interrupt_number] = &handler;
return;
}
if (s_interrupt_handler[interrupt_number]->is_shared_handler() && !s_interrupt_handler[interrupt_number]->is_sharing_with_others()) {
Kernel: Change HandlerPurpose to HandlerType Also, GenericInterruptHandler class requires to implement two new methods.
2020-03-05 19:13:55 +02:00
ASSERT(s_interrupt_handler[interrupt_number]->type() == HandlerType::SharedIRQHandler);
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
static_cast<SharedIRQHandler*>(s_interrupt_handler[interrupt_number])->register_handler(handler);
return;
Kernel: Change get_sharing_devices_count() in GenericInterruptHandler The new method' name is sharing_devices_count(). The Serenity Coding Style tends to not accept the word "get" in methods' names if possible.
2020-02-23 14:14:01 +02:00
}
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
if (!s_interrupt_handler[interrupt_number]->is_shared_handler()) {
Kernel: Allow to install a real IRQ handler on a spurious one IRQ 7 and 15 on the PIC architecture are used for spurious interrupts. IRQ 7 could also be used for LPT connection, and IRQ 15 can be used for the secondary IDE channel. Therefore, we need to allow to install a real IRQ handler and check if a real IRQ was asserted. If so, we handle them in the usual way. A note on this fix - unregistering or registering a new IRQ handler after we already registered one in the spurious interrupt handler is not supported yet.
2020-12-19 08:49:15 +02:00
if (s_interrupt_handler[interrupt_number]->type() == HandlerType::SpuriousInterruptHandler) {
static_cast<SpuriousInterruptHandler*>(s_interrupt_handler[interrupt_number])->register_handler(handler);
return;
}
Kernel: Change HandlerPurpose to HandlerType Also, GenericInterruptHandler class requires to implement two new methods.
2020-03-05 19:13:55 +02:00
ASSERT(s_interrupt_handler[interrupt_number]->type() == HandlerType::IRQHandler);
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
auto& previous_handler = *s_interrupt_handler[interrupt_number];
s_interrupt_handler[interrupt_number] = nullptr;
SharedIRQHandler::initialize(interrupt_number);
static_cast<SharedIRQHandler*>(s_interrupt_handler[interrupt_number])->register_handler(previous_handler);
static_cast<SharedIRQHandler*>(s_interrupt_handler[interrupt_number])->register_handler(handler);
return;
}
ASSERT_NOT_REACHED();
} else {
s_interrupt_handler[interrupt_number] = &handler;
}
}
void unregister_generic_interrupt_handler(u8 interrupt_number, GenericInterruptHandler& handler)
{
ASSERT(s_interrupt_handler[interrupt_number] != nullptr);
Kernel: Change HandlerPurpose to HandlerType Also, GenericInterruptHandler class requires to implement two new methods.
2020-03-05 19:13:55 +02:00
if (s_interrupt_handler[interrupt_number]->type() == HandlerType::UnhandledInterruptHandler) {
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything: The modifications in this commit were automatically made using the following command: find . -name '*.cpp' -exec sed -i -E 's/dbg\(\) << ("[^"{]*");/dbgln\(\1\);/' {} \;
2021-01-09 18:51:44 +01:00
dbgln("Trying to unregister unused handler (?)");
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
return;
}
if (s_interrupt_handler[interrupt_number]->is_shared_handler() && !s_interrupt_handler[interrupt_number]->is_sharing_with_others()) {
Kernel: Change HandlerPurpose to HandlerType Also, GenericInterruptHandler class requires to implement two new methods.
2020-03-05 19:13:55 +02:00
ASSERT(s_interrupt_handler[interrupt_number]->type() == HandlerType::SharedIRQHandler);
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
static_cast<SharedIRQHandler*>(s_interrupt_handler[interrupt_number])->unregister_handler(handler);
Kernel: Change get_sharing_devices_count() in GenericInterruptHandler The new method' name is sharing_devices_count(). The Serenity Coding Style tends to not accept the word "get" in methods' names if possible.
2020-02-23 14:14:01 +02:00
if (!static_cast<SharedIRQHandler*>(s_interrupt_handler[interrupt_number])->sharing_devices_count()) {
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
revert_to_unused_handler(interrupt_number);
}
return;
}
if (!s_interrupt_handler[interrupt_number]->is_shared_handler()) {
Kernel: Change HandlerPurpose to HandlerType Also, GenericInterruptHandler class requires to implement two new methods.
2020-03-05 19:13:55 +02:00
ASSERT(s_interrupt_handler[interrupt_number]->type() == HandlerType::IRQHandler);
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
revert_to_unused_handler(interrupt_number);
return;
}
ASSERT_NOT_REACHED();
Add IRQHandler class that can be subclasses to handle an IRQ. Also move Keyboard to a class implementation using this pattern.
2018-10-22 12:58:29 +02:00
}
AK: Rename the common integer typedefs to make it obvious what they are. These types can be picked up by including <AK/Types.h>: * u8, u16, u32, u64 (unsigned) * i8, i16, i32, i64 (signed)
2019-07-03 21:17:35 +02:00
void register_interrupt_handler(u8 index, void (*f)())
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
{
s_idt[index].low = 0x00080000 | LSW((f));
AK: Rename the common integer typedefs to make it obvious what they are. These types can be picked up by including <AK/Types.h>: * u8, u16, u32, u64 (unsigned) * i8, i16, i32, i64 (signed)
2019-07-03 21:17:35 +02:00
s_idt[index].high = ((u32)(f)&0xffff0000) | 0x8e00;
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
}
AK: Rename the common integer typedefs to make it obvious what they are. These types can be picked up by including <AK/Types.h>: * u8, u16, u32, u64 (unsigned) * i8, i16, i32, i64 (signed)
2019-07-03 21:17:35 +02:00
void register_user_callable_interrupt_handler(u8 index, void (*f)())
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
{
s_idt[index].low = 0x00080000 | LSW((f));
AK: Rename the common integer typedefs to make it obvious what they are. These types can be picked up by including <AK/Types.h>: * u8, u16, u32, u64 (unsigned) * i8, i16, i32, i64 (signed)
2019-07-03 21:17:35 +02:00
s_idt[index].high = ((u32)(f)&0xffff0000) | 0xef00;
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
}
More coding style changes.
2018-12-03 00:39:25 +01:00
void flush_idt()
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
{
Kernel: Run clang-format on everything.
2019-06-07 11:43:58 +02:00
asm("lidt %0" ::"m"(s_idtr));
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
}
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
static void idt_init()
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
{
s_idtr.address = s_idt;
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
s_idtr.limit = 256 * 8 - 1;
More coding style changes.
2018-12-03 00:39:25 +01:00
Kernel: Tidy up kernel entry points a little bit Now that we can see the kernel entry points all the time in profiles, let's tweak the names a little bit and switch to named exceptions.
2019-12-14 16:09:07 +01:00
register_interrupt_handler(0x00, divide_error_asm_entry);
CPU: Handle Debug exception We currently only care about debug exceptions that are triggered by the single-step execution mode. The debug exception is translated to a SIGTRAP, which can be caught and handled by the tracing thread.
2020-04-13 16:37:47 +03:00
register_user_callable_interrupt_handler(0x01, debug_asm_entry);
More coding style changes.
2018-12-03 00:39:25 +01:00
register_interrupt_handler(0x02, _exception2);
CPU: Handle breakpoint trap Also, start working on the debugger app.
2020-04-03 14:50:17 +03:00
register_user_callable_interrupt_handler(0x03, breakpoint_asm_entry);
More coding style changes.
2018-12-03 00:39:25 +01:00
register_interrupt_handler(0x04, _exception4);
register_interrupt_handler(0x05, _exception5);
Kernel: Tidy up kernel entry points a little bit Now that we can see the kernel entry points all the time in profiles, let's tweak the names a little bit and switch to named exceptions.
2019-12-14 16:09:07 +01:00
register_interrupt_handler(0x06, illegal_instruction_asm_entry);
register_interrupt_handler(0x07, fpu_exception_asm_entry);
More coding style changes.
2018-12-03 00:39:25 +01:00
register_interrupt_handler(0x08, _exception8);
register_interrupt_handler(0x09, _exception9);
register_interrupt_handler(0x0a, _exception10);
register_interrupt_handler(0x0b, _exception11);
register_interrupt_handler(0x0c, _exception12);
Kernel: Tidy up kernel entry points a little bit Now that we can see the kernel entry points all the time in profiles, let's tweak the names a little bit and switch to named exceptions.
2019-12-14 16:09:07 +01:00
register_interrupt_handler(0x0d, general_protection_fault_asm_entry);
register_interrupt_handler(0x0e, page_fault_asm_entry);
More coding style changes.
2018-12-03 00:39:25 +01:00
register_interrupt_handler(0x0f, _exception15);
register_interrupt_handler(0x10, _exception16);
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
for (u8 i = 0x11; i < 0x50; i++)
register_interrupt_handler(i, unimp_trap);
register_interrupt_handler(0x50, interrupt_80_asm_entry);
register_interrupt_handler(0x51, interrupt_81_asm_entry);
register_interrupt_handler(0x52, interrupt_82_asm_entry);
register_interrupt_handler(0x53, interrupt_83_asm_entry);
register_interrupt_handler(0x54, interrupt_84_asm_entry);
register_interrupt_handler(0x55, interrupt_85_asm_entry);
register_interrupt_handler(0x56, interrupt_86_asm_entry);
register_interrupt_handler(0x57, interrupt_87_asm_entry);
register_interrupt_handler(0x58, interrupt_88_asm_entry);
register_interrupt_handler(0x59, interrupt_89_asm_entry);
register_interrupt_handler(0x5a, interrupt_90_asm_entry);
register_interrupt_handler(0x5b, interrupt_91_asm_entry);
register_interrupt_handler(0x5c, interrupt_92_asm_entry);
register_interrupt_handler(0x5d, interrupt_93_asm_entry);
register_interrupt_handler(0x5e, interrupt_94_asm_entry);
register_interrupt_handler(0x5f, interrupt_95_asm_entry);
register_interrupt_handler(0x60, interrupt_96_asm_entry);
register_interrupt_handler(0x61, interrupt_97_asm_entry);
register_interrupt_handler(0x62, interrupt_98_asm_entry);
register_interrupt_handler(0x63, interrupt_99_asm_entry);
register_interrupt_handler(0x64, interrupt_100_asm_entry);
register_interrupt_handler(0x65, interrupt_101_asm_entry);
register_interrupt_handler(0x66, interrupt_102_asm_entry);
register_interrupt_handler(0x67, interrupt_103_asm_entry);
register_interrupt_handler(0x68, interrupt_104_asm_entry);
register_interrupt_handler(0x69, interrupt_105_asm_entry);
register_interrupt_handler(0x6a, interrupt_106_asm_entry);
register_interrupt_handler(0x6b, interrupt_107_asm_entry);
register_interrupt_handler(0x6c, interrupt_108_asm_entry);
register_interrupt_handler(0x6d, interrupt_109_asm_entry);
register_interrupt_handler(0x6e, interrupt_110_asm_entry);
register_interrupt_handler(0x6f, interrupt_111_asm_entry);
register_interrupt_handler(0x70, interrupt_112_asm_entry);
register_interrupt_handler(0x71, interrupt_113_asm_entry);
register_interrupt_handler(0x72, interrupt_114_asm_entry);
register_interrupt_handler(0x73, interrupt_115_asm_entry);
register_interrupt_handler(0x74, interrupt_116_asm_entry);
register_interrupt_handler(0x75, interrupt_117_asm_entry);
register_interrupt_handler(0x76, interrupt_118_asm_entry);
register_interrupt_handler(0x77, interrupt_119_asm_entry);
register_interrupt_handler(0x78, interrupt_120_asm_entry);
register_interrupt_handler(0x79, interrupt_121_asm_entry);
register_interrupt_handler(0x7a, interrupt_122_asm_entry);
register_interrupt_handler(0x7b, interrupt_123_asm_entry);
register_interrupt_handler(0x7c, interrupt_124_asm_entry);
register_interrupt_handler(0x7d, interrupt_125_asm_entry);
register_interrupt_handler(0x7e, interrupt_126_asm_entry);
register_interrupt_handler(0x7f, interrupt_127_asm_entry);
register_interrupt_handler(0x80, interrupt_128_asm_entry);
register_interrupt_handler(0x81, interrupt_129_asm_entry);
register_interrupt_handler(0x82, interrupt_130_asm_entry);
register_interrupt_handler(0x83, interrupt_131_asm_entry);
register_interrupt_handler(0x84, interrupt_132_asm_entry);
register_interrupt_handler(0x85, interrupt_133_asm_entry);
register_interrupt_handler(0x86, interrupt_134_asm_entry);
register_interrupt_handler(0x87, interrupt_135_asm_entry);
register_interrupt_handler(0x88, interrupt_136_asm_entry);
register_interrupt_handler(0x89, interrupt_137_asm_entry);
register_interrupt_handler(0x8a, interrupt_138_asm_entry);
register_interrupt_handler(0x8b, interrupt_139_asm_entry);
register_interrupt_handler(0x8c, interrupt_140_asm_entry);
register_interrupt_handler(0x8d, interrupt_141_asm_entry);
register_interrupt_handler(0x8e, interrupt_142_asm_entry);
register_interrupt_handler(0x8f, interrupt_143_asm_entry);
register_interrupt_handler(0x90, interrupt_144_asm_entry);
register_interrupt_handler(0x91, interrupt_145_asm_entry);
register_interrupt_handler(0x92, interrupt_146_asm_entry);
register_interrupt_handler(0x93, interrupt_147_asm_entry);
register_interrupt_handler(0x94, interrupt_148_asm_entry);
register_interrupt_handler(0x95, interrupt_149_asm_entry);
register_interrupt_handler(0x96, interrupt_150_asm_entry);
register_interrupt_handler(0x97, interrupt_151_asm_entry);
register_interrupt_handler(0x98, interrupt_152_asm_entry);
register_interrupt_handler(0x99, interrupt_153_asm_entry);
register_interrupt_handler(0x9a, interrupt_154_asm_entry);
register_interrupt_handler(0x9b, interrupt_155_asm_entry);
register_interrupt_handler(0x9c, interrupt_156_asm_entry);
register_interrupt_handler(0x9d, interrupt_157_asm_entry);
register_interrupt_handler(0x9e, interrupt_158_asm_entry);
register_interrupt_handler(0x9f, interrupt_159_asm_entry);
register_interrupt_handler(0xa0, interrupt_160_asm_entry);
register_interrupt_handler(0xa1, interrupt_161_asm_entry);
register_interrupt_handler(0xa2, interrupt_162_asm_entry);
register_interrupt_handler(0xa3, interrupt_163_asm_entry);
register_interrupt_handler(0xa4, interrupt_164_asm_entry);
register_interrupt_handler(0xa5, interrupt_165_asm_entry);
register_interrupt_handler(0xa6, interrupt_166_asm_entry);
register_interrupt_handler(0xa7, interrupt_167_asm_entry);
register_interrupt_handler(0xa8, interrupt_168_asm_entry);
register_interrupt_handler(0xa9, interrupt_169_asm_entry);
register_interrupt_handler(0xaa, interrupt_170_asm_entry);
register_interrupt_handler(0xab, interrupt_171_asm_entry);
register_interrupt_handler(0xac, interrupt_172_asm_entry);
register_interrupt_handler(0xad, interrupt_173_asm_entry);
register_interrupt_handler(0xae, interrupt_174_asm_entry);
register_interrupt_handler(0xaf, interrupt_175_asm_entry);
register_interrupt_handler(0xb0, interrupt_176_asm_entry);
register_interrupt_handler(0xb1, interrupt_177_asm_entry);
register_interrupt_handler(0xb2, interrupt_178_asm_entry);
register_interrupt_handler(0xb3, interrupt_179_asm_entry);
register_interrupt_handler(0xb4, interrupt_180_asm_entry);
register_interrupt_handler(0xb5, interrupt_181_asm_entry);
register_interrupt_handler(0xb6, interrupt_182_asm_entry);
register_interrupt_handler(0xb7, interrupt_183_asm_entry);
register_interrupt_handler(0xb8, interrupt_184_asm_entry);
register_interrupt_handler(0xb9, interrupt_185_asm_entry);
register_interrupt_handler(0xba, interrupt_186_asm_entry);
register_interrupt_handler(0xbb, interrupt_187_asm_entry);
register_interrupt_handler(0xbc, interrupt_188_asm_entry);
register_interrupt_handler(0xbd, interrupt_189_asm_entry);
register_interrupt_handler(0xbe, interrupt_190_asm_entry);
register_interrupt_handler(0xbf, interrupt_191_asm_entry);
register_interrupt_handler(0xc0, interrupt_192_asm_entry);
register_interrupt_handler(0xc1, interrupt_193_asm_entry);
register_interrupt_handler(0xc2, interrupt_194_asm_entry);
register_interrupt_handler(0xc3, interrupt_195_asm_entry);
register_interrupt_handler(0xc4, interrupt_196_asm_entry);
register_interrupt_handler(0xc5, interrupt_197_asm_entry);
register_interrupt_handler(0xc6, interrupt_198_asm_entry);
register_interrupt_handler(0xc7, interrupt_199_asm_entry);
register_interrupt_handler(0xc8, interrupt_200_asm_entry);
register_interrupt_handler(0xc9, interrupt_201_asm_entry);
register_interrupt_handler(0xca, interrupt_202_asm_entry);
register_interrupt_handler(0xcb, interrupt_203_asm_entry);
register_interrupt_handler(0xcc, interrupt_204_asm_entry);
register_interrupt_handler(0xcd, interrupt_205_asm_entry);
register_interrupt_handler(0xce, interrupt_206_asm_entry);
register_interrupt_handler(0xcf, interrupt_207_asm_entry);
register_interrupt_handler(0xd0, interrupt_208_asm_entry);
register_interrupt_handler(0xd1, interrupt_209_asm_entry);
register_interrupt_handler(0xd2, interrupt_210_asm_entry);
register_interrupt_handler(0xd3, interrupt_211_asm_entry);
register_interrupt_handler(0xd4, interrupt_212_asm_entry);
register_interrupt_handler(0xd5, interrupt_213_asm_entry);
register_interrupt_handler(0xd6, interrupt_214_asm_entry);
register_interrupt_handler(0xd7, interrupt_215_asm_entry);
register_interrupt_handler(0xd8, interrupt_216_asm_entry);
register_interrupt_handler(0xd9, interrupt_217_asm_entry);
register_interrupt_handler(0xda, interrupt_218_asm_entry);
register_interrupt_handler(0xdb, interrupt_219_asm_entry);
register_interrupt_handler(0xdc, interrupt_220_asm_entry);
register_interrupt_handler(0xdd, interrupt_221_asm_entry);
register_interrupt_handler(0xde, interrupt_222_asm_entry);
register_interrupt_handler(0xdf, interrupt_223_asm_entry);
register_interrupt_handler(0xe0, interrupt_224_asm_entry);
register_interrupt_handler(0xe1, interrupt_225_asm_entry);
register_interrupt_handler(0xe2, interrupt_226_asm_entry);
register_interrupt_handler(0xe3, interrupt_227_asm_entry);
register_interrupt_handler(0xe4, interrupt_228_asm_entry);
register_interrupt_handler(0xe5, interrupt_229_asm_entry);
register_interrupt_handler(0xe6, interrupt_230_asm_entry);
register_interrupt_handler(0xe7, interrupt_231_asm_entry);
register_interrupt_handler(0xe8, interrupt_232_asm_entry);
register_interrupt_handler(0xe9, interrupt_233_asm_entry);
register_interrupt_handler(0xea, interrupt_234_asm_entry);
register_interrupt_handler(0xeb, interrupt_235_asm_entry);
register_interrupt_handler(0xec, interrupt_236_asm_entry);
register_interrupt_handler(0xed, interrupt_237_asm_entry);
register_interrupt_handler(0xee, interrupt_238_asm_entry);
register_interrupt_handler(0xef, interrupt_239_asm_entry);
register_interrupt_handler(0xf0, interrupt_240_asm_entry);
register_interrupt_handler(0xf1, interrupt_241_asm_entry);
register_interrupt_handler(0xf2, interrupt_242_asm_entry);
register_interrupt_handler(0xf3, interrupt_243_asm_entry);
register_interrupt_handler(0xf4, interrupt_244_asm_entry);
register_interrupt_handler(0xf5, interrupt_245_asm_entry);
register_interrupt_handler(0xf6, interrupt_246_asm_entry);
register_interrupt_handler(0xf7, interrupt_247_asm_entry);
register_interrupt_handler(0xf8, interrupt_248_asm_entry);
register_interrupt_handler(0xf9, interrupt_249_asm_entry);
register_interrupt_handler(0xfa, interrupt_250_asm_entry);
register_interrupt_handler(0xfb, interrupt_251_asm_entry);
register_interrupt_handler(0xfc, interrupt_252_asm_entry);
register_interrupt_handler(0xfd, interrupt_253_asm_entry);
register_interrupt_handler(0xfe, interrupt_254_asm_entry);
register_interrupt_handler(0xff, interrupt_255_asm_entry);
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything: The modifications in this commit were automatically made using the following command: find . -name '*.cpp' -exec sed -i -E 's/dbg\(\) << ("[^"{]*");/dbgln\(\1\);/' {} \;
2021-01-09 18:51:44 +01:00
dbgln("Installing Unhandled Handlers");
CPU: Allow to use IRQs in range of 50 to 178
2020-03-06 03:20:51 +02:00
CPU: Use the new interrupt components Now we use the GenericInterruptHandler class instead of IRQHandler in the CPU functions. This commit adds an include to the ISR stub macros header file. Also, this commit adds support for IRQ sharing, so when an IRQHandler will try to register to already-assigned IRQ number, a SharedIRQHandler will be created to register both IRQHandlers.
2020-02-22 20:38:17 +02:00
for (u8 i = 0; i < GENERIC_INTERRUPT_HANDLERS_COUNT; ++i) {
new UnhandledInterruptHandler(i);
Add IRQHandler class that can be subclasses to handle an IRQ. Also move Keyboard to a class implementation using this pattern.
2018-10-22 12:58:29 +02:00
}
More coding style changes.
2018-12-03 00:39:25 +01:00
flush_idt();
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
}
AK: Rename the common integer typedefs to make it obvious what they are. These types can be picked up by including <AK/Types.h>: * u8, u16, u32, u64 (unsigned) * i8, i16, i32, i64 (signed)
2019-07-03 21:17:35 +02:00
void load_task_register(u16 selector)
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
{
Kernel: Run clang-format on everything.
2019-06-07 11:43:58 +02:00
asm("ltr %0" ::"r"(selector));
Import the "gerbert" kernel I worked on earlier this year. It's a lot crappier than I remembered it. It's gonna need a lot of work.
2018-10-16 11:01:38 +02:00
}
Add IRQHandler class that can be subclasses to handle an IRQ. Also move Keyboard to a class implementation using this pattern.
2018-10-22 12:58:29 +02:00
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
void handle_interrupt(TrapFrame* trap)
Add IRQHandler class that can be subclasses to handle an IRQ. Also move Keyboard to a class implementation using this pattern.
2018-10-22 12:58:29 +02:00
{
Kernel: Enable SMAP protection on IRQ and exception entry It would be nice to do this in the assembly code, but we have to check if the feature is available before doing a CLAC, so I've put this in the C++ code for now.
2020-01-08 07:27:37 +01:00
clac();
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
auto& regs = *trap->regs;
CPU: Allow to use IRQs in range of 50 to 178
2020-03-06 03:20:51 +02:00
ASSERT(regs.isr_number >= IRQ_VECTOR_BASE && regs.isr_number <= (IRQ_VECTOR_BASE + GENERIC_INTERRUPT_HANDLERS_COUNT));
Kernel: Make separate kernel entry points for each PIC IRQ Instead of having a common entry point and looking at the PIC ISR to figure out which IRQ we're servicing, just make a separate entryway for each IRQ that pushes the IRQ number and jumps to a common routine. This fixes a weird issue where incoming network packets would sometimes cause the mouse to stop working. I didn't track it down further than realizing we were sometimes EOI'ing the wrong IRQ.
2019-12-15 12:11:39 +01:00
u8 irq = (u8)(regs.isr_number - 0x50);
Kernel: Make use of interrupts as an entropy source Booting old computers without RDRAND/RDSEED and without a disk makes the system severely starved for entropy. Uses interrupts as a source to side-step that issue. Also warn whenever the system is starved of entropy, because that's a non-obvious failure mode.
2021-01-24 18:17:54 +01:00
s_entropy_source_interrupts.add_random_event(irq);
Kernel: Fix counting interrupts Move counting interrupts out of the handle_interrupt method so that it is done in all cases without the interrupt handler having to implement it explicitly. Also make the counter an atomic value as e.g. the LocalAPIC interrupts may be triggered on multiple processors simultaneously. Fixes #4297
2020-12-02 09:41:52 -07:00
auto* handler = s_interrupt_handler[irq];
ASSERT(handler);
handler->increment_invoking_counter();
handler->handle_interrupt(regs);
handler->eoi();
Add IRQHandler class that can be subclasses to handle an IRQ. Also move Keyboard to a class implementation using this pattern.
2018-10-22 12:58:29 +02:00
}
Move assertion failures out-of-line to reduce binary bloat.
2018-11-04 13:12:58 +01:00
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
void enter_trap_no_irq(TrapFrame* trap)
{
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
InterruptDisabler disable;
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
Processor::current().enter_trap(*trap, false);
}
void enter_trap(TrapFrame* trap)
{
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
InterruptDisabler disable;
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
Processor::current().enter_trap(*trap, true);
}
void exit_trap(TrapFrame* trap)
{
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
InterruptDisabler disable;
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
return Processor::current().exit_trap(*trap);
}
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
static void sse_init()
Kernel: Initialize the CPU to allow SSE on startup. I still need to add support for SSE to the context switching code, but now at least one process can use it.
2019-03-27 13:40:00 +01:00
{
asm volatile(
"mov %cr0, %eax\n"
"andl $0xfffffffb, %eax\n"
"orl $0x2, %eax\n"
"mov %eax, %cr0\n"
"mov %cr4, %eax\n"
"orl $0x600, %eax\n"
Kernel: Run clang-format on everything.
2019-06-07 11:43:58 +02:00
"mov %eax, %cr4\n");
Kernel: Initialize the CPU to allow SSE on startup. I still need to add support for SSE to the context switching code, but now at least one process can use it.
2019-03-27 13:40:00 +01:00
}
Kernel: Move CPU feature detection to Arch/x86/CPU.{cpp.h} We now refuse to boot on machines that don't support PAE since all of our paging code depends on it. Also let's only enable SSE and PGE support if the CPU advertises it.
2020-01-01 12:56:21 +01:00
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
u32 read_cr0()
{
u32 cr0;
asm("movl %%cr0, %%eax"
: "=a"(cr0));
return cr0;
}
u32 read_cr3()
{
u32 cr3;
asm("movl %%cr3, %%eax"
: "=a"(cr3));
return cr3;
}
void write_cr3(u32 cr3)
{
Kernel: Prevent sign bit extension when creating a PDPTE When doing the cast to u64 on the page directory physical address, the sign bit was being extended. This only beomes an issue when crossing the 2 GiB boundary. At >= 2 GiB, the physical address has the sign bit set. For example, 0x80000000. This set all the reserved bits in the PDPTE, causing a GPF when loading the PDPT pointer into CR3. The reserved bits are presumably there to stop you writing out a physical address that the CPU physically cannot handle, as the size of the reserved bits is determined by the physical address width of the CPU. This fixes this by casting to FlatPtr instead. I believe the sign extension only happens when casting to a bigger type. I'm also using FlatPtr because it's a pointer we're writing into the PDPTE. sizeof(FlatPtr) will always be the same size as sizeof(void*). This also now asserts that the physical address in the PDPTE is within the max physical address the CPU supports. This is better than getting a GPF, because CPU::handle_crash tries to do the same operation that caused the GPF in the first place. That would cause an infinite loop of GPFs until the stack was exhausted, causing a triple fault. As far as I know and tested, I believe we can now use the full 32-bit physical range without crashing. Fixes #4584. See that issue for the full debugging story.
2020-12-30 17:31:25 +00:00
// NOTE: If you're here from a GPF crash, it's very likely that a PDPT entry is incorrect, not this!
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
asm volatile("movl %%eax, %%cr3" ::"a"(cr3)
: "memory");
}
u32 read_cr4()
{
u32 cr4;
asm("movl %%cr4, %%eax"
: "=a"(cr4));
return cr4;
}
u32 read_dr6()
{
u32 dr6;
asm("movl %%dr6, %%eax"
: "=a"(dr6));
return dr6;
}
FPUState Processor::s_clean_fpu_state;
static Vector<Processor*>* s_processors;
static SpinLock s_processor_lock;
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
volatile u32 Processor::g_total_processors;
static volatile bool s_smp_enabled;
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
Vector<Processor*>& Processor::processors()
{
ASSERT(s_processors);
return *s_processors;
}
Kernel: Move CPU feature detection to Arch/x86/CPU.{cpp.h} We now refuse to boot on machines that don't support PAE since all of our paging code depends on it. Also let's only enable SSE and PGE support if the CPU advertises it.
2020-01-01 12:56:21 +01:00
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
Processor& Processor::by_id(u32 cpu)
Kernel: Move CPU feature detection to Arch/x86/CPU.{cpp.h} We now refuse to boot on machines that don't support PAE since all of our paging code depends on it. Also let's only enable SSE and PGE support if the CPU advertises it.
2020-01-01 12:56:21 +01:00
{
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
// s_processors does not need to be protected by a lock of any kind.
// It is populated early in the boot process, and the BSP is waiting
// for all APs to finish, after which this array never gets modified
// again, so it's safe to not protect access to it here
auto& procs = processors();
ASSERT(procs[cpu] != nullptr);
ASSERT(procs.size() > cpu);
return *procs[cpu];
}
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
[[noreturn]] static inline void halt_this()
{
for (;;) {
asm volatile("cli; hlt");
}
}
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
void Processor::cpu_detect()
{
// NOTE: This is called during Processor::early_initialize, we cannot
// safely log at this point because we don't have kmalloc
// initialized yet!
auto set_feature =
[&](CPUFeature f) {
m_features = static_cast<CPUFeature>(static_cast<u32>(m_features) | static_cast<u32>(f));
};
m_features = static_cast<CPUFeature>(0);
Kernel: Move CPU feature detection to Arch/x86/CPU.{cpp.h} We now refuse to boot on machines that don't support PAE since all of our paging code depends on it. Also let's only enable SSE and PGE support if the CPU advertises it.
2020-01-01 12:56:21 +01:00
CPUID processor_info(0x1);
Kernel: Fix Processor::features_string() stopping too early and detect more features The exit condition for the loop was sizeof(m_features) * 8, which was 32. Presumably this was supposed to mean 32 bits, but it actually made it stop as soon as it reached the 6th bit. Also add detection for more SIMD CPU features.
2020-08-31 14:27:38 +01:00
if (processor_info.edx() & (1 << 4))
set_feature(CPUFeature::TSC);
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (processor_info.edx() & (1 << 6))
set_feature(CPUFeature::PAE);
if (processor_info.edx() & (1 << 13))
set_feature(CPUFeature::PGE);
Kernel: Fix Processor::features_string() stopping too early and detect more features The exit condition for the loop was sizeof(m_features) * 8, which was 32. Presumably this was supposed to mean 32 bits, but it actually made it stop as soon as it reached the 6th bit. Also add detection for more SIMD CPU features.
2020-08-31 14:27:38 +01:00
if (processor_info.edx() & (1 << 23))
set_feature(CPUFeature::MMX);
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (processor_info.edx() & (1 << 25))
set_feature(CPUFeature::SSE);
Kernel: Fix Processor::features_string() stopping too early and detect more features The exit condition for the loop was sizeof(m_features) * 8, which was 32. Presumably this was supposed to mean 32 bits, but it actually made it stop as soon as it reached the 6th bit. Also add detection for more SIMD CPU features.
2020-08-31 14:27:38 +01:00
if (processor_info.edx() & (1 << 26))
set_feature(CPUFeature::SSE2);
if (processor_info.ecx() & (1 << 0))
set_feature(CPUFeature::SSE3);
if (processor_info.ecx() & (1 << 9))
set_feature(CPUFeature::SSSE3);
if (processor_info.ecx() & (1 << 19))
set_feature(CPUFeature::SSE4_1);
if (processor_info.ecx() & (1 << 20))
set_feature(CPUFeature::SSE4_2);
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (processor_info.ecx() & (1 << 30))
set_feature(CPUFeature::RDRAND);
Kernel: Detect syscall/sysenter support
2020-07-07 19:23:38 -06:00
if (processor_info.edx() & (1 << 11)) {
u32 stepping = processor_info.eax() & 0xf;
u32 model = (processor_info.eax() >> 4) & 0xf;
u32 family = (processor_info.eax() >> 8) & 0xf;
if (!(family == 6 && model < 3 && stepping < 3))
set_feature(CPUFeature::SEP);
Kernel: Add some CPU feature flags related to TSC In case we want to rely more on TSC in time keeping in the future, idk This adds: - RDTSCP, for when the RDTSCP instruction is available - CONSTANT_TSC, for when the TSC has a constant frequency, invariant under things like the CPU boosting its frequency. - NONSTOP_TSC, for when the TSC doesn't pause when the CPU enters sleep states. AMD cpus and newer intel cpus set the INVSTC bit (bit 8 in edx of extended cpuid 0x8000000008), which implies both CONSTANT_TSC and NONSTOP_TSC. Some older intel processors have CONSTANT_TSC but not NONSTOP_TSC; this is set based on cpu model checks. There isn't a ton of documentation on this, so this follows Linux terminology and http://blog.tinola.com/?e=54 CONSTANT_TSC: https://github.com/torvalds/linux/commit/39b3a7910556005a7a0d042ecb7ff98bfa84ea57 NONSTOP_TSC: https://github.com/torvalds/linux/commit/40fb17152c50a69dc304dd632131c2f41281ce44 qemu disables invtsc (bit 8 in edx of extended cpuid 0x8000000008) by default even if the host cpu supports it. It can be enabled by running with `SERENITY_QEMU_CPU=host,migratable=off` set.
2020-10-07 13:19:09 -04:00
if ((family == 6 && model >= 3) || (family == 0xf && model >= 0xe))
set_feature(CPUFeature::CONSTANT_TSC);
Kernel: Detect syscall/sysenter support
2020-07-07 19:23:38 -06:00
}
Kernel: Add some CPU feature flags related to TSC In case we want to rely more on TSC in time keeping in the future, idk This adds: - RDTSCP, for when the RDTSCP instruction is available - CONSTANT_TSC, for when the TSC has a constant frequency, invariant under things like the CPU boosting its frequency. - NONSTOP_TSC, for when the TSC doesn't pause when the CPU enters sleep states. AMD cpus and newer intel cpus set the INVSTC bit (bit 8 in edx of extended cpuid 0x8000000008), which implies both CONSTANT_TSC and NONSTOP_TSC. Some older intel processors have CONSTANT_TSC but not NONSTOP_TSC; this is set based on cpu model checks. There isn't a ton of documentation on this, so this follows Linux terminology and http://blog.tinola.com/?e=54 CONSTANT_TSC: https://github.com/torvalds/linux/commit/39b3a7910556005a7a0d042ecb7ff98bfa84ea57 NONSTOP_TSC: https://github.com/torvalds/linux/commit/40fb17152c50a69dc304dd632131c2f41281ce44 qemu disables invtsc (bit 8 in edx of extended cpuid 0x8000000008) by default even if the host cpu supports it. It can be enabled by running with `SERENITY_QEMU_CPU=host,migratable=off` set.
2020-10-07 13:19:09 -04:00
u32 max_extended_leaf = CPUID(0x80000000).eax();
ASSERT(max_extended_leaf >= 0x80000001);
Kernel: Move CPU feature detection to Arch/x86/CPU.{cpp.h} We now refuse to boot on machines that don't support PAE since all of our paging code depends on it. Also let's only enable SSE and PGE support if the CPU advertises it.
2020-01-01 12:56:21 +01:00
CPUID extended_processor_info(0x80000001);
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (extended_processor_info.edx() & (1 << 20))
set_feature(CPUFeature::NX);
Kernel: Add some CPU feature flags related to TSC In case we want to rely more on TSC in time keeping in the future, idk This adds: - RDTSCP, for when the RDTSCP instruction is available - CONSTANT_TSC, for when the TSC has a constant frequency, invariant under things like the CPU boosting its frequency. - NONSTOP_TSC, for when the TSC doesn't pause when the CPU enters sleep states. AMD cpus and newer intel cpus set the INVSTC bit (bit 8 in edx of extended cpuid 0x8000000008), which implies both CONSTANT_TSC and NONSTOP_TSC. Some older intel processors have CONSTANT_TSC but not NONSTOP_TSC; this is set based on cpu model checks. There isn't a ton of documentation on this, so this follows Linux terminology and http://blog.tinola.com/?e=54 CONSTANT_TSC: https://github.com/torvalds/linux/commit/39b3a7910556005a7a0d042ecb7ff98bfa84ea57 NONSTOP_TSC: https://github.com/torvalds/linux/commit/40fb17152c50a69dc304dd632131c2f41281ce44 qemu disables invtsc (bit 8 in edx of extended cpuid 0x8000000008) by default even if the host cpu supports it. It can be enabled by running with `SERENITY_QEMU_CPU=host,migratable=off` set.
2020-10-07 13:19:09 -04:00
if (extended_processor_info.edx() & (1 << 27))
set_feature(CPUFeature::RDTSCP);
Kernel: Detect syscall/sysenter support
2020-07-07 19:23:38 -06:00
if (extended_processor_info.edx() & (1 << 11)) {
// Only available in 64 bit mode
set_feature(CPUFeature::SYSCALL);
}
Kernel: Move CPU feature detection to Arch/x86/CPU.{cpp.h} We now refuse to boot on machines that don't support PAE since all of our paging code depends on it. Also let's only enable SSE and PGE support if the CPU advertises it.
2020-01-01 12:56:21 +01:00
Kernel: Add some CPU feature flags related to TSC In case we want to rely more on TSC in time keeping in the future, idk This adds: - RDTSCP, for when the RDTSCP instruction is available - CONSTANT_TSC, for when the TSC has a constant frequency, invariant under things like the CPU boosting its frequency. - NONSTOP_TSC, for when the TSC doesn't pause when the CPU enters sleep states. AMD cpus and newer intel cpus set the INVSTC bit (bit 8 in edx of extended cpuid 0x8000000008), which implies both CONSTANT_TSC and NONSTOP_TSC. Some older intel processors have CONSTANT_TSC but not NONSTOP_TSC; this is set based on cpu model checks. There isn't a ton of documentation on this, so this follows Linux terminology and http://blog.tinola.com/?e=54 CONSTANT_TSC: https://github.com/torvalds/linux/commit/39b3a7910556005a7a0d042ecb7ff98bfa84ea57 NONSTOP_TSC: https://github.com/torvalds/linux/commit/40fb17152c50a69dc304dd632131c2f41281ce44 qemu disables invtsc (bit 8 in edx of extended cpuid 0x8000000008) by default even if the host cpu supports it. It can be enabled by running with `SERENITY_QEMU_CPU=host,migratable=off` set.
2020-10-07 13:19:09 -04:00
if (max_extended_leaf >= 0x80000007) {
CPUID cpuid(0x80000007);
if (cpuid.edx() & (1 << 8)) {
set_feature(CPUFeature::CONSTANT_TSC);
set_feature(CPUFeature::NONSTOP_TSC);
}
}
Kernel: Prevent sign bit extension when creating a PDPTE When doing the cast to u64 on the page directory physical address, the sign bit was being extended. This only beomes an issue when crossing the 2 GiB boundary. At >= 2 GiB, the physical address has the sign bit set. For example, 0x80000000. This set all the reserved bits in the PDPTE, causing a GPF when loading the PDPT pointer into CR3. The reserved bits are presumably there to stop you writing out a physical address that the CPU physically cannot handle, as the size of the reserved bits is determined by the physical address width of the CPU. This fixes this by casting to FlatPtr instead. I believe the sign extension only happens when casting to a bigger type. I'm also using FlatPtr because it's a pointer we're writing into the PDPTE. sizeof(FlatPtr) will always be the same size as sizeof(void*). This also now asserts that the physical address in the PDPTE is within the max physical address the CPU supports. This is better than getting a GPF, because CPU::handle_crash tries to do the same operation that caused the GPF in the first place. That would cause an infinite loop of GPFs until the stack was exhausted, causing a triple fault. As far as I know and tested, I believe we can now use the full 32-bit physical range without crashing. Fixes #4584. See that issue for the full debugging story.
2020-12-30 17:31:25 +00:00
if (max_extended_leaf >= 0x80000008) {
// CPUID.80000008H:EAX[7:0] reports the physical-address width supported by the processor.
CPUID cpuid(0x80000008);
m_physical_address_bit_width = cpuid.eax() & 0xff;
} else {
// For processors that do not support CPUID function 80000008H, the width is generally 36 if CPUID.01H:EDX.PAE [bit 6] = 1 and 32 otherwise.
m_physical_address_bit_width = has_feature(CPUFeature::PAE) ? 36 : 32;
}
Kernel: Move CPU feature detection to Arch/x86/CPU.{cpp.h} We now refuse to boot on machines that don't support PAE since all of our paging code depends on it. Also let's only enable SSE and PGE support if the CPU advertises it.
2020-01-01 12:56:21 +01:00
CPUID extended_features(0x7);
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (extended_features.ebx() & (1 << 20))
set_feature(CPUFeature::SMAP);
if (extended_features.ebx() & (1 << 7))
set_feature(CPUFeature::SMEP);
if (extended_features.ecx() & (1 << 2))
set_feature(CPUFeature::UMIP);
if (extended_features.ebx() & (1 << 18))
set_feature(CPUFeature::RDSEED);
Kernel: Move CPU feature detection to Arch/x86/CPU.{cpp.h} We now refuse to boot on machines that don't support PAE since all of our paging code depends on it. Also let's only enable SSE and PGE support if the CPU advertises it.
2020-01-01 12:56:21 +01:00
}
Kernel: Start implementing x86 SMAP support Supervisor Mode Access Prevention (SMAP) is an x86 CPU feature that prevents the kernel from accessing userspace memory. With SMAP enabled, trying to read/write a userspace memory address while in the kernel will now generate a page fault. Since it's sometimes necessary to read/write userspace memory, there are two new instructions that quickly switch the protection on/off: STAC (disables protection) and CLAC (enables protection.) These are exposed in kernel code via the stac() and clac() helpers. There's also a SmapDisabler RAII object that can be used to ensure that you don't forget to re-enable protection before returning to userspace code. THis patch also adds copy_to_user(), copy_from_user() and memset_user() which are the "correct" way of doing things. These functions allow us to briefly disable protection for a specific purpose, and then turn it back on immediately after it's done. Going forward all kernel code should be moved to using these and all uses of SmapDisabler are to be considered FIXME's. Note that we're not realizing the full potential of this feature since I've used SmapDisabler quite liberally in this initial bring-up patch.
2020-01-05 18:00:15 +01:00
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
void Processor::cpu_setup()
Kernel: Clean up MemoryManager initialization a bit more Move the CPU feature enabling to functions in Arch/i386/CPU.cpp.
2020-01-17 23:56:13 +01:00
{
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
// NOTE: This is called during Processor::early_initialize, we cannot
// safely log at this point because we don't have kmalloc
// initialized yet!
cpu_detect();
Kernel: Move all CPU feature initialization into cpu_setup() ..and do it very very early in boot.
2020-01-18 10:11:29 +01:00
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (has_feature(CPUFeature::SSE))
Kernel: Move all CPU feature initialization into cpu_setup() ..and do it very very early in boot.
2020-01-18 10:11:29 +01:00
sse_init();
Kernel: Clean up MemoryManager initialization a bit more Move the CPU feature enabling to functions in Arch/i386/CPU.cpp.
2020-01-17 23:56:13 +01:00
asm volatile(
Kernel: Move all CPU feature initialization into cpu_setup() ..and do it very very early in boot.
2020-01-18 10:11:29 +01:00
"movl %%cr0, %%eax\n"
"orl $0x00010000, %%eax\n"
"movl %%eax, %%cr0\n" ::
: "%eax", "memory");
Kernel: Clean up MemoryManager initialization a bit more Move the CPU feature enabling to functions in Arch/i386/CPU.cpp.
2020-01-17 23:56:13 +01:00
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (has_feature(CPUFeature::PGE)) {
Kernel: Clean up MemoryManager initialization a bit more Move the CPU feature enabling to functions in Arch/i386/CPU.cpp.
2020-01-17 23:56:13 +01:00
// Turn on CR4.PGE so the CPU will respect the G bit in page tables.
asm volatile(
"mov %cr4, %eax\n"
"orl $0x80, %eax\n"
"mov %eax, %cr4\n");
}
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (has_feature(CPUFeature::NX)) {
Kernel: Move all CPU feature initialization into cpu_setup() ..and do it very very early in boot.
2020-01-18 10:11:29 +01:00
// Turn on IA32_EFER.NXE
asm volatile(
"movl $0xc0000080, %ecx\n"
"rdmsr\n"
"orl $0x800, %eax\n"
"wrmsr\n");
}
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (has_feature(CPUFeature::SMEP)) {
Kernel: Clean up MemoryManager initialization a bit more Move the CPU feature enabling to functions in Arch/i386/CPU.cpp.
2020-01-17 23:56:13 +01:00
// Turn on CR4.SMEP
asm volatile(
"mov %cr4, %eax\n"
"orl $0x100000, %eax\n"
"mov %eax, %cr4\n");
}
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (has_feature(CPUFeature::SMAP)) {
Kernel: Clean up MemoryManager initialization a bit more Move the CPU feature enabling to functions in Arch/i386/CPU.cpp.
2020-01-17 23:56:13 +01:00
// Turn on CR4.SMAP
asm volatile(
"mov %cr4, %eax\n"
"orl $0x200000, %eax\n"
"mov %eax, %cr4\n");
}
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (has_feature(CPUFeature::UMIP)) {
Kernel: Clean up MemoryManager initialization a bit more Move the CPU feature enabling to functions in Arch/i386/CPU.cpp.
2020-01-17 23:56:13 +01:00
asm volatile(
Kernel: Move all CPU feature initialization into cpu_setup() ..and do it very very early in boot.
2020-01-18 10:11:29 +01:00
"mov %cr4, %eax\n"
Kernel: Clean up MemoryManager initialization a bit more Move the CPU feature enabling to functions in Arch/i386/CPU.cpp.
2020-01-17 23:56:13 +01:00
"orl $0x800, %eax\n"
Kernel: Move all CPU feature initialization into cpu_setup() ..and do it very very early in boot.
2020-01-18 10:11:29 +01:00
"mov %eax, %cr4\n");
Kernel: Clean up MemoryManager initialization a bit more Move the CPU feature enabling to functions in Arch/i386/CPU.cpp.
2020-01-17 23:56:13 +01:00
}
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
if (has_feature(CPUFeature::TSC)) {
Kernel: Move all CPU feature initialization into cpu_setup() ..and do it very very early in boot.
2020-01-18 10:11:29 +01:00
asm volatile(
"mov %cr4, %eax\n"
"orl $0x4, %eax\n"
"mov %eax, %cr4\n");
}
Kernel: Detect APs and boot them into protected mode This isn't fully working, the APs pretend like they're fully initialized and are just halted permanently for now.
2020-06-04 09:10:16 -06:00
}
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
String Processor::features_string() const
{
StringBuilder builder;
auto feature_to_str =
[](CPUFeature f) -> const char*
{
switch (f) {
case CPUFeature::NX:
return "nx";
case CPUFeature::PAE:
return "pae";
case CPUFeature::PGE:
return "pge";
case CPUFeature::RDRAND:
return "rdrand";
case CPUFeature::RDSEED:
return "rdseed";
case CPUFeature::SMAP:
return "smap";
case CPUFeature::SMEP:
return "smep";
case CPUFeature::SSE:
return "sse";
case CPUFeature::TSC:
return "tsc";
Kernel: Add some CPU feature flags related to TSC In case we want to rely more on TSC in time keeping in the future, idk This adds: - RDTSCP, for when the RDTSCP instruction is available - CONSTANT_TSC, for when the TSC has a constant frequency, invariant under things like the CPU boosting its frequency. - NONSTOP_TSC, for when the TSC doesn't pause when the CPU enters sleep states. AMD cpus and newer intel cpus set the INVSTC bit (bit 8 in edx of extended cpuid 0x8000000008), which implies both CONSTANT_TSC and NONSTOP_TSC. Some older intel processors have CONSTANT_TSC but not NONSTOP_TSC; this is set based on cpu model checks. There isn't a ton of documentation on this, so this follows Linux terminology and http://blog.tinola.com/?e=54 CONSTANT_TSC: https://github.com/torvalds/linux/commit/39b3a7910556005a7a0d042ecb7ff98bfa84ea57 NONSTOP_TSC: https://github.com/torvalds/linux/commit/40fb17152c50a69dc304dd632131c2f41281ce44 qemu disables invtsc (bit 8 in edx of extended cpuid 0x8000000008) by default even if the host cpu supports it. It can be enabled by running with `SERENITY_QEMU_CPU=host,migratable=off` set.
2020-10-07 13:19:09 -04:00
case CPUFeature::RDTSCP:
return "rdtscp";
case CPUFeature::CONSTANT_TSC:
return "constant_tsc";
case CPUFeature::NONSTOP_TSC:
return "nonstop_tsc";
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
case CPUFeature::UMIP:
return "umip";
Kernel: Detect syscall/sysenter support
2020-07-07 19:23:38 -06:00
case CPUFeature::SEP:
return "sep";
case CPUFeature::SYSCALL:
return "syscall";
Kernel: Fix Processor::features_string() stopping too early and detect more features The exit condition for the loop was sizeof(m_features) * 8, which was 32. Presumably this was supposed to mean 32 bits, but it actually made it stop as soon as it reached the 6th bit. Also add detection for more SIMD CPU features.
2020-08-31 14:27:38 +01:00
case CPUFeature::MMX:
return "mmx";
case CPUFeature::SSE2:
return "sse2";
case CPUFeature::SSE3:
return "sse3";
case CPUFeature::SSSE3:
return "ssse3";
case CPUFeature::SSE4_1:
return "sse4.1";
case CPUFeature::SSE4_2:
return "sse4.2";
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
// no default statement here intentionally so that we get
// a warning if a new feature is forgotten to be added here
}
// Shouldn't ever happen
return "???";
};
bool first = true;
Kernel: Fix Processor::features_string() stopping too early and detect more features The exit condition for the loop was sizeof(m_features) * 8, which was 32. Presumably this was supposed to mean 32 bits, but it actually made it stop as soon as it reached the 6th bit. Also add detection for more SIMD CPU features.
2020-08-31 14:27:38 +01:00
for (u32 flag = 1; flag != 0; flag <<= 1) {
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
if ((static_cast<u32>(m_features) & flag) != 0) {
if (first)
first = false;
else
builder.append(' ');
auto str = feature_to_str(static_cast<CPUFeature>(flag));
builder.append(str, strlen(str));
}
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
}
return builder.build();
Kernel: List all CPUs in /proc/cpuinfo
2020-06-27 17:06:33 -06:00
}
Kernel: Split initialization of Processor structure We need to very early on initialize the Processor structure so that we can use RecursiveSpinLock early on.
2020-07-02 08:34:00 -06:00
void Processor::early_initialize(u32 cpu)
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
{
m_self = this;
m_cpu = cpu;
m_in_irq = 0;
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
m_in_critical = 0;
m_invoke_scheduler_async = false;
m_scheduler_initialized = false;
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
m_message_queue = nullptr;
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
m_idle_thread = nullptr;
m_current_thread = nullptr;
Kernel: Prevent recursive calls into the scheduler Upon leaving a critical section (such as a SpinLock) we need to check if we're already asynchronously invoking the Scheduler. Otherwise we might end up triggering another context switch as soon as leaving the scheduler lock. Fixes #2883
2020-08-01 14:37:40 -06:00
m_scheduler_data = nullptr;
Kernel: Add a quickmap region for each processor Threads need to be able to concurrently quickmap things.
2020-06-28 16:04:35 -06:00
m_mm_data = nullptr;
Kernel: Split initialization of Processor structure We need to very early on initialize the Processor structure so that we can use RecursiveSpinLock early on.
2020-07-02 08:34:00 -06:00
m_info = nullptr;
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
m_halt_requested = false;
if (cpu == 0) {
s_smp_enabled = false;
atomic_store(&g_total_processors, 1u, AK::MemoryOrder::memory_order_release);
} else {
atomic_fetch_add(&g_total_processors, 1u, AK::MemoryOrder::memory_order_acq_rel);
}
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
Kernel: Add mechanism to queue deferred function calls Function calls that are deferred will be executed before a thread enters a pre-emptable state (meaning it is not in a critical section and it is not in an irq handler). If it is not already in such a state, it will be called immediately. This is meant to be used from e.g. IRQ handlers where we might want to block a thread until an interrupt happens.
2020-11-01 13:08:25 -07:00
deferred_call_pool_init();
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
cpu_setup();
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
gdt_init();
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
ASSERT(is_initialized()); // sanity check
Kernel: Split initialization of Processor structure We need to very early on initialize the Processor structure so that we can use RecursiveSpinLock early on.
2020-07-02 08:34:00 -06:00
ASSERT(&current() == this); // sanity check
}
void Processor::initialize(u32 cpu)
{
ASSERT(m_self == this);
ASSERT(&current() == this); // sanity check
Kernel: Consolidate features into CPUFeature enum This allows us to consolidate printing out all the CPU features into one log statement. Also expose them in /proc/cpuinfo
2020-07-03 10:23:09 -06:00
klog() << "CPU[" << id() << "]: Supported features: " << features_string();
if (!has_feature(CPUFeature::RDRAND))
klog() << "CPU[" << id() << "]: No RDRAND support detected, randomness will be poor";
Kernel: Prevent sign bit extension when creating a PDPTE When doing the cast to u64 on the page directory physical address, the sign bit was being extended. This only beomes an issue when crossing the 2 GiB boundary. At >= 2 GiB, the physical address has the sign bit set. For example, 0x80000000. This set all the reserved bits in the PDPTE, causing a GPF when loading the PDPT pointer into CR3. The reserved bits are presumably there to stop you writing out a physical address that the CPU physically cannot handle, as the size of the reserved bits is determined by the physical address width of the CPU. This fixes this by casting to FlatPtr instead. I believe the sign extension only happens when casting to a bigger type. I'm also using FlatPtr because it's a pointer we're writing into the PDPTE. sizeof(FlatPtr) will always be the same size as sizeof(void*). This also now asserts that the physical address in the PDPTE is within the max physical address the CPU supports. This is better than getting a GPF, because CPU::handle_crash tries to do the same operation that caused the GPF in the first place. That would cause an infinite loop of GPFs until the stack was exhausted, causing a triple fault. As far as I know and tested, I believe we can now use the full 32-bit physical range without crashing. Fixes #4584. See that issue for the full debugging story.
2020-12-30 17:31:25 +00:00
klog() << "CPU[" << id() << "]: Physical address bit width: " << m_physical_address_bit_width;
Kernel: Split initialization of Processor structure We need to very early on initialize the Processor structure so that we can use RecursiveSpinLock early on.
2020-07-02 08:34:00 -06:00
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
if (cpu == 0)
idt_init();
else
flush_idt();
if (cpu == 0) {
ASSERT((FlatPtr(&s_clean_fpu_state) & 0xF) == 0);
asm volatile("fninit");
asm volatile("fxsave %0"
: "=m"(s_clean_fpu_state));
}
Kernel: List all CPUs in /proc/cpuinfo
2020-06-27 17:06:33 -06:00
m_info = new ProcessorInfo(*this);
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
{
ScopedSpinLock lock(s_processor_lock);
// We need to prevent races between APs starting up at the same time
if (!s_processors)
s_processors = new Vector<Processor*>();
if (cpu >= s_processors->size())
s_processors->resize(cpu + 1);
(*s_processors)[cpu] = this;
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
}
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
}
void Processor::write_raw_gdt_entry(u16 selector, u32 low, u32 high)
{
u16 i = (selector & 0xfffc) >> 3;
u32 prev_gdt_length = m_gdt_length;
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
if (i > m_gdt_length) {
m_gdt_length = i + 1;
ASSERT(m_gdt_length <= sizeof(m_gdt) / sizeof(m_gdt[0]));
m_gdtr.limit = (m_gdt_length + 1) * 8 - 1;
}
m_gdt[i].low = low;
m_gdt[i].high = high;
// clear selectors we may have skipped
while (i < prev_gdt_length) {
m_gdt[i].low = 0;
m_gdt[i].high = 0;
i++;
}
}
void Processor::write_gdt_entry(u16 selector, Descriptor& descriptor)
{
write_raw_gdt_entry(selector, descriptor.low, descriptor.high);
}
Descriptor& Processor::get_gdt_entry(u16 selector)
{
u16 i = (selector & 0xfffc) >> 3;
return *(Descriptor*)(&m_gdt[i]);
}
void Processor::flush_gdt()
{
m_gdtr.address = m_gdt;
m_gdtr.limit = (m_gdt_length * 8) - 1;
asm volatile("lgdt %0" ::"m"(m_gdtr)
: "memory");
}
const DescriptorTablePointer& Processor::get_gdtr()
{
return m_gdtr;
}
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
Vector<FlatPtr> Processor::capture_stack_trace(Thread& thread, size_t max_frames)
Kernel: Fix retreiving frame pointer from a thread If we're trying to walk the stack for another thread, we can no longer retreive the EBP register from Thread::m_tss. Instead, we need to look at the top of the kernel stack, because all threads not currently running were last in kernel mode. Context switches now always trigger a brief switch to kernel mode, and Thread::m_tss only is used to save ESP and EIP. Fixes #2678
2020-07-03 12:12:34 -06:00
{
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
FlatPtr frame_ptr = 0, eip = 0;
Vector<FlatPtr, 32> stack_trace;
auto walk_stack = [&](FlatPtr stack_ptr)
{
Kernel: Limit the size of stack traces Let's not allow infinitely long stack traces. Cap it at 4096 frames.
2021-02-02 16:05:18 +01:00
static constexpr size_t max_stack_frames = 4096;
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
stack_trace.append(eip);
size_t count = 1;
Kernel: Limit the size of stack traces Let's not allow infinitely long stack traces. Cap it at 4096 frames.
2021-02-02 16:05:18 +01:00
while (stack_ptr && stack_trace.size() < max_stack_frames) {
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
FlatPtr retaddr;
count++;
if (max_frames != 0 && count > max_frames)
break;
if (is_user_range(VirtualAddress(stack_ptr), sizeof(FlatPtr) * 2)) {
if (!copy_from_user(&retaddr, &((FlatPtr*)stack_ptr)[1]) || !retaddr)
break;
stack_trace.append(retaddr);
if (!copy_from_user(&stack_ptr, (FlatPtr*)stack_ptr))
break;
} else {
void* fault_at;
if (!safe_memcpy(&retaddr, &((FlatPtr*)stack_ptr)[1], sizeof(FlatPtr), fault_at) || !retaddr)
break;
stack_trace.append(retaddr);
if (!safe_memcpy(&stack_ptr, (FlatPtr*)stack_ptr, sizeof(FlatPtr), fault_at))
break;
}
}
};
auto capture_current_thread = [&]()
{
frame_ptr = (FlatPtr)__builtin_frame_address(0);
eip = (FlatPtr)__builtin_return_address(0);
walk_stack(frame_ptr);
};
// Since the thread may be running on another processor, there
// is a chance a context switch may happen while we're trying
// to get it. It also won't be entirely accurate and merely
// reflect the status at the last context switch.
ScopedSpinLock lock(g_scheduler_lock);
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
if (&thread == Processor::current_thread()) {
Kernel: Fix retreiving frame pointer from a thread If we're trying to walk the stack for another thread, we can no longer retreive the EBP register from Thread::m_tss. Instead, we need to look at the top of the kernel stack, because all threads not currently running were last in kernel mode. Context switches now always trigger a brief switch to kernel mode, and Thread::m_tss only is used to save ESP and EIP. Fixes #2678
2020-07-03 12:12:34 -06:00
ASSERT(thread.state() == Thread::Running);
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
// Leave the scheduler lock. If we trigger page faults we may
// need to be preempted. Since this is our own thread it won't
// cause any problems as the stack won't change below this frame.
lock.unlock();
capture_current_thread();
} else if (thread.is_active()) {
Kernel: Make Processor::id a static function This eliminates the window between calling Processor::current and the member function where a thread could be moved to another processor. This is generally not as big of a concern as with Processor::current_thread, but also slightly more light weight.
2021-01-26 20:44:01 -07:00
ASSERT(thread.cpu() != Processor::id());
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
// If this is the case, the thread is currently running
// on another processor. We can't trust the kernel stack as
// it may be changing at any time. We need to probably send
// an IPI to that processor, have it walk the stack and wait
// until it returns the data back to us
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
auto& proc = Processor::current();
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
smp_unicast(thread.cpu(),
[&]() {
Kernel: Make Processor::id a static function This eliminates the window between calling Processor::current and the member function where a thread could be moved to another processor. This is generally not as big of a concern as with Processor::current_thread, but also slightly more light weight.
2021-01-26 20:44:01 -07:00
dbgln("CPU[{}] getting stack for cpu #{}", Processor::id(), proc.get_id());
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
ProcessPagingScope paging_scope(thread.process());
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
ASSERT(&Processor::current() != &proc);
ASSERT(&thread == Processor::current_thread());
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
// NOTE: Because the other processor is still holding the
// scheduler lock while waiting for this callback to finish,
// the current thread on the target processor cannot change
// TODO: What to do about page faults here? We might deadlock
// because the other processor is still holding the
// scheduler lock...
capture_current_thread();
}, false);
Kernel: Fix retreiving frame pointer from a thread If we're trying to walk the stack for another thread, we can no longer retreive the EBP register from Thread::m_tss. Instead, we need to look at the top of the kernel stack, because all threads not currently running were last in kernel mode. Context switches now always trigger a brief switch to kernel mode, and Thread::m_tss only is used to save ESP and EIP. Fixes #2678
2020-07-03 12:12:34 -06:00
} else {
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
switch (thread.state()) {
case Thread::Running:
ASSERT_NOT_REACHED(); // should have been handled above
case Thread::Runnable:
case Thread::Stopped:
case Thread::Blocked:
case Thread::Dying:
case Thread::Dead: {
Kernel: Fix retreiving frame pointer from a thread If we're trying to walk the stack for another thread, we can no longer retreive the EBP register from Thread::m_tss. Instead, we need to look at the top of the kernel stack, because all threads not currently running were last in kernel mode. Context switches now always trigger a brief switch to kernel mode, and Thread::m_tss only is used to save ESP and EIP. Fixes #2678
2020-07-03 12:12:34 -06:00
// We need to retrieve ebp from what was last pushed to the kernel
// stack. Before switching out of that thread, it switch_context
// pushed the callee-saved registers, and the last of them happens
// to be ebp.
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
ProcessPagingScope paging_scope(thread.process());
Kernel: Fix retreiving frame pointer from a thread If we're trying to walk the stack for another thread, we can no longer retreive the EBP register from Thread::m_tss. Instead, we need to look at the top of the kernel stack, because all threads not currently running were last in kernel mode. Context switches now always trigger a brief switch to kernel mode, and Thread::m_tss only is used to save ESP and EIP. Fixes #2678
2020-07-03 12:12:34 -06:00
auto& tss = thread.tss();
u32* stack_top = reinterpret_cast<u32*>(tss.esp);
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
if (is_user_range(VirtualAddress(stack_top), sizeof(FlatPtr))) {
if (!copy_from_user(&frame_ptr, &((FlatPtr*)stack_top)[0]))
frame_ptr = 0;
} else {
void* fault_at;
if (!safe_memcpy(&frame_ptr, &((FlatPtr*)stack_top)[0], sizeof(FlatPtr), fault_at))
frame_ptr = 0;
}
Kernel: Fix retreiving frame pointer from a thread If we're trying to walk the stack for another thread, we can no longer retreive the EBP register from Thread::m_tss. Instead, we need to look at the top of the kernel stack, because all threads not currently running were last in kernel mode. Context switches now always trigger a brief switch to kernel mode, and Thread::m_tss only is used to save ESP and EIP. Fixes #2678
2020-07-03 12:12:34 -06:00
eip = tss.eip;
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
// TODO: We need to leave the scheduler lock here, but we also
// need to prevent the target thread from being run while
// we walk the stack
lock.unlock();
walk_stack(frame_ptr);
break;
}
default:
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.Everything:
2021-01-10 14:24:59 +01:00
dbgln("Cannot capture stack trace for thread {} in state {}", thread, thread.state_string());
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
break;
Kernel: Fix retreiving frame pointer from a thread If we're trying to walk the stack for another thread, we can no longer retreive the EBP register from Thread::m_tss. Instead, we need to look at the top of the kernel stack, because all threads not currently running were last in kernel mode. Context switches now always trigger a brief switch to kernel mode, and Thread::m_tss only is used to save ESP and EIP. Fixes #2678
2020-07-03 12:12:34 -06:00
}
}
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
return stack_trace;
Kernel: Fix retreiving frame pointer from a thread If we're trying to walk the stack for another thread, we can no longer retreive the EBP register from Thread::m_tss. Instead, we need to look at the top of the kernel stack, because all threads not currently running were last in kernel mode. Context switches now always trigger a brief switch to kernel mode, and Thread::m_tss only is used to save ESP and EIP. Fixes #2678
2020-07-03 12:12:34 -06:00
}
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
extern "C" void enter_thread_context(Thread* from_thread, Thread* to_thread)
{
ASSERT(from_thread == to_thread || from_thread->state() != Thread::Running);
ASSERT(to_thread->state() == Thread::Running);
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
Processor::set_current_thread(*to_thread);
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
auto& from_tss = from_thread->tss();
auto& to_tss = to_thread->tss();
asm volatile("fxsave %0"
: "=m"(from_thread->fpu_state()));
from_tss.fs = get_fs();
from_tss.gs = get_gs();
set_fs(to_tss.fs);
set_gs(to_tss.gs);
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
auto& processor = Processor::current();
Kernel/LibCore: Expose processor id where a thread last ran
2020-06-27 22:36:15 -06:00
auto& tls_descriptor = processor.get_gdt_entry(GDT_SELECTOR_TLS);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
tls_descriptor.set_base(to_thread->thread_specific_data().as_ptr());
tls_descriptor.set_limit(to_thread->thread_specific_region_size());
if (from_tss.cr3 != to_tss.cr3)
write_cr3(to_tss.cr3);
Kernel: Make Processor::id a static function This eliminates the window between calling Processor::current and the member function where a thread could be moved to another processor. This is generally not as big of a concern as with Processor::current_thread, but also slightly more light weight.
2021-01-26 20:44:01 -07:00
to_thread->set_cpu(processor.get_id());
Kernel: Make entering and leaving critical sections atomic We also need to store m_in_critical in the Thread upon switching, and we need to restore it. This solves a problem where threads moving between different processors could end up with an unexpected value.
2021-01-23 22:24:10 -07:00
processor.restore_in_critical(to_thread->saved_critical());
Kernel/LibCore: Expose processor id where a thread last ran
2020-06-27 22:36:15 -06:00
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
asm volatile("fxrstor %0"
::"m"(to_thread->fpu_state()));
// TODO: debug registers
// TODO: ioperm?
}
#define ENTER_THREAD_CONTEXT_ARGS_SIZE (2 * 4) // to_thread, from_thread
Kernel: Various context switch fixes These changes solve a number of problems with the software context swithcing: * The scheduler lock really should be held throughout context switches * Transitioning from the initial (idle) thread to another needs to hold the scheduler lock * Transitioning from a dying thread to another also needs to hold the scheduler lock * Dying threads cannot necessarily be finalized if they haven't switched out of it yet, so flag them as active while a processor is running it (the Running state may be switched to Dying while it still is actually running)
2020-07-05 14:32:07 -06:00
void Processor::switch_context(Thread*& from_thread, Thread*& to_thread)
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
{
ASSERT(!in_irq());
Kernel: Various context switch fixes These changes solve a number of problems with the software context swithcing: * The scheduler lock really should be held throughout context switches * Transitioning from the initial (idle) thread to another needs to hold the scheduler lock * Transitioning from a dying thread to another also needs to hold the scheduler lock * Dying threads cannot necessarily be finalized if they haven't switched out of it yet, so flag them as active while a processor is running it (the Running state may be switched to Dying while it still is actually running)
2020-07-05 14:32:07 -06:00
ASSERT(m_in_critical == 1);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
ASSERT(is_kernel_mode());
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-13 21:48:51 +01:00
Everywhere: Debug macros instead of constexpr. This was done with the following script: find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/dbgln<debug_([a-z_]+)>/dbgln<\U\1_DEBUG>/' {} \; find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/if constexpr \(debug_([a-z0-9_]+)/if constexpr \(\U\1_DEBUG/' {} \;
2021-01-23 23:59:27 +01:00
dbgln<CONTEXT_SWITCH_DEBUG>("switch_context --> switching out of: {} {}", VirtualAddress(from_thread), *from_thread);
Kernel: Make entering and leaving critical sections atomic We also need to store m_in_critical in the Thread upon switching, and we need to restore it. This solves a problem where threads moving between different processors could end up with an unexpected value.
2021-01-23 22:24:10 -07:00
from_thread->save_critical(m_in_critical);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
// Switch to new thread context, passing from_thread and to_thread
// through to the new context using registers edx and eax
asm volatile(
// NOTE: changing how much we push to the stack affects
// SWITCH_CONTEXT_TO_STACK_SIZE and thread_context_first_enter()!
"pushfl \n"
"pushl %%ebx \n"
"pushl %%esi \n"
"pushl %%edi \n"
"pushl %%ebp \n"
"movl %%esp, %[from_esp] \n"
"movl $1f, %[from_eip] \n"
"movl %[to_esp0], %%ebx \n"
"movl %%ebx, %[tss_esp0] \n"
"movl %[to_esp], %%esp \n"
"pushl %[to_thread] \n"
"pushl %[from_thread] \n"
"pushl %[to_eip] \n"
"cld \n"
"jmp enter_thread_context \n"
"1: \n"
"popl %%edx \n"
"popl %%eax \n"
"popl %%ebp \n"
"popl %%edi \n"
"popl %%esi \n"
"popl %%ebx \n"
"popfl \n"
: [from_esp] "=m" (from_thread->tss().esp),
[from_eip] "=m" (from_thread->tss().eip),
[tss_esp0] "=m" (m_tss.esp0),
"=d" (from_thread), // needed so that from_thread retains the correct value
"=a" (to_thread) // needed so that to_thread retains the correct value
: [to_esp] "g" (to_thread->tss().esp),
[to_esp0] "g" (to_thread->tss().esp0),
[to_eip] "c" (to_thread->tss().eip),
[from_thread] "d" (from_thread),
[to_thread] "a" (to_thread)
Kernel: Make entering and leaving critical sections atomic We also need to store m_in_critical in the Thread upon switching, and we need to restore it. This solves a problem where threads moving between different processors could end up with an unexpected value.
2021-01-23 22:24:10 -07:00
: "memory"
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
);
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-13 21:48:51 +01:00
Everywhere: Debug macros instead of constexpr. This was done with the following script: find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/dbgln<debug_([a-z_]+)>/dbgln<\U\1_DEBUG>/' {} \; find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/if constexpr \(debug_([a-z0-9_]+)/if constexpr \(\U\1_DEBUG/' {} \;
2021-01-23 23:59:27 +01:00
dbgln<CONTEXT_SWITCH_DEBUG>("switch_context <-- from {} {} to {} {}", VirtualAddress(from_thread), *from_thread, VirtualAddress(to_thread), *to_thread);
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
Processor::current().restore_in_critical(to_thread->saved_critical());
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
}
Everywhere: Switch from (void) to [[maybe_unused]] (#4473) Problem: - `(void)` simply casts the expression to void. This is understood to indicate that it is ignored, but this is really a compiler trick to get the compiler to not generate a warning. Solution: - Use the `[[maybe_unused]]` attribute to indicate the value is unused. Note: - Functions taking a `(void)` argument list have also been changed to `()` because this is not needed and shows up in the same grep command.
2020-12-20 16:09:48 -07:00
extern "C" void context_first_init([[maybe_unused]] Thread* from_thread, [[maybe_unused]] Thread* to_thread, [[maybe_unused]] TrapFrame* trap)
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
{
ASSERT(!are_interrupts_enabled());
ASSERT(is_kernel_mode());
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
Everywhere: Debug macros instead of constexpr. This was done with the following script: find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/dbgln<debug_([a-z_]+)>/dbgln<\U\1_DEBUG>/' {} \; find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/if constexpr \(debug_([a-z0-9_]+)/if constexpr \(\U\1_DEBUG/' {} \;
2021-01-23 23:59:27 +01:00
dbgln<CONTEXT_SWITCH_DEBUG>("switch_context <-- from {} {} to {} {} (context_first_init)", VirtualAddress(from_thread), *from_thread, VirtualAddress(to_thread), *to_thread);
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
Kernel: Various context switch fixes These changes solve a number of problems with the software context swithcing: * The scheduler lock really should be held throughout context switches * Transitioning from the initial (idle) thread to another needs to hold the scheduler lock * Transitioning from a dying thread to another also needs to hold the scheduler lock * Dying threads cannot necessarily be finalized if they haven't switched out of it yet, so flag them as active while a processor is running it (the Running state may be switched to Dying while it still is actually running)
2020-07-05 14:32:07 -06:00
ASSERT(to_thread == Thread::current());
Kernel: Fix some issues related to fixes and block conditions Fix some problems with join blocks where the joining thread block condition was added twice, which lead to a crash when trying to unblock that condition a second time. Deferred block condition evaluation by File objects were also not properly keeping the File object alive, which lead to some random crashes and corruption problems. Other problems were caused by the fact that the Queued state didn't handle signals/interruptions consistently. To solve these issues we remove this state entirely, along with Thread::wait_on and change the WaitQueue into a BlockCondition instead. Also, deliver signals even if there isn't going to be a context switch to another thread. Fixes #4336 and #4330
2020-12-07 21:29:41 -07:00
Scheduler::enter_current(*from_thread, true);
Kernel: Various context switch fixes These changes solve a number of problems with the software context swithcing: * The scheduler lock really should be held throughout context switches * Transitioning from the initial (idle) thread to another needs to hold the scheduler lock * Transitioning from a dying thread to another also needs to hold the scheduler lock * Dying threads cannot necessarily be finalized if they haven't switched out of it yet, so flag them as active while a processor is running it (the Running state may be switched to Dying while it still is actually running)
2020-07-05 14:32:07 -06:00
// Since we got here and don't have Scheduler::context_switch in the
// call stack (because this is the first time we switched into this
Kernel: Prevent recursive calls into the scheduler Upon leaving a critical section (such as a SpinLock) we need to check if we're already asynchronously invoking the Scheduler. Otherwise we might end up triggering another context switch as soon as leaving the scheduler lock. Fixes #2883
2020-08-01 14:37:40 -06:00
// context), we need to notify the scheduler so that it can release
Kernel: Fix possible context switch within first context switch of a thread We were enabling interrupts too early, before the first context switch to a thread was complete. This could then trigger another context switch within the context switch, which lead to a crash.
2021-01-25 13:17:43 -07:00
// the scheduler lock. We don't want to enable interrupts at this point
// as we're still in the middle of a context switch. Doing so could
// trigger a context switch within a context switch, leading to a crash.
Scheduler::leave_on_first_switch(trap->regs->eflags & ~0x200);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
}
Everywhere: void arguments to C functions Problem: - C functions with no arguments require a single `void` in the argument list. Solution: - Put the `void` in the argument list of functions in C header files.
2020-12-24 08:41:54 -07:00
extern "C" void thread_context_first_enter(void);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
asm(
// enter_thread_context returns to here first time a thread is executing
".globl thread_context_first_enter \n"
"thread_context_first_enter: \n"
// switch_context will have pushed from_thread and to_thread to our new
// stack prior to thread_context_first_enter() being called, and the
// pointer to TrapFrame was the top of the stack before that
" movl 8(%esp), %ebx \n" // save pointer to TrapFrame
" cld \n"
" call context_first_init \n"
" addl $" __STRINGIFY(ENTER_THREAD_CONTEXT_ARGS_SIZE) ", %esp \n"
" movl %ebx, 0(%esp) \n" // push pointer to TrapFrame
" jmp common_trap_exit \n"
);
Everywhere: void arguments to C functions Problem: - C functions with no arguments require a single `void` in the argument list. Solution: - Put the `void` in the argument list of functions in C header files.
2020-12-24 08:41:54 -07:00
void exit_kernel_thread(void)
Kernel: Allow passing a thread argument for new kernel threads This adds the ability to pass a pointer to kernel thread/process. Also add the ability to use a closure as thread function, which allows passing information to a kernel thread more easily.
2020-11-16 20:51:34 -07:00
{
Thread::current()->exit();
}
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
u32 Processor::init_context(Thread& thread, bool leave_crit)
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
{
ASSERT(is_kernel_mode());
Kernel: Various context switch fixes These changes solve a number of problems with the software context swithcing: * The scheduler lock really should be held throughout context switches * Transitioning from the initial (idle) thread to another needs to hold the scheduler lock * Transitioning from a dying thread to another also needs to hold the scheduler lock * Dying threads cannot necessarily be finalized if they haven't switched out of it yet, so flag them as active while a processor is running it (the Running state may be switched to Dying while it still is actually running)
2020-07-05 14:32:07 -06:00
ASSERT(g_scheduler_lock.is_locked());
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
if (leave_crit) {
Kernel: Various context switch fixes These changes solve a number of problems with the software context swithcing: * The scheduler lock really should be held throughout context switches * Transitioning from the initial (idle) thread to another needs to hold the scheduler lock * Transitioning from a dying thread to another also needs to hold the scheduler lock * Dying threads cannot necessarily be finalized if they haven't switched out of it yet, so flag them as active while a processor is running it (the Running state may be switched to Dying while it still is actually running)
2020-07-05 14:32:07 -06:00
// Leave the critical section we set up in in Process::exec,
// but because we still have the scheduler lock we should end up with 1
m_in_critical--; // leave it without triggering anything or restoring flags
ASSERT(in_critical() == 1);
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
}
Kernel: Allow passing a thread argument for new kernel threads This adds the ability to pass a pointer to kernel thread/process. Also add the ability to use a closure as thread function, which allows passing information to a kernel thread more easily.
2020-11-16 20:51:34 -07:00
u32 kernel_stack_top = thread.kernel_stack_top();
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
u32 stack_top = kernel_stack_top;
// TODO: handle NT?
ASSERT((cpu_flags() & 0x24000) == 0); // Assume !(NT | VM)
auto& tss = thread.tss();
bool return_to_user = (tss.cs & 3) != 0;
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
// make room for an interrupt frame
if (!return_to_user) {
// userspace_esp and userspace_ss are not popped off by iret
// unless we're switching back to user mode
stack_top -= sizeof(RegisterState) - 2 * sizeof(u32);
Kernel: Allow passing a thread argument for new kernel threads This adds the ability to pass a pointer to kernel thread/process. Also add the ability to use a closure as thread function, which allows passing information to a kernel thread more easily.
2020-11-16 20:51:34 -07:00
// For kernel threads we'll push the thread function argument
// which should be in tss.esp and exit_kernel_thread as return
// address.
stack_top -= 2 * sizeof(u32);
*reinterpret_cast<u32*>(kernel_stack_top - 2 * sizeof(u32)) = tss.esp;
*reinterpret_cast<u32*>(kernel_stack_top - 3 * sizeof(u32)) = FlatPtr(&exit_kernel_thread);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
} else {
stack_top -= sizeof(RegisterState);
}
// we want to end up 16-byte aligned, %esp + 4 should be aligned
stack_top -= sizeof(u32);
Kernel: Allow passing a thread argument for new kernel threads This adds the ability to pass a pointer to kernel thread/process. Also add the ability to use a closure as thread function, which allows passing information to a kernel thread more easily.
2020-11-16 20:51:34 -07:00
*reinterpret_cast<u32*>(kernel_stack_top - sizeof(u32)) = 0;
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
// set up the stack so that after returning from thread_context_first_enter()
// we will end up either in kernel mode or user mode, depending on how the thread is set up
// However, the first step is to always start in kernel mode with thread_context_first_enter
RegisterState& iretframe = *reinterpret_cast<RegisterState*>(stack_top);
iretframe.ss = tss.ss;
iretframe.gs = tss.gs;
iretframe.fs = tss.fs;
iretframe.es = tss.es;
iretframe.ds = tss.ds;
iretframe.edi = tss.edi;
iretframe.esi = tss.esi;
iretframe.ebp = tss.ebp;
iretframe.esp = 0;
iretframe.ebx = tss.ebx;
iretframe.edx = tss.edx;
iretframe.ecx = tss.ecx;
iretframe.eax = tss.eax;
iretframe.eflags = tss.eflags;
iretframe.eip = tss.eip;
iretframe.cs = tss.cs;
if (return_to_user) {
iretframe.userspace_esp = tss.esp;
iretframe.userspace_ss = tss.ss;
}
// make space for a trap frame
stack_top -= sizeof(TrapFrame);
TrapFrame& trap = *reinterpret_cast<TrapFrame*>(stack_top);
trap.regs = &iretframe;
trap.prev_irq_level = 0;
Kernel: Track previous mode when entering/exiting traps This allows us to determine what the previous mode (user or kernel) was, e.g. in the timer interrupt. This is used e.g. to determine whether a signal handler should be set up. Fixes #5096
2021-01-25 13:19:34 -07:00
trap.next_trap = nullptr;
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
stack_top -= sizeof(u32); // pointer to TrapFrame
*reinterpret_cast<u32*>(stack_top) = stack_top + 4;
Everywhere: Debug macros instead of constexpr. This was done with the following script: find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/dbgln<debug_([a-z_]+)>/dbgln<\U\1_DEBUG>/' {} \; find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/if constexpr \(debug_([a-z0-9_]+)/if constexpr \(\U\1_DEBUG/' {} \;
2021-01-23 23:59:27 +01:00
if constexpr (CONTEXT_SWITCH_DEBUG) {
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-13 21:48:51 +01:00
if (return_to_user) {
dbgln("init_context {} ({}) set up to execute at eip={}:{}, esp={}, stack_top={}, user_top={}:{}",
thread,
VirtualAddress(&thread),
iretframe.cs, tss.eip,
VirtualAddress(tss.esp),
VirtualAddress(stack_top),
iretframe.userspace_ss,
iretframe.userspace_esp);
} else {
dbgln("init_context {} ({}) set up to execute at eip={}:{}, esp={}, stack_top={}",
thread,
VirtualAddress(&thread),
iretframe.cs, tss.eip,
VirtualAddress(tss.esp),
VirtualAddress(stack_top));
}
}
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
// make switch_context() always first return to thread_context_first_enter()
// in kernel mode, so set up these values so that we end up popping iretframe
// off the stack right after the context switch completed, at which point
// control is transferred to what iretframe is pointing to.
tss.eip = FlatPtr(&thread_context_first_enter);
tss.esp0 = kernel_stack_top;
tss.esp = stack_top;
tss.cs = GDT_SELECTOR_CODE0;
tss.ds = GDT_SELECTOR_DATA0;
tss.es = GDT_SELECTOR_DATA0;
tss.gs = GDT_SELECTOR_DATA0;
tss.ss = GDT_SELECTOR_DATA0;
tss.fs = GDT_SELECTOR_PROC;
return stack_top;
}
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
extern "C" u32 do_init_context(Thread* thread, u32 flags)
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
{
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
ASSERT_INTERRUPTS_DISABLED();
thread->tss().eflags = flags;
return Processor::current().init_context(*thread, true);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
}
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
extern "C" void do_assume_context(Thread* thread, u32 flags);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
asm(
".global do_assume_context \n"
"do_assume_context: \n"
" movl 4(%esp), %ebx \n"
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
" movl 8(%esp), %esi \n"
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
// We're going to call Processor::init_context, so just make sure
// we have enough stack space so we don't stomp over it
" subl $(" __STRINGIFY(4 + REGISTER_STATE_SIZE + TRAP_FRAME_SIZE + 4) "), %esp \n"
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
" pushl %esi \n"
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
" pushl %ebx \n"
" cld \n"
" call do_init_context \n"
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
" addl $8, %esp \n"
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
" movl %eax, %esp \n" // move stack pointer to what Processor::init_context set up for us
" pushl %ebx \n" // push to_thread
" pushl %ebx \n" // push from_thread
" pushl $thread_context_first_enter \n" // should be same as tss.eip
" jmp enter_thread_context \n"
);
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
void Processor::assume_context(Thread& thread, u32 flags)
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
{
Everywhere: Debug macros instead of constexpr. This was done with the following script: find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/dbgln<debug_([a-z_]+)>/dbgln<\U\1_DEBUG>/' {} \; find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/if constexpr \(debug_([a-z0-9_]+)/if constexpr \(\U\1_DEBUG/' {} \;
2021-01-23 23:59:27 +01:00
dbgln<CONTEXT_SWITCH_DEBUG>("Assume context for thread {} {}", VirtualAddress(&thread), thread);
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-13 21:48:51 +01:00
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
ASSERT_INTERRUPTS_DISABLED();
Kernel: Prevent recursive calls into the scheduler Upon leaving a critical section (such as a SpinLock) we need to check if we're already asynchronously invoking the Scheduler. Otherwise we might end up triggering another context switch as soon as leaving the scheduler lock. Fixes #2883
2020-08-01 14:37:40 -06:00
Scheduler::prepare_after_exec();
Kernel: Various context switch fixes These changes solve a number of problems with the software context swithcing: * The scheduler lock really should be held throughout context switches * Transitioning from the initial (idle) thread to another needs to hold the scheduler lock * Transitioning from a dying thread to another also needs to hold the scheduler lock * Dying threads cannot necessarily be finalized if they haven't switched out of it yet, so flag them as active while a processor is running it (the Running state may be switched to Dying while it still is actually running)
2020-07-05 14:32:07 -06:00
// in_critical() should be 2 here. The critical section in Process::exec
// and then the scheduler lock
ASSERT(Processor::current().in_critical() == 2);
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
do_assume_context(&thread, flags);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
ASSERT_NOT_REACHED();
}
Everywhere: void arguments to C functions Problem: - C functions with no arguments require a single `void` in the argument list. Solution: - Put the `void` in the argument list of functions in C header files.
2020-12-24 08:41:54 -07:00
extern "C" void pre_init_finished(void)
Kernel: Fix some flaws that caused crashes or hangs during boot We need to halt the BSP briefly until all APs are ready for the first context switch, but we can't hold the same spinlock by all of them while doing so. So, while the APs are waiting on each other they need to release the scheduler lock, and then once signaled re-acquire it. Should solve some timing dependent hangs or crashes, most easily observed using qemu with kvm disabled.
2020-07-08 15:20:36 -06:00
{
ASSERT(g_scheduler_lock.own_lock());
Kernel: Prevent recursive calls into the scheduler Upon leaving a critical section (such as a SpinLock) we need to check if we're already asynchronously invoking the Scheduler. Otherwise we might end up triggering another context switch as soon as leaving the scheduler lock. Fixes #2883
2020-08-01 14:37:40 -06:00
// Because init_finished() will wait on the other APs, we need
Kernel: Fix some flaws that caused crashes or hangs during boot We need to halt the BSP briefly until all APs are ready for the first context switch, but we can't hold the same spinlock by all of them while doing so. So, while the APs are waiting on each other they need to release the scheduler lock, and then once signaled re-acquire it. Should solve some timing dependent hangs or crashes, most easily observed using qemu with kvm disabled.
2020-07-08 15:20:36 -06:00
// to release the scheduler lock so that the other APs can also get
// to this point
Kernel: Prevent recursive calls into the scheduler Upon leaving a critical section (such as a SpinLock) we need to check if we're already asynchronously invoking the Scheduler. Otherwise we might end up triggering another context switch as soon as leaving the scheduler lock. Fixes #2883
2020-08-01 14:37:40 -06:00
// The target flags will get restored upon leaving the trap
u32 prev_flags = cpu_flags();
Scheduler::leave_on_first_switch(prev_flags);
Kernel: Fix some flaws that caused crashes or hangs during boot We need to halt the BSP briefly until all APs are ready for the first context switch, but we can't hold the same spinlock by all of them while doing so. So, while the APs are waiting on each other they need to release the scheduler lock, and then once signaled re-acquire it. Should solve some timing dependent hangs or crashes, most easily observed using qemu with kvm disabled.
2020-07-08 15:20:36 -06:00
}
Everywhere: void arguments to C functions Problem: - C functions with no arguments require a single `void` in the argument list. Solution: - Put the `void` in the argument list of functions in C header files.
2020-12-24 08:41:54 -07:00
extern "C" void post_init_finished(void)
Kernel: Fix some flaws that caused crashes or hangs during boot We need to halt the BSP briefly until all APs are ready for the first context switch, but we can't hold the same spinlock by all of them while doing so. So, while the APs are waiting on each other they need to release the scheduler lock, and then once signaled re-acquire it. Should solve some timing dependent hangs or crashes, most easily observed using qemu with kvm disabled.
2020-07-08 15:20:36 -06:00
{
// We need to re-acquire the scheduler lock before a context switch
// transfers control into the idle loop, which needs the lock held
Kernel: Prevent recursive calls into the scheduler Upon leaving a critical section (such as a SpinLock) we need to check if we're already asynchronously invoking the Scheduler. Otherwise we might end up triggering another context switch as soon as leaving the scheduler lock. Fixes #2883
2020-08-01 14:37:40 -06:00
Scheduler::prepare_for_idle_loop();
Kernel: Fix some flaws that caused crashes or hangs during boot We need to halt the BSP briefly until all APs are ready for the first context switch, but we can't hold the same spinlock by all of them while doing so. So, while the APs are waiting on each other they need to release the scheduler lock, and then once signaled re-acquire it. Should solve some timing dependent hangs or crashes, most easily observed using qemu with kvm disabled.
2020-07-08 15:20:36 -06:00
}
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
void Processor::initialize_context_switching(Thread& initial_thread)
{
Kernel: Rename Process::is_ring0/3 to Process::is_kernel/user_process Since "rings" typically refer to code execution and user processes can also execute in ring 0, rename these functions to more accurately describe what they mean: kernel processes and user processes.
2020-09-10 09:46:24 -06:00
ASSERT(initial_thread.process().is_kernel_process());
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
auto& tss = initial_thread.tss();
m_tss = tss;
m_tss.esp0 = tss.esp0;
m_tss.ss0 = GDT_SELECTOR_DATA0;
// user mode needs to be able to switch to kernel mode:
m_tss.cs = m_tss.ds = m_tss.es = m_tss.gs = m_tss.ss = GDT_SELECTOR_CODE0 | 3;
m_tss.fs = GDT_SELECTOR_PROC | 3;
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
m_scheduler_initialized = true;
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
asm volatile(
Everywhere: Fix more typos
2020-10-02 22:14:37 +01:00
"movl %[new_esp], %%esp \n" // switch to new stack
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
"pushl %[from_to_thread] \n" // to_thread
"pushl %[from_to_thread] \n" // from_thread
"pushl $" __STRINGIFY(GDT_SELECTOR_CODE0) " \n"
"pushl %[new_eip] \n" // save the entry eip to the stack
"movl %%esp, %%ebx \n"
"addl $20, %%ebx \n" // calculate pointer to TrapFrame
"pushl %%ebx \n"
"cld \n"
Kernel: Fix some flaws that caused crashes or hangs during boot We need to halt the BSP briefly until all APs are ready for the first context switch, but we can't hold the same spinlock by all of them while doing so. So, while the APs are waiting on each other they need to release the scheduler lock, and then once signaled re-acquire it. Should solve some timing dependent hangs or crashes, most easily observed using qemu with kvm disabled.
2020-07-08 15:20:36 -06:00
"pushl %[cpu] \n" // push argument for init_finished before register is clobbered
"call pre_init_finished \n"
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
"call init_finished \n"
"addl $4, %%esp \n"
Kernel: Fix some flaws that caused crashes or hangs during boot We need to halt the BSP briefly until all APs are ready for the first context switch, but we can't hold the same spinlock by all of them while doing so. So, while the APs are waiting on each other they need to release the scheduler lock, and then once signaled re-acquire it. Should solve some timing dependent hangs or crashes, most easily observed using qemu with kvm disabled.
2020-07-08 15:20:36 -06:00
"call post_init_finished \n"
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
"call enter_trap_no_irq \n"
"addl $4, %%esp \n"
"lret \n"
:: [new_esp] "g" (tss.esp),
[new_eip] "a" (tss.eip),
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
[from_to_thread] "b" (&initial_thread),
[cpu] "c" (id())
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
);
Everywhere: Switch from (void) to [[maybe_unused]] (#4473) Problem: - `(void)` simply casts the expression to void. This is understood to indicate that it is ignored, but this is really a compiler trick to get the compiler to not generate a warning. Solution: - Use the `[[maybe_unused]]` attribute to indicate the value is unused. Note: - Functions taking a `(void)` argument list have also been changed to `()` because this is not needed and shows up in the same grep command.
2020-12-20 16:09:48 -07:00
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
ASSERT_NOT_REACHED();
}
void Processor::enter_trap(TrapFrame& trap, bool raise_irq)
{
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
ASSERT_INTERRUPTS_DISABLED();
ASSERT(&Processor::current() == this);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
trap.prev_irq_level = m_in_irq;
if (raise_irq)
m_in_irq++;
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
auto* current_thread = Processor::current_thread();
if (current_thread) {
auto& current_trap = current_thread->current_trap();
Kernel: Track previous mode when entering/exiting traps This allows us to determine what the previous mode (user or kernel) was, e.g. in the timer interrupt. This is used e.g. to determine whether a signal handler should be set up. Fixes #5096
2021-01-25 13:19:34 -07:00
trap.next_trap = current_trap;
current_trap = &trap;
// The cs register of this trap tells us where we will return back to
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
current_thread->set_previous_mode(((trap.regs->cs & 3) != 0) ? Thread::PreviousMode::UserMode : Thread::PreviousMode::KernelMode);
Kernel: Track previous mode when entering/exiting traps This allows us to determine what the previous mode (user or kernel) was, e.g. in the timer interrupt. This is used e.g. to determine whether a signal handler should be set up. Fixes #5096
2021-01-25 13:19:34 -07:00
} else {
trap.next_trap = nullptr;
}
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
}
void Processor::exit_trap(TrapFrame& trap)
{
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
ASSERT_INTERRUPTS_DISABLED();
ASSERT(&Processor::current() == this);
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
ASSERT(m_in_irq >= trap.prev_irq_level);
m_in_irq = trap.prev_irq_level;
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
smp_process_pending_messages();
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
if (!m_in_irq && !m_in_critical)
check_invoke_scheduler();
Kernel: Track previous mode when entering/exiting traps This allows us to determine what the previous mode (user or kernel) was, e.g. in the timer interrupt. This is used e.g. to determine whether a signal handler should be set up. Fixes #5096
2021-01-25 13:19:34 -07:00
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
auto* current_thread = Processor::current_thread();
if (current_thread) {
auto& current_trap = current_thread->current_trap();
Kernel: Track previous mode when entering/exiting traps This allows us to determine what the previous mode (user or kernel) was, e.g. in the timer interrupt. This is used e.g. to determine whether a signal handler should be set up. Fixes #5096
2021-01-25 13:19:34 -07:00
current_trap = trap.next_trap;
if (current_trap) {
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
ASSERT(current_trap->regs);
Kernel: Track previous mode when entering/exiting traps This allows us to determine what the previous mode (user or kernel) was, e.g. in the timer interrupt. This is used e.g. to determine whether a signal handler should be set up. Fixes #5096
2021-01-25 13:19:34 -07:00
// If we have another higher level trap then we probably returned
// from an interrupt or irq handler. The cs register of the
// new/higher level trap tells us what the mode prior to it was
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
current_thread->set_previous_mode(((current_trap->regs->cs & 3) != 0) ? Thread::PreviousMode::UserMode : Thread::PreviousMode::KernelMode);
Kernel: Track previous mode when entering/exiting traps This allows us to determine what the previous mode (user or kernel) was, e.g. in the timer interrupt. This is used e.g. to determine whether a signal handler should be set up. Fixes #5096
2021-01-25 13:19:34 -07:00
} else {
// If we don't have a higher level trap then we're back in user mode.
// Unless we're a kernel process, in which case we're always in kernel mode
Kernel: Make Thread::current smp-safe Change Thread::current to be a static function and read using the fs register, which eliminates a window between Processor::current() returning and calling a function on it, which can trigger preemption and a move to a different processor, which then causes operating on the wrong object.
2021-01-26 14:16:07 -07:00
current_thread->set_previous_mode(current_thread->process().is_kernel_process() ? Thread::PreviousMode::KernelMode : Thread::PreviousMode::UserMode);
Kernel: Track previous mode when entering/exiting traps This allows us to determine what the previous mode (user or kernel) was, e.g. in the timer interrupt. This is used e.g. to determine whether a signal handler should be set up. Fixes #5096
2021-01-25 13:19:34 -07:00
}
}
Kernel: Fix signal delivery When delivering urgent signals to the current thread we need to check if we should be unblocked, and if not we need to yield to another process. We also need to make sure that we suppress context switches during Process::exec() so that we don't clobber the registers that it sets up (eip mainly) by a context switch. To be able to do that we add the concept of a critical section, which are similar to Process::m_in_irq but different in that they can be requested at any time. Calls to Scheduler::yield and Scheduler::donate_to will return instantly without triggering a context switch, but the processor will then asynchronously trigger a context switch once the critical section is left.
2020-07-03 05:19:50 -06:00
}
void Processor::check_invoke_scheduler()
{
ASSERT(!m_in_irq);
ASSERT(!m_in_critical);
if (m_invoke_scheduler_async && m_scheduler_initialized) {
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
m_invoke_scheduler_async = false;
Scheduler::invoke_async();
}
}
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
void Processor::flush_tlb_local(VirtualAddress vaddr, size_t page_count)
{
auto ptr = vaddr.as_ptr();
while (page_count > 0) {
asm volatile("invlpg %0"
:
Kernel: Aggregate TLB flush requests for Regions for SMP Rather than sending one TLB flush request for each page, aggregate them so that we're not spamming the other processors with FlushTLB IPIs.
2020-07-06 12:47:08 -06:00
: "m"(*ptr)
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
: "memory");
ptr += PAGE_SIZE;
page_count--;
}
}
Kernel: Ignore TLB flush requests for user addresses of other processes If a TLB flush request is broadcast to other processors and the addresses to flush are user mode addresses, we can ignore such a request on the target processor if the page directory currently in use doesn't match the addresses to be flushed. We still need to broadcast to all processors in that case because the other processors may switch to that same page directory at any time.
2021-01-02 12:27:38 -07:00
void Processor::flush_tlb(const PageDirectory* page_directory, VirtualAddress vaddr, size_t page_count)
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
{
if (s_smp_enabled)
Kernel: Ignore TLB flush requests for user addresses of other processes If a TLB flush request is broadcast to other processors and the addresses to flush are user mode addresses, we can ignore such a request on the target processor if the page directory currently in use doesn't match the addresses to be flushed. We still need to broadcast to all processors in that case because the other processors may switch to that same page directory at any time.
2021-01-02 12:27:38 -07:00
smp_broadcast_flush_tlb(page_directory, vaddr, page_count);
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
else
flush_tlb_local(vaddr, page_count);
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
}
static volatile ProcessorMessage* s_message_pool;
void Processor::smp_return_to_pool(ProcessorMessage& msg)
{
ProcessorMessage* next = nullptr;
do {
msg.next = next;
} while (!atomic_compare_exchange_strong(&s_message_pool, next, &msg, AK::MemoryOrder::memory_order_acq_rel));
}
ProcessorMessage& Processor::smp_get_from_pool()
{
ProcessorMessage* msg;
// The assumption is that messages are never removed from the pool!
for (;;) {
msg = atomic_load(&s_message_pool, AK::MemoryOrder::memory_order_consume);
if (!msg) {
if (!Processor::current().smp_process_pending_messages()) {
// TODO: pause for a bit?
}
continue;
}
// If another processor were to use this message in the meanwhile,
// "msg" is still valid (because it never gets freed). We'd detect
// this because the expected value "msg" and pool would
// no longer match, and the compare_exchange will fail. But accessing
// "msg->next" is always safe here.
if (atomic_compare_exchange_strong(&s_message_pool, msg, msg->next, AK::MemoryOrder::memory_order_acq_rel)) {
// We successfully "popped" this available message
break;
}
}
ASSERT(msg != nullptr);
return *msg;
}
Kernel: Track processor idle state and wake processors when waking threads Attempt to wake idle processors to get threads to be scheduled more quickly. We don't want to wait until the next timer tick if we have processors that aren't doing anything.
2020-10-28 16:06:16 -06:00
Atomic<u32> Processor::s_idle_cpu_mask{ 0 };
u32 Processor::smp_wake_n_idle_processors(u32 wake_count)
{
ASSERT(Processor::current().in_critical());
ASSERT(wake_count > 0);
if (!s_smp_enabled)
return 0;
// Wake at most N - 1 processors
if (wake_count >= Processor::count()) {
wake_count = Processor::count() - 1;
ASSERT(wake_count > 0);
}
u32 current_id = Processor::current().id();
u32 did_wake_count = 0;
auto& apic = APIC::the();
while (did_wake_count < wake_count) {
// Try to get a set of idle CPUs and flip them to busy
u32 idle_mask = s_idle_cpu_mask.load(AK::MemoryOrder::memory_order_relaxed) & ~(1u << current_id);
u32 idle_count = __builtin_popcountl(idle_mask);
if (idle_count == 0)
break; // No (more) idle processor available
u32 found_mask = 0;
for (u32 i = 0; i < idle_count; i++) {
u32 cpu = __builtin_ffsl(idle_mask) - 1;
idle_mask &= ~(1u << cpu);
found_mask |= 1u << cpu;
}
idle_mask = s_idle_cpu_mask.fetch_and(~found_mask, AK::MemoryOrder::memory_order_acq_rel) & found_mask;
if (idle_mask == 0)
continue; // All of them were flipped to busy, try again
idle_count = __builtin_popcountl(idle_mask);
for (u32 i = 0; i < idle_count; i++) {
u32 cpu = __builtin_ffsl(idle_mask) - 1;
idle_mask &= ~(1u << cpu);
// Send an IPI to that CPU to wake it up. There is a possibility
// someone else woke it up as well, or that it woke up due to
// a timer interrupt. But we tried hard to avoid this...
apic.send_ipi(cpu);
did_wake_count++;
}
}
return did_wake_count;
}
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
void Processor::smp_enable()
{
size_t msg_pool_size = Processor::count() * 100u;
size_t msg_entries_cnt = Processor::count();
auto msgs = new ProcessorMessage[msg_pool_size];
auto msg_entries = new ProcessorMessageEntry[msg_pool_size * msg_entries_cnt];
size_t msg_entry_i = 0;
for (size_t i = 0; i < msg_pool_size; i++, msg_entry_i += msg_entries_cnt) {
auto& msg = msgs[i];
msg.next = i < msg_pool_size - 1 ? &msgs[i + 1] : nullptr;
msg.per_proc_entries = &msg_entries[msg_entry_i];
for (size_t k = 0; k < msg_entries_cnt; k++)
msg_entries[msg_entry_i + k].msg = &msg;
}
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
atomic_store(&s_message_pool, &msgs[0], AK::MemoryOrder::memory_order_release);
// Start sending IPI messages
s_smp_enabled = true;
}
void Processor::smp_cleanup_message(ProcessorMessage& msg)
{
switch (msg.type) {
Meta+Kernel: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
case ProcessorMessage::CallbackWithData:
if (msg.callback_with_data.free)
msg.callback_with_data.free(msg.callback_with_data.data);
break;
default:
break;
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
}
}
bool Processor::smp_process_pending_messages()
{
bool did_process = false;
u32 prev_flags;
enter_critical(prev_flags);
Meta+Kernel: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
if (auto pending_msgs = atomic_exchange(&m_message_queue, nullptr, AK::MemoryOrder::memory_order_acq_rel)) {
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
// We pulled the stack of pending messages in LIFO order, so we need to reverse the list first
auto reverse_list =
[](ProcessorMessageEntry* list) -> ProcessorMessageEntry*
{
ProcessorMessageEntry* rev_list = nullptr;
while (list) {
auto next = list->next;
list->next = rev_list;
rev_list = list;
list = next;
}
return rev_list;
};
pending_msgs = reverse_list(pending_msgs);
// now process in the right order
ProcessorMessageEntry* next_msg;
for (auto cur_msg = pending_msgs; cur_msg; cur_msg = next_msg) {
next_msg = cur_msg->next;
auto msg = cur_msg->msg;
Meta+Kernel: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
Everywhere: Debug macros instead of constexpr. This was done with the following script: find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/dbgln<debug_([a-z_]+)>/dbgln<\U\1_DEBUG>/' {} \; find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/if constexpr \(debug_([a-z0-9_]+)/if constexpr \(\U\1_DEBUG/' {} \;
2021-01-23 23:59:27 +01:00
dbgln<SMP_DEBUG>("SMP[{}]: Processing message {}", id(), VirtualAddress(msg));
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
switch (msg->type) {
Meta+Kernel: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
case ProcessorMessage::Callback:
msg->callback.handler();
break;
case ProcessorMessage::CallbackWithData:
msg->callback_with_data.handler(msg->callback_with_data.data);
break;
case ProcessorMessage::FlushTlb:
Kernel: Ignore TLB flush requests for user addresses of other processes If a TLB flush request is broadcast to other processors and the addresses to flush are user mode addresses, we can ignore such a request on the target processor if the page directory currently in use doesn't match the addresses to be flushed. We still need to broadcast to all processors in that case because the other processors may switch to that same page directory at any time.
2021-01-02 12:27:38 -07:00
if (is_user_address(VirtualAddress(msg->flush_tlb.ptr))) {
// We assume that we don't cross into kernel land!
ASSERT(is_user_range(VirtualAddress(msg->flush_tlb.ptr), msg->flush_tlb.page_count * PAGE_SIZE));
if (read_cr3() != msg->flush_tlb.page_directory->cr3()) {
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-13 21:48:51 +01:00
// This processor isn't using this page directory right now, we can ignore this request
Everywhere: Debug macros instead of constexpr. This was done with the following script: find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/dbgln<debug_([a-z_]+)>/dbgln<\U\1_DEBUG>/' {} \; find . \( -name '*.cpp' -o -name '*.h' -o -name '*.in' \) -not -path './Toolchain/*' -not -path './Build/*' -exec sed -i -E 's/if constexpr \(debug_([a-z0-9_]+)/if constexpr \(\U\1_DEBUG/' {} \;
2021-01-23 23:59:27 +01:00
dbgln<SMP_DEBUG>("SMP[{}]: No need to flush {} pages at {}", id(), msg->flush_tlb.page_count, VirtualAddress(msg->flush_tlb.ptr));
Kernel: Ignore TLB flush requests for user addresses of other processes If a TLB flush request is broadcast to other processors and the addresses to flush are user mode addresses, we can ignore such a request on the target processor if the page directory currently in use doesn't match the addresses to be flushed. We still need to broadcast to all processors in that case because the other processors may switch to that same page directory at any time.
2021-01-02 12:27:38 -07:00
break;
}
}
Meta+Kernel: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
flush_tlb_local(VirtualAddress(msg->flush_tlb.ptr), msg->flush_tlb.page_count);
break;
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
}
bool is_async = msg->async; // Need to cache this value *before* dropping the ref count!
auto prev_refs = atomic_fetch_sub(&msg->refs, 1u, AK::MemoryOrder::memory_order_acq_rel);
ASSERT(prev_refs != 0);
if (prev_refs == 1) {
// All processors handled this. If this is an async message,
// we need to clean it up and return it to the pool
if (is_async) {
smp_cleanup_message(*msg);
smp_return_to_pool(*msg);
}
}
Kernel: Make m_halt_requested an atomic variable We need to make sure the change to this variable is visible to all processors instantly.
2020-10-27 20:39:08 -06:00
if (m_halt_requested.load(AK::MemoryOrder::memory_order_relaxed))
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
halt_this();
}
did_process = true;
Kernel: Make m_halt_requested an atomic variable We need to make sure the change to this variable is visible to all processors instantly.
2020-10-27 20:39:08 -06:00
} else if (m_halt_requested.load(AK::MemoryOrder::memory_order_relaxed)) {
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
halt_this();
}
leave_critical(prev_flags);
return did_process;
}
bool Processor::smp_queue_message(ProcessorMessage& msg)
{
// Note that it's quite possible that the other processor may pop
// the queue at any given time. We rely on the fact that the messages
// are pooled and never get freed!
auto& msg_entry = msg.per_proc_entries[id()];
ASSERT(msg_entry.msg == &msg);
ProcessorMessageEntry* next = nullptr;
do {
msg_entry.next = next;
} while (!atomic_compare_exchange_strong(&m_message_queue, next, &msg_entry, AK::MemoryOrder::memory_order_acq_rel));
return next == nullptr;
}
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
void Processor::smp_broadcast_message(ProcessorMessage& msg)
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
{
auto& cur_proc = Processor::current();
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-13 21:48:51 +01:00
Kernel: Make Processor::id a static function This eliminates the window between calling Processor::current and the member function where a thread could be moved to another processor. This is generally not as big of a concern as with Processor::current_thread, but also slightly more light weight.
2021-01-26 20:44:01 -07:00
dbgln<SMP_DEBUG>("SMP[{}]: Broadcast message {} to cpus: {} proc: {}", cur_proc.get_id(), VirtualAddress(&msg), count(), VirtualAddress(&cur_proc));
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-13 21:48:51 +01:00
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
atomic_store(&msg.refs, count() - 1, AK::MemoryOrder::memory_order_release);
ASSERT(msg.refs > 0);
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
bool need_broadcast = false;
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
for_each(
Meta+Kernel: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
[&](Processor& proc) -> IterationDecision {
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
if (&proc != &cur_proc) {
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
if (proc.smp_queue_message(msg))
need_broadcast = true;
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
}
return IterationDecision::Continue;
});
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
// Now trigger an IPI on all other APs (unless all targets already had messages queued)
if (need_broadcast)
APIC::the().broadcast_ipi();
}
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
void Processor::smp_broadcast_wait_sync(ProcessorMessage& msg)
{
auto& cur_proc = Processor::current();
ASSERT(!msg.async);
// If synchronous then we must cleanup and return the message back
// to the pool. Otherwise, the last processor to complete it will return it
while (atomic_load(&msg.refs, AK::MemoryOrder::memory_order_consume) != 0) {
// TODO: pause for a bit?
// We need to process any messages that may have been sent to
// us while we're waiting. This also checks if another processor
// may have requested us to halt.
cur_proc.smp_process_pending_messages();
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
}
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
smp_cleanup_message(msg);
smp_return_to_pool(msg);
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
}
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
void Processor::smp_broadcast(void (*callback)(void*), void* data, void (*free_data)(void*), bool async)
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
{
auto& msg = smp_get_from_pool();
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
msg.async = async;
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
msg.type = ProcessorMessage::CallbackWithData;
msg.callback_with_data.handler = callback;
msg.callback_with_data.data = data;
msg.callback_with_data.free = free_data;
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
smp_broadcast_message(msg);
if (!async)
smp_broadcast_wait_sync(msg);
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
}
Kernel: Tell compiler about invisible calls This makes the Kernel build cleanly with -Wmissing-declarations.
2020-08-11 23:33:37 +02:00
void Processor::smp_broadcast(void (*callback)(), bool async)
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
{
auto& msg = smp_get_from_pool();
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
msg.async = async;
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
msg.type = ProcessorMessage::CallbackWithData;
msg.callback.handler = callback;
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
smp_broadcast_message(msg);
if (!async)
smp_broadcast_wait_sync(msg);
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
}
Kernel: Implement capturing stack trace on a different CPU When trying to get a stack trace of a thread on another CPU we send a SMP message to that processor to capture the stack trace for us.
2020-10-31 19:32:38 -06:00
void Processor::smp_unicast_message(u32 cpu, ProcessorMessage& msg, bool async)
{
auto& cur_proc = Processor::current();
Kernel: Make Processor::id a static function This eliminates the window between calling Processor::current and the member function where a thread could be moved to another processor. This is generally not as big of a concern as with Processor::current_thread, but also slightly more light weight.
2021-01-26 20:44:01 -07:00
ASSERT(cpu != cur_proc.get_id());
Kernel: Implement capturing stack trace on a different CPU When trying to get a stack trace of a thread on another CPU we send a SMP message to that processor to capture the stack trace for us.
2020-10-31 19:32:38 -06:00
auto& target_proc = processors()[cpu];
msg.async = async;
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-13 21:48:51 +01:00
Kernel: Make Processor::id a static function This eliminates the window between calling Processor::current and the member function where a thread could be moved to another processor. This is generally not as big of a concern as with Processor::current_thread, but also slightly more light weight.
2021-01-26 20:44:01 -07:00
dbgln<SMP_DEBUG>("SMP[{}]: Send message {} to cpu #{} proc: {}", cur_proc.get_id(), VirtualAddress(&msg), cpu, VirtualAddress(&target_proc));
Everywhere: Replace a bundle of dbg with dbgln. These changes are arbitrarily divided into multiple commits to make it easier to find potentially introduced bugs with git bisect.
2021-01-13 21:48:51 +01:00
Kernel: Implement capturing stack trace on a different CPU When trying to get a stack trace of a thread on another CPU we send a SMP message to that processor to capture the stack trace for us.
2020-10-31 19:32:38 -06:00
atomic_store(&msg.refs, 1u, AK::MemoryOrder::memory_order_release);
if (target_proc->smp_queue_message(msg)) {
APIC::the().send_ipi(cpu);
}
if (!async) {
// If synchronous then we must cleanup and return the message back
// to the pool. Otherwise, the last processor to complete it will return it
while (atomic_load(&msg.refs, AK::MemoryOrder::memory_order_consume) != 0) {
// TODO: pause for a bit?
Kernel: Fix deadlock when unicasting/broadcasting SMP message When two processors send each others a SMP message at the same time they need to process messages while waiting for delivery of the message they just sent, or they will deadlock.
2020-11-01 10:14:27 -07:00
// We need to process any messages that may have been sent to
// us while we're waiting. This also checks if another processor
// may have requested us to halt.
cur_proc.smp_process_pending_messages();
Kernel: Implement capturing stack trace on a different CPU When trying to get a stack trace of a thread on another CPU we send a SMP message to that processor to capture the stack trace for us.
2020-10-31 19:32:38 -06:00
}
smp_cleanup_message(msg);
smp_return_to_pool(msg);
}
}
void Processor::smp_unicast(u32 cpu, void (*callback)(void*), void* data, void (*free_data)(void*), bool async)
{
auto& msg = smp_get_from_pool();
msg.type = ProcessorMessage::CallbackWithData;
msg.callback_with_data.handler = callback;
msg.callback_with_data.data = data;
msg.callback_with_data.free = free_data;
smp_unicast_message(cpu, msg, async);
}
void Processor::smp_unicast(u32 cpu, void (*callback)(), bool async)
{
auto& msg = smp_get_from_pool();
msg.type = ProcessorMessage::CallbackWithData;
msg.callback.handler = callback;
smp_unicast_message(cpu, msg, async);
}
Kernel: Ignore TLB flush requests for user addresses of other processes If a TLB flush request is broadcast to other processors and the addresses to flush are user mode addresses, we can ignore such a request on the target processor if the page directory currently in use doesn't match the addresses to be flushed. We still need to broadcast to all processors in that case because the other processors may switch to that same page directory at any time.
2021-01-02 12:27:38 -07:00
void Processor::smp_broadcast_flush_tlb(const PageDirectory* page_directory, VirtualAddress vaddr, size_t page_count)
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
{
auto& msg = smp_get_from_pool();
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
msg.async = false;
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
msg.type = ProcessorMessage::FlushTlb;
Kernel: Ignore TLB flush requests for user addresses of other processes If a TLB flush request is broadcast to other processors and the addresses to flush are user mode addresses, we can ignore such a request on the target processor if the page directory currently in use doesn't match the addresses to be flushed. We still need to broadcast to all processors in that case because the other processors may switch to that same page directory at any time.
2021-01-02 12:27:38 -07:00
msg.flush_tlb.page_directory = page_directory;
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
msg.flush_tlb.ptr = vaddr.as_ptr();
msg.flush_tlb.page_count = page_count;
Kernel: Flush TLBs concurrently Instead of flushing the TLB on the current processor first and then notifying the other processors to do the same, notify the others first, and while waiting on the others flush our own.
2020-12-02 15:03:07 -07:00
smp_broadcast_message(msg);
// While the other processors handle this request, we'll flush ours
flush_tlb_local(vaddr, page_count);
// Now wait until everybody is done as well
smp_broadcast_wait_sync(msg);
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
}
void Processor::smp_broadcast_halt()
{
// We don't want to use a message, because this could have been triggered
// by being out of memory and we might not be able to get a message
for_each(
Meta+Kernel: Make clang-format-10 clean
2020-09-18 09:49:51 +02:00
[&](Processor& proc) -> IterationDecision {
Kernel: Make m_halt_requested an atomic variable We need to make sure the change to this variable is visible to all processors instantly.
2020-10-27 20:39:08 -06:00
proc.m_halt_requested.store(true, AK::MemoryOrder::memory_order_release);
Kernel: Add SMP IPI support We can now properly initialize all processors without crashing by sending SMP IPI messages to synchronize memory between processors. We now initialize the APs once we have the scheduler running. This is so that we can process IPI messages from the other cores. Also rework interrupt handling a bit so that it's more of a 1:1 mapping. We need to allocate non-sharable interrupts for IPIs. This also fixes the occasional hang/crash because all CPUs now synchronize memory with each other.
2020-07-06 07:27:22 -06:00
return IterationDecision::Continue;
});
// Now trigger an IPI on all other APs
APIC::the().broadcast_ipi();
}
void Processor::Processor::halt()
{
if (s_smp_enabled)
smp_broadcast_halt();
halt_this();
}
Kernel: Add mechanism to queue deferred function calls Function calls that are deferred will be executed before a thread enters a pre-emptable state (meaning it is not in a critical section and it is not in an irq handler). If it is not already in such a state, it will be called immediately. This is meant to be used from e.g. IRQ handlers where we might want to block a thread until an interrupt happens.
2020-11-01 13:08:25 -07:00
void Processor::deferred_call_pool_init()
{
size_t pool_count = sizeof(m_deferred_call_pool) / sizeof(m_deferred_call_pool[0]);
for (size_t i = 0; i < pool_count; i++) {
auto& entry = m_deferred_call_pool[i];
entry.next = i < pool_count - 1 ? &m_deferred_call_pool[i + 1] : nullptr;
entry.was_allocated = false;
}
m_pending_deferred_calls = nullptr;
m_free_deferred_call_pool_entry = &m_deferred_call_pool[0];
}
void Processor::deferred_call_return_to_pool(DeferredCallEntry* entry)
{
ASSERT(m_in_critical);
ASSERT(!entry->was_allocated);
entry->next = m_free_deferred_call_pool_entry;
m_free_deferred_call_pool_entry = entry;
}
DeferredCallEntry* Processor::deferred_call_get_free()
{
ASSERT(m_in_critical);
if (m_free_deferred_call_pool_entry) {
// Fast path, we have an entry in our pool
auto* entry = m_free_deferred_call_pool_entry;
m_free_deferred_call_pool_entry = entry->next;
ASSERT(!entry->was_allocated);
return entry;
}
auto* entry = new DeferredCallEntry;
entry->was_allocated = true;
return entry;
}
void Processor::deferred_call_execute_pending()
{
ASSERT(m_in_critical);
if (!m_pending_deferred_calls)
return;
auto* pending_list = m_pending_deferred_calls;
m_pending_deferred_calls = nullptr;
// We pulled the stack of pending deferred calls in LIFO order, so we need to reverse the list first
auto reverse_list =
[](DeferredCallEntry* list) -> DeferredCallEntry*
{
DeferredCallEntry* rev_list = nullptr;
while (list) {
auto next = list->next;
list->next = rev_list;
rev_list = list;
list = next;
}
return rev_list;
};
pending_list = reverse_list(pending_list);
do {
// Call the appropriate callback handler
if (pending_list->have_data) {
pending_list->callback_with_data.handler(pending_list->callback_with_data.data);
if (pending_list->callback_with_data.free)
pending_list->callback_with_data.free(pending_list->callback_with_data.data);
} else {
pending_list->callback.handler();
}
// Return the entry back to the pool, or free it
auto* next = pending_list->next;
if (pending_list->was_allocated)
delete pending_list;
else
deferred_call_return_to_pool(pending_list);
pending_list = next;
} while (pending_list);
}
void Processor::deferred_call_queue_entry(DeferredCallEntry* entry)
{
ASSERT(m_in_critical);
entry->next = m_pending_deferred_calls;
m_pending_deferred_calls = entry;
}
void Processor::deferred_call_queue(void (*callback)())
{
// NOTE: If we are called outside of a critical section and outside
// of an irq handler, the function will be executed before we return!
ScopedCritical critical;
auto& cur_proc = Processor::current();
auto* entry = cur_proc.deferred_call_get_free();
entry->have_data = false;
entry->callback.handler = callback;
cur_proc.deferred_call_queue_entry(entry);
}
void Processor::deferred_call_queue(void (*callback)(void*), void* data, void (*free_data)(void*))
{
// NOTE: If we are called outside of a critical section and outside
// of an irq handler, the function will be executed before we return!
ScopedCritical critical;
auto& cur_proc = Processor::current();
auto* entry = cur_proc.deferred_call_get_free();
entry->have_data = true;
entry->callback_with_data.handler = callback;
entry->callback_with_data.data = data;
entry->callback_with_data.free = free_data;
cur_proc.deferred_call_queue_entry(entry);
}
Kernel: Implement software context switching and Processor structure Moving certain globals into a new Processor structure for each CPU allows us to eventually run an instance of the scheduler on each CPU.
2020-06-27 13:42:28 -06:00
void Processor::gdt_init()
{
m_gdt_length = 0;
m_gdtr.address = nullptr;
m_gdtr.limit = 0;
write_raw_gdt_entry(0x0000, 0x00000000, 0x00000000);
write_raw_gdt_entry(GDT_SELECTOR_CODE0, 0x0000ffff, 0x00cf9a00); // code0
write_raw_gdt_entry(GDT_SELECTOR_DATA0, 0x0000ffff, 0x00cf9200); // data0
write_raw_gdt_entry(GDT_SELECTOR_CODE3, 0x0000ffff, 0x00cffa00); // code3
write_raw_gdt_entry(GDT_SELECTOR_DATA3, 0x0000ffff, 0x00cff200); // data3
Descriptor tls_descriptor;
tls_descriptor.low = tls_descriptor.high = 0;
tls_descriptor.dpl = 3;
tls_descriptor.segment_present = 1;
tls_descriptor.granularity = 0;
tls_descriptor.zero = 0;
tls_descriptor.operation_size = 1;
tls_descriptor.descriptor_type = 1;
tls_descriptor.type = 2;
write_gdt_entry(GDT_SELECTOR_TLS, tls_descriptor); // tls3
Descriptor fs_descriptor;
fs_descriptor.set_base(this);
fs_descriptor.set_limit(sizeof(Processor));
fs_descriptor.dpl = 0;
fs_descriptor.segment_present = 1;
fs_descriptor.granularity = 0;
fs_descriptor.zero = 0;
fs_descriptor.operation_size = 1;
fs_descriptor.descriptor_type = 1;
fs_descriptor.type = 2;
write_gdt_entry(GDT_SELECTOR_PROC, fs_descriptor); // fs0
Descriptor tss_descriptor;
tss_descriptor.set_base(&m_tss);
tss_descriptor.set_limit(sizeof(TSS32));
tss_descriptor.dpl = 0;
tss_descriptor.segment_present = 1;
tss_descriptor.granularity = 0;
tss_descriptor.zero = 0;
tss_descriptor.operation_size = 1;
tss_descriptor.descriptor_type = 0;
tss_descriptor.type = 9;
write_gdt_entry(GDT_SELECTOR_TSS, tss_descriptor); // tss
flush_gdt();
load_task_register(GDT_SELECTOR_TSS);
asm volatile(
"mov %%ax, %%ds\n"
"mov %%ax, %%es\n"
"mov %%ax, %%gs\n"
"mov %%ax, %%ss\n" ::"a"(GDT_SELECTOR_DATA0)
: "memory");
set_fs(GDT_SELECTOR_PROC);
// Make sure CS points to the kernel code descriptor.
asm volatile(
"ljmpl $" __STRINGIFY(GDT_SELECTOR_CODE0) ", $sanity\n"
"sanity:\n");
}
void Processor::set_thread_specific(u8* data, size_t len)
{
auto& descriptor = get_gdt_entry(GDT_SELECTOR_TLS);
descriptor.set_base(data);
descriptor.set_limit(len);
}
Kernel: Move all code into the Kernel namespace
2020-02-16 01:27:42 +01:00
}
#ifdef DEBUG
void __assertion_failed(const char* msg, const char* file, unsigned line, const char* func)
{
asm volatile("cli");
Kernel: Use klog() instead of kprintf() Also, duplicate data in dbg() and klog() calls were removed. In addition, leakage of virtual address to kernel log is prevented. This is done by replacing kprintf() calls to dbg() calls with the leaked data instead. Also, other kprintf() calls were replaced with klog().
2020-03-01 21:45:39 +02:00
klog() << "ASSERTION FAILED: " << msg << "\n"
<< file << ":" << line << " in " << func;
Kernel: Move all code into the Kernel namespace
2020-02-16 01:27:42 +01:00
// Switch back to the current process's page tables if there are any.
// Otherwise stack walking will be a disaster.
Kernel: Turn Thread::current and Process::current into functions This allows us to query the current thread and process on a per processor basis
2020-06-28 15:34:31 -06:00
auto process = Process::current();
if (process)
MM.enter_process_paging_scope(*process);
Kernel: Move all code into the Kernel namespace
2020-02-16 01:27:42 +01:00
Kernel::dump_backtrace();
Kernel: Halt all processors on assertion failure
2020-10-25 22:27:27 -06:00
Processor::halt();
Kernel: Move all code into the Kernel namespace
2020-02-16 01:27:42 +01:00
}
#endif
Kernel: Add the NonMaskableInterruptDisabler class This class will be used later to disable NMIs when we initialize the RTC timer.
2020-02-26 02:04:05 +02:00
NonMaskableInterruptDisabler::NonMaskableInterruptDisabler()
{
IO::out8(0x70, IO::in8(0x70) | 0x80);
}
NonMaskableInterruptDisabler::~NonMaskableInterruptDisabler()
{
IO::out8(0x70, IO::in8(0x70) & 0x7F);
}
Reference in a new issue Copy permalink