-
Notifications
You must be signed in to change notification settings - Fork 6
x86_64 System Bring up Walkthrough
- Kernel and core-specific memory begins 1MB into physical memory.
- 4GB of virtual memory is initialized using PML4 paging (as described in the Intel manual).
- Execution begins on a single core with interrupts disabled. Core0 spins up all additional core after preinit.
The single execution path moves from the Multiboot bootloader (GRUB) to an assembly script running in 32 bit mode (/l0/lrt/bare/arch/amd64/start.S). This code pulls Multiboot settings values into the system registers and banches into the "init32" C-language function. The runtime is not expected to return from the init32 function.
The primary role of the init32 function (/l0/lrt/bare/arch/amd64/init32.c) is to setup the virtual memory paging structure. The first 1GB of physical memory is mapped using 2MB pages. All memory is mapped as present and readable/writable. To setup the virtual mapping, we enable PAE mode, longmode and paging. Next we configure and load a long-mode sector into the global descriptor table (GDT) and do a long jump (segment switch) into init64(). At the point core0 is are running in 64-bit longmode.
The init64 function (/l0/lrt/bare/arch/amd64/init64.c) handles the bulk of early system bring up. Namely: Zero out segment selectors Clear the bss segment, the part of the data segment containing statically-allocated variables (as required by the C standard.) Initialize C++ ctors Initialize serial / stdout Initalize a stripped-down ACPI and, from this, get a system core count Run the Ebblib preinit functions: (1.1) memory, (1.2) event and (1.3) translation. Spin up the secondary system (1.4) cores. Finally, each all cores complete the EBBlib Init Functions.
The lrt_mem_preinit(int cores) function _(/l0/lrt/bare/arch/amd64/mem.c) _ divides the available virtual memory (lower 4 gigs, virtual and physical mapped 1:1) up between the cores and stores the partition information in a shared, global array called bootmem.
The lrt_event_preinit(int cores) function (/l0/lrt/bare/arch/amd64/event.c) takes core are the system-wide configuration of interrupts. To do this, we first disable the unnecessary interrupt-related hardware: pic, pit & rtc (pit and rtc are timers that fire interrupts). Next, we allocate the idt and enable the lapic and ioapic. Interrupts arerequired for SMP bringup (via IPI.) Finally, this function also allocates the alt-stack (user for interception & hot-swap) in the memory of core0.
The lrt_trans_preinit(int cores) function (/l0/lrt/bare/arch/amd64/trans.c) allocate the (Ebb) global and local description tables (for all cores) and map then into the page tables.
The role of the init_smp() function (/l0/lrt/bare/arch/amd64/init64.c) is to spin up and begin the execution of the remaning local available cores.We do this by sending each core two IPIs. The first to "wake up" each core and the second to tell the core the starting address (which is in /l0/lrt/bare/arch/amd64/smp.S). After initialization, each core enters the event loop.
Each core is spun up one at a time. This is done by having core0 lock after signaling a core. Once that core has finished the initialization, core0 is released and will spin up the next core.
For specifics on the secondary core init process, look at the section on Secondary core bring-up.
###From this point on, all initialization code is executed in parallel across all cores.
By the end of init64, each core has begun execution and the system is (just about) in a fully initialize state. At this point, each core initializes its local environment for events,memory and translations and, finally, either starts execution the application or enters a loop and waits for events.
The first function call after core0 as spun up the secondary cores, as well as one of the first functions executed by each a secondary core, is the lrt_event_init() function (/l0/lrt/bare/arch/amd64/event.c). Within this function, each core sets its location id (core #), allocates a local stack, and makes a call to lrt_start() (/l0/lrt/bare/arch/amd64/lrt_start.c). Within the lrt_start function, each core initializes memory and translation before either starting the application or returning to lrt_event_init where it will event the event loop.
[This function is empty for the time being.]
Lrt_start() makes a call to lrt_trans_init() (l0/lrt/trans.c) that makes a call a lrt_trans_specific_init() (/l0/lrt/bare/arch/amd64/trans.c). Within this function, each core allocates and maps its own translation table, copies over the pre-init data (setup in init32 and lrt_trans_preinit) and add its local memory translation table and the global translation table into virtual memory. At this point, all cores have their own virtual memory translation structures.
The address of each local translation table is stored in global array (shared between cores) called lmem_table.
After the memory and translation initialization, the core are fully initialized and are ready to begin execution of the app. The app developer has the ability to control the cores that begin execution of the apps (by running the ".start" function) and the core than do not run the app but instead enter the event loop. Macros are available (/l1/App.h) for basic declarations of app_start.
This subsection describes bring-up process of the SMP CPU cores that are spun-up during the x86_64 system bring-up.
Initially the core is awoken by core0 (via IPI) and pointed towards smp_start in (/l0/lrt/bare/arch/amd64/smp.S). The secondary cores are spun up one at a time and each core is responsible for signaling core0 that it is done. This is done through a global counter value modified within lrt_event_init.
The secondary cores begin execution in 16-bit legacy mode with interrupts disabled. alloc a temporary stack for smp bringup
copy code of smp.S to defind point in memory. send 2 ipis, 1) init, 2) start execution (at point in mem)
Each core beings execution in assembly function _smp_start (./smp.S) and will eventually jump to lrt_event_init()
As this is an event driven system, the native state of a core is waiting for an event, hence, the even_loop will dump most of the cores directly into their event loop.