Skip to content

Commit 9f60095

Browse files
committed
crashdump: enable crashdump feature for release builds
- Improve documentation and describe how an user can create separate files for debug information - Change the output file directory to be configurable using HYPERLIGHT_CORE_DUMP_DIR environment variable - Change output file name to include a timestamp Signed-off-by: Doru Blânzeanu <[email protected]>
1 parent 9fcf76c commit 9f60095

File tree

5 files changed

+184
-20
lines changed

5 files changed

+184
-20
lines changed

Cargo.lock

Lines changed: 2 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/how-to-debug-a-hyperlight-guest.md

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -275,3 +275,149 @@ To do this in vscode, the following configuration can be used to add debug confi
275275
press the `pause` button. This is a known issue with the `CodeLldb` extension [#1245](https://github.com/vadimcn/codelldb/issues/1245).
276276
The `cppdbg` extension works as expected and stops at the entry point of the program.**
277277

278+
## Compiling guests with debug information for release builds
279+
280+
This section explains how to compile a guest with debugging information but still have optimized code, and how to separate the debug information from the binary.
281+
282+
### Creating a release build with debug information
283+
284+
To create a release build with debug information, you can add a custom profile to your `Cargo.toml` file:
285+
286+
```toml
287+
[profile.release-with-debug]
288+
inherits = "release"
289+
debug = true
290+
```
291+
292+
This creates a new profile called `release-with-debug` that inherits all settings from the release profile but adds debug information.
293+
294+
### Splitting debug information from the binary
295+
296+
To reduce the binary size while still having debug information available, you can split the debug information into a separate file.
297+
This is useful for production environments where you want smaller binaries but still want to be able to debug crashes.
298+
299+
Here's a step-by-step guide:
300+
301+
1. Build your guest with the release-with-debug profile:
302+
```bash
303+
cargo build --profile release-with-debug
304+
```
305+
306+
2. Locate your binary in the target directory:
307+
```bash
308+
TARGET_DIR="target"
309+
PROFILE="release-with-debug"
310+
ARCH="x86_64-unknown-none" # Your target architecture
311+
BUILD_DIR="${TARGET_DIR}/${ARCH}/${PROFILE}"
312+
BINARY=$(find "${BUILD_DIR}" -type f -executable -name "guest-binary" | head -1)
313+
```
314+
315+
3. Extract debug information into a full debug file:
316+
```bash
317+
DEBUG_FILE_FULL="${BINARY}.debug.full"
318+
objcopy --only-keep-debug "${BINARY}" "${DEBUG_FILE_FULL}"
319+
```
320+
321+
4. Create a symbols-only debug file (smaller, but still useful for stack traces):
322+
```bash
323+
DEBUG_FILE="${BINARY}.debug"
324+
objcopy --keep-file-symbols "${DEBUG_FILE_FULL}" "${DEBUG_FILE}"
325+
```
326+
327+
5. Strip debug information from the original binary but keep function names:
328+
```bash
329+
objcopy --strip-debug "${BINARY}"
330+
```
331+
332+
6. Add a debug link to the stripped binary:
333+
```bash
334+
objcopy --add-gnu-debuglink="${DEBUG_FILE}" "${BINARY}"
335+
```
336+
337+
After these steps, you'll have:
338+
- An optimized binary with function names for basic stack traces
339+
- A symbols-only debug file for stack traces
340+
- A full debug file for complete source-level debugging
341+
342+
### Analyzing core dumps with the debug files
343+
344+
When you have a core dump from a crashed guest, you can analyze it with different levels of detail using either GDB or LLDB.
345+
346+
#### Using GDB
347+
348+
1. For basic analysis with function names (stack traces):
349+
```bash
350+
gdb ${BINARY} -c /path/to/core.dump
351+
```
352+
353+
2. For full source-level debugging:
354+
```bash
355+
gdb -s ${DEBUG_FILE_FULL} ${BINARY} -c /path/to/core.dump
356+
```
357+
358+
#### Using LLDB
359+
360+
LLDB provides similar capabilities with slightly different commands:
361+
362+
1. For basic analysis with function names (stack traces):
363+
```bash
364+
lldb ${BINARY} -c /path/to/core.dump
365+
```
366+
367+
2. For full source-level debugging:
368+
```bash
369+
lldb -o "target create -c /path/to/core.dump ${BINARY}" -o "add-dsym ${DEBUG_FILE_FULL}"
370+
```
371+
372+
3. If your debug symbols are in a separate file:
373+
```bash
374+
lldb ${BINARY} -c /path/to/core.dump
375+
(lldb) add-dsym ${DEBUG_FILE_FULL}
376+
```
377+
378+
### VSCode Debug Configurations
379+
380+
You can configure VSCode (in `.vscode/launch.json`) to use these files by modifying the debug configurations:
381+
382+
#### For GDB
383+
384+
```json
385+
{
386+
"name": "[GDB] Load core dump with full debug symbols",
387+
"type": "cppdbg",
388+
"request": "launch",
389+
"program": "${input:program}",
390+
"coreDumpPath": "${input:core_dump}",
391+
"cwd": "${workspaceFolder}",
392+
"MIMode": "gdb",
393+
"externalConsole": false,
394+
"miDebuggerPath": "/usr/bin/gdb",
395+
"setupCommands": [
396+
{
397+
"description": "Enable pretty-printing for gdb",
398+
"text": "-enable-pretty-printing",
399+
"ignoreFailures": true
400+
}
401+
]
402+
}
403+
```
404+
405+
#### For LLDB
406+
407+
```json
408+
{
409+
"name": "[LLDB] Load core dump with full debug symbols",
410+
"type": "lldb",
411+
"request": "launch",
412+
"program": "${input:program}",
413+
"cwd": "${workspaceFolder}",
414+
"processCreateCommands": [],
415+
"targetCreateCommands": [
416+
"target create -c ${input:core_dump} ${input:program}"
417+
],
418+
"postRunCommands": [
419+
// if debug symbols are in a different file
420+
"add-dsym ${input:debug_file_path}"
421+
]
422+
}
423+
```

src/hyperlight_host/Cargo.toml

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -38,10 +38,10 @@ vmm-sys-util = "0.13.0"
3838
crossbeam = "0.8.0"
3939
crossbeam-channel = "0.5.15"
4040
thiserror = "2.0.12"
41-
tempfile = { version = "3.20", optional = true }
41+
chrono = { version = "0.4", optional = true }
4242
anyhow = "1.0"
4343
metrics = "0.24.2"
44-
elfcore = { git = "https://github.com/dblnz/elfcore.git", rev = "1a57f04272dd54bc06df638f4027debe6d0f694f" }
44+
elfcore = { git = "https://github.com/hyperlight-dev/elfcore.git", rev = "cef4c80e26bf4b2a5599e50d2d1730965f942c13" }
4545

4646
[target.'cfg(windows)'.dependencies]
4747
windows = { version = "0.61", features = [
@@ -123,7 +123,8 @@ function_call_metrics = []
123123
executable_heap = []
124124
# This feature enables printing of debug information to stdout in debug builds
125125
print_debug = []
126-
crashdump = ["dep:tempfile"] # Dumps the VM state to a file on unexpected errors or crashes. The path of the file will be printed on stdout and logged. This feature can only be used in debug builds.
126+
# Dumps the VM state to a file on unexpected errors or crashes. The path of the file will be printed on stdout and logged.
127+
crashdump = ["dep:chrono"]
127128
kvm = ["dep:kvm-bindings", "dep:kvm-ioctls"]
128129
mshv2 = ["dep:mshv-bindings2", "dep:mshv-ioctls2"]
129130
mshv3 = ["dep:mshv-bindings3", "dep:mshv-ioctls3"]

src/hyperlight_host/build.rs

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,8 +93,7 @@ fn main() -> Result<()> {
9393
gdb: { all(feature = "gdb", debug_assertions, any(feature = "kvm", feature = "mshv2", feature = "mshv3"), target_os = "linux") },
9494
kvm: { all(feature = "kvm", target_os = "linux") },
9595
mshv: { all(any(feature = "mshv2", feature = "mshv3"), target_os = "linux") },
96-
// crashdump feature is aliased with debug_assertions to make it only available in debug-builds.
97-
crashdump: { all(feature = "crashdump", debug_assertions) },
96+
crashdump: { all(feature = "crashdump") },
9897
// print_debug feature is aliased with debug_assertions to make it only available in debug-builds.
9998
print_debug: { all(feature = "print_debug", debug_assertions) },
10099
// the following features are mutually exclusive but rather than enforcing that here we are enabling mshv3 to override mshv2 when both are enabled

src/hyperlight_host/src/hypervisor/crashdump.rs

Lines changed: 31 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
use std::cmp::min;
22

3+
use chrono;
34
use elfcore::{
45
ArchComponentState, ArchState, CoreDumpBuilder, CoreError, Elf64_Auxv, ProcessInfoSource,
56
ReadProcessMemory, ThreadView, VaProtection, VaRegion,
67
};
7-
use tempfile::NamedTempFile;
88

99
use super::Hypervisor;
1010
use crate::mem::memory_region::{MemoryRegion, MemoryRegionFlags};
@@ -87,12 +87,12 @@ impl GuestView {
8787
let filename = ctx
8888
.filename
8989
.as_ref()
90-
.map_or(|| "<unknown>".to_string(), |s| s.to_string());
90+
.map_or("<unknown>".to_string(), |s| s.to_string());
9191

9292
let cmd = ctx
9393
.binary
9494
.as_ref()
95-
.map_or(|| "<unknown>".to_string(), |s| s.to_string());
95+
.map_or("<unknown>".to_string(), |s| s.to_string());
9696

9797
// The xsave state is checked as it can be empty
9898
let mut components = vec![];
@@ -246,10 +246,6 @@ impl ReadProcessMemory for GuestMemReader {
246246
pub(crate) fn crashdump_to_tempfile(hv: &dyn Hypervisor) -> Result<()> {
247247
log::info!("Creating core dump file...");
248248

249-
// Create a temporary file with a recognizable prefix
250-
let temp_file = NamedTempFile::with_prefix("hl_core_")
251-
.map_err(|e| new_error!("Failed to create temporary file: {:?}", e))?;
252-
253249
// Get crash context from hypervisor
254250
let ctx = hv
255251
.crashdump_context()
@@ -262,16 +258,37 @@ pub(crate) fn crashdump_to_tempfile(hv: &dyn Hypervisor) -> Result<()> {
262258
// Create and write core dump
263259
let core_builder = CoreDumpBuilder::from_source(guest_view, memory_reader);
264260

261+
// Generate timestamp string for the filename using chrono
262+
let timestamp = chrono::Local::now().format("%Y%m%d_%H%M%S").to_string();
263+
264+
// Determine the output directory based on environment variable
265+
let output_dir = if let Ok(dump_dir) = std::env::var("HYPERLIGHT_CORE_DUMP_DIR") {
266+
// Create the directory if it doesn't exist
267+
let path = std::path::Path::new(&dump_dir);
268+
if !path.exists() {
269+
std::fs::create_dir_all(path)
270+
.map_err(|e| new_error!("Failed to create core dump directory: {:?}", e))?;
271+
}
272+
std::path::PathBuf::from(dump_dir)
273+
} else {
274+
// Fall back to the system temp directory
275+
std::env::temp_dir()
276+
};
277+
278+
// Create the filename with timestamp
279+
let filename = format!("hl_core_{}.elf", timestamp);
280+
let file_path = output_dir.join(filename);
281+
282+
// Create the file
283+
let file = std::fs::File::create(&file_path)
284+
.map_err(|e| new_error!("Failed to create core dump file: {:?}", e))?;
285+
286+
// Write the core dump directly to the file
265287
core_builder
266-
.write(&temp_file)
288+
.write(&file)
267289
.map_err(|e| new_error!("Failed to write core dump: {:?}", e))?;
268290

269-
let persist_path = temp_file.path().with_extension("elf");
270-
temp_file
271-
.persist(&persist_path)
272-
.map_err(|e| new_error!("Failed to persist core dump file: {:?}", e))?;
273-
274-
let path_string = persist_path.to_string_lossy().to_string();
291+
let path_string = file_path.to_string_lossy().to_string();
275292

276293
println!("Core dump created successfully: {}", path_string);
277294
log::error!("Core dump file: {}", path_string);

0 commit comments

Comments
 (0)