book

Linux Kernel Debugging

by Kaiwan N. Billimoria

August 2022

Intermediate to advanced

638 pages

13h 58m

English

Packt Publishing

Read now

Unlock full access

ContributorsAbout the authorAbout the reviewer
Who this book is forWhat this book coversTo get the most out of this bookDownload the example code filesDownload the color imagesConventions usedGet in touchShare Your Thoughts
Technical requirementsCloning this book's code repositorySoftware debugging – what it is, origins, and mythsSoftware bugs – a few actual casesPatriot missile failureThe ESA's unmanned Ariane 5 rocketMars Pathfinder reset issueThe Boeing 737 MAX aircraft – the MCAS and lack of training of the flight crewOther casesSetting up the workspaceRunning Linux as a native or guest OSRunning Linux as a guest OSInstalling the Oracle VirtualBox guest additionsInstalling required software packagesA tale of two kernelsA production and a debug kernelSetting up our custom production kernelSetting up our custom debug kernelSeeing the difference – production and debug kernel configDebugging – a few quick tipsA programmer's checklist – seven rulesSummaryFurther reading
Technical requirementsClassifying bug typesTypes of bugs – the classic viewTypes of bugs – the memory viewTypes of bugs – the CVE/CWE security-related viewTypes of bugs – the Linux kernelKernel debugging – why there are different approaches to itSummarizing the different approaches to kernel debuggingThe development phaseUnit testing and/or QA phasesCategorizing into different scenariosSummaryFurther reading
Technical requirementsThe ubiquitous kernel printkUsing the printk API's logging levelsLeveraging the pr_<foo> convenience macrosUnderstanding where the printk output goesPractically using the printk format specifiers – a few quick tipsLeveraging the printk for debug purposesWriting debug messages to the kernel logDebug printing – quick and useful tipsDevice drivers – using the dev_dbg() macroTrying our kernel module on the custom production kernelRate limiting the printkUsing the kernel's powerful dynamic debug featureDynamic debugging via module parametersSpecifying what and how to print debug messagesExercising dynamic debugging on a kernel module on a production kernelRemaining printk miscellanyPrinting before console init – the early printkDesignating the printk to some known presets Printing exactly onceEmitting a printk from userspaceEasily dumping buffer contentRemaining points – bootloader log peeking, LED flashing, and moreSummaryFurther reading
Understanding kprobes basicsWhat we intend to doUsing static kprobes – traditional approaches to probingDemo 1 – static kprobe – trapping into the file open the traditional static kprobes way – simplest caseDemo 2 – static kprobe – specifying the function to probe via a module parameterUnderstanding the basics of the Application Binary Interface (ABI)Using static kprobes – demo 3 and demo 4Demo 3 – static kprobe – probing the file open syscall and retrieving the filename parameterDemo 4 – semi-automated static kprobe via our helper scriptGetting started with kretprobesKprobes miscellanyKprobes – limitations and downsidesInterface stabilityThe easier way – dynamic kprobes or kprobe-based event tracingKprobe-based event tracing – minimal internal detailsSetting up a dynamic kprobe (via kprobe events) on any functionUsing dynamic kprobe event tracing on a kernel moduleSetting up a return probe (kretprobe) with kprobe-perfTrapping into the execve() API – via perf and eBPF toolingSystem calls and where they land in the kernelObservability with eBPF tools – an introductionSummaryFurther reading
Technical requirementsWhat's the problem with memory anyway?Tools to catch kernel memory issues – a quick summaryUsing KASAN and UBSAN to find memory bugsUnderstanding KASAN – the basicsRequirements to use KASANConfiguring the kernel for Generic KASAN modeBug hunting with KASANUsing the UBSAN kernel checker to find Undefined BehaviorBuilding your kernel and modules with ClangUsing Clang 13 on Ubuntu 21.10Catching memory defects in the kernel – comparisons and notes (Part 1)Miscellaneous notesSummaryFurther reading
Technical requirementsDetecting slab memory corruption via SLUB debugConfiguring the kernel for SLUB debugLeveraging SLUB debug features via the slub_debug kernel parameterRunning and tabulating the SLUB debug test casesInterpreting the kernel's SLUB debug error reportLearning how to use the slabinfo and related utilitiesFinding memory leakage issues with kmemleakConfiguring the kernel for kmemleakUsing kmemleakA few tips for developers regarding dynamic kernel memory allocationCatching memory defects in the kernel – comparisons and notes (Part 2)Miscellaneous notesSummaryFurther reading

Technical requirementsGenerating a simple kernel bug and OopsThe procmap utilityWhat's this NULL trap page anyway?A simple Oops v1 – dereferencing the NULL pointerDoing a bit more of an Oops – our buggy module v2A kernel Oops and what it signifiesThe devil is in the details – decoding the OopsLine-by-line interpretation of an OopsTools and techniques to help determine the location of the OopsUsing objdump to help pinpoint the Oops code locationUsing GDB to help debug the OopsUsing addr2line to help pinpoint the Oops code locationTaking advantage of kernel scripts to help debug kernel issuesLeveraging the console device to get the kernel log after Oopsing in IRQ contextAn Oops on an ARM Linux system and using netconsoleFiguring out the actual buggy code location (on ARM)A few actual OopsesSummaryFurther reading
Technical requirementsLocking and lock debuggingLocking – a quick summarization of key pointsUnderstanding data races – delving deeperCatching concurrency bugs with KCSANWhat KCSAN does, in a nutshellConfiguring the kernel for KCSANUsing KCSANKnee-jerk reactions to KCSAN reports – please don't!A few actual use cases of kernel bugs due to locking defectsDefects identified by KCSANIdentifying locking rules and bugs from the LDV projectIdentifying locking bugs from the Linux kernel BugzillaIdentifying some locking defects from various blog articles and the likeSummaryFurther reading
Technical requirementsKernel tracing technology – an overviewUsing the ftrace kernel tracerAccessing ftrace via the filesystemConfiguring the kernel for ftraceUsing ftrace to trace the flow of the kernelUseful ftrace filtering optionsCase 1 – tracing a single ping with raw ftraceCase 2 – tracing a single ping with raw ftrace via the set_event interfaceUsing trace_printk() for debuggingFtrace – miscellaneous remaining points via FAQsFtrace use casesUsing the trace-cmd, KernelShark, and perf-tools ftrace frontendsAn introduction to using trace-cmdUsing the KernelShark GUIAn introduction to using perf-toolsAn introduction to kernel tracing with LTTng and Trace CompassA quick introduction to recording a kernel tracing session with LTTngUsing the Trace Compass GUI to visualize the single ping LTTng traceSummaryFurther reading
Technical requirementsPanic! – what happens when a kernel panicsLet's panicTo the rescue with netconsoleInterpreting the panic outputKernel parameters, tunables, and configs that affect kernel panicWriting a custom kernel panic handler routineLinux kernel panic notifier chains – the basicsSetting up our custom panic handler within a moduleDetecting lockups and CPU stalls in the kernelA short note on watchdogsEmploying the kernel's hard and soft lockup detectorEmploying the kernel's hung task and workqueue stall detectorsLeveraging the kernel hung task detectorDetecting workqueue stallsSummaryFurther reading
Technical requirementsConceptually understanding how KGDB worksSetting up an ARM target system and kernel for KGDBBuilding a minimal custom ARM Linux target system with SEALSConfiguring the kernel for KGDBTesting the target systemDebugging the kernel with KGDBRunning our target (emulated) ARM32 systemRunning and working with the remote GDB client on the host systemDebugging kernel modules with KGDBInforming the GDB client about the target module's locations in memoryStep by step – debugging a buggy module with KGDB[K]GDB – a few tips and tricksSetting up and using GDB scripts with CONFIG_GDB_SCRIPTSKGDB target remote :1234 command doesn't work on physical systemsSetting the system root with sysrootUsing GDB's TUI modeWhat to do when the <value optimized out> GDB response occursGDB convenience routinesGDB custom macros in its startup fileFancy breakpoints and hardware watchpointsMiscellaneous GDB tipsSummaryFurther reading
An introduction to the kdump/crash frameworkWhy use kdump/crash?Understanding the kdump/crash basic frameworkA mention on performing static analysis on kernel codeExamples using cppcheck and checkpatch.pl for static analysisAn introduction to kernel code coverage tools and testing frameworksWhy is code coverage important?A brief note on kernel testingMiscellaneous – using journalctl, assertions, and warningsLooking up system logs with journalctlAssertions, warnings, and BUG() macrosSummaryFurther reading
Other Books You May EnjoyPackt is searching for authors like youShare Your Thoughts

Content preview from Linux Kernel Debugging

Chapter 10: Kernel Panic, Lockups, and Hangs

It's unpleasant – that queasy feeling deep in the pit of your stomach, the cold sweat forming on your brow – when you get that dreaded kernel panic message on the console, and those absolute, unforgiving pixels, with the hard cold eye of a god, tell you that the system is effectively dead:

Kernel panic - not syncing: [...]

Why, oh why? – your lamentations are futile. Unless, unless... you don't panic (pun intended), read this chapter, figure out what's going on (by writing your own custom panic handler to help with that), and get on with your life, dude!

In addition to understanding and dealing with kernel panics, we also delve into the causes of kernel lockups, hung tasks, stalls, and how to configure ...