Linux Kernel Debugging and Security (LFD440)


Seminar - Ziel

This instructor-led course focuses on the important tools used for debugging and monitoring the kernel, and how security features are implemented and controlled.

This four day course includes extensive hands-on exercises and demonstrations designed to give you the necessary tools to develop and debug Linux kernel code.

Teilnehmer - Zielgruppe

This course is for experienced developers who need to understand the methods and internal infrastructure of the Linux kernel.

Kurs - Voraussetzungen

To make the most of this course, you should:

  • Be proficient in the C programming language.
  • Be familiar with basic Linux (UNIX) utilities such as ls, grep and tar.
  • Be comfortable using any of the available text editors (e.g. emacs, vi, etc.).
  • Experience with any major Linux distribution is helpful but not strictly required.
  • Have experience equivalent to having taken LFD420: Linux Kernel Internals and Development.

Pre-class preparation material will be provided before class.


  • 4 Tage
  • 09:00 Uhr bis 17:00 Uhr


  • nach Absprache

Seminar-Inhalt / Agenda


  • Objectives
  • Who You Are
  • The Linux Foundation
  • Linux Foundation Training
  • Certification Programs and Digital Badging
  • Linux Distributions
  • Platforms
  • Preparing Your System
  • Using and Downloading a Virtual Machine
  • Things change in Linux
  • Documentation and Links


  • Procedures
  • Kernel Versions
  • Kernel Sources and Use of git

How to Work in OSS Projects **

  • Overview on How to Contribute Properly
  • Stay Close to Mainline for Security and Quality
  • Study and Understand the Project DNA
  • Figure Out What Itch You Want to Scratch
  • Identify Maintainers and Their Work Flows and Methods
  • Get Early Input and Work in the Open
  • Contribute Incremental Bits, Not Large Code Dumps
  • Leave Your Ego at the Door: Don’t Be Thin-Skinned
  • Be Patient, Develop Long Term Relationships, Be Helpful

Kernel Features

  • Components of the Kernel
  • User-Space vs. Kernel-Space
  • What are System Calls?
  • Available System Calls
  • Scheduling Algorithms and Task Structures
  • Process Context
  • Labs

Monitoring and Debugging

  • Debuginfo Packages
  • Tracing and Profiling
  • sysctl
  • SysRq Key
  • oops Messages
  • Kernel Debuggers
  • debugfs
  • Labs

The proc Filesystem **

  • What is the proc Filesystem?
  • Creating and Removing Entries
  • Reading and Writing Entries
  • The seq file Interface **
  • Labs


  • kprobes
  • kretprobes
  • SystemTap **
  • Labs


  • What is ftrace?
  • ftrace, trace-cmd and kernelshark
  • Available Tracers
  • Using ftrace
  • Files in the Tracing Directory
  • Tracing Options
  • Printing with trace printk()
  • Trace Markers
  • Dumping the Buffer
  • trace-cmd
  • Labs


  • What is perf?
  • perf stat
  • perf list
  • perf record
  • perf report
  • perf annotate
  • perf top
  • Labs


  • BFP
  • eBPF
  • Installation
  • bcc Tools
  • bpftrace
  • Labs


  • Crash
  • Main Commands
  • Labs

Kernel Core Dumps

  • Generating Kernel Core Dumps
  • kexec
  • Setting Up Kernel Core Dumps
  • Labs


  • What is Virtualization?
  • Rings of Virtualization
  • Hypervisors


  • What is QEMU?
  • Emulated Architectures
  • Image Formats
  • Third Party Hypervisor Integration
  • Labs

Linux Kernel Debugging Tools

  • Linux Kernel (built-in) tools and helpers
  • kdb
  • qemu+gdb
  • kgdb: hardware+serial+gdb
  • Labs

Embedded Linux**

  • Embedded and Real Time Operating Systems
  • Why Use Linux?
  • Making a Small Linux Environment
  • Real Time Linuxes


  • What are Notifiers?
  • Data Structures
  • Callbacks and Notifications
  • Creating Notifier Chains
  • Labs

CPU Frequency Scaling**

  • What is Frequency and Voltage Scaling?
  • Notifiers
  • Drivers
  • Governors
  • Labs

Netlink Sockets**

  • What are netlink Sockets?
  • Opening a netlink Socket
  • netlink Messages
  • Labs

Introduction to Linux Kernel Security

  • Linux Kernel Security Basics
  • Discretionary Access Control (DAC)
  • POSIX Capabilities
  • Namespaces
  • Linux Security Modules (LSM)
  • Netfilter
  • Cryptographic Methods
  • The Kernel Self Protection Project

Linux Security Modules (LSM)

  • What are Linux Security Modules?
  • LSM Basics
  • LSM Choices
  • How LSM Works
  • An LSM Example: Tomoyo


  • SELinux
  • SELinux Overview
  • SELinux Modes
  • SELinux Policies
  • Context Utilities
  • SELinux and Standard Command Line Tools
  • SELinux Context Inheritance and Preservation**
  • restorecon**
  • semanage fcontext**
  • Using SELinux Booleans**
  • getsebool and setsebool**
  • Troubleshooting Tools
  • Labs


  • What is AppArmor?
  • Checking Status
  • Modes and Profiles
  • Profiles
  • Utilities


  • What is netfilter?
  • Netfilter Hooks
  • Netfilter Implementation
  • Hooking into Netfilter
  • Iptables
  • Labs

The Virtual File System

  • What is the Virtual File System?
  • Available Filesystems
  • Special Filesystems
  • The tmpfs Filesystem
  • The ext2/ext3 Filesystem
  • The ext4 Filesystem
  • The btrfs Filesystem
  • Common File Model
  • VFS System Calls
  • Files and Processes
  • Mounting Filesystems

Filesystems in User-Space (FUSE)**

  • What is FUSE?
  • Writing a Filesystem
  • Labs

Journaling Filesystems**

  • What are Journaling Filesystems?
  • Available Journaling Filesystems
  • Contrasting Features
  • Labs

Closing and Evaluation Survey

  • Evaluation Survey

Kernel Architecture I

  • UNIX and Linux **
  • Monolithic and Micro Kernels
  • Object-Oriented Methods
  • Main Kernel Tasks
  • User-Space and Kernel-Space

Kernel Programming Preview

  • Error Numbers and Getting Kernel Output
  • Task Structure
  • Memory Allocation
  • Transferring Data between User and Kernel Spaces
  • Linked Lists
  • Jiffies
  • Labs


  • What are Modules?
  • A Trivial Example
  • Compiling Modules
  • Modules vs Built-in
  • Module Utilities
  • Automatic Loading/Unloading of Modules
  • Module Usage Count
  • Module Licensing
  • Exporting Symbols
  • Resolving Symbols **
  • Labs

Kernel Architecture II

  • Processes, Threads, and Tasks
  • Kernel Preemption
  • Real Time Preemption Patch
  • Labs

Kernel Configuration and Compilation

  • Installation and Layout of the Kernel Source
  • Kernel Browsers
  • Kernel Configuration Files
  • Kernel Building and Makefiles
  • initrd and initramfs
  • Labs

Kernel Style and General Considerations

  • Coding Style
  • Using Generic Kernel Routines and Methods
  • Making a Kernel Patch
  • sparse
  • Using likely() and unlikely()
  • Writing Portable Code, CPU, 32/64-bit, Endianness
  • Writing for SMP
  • Writing for High Memory Systems
  • Power Management
  • Keeping Security in Mind
  • Labs

Race Conditions and Synchronization Methods

  • Concurrency and Synchronization Methods
  • Atomic Operations
  • Bit Operations
  • Spinlocks
  • Seqlocks
  • Disabling Preemption
  • Mutexes
  • Semaphores
  • Completion Functions
  • Read-Copy-Update (RCU)
  • Reference Counts
  • Labs

Virtual Memory Management

  • Systems With and Without MMU and the TLB
  • Memory Addresses
  • High and Low Memory
  • Memory Zones
  • Special Device Nodes
  • NUMA
  • Paging
  • Page Tables
  • page structure
  • Labs

Memory Allocation

  • Requesting and Releasing Pages
  • Buddy System
  • Slabs and Cache Allocations
  • Memory Pools
  • kmalloc()
  • vmalloc()
  • Early Allocations and bootmem()
  • Memory Defragmentation
  • Labs

Weitere Schulungen zu Thema Linux Foundation

Kubernetes for App Developers (LFD459)

- u.a. in Nürnberg, Berlin, Stuttgart, München, Köln

This course will teach you how to containerize, host, deploy, and configure an application in a multi-node cluster. It also serves as preparation for the Certified Kubernetes Application Developer (CKAD) exam. Starting with a simple Python script, this course will show you how ...

ONAP Essentials (LFS463)

- u.a. in Nürnberg, Berlin, Stuttgart, München, Köln

This course aims to provide you the conceptual and hands-on skills around ONAP, focusing -on: -The basics of Network Function Virtualization (NFV) -An introduction to The Linux Foundation ONAP project -Overview of the ONAP project’s architecture, subprojects and demos In ...

Linux for System Administrators (LFS301)

- u.a. in Köln, Paderborn, Dresden, Darmstadt, Freiburg

Linux system administration is one of the most in-demand skills in IT. Whether you’re looking for expert prep for the Linux Foundation Certified System Administration (LFCS) certification, need training to help start a new Linux IT career, transition to Linux from another ...

Linux Kernel Internals and Development (LFD420)

- u.a. in Bremen, Hannover, Leipzig, Paderborn, Koblenz

Learn how to develop for the Linux kernel. In this instructor-led course you’ll learn how Linux is architected, the basic methods for developing on the kernel, and how to efficiently work with the Linux developer community. If you are interested in learning about the Linux ...