KConfigTune

KConfigTune is an AI-driven Linux kernel configuration optimizer. It uses workload profiling, a neural performance surrogate, and reinforcement learning to suggest kernel .config options that are expected to reduce latency for a specific workload.

In simple terms: KConfigTune observes how an application behaves, learns how kernel configuration choices affect performance, then generates a tuned Linux kernel configuration for that application.

This Project was done by the following Students of Higher Management Studies Buea

• ELEH GILBERT KOME MATID: UBA/HIMS/25E/020
• STEPHEN EMMANGHE OTANG MATID: UBa/HIMs/25E/026
• FAVOUR CHELSEA YANKEH LAMBI: UBa/HIMS/235E/027

Table of Contents

• Project Purpose
• What Problem This Project Solves
• How KConfigTune Works
• Core Concepts Explained
• Project Architecture
• Installation
• Step-by-Step Usage Guide
• Command Reference
• Data Formats
• Folder Structure
• Production and Safety Notes
• Development Guide
• Presentation Summary
• Research Links

Project Purpose

Linux kernels are highly configurable. A kernel configuration file, normally called .config, controls thousands of options such as CPU scheduling, memory-management behavior, networking algorithms, filesystem support, tracing, debug features, and security mitigations.

Different workloads benefit from different kernel settings:

• A database may care more about I/O latency, memory behavior, and NUMA.
• A web server may care more about networking, CPU scheduling, and throughput.
• An AI inference workload may care more about CPU utilization, memory bandwidth, and accelerator-related options.

Manually searching for a good kernel configuration is difficult because:

• The search space is large.
• Kernel options can depend on each other.
• Building and testing a kernel is slow.
• A setting that helps one workload can hurt another.

KConfigTune automates part of this process by combining:

• Workload profiling
• Machine learning
• Reinforcement learning
• Kernel configuration validation
• Optional real kernel compilation and QEMU boot verification

What Problem This Project Solves

The main problem is kernel configuration optimization.

Instead of using a one-size-fits-all kernel configuration, KConfigTune tries to produce a workload-aware configuration. The goal is to make the kernel better matched to the application that will run on it.

The project focuses on 47 selected Linux kernel options. These options were chosen because they commonly affect performance-sensitive areas such as:

• Scheduling and latency
• Memory management
• CPU frequency and power behavior
• I/O schedulers
• NUMA behavior
• Networking
• Filesystems
• GPU and virtualization support
• Security mitigations
• Tracing and debugging

KConfigTune does not attempt to tune every Linux kernel option. Instead, it uses a controlled set of important options so the learning problem stays practical and explainable.

How KConfigTune Works

KConfigTune has two execution paths:

• A fast learning path
• A slow validation path

The fast path is used during AI training. It avoids compiling the kernel at every step by using a neural surrogate model to predict latency.

The slow path is used when real validation is needed. It compiles an actual kernel, optionally boots it in QEMU, and can be used to collect real performance data.

High-Level Workflow

1. Profile workload
       |
       v
2. Convert workload behavior into a 6-value feature vector
       |
       v
3. Train or load surrogate latency model
       |
       v
4. Train PPO reinforcement-learning agent
       |
       v
5. Generate optimized kernel configuration
       |
       v
6. Validate and write .config
       |
       v
7. Optionally compile and boot-test the kernel

Step 1: Profile the Workload

The profiler uses Linux perf stat counters and system statistics to describe the workload numerically.

It produces a 6-dimensional workload feature vector:

Feature	Meaning	Why it matters
ipc	Instructions per cycle	Shows how efficiently the CPU is executing instructions.
llc_miss_rate	Last-level cache miss rate	High values suggest memory/cache pressure.
branch_miss_rate	Branch misprediction rate	High values can indicate control-flow inefficiency.
io_wait_pct	CPU time waiting on I/O	Important for databases and storage-heavy applications.
mem_bandwidth_norm	Normalized memory bandwidth estimate	Helps identify memory-intensive workloads.
cpu_util	CPU utilization	Shows how busy the CPU is during the workload.

This vector is the workload fingerprint used by the rest of the system.

Step 2: Represent Kernel Options as a Vector

The project tracks 47 kernel options. Each option is represented as one binary value:

• 1 means the option is enabled.
• 0 means the option is disabled.

Together, these 47 values form the kernel configuration vector.

Step 3: Build the State Vector

The reinforcement-learning environment combines:

• 6 workload features
• 47 kernel option values

This creates a 53-dimensional state vector:

state = workload_features(6) + kernel_config_vector(47)
state_dimension = 53

The agent reads this state and decides which kernel option to flip.

Step 4: Predict Latency With a Surrogate Model

Compiling and benchmarking a Linux kernel can take many minutes. Reinforcement learning may need thousands or millions of environment interactions, so using real compilation for every step would be too slow.

KConfigTune solves this with a neural surrogate model.

A surrogate model is a machine-learning model that approximates an expensive real process. In this project, the expensive process is:

kernel config + workload -> compile kernel -> run benchmark -> measure latency

The surrogate learns the shorter approximation:

kernel config + workload -> predicted latency

The model architecture is a multilayer perceptron:

Input:  53 values
Hidden: 512 -> 256 -> 128
Output: 1 predicted latency value in milliseconds

Step 5: Train the PPO Agent

KConfigTune uses PPO, which stands for Proximal Policy Optimization.

PPO is a reinforcement-learning algorithm. Reinforcement learning trains an agent to make decisions by giving it rewards for useful behavior and penalties for poor behavior.

In this project:

• The agent sees the current workload and kernel configuration.
• The agent chooses one of 47 actions.
• Each action flips one kernel option on or off.
• The surrogate predicts the new latency.
• The environment gives a reward based on latency improvement.

The reward is approximately:

reward = latency improvement - cost for changing options

The cost discourages unnecessary configuration changes.

Step 6: Generate a Kernel .config

After training, the agent can generate a configuration for a profiled workload. The generated vector is passed to KConfigManager, which writes a .config file.

If a Linux kernel source tree is provided, KConfigTune can also validate some configuration dependencies before writing the file.

Step 7: Apply and Verify

The optional apply command copies the generated .config into a kernel source tree, runs make olddefconfig, compiles the kernel, and can optionally boot-test the result in QEMU.

This is the slow path and should be treated carefully, especially if using the --install option.

Core Concepts Explained

Linux Kernel

The Linux kernel is the core layer between hardware and software. It manages CPU scheduling, memory, filesystems, networking, devices, security boundaries, and many low-level system behaviors.

Kernel Configuration

A kernel configuration is a set of options that decide which features are included in a kernel and how certain kernel behaviors are enabled.

The configuration is stored in a file named .config.

Example:

CONFIG_PREEMPT=y
# CONFIG_DEBUG_KERNEL is not set
CONFIG_TCP_CONG_BBR=y

Kconfig

Kconfig is the configuration system used by the Linux kernel. It defines available options, option types, dependencies, defaults, and relationships between symbols.

KConfigTune uses kconfiglib to work with Kconfig data when a kernel source tree is available.

Workload

A workload is the real application or benchmark being optimized for. Examples:

• PostgreSQL database workload
• Nginx or HTTP server workload
• AI inference process
• Generic CPU benchmark

KConfigTune does not tune for an abstract machine only. It tunes based on measured workload behavior.

Profiling

Profiling means measuring how a program behaves while it runs. KConfigTune uses profiling to turn a workload into measurable features such as CPU utilization, cache misses, branch misses, and I/O wait.

Feature Vector

A feature vector is a list of numbers that represents something in a form a machine-learning model can understand.

For KConfigTune, the workload feature vector has 6 values.

Surrogate Model

A surrogate model is a fast approximation of a slow process.

Here, it predicts latency without compiling and benchmarking a real kernel for every possible configuration.

Reinforcement Learning

Reinforcement learning trains an agent through interaction. The agent chooses actions, receives rewards, and gradually learns which actions are useful.

In KConfigTune, the agent learns which kernel options to enable or disable.

PPO

PPO, or Proximal Policy Optimization, is a reinforcement-learning algorithm designed to update an agent carefully. It improves the policy while limiting overly large policy changes that can destabilize training.

Actor-Critic

The PPO agent uses an actor-critic neural network.

• The actor decides which action to take.
• The critic estimates how good the current state is.

In this project:

• Actor output: probability distribution over 47 kernel-option actions.
• Critic output: estimated value of the current state.

Reward

The reward tells the agent whether an action helped.

KConfigTune rewards lower predicted latency and penalizes unnecessary option flips. This encourages useful, minimal configuration changes.

QEMU

QEMU is a machine emulator. KConfigTune can use it to boot-test a compiled kernel before installing it on real hardware.

This is safer than immediately booting a newly generated kernel on a production machine.

Project Architecture

KConfigTune follows a modular architecture. Each module has a clear responsibility and belongs to a specific domain.

kconfigtune/
  cli.py                  Command-line interface
  config.py               Shared constants and hyperparameter dataclasses

  profiler/
    workload_profiler.py  Workload measurement and feature extraction

  surrogate/
    dataset.py            Training-pair storage and loading
    surrogate_model.py    Neural latency prediction model
    surrogate_trainer.py  Training loop for the surrogate model

  agent/
    environment.py        Gymnasium environment for config optimization
    actor_critic.py       PPO policy and value neural network
    ppo.py                PPO training loop
    replay_buffer.py      Rollout storage and advantage calculation

  kernel/
    kconfig_manager.py    Conversion between vectors and .config files
    kernel_compiler.py    Kernel compilation and QEMU verification

  benchmarks/
    benchmark_runner.py   Benchmark wrappers for real performance testing

Separation of Concerns

The project follows separation of concerns:

• Profiling code only measures workloads.
• Surrogate code only manages latency prediction.
• Agent code only handles reinforcement learning.
• Kernel code only handles .config, compilation, and validation.
• Benchmark code only runs and parses benchmarks.
• CLI code only connects user commands to the correct modules.

This makes the project easier to test, explain, maintain, and extend.

Installation

Requirements

Recommended environment:

• Python 3.11 or newer
• Linux for real profiling and kernel compilation
• perf for workload profiling
• Linux kernel source tree for real .config validation and compilation
• Optional NVIDIA CUDA GPU for faster training
• QEMU for boot verification

Python dependencies are declared in pyproject.toml.

Important packages include:

• torch for neural networks
• numpy and pandas for data handling
• gymnasium for the reinforcement-learning environment interface
• kconfiglib for Linux Kconfig handling
• tensorboard for training logs
• rich for readable CLI output

Install in Editable Mode

From the project root:

pip install -e .

For development tools:

pip install -e ".[dev]"

If you want optional eBPF support:

pip install -e ".[ebpf]"

Note: eBPF support may also require system-level BCC packages.

Verify the CLI

kconfigtune --help

You should see commands such as:

• profile
• train
• generate
• apply
• collect

Step-by-Step Usage Guide

This section shows the normal end-to-end workflow.

Step 1: Profile a Running Process

If the workload is already running and you know its process ID:

kconfigtune profile --pid 1234 --duration 30 --output workload.json

What this does:

• Attaches perf to process 1234
• Measures the process for 30 seconds
• Extracts CPU, cache, branch, memory, and I/O features
• Saves the workload fingerprint to workload.json

Step 2: Profile a Command

If you want KConfigTune to launch the workload:

kconfigtune profile --duration 30 --output workload.json --command python benchmark.py

What this does:

• Starts the given command
• Profiles it while it runs
• Saves the workload features

Step 3: Train the Agent

kconfigtune train \
  --workload workload.json \
  --episodes 500 \
  --checkpoint-dir checkpoints \
  --log-dir logs \
  --device cpu

What this does:

• Loads the workload profile
• Loads training data from data/training_pairs
• If no real data exists, creates synthetic bootstrap data
• Trains or loads the surrogate model
• Creates the reinforcement-learning environment
• Trains the PPO agent
• Saves model checkpoints

Expected outputs:

checkpoints/surrogate.pt
checkpoints/agent_final.pt
logs/ppo/

Use --device cuda if CUDA is available and configured.

Step 4: Generate an Optimized Kernel Configuration

kconfigtune generate \
  --workload workload.json \
  --checkpoint-dir checkpoints \
  --output optimized.config \
  --device cpu

What this does:

• Loads the trained surrogate model
• Loads the trained PPO agent
• Reads the workload profile
• Uses the agent to choose kernel options
• Writes an optimized .config fragment or full config

If you have a Linux kernel source tree, pass it for better validation:

kconfigtune generate \
  --workload workload.json \
  --checkpoint-dir checkpoints \
  --kernel-src /usr/src/linux \
  --output optimized.config

Step 5: Compile the Kernel

kconfigtune apply \
  --config optimized.config \
  --kernel-src /usr/src/linux \
  --output-dir build-output

What this does:

• Copies optimized.config into the kernel source tree as .config
• Runs make olddefconfig
• Runs a parallel kernel build
• Copies the resulting bzImage to the output directory

Step 6: Boot-Test With QEMU

kconfigtune apply \
  --config optimized.config \
  --kernel-src /usr/src/linux \
  --output-dir build-output \
  --qemu-image /path/to/initrd.img \
  --verify-qemu

What this does:

• Compiles the kernel
• Starts QEMU with the generated kernel image
• Checks whether the kernel reaches a bootable state

Step 7: Install the Kernel

kconfigtune apply \
  --config optimized.config \
  --kernel-src /usr/src/linux \
  --install

Use this only on a dedicated test machine. Installing kernels can modify /boot and bootloader configuration.

Command Reference

profile

Profiles a workload and writes a workload feature JSON file.

kconfigtune profile [--pid PID | --command COMMAND...] [--duration SECONDS] [--output PATH]

Common options:

Option	Purpose
--pid	Attach to an already-running process.
--command	Launch and profile a command.
--duration	Number of seconds to profile.
--output	Path to save workload features.

train

Trains the surrogate model and PPO agent.

kconfigtune train --workload workload.json --episodes 500

Common options:

Option	Purpose
--workload	Workload profile JSON from profile.
--episodes	Number of PPO training episodes.
--steps-per-episode	Number of option flips per episode.
--data-dir	Directory containing real training pairs.
--checkpoint-dir	Directory for saved model weights.
--log-dir	Directory for TensorBoard logs.
--device	cpu or cuda.
--retrain-surrogate	Force surrogate retraining.

generate

Generates a tuned kernel configuration.

kconfigtune generate --workload workload.json --output optimized.config

Common options:

Option	Purpose
--workload	Workload profile JSON.
--output	Path for generated .config.
--checkpoint-dir	Directory containing surrogate.pt and agent_final.pt.
--kernel-src	Optional kernel tree for validation.
--steps	Number of greedy inference steps.
--device	cpu or cuda.

apply

Compiles a kernel using the generated configuration.

kconfigtune apply --config optimized.config --kernel-src /usr/src/linux

Common options:

Option	Purpose
--config	Generated .config file.
--kernel-src	Linux kernel source tree.
--output-dir	Directory where bzImage should be copied.
--qemu-image	Initrd/root filesystem for QEMU boot test.
--verify-qemu	Boot-test the compiled kernel in QEMU.
--install	Install the kernel. Use with caution.

collect

Collects real training pairs by compiling and benchmarking configurations.

kconfigtune collect \
  --n-samples 50 \
  --kernel-src /usr/src/linux \
  --workload workload.json \
  --output-dir data/training_pairs

This is expected to be slow because it uses real kernel compilation and real benchmarking.

Note: the current CLI delegates this command to scripts/collect_training_data.py. Ensure that script exists in your working tree before using collect.

Data Formats

Workload Profile JSON

Generated by:

kconfigtune profile --output workload.json

Example:

{
  "ipc": 0.42,
  "llc_miss_rate": 0.18,
  "branch_miss_rate": 0.03,
  "io_wait_pct": 0.12,
  "mem_bandwidth_norm": 0.31,
  "cpu_util": 0.76
}

Training Pair JSON

A training pair connects a kernel configuration and a workload profile to a measured latency value.

Example:

{
  "config_vec": [0, 1, 1, 0],
  "workload_vec": [0.42, 0.18, 0.03, 0.12, 0.31, 0.76],
  "latency_ms": 142.7,
  "workload_type": "database"
}

In the real project, config_vec must contain 47 values.

Generated Config

Generated by:

kconfigtune generate --output optimized.config

Example:

CONFIG_PREEMPT=y
# CONFIG_DEBUG_KERNEL is not set
CONFIG_TCP_CONG_BBR=y

If no kernel source tree is available, KConfigTune can write a stub .config fragment containing only the tracked options. Run make olddefconfig in a kernel source tree to resolve missing options.

Folder Structure

Current repository layout:

.
├── README.md
├── PROJECT_STRUCTURE.md
├── pyproject.toml
└── kconfigtune/
    ├── cli.py
    ├── config.py
    ├── agent/
    │   ├── actor_critic.py
    │   ├── environment.py
    │   ├── ppo.py
    │   └── replay_buffer.py
    ├── benchmarks/
    │   └── benchmark_runner.py
    ├── kernel/
    │   ├── kconfig_manager.py
    │   └── kernel_compiler.py
    ├── profiler/
    │   └── workload_profiler.py
    └── surrogate/
        ├── dataset.py
        ├── surrogate_model.py
        └── surrogate_trainer.py

Production and Safety Notes

KConfigTune is designed for production-minded experimentation, but generated kernel configurations should always be validated before use.

Important safety practices:

• Do not install generated kernels directly on critical production machines.
• Compile and boot-test in an isolated environment first.
• Use QEMU verification when possible.
• Keep a known-good boot entry available.
• Review generated .config files before installation.
• Collect real benchmark data for important workloads instead of relying only on synthetic bootstrap data.
• Treat surrogate predictions as estimates, not guarantees.

Why Synthetic Data Exists

If data/training_pairs is empty, training can use a synthetic dataset. This is useful for development and demonstrations, but it is not a replacement for real benchmark data.

For serious tuning, collect real pairs:

config vector + workload vector -> measured latency

The surrogate becomes more useful as the real dataset becomes more accurate and representative.

Current Implementation Notes

• The project currently tracks 47 kernel options.
• The workload vector has 6 features.
• The combined RL state has 53 values.
• The action space has 47 discrete actions.
• Each action flips one tracked kernel option.
• The default PPO training configuration uses 500 episodes.
• The surrogate model predicts latency in milliseconds.

Development Guide

Run Formatting and Static Checks

Development dependencies include ruff and mypy.

ruff check .
mypy kconfigtune

Run Tests

If tests are present in your working tree:

pytest

View Training Logs

Training logs are written for TensorBoard:

tensorboard --logdir logs

Then open the URL printed by TensorBoard.

Adding New Kernel Options

To add more tunable options:

1. Add the option name to CONFIG_OPTIONS in kconfigtune/config.py.
2. Update the expected option count if needed.
3. Review dependency rules in kconfigtune/kernel/kconfig_manager.py.
4. Retrain the surrogate model because the state dimension changes.
5. Retrain the PPO agent because the action space changes.

Adding a New Workload Type

To support another workload category:

1. Add the workload type constant in kconfigtune/config.py.
2. Update classification logic in WorkloadProfiler.classify.
3. Add or update benchmark logic in BenchmarkRunner.
4. Collect real training data for that workload.
5. Retrain the surrogate and agent.

Presentation Summary

KConfigTune is a modular system for AI-assisted Linux kernel tuning.

The key idea is to avoid blindly testing thousands of kernel configurations. Instead, the project:

1. Profiles the target workload.
2. Converts workload behavior into numerical features.
3. Uses a surrogate neural network to predict performance quickly.
4. Uses PPO reinforcement learning to search for better kernel options.
5. Writes a Linux .config file from the learned configuration vector.
6. Optionally compiles and validates the kernel with QEMU.

The project is explainable because each part has a clear role:

• The profiler describes the workload.
• The surrogate predicts latency.
• The RL environment defines the optimization problem.
• The PPO agent learns configuration decisions.
• The Kconfig manager writes and validates kernel configs.
• The compiler verifies generated configs in the real kernel toolchain.

This makes KConfigTune useful both as a research/engineering prototype and as a presentation-friendly example of applying machine learning to systems performance optimization.

Research Links

OUr Research Links:

• https://arxiv.org/html/2505.07487v1
• https://arxiv.org/abs/2505.07487
• https://www.researchgate.net/profile/Khadimul-Islam-Zim/publication/400517845_Surrogate_Modeling_with_Artificial_Neural_Networks_for_Efficient_Optimization/links/6985da735d60ab48356bc36b/Surrogate-Modeling-with-Artificial-Neural-Networks-for-Efficient-Optimization.pdf?origin=publication_detail&_tp=eyJjb250ZXh0Ijp7ImZpcnN0UGFnZSI6Il9kaXJlY3QiLCJwYWdlIjoicHVibGljYXRpb25Eb3dubG9hZCIsInByZXZpb3VzUGFnZSI6InB1YmxpY2F0aW9uIn19
• https://arxiv.org/html/2603.02145v1?hl=en-US#:~:text=Suggested%20approach%20of%20kernel%20ML,ML%20model%20user%2Dspace%20thread