Imported from kernelorg-mirror/utils_rt-tests_rt-tests (
AGENTS.md). Install upstream withnpx skills add kernelorg-mirror/utils_rt-tests_rt-tests. Copyright stays with the author.
SPDX-License-Identifier: GPL-2.0-only
Copyright 2026 John Kacur jkacur@redhat.com
AGENTS.md - AI Coding Assistant Guide for rt-tests
This document provides guidance for AI coding assistants working with the rt-tests codebase.
Project Overview
rt-tests is a collection of programs for testing real-time Linux kernel behavior. These tools measure scheduling latency, test priority inheritance, validate deadline scheduling, and stress-test various real-time features of the Linux kernel.
- License: GPL-2.0-only (some files GPL-2.0-or-later)
- Language: C (with some Python utilities)
- Maintainers: John Kacur jkacur@redhat.com, Clark Williams williams@redhat.com
- Repository: https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git
- Mailing List: linux-rt-users@vger.kernel.org
Architecture
Directory Structure
rt-tests/
├── src/ # Source code organized by program
│ ├── cyclictest/ # Cyclictest program (main latency tester)
│ ├── hackbench/ # Scheduler benchmark/stress test
│ ├── hwlatdetect/ # Hardware latency detector (Python)
│ ├── pi_tests/ # Priority inheritance tests
│ ├── pmqtest/ # POSIX message queue test
│ ├── ptsematest/ # POSIX semaphore test
│ ├── queuelat/ # Network queue latency detector
│ ├── rt-migrate-test/ # RT migration test
│ ├── sched_deadline/ # SCHED_DEADLINE tests
│ ├── signaltest/ # Signal latency test
│ ├── sigwaittest/ # Signal wait test
│ ├── ssdd/ # Stress, Strain, Duty, Duration test
│ ├── svsematest/ # SysV semaphore test
│ ├── oslat/ # OS latency detector
│ ├── backfire/ # Backfire test utility
│ ├── lib/ # Shared library code
│ │ ├── rt-utils.c # Common utilities (affinity, prio, tracing)
│ │ ├── rt-numa.c # NUMA support functions
│ │ ├── rt-error.c # Error handling
│ │ ├── rt-sched.c # Scheduler utilities
│ │ ├── rt-get_cpu.c # CPU detection
│ │ └── histogram.c # Histogram generation
│ ├── include/ # Common header files
│ │ ├── rt-utils.h # Utility function declarations
│ │ ├── rt-numa.h # NUMA declarations
│ │ ├── rt-error.h # Error handling declarations
│ │ ├── rt-sched.h # Scheduler declarations
│ │ ├── rt-get_cpu.h # CPU detection declarations
│ │ ├── histogram.h # Histogram declarations
│ │ ├── pi_stress.h # PI stress test definitions
│ │ ├── pip_stress.h # PIP stress test definitions
│ │ └── bionic.h # Android Bionic support
│ └── arch/ # Architecture-specific code
│ └── android/ # Android-specific support
├── Makefile # Build system
├── README.markdown # Usage documentation
├── MAINTAINERS # Maintainer contact information
├── COPYING # License text
├── Dockerfile # Container build configuration
├── get_pylib.py # Python library path helper
├── feature/ # Feature detection for build
└── bld/ # Build output directory
Core Programs
cyclictest - High-resolution timer latency tester (primary tool):
- Measures scheduling latency and timer jitter
- Multi-threaded with per-CPU thread support
- Integrates with kernel tracing (ftrace)
- Supports histogram output
- Can trigger trace on latency threshold (
--breaktrace) - Options for affinity, priority, interval, duration
- Man page: src/cyclictest/cyclictest.8
hackbench - Scheduler stress test and benchmark:
- Creates groups of processes/threads exchanging messages
- Stresses scheduler and IPC mechanisms
- Useful for finding worst-case latencies under load
pi_stress / pip_stress - Priority inheritance tests:
- Tests kernel priority inheritance mechanisms
- Validates priority ceiling protocol
- Helps debug priority inversion issues
pmqtest / ptsematest / svsematest - IPC latency tests:
- Test POSIX message queues, POSIX semaphores, SysV semaphores
- Measure wakeup latency through various IPC mechanisms
hwlatdetect - Hardware/firmware latency detector (Python):
- Detects System Management Interrupts (SMIs) and firmware delays
- Uses kernel hwlat detector module
- Runs in debugfs environment
queuelat - Network queue latency detector:
- Measures latency caused by network queue processing
- Tests impact of network load on real-time performance
deadline_test / cyclicdeadline - SCHED_DEADLINE tests:
- Test and validate SCHED_DEADLINE scheduler
- Deadline-based real-time scheduling
oslat - OS latency detector:
- Detects OS-induced latencies
- Measures worst-case OS interference
signaltest / sigwaittest - Signal latency tests:
- Test signal delivery latency
- Measure real-time signal performance
ssdd - Stress, Strain, Duty, Duration:
- Configurable stress test with various patterns
- Tests system under different workloads
rt-migrate-test - RT task migration test:
- Tests real-time task migration behavior
- Validates scheduler migration logic
Shared Library Components
rt-utils.c / rt-utils.h - Common utilities:
parse_cpumask()- Parse CPU affinity masksnuma_initialize()- Initialize NUMA supportenable_trace_mark()- Enable kernel trace markerstracemark()- Write trace markersget_tracers()- Get available kernel tracerssetkfifo()- Set SCHED_FIFO priority- Priority, affinity, and tracing helpers
rt-numa.c / rt-numa.h - NUMA support:
- CPU topology detection
- NUMA node management
- Per-node thread allocation
rt-error.c / rt-error.h - Error handling:
fatal()- Fatal error with exiterr_exit()- Error with errnowarn()- Warning messagesinfo()- Informational messages
histogram.c / histogram.h - Latency histogram generation:
- Collect latency samples into bins
- Format histogram output
- Calculate statistics (min, max, avg)
rt-sched.c / rt-sched.h - Scheduler utilities:
- Scheduler policy helpers
- Priority validation
rt-get_cpu.c / rt-get_cpu.h - CPU detection:
- Get current CPU number
- CPU affinity helpers
Python Utilities
hwlatdetect.py - Hardware latency detector:
- Detects SMIs and firmware-induced delays
- Mounts/unmounts debugfs as needed
- Uses kernel's hwlat_detector module
- Supports duration, threshold, and window parameters
get_cyclictest_snapshot.py - Cyclictest snapshot utility:
- Captures cyclictest state while running
- Reads /dev/shm/cyclictest-* files
- Provides statistics and thread status
get_pylib.py - Python library path helper:
- Determines correct Python library installation path
- Used by Makefile for Python tool installation
Development Guidelines
Code Style
- Follow Linux kernel coding style (K&R style)
- Use tabs for indentation (width 8)
- Maximum line length: 80 characters
- SPDX license identifier at top of each file:
// SPDX-License-Identifier: GPL-2.0-only - Function names: lowercase with underscores (e.g.,
parse_cpumask) - Variable names: lowercase with underscores
- Constants: UPPERCASE with underscores
- Checking code style: Use the kernel's checkpatch.pl script to verify code style compliance:
/path/to/kernel/scripts/checkpatch.pl --no-tree -f <file.c>
C Standards and Dependencies
- C Standard: C99 or later
- POSIX: POSIX.1-2001 with real-time extensions
- Required Libraries:
- pthread (POSIX threads)
- rt (POSIX real-time extensions)
- numa (NUMA support, optional)
- dl (dynamic loading)
- Optional Libraries:
- cpupower (for deepest idle state support)
Python Code
- Python Version: Python 3.8+
- Style: PEP 8 Python style guide
- Shebang:
#!/usr/bin/python3 - Use standard library modules when possible
- Include docstrings for modules, classes, and functions
Error Handling
- Use
err_exit()for fatal errors with errno - Use
fatal()for fatal errors without errno - Use
warn()for warnings that don't stop execution - Check return values from all system calls
- Handle interrupted system calls (EINTR)
- Clean up resources (threads, memory, file descriptors) on error
Real-Time Programming Considerations
- Memory Locking: Lock memory with
mlockall(MCL_CURRENT | MCL_FUTURE)to prevent page faults - Priority: Use SCHED_FIFO or SCHED_DEADLINE for real-time threads
- Affinity: Pin threads to specific CPUs to avoid migration overhead
- Stack Prefault: Touch stack pages before real-time operation
- Avoid Blocking: Minimize system calls and library calls in RT critical sections
- Signal Handling: Use real-time signals when possible
- Trace Integration: Support ftrace integration for latency debugging
Build and Installation
Build Commands
# Install build dependencies (Debian/Ubuntu)
sudo apt-get install build-essential libnuma-dev
# Install build dependencies (Fedora/RHEL)
sudo dnf install gcc make numactl-devel kernel-tools-libs-devel
# Build all programs
make
# Build with debugging
make DEBUG=1
# Build specific program
make cyclictest
# Clean build artifacts
make clean
# Generate tags
make tags
Installation
# Install to /usr/local (default)
sudo make install
# Install to custom prefix
sudo make prefix=/usr install
# Install with compressed man pages
sudo make MAN_COMPRESSION=gzip install
Cross-Compilation
# Set cross-compiler
make CROSS_COMPILE=aarch64-linux-gnu-
# Build for Android
make ostype=android
Testing
Running cyclictest
# Basic test: one thread per CPU, SCHED_FIFO priority 99
sudo ./cyclictest -a -t -p99
# With histogram output
sudo ./cyclictest -a -t -p99 -h 400 -q
# With trace on latency spike
sudo ./cyclictest -a -t -p99 --tracemark --breaktrace=100
# Long duration test (24 hours)
sudo ./cyclictest -a -t -p99 -D 24h -h 400 -q
Running hwlatdetect
# Detect hardware latency for 60 seconds
sudo ./hwlatdetect --duration=60
# With threshold
sudo ./hwlatdetect --duration=60 --threshold=10
Testing Priority Inheritance
# Run pi_stress test
sudo ./pi_stress
# Run with specific parameters
sudo ./pi_stress --inversions=100 --duration=60
Key Files
src/cyclictest/cyclictest.c- Main latency testing toolsrc/lib/rt-utils.c- Common utility functionssrc/lib/rt-numa.c- NUMA supportsrc/include/rt-utils.h- Utility declarationssrc/hwlatdetect/hwlatdetect.py- Hardware latency detectorMakefile- Build configurationREADME.markdown- Usage documentation
Common Gotchas
- Root Privileges: Most tests require root to set real-time priorities and lock memory
- Kernel Configuration: RT tests require kernel with CONFIG_PREEMPT_RT or CONFIG_PREEMPT for best results
- CPU Isolation: For accurate results, isolate CPUs from scheduler (isolcpus kernel parameter)
- Power Management: Disable CPU frequency scaling and idle states for consistent results
- NUMA Systems: Be aware of memory locality on NUMA systems
- Tracing Overhead: Kernel tracing adds overhead; use only when needed
- Signal Handling: Proper cleanup on SIGINT/SIGTERM is important
- Memory Locking: Must lock memory before enabling real-time priority
Build System Notes
The Makefile uses feature detection to conditionally enable features:
- libcpupower: Detected via feature/test-feature.mak, enables
--deepest-idle-stateoption - NUMA: Detected at build time, links with libnuma if available
- Python: Detected via get_pylib.py, determines if Python utilities can be installed
- Architecture: Detected via
$(CC) -dumpmachinefor platform-specific code
Makefile Variables
VERSION: Current version numberDESTDIR: Installation destination (for packaging)prefix: Installation prefix (default: /usr/local)DEBUG: Enable debug build (DEBUG=1)CROSS_COMPILE: Cross-compiler prefixno_libcpupower: Disable libcpupower (no_libcpupower=1)
Support and Contact
- Maintainers:
- John Kacur jkacur@redhat.com
- Clark Williams williams@redhat.com
- Mailing List: linux-rt-users@vger.kernel.org
- Bug Reports: Send to maintainers with mailing list CC'd
- Patch Submissions: Send to maintainers with mailing list CC'd
Git Workflow
- Main Branch:
master - Repository: https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git
- Patch Submission: Use standard git format-patch/send-email workflow
- Commit Format: Follow kernel commit message format
Development Workflow
- Make changes to code
- Build and test:
make && sudo ./cyclictest -a -t -p99 - Run relevant tests to verify changes
- Update man pages if changing command-line options
- Commit with descriptive message
- Submit patches to maintainers with mailing list CC'd
Additional Notes for AI Assistants
- Real-Time Focus: All programs are designed for real-time system testing and analysis
- Kernel Integration: Heavy integration with Linux kernel features (ftrace, debugfs, scheduler)
- Thread Safety: Most programs are multi-threaded; be aware of synchronization
- Signal Handling: Programs must handle SIGINT/SIGTERM for graceful shutdown
- Resource Cleanup: Always clean up threads, memory, and file descriptors
- Man Page Updates: Update man pages when changing options or behavior
- Backwards Compatibility: Maintain compatibility with existing scripts and tools
- Performance Critical: Some code paths are performance-critical; avoid adding overhead
- Architecture Support: Code should work on x86_64, aarch64, ppc64le, s390x, RISC-V
- Android Support: Some programs support Android via Bionic compatibility layer
Related Projects
Projects in the RT ecosystem:
- rteval: Real-time evaluation suite - https://git.kernel.org/pub/scm/utils/rteval/rteval.git
- tuna: Thread and IRQ tuning tool - https://git.kernel.org/pub/scm/utils/tuna/tuna.git
- rtla: Real-time Linux analysis tool - https://www.kernel.org/doc/html/latest/tools/rtla/
- rv: Runtime verification tool - https://www.kernel.org/doc/html/latest/tools/rv/
- Linux RT Kernel: PREEMPT_RT patches - https://kernel.org/pub/linux/kernel/projects/rt/
References
- Linux Real-Time Wiki: https://wiki.linuxfoundation.org/realtime/start
- OSADL Latency Plots: https://www.osadl.org/QA-Farm-Realtime.qa-farm-about.0.html
- RT Kernel Documentation: https://www.kernel.org/doc/Documentation/scheduler/sched-rt-group.txt
- High-resolution timers: https://www.kernel.org/doc/html/latest/timers/hrtimers.html
Last Updated: 2026-06-17 Document Version: 1.0
