Real-time in Linux

2026-05-31 922 words 5 minutes

/posts/real-time-linux/linux-real-time-gemini.png

Contents

This post serves as a quick, summarized reference for building real-time applications in Linux. It is designed assuming minimal prior knowledge of real-time systems. Because this is a concise guide meant for fast navigation, it focuses on the essentials rather than deep theoretical explanations. I hope you find this summary helpful for your projects!

Priorities and scheduling policies

Linux have 99 priorities for real-time: from 1 (lowest) to 99 (highest). However, you should avoid using 99 because there are some kernel threads working at that priority.

The scheduling policies define which task (thread or process) when 2 or more are “ready to run” and both share the same priority.

SCHED_FIFO: First come, first served.
SCHED_RR: Tasks shares the CPU using time slicing.
SCHED_DEADLINE: For scheduling EDF (Earliest Deadline First) tasks.

NOTE: You should avoid Runtime throttling. Otherwise, the kernel limits the CPU utilizations up to 95% (950ms of 1 second). Therefore, set /proc/sys/kernel/sched_rt_runtime_us to -1.

Furthermore, you can use chrt for showing or changing the scheduling attributes of a running process.

Allow user to create real-time threads

In Fedora:

        
sudo dnf install realtime-setup
sudo usermod -aG realtime $USER

In Ubuntu, create the following file /etc/security/limits.d/realtime.conf:

@realtime       soft    cpu             unlimited
@realtime       -       rtprio          99
@realtime       -       nice            -20
@realtime       -       memlock         unlimited

Also, add your user to the realtime group (as showed before for Fedora).

Synchronization between threads/processes

Make sure you are using priority inheritance for avoiding priority inversion. This is not default for pthread_mutex_t. Use pthread_mutexattr_setprotocol() configuring it.

NOTE: Avoid using SIGNALS. Use them just for stopping a process.

Creation of periodic tasks

You should use CLOCK_MONOTONIC because it is not affected by time setting (like CLOCK_REALTIME). Also, use TIMER_ABSTIME to avoid execution jitter.

Example code:

        
        
        
    
#include <time.h>

#define CYCLIC_TIME_NS (100 * 1000 * 1000) // 100ms

void cyclic_task(void)
{
    struct timespec tv;
    clock_gettime(CLOCK_MONOTONIC, &tv);
    while(true)
    {
        // Do some work

        tv.tv_nsec += CYCLIC_TIME_NS;
        // Normalize tv (tv_nsec should get values smaller than 1 second)
        clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &tv, NULL);
    }
}

Page faulting

Linux does not allocate memory in the physical RAM until it is touched (read/written). When a process tries to access a piece of memory for the first time, an exception known as “page fault” occurs. Linux handles this exception by allocating a page (usually 4KiB) into physical RAM for the process. This is valid for: Text segment, uninitialized/initialized data segments, heap, and stack. Page faults take some time and, therefore, they have to be avoided in real-time applications. Page faults are avoided by doing the following:

Tune glibc malloc: Request memory directly from the kernel.

        
#include <malloc.h>

mallopt(M_MMAP_MAX, 0);
mallopt(M_TRIM_THRESHOLD, -1); // Disabling recycling of unused memory by trimming.

Lock down allocated pages: The memory should never go back to the kernel for recycling. (If the kernel goes low in physical RAM, it starts swapping. In this case, the text segment could be paged out. This should never happen.)
```
#include <sys/mman.h>

mlockall(MCL_CURRENT | MCL_FUTURE);
```

Pre-fault the heap and the stack. This is required once during initialization.

        
        
        
    
#include <stdlib.h>
#include <unistd.h>

#define MAX_SAFE_STACK (512 * 1024) // 512kB

void prefault_heap(int size)
{
    char* dummy;
    int i;

    dummy = malloc(size);
    if (!dummy)
    {
        return;
    }
    for (i = 0; i < size; i += sysconf(_SC_PAGESIZE))
    {
        dummy[i] = 1;
    }
    free(dummy);
}

void prefault_stack(void)
{
    unsigned char dummy[MAX_SAFE_STACK];
    int i;

    for (i = 0; i < MAX_SAFE_STACK; i += sysconf(_SC_PAGESIZE))
    {
        dummy[i] = 1;
    }
}

CPU affinity

In order to avoid interference over real-time tasks, there are several strategies can be taken:

Isolate activities on cores can increase determinism (i.e. some cores for real-time tasks, the others for non-real-time tasks). Therefore, kernel threads can be assigned to different cores than the ones for real-time tasks. However, you have to pay attention to the architecture while selecting the cores because of caches shared between them. For binding tasks to cores, you can use taskset or cpuset (cgroups). With taskset you can bind a single process to a specific core from user level. With cpuset you can handle a group of process but you will need admin rights. cpuset restrict the taskset.
You can route the hardware interrupt to specific cores:
- /proc/irq/irq-number/smp_affinity: The default value for that interrupt handler.
- /proc/irq/irq-number/effective_affinity: Current value. Write into smp_affinity and then check effective_affinity (some hardware is not capable of arbitrarily assigning processors to hardware interrupt handlers). Furthermore, /proc/irq/default_smp_affinity sets the default affinity for hardware interrupts.
You can configure maxcpus to limit the number of cores the kernel can see. This can be used for using the rest of the cores for bare metal real-time applications. maxcpus is configured in /etc/default/grub.

Tools for evaluation

Finally, here are some tools that you can use for evaluating your real-time application:

cyclictest: Tracks latencies at a given priority level. Use cyclictest -S -m -p <prio> --secaligned --default-system.
hackbench: For creating scheduling load.
You can invoke OMM (Out-Of-Memory) killer.
trace-cmd and kernelshark: Tracing and visualization tools.
RV monitor rtapp: Runtime verification monitor that checks if any real-time application has some problem (e.g. page fault). See rtapp doc.

        
# Enable rtapp monitor
echo rtapp > /sys/kernel/tracing/rv/enabled_monitors

# Disable rtapp monitor
echo -rtapp > /sys/kernel/tracing/rv/enabled_monitors

# Enable printk as reactor
echo printk > /sys/kernel/tracing/rv/monitors/rtapp/reactors

# See events (-k kernel messages, -f follows the log in real-time)
journalctl -k -f