The Linux Watchdog driver API¶
This document describes the state of the Linux 2.4.18 kernel.
Introduction¶
A Watchdog Timer (WDT) is a hardware circuit that can reset the computer system in case of a software fault. You probably knew that already.
Usually a userspace daemon will notify the kernel watchdog driver via the /dev/watchdog special device file that userspace is still alive, at regular intervals. When such a notification occurs, the driver will usually tell the hardware watchdog that everything is in order, and that the watchdog should wait for yet another little while to reset the system. If userspace fails (RAM error, kernel bug, whatever), the notifications cease to occur, and the hardware watchdog will reset the system (causing a reboot) after the timeout occurs.
The Linux watchdog API is a rather ad-hoc construction and different drivers implement different, and sometimes incompatible, parts of it. This file is an attempt to document the existing usage and allow future driver writers to use it as a reference.
The simplest API¶
All drivers support the basic mode of operation, where the watchdog activates as soon as /dev/watchdog is opened and will reboot unless the watchdog is pinged within a certain time, this time is called the timeout or margin. The simplest way to ping the watchdog is to write some data to the device. So a very simple watchdog daemon would look like this source file: see samples/watchdog/watchdog-simple.c
A more advanced driver could for example check that a HTTP server is still responding before doing the write call to ping the watchdog.
When the device is closed, the watchdog is disabled, unless the "Magic Close" feature is supported (see below). This is not always such a good idea, since if there is a bug in the watchdog daemon and it crashes the system will not reboot. Because of this, some of the drivers support the configuration option "Disable watchdog shutdown on close", CONFIG_WATCHDOG_NOWAYOUT. If it is set to Y when compiling the kernel, there is no way of disabling the watchdog once it has been started. So, if the watchdog daemon crashes, the system will reboot after the timeout has passed. Watchdog devices also usually support the nowayout module parameter so that this option can be controlled at runtime.
Magic Close feature¶
If a driver supports "Magic Close", the driver will not disable the watchdog unless a specific magic character ‘V’ has been sent to /dev/watchdog just before closing the file. If the userspace daemon closes the file without sending this special character, the driver will assume that the daemon (and userspace in general) died, and will stop pinging the watchdog without disabling it first. This will then cause a reboot if the watchdog is not re-opened in sufficient time.
The ioctl API¶
All conforming drivers also support an ioctl API.
Pinging the watchdog using an ioctl:
All drivers that have an ioctl interface support at least one ioctl, KEEPALIVE. This ioctl does exactly the same thing as a write to the watchdog device, so the main loop in the above program could be replaced with:
the argument to the ioctl is ignored.
Setting and getting the timeout¶
For some drivers it is possible to modify the watchdog timeout on the fly with the SETTIMEOUT ioctl, those drivers have the WDIOF_SETTIMEOUT flag set in their option field. The argument is an integer representing the timeout in seconds. The driver returns the real timeout used in the same variable, and this timeout might differ from the requested one due to limitation of the hardware:
This example might actually print "The timeout was set to 60 seconds" if the device has a granularity of minutes for its timeout.
Starting with the Linux 2.4.18 kernel, it is possible to query the current timeout using the GETTIMEOUT ioctl:
Pretimeouts¶
Some watchdog timers can be set to have a trigger go off before the actual time they will reset the system. This can be done with an NMI, interrupt, or other mechanism. This allows Linux to record useful information (like panic information and kernel coredumps) before it resets:
Note that the pretimeout is the number of seconds before the time when the timeout will go off. It is not the number of seconds until the pretimeout. So, for instance, if you set the timeout to 60 seconds and the pretimeout to 10 seconds, the pretimeout will go off in 50 seconds. Setting a pretimeout to zero disables it.
There is also a get function for getting the pretimeout:
Not all watchdog drivers will support a pretimeout.
Get the number of seconds before reboot¶
Some watchdog drivers have the ability to report the remaining time before the system will reboot. The WDIOC_GETTIMELEFT is the ioctl that returns the number of seconds before reboot:
Environmental monitoring¶
All watchdog drivers are required return more information about the system, some do temperature, fan and power level monitoring, some can tell you the reason for the last reboot of the system. The GETSUPPORT ioctl is available to ask what the device can do:
the fields returned in the ident struct are:
|
identity |
a string identifying the watchdog driver |
|
firmware_version |
the firmware version of the card if available |
|
options |
a flags describing what the device supports |
the options field can have the following bits set, and describes what kind of information that the GET_STATUS and GET_BOOT_STATUS ioctls can return.
|
WDIOF_OVERHEAT |
Reset due to CPU overheat |
The machine was last rebooted by the watchdog because the thermal limit was exceeded:
|
WDIOF_FANFAULT |
Fan failed |
A system fan monitored by the watchdog card has failed
|
WDIOF_EXTERN1 |
External relay 1 |
External monitoring relay/source 1 was triggered. Controllers intended for real world applications include external monitoring pins that will trigger a reset.
|
WDIOF_EXTERN2 |
External relay 2 |
External monitoring relay/source 2 was triggered
|
WDIOF_POWERUNDER |
Power bad/power fault |
The machine is showing an undervoltage status
|
WDIOF_CARDRESET |
Card previously reset the CPU |
The last reboot was caused by the watchdog card
|
WDIOF_POWEROVER |
Power over voltage |
The machine is showing an overvoltage status. Note that if one level is under and one over both bits will be set — this may seem odd but makes sense.
|
WDIOF_KEEPALIVEPING |
Keep alive ping reply |
The watchdog saw a keepalive ping since it was last queried.
|
WDIOF_SETTIMEOUT |
Can set/get the timeout |
The watchdog can do pretimeouts.
|
WDIOF_PRETIMEOUT |
Pretimeout (in seconds), get/set |
For those drivers that return any bits set in the option field, the GETSTATUS and GETBOOTSTATUS ioctls can be used to ask for the current status, and the status at the last reboot, respectively:
Note that not all devices support these two calls, and some only support the GETBOOTSTATUS call.
Some drivers can measure the temperature using the GETTEMP ioctl. The returned value is the temperature in degrees fahrenheit:
Finally the SETOPTIONS ioctl can be used to control some aspects of the cards operation:
Watchdog (сторожевой таймер) в Linux
Watchdog (сторожевой таймер) — аппаратно реализованная схема контроля над зависанием системы.
Сторожевые таймеры используются в системах, которые должны работать без надзора человека. Такие системы должны быть само восстанавливающимися без участия оператора.
В контроллерах JetHome JetHub на базе процессоров Amlogic поддерживается драйвер аппаратного watchdog meson_wdt .
Проверка устройств
При корректно запущенном модулей watchdog в системе /dev должны быть видны устройства /dev/watchdog и /dev/watchdog0 :
Установка сервиса Watchdog
В прошивках начиная с версии Armbian 22.02 watchdog предустановлен, ручная установка не требуется.
Для установки сервиса watchdog выполните следующие команды:
Создайте папку для лог-файлов watchdog:
Проверьте настройки для сервиса в файле /etc/default/watchdog :
Конфигурационные файлы
Внесите необходимые изменения в конфигурационный файл /etc/watchdog.conf :
-
Раскомментируйте использование устройства /dev/watchdog (иначе сервис watchdog не будет использовать аппаратный таймер для перезагрузки контроллера).
-
Установите необходимые проверки и таймауты.
Ниже приведен пример конфигурационного файла, с установленным таймаутом 15 секунд на зависание:
Примечание
Значение watchdog-timeout определяет через какое время после аварии сервиса watchdog аппаратный таймер перезагрузит контроллер.
The Linux Process Journey — watchdogd
This kernel thread “watchdogd” is used in order to let the kernel know that a serious problem has occurred so the kernel can restart the system. It is sometimes called COP (Computer Operating Properly). The way it is implemented is by opening “/dev/watchdog”, then writing at least once a minute. Every time there is a write the restart of the system is delayed.
In case of inactivity for a minute the watchdog should restart the system. Due to the fact we are not talking about a hardware watchdog the compilation of the operation depends on the state of the machine. You should know that the watchdog implementation could be software only (there are cases in which it won’t restart the machine due to failure) or using a driver/module in case of hardware support (you can read more about it here — https://github.com/torvalds/linux/blob/master/Documentation/watchdog/watchdog-api.rst).
If we are talking about hardware support then the watchdog module is specific for a chip or a device hardware. It is most relevant to systems that need the ability to restart themselves without any human intervention (as opposed to a PC we can reboot easily) — think about an unmanned aircraft. We need to be careful because a problem in the watchdog configuration can lead to unpredictable reboot, reboot loops and even file corruption due to hard restart (https://linuxhint.com/linux-kernel-watchdog-explained/).
The relationship between the hardware and software is as follows: the hardware is responsible to set up the timer and the software is responsible to reset the timer. When the timer gets to a specific value (configured ahead) and it is not elapsed by the software the hardware will restart the system. For an example of using hardware for this functionality you can read the following link https://developer.toradex.com/linux-bsp/how-to/linux-features/watchdog-linux/.
The software part is being conducted by the “watchdogd” (the software watchdog daemon) which opens “/dev/watchdog” and writes to it in order to postpone the restart of the system by the hardware — for more information you can read https://linux.die.net/man/8/watchdog. Examples for different watchdog drives/modules for specific chips can be found in the source tree of linux here https://elixir.bootlin.com/linux/v6.0.11/source/drivers/watchdog. Some example are apple_wdt (Apple’s SOC), ath79_wdt (Atheros AR71XX/AR724X/AR913X) and w83977f_wdt (Winbond W83977F I/O Chip).
We can stop the watchdog without restarting the system by closing “/dev/watchdog”. It is not possible if the kernel was compiled with “CONFIG_WATCHDOG_NOWAYOUT” enabled.
Overall, in order for the watchdog to operate the kernel needs to be compiled with CONFIG_WATCHDOG=y and “/dev/watchdog” character device should be created (with major number of 10 and minor number of 130 — checkout “man mknod” if you want to create it).
Linux Kernel Watchdog Explained
The Linux kernel watchdog is used to monitor if a system is running. It is supposed to automatically reboot hanged systems due to unrecoverable software errors. The watchdog module is specific to the hardware or chip being used. Personal computer users don’t need watchdog as they can reset the system manually. However, it is useful for systems that are mission critical and need the ability to reboot themselves without human intervention. For example, servers on a remote location or embedded equipment on a spacecraft that need automatic hardware reset capabilities.
Warning: Proceed with Caution
Wrong configurations of a watchdog on your system can cause problems like:
- Endless reboot loop
- File corruption due to hard reset
- Unpredictable random reboots
So avoid using live servers to test Linux kernel watchdog.
Watchdog Module
Watchdog functionality on the hardware side sets up a timer that times out after a predetermined period. The watchdog software then periodically refreshes the hardware timer. If the software stops refreshing, then after the predetermined period, the timer performs a hardware reset of the device. In order for a watchdog timer to be functional, the motherboard manufacturer has to use the chip’s watchdog functionality. Often the documentation from the manufacturer is not clear about whether the functionality was implemented. In that case, you have to test it out.
Also, you need the right watchdog kernel module to be loaded in your Linux system. Different chips use different modules. For example:
- Intel chipsets might use the “iTCO_wdt” module
- HP hardware might use “hpwdt”
- IBM mainframes might use “vmwatchdog”
- Xen VM might use “xen_wdt”
After the module is loaded, you can check /dev/watchdog on the Linux system. If this file is present, that means the watchdog kernel device driver or module was loaded. The system periodically keeps writing to /dev/watchdog. It is also called “kicking or feeding the watchdog”. If the system fails to kick or feed the watchdog, then after a while the system is hard reset.
Watchdog Daemon
The watchdog daemon opens the device and provides the necessary refresh to keep the system from resetting. It can test process table space, memory usage, file accessibility, work overload, file table overflow, IP address ping, network interface traffic, temperature, running processes and more. If the tests fail, then watchdog causes a shutdown.
Starting and Stopping Watchdog
Watchdog daemon should start at boot time and put itself in the background. You can check if it is running:
If the kernel is NOT compiled with CONFIG_WATCHDOG_NOWAYOUT, then if you close the /dev/watchdog properly, it will not cause a reboot. You can write the character V into /dev/watchdog and then close the file. This should stop the watchdog.
Testing the Watchdog
If you want to test if the hardware watchdog is working, you can do the following from your administrator command prompt:
And press “enter” twice and wait. The prompt will not come back. After awhile depending on your kernel’s setting, the system should perform the hard reboot.
References:
About the author
Zak H
Zak H. lives in Los Angeles. He enjoys the California sunshine and loves working in emerging technologies and writing about Linux and DevOps topics.