DAMON: Data Access Monitor

With increasingly data-intensive workloads and limited DRAM capacity, optimal memory management based on dynamic access patterns is becoming increasingly important. Such mechanisms are only possible if accurate and efficient dynamic access pattern monitoring is available.

DAMON is a Data Access MONitoring framework subsystem for the Linux kernel developed for such memory management. It is designed with some key mechanism (refer to Design for the detail) that make it

  • accurate (the monitoring output is useful enough for DRAM level memory management; It might not be appropriate for CPU Cache levels, though),
  • light-weight (the monitoring overhead is low enough to be applied online), and
  • scalable (the upper-bound of the overhead is in constant range regardless of the size of target workloads).

Therefore, DAMON can be used to develop memory management based on any access pattern. To make it easier to develop such systems, DAMON provides a feature called DAMON-based Operation Schemes (DAMOS). This allows DAMON users to develop and execute access-aware memory management without code but with a simple setup. Simple mechanisms based on DAMOS can achieve up to 12% performance improvement and 91% memory savings.

Detailed evaluation of DAMON and DAMON-based system optimizations are available at another post.

Table Of Contents

Demo Video

damo monitor for water_nsquared

Demo Screenshot

masim stairs heatmap in ascii

Recent News

Below are only a short list of recent news. For complete list of the news, please refer to a dedecated post.

2024-03-13: Memory management subsystem pull request for Linux 6.9-rc1 has been posted. To quote Andrew’s summary for DAMON part:

  • More DAMON work from SeongJae Park in the series
    • “mm/damon: make DAMON debugfs interface deprecation unignorable”
    • “selftests/damon: add more tests for core functionalities and corner cases”
    • “Docs/mm/damon: misc readability improvements”
    • “mm/damon: let DAMOS feeds and tame/auto-tune itself”

2024-03-06: LSF/MM/BPF 2024 DAMON discussion topic is accepted

2024-03-04: DAMO v2.2.4 is released with a new feature for access pattern-based profiling. For example, users can know which code is making their program’s memory usage unexpectedly high, or which code is intensively accessing memory, and optimize.

2024-02-27: DAMON user-space tool, DAMO, has downloaded from PyPI more than 2,000 times last month. damo_2000_monthly_downloads

2024-02-21: Yet another academic paper exploring DAMON for tiered memory management will be presented at EuroSys 2024

2024-02-20: DAMO v2.2.2 is released with a new command, replay. It will hopefully help reproducing the real-world memory access pattern for analysis and benchmarks.

Getting Started

You can start using DAMON by

By following those, you will be able to know if DAMON works on your machine and how you can use it.

Install

DAMON is implemented in the Linux kernel, so you should install DAMON-enabled Linux kernel to use it. To check if you’re already running DAMON-enabled kernel, you could:

$ if grep CONFIG_DAMON /boot/config-$(uname -r); then echo "installed"; fi

DAMON is enabled in Amazon Linux kernels that based on v5.4 or newer Linux kernels. If you’re using Amazon Linux 2 and not using DAMON-enabled kernel on the distro, you can install DAMON-enabled kernels using its package manager, amazon-linux-extras. For example, if you want to use DAMON with Amazon Linux v5.10 kernel, below commands will work.

$ sudo yum install -y amazon-linux-extras
$ sudo amazon-linux-extras install kernel-5.10
$ sudo reboot

Note that AL2 v5.10.y contains all DAMON features that available on mainline while AL2 5.4.y contains only a part of DAMON features. AL2 v5.10.y kernels are recommended for the reason.

If your package system doesn’t support DAMON-enabled kernel, you can fetch a DAMON-merged Linux kernel source tree (it has merged in Linux mainline tree since v5.15), build, and install it. Note that you should enable kernel configuration options for DAMON, depending on your demands. For example:

$ cd $THE_FETCHED_DAMON_KERNEL_SOURCE_TREE
$ make olddefconfig
$ echo 'CONFIG_DAMON=y' >> /.config
$ echo 'CONFIG_DAMON_VADDR=y' >> /.config
$ echo 'CONFIG_DAMON_PADDR=y' >> /.config
$ echo 'CONFIG_DAMON_SYSFS=y' >> /.config
$ echo 'CONFIG_DAMON_DBGFS=y' >> /.config
$ echo 'CONFIG_DAMON_RECLAIM=y' >> /.config
$ echo 'CONFIG_DAMON_LRU_SORT=y' >> /.config
$ make -j$(nproc)
$ sudo make modules_install install

Source Code

There are several Linux kernel source trees having DAMON for different users. You may pick one among those based on your needs.

For users who want a stable version of DAMON, below trees are recommended.

  • Linus Torvalds’ mainline tree contains the latest stable version of DAMON since v5.15.
  • Amazon Linux kernel v5.10.y tree, v5.4.y tree and all trees that based on v5.15 or newer Linux kernel contain stable versions of DAMON.

Note that AL2 v5.10.y contains all DAMON features that available on the latest major releases of Linux while AL2 5.4.y contains only a part of DAMON features, so AL2 v5.10.y is more recommended.

If you have interests in DAMON features under development, below trees will be appropriate. These trees contain latest version of DAMON which having features under development.

  • mm-unstable contains the latest DAMON patches, which are under testing with other unstable memory management subsystem patches. So this tree is likely unstable and frequently updated, but would be a good baseline for your DAMON development.
  • damon/next contains the latest changes, which might not tested at all. So this tree is likely more unstable and frequently updated than mm-unstable. This tree also contains some changes that exist only for DAMON hacks itself rather than eventually be merged in the mainline.

For people who have interest in DAMON features under development but use LTS kernels as their baseline, three were trees that based on three latest LTS kernels.

NOTE: Below trees were continuously getting DAMON backports on latest 5.4.y, 5.10.y, and 5.15.y. But those are deprecated as of 2022-09-03, and therefore there will be no update to the trees.

The source code of DAPTRACE, which is a prototype of DAMON, is also available.

User-space Tool

A user-space tool for DAMON, which is called DAMO (Data Access Monitoring Operator), is available at Github and PyPi. You may start using DAMON by following the Getting Started of the tool for start.

Tests Package

There is a tests suite for correctness verification and performance evaluation of DAMON. Those are actively used for the development of DAMON. Using that, you can test DAMON on your machine with only a few simple commands or deeply understand the user interface of DAMON.

So, if you finished the tutorial but have no idea about what to do next, running the tests would be a good start. If any test fails, please report that to the maintainer via mail (sj@kernel.org or damon@lists.linux.dev) or github.

Official Document

The official document of DAMON in the latest mainline for users/sysadmins and kernel programmers are available. Those for next major release are also available(users/sysadmins doc, kernel programmers doc).

Showcase Website

There is a showcase website that you can get more information of DAMON. The site hosts

Evaluation Results

Evaluation of DAMON and DAMON-based system optimizations are available.

DAMON-based System Optimization Guide

A guide for DAMON-based system optimizations are also available.

Profile-Guided Optimization Example

An example of DAMON-based profile-guided optimization is also available.

Community

DAMON is maintained and developed by its own community, which is a sub-set of the linux kernel development community.

The community is mainly driven by the mailing list (damon@lists.linux.dev). All the patches are posted there and reviewed. Almost every discussions are also made there. You can subscribe to the mailing list via subspace.kernel.org following the instruction. You could also read the mails without subscription via lore.

The community also have an open, regular, and informal virtual bi-weekly meeting series for DAMON community called DAMON Beer/Coffee/Tea chat series.

Publications and Presentations

Below are featured and/or upcoming publications and presentations covering DAMON. For more complete list of those, please refer to another dedicated post.

  • SeongJae Park, DAMO[N,S]?: Implementing Self-Driven Data Access-Aware Efficient Linux System. In Open Source Summit North America, Apr 2024. Link
  • SeongJae Park, DAMON: Current Status and Future Plans. In Kernel Summit, Nov 2023. Slides, Video, Link
  • SeongJae Park, Data Access Monitoring Operator (DAMO): User-Space Tool/Python Library for Access-Aware Profiling and Optimization of Your Linux Systems. In Open Source Summit Europe, Sep 2023. Slides, Video, Link
  • SeongJae Park, DAMON, DAMOS, and DAMO: Kernel Subsystems and User-Space Tools for Data Access-Aware System Analysis/Optimizations. In Open Source Summit North America, May 2023. Slides, Video, Link
  • SeongJae Park, DAMON updates and future plans. In Linux Storage | Filesystem | MM & BPF Summit, May 2023. Slides, Video, Link
  • SeongJae Park, Current Status and Future Plans of DAMON. In The Linux Kernel Summit, September 2022. Slides, Video, Link
  • Jonathan Corbet, LRU-list manipulation with DAMON. In Linux Weekly News, August 2022. Article
  • SeongJae Park, Madhuparna Bhowmik, Alexandru Uta, DAOS: Data Access-aware Operating System. In The 31st International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC'22), June 2022. Paper, Slides, Poster
  • Jonathan Corbet, Using DAMON for proactive reclaim. In Linux Weekly News, July 2021. Article
  • Jonathan Corbet, Memory-management optimization with DAMON. In Linux Weekly News, February 2020. Article
  • SeongJae Park, Yunjae Lee, Heon Y. Yeom, Profiling Dynamic Data Access Patterns with Controlled Overhead and Quality. In 20th ACM/IFIP International Middleware Conference Industry, December 2019. Paper
Avatar
SeongJae Park
Kernel Development Engineer

SeongJae Park is a programmer who loves to analyze and develop systems.

Related