Introduction

Welcome to the Alumet user guide! If you want to measure something with Alumet, you have come to the right place.

To skip the introduction and install Alumet, click here.

What is Alumet?

Alumet is a modular framework for local and distributed measurement.

Diagram of Alumet (high-level view)

Alumet provides a unified interface for gathering measurements with sources (on the left), transforming the data with models (in the middle) and writing the result to various outputs (on the right). The elements (colored rectangles) are created by plugins, on top of a standard framework.

Key points:

  • The framework is generic and extensible: you can write your own plugins if you need to, or take what you need among the numerous existing plugins. Alumet can easily be extended in order to make new research experiments: add new probes, statistical models, transform functions, export formats, etc. using a high-level API.
  • Alumet is efficient: written in Rust and optimized for low-latency measurement. (scientific publication pending with benchmarks)
  • Alumet is more correct than some existing "software powermeters": our plugins fix some bugs that are hard to detect1.
  • It produces good operational tools: the end result is (or aims to be) a ready-to-use measurement tool that is robust, efficient and scalable.
  • You have control over the tool: the methodology is transparent, the configuration is clear, and you are free to change whatever you want.

Please read the developer book to learn more about the creation of plugins.

What makes Alumet more efficient?

The L in Alumet stands for Lightweight. Why is Alumet "lightweight" compared to other measurement tools?

  1. Optimized pipeline: Alumet is written in Rust, optimized for minimal latency and low memory consumption.
  2. Efficient interfaces: When we develop a new measurement source, we try to find the most efficient way of measuring what we're interested in. As a result, many plugins are based on low-level interfaces, such as the Linux perf_events interface, instead of slower higher-level wrappers. In particular, we try to remove useless intermediate levels, such as calling an external program and parsing its text output.
  3. Pay only for what you need: Alumet's modularity allows you to create a bespoke measurement tool by choosing the plugins that suit your needs, and removing the rest. You don't need a mathematical model that assigns the energy consumption of hardware components to processes? Remove it, and enjoy an even smaller disk footprint, CPU overhead, memory use and energy consumption.

Does it work on my machine ?

For now, Alumet works in the following environments:

  • Operating Systems: Linux, macOS2, Windows2
  • Hardware components3:
    • CPUs: Intel x86 processors (Sandy Bridge or more recent), AMD x86 processors (Zen 1 or more recent), NVIDIA Jetson CPUs (any model)
    • GPUs: NVIDIA dedicated GPUs, NVIDIA Jetson GPUs (any model)

(nice compatibility table coming soon)

  1. Guillaume Raffin, Denis Trystram. Dissecting the software-based measurement of CPU energy consumption: a comparative analysis. 2024. ⟨hal-04420527v2⟩. ↩

  2. While the core of Alumet is cross-platform, many plugins only work on Linux, for example the RAPL and perf plugins. There is no macOS-specific nor Windows-specific plugin for the moment, so Alumet will not be able to measure interesting metrics on these systems. ↩ ↩2

  3. If your computer contains both supported and unsupported components, you can still use Alumet (with the plugins corresponding to the supported components). It will simply not measure the unsupported components. ↩

The main parts of Alumet: core, plugins, agents

One of the key features of the Alumet framework is its extensibility. Thanks to a clear separation between the "core" and the "plugins", Alumet allows to build measure-made measurement tools for a wide range of situations.

This page offers a simple, high-level view of the main concepts. For a more detailed explanation, read the Alumet Architecture chapter of the Alumet Developer Book.

Alumet core

The core of Alumet is a Rust library that implements:

  • a generic and "universal" measurement model
  • a concurrent measurement pipeline based on asynchronous tasks
  • a plugin system to populate the pipeline with various elements
  • a resilient way to handle errors
  • and various utilities

Alumet plugins

On top of this library, we build plugins, which use the core to provide environment-specific features such as:

  • gathering measurements from the operating system
  • reading data from hardware probes
  • applying a statistical model on the data
  • filtering the data
  • writing the measurements to a file or database
  • …

Alumet agent(s)

But Alumet core and Alumet plugins are not executable! You cannot run them to obtain your measurements. To get an operational tool, we combine them in an agent: a runnable application.

We provide a "standard" agent that you can download and use right away. See Installing Alumet. You can also build your own customized agent, it only takes a few lines of codes. Refer to the Developer Book.

Installing Alumet agent

⚠️  Alumet is currently in Beta.

If you have trouble using Alumet, do not hesitate to discuss with us, we will help you find a solution. If you think that you have found a bug, please open an issue in the repository.

There are three main ways to install the standard Alumet agent1:

  1. 📦 Download a pre-built package. This is the simplest method.
  2. 🐳 Pull a docker image.
  3. 🔵 Deploy in a K8S cluster with a helm chart.
  4. 🧑‍💻 Use cargo to compile and install Alumet from source. This requires a Rust toolchain, but enables the use of the most recent version of the code without waiting for a new release.

Option 1: Installing with a pre-built package

Go to the latest release on Alumet's GitHub page. In the Assets section, find the package that corresponds to your system. For instance, if you run Ubuntu 22.04 on a 64-bits x86 CPU, download the file that ends with amd64_ubuntu_22.04.deb.

You can then install the package with your package manager. For instance, on Ubuntu:

sudo apt install ./alumet-agent*amd64_ubuntu_22.04.deb

We currently have packages for multiples versions of Debian, Ubuntu, RHEL and Fedora. We intend to provide even more packages in the future.

What if I have a more recent OS?

The packages that contain the Alumet agent have very few dependencies, therefore an older package should work fine on a newer system. For example, if you have Ubuntu 25.04, it's fine to download and install the package for Ubuntu 24.04.

To simplify maintenance, we don't release one package for each OS version, but we focus on LTS ones.

My OS is not supported, what do I do?

Alumet should work fine on nearly all Linux distributions, but we do not provide packages for every single one of them. Use another installation method (see below). For instance, if you are using Ubuntu on ARM devices (for example Jetson edge devices), you should compile the agent from source.

Alumet core is OS-agnostic, but the standard Alumet agent does not support Windows nor macOS yet1.

Option 2: Installing with Podman/Docker

Every release is published to the container registry of the alumet-dev organization.

Pull the latest image with the following command (replace podman with docker if you use docker):

podman pull ghcr.io/alumet-dev/alumet-agent

View more variants of the container image on the alumet-agent image page.

Privileges required when running

Because Alumet has low-level interactions with the system, it requires some privileges. The packages take care of this setup, but with a container image, you need to grant these capabilities manually.

To run alumet-agent, you need to execute (again, replace podman with docker if you use docker):

podman run --cap-add=CAP_PERFMON,CAP_SYS_NICE ghcr.io/alumet-dev/alumet-agent

Launcher script (optional)

Let's simplify your work and make a shortcut: create a file alumet-agent somewhere. We recommend $HOME/.local/bin/ (make sure that it is in your path).

#!/usr/bin/bash
podman run --cap-add=CAP_PERFMON,CAP_SYS_NICE ghcr.io/alumet-dev/alumet-agent

Give it the permission to execute with chmod +x $HOME/.local/bin/alumet-agent, and voilà! You should now be able to run the alumet-agent command directly.

Option 3: Installing in a K8S cluster with Helm

To deploy Alumet in a Kubernetes cluster, you can use our Helm chart to setup a database, an Alumet relay server, and multiple Alumet clients. Please refer to Distributed deployment with the relay mode for more information.

Quick install steps:

helm repo add alumet https://alumet-dev.github.io/helm-charts
helm install alumet-distributed alumet/alumet

Here, alumet-distributed is the name of your Helm release, you can put the name you want, or use --generate-name to obtain a new, unique name. See the Helm documentation.

Option 4: Installing from source

Prerequisite: you need to install the Rust toolchain.

Use cargo to compile the Alumet agent.

cargo install --git https://github.com/alumet-dev/alumet.git alumet-agent

It will be installed in the ~/.cargo/bin directory. Make sure to add it to your PATH.

To debug Alumet more easily, compile the agent in debug mode by adding the --debug flag (performance will decrease and memory usage will increase). For more information on how to help us with this ambitious project, refer to the Alumet Developer Book.

Privileges required

Because Alumet has low-level interactions with the system, it requires some privileges. The packages take care of this setup, but with a container image, you need to grant these capabilities manually.

The easiest way to do is is to use setcap as root before running Alumet:

sudo setcap 'cap_perfmon=ep cap_sys_nice=ep' ~/.cargo/bin/alumet-agent

This grants the capabilities to the binary file ~/.cargo/bin/alumet-agent. You will then be able to run the agent directly.

Alternatively, you can also run the Alumet agent without doing setcap, and it will tell you what to do, depending on the plugins that you have enabled.

NOTE: running Alumet as root also works, but is not recommended. A good practice regarding security is to grant the least amount of privileges required.

Post-install steps

Once the Alumet agent is installed, head over to Running Alumet.

  1. See also difference between Alumet core and Alumet agent. ↩ ↩2

Running Alumet agent

To start using the Alumet agent, let us run it in a Terminal.

First, run alumet-agent --help to see the available commands and options.

There are two commands that allow to measure things with Alumet. They correspond to two measurement "modes":

  • The run mode monitors the system.
  • The exec mode spawns a process and observes it.

Monitoring the system with the run mode

In run mode, the Alumet agent uses its plugins to monitor the entire system (to the extent of what the plugins do). To choose the plugins to run, pass the --plugins flag before the run command (this is because the list of plugins apply to every command of the agent, it's not specific to run).

Example:

alumet-agent --plugins procfs,csv run

This will start the agent with two plugins:

  • procfs, which collects information about the processes
  • csv, which stores the measurements in a local CSV file

Stopping

To stop the agent, simply press Ctrl+C.

CSV file

The default CSV file is alumet-output.csv. To change the path of the file, use the --output-file option.

alumet-agent --plugins procfs,csv --output-file "measurements-experiment-1.csv" run

Unlike some other measurement tools, Alumet saves measurements periodically, and provides the full data (unless you use plugins to aggregate or filter the measurements that you want to save, of course).

Default command

Since run is the default command, it can be omitted. That is, the above example is equivalent to:

alumet-agent --plugins procfs,csv

Observing a process with the exec mode

In exec mode, the Alumet agent spawns a single process and uses its plugins to observe it.

The plugins are informed that they must concentrate on the spawned process instead of monitoring the whole system. For instance, the procfs plugin will mainly gather measurements related to the spawned process. It will also obtain some system measurements, but will not monitor all the processes of the system.

Example:

alumet-agent --plugins procfs,csv exec sleep 5

Everything after exec is part of the process to spawn, here sleep 5, which will do nothing for 5 seconds. When the process exits, the Alumet agent stops automatically.

One just before, one just after

To guarantee that you obtain interesting measurements even if the process is short-lived, some plugins (especially the ones that measure the energy consumption of the hardware) will perform one measurement just before the process is started and one measurement just after it terminates.

Of course, if the process lives long enough, the measurement sources will produce intermediate data points on top of those two "mandatory" measurements.

Understanding the measurements: how to read the CSV file

The CSV file produced by this simple setup of Alumet looks like this. The output has been aligned and spaced to make it easier to understand, scroll on the right to see it all.

metric                 ; timestamp                     ; value      ; resource_kind; resource_id; consumer_kind; consumer_id; __late_attributes
mem_total_kB           ; 2025-04-25T15:08:53.949565834Z; 16377356288; local_machine;            ; local_machine;            ; 
mem_free_kB            ; 2025-04-25T15:08:53.949565834Z;   884572160; local_machine;            ; local_machine;            ; 
mem_available_kB       ; 2025-04-25T15:08:53.949565834Z;  8152973312; local_machine;            ; local_machine;            ; 

kernel_new_forks       ; 2025-04-25T15:08:53.949919481Z;           2; local_machine;            ; local_machine;            ; 
kernel_n_procs_running ; 2025-04-25T15:08:53.949919481Z;           6; local_machine;            ; local_machine;            ; 
kernel_n_procs_blocked ; 2025-04-25T15:08:53.949919481Z;           0; local_machine;            ; local_machine;            ; 

process_cpu_time_ms    ; 2025-04-25T15:08:53.99636522Z ;           0; local_machine;            ; process      ;       65387; cpu_state=user
process_cpu_time_ms    ; 2025-04-25T15:08:53.99636522Z ;           0; local_machine;            ; process      ;       65387; cpu_state=system
process_cpu_time_ms    ; 2025-04-25T15:08:53.99636522Z ;           0; local_machine;            ; process      ;       65387; cpu_state=guest
process_memory_kB      ; 2025-04-25T15:08:53.99636522Z ;        2196; local_machine;            ; process      ;       65387; memory_kind=resident
process_memory_kB      ; 2025-04-25T15:08:53.99636522Z ;        2196; local_machine;            ; process      ;       65387; memory_kind=shared
process_memory_kB      ; 2025-04-25T15:08:53.99636522Z ;       17372; local_machine;            ; process      ;       65387; memory_kind=vmsize

The first line contains the name of the columns. Here is what they mean.

  • metric: the unique name of the metric that has been measured. With the default configuration of the csv plugin, the unit of the metric is appended to its name in the CSV (see Common plugin options). For instance, process_cpu_time is in milliseconds. Some metrics, such as kernel_n_procs_running, have no unit, they're numbers without a dimension.
  • timestamp: when the measurement has been obtained. Timestamps are serialized as RFC 3339 date+time values with nanosecond resolution, in the UTC timezone (hence the Z at the end).
  • value: the value of the measurement. For instance, kernel_n_procs_running has a value of 6 at 2025-04-25T15:08:53.949919481.
  • resource_kind and resource_id indicate the "resource" that this measurement is about. It's usually a piece of hardware. The special data local_machine (with an empty value for the resource id) means that this measurement is about the entire system. If a CPU-related plugin was enabled, it would have produced measurements with a resource kind of cpu_package and a resource id corresponding to that package.
  • consumer_kind and consumer_id indicate the "consumer" that this measurement is about: who consumed the resource? It's usually a piece of software. For instance, local_machine means that there was no consumer, the measurement is a global system-level measurement such as the total amount of memory or the temperature of a component. process;1 (kind process, id 65387) means that this measurement concerns the consumption of the process with pid 65387.

Additional attributes in the CSV output

Alumet measurements can contain an arbitrary number of additional key-value pairs, called attributes. If they are known early, they will show up as separate CSV columns. However, it's not always the case. If an attribute appears after the CSV header has been written, it will end up in the __late_attributes column. Attributes in __late_attributes should be treated just like attributes in separate CSV columns.

In the output example, measurements with the process_cpu_time metric have an additional attribute named cpu_state, with a string value. It refines the perimeter of the measurement by indicating which type of CPU time the measurement is about. In particular, Linux has separate counters for the time spent by the CPU for a particular process in user code or in kernel code.

Configuration file

Alumet and its plugins can be configured by modifying a TOML file. If you don't know TOML, it's a configuration format that aims to be "a config file for humans", easy to read and to modify.

Where is the configuration file?

Packaged Alumet agent

Pre-built Alumet packages contain a default configuration file that applies to every Alumet agent on the system. It is located at /etc/alumet/alumet-config.toml.

Standalone binary or Docker image

Installing the Alumet agent with cargo or using one of our Docker images (see Installing Alumet) does not set up a default global configuration file.

Instead, the agent automatically generates a local configuration file alumet-config.toml, in its working directory, on startup. The content of the configuration depends on the plugins that you enable with the --plugins flag.

Overriding the configuration file 📄

You can override the configuration file in two ways:

  • Pass the --config argument.
  • Set the environment variable ALUMET_CONFIG.

The flag takes precedence over the environment variable.

If you specify a path that does not exist, the Alumet agent will attempt to create it with a set of default values.

Example with the flag:

alumet-agent --plugins procfs,csv --config my-config.toml exec sleep 1

Example with the environment variable:

ALUMET_CONFIG='my-config.toml' alumet-agent --plugins procfs,csv exec sleep 1

Regenerating the configuration file 🔁

If the config file does not exist, it will be automatically generated.

But it is sometimes useful to manually generate the configuration file, for instance when you want to review or modify the configuration before using it. It is also useful when you change your environment in a way that is potentially incompatible with the current configuration, for example when updating the agent to the next major version.

The following command regenerates the configuration file:

alumet-agent config regen

It takes into account, among others, the list of plugins that must be enabled (specified by --plugins). Disabled plugins will not be present in the generated config.

Common plugin options

Each plugin is free to declare its own configuration options. Nevertheless, some options are very common and can be found in multiple plugins.

Please refer to the documentation of each plugin for an accurate description of its possible configuration settings.

Source trigger options

Plugins that provide measurements sources often do so by delegating the trigger management to Alumet. In other words, it is Alumet (core) that wakes up the measurement sources when needed, and then the sources do what they need to do to gather the data.

In that case, the plugin will most likely provide the following settings:

  • poll_interval: the interval between each "wake up" of the plugin's measurement source.
  • flush_interval: how long to wait before sending the measurements to the next step in Alumet's pipeline (if there is at least one transform in the pipeline, the next step is the transform step, otherwise it's directly the output step).

These two settings are given as a string with a special syntax that represent a duration. For example, the value "1s" means one second, and the value "1ms" means one millisecond.

Here is an example with the rapl plugin, which provide one measurement source:

[plugins.rapl]
# Interval between each measurement of the RAPL plugin.
# Most plugins that provide measurement sources also provide this configuration option.
poll_interval = "1s"

# Measurements are kept in a buffer and are only sent to the next step of the Alumet
# pipeline when the flush interval expires.
flush_interval = "5s"

# Another option (specific to the RAPL plugin, included for exhaustivity purposes)
no_perf_events = false

Note that the rapl plugin defines a specific option no_perf_events on top of the common configuration options for measurement sources.

Output formatting options

Plugins that provide outputs are often able to slightly modify the data before it is finally exported. Here are some options that are quite common among output plugins:

  • append_unit_to_metric_name (boolean): If set to true, append the unit of the metric to its name.
  • use_unit_display_name (boolean): If set to true, use the human-readable display name of the metric unit when appending it to its name. If set to false, use the machine-readable ASCII name of the unit. This distinction is based on the Unified Code for Units of Measure, aka UCUM. This setting does nothing if append_unit_to_metric_name is false.
    • Example: the human-readable display name of the "microsecond" unit is µs, while its machine-readable unique name is us.

Here is an example with the csv plugin, which exports measurements to a local CSV file.

[plugins.csv]
# csv-specific: path to the CSV file
output_path = "alumet-output.csv"

# csv-specific: always flush the buffer after each operation
# (this makes the data visible in the file with less delay)
force_flush = true

# Common options, described above
append_unit_to_metric_name = true
use_unit_display_name = true

# csv-specific: column delimited
csv_delimiter = ";"

OpenTelemetry Output Plugin

Description

OpenTelemetry (OTEL) is an open source observability framework and toolset designed to ease the integration of observability backends such as Jaeger, Prometheus, Elasticsearch, OpenSearch and more. While it offers vendor/tool-agnostic and auto-instrumentation capabilities, the backend (storage) and the frontend (visualization) of telemetry data are intentionally left to other tools.

The plugin developed is an exporter (push) which can be connected to the OTEL framework via a receiver.

Configuration

The plugin can be configured via the alumet's config.toml file with the following options:

[plugins.opentelemetry]
collector_host = "http://localhost:4317"
prefix = ""
suffix = "_alumet"
append_unit_to_metric_name = true
use_unit_display_name = true
add_attributes_to_labels = true
push_interval_seconds = 15

Examples

The plugin has been tested on both, local environment using docker-compose.yaml and a K8s.

OpenSearch local example

OpenSearch is a distributed search and analytics engine that can be used as vector database, full-text search and observability backend for logs, metrics and traces.

The connection to OpenSearch was done following the official Data Prepper tutorial which is an additional tool that translates OTEL protocol to OpenSearch protocol, as described here.

Notes:

  • For clarity, I disconnected traces and metrics from other sources to better visualize in OpenSearch what comes from alumet.
  • Also, the "logging" exporter is deprecated and needs to be updated.
# data-prepper/examples/metrics-ingestion-otel/otel-collector-config.yml
receivers:
  # hostmetrics:
  #   collection_interval: 60s
  #   scrapers:
  #     cpu:
  #     memory:
  # prometheus:
  #   config:
  #     scrape_configs:
  #       - job_name: data-prepper
  #         metrics_path: /metrics/sys
  #         scrape_interval: 60s
  #         static_configs:
  #           - targets: ['data-prepper:4900']
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  debug: # Appears as "logging" but is deprecated
    verbosity: detailed

Alumet was left with the default configuration and resulted in the correct population of the OpenSearch database and ability to explore the data via the dashboards as shown in the figure below.

demo

Thanos K8s example

Thanos is a set of components that can be composed into a highly available metric system with unlimited storage capacity for Time Series Data Base (TSDB), which can be added seamlessly on top of existing Prometheus deployments. The component "Thanos Receive" has the Prometheus Remote Write API built in, on top of the functionality for long-term-storage and downsampling. Since the OTEL Exporter natively implements the Prometheus Remote Write protocol, it can directly upload TSDB blocks to the object storage bucket of Thanos even without an underlying Prometheus.

Alumet was configured as following and deployed in the host of a single-node K8s cluster which had an OTEL collector using the operator and Thanos.

# Alumet config
[plugins.opentelemetry]
collector_host = "http://my-otel-collector-exposed-service"
prefix = ""
suffix = "_alumet"
append_unit_to_metric_name = true
use_unit_display_name = true
add_attributes_to_labels = true
push_interval_seconds = 15

# Otel values.yaml
# Top level field related to the OpenTelemetry Operator
opentelemetry-operator:
  # Field indicating whether the operator is enabled or not
  enabled: true
  manager:
    collectorImage:
      repository: otel/opentelemetry-collector-contrib
  # Sub-field for admission webhooks configuration
  admissionWebhooks:
    # Use Helm to automatically generate self-signed certificate.
    certManager:
      enabled: false
    autoGenerateCert:
      enabled: true
collectors:
  otelgateway:
    suffix: gateway
    replicas: 1
    mode: deployment
    enabled: true
    config:
      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
      processors:
        batch:
          send_batch_size: 1024
          timeout: 1s
      exporters:
        prometheusremotewrite:
          endpoint: "http://my-thanos-receive-service.monitoring.svc.cluster.local:19291/api/v1/receive"

      service:
        pipelines:
          metrics:
            receivers:
              - otlp
            processors:
              - batch
            exporters: 
              - prometheusremotewrite

Prometheus Exporter Output Plugin

Description

Prometheus is an open source system for monitoring and alerting based on the collection (pull-based) of metrics in a Time Series Data Base (TSDB).

The plugin developed is an standard prometheus exporter which is exposed as a web service (with endpoint /metrics) to be reached by the prometheus system to pull the alumet's metrics.

Configuration

The plugin can be configured via the alumet's config.toml file with the following options:

[plugins.prometheus-exporter]
host = "0.0.0.0"
prefix = ""
suffix = "_alumet"
port = 9091
append_unit_to_metric_name = true
use_unit_display_name = true
add_attributes_to_labels = true

Examples

The plugin has been tested on both, local environment and a K8s.

Simple local example

The plugin can be tested in a local environment with the Alumet's default configuration for it and results in the ability of downloading the metrics from http://localhost:9091/metrics.

Note that the list of metrics available will vary based on the input plugins used.

# HELP mem_total_kB_alumet ?.
# TYPE mem_total_kB_alumet gauge
mem_total_kB_alumet{resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 12372303872.0
# HELP mem_free_kB_alumet ?.
# TYPE mem_free_kB_alumet gauge
mem_free_kB_alumet{resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 1491062784.0
# HELP mem_available_kB_alumet ?.
# TYPE mem_available_kB_alumet gauge
mem_available_kB_alumet{resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 5022056448.0
# HELP cached_kB_alumet ?.
# TYPE cached_kB_alumet gauge
cached_kB_alumet{resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 3523051520.0
# HELP swap_cached_kB_alumet ?.
# TYPE swap_cached_kB_alumet gauge
swap_cached_kB_alumet{resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 54939648.0
# HELP active_kB_alumet ?.
# TYPE active_kB_alumet gauge
active_kB_alumet{resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 2447753216.0
# HELP inactive_kB_alumet ?.
# TYPE inactive_kB_alumet gauge
inactive_kB_alumet{resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 7838990336.0
# HELP mapped_kB_alumet ?.
# TYPE mapped_kB_alumet gauge
mapped_kB_alumet{resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 230359040.0
# HELP kernel_cpu_time_ms_alumet busy CPU time.
# TYPE kernel_cpu_time_ms_alumet gauge
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="11",resource_kind="cpu_core"} 4950.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="1",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="7",resource_kind="cpu_core"} 4900.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="2",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="11",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="11",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="0",resource_kind="cpu_core"} 30.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="3",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="9",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="9",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 270.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="7",resource_kind="cpu_core"} 10.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="6",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="2",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="5",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="2",resource_kind="cpu_core"} 50.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="11",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="4",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="5",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="0",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="2",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="3",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="8",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="11",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="5",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="3",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="4",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="2",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="6",resource_kind="cpu_core"} 50.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="1",resource_kind="cpu_core"} 20.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="2",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="8",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="6",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="9",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="8",resource_kind="cpu_core"} 10.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="0",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="9",resource_kind="cpu_core"} 4990.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="10",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="8",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="0",resource_kind="cpu_core"} 4960.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="11",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="7",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="9",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="4",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="10",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="10",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="6",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="4",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="5",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 130.0
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 59500.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="7",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="10",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="3",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="7",resource_kind="cpu_core"} 60.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="8",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="0",resource_kind="cpu_core"} 20.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="0",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="10",resource_kind="cpu_core"} 4960.0
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="1",resource_kind="cpu_core"} 4980.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="10",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="3",resource_kind="cpu_core"} 5000.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="11",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="0",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="9",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="10",resource_kind="cpu_core"} 30.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="1",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="6",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="9",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="11",resource_kind="cpu_core"} 50.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="0",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="4",resource_kind="cpu_core"} 20.0
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="4",resource_kind="cpu_core"} 4980.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="10",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="5",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="8",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="9",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="11",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="1",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="3",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest_nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="5",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="4",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="7",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="5",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="6",resource_kind="cpu_core"} 4930.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="7",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="1",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="2",resource_kind="cpu_core"} 4880.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="1",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="4",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="6",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="2",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="6",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="5",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="3",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="0",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="1",resource_kind="cpu_core"} 10.0
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="5",resource_kind="cpu_core"} 5000.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="7",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="9",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="6",resource_kind="cpu_core"} 20.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="2",resource_kind="cpu_core"} 60.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="3",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="nice",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="7",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="user",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="10",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="irq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="3",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="system",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="1",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="4",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="idle",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="8",resource_kind="cpu_core"} 4980.0
kernel_cpu_time_ms_alumet{cpu_state="steal",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="8",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="guest",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="8",resource_kind="cpu_core"} 0.0
kernel_cpu_time_ms_alumet{cpu_state="softirq",resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 40.0
# HELP kernel_context_switches_alumet number of context switches.
# TYPE kernel_context_switches_alumet gauge
kernel_context_switches_alumet{resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 4385.0
# HELP kernel_new_forks_alumet number of fork operations.
# TYPE kernel_new_forks_alumet gauge
kernel_new_forks_alumet{resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 33.0
# HELP kernel_n_procs_running_alumet number of processes in a runnable state.
# TYPE kernel_n_procs_running_alumet gauge
kernel_n_procs_running_alumet{resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 2.0
# HELP kernel_n_procs_blocked_alumet numbers of processes that are blocked on I/O operations.
# TYPE kernel_n_procs_blocked_alumet gauge
kernel_n_procs_blocked_alumet{resource_consumer_id="",resource_consumer_kind="local_machine",resource_id="",resource_kind="local_machine"} 0.0
# EOF

K8s example

For the K8s demo, Alumet was installed with the it's default configuration in the host of a NUC where there was a single-node k3s cluster. After installing the default kube-prometheus-stack, the following manifest was deployed to scrape the exporter:

apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: alumet-host-scrape
  labels:
    release: promstack
spec:
  staticConfigs:
    - labels:
        job: alumet-host
      targets:
        - my_host_ip:9091

For the demo, the "stress --cpu 8 --io 4 --vm 2 --vm-bytes 128M" command was used to ensure that data was pulled correctly. This resulted in Grafana and Prometheus to correctly capture the behavior of the system as shown in the figure below.

demo