What Is IRONdb?

IRONdb is a distributed time-series database focusing on simplistic operation, resiliency, continued operations in the event of component failure, and embedding analytics and computation.

Getting Started

Administration

API

Integrations

Tools

Although numeric shards can be configured with , this only removes entire shards once they are past the window. In cases where one has all data for a significant number of metrics, the storage space they occupy in rollups may be recovered by performing a compaction of one or more NNTBS shards using a map of active IDs from the .

Compaction is performed by running the shard_compactor tool. It has two required arguments:

• -d <nntbs_dir> - The path where NNTBS shards are stored. This is typically found under /irondb/nntbs, or /snowth/nntbs on deployments hosted by Circonus. The directory name matches the node's cluster UUID.

API description: See "Rebalance" in the

Getting Topology Rebalance State

This API call is for viewing the current topology rebalance state.

Data will be returned as a JSON document. The fields in this document are described below.

Starting with release 0.12, IRONdb supports tracking of metric activity without the expense of reading all known time series data to find active ranges. The activity of a metric is tracked at a 5 minute granularity. Any ingestion of a metric will mark that 5 minute period that the timestamp falls into as active for that metric. Activity periods are stored in the surrogate database.

This activity tracking also coalesces nearby active ranges. Any activity on a metric within an 8 hour window marks that metric as active for that 8 hour span. For example, if you have a metric that arrived with the timestamp: 2018-07-03T11:00:01:123Z and then nothing else arrived until 2018-07-03T19:00:02:123Z, the metric would be considered inactive in the 8 hour span between these 2 timestamps. If, later, some late data arrives and we see a timestamp at: 2018-07-03T14:00:01:123Z, then the entire 8 hour span is considered active for purposes of querying.

See Searching Tags on how to query activity periods for a given list of metrics.

This activity tracking only applies to data ingested after the upgrade to 0.12 or later. Any data ingested prior to installation of 0.12 will be invisible to the activity tracking code. However, IRONdb also ships with an API to rebuild activity tracking data by reading the actual datapoints for a metric to determine its activity ranges. Since this is an expensive operation it has to be triggered for a list of metrics by an operator.

Rebuilding Activity Data

Do not trigger this API until you have upgraded all IRONdb nodes to 0.12 or later.

URI

• /surrogate/activity_rebuild

Method

POST

Inputs

A JSON document which lists the set of metrics to rebuild activity data for, with the syntax:

The above will rebuild activity for the 3 metrics listed in the document.

IRONdb supports remote write and read capabilities to provide long-term metric storage for Prometheus deployments. One IRONdb cluster can support many individual Prometheus instances.

Load balancing requests

Both read and write requests to IRONdb can safely go to any node in an IRONdb cluster. To ensure high availability and distribute load, users are encouraged to put a load balancer between the Prometheus nodes and the cluster.

Prometheus Ingestion

IRONdb has native endpoints for accepting remote write data from a Prometheus installation. Once the Prometheus module is enabled, data can be sent to IRONdb by setting the Prometheus remote_write endpoint to:

http://irondbnode:8112/module/prometheus/write/<accountid>/<uuid>

Namespacing

Prometheus data is not namespaced by nature. This can create confusion if different copies of Prometheus have identically named metrics. Inside of IRONdb, we require that all data be namespaced under a UUID. This UUID can be created using uuidgen on a typical UNIX(like) system or via any external tool or website that generates UUIDs. Each distinct set of Prometheus data should have its own UUID. For high-availability in Prometheus it is the recommended pratice to have two copies collecting the same data. While these two instances do not contain the same data, they do represent the same metrics, and so should share a common UUID for their namespace. One may wish to send both of these instances into IRONdb where they simply become more samples in the given metric stream.

All metrics live under a numeric identifier (one can think of this like an account ID). Metric names can only be associated with one "account ID". This allows separate client instances that completely segregate data.

Writing Prometheus Data to IRONdb

To configure a Prometheus instance to write to IRONdb the Prometheus YAML configuration file will need to be updated. The remote_write section's url field should be set to http://irondbnode:8112/module/prometheus/write/<accountid>/<uuid>.

This should look something like:

Reading Prometheus Data from IRONdb

To configure a Prometheus instance to use IRONdb as a remote datasource, the Prometheus YAML configuration file will need to be updated. The remote_read section's url field should be set to http://irondbnode:8112/module/prometheus/read/<accountid>/<uuid>.

This should look something like:

But with an account ID and UUID value matching what was configured in the remote write URL.

In contrast to or , operational needs may call for migrating a cluster to a new set of machines entirely. This may be due to hardware lifecycle requirements and/or the desire to modify the topology all at once.

As with individual node reconstitution, this is a "pull"-type operation, where the new cluster's nodes pull the necessary metric data from the source cluster. The following procedure will be run on each of the new cluster's nodes. Multiple new-cluster nodes can reconstitute simultaneously if the source cluster has sufficient read capacity, but exercise care, since every reconstituting node will read from every source cluster node.

Prerequisites

Reconstitution requires that at least one replica of every metric stream stored on the existing cluster be available. A reconstitute operation cannot complete if more than W-1 nodes of the existing cluster are unavailable (

This is intended as a general guide to determining how many nodes and how much storage space per node you require for your workload. Please contact Apica if you have questions arising from your specific needs.

Key Terminology

• T is the number of unique metric streams.

• N is the number of nodes participating in the cluster.

• W is the number of times a given measurement is stored across the cluster.

  • For example, if you have 1 GB of metric data, you must have W GB of storage space across the cluster.

The value of W determines the number of nodes that can be unavailable before metric data become inaccessible. A cluster with W write copies can survive W-1 node failures before a partial data outage will occur.

Metric streams are distributed approximately evenly across the nodes in the cluster. In other words, each node is responsible for storing approximately (T*W)/N metric streams. For example, a cluster of 4 nodes with 100K streams and W=2 would store about 50K streams per node.

Rules of Thumb

• Nodes should be operated at no more than 70% capacity.

• Favor ZFS striped mirrors over other pool layouts. This provides the highest performance in IOPS.

• W must be >= 2

• N must be >=

Storage Space

The system stores three types of data: text, numeric (statistical aggregates), and histograms. Additionally there are two tiers of data storage: near-term and long-term. Near-term storage is called the and stores at full resolution (however frequently measurements were collected.) Long-term resolution is determined by the .

The default configuration for the raw database is to collect data into shards (time buckets) of 1 week, and to retain those shards for 4 weeks before rolling them up into long-term storage. At 1-minute collection frequency, a single numeric stream would require approximately 118 KiB per 1-week shard, or 472 KiB total, before being rolled up to long-term storage.

These numbers represent uncompressed data. With our default LZ4 compression setting in ZFS, we see 3.5x-4x compression ratios for numeric data.

The following modeling is based on an observed distribution of all data types, in long-term storage, across many clients and may be adjusted from time to time. This would be in addition to the raw database storage above.

All sizing above represents uncompressed data.

Sizing Example

Suppose we want to store 100,000 metric streams at 1-minute resolution for 5 years. We'd like to build a 4-node cluster with a W value of 2.

Hardware Choices

Apica recommends server-class hardware for all production deployments. This includes, but is not limited to, features like ECC memory and hot-swappable hard drives.

• See for general advice.

  • Specifically, hardware RAID should be . ZFS should be given access to raw hard drive devices whenever possible.

In addition to the overall storage space requirements above, consideration must be given to the IOPS requirements. The minimum IOPS required is the primary write load of ingesting metric data (approximately 12 bytes per measurement point), but there is additional internal work such as parsing and various database accounting operations that can induce disk reads beyond the pure writing of measurement data. After initial ingestion there are other operations, such as searching, rollups, and maintenance activity like reconstitution and ZFS scrubbing that require additional IOPS. Ensure that the hardware you choose for your nodes has the capacity to allow for these operations without significantly impacting ongoing ingestion.

ZFS's helps by absorbing some portion of the read load, so the more RAM available to the system, the better.

Hardware Profiles

The following are sample profiles to guide you in selecting the right combination of hardware and cluster topology for your needs.

Assumptions:

• 10-second collection frequency

• 4 weeks of near-term (full-resolution) storage

• 2 years of historical data at 1-minute resolution

• striped-mirror ZFS pool layout

If an IRONdb node or its data is damaged or lost, its data may be rebuilt from replicas elsewhere in the cluster. This process is known as "reconstituting" a node.

Prerequisites

Reconstitution requires that at least one replica of every metric stream stored on the reconstituting node be available. A reconstitute operation cannot complete if more than W-1 nodes are unavailable, including the node being reconstituted (W is the number of write_copies configured for the current topology.)

For example, given a cluster of 10 nodes (N=10) with 3 write copies (W=3), a node may be reconstituted if at least N-(W-1), or 8, other nodes are available and healthy.

As this can be a long-running procedure, a terminal multiplexer such as tmux or screen is recommended to avoid interruption.

Reconstitute Procedure

1. Log into the IRONdb node you wish to reconstitute as root or a privileged user. Make sure the IRONdb package is .

   • Note: If the entire old node was replaced (e.g., due to a hardware failure), or the ZFS pool has been recreated (due to hardware failure or administrative action), then you should repeat and then . The installer will not interfere with an existing irondb.conf file but will ensure that all necessary ZFS datasets and node-id subdirectories have been created.

2. Make note of this node's topology UUID, found in the

Deleting All Data for a Metric or a Set of Metrics

This API call is for deleting all of the data from an IRONdb node for a specific metric or for a set of metrics (when a tag query is specified). It will remove data for the matching metric(s) throughout all timestamps and all rollups that have been provided by the user, no matter what the data type. In addition, it will remove all record of the metric name(s) with their tags and metadata. This call is intended for removing misnamed/experimental metrics or old metrics which are obsolete and can be safely removed.

When used for deletion of a single metric, this call will return a JSON object that reports if the request succeeded or not.

When used with wildcards or a tag query, this call always returns a JSON object which describes the matching metrics and the actions taken or errors received on the deletion. A list of the possible result statuses for each metric and what they mean can be found here. For safety, explicit confirmation is required in the headers to actually force the data deletion.

It is highly recommended to perform the deletion API call without confirmation as a first step, in order to review what would actually be deleted (and hopefully avoid accidentally deleting more data than intended).

Deletion is currently only supported on a single node per API call. To delete data from the entire cluster, issue the same API call to each node.

API description: See "Data Deletion" in the

Single Metric Example

In this example:

• full : This tells the system that full data and metadata will be removed for the specified metric.

• canonical : This tells the system to delete a single metric that matches the given UUID and metric name.

• 1234 : Delete data only for the given account id

Sample Output for Single Metric Example

Query Example

In this example:

• full : This tells the system that all data and metadata for the matching metrics will be removed.

• tags : This tells the system that this is a tag query.

• 1234 : Delete data only for the given account id

Sample Output for Query Example

Wildcard, Tag Query and Check Delete Result Statuses

When doing a delete which could affect multiple metrics, the returned JSON response will indicate the final status for each metric which matched the request. A list of these statuses and a description is given below. Note that, in many cases, the "payload' field will contain further details.

• Bad request : The URI did not conform to expected syntax or inputs for the API

• Deleted : Data was found and the deletion completed successfully

• Found : Data was found that can be deleted if request is submitted again with delete confirmation

0.1.4

2025-09-02

• Add support for Prometheus data - both an API endpoint and from Kafka using the libmtev Kafka module.

• Added ability to configure irondb-relay to drain journals on shutdown instead of just exiting. The default is still to just exit.

0.1.3

2025-04-01

• Update error logging to be more accurate and provide more detail.

0.1.2

2025-03-07

• Update Docker base image to be Ubuntu 22.04.

• Improve graphite read error messages.

0.1.1

2024-03-27

• Update libmtev dependency, which fixes potential memory corruption issues.

0.1.0

2024-01-31

• Add TLS support

0.0.57

2024-01-25

• Fix Docker build to bust apt caches and avoid errors.

• Update setup script to better support HTTPS URLs in the bootstrap list.

• Add C++ guards to headers and convert send code to C++ to take advantage of libsnowth features.

0.0.56

2023-11-06

• Add Docker support.

0.0.55

2023-09-05

• Use new libsnowth_init function to avoid potential buffer overflow.

0.0.54

2023-06-06

• Remove unused DH parameter files from configuration.

0.0.53

2023-03-06

• Fix simdjson linking.

0.0.52

2022-09-14

• Fix log rotation.

0.0.51

2022-06-09

• Initialize metric_t structures to avoid data corruption.

0.0.50

2022-02-07

• Replace deprecated mtev_atomic* types and functions with compatibles ones from ConcurrencyKit (libck).

0.0.49

2022-02-04

• Fix an issue where some jlog subscribers were not advanced when they did not have work to do. This led to increased disk usage from processed segments that could not be removed.

0.0.48

2021-04-09

• Bring setup and start scripts into the repo.

0.0.47

2021-03-24

• Improved error handling/data parsing.

• Accept UTF-8 Graphite data.

• Move debug/parsing log to debug/parsing/graphite and add error/parsing/graphite log to catch parsing errors.

Each IRONdb node exposes a wealth of information about its internal operation. There are two ways to obtain this data: pulling JSON from a REST stats endpoint, or having IRONdb push its own stats into a particular account/check using a loadable module. In both cases, the metrics exposed are the same.

The types of statistics available are described in the section of the Operations page.

The JSON endpoint is best for viewing live information. The internal monitor module is best suited to long-term trending in standalone IRONdb deployments. Its metrics may be retrieved using one of the type-specific .

Both methods are described below.

JSON

JSON-formatted metrics are available from two REST endpoints, each having two format options:

This guide assumes a server with the following storage configuration:

• One or more OS drives with ext4 on LVM, md, etc., depending on installer choices and/or operator preference.

• 12 data drives attached via a SAS or SATA HBA (non-RAID) that will be used exclusively for ZFS.

Writing Raw Data

The raw API accepts direct input of measurement data at arbitrary frequencies. It stores every measurement as it was received, for a configurable amount of time, before aging it out to a rollup format.

Metric records are in one of several formats, and are accepted as either tab-separated values or as FlatBuffer messages.

API description: See "Data Submission" in the

This is the plugin for IRONdb 0.17.1 and newer. It is evolving and we continue to track its API.

Installation

• The default location for the plugins directory is /var/lib/grafana/plugins, though the location may be different in your installation, see for more plugin information.

The essential steps to changing the topology of an existing IRONdb cluster are as follows:

• Create your new topology.

• Load the new topology to all nodes that will be part of the new cluster.

• Start the "rebalance" operation on each node, which begins the migration of metric data to the new topology. Depending on the amount of stored data, this process may take a long time.

Rebalancing involves recalculating the node ownership for each individual metric stream, and then sending that stream to the new owning node. All metric data remain available during a rebalance, under the old topology. New, incoming metric data is replicated to

Reference to available options and arguments.

• Synopsis

• Process Control

• Operating Modes

To obtain the most current usage summary: /opt/circonus/sbin/snowthd -h

Synopsis

Process Control Options

• -k <start|stop|status>

status will exit 0 if the process is running, non-zero otherwise.

Operating Mode Options

These options are mutually exclusive of one another. One or the other is required.

• -i <uuid>

Identify this node with <uuid>. This is the normal mode of operation.

• -e

Boot the node in ephemeral mode. Ephemeral nodes are read-only participants in the cluster. They do not appear in the cluster topology, and do not accept incoming metrics, but may be used to read metric data from other nodes and perform intensive computation that would add unreasonable load to the main nodes.

Loader Options

These options imply foreground operation and perform a specific task, then exit. They are only valid in identified mode (-i).

• -m

Merge text reconstitution files. DEPRECATED

• -H

Merge histogram reconstitution files. DEPRECATED

The above 2 options were used in a previous version of the reconstitute process and are no longer strictly required. They may be removed in a future version.

Maintenance Options

These options imply foreground operation and perform a specific task, then exit. They are only valid in identified mode (-i).

• -r text/metrics

Repair text inventory.

• -r text/changelog

Repair text datastore.

• -r hist/metrics

Repair histogram inventory.

• -r hist/<rollup>

Repair a histogram rollup. The value is one of the existing histogram rollup periods from the config file, e.g., hist/60 to repair the 1-minute histogram rollups.

• -j

Journal-drain mode. Does not start a network listener, so this node will appear "down" to its peers, but will send any pending journal data to them. This is useful if you are planning to retire and replace a cluster node, and want to ensure that it has sent all outgoing journal data without accepting any new input.

Behavioral Options

These determine optional behavior, and are not required.

• -c <file>

Load configuration from <file>. Must be a full path. If not specified, the default path is /opt/circonus/etc/snowth.conf.

• -d

Activate additional debug logging. Use with caution; can generate a large volume of logs.

• -D

Stay in the foreground, rather than daemonizing. If specified once, run as a single process with no watchdog. If specified twice, run as a parent/child pair, with the parent (watchdog) process in the foreground.

See the for details on foreground operation.

• -u <user>

Drop privileges after start and run as this user.

• -g <group>

Drop privileges after start and run as this group.

• -t <path>

Chroot to <path> for operation. Ensure that log file locations may be accessed within the chrooted environment.

• -l <logname>

Enable <logname>, even if it is disabled in the configuration file. The specified log stream must exist.

• -L <logname>

Disable <logname>, even if it is enabled in the configuration file. The specified log stream must exist.

Reconstitute Options

These operations are used when .

• -B

Enable reconstitute mode.

• -T <topo_hash>

Reconstitute from this remote/foreign topology. Used when creating a new cluster from an existing one.

• -O <ip>[:<port>]

Bootstrap remote reconstitute from this node in the source cluster. Used when creating a new cluster from an existing one. The reconstituting node will fetch information about the source cluster's topology from this node, but actual metric data will be fetched from all source cluster nodes.

• -A <type>

Reconstitute one type of data, or all if the option is omitted. May be specified multiple times to reconstitute multiple data types.

• -S <node_uuid>

Skip the specified node(s) when pulling data for reconstitute. This is useful if a node is unavailable at the time a reconstitute is started. May be specified multiple times to skip more than one node. Use with caution. If the number of skipped nodes exceeds the number of data copies, the reconstitute may be incomplete.

IRONdb has native endpoints for accepting OpenTSDB-style data.

Ingestion Format

There are 2 methods for ingesting OpenTSDB data into IRONdb:

1. RESTful HTTP POST of OpenTSDB JSON formatted datapoint(s)

2. Network socket listener akin to the normal OpenTSDB telnet method

For the HTTP method, POST a JSON object (or an array of JSON objects) to the RESTful API endpoint (see the section below - Writing OpenTSDB with HTTP). Each datapoint should be encoded as follows:

At least one tag key/value pair is required. Multiple datapoints can be sent as a JSON array, separated by commas, and the entire POST enclosed in square brackets.

For example:

In the case of the telnet method, telnet put commands in the normal OpenTSDB format are accepted:

put<space>metric_name<space>timestamp<space>value<space>tag_key=tag_value{<space>tag_key2=tag_value2...<space>tag_keyn=tag_valuen}

At least one tag key/value pair must be included. For example:

put my.metric.name<space>1480371755<space>12345.56<space>datacenter=east

If you desire higher resolution data capture, you can suffix the timestamp with a period, followed by the number of milliseconds in the second, or simply just use 13 numeric digits without the period (the last three digits will become the millseconds). For example:

put my.metric.name<space>1480371964.123<space>12345.56<space>datacenter=east

Or just:

put my.metric.name<space>1480371964123<space>12345.56<space>datacenter=east

These two examples mean 123 milliseconds into the timestamp 1480371964 or November 28, 2016 10:26:04 and 123ms PM UTC

Note that, while it resembles a floating point number, this is not a float.

For data safety reasons, we recommend that you use the RESTful POST interface to send OpenTSDB-formatted JSON data. The network socket listener provides no feedback to the sender about whether or not data was actually ingested (or indeed even made it off the sender machine and was not stuck in an outbound socket buffer) because there is no acknowledgement mechanism on a raw socket.

The HTTP interface, on the other hand, will provide feedback about whether data was safely ingested and will not respond until data has actually been written by the underlying database.

Namespacing

Both of the interfaces require you to namespace your OpenTSDB data. This lets you associate a UUID/Name and numeric identifier with the incoming metrics. This is useful, for example, if you want to use a single IRONdb installation to service multiple different internal groups in your organization but keep metrics hidden across the various groups.

All metrics live under a numeric identifier (you can think of this like an account_id). Metric names can only be associated with an "account_id". This allows you have separate client instances that segregate queries for metric names, or combine them all together under a single "account_id", or even separate your internal groups but recombine them under the client for visualization purposes. It's really up to you.

Furthermore, IRONdb requires associating incoming OpenTSDB data with a UUID and Name to make OpenTSDB data match data ingested from native sources more closely on the Apica platform. We hide the complexity of this on the rendering side, so you only have to worry about this mapping on the ingestion side. This UUID can be created using uuidgen on a typical UNIX(like) system or via any external tool or website that generates UUIDs.

When we store these metric names inside IRONdb, we prefix them with our standard collection category ("reconnoiter" will be automatically assigned) and the "Name" of the check. You can see this in the examples below in more detail.

Adding these additional fields allow us to disambiguate metric names from potential duplicate names collected from other sources.

Optional Configuration

OpenTSDB ingestion will, by default, accept timestamps up to 1 year in the past. This value may be changed through .

Writing OpenTSDB Data with HTTP

OpenTSDB data is sent by POSTing a JSON object or an array of JSON objects using the format described above to the OpenTSDB ingestion endpoint:

http://<irondb_machine:port>/opentsdb/<account_id>/<uuid>/<check_name>

For example:

http://192.168.1.100:4242/opentsdb/1/8c01e252-e0ed-40bd-d4a3-dc9c7ed3a9b2/dev

This will place all metrics under account_id 1 with that UUID and call them dev.

http://192.168.1.100:4242/opentsdb/1/45e77556-7a1b-46ef-f90a-cfa34e911bc3/prod

This will place all metrics under account_id 1 with that UUID and call them prod.

Writing OpenTSDB Data with Network Listener

The network listener requires that we associate an account_id, uuid, and name with a network port. This is added to the during initial installation, for the default OpenTSDB text protocol port (4242). Additional stanzas may be added, associating different IDs with different ports to segregate incoming traffic.

You can then use:

to send metrics to IRONdb, and it will store the datapoint under the supplied metric name with the account, uuid, and name that was provided by the configuration for the port that was used.

Retrieving and Transforming Data

The /fetch API provides fast, one-request access to common complex data extraction requirements. It allows for fetch submissions in both FlatBuffers and JSON formats, and returns DF4 output format available in both FlatBuffers and JSON encoding.

API description: See "Retrieving and Transforming Data" under Developer API

Transforms

Numeric (kind = numeric)

• average - the average of measurements in the period.

• sum - the sum of measurements in the period.

• count - the number of measurements in the period.

Histogram (kind = histogram)

• none - pass the input through unmodified.

• count - the number of samples in each histogram.

• rate - the number of samples per second in each histogram (count/period).

Text (kind = text)

• none - pass the input through unmodified.

• count - return a numeric count of the number of text entries in the period.

• count_cumulative - return a cumulative count of text entries starting at zero for the first period requested.

Reductions

pass - pass the inputs to outputs unmodified

• method_params none

• Inputs can be numeric, histogram, or text.

groupby_mean - group inputs and calculate a mean over the grouping

• method_params a list of tag categories on which to perform grouping

• Inputs must be numeric.

groupby_sum - group inputs and calculate a sum over the grouping

• method_params a list of tag categories on which to perform grouping

• Inputs must be numeric.

groupby_merge - group inputs and merge into a histogram stream

• method_params a list of tag categories on which to perform grouping

• Inputs must be either numeric or histogram.

mean - calculate the mean across input streams

• method_params none

• Inputs must be numeric.

merge - group inputs and merge into a histogram stream

• method_params none

• Inputs must be either numeric or histogram.

sum - calculate the sum across input streams

• method_params none

• Inputs must be numeric.

topk - filter a set of inputs to the top K

• method_params : [ K, <mech>, <mech_param> ]

• Inputs must be either numeric or histogram.

• Allowable mech values are mean (default), max, or quantile

Retrieving Graphite Data

Fetches Graphite-style data. The data returned is always average data and this endpoint will scale the rollup_span to match the time range of data requested.

See .

A plugin for using graphite with the IRONdb from Apica.

Requires Graphite-web 1.1.X.

Installation

First, checkout the code:

(With no options provided, the install will look for a library in /opt/circonus)

Then, to install using library for FlatBuffers:

Or:

API description: See "Internal Observability" in the Administration API

Getting Node State

This API call is for viewing the system state of the current node.

Data will be returned as a JSON document. The fields in this document are described below.

State Output

• identity : The UUID that identifies this node.

• current : The current topology in which this node resides.

• next : The next topology for this node. A value of "-" indicates there is no next topology.

Retrieving Gossip JSON Data

This API call retrieves gossip information from a IRONdb node. Gossip data is information on how the nodes are communicating with each other and if any nodes are behind other nodes with regards to data replication.

Data will be returned as an array of JSON objects. The format of these objects is described below.

API description: See "Internal Observability" in the

Gossip JSON Output

Each object in the array has the following form:

• id : The UUID of the node whose gossip information follows.

• gossip_time : The last time, in seconds, that this node received a gossip message.

• gossip_age : The difference, in seconds, between the last time this node received a gossip message and the current time.

Retrieving Gossip XML Data

This API call retrieves gossip information from a Snowth node. Gossip data is information on how the nodes are communicating with each other and if any nodes are behind other nodes with regards to data replication.

Data will be returned an XML object. The format of this object is described below.

API description: See "Internal Observability" in the

Gossip XML Output

• <nodes> : The top-level element for the topology.

  • <node> : The container for all the information for a single node in the cluster. There will x of these elements, where "x" is the number of nodes in the cluster.

    • Attributes:

IRONdb is a drop-in replacement for Graphite's Whisper database.

It supports ingestion from Carbon sources like carbon-relay and carbon-c-relay. Graphite-irondb is a storage finder plugin that allows IRONdb to seamlessly integrate with an organization's existing Graphite-web deployment.

The IRONdb Relay is a scalable, drop-in replacement for carbon-relay or carbon-c-relay.

Graphite Ingestion

The format for ingestion is the typical Carbon plaintext format:

dot.separated.metric.name<space>12345.56<space>1480371755

If you desire higher resolution data capture, IRONdb does support a variant of the unix epoch timestamp (3rd field) where you can suffix the timestamp with a period, followed by the number of milliseconds in the second. For example:

dot.separated.metric.name<space>12345.56<space>1480371964.123

This example means 123 milliseconds into the timestamp 1480371964 or November 28, 2016 10:26:04 and 123ms PM UTC

Note that, while it resembles a floating point number, this is not a float.

Starting with IRONdb release 0.12 you can also ingest tagged graphite data. Tagged graphite data has the following format:

dot.separated.metric.name;category1=value1;category2=value2

Where tags are appended to the normal name and are separated by semicolons (;).

For more info on the graphite tag format see: .

Namespacing

Graphite ingestion into IRONdb requires namespacing your graphite data. This lets you associate a UUID/Name and numeric identifier with the incoming metrics. This is useful, for example, if you want to use a single IRONdb installation to service multiple different internal groups in your organization but keep metrics hidden across the various groups.

All metrics live under a numeric identifier (you can think of this like an account_id). Metric names can only be associated with an "account_id". This allows you have separate graphite-web or Grafana instances that segregate queries for metric names, or combine them all together under a single "account_id", or even separate your internal groups but recombine them under graphite-web/Grafana for visualization purposes. It's really up to you.

Optional Configuration

Graphite ingestion will, by default, accept timestamps up to 1 year in the past. When retrieving Graphite data, a floor of 1-minute resolution is used, to prevent gaps if the requested period is shorter. These values may be changed through .

Writing Graphite Data with Network Listener

The network listener requires that we associate an account_id, uuid, and name with a network port. This is added to the during initial installation, for the default Graphite text protocol port (2003). Additional stanzas may be added, associating different IDs with different ports to segregate incoming traffic.

You can then use:

to send metrics to IRONdb.

See also the

Graphite Rendering

IRONdb has a graphite-web Storage Backend which makes the following Graphite Rendering seamless with an existing graphite-web installation. The Storage Backend requires graphite 0.10 or newer and can be obtained :

Follow the instructions in the README in that repo to install and utilize the IRONdb graphite storage backend.

That Storage Backend plugin simply utilizes the endpoints described below.

Query Result Limits

All query results are subject to limits to control the number of results returned. If not otherwise specified, queries will be limited to the first 10,000 results returned.

This limit may be changed by setting a request header, x-snowth-advisory-limit, with one of the following values:

• A positive integer representing the desired limit

• -1 or "none" to remove the limit

If the header contains any other value or is not present, the default of 10,000 will be used.

Searching for Metric Names

Graphite metrics can be fetched (rendered) from IRONdb using the following endpoints. Glob style wildcards are supported.

http://<host:port>/graphite/<account_id>/<optional_query_prefix>/metrics/find?query=foo.*

This will return a JSON document with metrics matching the prefix: foo. which terminate at that level. Continuing on the example in Graphite Ingestion, the above example could return the following:

When a metric is a leaf node, leaf will be true and that metric will be queryable for actual datapoints.

The optional_query_prefix can be used to simplify metric names. You can place any non-glob part of the prefix of a query into the optional_query_prefix and that prefix will be auto-prefixed to any incoming query for metric names. For example:

http://<host:port>/graphite/1/foo./metrics/find?query=*

Will return:

Note that the optional_query_prefix is omitted from the response json. You would use this feature to simplify all metric names in graphite-web or Grafana.

If you do not want to utilize the optional_query_prefix you can leave it off the URL:

http://<host:port>/graphite/1/metrics/find?query=foo.*

Searching for Tags

Graphite metrics can be fetched (rendered) from IRONdb using multi-dimensional tag queries.

http://<host:port>/graphite/<account_id>/<optional_query_prefix>/tags/find?query=<tag query>

This will return a JSON document with metrics matching the <tag query>. Tag query syntax is the same as supported by Graphite version >= 1.1. See

The syntax is:

http://<host:port>/graphite/1/tags/find?query=category1=value1

Retrieving Datapoints

There are 2 methods for retrieving datapoints from IRONdb. A GET and a POST.

GET

For retrieving an individual metric name, use:

http://<host:port>/graphite/<account_id>/<optional_query_prefix>/series?start=<start_timestamp&end=<end_timestamp>&name=<metric_name>

where <start_timestamp> and <end_timestamp> are expressed in unix epoch seconds, and <metric_name> is the originally ingested leaf node returned from the /metrics/find query above. optional_query_prefix follows the same rules as described in the prior section.

POST

For fetching batches of time series data all at once, IRONdb provide a POST interface to send multiple names at the same time. To use this, POST a json document of Content-type: application/json to the following url:

http://<host:port>/graphite/<account_id>/<optional_query_prefix>/series_multi

The document format:

optional_query_prefix follows the same rules as the prior sections. If you provide an optional_query_prefix you would omit that portion of the metric name from the names in the JSON document. For example:

http://<host:port>/graphite/1/graphite./series_multi

The document format:

Native Whisper Read Support

IRONdb has the capability of reading Whisper database files directly, making historical Graphite data available to be queried. Writing new data to Whisper format is not supported.

To make an existing hierarchy of Whisper content available, the starting directory must be made available to all IRONdb nodes. Depending on operator preference, this may involve copying the directory structure and its files to each IRONdb node, or making a shared mountpoint available over a networked filesystem such as NFS, and mounting it at the same location on each IRONdb node. In all cases, the filesystem should be mounted read-only.

Multiple collections of Whisper data are also supported, such as from disparate Graphite installations. Each collection can be exposed to IRONdb individually, and may be segregated from one another using different IRONdb check UUIDs and/or account IDs. See above for details on how check UUIDs and account IDs are used.

To configure one or more Whisper directories, see .

Once Whisper directories are configured, they must be scanned and indexed in order for IRONdb to actually find and read them. The whisper_loader tool will read the IRONdb configuration and build an inventory. The inventory file records each metric name, along with the time range it covers, the aggregation function it uses, and the check UUID and account ID that it will be associated with.

NOTE: IRONdb only supports average and sum aggregation functions. Whisper databases using min, max, or last will be treated as if they were using average.

This inventory is then used as input on each IRONdb node to populate its local metric name index. The IRONdb service must be running on all nodes.

Full usage information may be obtained via:

Procedure:

1. Make the desired Whisper directory (or directories) visible on each IRONdb node. The directory structure must look the same to each node, whether via locally copied files or shared filesystem mount.

2. Select one IRONdb node on which to run the loader tool in "discovery mode", and run it:

3. Copy the inventory file to the remaining IRONdb nodes.

4. On each IRONdb node, including the one where discovery was done, run the tool in "submit mode", which will read the inventory file and create local metric name index entries:

As with ordinary metric ingestion, each Whisper metric will be "owned" by a subset of IRONdb nodes. As the inventory is processed in submit mode, any metric that is not owned by the local node will simply be skipped.

Canonical Metric Names

Canonical Metric Names in IRONdb are the combination of a metric name and tags. For a general overview, canonical metric names would follow the following BNF description:

To be canonical:

• A full canonical metric name must be less than 4095 characters in length.

• <tagsets> must have duplicate <tag> items removed, and then sorted lexicographically by category, and then value.

Submissions will be canonicalized before storage.

Examples:

• my_metric_name

• my_metric_name|ST[color:blue,env:prod]

• my_metric_name|MT{}|ST[env:prod]|MT{foo}|ST[color:blue]

The final example would canonicalize into the previous example since measurement-tags are not currently stored.

Metric Names

Metric names in Circonus may be an string of bytes other than a null character, or the stream-tag or measurement-tags identifiers (|ST[ or |MT{).

Stream Tags

Stream tags, as part of the metric name, are considered part of the unique identifier for the metric stream.

Measurement Tags

While part of the specification, Measurement Tags are experimental and should not be used at this time. They are not part of the unique identifier of a metric stream.

Tags

Tags in IRONdb are represented as category:value pairs that are separated by the colon (:) character.

Category strings may contain upper- and lowercase letters (A-Z and a-z), numerals (0-9), and the following characters:

Tag values allow all of the above characters plus colon (:) and equals (=).

Any tag characters that do not fall into this set can still be ingested if they are quoted, or base64 encoded and passed in a special wrapper format. More on this below.

Tags are ingested into IRONdb by placing the tags after the metric name with a tag separator character sequence: |ST and enclosed in square brackets []. Commas separate each tag.

Examples:

Tags (including category, colon, and value) are limited to 256 characters for each tag-pair. Tag-pairs exceeding that length will be truncated.

Base64 Encoding

Tags that contain characters outside of the acceptable set can be ingested, or searched for, by base64 encoding. To store a metric like:

The tilde ~, parens (), and greater/less <> are outside of the acceptable character set. The category and value can be encoded separately as base64 and enclosed in b"". For example:

It is always safe to encode all incoming tags in this way, the server will decide if the name is safely representable without encoding and store the metric name decoded if it can.

Quoting

For searching, but not ingestion, tags that contain characters outside of the acceptable set can also be quoted with double-quotes. Double-quoted strings accept all printable ASCII characters other than " and \, which must be escaped as \" and \\, respectively.

To search for a metric like:

The tilde ~, parens (), and greater/less <> are outside of the acceptable character set. The category and value can be quoted separately with "". For example:

See

Tag Queries

Tag queries can be used to find or perform deletion of metrics using a boolean tag search.

Query Syntax

A query follows this eBNF syntax:

A not clause may only contain a single expression, whereas and/or may each contain a list of expressions. Each expression may be a literal key:value to match, a regular expression, or a glob match syntax.

Regular expressions follow the PCRE2 syntax and are of the form:

Note that you can apply regular expressions independently to category or value or both:

Glob syntax supports the wildcard "*" and can be used as a completer:

The last will match every tag and pull everything for the account.

There are several special tags:

• __name

• __check_uuid

• __activity

Which do not explicitly appear in metric names but can be used to find metrics anyway. For example, you could query activity periods for all metrics within a given __check_uuid even if none of those metrics were submitted with tags.

The __activity tag uses a special syntax to select only metrics that have data (also know as activity) in a specific time range (start and end both inclusive). The value of the __activity tag in the search expression must take one of the following formats:

• <start seconds>-<end seconds> (hyphen format)

  • <start seconds>: Seconds since Unix epoch. May contain decimal precision. May be omitted to mean "the beginning of time". Note that a value of 1 shares this meaning.

  • <end seconds>: Seconds since Unix epoch. May contain decimal precision. May be omitted to mean "the end of time".

An example to find metrics named query_count with data between 1569869100 to 1569870000 would be:

and(__name:query_count,__activity:1569869100-1569870000)

An example to find metrics named query_count with data between two weeks ago and one week ago would be:

and(__name:query_count,__activity:-2w:-1w)

If your query segment uses an unsupported tag character you must enclose the segment in double-quotes, or use base64 notation:

and("foo$%^":"bar$%^") and(b"Zm9vJCVe":b"YmFyJCVe")

Note that the asterisk (*) for glob syntax is supported and stays a glob even if quoted or base64 encoded. To remove this behavior use the [exact] qualifier.

and([exact]"foo*":"bar") and(b[exact]"Zm9vKg==":b"YmFy")

If using regular expression patterns, the / / should not be encoded. The regex pattern however, may be base64 encoded if it uses a character that otherwise will violate parse rules. To perform a regex match in this form would look like b/KGZvb3xiYXIp/.

Query Examples

You have ingested the following metrics:

To find all of the metrics under app:myapp your query would be:

and(app:myapp)

To find all of the metrics in us-east regardless of sub-region you would do:

and(region:us-east-*) in glob syntax or:

and(region:/us-east-.*/) in regex syntax.

To find bar or quux you could either do:

or(__name:bar,__name:quux)

or:

or(and(region:us-east-2,app:,myapp),and(region:us-west-2,app:yourapp))

match impl Search Options

While primarily used for the __name tag, there are other options that can be invoked for specific search types on tag categories or values. These are known as "match impl" and have four options and can be activated with an optional [<type>] invocation at the beginning of the value.

• default - Literal matches with glob (*) support - as its name implies, this is the default form

• exact - Literal without glob support - useful for matching metrics with a * character

• re - The following string is a regex - this is synonymous with

These options are applied to whatever immediately follows them barring delimiting characters, so using them with unencoded values is straightforward:

example: and(__name:[graphite]prod.thing.nyc2.meter.worker.counter) example: and(__name:[graphite]prod.*.*.,mycategory:[re]foo.*bar[0-9]{5})

When using Base64 encoding, the same logic applies, therefore given a Base64 string as above b"Zm9vKg==", the correct application of the match impl would be b[<type>]"Zm9vKg==":

example: and(__name:b[exact]"Zm9vKg==")

Note that, in accordance with the above, if the match impl is placed before the b in a Base64 string, it will result in matching the Base64 string as though it were not encoded.

System Requirements

IRONdb requires one of the following operating systems:

• Ubuntu 22.04 LTS

Additionally, IRONdb requires the filesystem. This is available natively on Ubuntu.

Hardware requirements will necessarily vary depending upon system scale and cluster size. An appendix with general guidelines for

By default, IRONdb listens externally on TCP ports 2003 and 4242, TCP and UDP port 8112, and locally on TCP port 32322. These ports can be changed via configuration files. There are normally two processes, a parent and child. The parent process monitors the child, restarting it if it crashes. The child process provides the actual services, and is responsible for periodically "heartbeating" to the parent to show that it is making progress.

IRONdb is sensitive to CPU and IO limits. If either resource is limited, you may see a process being killed off when it does not heartbeat on time. These are known as "watchdog" events.

Service Management

The IRONdb service is called circonus-irondb.

To view service status: /bin/systemctl status circonus-irondb

To start the service: /bin/systemctl start circonus-irondb

To stop the service: /bin/systemctl stop circonus-irondb

To restart the service: /bin/systemctl restart circonus-irondb

To disable the service from running at system boot: /bin/systemctl disable circonus-irondb

To enable the service to run at system boot: /bin/systemctl enable circonus-irondb

Logs

Log files are located under /irondb/logs and include the following files:

• accesslog

• errorlog

• startuplog

The access logs are useful to verify activity going to the server in question. Error logs record, among other things, crashes and other errant behavior, and may contain debugging information important for support personnel. The startup log records various information about database initialization and other data that are typically of interest to developers and operators. Logs are automatically rotated and retained based on configuration attributes in /opt/circonus/etc/irondb.conf.

If the child process becomes unstable, verify that the host is not starved for resources (CPU, IO, memory). Hardware disk errors can also impact IRONdb's performance. Install the smartmontools package and run /usr/sbin/smartctl -a /dev/sdX, looking for errors and/or reallocated-sector counts.

Crash Handling

Application crashes are, by default, automatically reported to Apica, using technology. When the crash occurs, a tracer program quickly gathers a wealth of detailed information about the crashed process and sends a report to Apica, in lieu of obtaining a full core dump.

If you have disabled crash reporting in your environment, you can still enable traditional core dumping.

Debugging Mode

If instability continues, you may run IRONdb as a single process in the foreground, with additional debugging enabled.

First, ensure the service is disabled: /usr/bin/systemctl stop circonus-irondb

Then, run the following as root:

Running IRONdb in the foreground with debugging should make the error apparent, and Apica Support can help diagnose your problem. Core dumps are also useful in these situations (see above).

Replication

In a multi-node cluster, IRONdb nodes communicate with one another using port 8112. Metric data are replicated over TCP, while intra-cluster state (a.k.a. ) is exchanged over UDP. The replication factor is determine by the number of defined in the cluster's toplogy. When a node receives a new metric data point, it calculates which nodes should "own" this particular stream, and, if necessary, writes out the data to a local, per-node journal. This journal is then read behind and replayed to the destination node.

When a remote node is unavailable, its corresponding journal on the remaining active nodes continues to collect new metric data that is being ingested by the cluster. When that node comes back online, its peers begin feeding it their backlog of journal data, in addition to any new ingestion which is coming directly to the returned node.

Proxying

Clients requesting metric data from IRONdb need not know the specific location of a particular stream's data in order to fetch it. Instead, they may request it from any node, and if the data are not present on that node, the request is transparently proxied to a node that does have the data. Because nodes can fail and need to catch up with their peers, proxying favors remote nodes that are the most up to date. This is determined from the gossip data, which includes a latency metric, indicating the most recent replication message that this node has seen from each of its peers. The node performing the proxying decides which of the other nodes that own the given metric has the most recent data.

If gossip state is unavailable, such as due to a network partition, the node handling the request may return less recent data, if it proxies to a node that happens to be behind, or none at all, if the requested data is not available locally and all other owning nodes are unavailable.

Operations Dashboard

IRONdb comes with a built-in operational dashboard accessible via port 8112 in your browser, e.g., http://irondb-host:8112. This interface provides real-time information about the IRONdb cluster. There are a number of tabs in the UI, which display different aspects about the node's current status.

Overview Tab

The "Overview" tab displays a number of tiles representing the current ingestion throughput, available rollup dimensions, license information, and storage statistics.

Ingestion

Read (Get) and Write (Put) throughput, per second.

• "Batch" is an operation that reads or writes one or more metric streams.

• "Tuple" is an individual measurement.

Therefore, a write operation that PUTs data for 10 different streams in a single operation counts as 1 Batch and 10 Tuples.

License Info

Displays details of the node's .

Numeric Rollups

Displays throughput for both reads and writes per second for numeric rollup data.

• "Cache Size" is the number of open file handles for numeric rollup data. A given stream's data may be stored in multiple files, one for each configured rollup period in which that stream's data has been recorded.

• "Rollups" is the list of available rollup periods.

Histogram Rollups

Displays throughput for both reads and writes per second for histogram rollup data.

• "Rollups" is the list of available rollup periods.

Text Changesets

Displays throughput for both reads and writes per second for text data.

Storage

Disk space used and performance data per data type and rollup dimension.

Each icon under "Performance" displays a histogram of the associated operation (Get/Put/Proxy) latency since the server last started. "Get" operations are reads, "Put" are writes, and "Proxy" are operations that require fetching data from a different node than the one which received the request.

Latencies are plotted on the x-axis as seconds, with suffixes "m" for milliseconds, "μ" for microseconds, and "n" for nanoseconds. Counts of operations in each latency bucket are on the y-axis. The mean latency for the set is displayed as a vertical green line.

Hovering over the x-axis will display a shaded region representing quantile bands and the latency values that fall within them. The quantiles are divided into four bands: p(0)-p(25), p(25)-p(50), p(50)-p(75), and p(75)-p(100). To avoid losing detail, the maximum x-axis values are not displayed, but the highest latency value may be seen by hovering over the p(75)-p(100) quantile band.

Hovering over an individual latency bar will display three lines at the top right corner of the histogram. These represent the number of operations that had less than, equal to, or greater than the current latency, and what percentage of the total each count represents.

The Used, Total, and Compress Ratio figures represent how much disk space is occupied by each data type or rollup, the total filesystem space available on the node, and the ratio of the original size to the compressed size stored on disk. The compression ratio is determined from the underlying ZFS filesystem.

Replication Latency Tab

Two types of latency are displayed here: "replication latency" and "gossip age". Replication latency is the difference between the current time on each node and the timestamp of the most recently received metric in the from a remote node. Replication status information is exchanged between nodes using "gossip" messages, and the difference between the current time and the timestamp of the last gossip message received is the "gossip age". Gossip messages contain all replication state for a given node relative to all other nodes, so the state of the entire cluster can be seen from any node's UI.

Each node in the cluster is listed in a heading derived from the , and a gossip age in parentheses (see below). The node's latency summary is displayed at the right end of the heading line, and is an average of the replication latency between this node and all remote nodes. This is intended as a quick "health check" as to whether this node is significantly behind or not.

Clicking on the heading exposes a list of peer nodes, also from the topology configuration, and a replication latency indicator for each. Each peer's latency may be understood as "how far behind" the selected node is from that peer's current ingestion. In the example above, we can say that node "171" is 0 seconds behind from its peers "172" and "173".

All nodes should be running NTP or similar time synchronization. For example, if a remote node is shown as "(0.55 seconds old)", that means that a gossip message was received from that node 0.55 seconds ago, relative to the current node. Nodes that have persisently high gossip age, or peer latencies that do not drop to zero, may have clock skew.

Packet loss is another possible cause of replication latency. If a remote node's gossip latency varies widely, it could mean that gossip packets are being lost between hosts.

If the current node has never received a gossip message from a remote node since starting, that node will be displayed with a black bar, and the latency values will be reported as "unknown". This indicates that the remote node is either down or there is a network problem preventing communication with that node. Check that port 8112/udp is permitted between all cluster nodes.

Display Colors

Both gossip age and replication latency are also indicated using color.

The heading of the node being viewed will always be displayed in blue.

Gossip ages for remote nodes are colored in the heading as follows:

• Green means a difference of less than 2 seconds

• Yellow means a difference of more than 2 seconds and less than 8 seconds

• Red means a difference of more than 8 seconds

• Black means no gossip packets have been received from the remote host since this host last booted.

Latency summaries in the heading are colored as follows:

• If the node is behind W or more nodes by more than 4.5 minutes, then the summary is "latencies danger", and colored red.

• If the node is behind W-1 or more nodes by more than 30 seconds, then the summary is "latencies warning", and colored yellow.

• Otherwise, the average of all peer latencies is displayed, and colored green.

Replication latency indicators for individual remote nodes are colored as follows:

• Green for less than 30 seconds behind

• Yellow for more than 30 seconds but less than 270 seconds (4.5 minutes) behind

• Red for more than 270 seconds (4.5 minutes) behind

Topology Tab

Displays the layout of the topology ring, and the percentage of the key space for which each node is primarily responsible (coverage.) The ideal distribution is 1/N, but since the system uses consistent hashing to map metric names to nodes, the layout will be slightly imperfect.

An individual stream may be located by entering its UUID and Metric Name in the Locate Metrics tile, and then clicking the Locate button. Numbers indicating the primary and secondary owners of the metric (or more if more write copies are configured) will appear next to the corresponding node.

Extensions Tab

Displays a list of the loaded Lua extensions that provide many of the features of IRONdb.

Internals Tab

Shows internal application information, which is useful for troubleshooting performance problems. This information is divided into panels by the type of information contained within. These panels are described below.

Logs

The Logs panel of the Internals tab shows recent entries from the . When the Internals tab is first displayed, the Logs panel is expanded by default.

Job Queues

The Job Queues panel lists libmtev (aka "jobqs"), which are groups of one or more threads dedicated to a particular task, such as writing to the database, or performing data replication. These tasks may potentially block for "long" periods of time and so must be handled asynchronously to avoid stalling the application's event loop.

Job queues have names that indicate what they are used for, and concurrency attributes that control the number of threads to use in different scenarios.

At the top right of the Joq Queues panel is a toggle that controls whether to display jobqs currently in use ("Used") or all existing jobqs ("All"). The default is to show only in-use jobqs.

The toggle first appeared in version 0.15.1

Each row in the panel represents a job queue, with the following columns:

• Queue: the jobq name, preceded by a gauge of jobs that are either in-flight or backlogged (waiting to be enqueued.)

• Concurrency: the number of threads devoted to this jobq. This may be expressed as a pair of numbers separated by an arrow, indicating the current thread count (left) out of a potential maximum thread count (right). It may also be shown as a single number, meaning either that the queue is of a fixed size, or that a dynamic queue is at its maximum concurrency.

• Processed: a counter of jobs processed through this jobq since the application last booted.

• Waiting: information on jobs waiting in the queue. From left to right, three pieces of information are visible:

Sockets

The Sockets panel displays information on active sockets. These include both internal file descriptors for the , as well as network connections for REST API listeners and clients.

Each row in the panel corresponds to one socket, with the following columns:

• FD: the file descriptor number that corresponds to the socket, and the value of the . The mask determines what type of activity will trigger the callback associated with the socket. Typical values are (R)ead, (W)rite, and (E)xception. If multiple values are set, they are separated by a vertical bar.

• Opset: the "style" of socket determines the set of operations that may be performed on the socket. Typical values are "POSIX", which means the standard set of POSIX-compliant calls like accept() and close() are available, and "SSL", which adds SSL/TLS operations. The vast majority of sockets in IRONdb will be of the POSIX type.

• Callback: the libmtev function that will be called when the socket is triggered by activity matching the socket's mask. For example, if a socket has the Read mask, and there is data on the socket to read, the associated callback function will be invoked to handle reading that data.

Network sockets:

Timers

The Timers panel displays information on . IRONdb does not make extensive use of timed events so this panel is often empty.

Each row in the panel lists a timed event, with the following columns:

• Callback: the libmtev function that will be called when the appointed time arrives.

• When: the time that the callback should fire.

Stats

The Stats panel displays all statistics application statistics that have been registered into the system. These are collected and maintained by the library. Statistics accumulate over the lifetime of the process, and are reset when the process restarts.

At the top of the panel is a Filter field where you can enter a substring or regex pattern to match statistics. Only those statistics matching the pattern will be displayed. This is a useful way to narrow down the list of statistics, which can be quite long.

The filter field first appeared in version 0.15.4.

Stats are namespaced to indicate what they represent:

• mtev: internal libmtev statistics

  • eventer: stats related to the operation of the event system

    • callbacks: each named callback registered in the system gets a "latency" statistic that is a cumulative histogram of all latency values for this callback since boot.

    • jobq: each jobq registered in the system gets a set of stats that convey various information about that jobq. The same information appears in the Job Queues panel, without the

The IRONdb-relay, like the carbon-relay or the carbon-c-relay is a metrics data router that takes carbon TEXT format metrics and routes them to the appropriate IRONdb storage node.

Since IRONdb uses SHA256 hashing to route metrics to IRONdb nodes, it is incompatible with routing options that exist in carbon-c-relay and carbon-relay. In addition, it provides advanced aggregation and filtering functions for Graphite metrics.

The IRONdb-relay is also capable of accepting Prometheus Snappy-compressed protocol buffers, decoding them, and routing the data to the appropriate IRONdb-relay storage node. It can accept this data either via a dedicated API endpoint or by pulling data from Kafka using the libmtev Kafka module.

Changelog

Features

• Ingests TEXT carbon format metrics on a configurable port

  foo.bar.baz 1234.56 1507724786

• Ingests Prometheus Snappy-compressed protocol buffers via an API endpoint or via Kafka.

• Routes to primary owner of the metric name and then subsequent nodes if the primary is down.

• of incoming metrics based on regular expressions with support for SUM, AVG, MIN, MAX, p0, p25, p50, p95, p99, p100 for carbon format metrics.

• of metrics based on regular expressions

• Durable delivery of metrics using write ahead logs

Installation

IRONdb-relay requires one of the following operating systems:

• Ubuntu 22.04 LTS

The following network protocols and ports are utilized. These are defaults and may be changed via configuration files.

• 2003/tcp (Carbon plaintext submission)

• 8112/tcp (admin UI, HTTP REST API)

• If the IRONdb cluster uses , 8443/tcp will be used for ingestion to IRONdb.

System Tuning

You should follow the same system tuning as outline in the .

Configure Software Sources

Use the same software source as the .

Install Package

• /usr/bin/apt-get install circonus-platform-irondb-relay

Run Installer

Prepare site-specific information for setup. These values may be set via shell environment variables, or as arguments to the setup script. The environment variables are listed below.

• IRONDB_CHECK_UUID

  (required) Check ID for Graphite metric ingestion, which must be the same on all cluster nodes. You may use the uuidgen command that comes with your OS, or generate a UUID with an external tool or website.

• IRONDB_CHECK_NAME

  (required) The string that will identify Graphite-compatible metrics stored in the check identified by IRONDB_CHECK_UUID. For example, if you submit a metric named "my.metric.1", and the check is named "test", the resulting metric name in IRONdb will be "graphite.test.my.metric.1".

Run the setup script. All required options must be present, either as environment variables or via command-line arguments. A mix of environment variables and arguments is permitted, but environment variables take precedence over command-line arguments. Use the -h option to view a usage summary:

If your IRONdb cluster , then specify the node list as https://<FQDN>:8443 URLs, and, if necessary, place the CA certificate that corresponds to the cluster's client-facing listener as /opt/circonus/etc/ssl/irondb-ca.crt. The CA cert is necessary if your certificates are issued by an internal CA, as opposed to a public CA that is trusted by the operating system.

The setup script will configure your IRONdb-relay instance and start the service. See the section for details.

If you selected the TLS option irondb-relay listeners, the service will not be started automatically, and you will need to install a private key and certificate before starting the service.

Configuration

IRONdb-relay is implemented using , a framework for building high-performance C applications. You may wish to review the libmtev for an overview of how libmtev applications are configured generally.

This document deals with options that are specific to IRONdb-relay, but links to relevant libmtev documentation where appropriate.

Default values are those that are present in the default configuration produced during initial installation.

irondb-relay.conf

This is the primary configuration file that IRONdb-relay reads at start. It includes additional configuration files which are discussed later. It is located at /opt/circonus/etc/irondb-relay.conf

irondb-relay

IRONdb-relay's libmtev application name. This is a required node and must not be changed.

irondb-relay lockfile

Path to a file that prevents multiple instances of the application from running concurrently. You should not need to change this.

Default: /irondb-relay/logs/irondb-relay.lock

eventer