IRONdb Relay
Last updated
Was this helpful?
Last updated
Was this helpful?
The IRONdb-relay, like the carbon-relay or the carbon-c-relay is a metrics data router that takes carbon TEXT format metrics and routes them to the appropriate IRONdb storage node.
Since IRONdb uses SHA256 hashing to route metrics to IRONdb nodes, it is incompatible with routing options that exist in carbon-c-relay and carbon-relay. In addition, it provides advanced aggregation and filtering functions.
Ingests TEXT carbon format metrics on a configurable port
foo.bar.baz 1234.56 1507724786
Routes to primary owner of the metric name and then subsequent nodes if the primary is down.
of incoming metrics based on regular expressions with support for SUM, AVG, MIN, MAX, p0, p25, p50, p95, p99, p100
of metrics based on regular expressions
Durable delivery of metrics using write head logs
IRONdb-relay requires one of the following operating systems:
Ubuntu 22.04 LTS
The following network protocols and ports are utilized. These are defaults and may be changed via configuration files.
2003/tcp (Carbon plaintext submission)
8112/tcp (admin UI, HTTP REST API)
If the IRONdb cluster uses , 8443/tcp will be used for ingestion to IRONdb.
/usr/bin/apt-get install circonus-platform-irondb-relay
Prepare site-specific information for setup. These values may be set via shell environment variables, or as arguments to the setup script. The environment variables are listed below.
IRONDB_CHECK_UUID
IRONDB_CHECK_NAME
(required) The string that will identify Graphite-compatible metrics stored in the check identified by IRONDB_CHECK_UUID. For example, if you submit a metric named "my.metric.1", and the check is named "test", the resulting metric name in IRONdb will be "graphite.test.my.metric.1".
IRONDB_BOOTSTRAP
(required) The comma separated list of IRONdb nodes (ipaddress:port or https://FQDN:port URL) to use to discover the topology of the IRONdb cluster. It's a good practice to list all IRONdb nodes in this list to adjust to down nodes.
IRONDB_RELAY_TLS
IRONDB_CRASH_REPORTING
IRONDB_RELAY_DURABLE
(optional) Control enablement of durable delivery. Default is "false". If set to "true", will cause IRONdb-relay to use the disk to persist all incoming metrics to the file system before sending them on to IRONdb nodes.
Run the setup script. All required options must be present, either as environment variables or via command-line arguments. A mix of environment variables and arguments is permitted, but environment variables take precedence over command-line arguments. Use the -h option to view a usage summary:
If you selected the TLS option irondb-relay listeners, the service will not be started automatically, and you will need to install a private key and certificate before starting the service.
This document deals with options that are specific to IRONdb-relay, but links to relevant libmtev documentation where appropriate.
Default values are those that are present in the default configuration produced during initial installation.
This is the primary configuration file that IRONdb-relay reads at start. It includes additional configuration files which are discussed later. It is located at /opt/circonus/etc/irondb-relay.conf
IRONdb-relay's libmtev application name. This is a required node and must not be changed.
Path to a file that prevents multiple instances of the application from running concurrently. You should not need to change this.
Default: /irondb-relay/logs/irondb-relay.lock
By default, the following log files are written and automatically rotated, with the current file having the base name and rotated files having an epoch-timestamp suffix denoting when they were created:
/irondb-relay/logs/errorlog: Output from the daemon process, including not just errors but also operational warnings and other information that may be useful to Apica Support.
Rotated: 24 hours
Retained: 1 week
There are 2 modules provided with IRONdb-relay:
filter
Will allow you to setup whitelist/blacklist filtering for metrics
Enable the module under the <modules> section of your config by adding the line:
<generic image="filter" name="filter_hook"></generic>
Create your filter config
Add a <filter> block to your irondb-relay.conf file. A <filter> can have exactly one <ruleset> block. A <ruleset> block can have any number of <rule> blocks. A <rule> block consists of a <match_regex> or <match_all> directive and a <result>. <rule> blocks are processed in order and processing stops at the first matching <rule>.
Depending on whether you want a whitelist or a blacklist you would either configure your filter to whitelist a set of regexes and then have a <match_all> rule to deny everything else, or you would configure your filter to have a rule to match metrics you want to blacklist then have a final <match_all> rule to allow the remainder.
An example of a blacklist would resemble:
The above would blacklist everything that starts relay_test.agent.2 and allow everything else.
For best performance, it is wise to organize your <rule> blocks in descending order based on the expected frequency of matching. You want the <rule>s that match more often to be at the beginning of the list and the <rule>s that match infrequently to be lower down in the list.
aggregation_hook
Will allow you to perform aggregation on incoming metrics and produce new metrics as the result.
Enable the module under the <modules> section of your irondb-relay.conf by adding the line:
<generic image="aggregation_hook" name="aggregation_hook"></generic>
Create your aggregation config
Add an <aggregation> block to your irondb-relay.conf file. An <aggregation> can have exactly one <matchers> block which itself can contain any number of <matcher> blocks. A <matcher> block consists of the following:
<match_regex> - the regular expression (including captures) you want to match incoming metrics
<flush_seconds> - how long to aggregate matching records for
<flush_name_template> - the template to use for the outgoing name of the aggregated metric result
<flush_functions> - a comma separate list of functions you want applied to the matching metric values
<flush_original> - whether or not you want to let the original incoming metric to also be sent to IRONdb
<jitter_ms> - to prevent collisions among multiple relays which might be aggregating the same metrics, set this to a unique value per irondb-relay instance
<idle_cycles> - how many multiples of <flush_seconds> should the relay wait before giving up on any new incoming metrics that would fall into this aggregation window
For <flush_name_template> you can use capture references (\1) and a special sequence ${FF} to create the outgoing metric name.
An example:
The above first <matcher> matches incoming metrics that start relay_test.agent., followed by any number of digits, followed by .metrics. and finally capturing the trailing sequence of any number of digits. An example metric that would match: relay_test.agent.5.metrics.27. All incoming rows that fit this regex will have their values averaged (<flush_functions>avg</flush_functions>) for 10 seconds. The original source rows will not be sent to IRONdb and after 10 seconds a new row will be produced that looks like: agg.all_agents_metrics.27_avg and it's value will be the average of metrics.27 from all agents that this relay saw in that 10 second window.
The 2nd <matcher> performs the same match but uses sum instead of avg and uses a different <flush_name_template>.
The supported <flush_functions> are: sum,avg,min,max,p0,p25,p50,p95,p99,p100,histogram
sum is the sum of values of the matching rows.
avg is the mean
min is the smallest value
max is the largest
p0 is a synonym for min
p100 is a synonym for max
p25 is the 25th percentile value
p50 is the 50th percentile value
p95 is the 95th percentile value
p99 is the 99th percentile value
histogram is the complete distribution of all values
With histogram IRONdb will be able to store the histogram data but there currently is no facility in graphite-web to render this data.
A note on flushing results to IRONdb
This config has a single attribute: durable="true|false". If set to "true" it will use the <journal> settings below to journal every row destined for IRONdb nodes. If set to "false", it will bypass the journaling and directly send to IRONdb. If set to "false", the relay will do its best to make sure data arrives at one of the IRONdb nodes if the primary doesn't respond or is down but there is no guarantee of delivery.
Each listener below is configured within a <listener> node. Additional listeners may be configured if desired, or the specific address and/or port may be modified to suit your environment.
TLS Configuration
This section will be present when TLS operation has been activated via the setup script. These settings apply to any and all listeners that have the ssl attribute set to "on".
Place the following files in the /opt/circonus/etc/ssl directory:
relay.key - An RSA private key.
relay.crt - A certificate issued for this relay's listeners. Its commonName (CN) should be the node's FQDN, or whatever name clients will be using to connect to this node.
relay-ca.crt - The Certificate Authority's public certificate, sometimes referred to as an intermediate or chain cert, that issued relay.crt.
These files must be readable by the unprivileged user that irondb-relay runs as, typically nobody.
Main listener
The main listener serves multiple functions:
JSON-formatted node statistics (http://thisnode:thisport/stats.json)
Main listener address
The IP address on which to listen, or the special * to listen on any local IP address.
Default: *
Main listener port
The port number to listen on. For the main listener this will utilize both TCP and UDP.
Default: 8112
Main listener backlog
The size of the queue of pending connections. This is used as an argument to the standard listen(2) system call. If a new connection arrives when this queue is full, the client may receive an error such as ECONNREFUSED.
Default: 100
Main listener type
The type of libmtev listener this is. The main listener is configured to be only a REST API listener. This value should not be changed.
Default: http_rest_api
Main listener ssl
If set to "on", SSL/TLS will be enabled for this listener.
Default: off
Graphite listener
Graphite listener address
The IP address on which to listen, or the special * to listen on any local IP address.
Default: *
Graphite listener port
The TCP port number to listen on.
Default: 2003
Graphite listener type
The type of listener. IRONdb implements a Graphite-compatible handler in libmtev, using the custom type "graphite".
Default: graphite
Graphite listener ssl
If set to "on", SSL/TLS will be enabled for this listener.
Default: off
Graphite listener config
check_uuid is the identifier for all metrics ingested via this listener.
account_id is also part of namespacing, for disambiguation.
CLI listener
CLI listener address
The IP address on which to listen, or the special * to listen on any local IP address.
Default: 127.0.0.1
CLI listener port
The TCP port number to listen on.
Default: 32322
CLI listener type
The CLI listener uses the built-in libmtev type "mtev_console" to allow access to the telnet console.
Default: mtev_console
Journals are write-ahead logs for replicating metric data to IRONdb nodes. Each IRONdb-relay has one journal for each of the IRONdb nodes.
journal concurrency
Establishes this number of concurrent threads for writing to each peer journal, improving ingestion throughput.
Default: 4
A concurrency of 4 is enough to provide up to 700K measurements/second throughput, and is not likely to require adjustment except in the most extreme cases.
journal replicate_concurrency
Establishes this number of concurrent threads for writing from the journals into the IRONdb cluster, improving throughput.
Default: 1
journal max_bundled_messages
Outbound journal messages will be sent in batches of up to this number, improving replication speed.
Default: 25000
journal pre_commit_size
An in-memory buffer of this number of bytes will be used to hold new journal writes, which will be flushed to the journal when full. This can improve ingestion throughput, at the risk of losing up to this amount of data if the system should fail before commit. To disable the pre-commit buffer, set this attribute to 0.
Default: 131072 (128 KB)
watchdog
IRONdb-relay comes with a built-in operational dashboard accessible via port 8112 (default) in your browser, e.g., http://irondb-relay-host:8112. This interface provides real-time information about the IRONdb-relay. There are a number of tabs in the UI, which display different aspects about the node's current status.
The node's version info is displayed at top right.
The "Overview" tab displays top level statistics about the relay.
Inflow
Socket accepts - how many connections have been made to this relay since startup
Received - how many individual rows have been sent into this relay
Parsed - the number of rows that we successfully parsed and validated
Parse errors - the number of parse failures
Outflow
Rows sent - the number of rows sent to IRONdb nodes
Batches sent - rows are sent in batches, this is the count
Batches OK - successful batch count
Batch timeouts - the count of batches that timed out while sending to IRONdb nodes
Each IRONdb node will be listed along with the number of journal reads and writes and how far behind this relay is in sending to each IRONdb node. Ideally we should have Seconds behind under 10 seconds.
If you have the filter module enabled, lists each filter in your current <ruleset> and how many rows it has processed.
If you have the aggregation_hook module enabled, lists each aggregation and how many rows it has seen, matched, skipped, and generated.
Shows internal application information, such as recent error logging, job queues, open sockets, and timers. This data is used by Apica Support when troubleshooting issues.
You should follow the same system tuning as outline in the .
Use the same software source as the .
(required) Check ID for Graphite metric ingestion, which must be the same on all cluster nodes. You may use the uuidgen command that comes with your OS, or generate a UUID with an external tool or website.
(optional) Configures listeners to require TLS where applicable. Default is "off". If set to "on", both the Carbon submission port and the admin UI port will expect TLS connections from clients. An SSL certificate will be required before the service can be started. See below for details.
(optional) Control enablement of automated crash reporting. Default is "on". IRONdb utilizes sophisticated crash tracing technology to help diagnose errors. Enabling crash reporting requires that the system be able to connect out to the Apica reporting endpoint: . If your site's network policy forbids this type of outbound connectivity, set the value to "off".
If your IRONdb cluster , then specify the node list as https://<FQDN>:8443 URLs, and, if necessary, place the CA certificate that corresponds to the cluster's client-facing listener as /opt/circonus/etc/ssl/irondb-ca.crt. The CA cert is necessary if your certificates are issued by an internal CA, as opposed to a public CA that is trusted by the operating system.
The setup script will configure your IRONdb-relay instance and start the service. See the section for details.
IRONdb-relay is implemented using , a framework for building high-performance C applications. You may wish to review the libmtev for an overview of how libmtev applications are configured generally.
Libmtev eventer system configuration. See the .
Libmtev logging configuration. See the .
Libmtev module configuration. See the
The very first row that matches and creates an aggregation "window" will start the flush timer. <flush_seconds> later the result will be sent to IRONdb. It is possible after this initial flush that some late data arrives that would normally fit into that same aggregation window. This is where <idle_cycles> comes into play. The relay will retain the aggregation window until no more matching rows are seen for <idle_cycles> cycles. If matching rows are seen the aggregation window is updated with the new values and the record is re-sent to IRONdb. To control the behavior of conflicts within the database when this happens please see the section of the IRONdb configuration.
Libmtev network listener configuration. See the .
See for specific details on each option.
The Graphite listener operates a Carbon-compatible submission pathway using the .
Multiple Graphite listeners may be configured on unique ports and associated with different check UUIDs. See the section on for details. The graphite listener config here should be kept in sync with the for the IRONdb nodes themselves.
These configuration items control which check UUID, name, and account ID are associated with this listener. The first Graphite listener is configured during .
check_name is a meaningful name that is used in .
The CLI listener provides a local for interacting with libmtev subsystems, including modifying configuration. As there is no authentication mechanism available for this listener, it is recommended that it only be operated on the localhost interface.
The watchdog configuration specifies a handler, known as a "glider", that is to be invoked when a child process crashes or hangs. See the .
If is turned on, the glider is what invokes the tracing, producing one or more files in the tracedir. Otherwise, it just reports the error and exits.
Aggregation gated - the number of rows that were not send on to IRONdb because of or
If <send durable="true" /> is set in the , this tab will contain information about replication lag.