Apica Docs
  • Welcome to Apica Docs!
  • PRODUCT OVERVIEW
    • Ascent Overview
    • Ascent User Interface
  • TECHNOLOGIES
    • Ascent with Kubernetes
      • Kubernetes is a Game-Changer
      • Ascent: Built on Kubernetes
    • Ascent with OpenTelemetry
      • Why Implement OpenTelemetry?
      • Common Use Cases for OpenTelemetry
      • How to Get Started with OpenTelemetry
      • Best Practices for OpenTelemetry Implementations
  • RELEASE NOTES
    • Release Notes
      • Ascent 2.10.3
      • Ascent 2.10.2
      • Ascent 2.9.0
      • Ascent 2.8.1
      • Ascent 2.8.0
      • Ascent 2.7.0
      • Ascent 2.6.0
      • Ascent 2.5.0
      • Ascent 2.4.0
      • Ascent 2.3.0
      • Ascent 2.2.0
      • Ascent 2.1.0
        • Data Fabric
          • Releases-old
        • Synthetic Monitoring
        • Advanced Scripting Engine
        • IRONdb
      • Synthetic Monitoring
  • GETTING STARTED
    • Getting Started with Ascent
      • Getting Started with Metrics
      • Getting Started with Logs
        • OpenTelemetry
    • Ascent Deployment Overview
    • Quickstart with Docker-Compose
    • On-Premise PaaS deployment
      • On-Premise PaaS Deployment Architecture
      • Deploying Apica Ascent PaaS on Kubernetes
      • Deploying Apica Ascent PaaS on MicroK8s
      • Deploying Apica Ascent PaaS on AWS
      • Deploying Apica Ascent EKS on AWS using CloudFormation
      • Deploying Ascent on AWS EKS with Aurora PostgreSQL and ElastiCache Redis using Cloud Formation
        • Deploying Apica Ascent on AWS EKS with Aurora PostgreSQL and ElastiCache Redis using CloudFormation
        • Apica Ascent on AWS EKS (Private Endpoint) with Aurora PostgreSQL and ElastiCache Redis on prod VPC
      • Deploying Apica Ascent EKS on AWS using custom AMI
      • Deploying Apica Ascent EKS with AWS ALB
      • Deploying Apica Ascent PaaS in Azure Kubernetes Service
        • Azure Blob Storage Lifecycle Management
      • Deploying Apica Ascent with OpenShift
    • Boomi RTO Quick Start Guide
      • RTO Dashboarding
      • Alerting on RTO Metrics
      • Alerting on RTO Logs
    • Dashboards & Visualizations
  • DATA SOURCES
    • Data Source Overview
    • API
      • JSON Data source
      • RSS
    • AWS
      • Amazon Athena
      • Amazon CloudWatch ( YAML )
      • Amazon Elasticsearch Service
      • Amazon Redshift
      • MySQL Server (Amazon RDS)
    • NoSQL Data Sources
      • MongoDB
    • OLAP
      • Data Bricks
      • Druid
      • Snowflake
    • SQL Data Sources
      • PostgreSQL
      • Microsoft SQL Server
      • MySQL Server
    • Time Series Databases
      • Prometheus Compatible
      • Elasticsearch
      • InfluxDB
    • Ascent Synthetics
      • Checks
    • Ascent Logs
      • Logs
  • INTEGRATIONS
    • Integrations Overview
      • Generating a secure ingest token
      • Data Ingest Ports
    • List of Integrations
      • Apache Beam
        • Export Metrics to Prometheus
          • Pull Mechanism via Push-Gateway
        • Export Events to Apica Ascent
      • Apica ASM
      • Apica Ascent Observability Data Collector Agent
      • AWS
        • AWS CloudWatch
        • AWS ECS
          • Forwarding AWS ECS logs to Apica Ascent using AWS FireLens
          • ECS prometheus metrics to Apica Ascent
        • AWS S3
      • Azure
        • Azure Databricks
        • Azure Eventhub
        • Azure Event Hubs
      • Docker Compose
      • Docker Swarm logging
      • Docker Syslog log driver
      • F5 Big-Ip System
      • Filebeat
      • Fluent Bit
        • Forwarding Amazon-Linux logs to Apica Ascent using Fluent Bit
        • Fluent Bit installation on Ubuntu
        • Enabling IoT(MQTT) Input (PAAS)
        • IIS Logs on Windows
      • Fluentd
      • FortiNet Firewalls
      • GCP PubSub
      • GCP Cloud Logging
      • IBM QRadar
      • ilert
      • Incident Management
        • Webhooks
      • Jaeger
      • Kafka
      • Kinesis
      • Kubernetes
      • Logstash
      • MQTT
      • Network Packets
      • OpenTelemetry
      • Object store (S3 Compatible)
      • Oracle OCI Infrastructure Audit/Logs
      • Oracle Data Integrator (ODI)
      • OSSEC Variants (OSSEC/WAZUH/ATOMIC)
        • Apica Ascent-OSSEC Agent for Windows
      • Palo Alto Firewall
      • Prometheus
        • Spring Boot
        • Prometheus on Windows
        • Prometheus Remote Write
        • MongoDB Exporter
        • JMX Exporter
      • Rsyslogd
      • Syslog
      • Syslog-ng
      • Splunk Universal Forwarder
      • Splunk Heavy Forwarder
      • SNMP
      • Splunk Forwarding Proxy
      • Vault
        • Audit Vault Logs - AWS
        • Audit Vault Logs - OCI
        • Audit Vault Metrics
    • Apica API DOCS
  • DATA MANAGEMENT
    • Data Management Overview
    • Data Explorer Overview
      • Query Builder
      • Widget
      • Alerts
      • JSON Import
      • Creating Json Schema
        • Visualization
          • Line chart
          • Bar chart
          • Area chart
          • Scatter chart
          • Status chart
          • Counter chart
          • Stat chart
          • Size chart
          • Dense Status chart
          • Honeycomb chart
          • Gauge chart
          • Pie chart
          • Disk chart
          • Table chart
          • Date time chart
      • Time-Series AI/ML
        • Anomaly Detection
        • Averaging
        • Standard Deviation(STD)
      • Data Explorer Dashboard
        • Create a Dashboard
        • Editing Dashboard
          • Dashboard level filters
    • Timestamp handling
      • Timestamp bookmark
    • Large log/events/metrics/traces
  • OBSERVE
    • Monitoring Overview
      • Connecting Prometheus
      • Connecting Amazon Managed Service for Prometheus
      • Windows Redis Monitoring
      • Writing queries
        • Query Snippets
      • Query API
      • Use Apica API to ingest JSON data
    • Distributed Tracing
      • Traces
      • Spans
      • Native support for OTEL Traces
      • Windows .NET Application Tracing
      • Linux+Java Application Tracing
    • Log Management
      • Terminology
      • Explore Logs
      • Topology
      • Apica Ascent Search Cheat Sheet
      • Share Search Results
      • Severity Metrics
      • Log2Metrics
      • Native support for OTEL Logs
      • Reports
        • Accessing Reports results via API
      • Role-Based Access Control (RBAC)
      • Configuring RBAC
    • AI and LLM Observability
      • AI Agent Deployment
      • Ascent AI Agent Monitoring
      • Ascent Quick Start Guide
    • Synthetic Check Monitoring
      • Map View
      • List View
      • Alerting for Check Results
  • Flow
    • Overview
    • Pipeline Management
      • Configuring Pipelines
      • Visualize Pipelines
      • Pipeline Overview Dashboard
      • Forwarding Data
    • OpenTelemetry Ingest
      • OpenTelemetry Logs / Traces
      • OpenTelemetry Metrics
        • Transforming Metrics through Code Rules
    • Vault
      • Certificates
      • Variables
      • Lookups
    • Rules
      • FILTER
      • EXTRACT
      • SIEM and TAG
      • REWRITE
      • CODE
      • FORWARD
        • Rename Attributes
      • STREAM
    • Functions
      • ascent.encode
      • ascent.decode
      • ascent.persist
      • Ascent.variables
      • ascent.crypto
      • Ascent.mask
      • Ascent.net
      • Ascent.text
      • Ascent.time
      • Ascent.lookups
    • List of Forwarders
    • OpenTelemetry Forwarding
      • Metrics
      • Traces
      • Logs
    • Splunk Forwarding
      • Apica UF Proxy App Extension
        • Standalone Instance
        • List of Indexer Instances
        • Indexer Discovery
      • Metric Indexes
      • Non Metric Indexes
      • Syslog Forwarding
    • Real-Time Stream Forwarding
      • AWS Kinesis
      • Azure Eventhub
      • Google Pub/Sub
    • Security Monitor Forwarding
      • Arc Sight
      • RSA New Witness
    • Forwarding to Monitoring Tools
      • Datadog Forwarding
      • New Relic Forwarding
      • Dynatrace Forwarding
      • Elasticsearch Forwarding
      • Coralogix Forwarding
      • Azure Log Analytics Forwarding
    • Object Store Forwarding
      • S3 Compatible
      • Azure Blob Storage
    • Forwarding to Data Warehouse
      • GCP Bigquery
  • Customized Forwarders
    • JS Code Forwarding
  • LAKE
    • Powered by Instastore™
  • FLEET MANAGEMENT
    • Overview
    • Agents
    • Configurations
    • Packages
    • Fleet Repository Management
    • Advanced Search
    • List of Agents
      • Datadog Agent
      • Fluent-bit Agent
      • Grafana Alloy
      • OpenTelemetry Collector
      • OpenTelemetry Kubernetes
      • Prometheus Agent
  • COMMAND LINE INTERFACE
    • apicactl Documentation
  • AUTONOMOUS INSIGHTS
    • Time Series AI-ML
      • Anomaly Detection
      • Averaging
      • Standard Deviation(STD)
      • Forecasting
      • AI-ML on PromQL Query Data Set
      • Statistical Data Description
    • Pattern-Signature (PS)
      • Log PS Explained
        • Unstructured Logs
        • Semi-structured JSON
        • Reduce Logs Based on PS
        • Log PS Use Cases
          • Log Outlier Isolation
          • Log Trending Analysis
          • Simple Log Compare
      • Config PS
        • Config JSON PS
    • ALIVE Log Visualization
      • ALIVE Pattern Signature Summary
      • ALIVE Log Compare
    • Log Explained using Generative AI
      • Configuring Generative AI Access
      • GenAI Example Using Log Explain
    • Alerts
    • Alerts (Simple/Anomaly)
    • Alerts On Logs
    • Rule Packs
    • AI-powered Search
  • PLATFORM DOCS
    • Synthetic Monitoring Overview
      • Getting Started with ASM
        • Achieving 3 Clicks to Issue Resolution via ASM
        • FAQ - Frequently Asked Questions
        • Creating A New Check
          • Creating a New Real Browser Check
      • Explore the Platform
        • API Details
        • Check Types
          • Android Check
          • Command Check
          • Compound Check
          • Browser Check
          • Desktop Application Check
          • AWS Lambda Check
          • DNS Resolver Check
          • DNS Security Check
          • Domain Availability Check
          • Domain Delegation Check
          • Domain Expiration Date Check
          • Hostname Integrity Check
          • iPad Check
          • iPhone Check
          • Ping Check
          • Port Check
          • Postman Check
          • Response Time Check
          • SSL Certificate Expiration Check
          • Scripted Check
        • Dashboards
        • Integrations
          • DynaTrace Integration
          • Google Analytics Integration
          • Akamai Integration
          • Centrify Integration
          • AppDynamics Integration
          • PagerDuty Integration
          • ServiceNow Integration
          • Splunk Integration
        • Metrics
          • Analyze Site
          • Result Values
          • Trends
          • Analyze Metrics
        • Monitoring
          • Integrating ASM Metrics into Grafana Using Apica Panels
            • Understanding the ASM Imported Dashboards
            • Using the Apica Panels Dashboards
          • Understanding ASM Check Host Locations
        • Navigation
          • Manage Menu
        • Reports
        • Use Cases
      • Configurations
        • Configuring Checks
          • Understanding Check Results
            • Understanding ZebraTester Check Results
            • Understanding Browser Check Results
            • Understanding Check Details
          • Editing Checks
            • Editing Browser Checks
            • Editing ZebraTester Checks
          • Using Regular Expressions Within the ASM Platform
          • Understanding the Edit Scenario Page
          • Comparing Selenium IDE Scripts to ASM Scenarios
          • Configuring Apica DNS Check Types
          • Implementing Tags Effectively Within ASM
          • Storing and Retrieving Information Using the ASM Dictionary
        • Configuring Users
          • Configuring SSO Within ASM
        • Configuring Alerts
          • Configuring Webhook Alerts
      • How-To Articles
        • ASM Monitoring Best Practices
        • API Monitoring Guide
        • IT Monitoring Guide
        • Monitor Mission-Critical Applications through the Eyes of Your Users
        • How To Mask Sensitive Data in ASM
        • How to Mask Sensitive Data When Using Postman Checks
        • How to Handle URL Errors in a Check
        • How To Set Up SSO Using Azure AD
        • How to Set Up SSO Using Centrify
        • ASM Scenarios How-To
          • How To Pace a Selenium Script
          • How to Utilize XPath Within a Selenium Script
          • How to Mask Sensitive Information Within an ASM Scenario
          • Handling Elements Which Do Not Appear Consistently
          • How to Handle HTML Windows in ASM Scenarios
    • ZebraTester Scripting
      • ZebraTester Overview
      • Install ZebraTester
        • Download ZebraTester
          • Core ZebraTester V7.5-A Documentation
          • Core ZebraTester V7.0-B Documentation
          • Core ZebraTester V7.0-A Documentation
          • Core ZebraTester V5.5-Z Documentation
          • Core ZebraTester V5.5-F Documentation
        • Download the ZebraTester Recorder Extension
        • Windows Installation
          • ZebraTester on Windows
          • Generate Private CA Root Certificate
          • Windows System Tuning
          • Install a new ZT version on Windows Server
          • Install/Uninstall ZT Windows Installer Silently
        • macOS Installation
          • macOS Preinstallation Instructions
          • Generate Private CA Root Cert (macOS X)
          • System Tuning (macOS)
          • Import a CA Root Certificate to an iOS device
          • Memory Configuration Guidelines for ZebraTester Agents
      • ZebraTester User Guide
        • Menu and Navigation Overview
        • 1. Get a Load Test Session
          • Recording Web Surfing Sessions with ZebraTester
            • Further Hints for Recording Web Surfing Sessions
            • Recording Extension
              • Record Web Session
              • Cookies and Cache
              • Proxy
              • Page Breaks
              • Recording Extension Introduction
              • Troubleshooting
            • Add URL to ZebraTester
            • Page Scanner
          • Next Steps after Recording a Web Surfing Session
        • 2. Scripting the Load Test Session
          • 1. Assertions - HTTP Response Verificaton
          • 2. Correlation - Dynamic Session Parameters
            • 2b. Configuring Variable Rules
            • 2a. Var Finder
          • 3. Parameterization: Input Fields, ADR and Input Files
            • ADR
          • 4. Execution Control - Inner Loops
          • 5. Execution Control - URL Loops
          • 6. Execution Control -User-Defined Transactions And Page Breaks
          • 7. Custom Scripting - Inline Scripts
          • 8. Custom Scripting - Load Test Plug-ins
            • ZebraTester Plug-in Handbooks
          • Modular Scripting Support
        • 3. Recording Session Replay
        • 4. Execute the Load Test
          • Executing a First Load Test
          • Executing Load Test Programs
            • Project Navigator
              • Configuration of the Project Navigator Main Directory
            • Real-Time Load Test Actions
            • Real-Time Error Analysis
            • Acquiring the Load Test Result
            • More Tips for Executing Load Tests
          • Distributed Load Tests
            • Exec Agents
            • Exec Agent Clusters
          • Multiple Client IP Addresses
            • Sending Email And Alerts
            • Using Multiple Client IP Addresses per Load-Releasing System
        • 5. Analyzing Results
          • Detail Results
          • Load Test Result Detail-Statistics and Diagrams
          • Enhanced HTTP Status Codes
          • Error Snapshots
          • Load Curve Diagrams
          • URL Exec Step
          • Comparison Diagrams
            • Analysis Load Test Response Time Comparison
            • Performance Overview
            • Session Failures
        • Programmatic Access to Measured Data
          • Extracting Error Snapshots
          • Extracting Performance Data
        • Web Tools
        • Advanced Topics
          • Execute a JMeter Test Plan in ZebraTester
          • Credentials Manager for ZebraTester
          • Wildcard Edition
          • Execution Plan in ZebraTester
          • Log rotation settings for ZebraTester Processes
          • Modify Session
          • Modular Scripting Support
          • Understanding Pacing
          • Integrating ZebraTester with GIT
            • GitHub Integration Manual V5.4.1
      • ZebraTester FAQ
      • ZebraTester How-to articles
        • How to Combine Multiple ZebraTester Scripts Into One
        • Inline Scripting
        • How to Configure a ZebraTester Script to Fetch Credentials from CyberArk
        • How to Configure a ZebraTester Scenario to Fetch Credentials from CyberArk
        • How to Convert a HAR file into a ZebraTester Script
        • How to Convert a LoadRunner Script to ZebraTester
        • How to Import the ZT Root Certificate to an iOS device
        • How to iterate over JSON objects in ZebraTester using Inline Scripts
        • How to round a number to a certain number of decimal points within a ZebraTester Inline Script
        • How to Use a Custom DNS Host File Within a ZebraTester Script
        • How to Move a ZebraTester Script to an Older Format
        • API Plugin Version
        • Setting up the Memu Player for ZebraTester Recording
        • Inline Script Version
      • Apica Data Repository (ADR) aka Apica Table Server
        • ADR related inline functions available in ZT
        • Apica Data Repository Release Notes
        • REST Endpoint Examples
        • Accessing the ADR with Inline Scripts
      • ZebraTester Plugin Repository
      • Apica YAML
        • Installing and Using the ApicaYAML CLI Tool
        • Understanding ApicaYAML Scripting and Syntax
    • Load Testing Overview
      • Getting Started with ALT
      • Creating / Running a Single Load Test
      • Running Multiple Tests Concurrently
      • Understanding Loadtest Results
    • Test Data Orchestrator (TDO)
      • Technical Guides
        • Hardware / Environment Requirements
        • IP Forwarding Instructions (Linux)
        • Self-Signed Certificate
        • Windows Server Install
        • Linux Server Install
        • User Maintenance
        • LDAP Setup
        • MongoDB Community Server Setup
        • TDX Installation Guide
      • User Documentation
        • End User Guide for TDO
          • Connecting to Orson
          • Coverage Sets and Business Rules
          • Data Assembly
          • Downloading Data
        • User Guide for TDX
          • Connecting to TDX
          • Setting up a Data Profile
          • Extracting Data
          • Analyzing Data Patterns
          • Performing Table Updates
        • API Guide
          • API Structure and Usage
          • Determining Attribute APIs
            • Create Determining Attribute (Range-based)
            • Create Determining Attribute (Value-based)
            • Update Determining Attributes
            • Get Determining Attribute Details
            • Delete a Determining Attribute
          • Coverage Set API’s
            • Create Coverage Set
            • Update Coverage Set
            • Get All Coverage Set Details
            • Get Single Coverage Set Details
            • Lock Coverage Set
            • Unlock Coverage Set
            • Delete Coverage Set
          • Business Rule API’s
            • Create Business Rule
            • Update Business Rule
            • Get Business Rule Details
            • Get All Business Rules
            • Delete Business Rule
          • Workset API's
            • Create Workset
            • Update Workset
            • Get All Worksets
            • Get Workset Details
            • Unlock Workset
            • Clone Workset
            • Delete Workset
          • Data Assembly API's
            • Assemble Data
            • Check Assembly Process
          • Data Movement API's
            • Ingest (Upload) Data Files
            • Download Data Files
              • HTML Download
              • CSV Download
              • Comma Delimited with Sequence Numbers Download
              • Pipe Delimited Download
              • Tab Delimited with Sequence Numbers Download
              • EDI X12 834 Download
              • SQL Lite db Download
              • Alight File Format Download
          • Reporting API's
            • Session Events
            • Rules Events
            • Coverage Events
            • Retrieve Data Block Contents
            • Data Assembly Summary
        • Workflow Guide
      • Release Notes
        • Build 1.0.2.0-20250213-1458
  • IRONdb
    • Getting Started
      • Installation
      • Configuration
      • Cluster Sizing
      • Command Line Options
      • ZFS Guide
    • Administration
      • Activity Tracking
      • Compacting Numeric Rollups
      • Migrating To A New Cluster
      • Monitoring
      • Operations
      • Rebuilding IRONdb Nodes
      • Resizing Clusters
    • API
      • API Specs
      • Data Deletion
      • Data Retrieval
      • Data Submission
      • Rebalance
      • State and Topology
    • Integrations
      • Graphite
      • Prometheus
      • OpenTSDB
    • Tools
      • Grafana Data Source
      • Graphite Plugin
      • IRONdb Relay
      • IRONdb Relay Release Notes
    • Metric Names and Tags
    • Release Notes
    • Archived Release Notes
  • Administration
    • E-Mail Configuration
    • Single Sign-On with SAML
    • Port Management
    • Audit Trail
      • Events Trail
      • Alerts Trail
Powered by GitBook
On this page
  • irondb.conf
  • snowth
  • cache
  • logs
  • listeners
  • pools
  • REST Configuration
  • raw_database
  • nntbs
  • histogram_ingest
  • histogram
  • surrogate_database
  • metric_name_database
  • journal
  • topology
  • Module Config
  • Graphite Config
  • OpenTSDB Config
  • TLS Configuration
  • Converting To TLS
  • Included Files
  • circonus-watchdog.conf
  • irondb-eventer.conf
  • irondb-eventer-site.conf
  • irondb-modules.conf
  • irondb-modules-site.conf
  • irondb-extensions-site.conf
  • licenses.conf

Was this helpful?

Edit on GitHub
Export as PDF
  1. IRONdb
  2. Getting Started

Configuration

Configuration files and options.

PreviousInstallationNextCluster Sizing

Last updated 22 days ago

Was this helpful?

IRONdb is implemented using , a framework for building high-performance C applications. You may wish to review the libmtev for an overview of how libmtev applications are configured generally.

This document deals with options that are specific to IRONdb, but links to relevant libmtev documentation where appropriate.

Default values are those that are present in the default configuration produced during initial installation.

Time periods are specified as second-resolution .

irondb.conf

This is the primary configuration file that IRONdb reads at start. It includes additional configuration files which are discussed later.

snowth

<snowth lockfile="/irondb/logs/snowth.lock" text_size_limit="512">

IRONdb's libmtev application name. This is a required node and must not be changed.

snowth lockfile

Path to a file that prevents multiple instances of the application from running concurrently. You should not need to change this.

Default: /irondb/logs/snowth.lock

snowth text_size_limit

The maximum length of a text-type metric value. Text metric values longer than this limit will be truncated.

Default: 512

Text-type metrics are supported in IRONdb but Graphite currently has no way to render these when using a Storage Finder plugin.

cache

<cache cpubuckets="128" size="0"/>

An LRU cache of open filehandles for numeric metric rollups. This can improve rollup read latency by keeping the on-disk files for frequently-accessed streams open.

The cache is divided up into the specified number of "buckets" to facilitate concurrent access by multiple threads. This parameter rarely requires tuning.

Default: 128

logs

By default, the following log files are written and automatically rotated, with the current file having the base name and rotated files having an epoch-timestamp suffix denoting when they were created:

  • /irondb/logs/errorlog: Output from the daemon process, including not just errors but also operational warnings and other information that may be useful to Apica Support.

    • Rotated: 24 hours

    • Retained: 1 week

  • /irondb/logs/startuplog: Additional non-error initialization output.

    • Rotated: 24 hours

    • Retained: 1 week

  • /irondb/logs/accesslog: Logs from the REST API, including metric writes and reads as well as inter-node communication.

    • Rotated: 1 hour

    • Retained: 1 week

Sometimes it may be desirable to log data submissions that are older than some threshold, in order to identify the source. Submitting "old" data can cause issues with rollups being interrupted, as well as introducing unwanted changes to historical data. IRONdb has a debug-level logging facility for recording such submissions.

Since version 0.20.2 a configuration to log such submissions has been available. It is not active by default, but can be activated by setting disabled="false" on the debug/old_data log:

<log name="debug/old_data" disabled="false"/>
<old_data_logging metric_age_threshold="7d"/>

listeners

Each listener below is configured within a <listener> node. Additional listeners may be configured if desired, or the specific address and/or port may be modified to suit your environment.

<listener address="*" port="8112" backlog="100" type="http_rest_api" accept_thread="on" fanout="true" ssl="off">
  <config>
    <document_root>/opt/circonus/share/snowth-web</document_root>
  </config>
</listener>

The main listener serves multiple functions:

  • JSON-formatted node statistics (http://thisnode:thisport/stats.json)

The IP address on which to listen, or the special * to listen on any local IP address.

Default: *

The port number to listen on. For the main listener this will utilize both TCP and UDP.

Default: 8112

The size of the queue of pending connections. This is used as an argument to the standard listen(2) system call. If a new connection arrives when this queue is full, the client may receive an error such as ECONNREFUSED.

Default: 100

The type of libmtev listener this is. The main listener is configured to be only a REST API listener. This value should not be changed.

Default: http_rest_api

If set to on, IRONdb will dedicate an eventer thread to handling incoming connections. This improves performance by ensuring that a new connection will be fully processed in blocking fashion, without preemption.

Default: off

If set to true, new events from accepted connections will be fanned out across all threads in the event pool owning the listening socket (usually the default event pool).

Default: false

Default: off

<listener address="*" port="2003" type="graphite">
  <config>
    <check_uuid>3c253dac-7238-41a1-87d7-2e546f3b4318</check_uuid>
    <account_id>1</account_id>
  </config>
</listener>

The IP address on which to listen, or the special * to listen on any local IP address.

Default: *

The TCP port number to listen on.

Default: 2003

The type of listener. IRONdb implements a Graphite-compatible handler in libmtev, using the custom type "graphite".

Default: graphite

  • account_id is also part of namespacing, for disambiguation.

Its configuration is identical to the plaintext listener, except the type is graphite_pickle.

<listener address="127.0.0.1" port="32322" type="mtev_console">
  <config>
    <line_protocol>telnet</line_protocol>
  </config>
</listener>

The IP address on which to listen, or the special * to listen on any local IP address.

Default: 127.0.0.1

The TCP port number to listen on.

Default: 32322

The CLI listener uses the built-in libmtev type "mtev_console" to allow access to the telnet console.

Default: mtev_console

pools

NOTE: As of version 0.20.0, resource configuration from this stanza is deprecated. Fresh installations will no longer contain this stanza.

Values from these attributes will still be respected until a future release. Deprecation messages will be logged for each pools attribute encountered in the configuration, and will include the name of the jobq that corresponds to that attribute.

<pools>
  <rollup concurrency="1"/>
  <nnt_put concurrency="16"/>
  <raw_writer concurrency="4"/>
  <raw_reader concurrency="16"/>
  <rest_graphite_numeric_get concurrency="4"/>
  <rest_graphite_find_metrics concurrency="4"/>
  <rest_graphite_fetch_metrics concurrency="10"/>
</pools>

Resource pools within IRONdb are used for various functions, such as reading and writing metric data. Some aspects of pool behavior are configurable, typically to adjust the number of worker threads to spawn.

The defaults presented are widely applicable to most workloads, but may be adjusted to improve throughput. Use caution when raising these values too high, as it could produce thrashing and decrease performance.

Deprecated

Use jobq_rollup_raw to preserve customizations.

The number of unique metric names (UUID + metric name) to process in parallel when performing rollups. A higher number generally causes the rollup operation to finish more quickly, but has the potential to overwhelm the storage subsystem if set too high.

Default: 1

These tasks compete with other readers of the raw_database, so if rollup concurrency is set higher than 4x raw_writer concurrency, it cannot be reached.

Deprecated

This attribute is obsolete and may be removed from configuration files.

The number of threads used for writing to numeric rollup files. Writes to a given rollup file will always occur in the same queue.

Default: the number of physical CPU cores present during installation

Deprecated

Use jobq_data_write to preserve customizations.

The number of threads used for writing to the raw metrics database. Additionally, by default, IRONdb will use 4x this number of threads for reading from the raw metrics database.

Default: 4

Deprecated

Use jobq_data_read to preserve customizations.

The number of threads used for reading from the raw metrics database.

Default: (raw_writer concurrency * 4)

Deprecated

Use jobq_snowth_graphite_numeric_get to preserve customizations.

The number of threads used for handling Graphite fetches. This is a general queue for all fetch operations, and there are two other thread pools for specific tasks within a fetch operation (see below.)

Default: 4

Deprecated

Use jobq_snowth_graphite_find_metrics_local and jobq_snowth_graphite_find_metrics_remote to preserve customizations. The value for this pools attribute was interpreted as the remote concurrency, which was divided by 4 to get the local concurrency (minimum 1).

The number of threads used for resolving metric names prior to fetch.

Default: 4

Deprecated

Use jobq_snowth_graphite_fetch_metrics_local and jobq_snowth_graphite_fetch_metrics_remote to preserve customizations. The value for this pools attribute was interpreted as the remote concurrency, which was divided by 4 to get the local concurrency (minimum 1).

The number of threads used for actually fetching Graphite metrics, including those local to the node and those residing on remote nodes.

Default: 10

REST Configuration

This is the node under which REST API configuration items are organized.

<rest>
  <acl>
    <rule type="allow" />
  </acl>
  <delete max_advisory_limit="10000" />
</rest>

This is the node used to configure DELETE endpoint behavior.

<rest>
  <delete max_advisory_limit="<val>"/>
</rest>

max_advisory_limit="<val>" attribute is used to configure how many deletes may be attempted by this operation where <val> may not be exceeded via X-Snowth-Advisory-Limit. Currently, this only affects the /full/tags endpoint.

raw_database

<raw_database location="/irondb/raw_db/{node}"
              data_db="nomdb"
              granularity="1w"
              recordsize="1h"
              min_delete_age="4w"
              delete_after_quiescent_age="1d"
              rollup_after_quiescent_age="8h"
              startup_rollup_delay="30m"
              max_clock_skew="1w"
              conflict_resolver="abs_biggest"
              rollup_strategy="raw_iterator"
              sync_after_full_rollup_finishes="false"
              sync_after_column_family_rollup_finishes="false"
              suppress_rollup_filter="and(__rollup:false)"
/>

The location and data_db attributes should not be modified.

Granularity controls the sharding of the raw numeric database. A shard is the unit of data that will be rolled up and removed after a configurable age and period of quiescence (no new writes coming in for that shard.)

Do not change granularity after starting to collect data, as this will result in data loss.

Default: 1 week

Recordsize controls the amount of data stored in an individual raw record.

Do not change recordsize after starting to collect data, as this will result in data loss.

Default: 1 hour

The minimum age that a shard must be before it is considered for deletion.

Default: 4 weeks

The period after which a shard, if it has been rolled up and not subsequenty written to, may be deleted.

Default: 1 day

The period the system will delay after the last write to a raw shard before attempting to roll it up. New writes to the time period/shard will interrupt the rollup process and reset the quiescent timer which must again reach the rollup_after_quiescent_age before a re-roll will be attempted.

Default: 8 hours

If an irondb instance restarted while it was doing a rollup, it will restart that rollup after it finishes booting, however it will wait startup_rollup_delay before doing so. This gives the node time to catch-up on ingestion, populate caches, and other operations it may need to do after a restart.

Default: 30 minutes

Allow the submission of metrics timestamped up to this amount of time in the future, to accommodate clients with incorrect clocks.

Default: 1 week

When a metric gets written more than one time at the exact millisecond offset you have a conflict we have to resolve. All operations in IRONdb are commutative and this lets us avoid complicated consensus algorithms for data. Conflicts, therefore, need to choose a winner and this choice needs to be consistent across the cluster. IRONdb gives you the following choices for conflict resolution should a datapoint appear more than once at the same millisecond.

  • abs_biggest - save the largest by absolute value.

  • abs_smallest - save the smallest by absolute value.

  • last_abs_smallest - same as last_abs_biggest but smallest instead.

  • last_biggest - same as last_abs_biggest but uses the largest without absolute value.

  • last_smallest - same as last but smallest.

  • biggest - the larger value without absolute.

  • smallest - the smaller value without absolute.

This setting should be the same on all nodes of the IRONdb cluster.

This value should never be changed when data is "in flight", that is, while a cluster is actively ingesting data, or there are nodes down, or nodes are suffering replication latency.

If you wish to change this setting after beginning to collect data, the following conditions must be met:

  • All nodes must be running and available.

  • All ingestion must be stopped.

Once these conditions are met:

  1. Bring down all nodes.

  2. Change the value of this option in the configuration file for each node.

  3. Restart all nodes.

Default: "abs_biggest"

Control how rollups are performed. By default, all levels of rollup data are calculated from the raw database as it is iterated.

Prior to version 0.12 the default if not specified was that the lowest level of rollup was computed and then IRONdb would read this lowest level data and compute higher level rollups. This rollup strategy has been removed.

Default: "raw_iterator"

Enables an LMDB sync to disk after each raw shard finishes rolling up. Each shard that the raw shard rolls up into will be synced.

Default: "false"

Enables an LMDB sync to disk after each column family within a raw shard finishes rolling up. Each shard that the raw shard rolls up into will be synced.

Default: "false"

Default: and(__rollup:false)

Introduced in IRONdb version 0.19.2

nntbs

<nntbs path="/irondb/nntbs/{node}">
  <shard period="60" size="1d" retention="52w" />
  <shard period="300" size="5d" retention="104w" />
  <shard period="1800" size="30d" retention="104w" />
  <shard period="10800" size="180d" retention="520w" />
</nntbs>

Each shard specifies a rollup using a given granularity in seconds (period).

NOTE: for installations with a high cardinality of metric names you will want to reduce the size parameters to keep the shards small to ensure performance remains consistent.

The retention setting for each shard determines how long to keep this data on disk before deleting it permanently. retention is optional and if you don't provide it, IRONdb will keep the data forever. When a timeshard is completely past the retention limit based on the current time, the entire shard is removed from disk. In the above example, 60-second rollups are retained for 52 weeks (1 year), 5- and 30-minute rollups are retained for 104 weeks (2 years), and 3-hour rollups are retained for 520 weeks (10 years). Retention uses the same time duration specifications as size above.

Whatever settings are chosen here cannot be changed after the database starts writing data into NNTBS (except for retention). If you change your mind about sizing you will have to wipe and reconstitute each node in order to apply new settings.

histogram_ingest

<histogram_ingest location="/irondb/hist_ingest/{node}"
              data_db="nomdb"
              granularity="7d"
              min_delete_age="4w"
              delete_after_quiescent_age="1d"
              rollup_after_quiescent_age="8h"
              max_clock_skew="1w"
/>

The location and data_db attributes should not be modified.

Granularity controls the sharding of the raw histogram database. A shard is the unit of data that will be rolled up and removed after a configurable age and period of quiescence (no new writes coming in for that shard.)

Do not change granularity after starting to collect data, as this will result in data loss.

Default: 1 week

The minimum age that a shard must be before it is considered for deletion.

Default: 4 weeks

The period after which a shard, if it has been rolled up and not subsequenty written to, may be deleted.

Default: 1 day

The period the system will delay after the last write to a shard before attempting to roll it up. New writes to the time period/shard will interrupt the rollup process and reset the quiescent timer which must again reach the rollup_after_quiescent_age before a re-roll will be attempted.

Default: 8 hours

Allow the submission of metrics timestamped up to this amount of time in the future, to accommodate clients with incorrect clocks.

Default: 1 week

histogram

<histogram location="/irondb/hist_rollup/{node}">
  <rollup period="60" granularity="7d"/>
  <rollup period="300" granularity="30d"/>
  <rollup period="1800" granularity="12w"/>
  <rollup period="10800" granularity="52w"/>
  <rollup period="86400" granularity="260w"/>
</histogram>

Whatever settings are chosen here cannot be changed after the database starts writing data (except for retention). If you change your mind about sizing you will have to wipe and reconstitute each node in order to apply new settings.

The period defines the time interval, in seconds, for which histogram metrics will be aggregated into the rollup.

NOTE: for installations with a high cardinality of metric names you will want to reduce the granularity parameters to keep the shards small to ensure performance remains consistent.

retention is optional and the default behavior is to keep the rollup data forever.

When a rollup timeshard is completely past the retention limit based on the current time, the entire shard is removed from disk.

Introduced in IRONdb version 0.23.7

surrogate_database

IMPORTANT NOTE: Any node running 0.23.7 or earlier MUST do a surrogate3 migration PRIOR to upgrading to 1.0.0. This is due to removal of support for the previous surrogate database format. See the "db_type" section below for details.

Data files are stored on disk and memory-mapped on demand when metrics are referenced by queries (read) or ingestion (write).

<surrogate_database location="/irondb/surrogate_db/{node}" db_type="s3"/>

Historically, the surrogate database was implemented as RocksDB, and loaded into RAM on startup. This is referred to as "surrogate2". Support for surrogate2 was removed in version 1.0.0.

To migrate a node to surrogate3, remove any surrogate2-specific attributes, if used, from the <surrogate_database> config (cache_init_size, load_concurrency, activity_tracking, asynch_update_journal_path) and change db_type to s3. When the node boots, the database will be migrated into on-disk files that no longer require keeping all entries in RAM. This migration is one-way and must be performed on versions 0.23.5-0.23.7 before upgrading to 1.0.0. The node will not be available for new ingestion during the migration, and the time required will be proportional to the number of unique metrics stored in the surrogate database.

flat_buffers is a legacy value that also indicates surrogate3, but is discouraged in favor of s3.

Default: s3

Default: false

This is the upper bound on whether a metric will be considered as a "latest value" candidate. By default if a metric timestamp is more than 4 hours in the future, it will be ignored for consideration as a replacement for the latest value. These values are only updated at ingestion time.

This value can be from 0s (ignore any future timestamps) to 4h (maximum).

Default: 4h

metric_name_database

This database stanza controls where IRONdb keeps certain aspects of its indexes.

<metric_name_database location="/irondb/metric_name_db/{node}"
              enable_level_indexing="true"
              materialize_after="100000"
              query_cache_size="1000"
              query_cache_timeout="900"
              enable_saving_bad_level_index_jlog_messages="false"
/>

The database of stored metric names. This database is used to satisfy graphite /metrics/find queries. By default, this database will cache 1000 queries for 900 seconds. Any newly arriving metric names will invalidate the cache so subsequent queries are correct.

Level indexing is used for graphite-style query acceleration. For large clusters that do not user graphite-style metrics, it may improve memory/CPU utilization to disable this index.

Default: true

The number of mutations that must occur before the system will flush to disk and trigger a compaction to occur, draining the jlog of queued updates.

Default: 100,000

The location on disk where the database files reside.

The number of incoming graphite/find queries to cache the results for.

Default: 1000

The number of seconds that cached queries should remain in the cache before being expired.

Default: 900

Enables saving of invalid jlog messages found when attempting to replay the jlog in the metric name database to build the indexes. The messages will be saved within the metric name database location for the account on which the error occurred in a folder called bad_flatbuffer_messages.

Default: "false"

journal

<journal concurrency="4"
         replicate_concurrency="4"
         max_bundled_messages="50000"
         max_total_timeout_ms="10000"
         pre_commit_size="131072"
         send_compressed="true"
         use_indexer="false"
/>

Journals are write-ahead logs for replicating metric data to other nodes. Each node has one journal for each of its cluster peers.

Establishes this number of concurrent threads for writing to each peer journal, improving ingestion throughput.

Default: 4

A concurrency of 4 is enough to provide up to 700K measurements/second throughput, and is not likely to require adjustment except in the most extreme cases.

Attempt to maintain this number of in-flight HTTP transactions, per peer journal, for posting replication data to peers. Higher concurrency helps keep up with ingestion at scale.

Each thread reads a portion of the journal log and is responsible for sending that portion to the peer. When it finishes its portion, and there are fewer than replicate_concurrency other jobs in flight for that peer, it skips ahead to the next "unclaimed" portion of the log and resumes sending.

Default: 4

Prior to version 0.15.3, the default was 1.

Outbound journal messages will be sent in batches of up to this number, improving replication speed.

Default: 50000

A node sending replication journals to its peers will allow up to this amount of time, in milliseconds, for the remote node to receive and process a batch. If nodes are timing out while processing incoming journal batches, increasing this timeout may give them enough time, avoiding repeatedly sending the same batch.

Default: 10000 (10 seconds)

An in-memory buffer of this number of bytes will be used to hold new journal writes, which will be flushed to the journal when full. This can improve ingestion throughput, at the risk of losing up to this amount of data if the system should fail before commit. To disable the pre-commit buffer, set this attribute to 0.

Default: 131072 (128 KB)

When sending journal messages to a peer, compress the messages before sending to save bandwidth, at the cost of sligtly more CPU usage. The bandwidth savings usually outweigh the cost of compression.

Default: true

Note that this will spawn one extra thread per journal (there is one journal for every remote node in the cluster.) For example, activating this feature will spawn 15 additional threads on each node in a 16-node cluster.

Default: false

topology

<topology path="/opt/circonus/etc/irondb-topo"
          active="(hash value)"
          next=""
          redo="/irondb/redo/{node}"
/>

No manual configuration of these settings is necessary.

Module Config

Graphite Config

<graphite min_rollup_span_ms="60000" max_ingest_age="365d">
  <whisper directory="/opt/graphite/storage/whisper"
           check_uuid="3c253dac-7238-41a1-87d7-2e546f3b4318"
           account_id="1"
           end_epoch_time="1780000000"
  />
</graphite>

Default: 1 year

The smallest rollup period that is being collected. This prevents gaps when requesting data at shorter intervals.

Default: 1 minute

Note that regardless of filesystem choice, it is highly desirable to mount it read-only on each cluster node. This becomes a requirement if using a shared storage volume in the cloud.

Multiple whisper entitites may be configured, each representing a logically distinct Graphite installation. Using different values for check_uuid and (potentially) account_id will segregate these metrics from others.

The directory attribute is required, and indicates the start of a hierarchy of directories containing Whisper database files. This path may exist on the local filesystem, or on a network-mounted filesystem.

For example, to locate a Whisper database stored at /opt/graphite/storage/whisper/foo/bar.wsp, set the directory attribute to "/opt/graphite/storage/whisper". The metric will be indexed as foo.bar.

Each whisper entity must have a unique, non-overlapping directory value. For example, it is an error to configure one with /foo and another with /foo/bar.

The end_epoch_time is optional and represents the last timestamp for which there is whisper data. The timestamp is provided as an epoch timestamp, in seconds. If a fetch has a start time after the provided time, the node will not look in the whisper file in order to be more efficient. If this field is not provided, the whisper files will be checked regardless of the start time of the fetch.

OpenTSDB Config

<opentsdb max_ingest_age="365d"/>

Default: 1 year

TLS Configuration

As of version 1.1.0, IRONdb supports TLS for both client and intra-cluster communications. This is currently an alpha feature, for testing only.

Due to certificate verification requirements, two sets of cryptographic keys and associated certificates are required:

  1. Intra-cluster communication: cluster nodes exchange information and replicate metric data using port 8112, and they use the node UUID as the hostname for all requests. When TLS is used, the certificates for this listener must use the node UUID as the certificate CommonName (CN).

  2. External client connections: since it would be awkward for external clients to verify a CN that is just a UUID, a second listener is added, using port 8443 and having its certificate CN set to the host's FQDN. This matches the expectation of clients connecting to the node to submit metrics or run queries.

The following files must be present on each node in order for the service to work properly with TLS. Place them in /opt/circonus/etc/ssl:

  • cluster.key - An RSA key for the intra-cluster listener.

  • cluster.crt - A certificate issued for the intra-cluster listener. Its commonName (CN) must be the node's UUID.

  • cluster-ca.crt - The Certificate Authority's public certificate, sometimes referred to as an intermediate or chain cert, that issued cluster.crt.

  • client.key - An RSA key for the external client listener.

  • client.crt - A certificate issued for the external client listener. Its commonName (CN) should match the hostname used to connect to the node, typically its FQDN.

  • client-ca.crt - The Certificate Authority's public certificate, sometimes referred to as an intermediate or chain cert, that issued client.crt.

Converting To TLS

To update an existing cluster to use TLS, several things need to change.

  1. A modified topology configuration that indicates TLS should be used for intra-cluster communication.

  2. Changes to listener configuration to specify locations for key, certificate, and CA chain certificate, add a new listener port for external clients, and to activate TLS.

The first two items will be done on all IRONdb nodes. The third item will vary depending on the specifics of the metric submission pipeline(s) and visualization platforms.

NOTE: because of the nature of this change, there will be disruption to cluster availability as the new configuration is rolled out. Nodes with TLS active will not be able to communicate with nodes that do not have TLS active, and vice versa.

The active topology for a cluster will be located in the/opt/circonus/etc/irondb-topo directory, as a file whose name matches the topology hash. This hash is recorded in /opt/circonus/etc/irondb.conf as the value for the active attribute within the <topology> stanza, e.g.

  <!-- Cluster definition -->
  <topology path="/opt/circonus/etc/irondb-topo"
            active="98e4683192dca2a2c22b9a87c7eb6acecd09ece89f46ce91fd5eb6ba19de50fb"
            next=""
            redo="/irondb/redo/{node}"
  />

Edit the /opt/circonus/etc/irondb-topo/<hash> file and add the use_tls="true" attribute to the nodes line:

-<nodes write_copies="2">
+<nodes write_copies="2" use_tls="true">

Distribute the updated file to all nodes in the cluster.

In /opt/circonus/etc/irondb.conf, locate the <listeners> stanza. The listeners that will be changing are the ones for port 8112 and, if used, the Graphite listener on port 2003.

In a default configuration, the non-TLS listeners look like this:

    <listener address="*" port="8112" backlog="100" type="http_rest_api" accept_thread="on" fanout="true">
      <config>
        <document_root>/opt/circonus/share/snowth-web</document_root>
      </config>
    </listener>

   <listener address="*" port="2003" type="graphite">
      <config>
        <check_uuid>6a07fd71-e94d-4b67-a9bc-29ac4c1739e9</check_uuid>
        <account_id>1</account_id>
      </config>
    </listener>

The Graphite check_uuid and account_id may differ from the above. Preserve those values in the new listener config.

Replace the above listener configs with this, ensuring that it is within the opening and closing listeners tags, and substituting your Graphite check UUID and account ID from the original config:

    <!--
      Intra-cluster listener. Used for gossip and replication.
    -->
    <cluster>
      <sslconfig>
        <!-- Certificate CNs MUST match node UUIDs assigned in the current topology. -->
        <certificate_file>/opt/circonus/etc/ssl/cluster.crt</certificate_file>
        <key_file>/opt/circonus/etc/ssl/cluster.key</key_file>
        <ca_chain>/opt/circonus/etc/ssl/cluster-ca.crt</ca_chain>
        <layer_openssl_10>tlsv1.2</layer_openssl_10>
        <layer_openssl_11>tlsv1:all,>=tlsv1.2,cipher_server_preference</layer_openssl_11>
        <ciphers>ECDHE+AES128+AESGCM:ECDHE+AES256+AESGCM:DHE+AES128+AESGCM:DHE+AES256+AESGCM:!DSS</ciphers>
      </sslconfig>
      <listener address="*" port="8112" backlog="100" type="http_rest_api" accept_thread="on" fanout="true" ssl="on">
        <config>
          <document_root>/opt/circonus/share/snowth-web</document_root>
        </config>
      </listener>
    </cluster>

    <!-- Client-facing listeners. -->
    <clients>
      <sslconfig>
        <!-- Certificate CNs should be the FQDN of the node. -->
        <certificate_file>/opt/circonus/etc/ssl/client.crt</certificate_file>
        <key_file>/opt/circonus/etc/ssl/client.key</key_file>
        <ca_chain>/opt/circonus/etc/ssl/client-ca.crt</ca_chain>
        <layer_openssl_10>tlsv1.2</layer_openssl_10>
        <layer_openssl_11>tlsv1:all,>=tlsv1.2,cipher_server_preference</layer_openssl_11>
        <ciphers>ECDHE+AES128+AESGCM:ECDHE+AES256+AESGCM:DHE+AES128+AESGCM:DHE+AES256+AESGCM:!DSS</ciphers>
      </sslconfig>

      <!-- Used for HTTP metric submission, admin UI. -->
      <listener address="*" port="8443" backlog="100" type="http_rest_api" accept_thread="on" fanout="true" ssl="on">
        <config>
          <document_root>/opt/circonus/share/snowth-web</document_root>
        </config>
      </listener>

      <!--
        Graphite listener
          This installs a network socket graphite listener under the account
          specified by <account_id>.
      -->
      <listener address="*" port="2003" type="graphite" ssl="on">
        <config>
          <check_uuid>GRAPHITE_CHECK_UUID</check_uuid>
          <account_id>ACCOUNT_ID</account_id>
        </config>
      </listener>
    </clients>

Generate and/or obtain the above key and certificate files, ensuring they are placed in the correct location as set in the listener sslconfig configuration.

Included Files

circonus-watchdog.conf

<watchdog glider="/opt/circonus/bin/backwash" tracedir="/opt/circonus/traces"/>

irondb-eventer.conf

This file contains default settings for event loops and job queues. Overrides should be placed in irondb-eventer-site.conf.

Settings in here should generally not be changed unless directed by Apica Support.

Many parts of IRONdb's functionality are handled within pools of threads that form "job queues" (abbreviated as jobq). Any actions that may block for some period of time, such as querying for data, performing rollups, etc. are handled asynchronously via these queues.

The value of each jobq_NAME is one or more comma-separated values:

concurrency[,min[,max[,memory_safety[,backlog]]]]

Concurrency is required; all others are optional, but position is significant. For example, overriding the backlog value will require min, max, and memory_safety to be filled in as well.

As with event loop settings, the job queue defaults are suitable for a wide range of workloads, so changes should be carefully tested to ensure they do not reduce performance or cause instability.

To override a jobq named foo, which might be defined by default as:

<jobq_foo>4,1,24</jobq_foo>

Place a line in the site configuration file with one or more different values, preserving the others:

<jobq_foo>8,1,24</jobq_foo>

The above would increase the desired concurrency from 4 to 8, keeping the minimum of 1 and maximum of 24.

irondb-eventer-site.conf

See the comment at the top of the file for how to override eventer settings. This file is included from irondb-eventer.conf.

This file's contents will be preserved across package updates.

irondb-modules.conf

Settings in this file should not be changed.

irondb-modules-site.conf

See the comment at the top of the file for how to configure optional modules. This file is included from irondb-modules.conf.

This file's contents will be preserved across package updates.

irondb-extensions-site.conf

See the comment at the top of the file for how to add or override extension configuration. This file is included from irondb-modules.conf.

This file's contents will be preserved across package updates.

licenses.conf

In a cluster, the license configuration must be the same on all cluster nodes.

If no license is configured, an embedded license is used, which enables all features described below with a limit of 25,000 active streams (max_streams).

The IRONdb license governs the following functionality:

Name: <expiry>

After this unix timestamp the license is invalid and will no longer work for any of the below.

Name: <max_streams>

How many unique time series (uniquely named streams of data) this installation can ingest in the most recent 5-minute period.

This number applies to all nodes in the cluster although each node applies this restriction individually. The math for unique streams is an estimate in the past 5 minutes and you are given a 15% overage before ingestion is affected.

If this license is violated, ingestion will stop for the remainder of the 5-minute period that the violation was detected. After the 5-minute period ends, the counter will reset to test the new 5-minute period.

Name: <lua_extension>

Whether or not Lua extensions will operate.

Name: <stream_tags>

Whether or not stream tag related API calls and stream tag ingestion will work. If you do not have this license and stream tagged data arrives it will be silently discarded.

Name: <histograms>

Whether or not histograms can be ingested. If you do not have this license and attempt to ingest histogram data it will be silently discarded.

Name: <text>

Whether or not text metrics can be ingested. If you do not have this license and attempt to ingest text data it will be silently discarded.

cache cpubuckets

Libmtev logging configuration. See the .

Logging old data submission

The threshold for what is considered "old" is controlled by metric_age_threshold. The value is a string representing an offset into the past from "now". The default is 7 days. Any data submitted with a timestamp that is further in the past will be logged.

Libmtev network listener configuration. See the .

Main listener

(TCP) and gossip (UDP)

Main listener address

Main listener port

Main listener backlog

Main listener type

Main listener accept_thread

Main listener fanout

Main listener ssl

When set to on, the listener will expect incoming connections to use Transport Layer Security (TLS), also known as "SSL". Additional TLS configuration is required. See .

Graphite listener

The Graphite listener operates a Carbon-compatible submission pathway using the .

Multiple Graphite listeners may be configured on unique ports and associated with different check UUIDs. See the section on for details.

Graphite listener address

Graphite listener port

Graphite listener type

Graphite listener config

These configuration items control which check UUID, name, and account ID are associated with this listener. The first Graphite listener is configured during .

check_uuid is a UUID the will be associated with all metrics ingested via this listener.

Pickle listener

The Pickle listener operates a Carbon-compatible submission pathway using the .

CLI listener

The CLI listener provides a local for interacting with libmtev subsystems, including modifying configuration. As there is no authentication mechanism available for this listener, it is recommended that it only be operated on the localhost interface.

CLI listener address

CLI listener port

CLI listener type

The value of the "concurrency" attribute is the first value in jobq configuration. See for details.

If in doubt, .

pools rollup concurrency

pools nnt_put concurrency

pools raw_writer concurrency

pools raw_reader concurrency

pools rest_graphite_numeric_get concurrency

pools rest_graphite_find_metrics concurrency

pools rest_graphite_fetch_metrics concurrency

DELETE Configuration

Raw numeric metrics database. This stores all ingested numeric metrics at full resolution for a configurable period of time, after which the values are rolled up and stored in one or more .

raw_database granularity

raw_database recordsize

raw_database min_delete_age

raw_database delete_after_quiescent_age

raw_database rollup_after_quiescent_age

raw_database startup_rollup_delay

raw_database max_clock_skew

raw_database conflict_resolver

last_abs_biggest - if used with the aggregation capabilities the datapoints can track a generation counter. This resolver considers the generation of the datapoint and then uses the largest by absolute value if the generations collide. If you are not using the relay, this will fall back to the same behavior as abs_biggest.

All from all nodes must be completely drained and applied on the destination node.

raw_database rollup_strategy

raw_database sync_after_full_rollup_finishes

raw_database sync_after_column_family_rollup_finishes

raw_database suppress_rollup_filter

Metrics that match this are never rolled up and only exist in the raw database. Raw only metrics are supported for both numeric and histogram metric types. When raw shards are deleted, a verify step is done on any metric that matches the filter to determine if there is any remaining data for that metric. If there is no remaining data, the metric will be completely deleted from the .

NNTBS is the rollup storage engine for data once it proceeds past the .

Shard size is the included in one shard. The minimum size for a shard is 127 * period; for a 60-second period, this would be 7620 seconds. Whatever time span you provide here will be rounded up to that multiple. For example, if you provided 1d for the period=60 shard as in the defaults above, you would actually get 91440 seconds per shard instead of 86400.

Raw histogram metrics database. This stores all ingested histogram metrics at full resolution for a configurable period of time, after which the values are rolled up and stored in one or more .

histogram_ingest granularity

histogram_ingest min_delete_age

histogram_ingest delete_after_quiescent_age

histogram_ingest rollup_after_quiescent_age

histogram_ingest max_clock_skew

The histogram rollup database for data once it proceeds past the . Rollups must be individually configured with a period, granularity, and optional retention period.

histogram rollup period

histogram rollup granularity

Shard granularity is the included in one shard. The granularity must be divisible by the period and will be rounded up if not compatible.

histogram rollup retention

Shard retention is the that determines how long to keep this rollup data on disk before deleting it permanently.

The surrogate database contains bidirectional mappings between full metric names (including tags) and integer-based keys which are used internally to refer to metrics. It also records on each metric.

surrogate_database db_type

surrogate_database implicit_latest

Toggle for maintaining an in-memory copy of the latest values for all newly seen metrics values during ingestion. If set to false, it will only maintain latest values for metrics that have been specifically "asked for" via a .

surrogate_database latest_future_bound

metric_name_database enable_level_indexing

metric_name_database materialize_after

metric_name_database location

metric_name_database query_cache_size

metric_name_database query_cache_timeout

metric_name_database enable_saving_bad_level_index_jlog_messages

journal concurrency

journal replicate_concurrency

journal max_bundled_messages

journal max_total_timeout_ms

journal pre_commit_size

journal send_compressed

journal use_indexer

Spawn a dedicated read-ahead thread to build indexes of upcoming segments in the write-ahead log for each remote node. This is only needed in the most extreme cases where the highest replication throughput is required. Almost all other installations will not notice any slowdown from indexing "on demand", as new segments are encountered.

The topology node instructs IRONdb where to find its current cluster configuration. The path is the directory where the imported topology config lives, which was created during setup. active indicates the hash of the currently-active topology. next is currently unused. The redo path is where are located for this topology.

The that provide support for ingesting Graphite and/or OpenTSDB data have optional configuration, described below. These settings are placed in the main irondb.conf file, as children of the <snowth> node (i.e., peers of <logs>, <topology>, etc.) If omitted, the defaults shown below will be used.

graphite max_ingest_age

The maximum offset into the past from "now" that will be accepted. Value may be any valid . If importing older data, it may be necessary to increase this value.

graphite min_rollup_span_ms

graphite whisper

The whisper entity configures . Each entity refers to the top of a directory hierarchy containing Whisper database files. This directory may exist on a local filesystem, or on a shared network-filesystem mountpoint. Any Whisper databases discovered in scanning this directory hierarchy with the whisper_loader tool (see link above) will be indexed for searching and querying.

graphite whisper directory

graphite whisper check_uuid

The check_uuid attribute is required, and the contained metrics within IRONdb. This UUID may be arbitrarily chosen, but if the metrics in this collection are the same as those being currently ingested directly into IRONdb, it may be desirable to use the same check_uuid value as the corresponding .

graphite whisper account_id

The account_id attribute is required, and the contained metrics within IRONdb. This ID may be arbitrarily chosen, but if the metrics in this collection are the same as those being currently ingested directly into IRONdb, it may be desirable to use the same account_id value as the corresponding .

graphite whisper end_epoch_time

opentsdb max_ingest_age

The maximum offset into the past from "now" that will be accepted. Value may be any valid . If importing older data, it may be necessary to increase this value.

The will automatically configure TLS listeners on a fresh installation when the -t option or the IRONDB_TLS environment variable is set to on.

Changes to metric submission pipelines and any visualization tools to use the new, externally-verifiable listener. This could include tools such as graphite-web or Grafana, as well as .

Update Topology

Update Listeners

watchdog

The watchdog configuration specifies a handler, known as a "glider", that is to be invoked when a child process crashes or hangs. See the .

If is turned on, the glider is what invokes the tracing, producing one or more files in the tracedir. Otherwise, it just reports the error and exits.

The eventer configuration contains .

Event Loop Configuration

Job Queue Configuration

Contains options for vendor-supplied .

This file holds any and all licenses that apply to this IRONdb node. Refer to the for details on obtaining and installing licenses.

Licensed Features

License Term

Ingest Cardinality

Enablement of Lua Extensions

Stream Tags Support

Histogram Support

Text Metric Support

Obtain A License

If you are interested in any of the above functionality and do not currently have a license please contact to upgrade your license.

libmtev
configuration documentation
libmtev time durations
​
​
​
libmtev logging documentation
​
libmtev time duration
libmtev listener documentation
​
HTTP REST API
​
​
​
​
​
​
​
​
plaintext format
Graphite ingestion
​
​
​
​
initial installation
well-formed, non-nil
​
pickle format
​
telnet console
​
​
​
contact support
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
IRONdb-relay
​
​
​
​
time span
​
​
​
​
​
​
​
time span
​
time span
collection activity periods
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
​
JLog
integration modules
​
libmtev time duration
​
​
​
​
​
​
​
libmtev time duration
IRONdb Relay
​
​
​
libmtev watchdog documentation
libmtev eventer configuration
​
​
libmtev dynamically-loadable modules
​
​
​
​
​
​
​
​
sales@apica.io
TLS Configuration
Job Queue Configuration
period-specific files
journals
raw database
period-specific files
raw histogram database
journals
surrogate database
listener
listener
tag query
tag search
read access to Whisper database files
namespaces
namespaces
installer script
installation steps
Cluster replication
Operations Dashboard
crash handling