Apica Docs
  • Welcome to Apica Docs!
  • PRODUCT OVERVIEW
    • Ascent Overview
    • Ascent User Interface
  • TECHNOLOGIES
    • Ascent with Kubernetes
      • Kubernetes is a Game-Changer
      • Ascent: Built on Kubernetes
    • Ascent with OpenTelemetry
      • Why Implement OpenTelemetry?
      • Common Use Cases for OpenTelemetry
      • How to Get Started with OpenTelemetry
      • Best Practices for OpenTelemetry Implementations
  • RELEASE NOTES
    • Release Notes
      • Ascent 2.10.3
      • Ascent 2.10.2
      • Ascent 2.9.0
      • Ascent 2.8.1
      • Ascent 2.8.0
      • Ascent 2.7.0
      • Ascent 2.6.0
      • Ascent 2.5.0
      • Ascent 2.4.0
      • Ascent 2.3.0
      • Ascent 2.2.0
      • Ascent 2.1.0
        • Data Fabric
          • Releases-old
        • Synthetic Monitoring
        • Advanced Scripting Engine
        • IRONdb
      • Synthetic Monitoring
  • GETTING STARTED
    • Getting Started with Ascent
      • Getting Started with Metrics
      • Getting Started with Logs
        • OpenTelemetry
    • Ascent Deployment Overview
    • Quickstart with Docker-Compose
    • On-Premise PaaS deployment
      • On-Premise PaaS Deployment Architecture
      • Deploying Apica Ascent PaaS on Kubernetes
      • Deploying Apica Ascent PaaS on MicroK8s
      • Deploying Apica Ascent PaaS on AWS
      • Deploying Apica Ascent EKS on AWS using CloudFormation
      • Deploying Ascent on AWS EKS with Aurora PostgreSQL and ElastiCache Redis using Cloud Formation
        • Deploying Apica Ascent on AWS EKS with Aurora PostgreSQL and ElastiCache Redis using CloudFormation
        • Apica Ascent on AWS EKS (Private Endpoint) with Aurora PostgreSQL and ElastiCache Redis on prod VPC
      • Deploying Apica Ascent EKS on AWS using custom AMI
      • Deploying Apica Ascent EKS with AWS ALB
      • Deploying Apica Ascent PaaS in Azure Kubernetes Service
        • Azure Blob Storage Lifecycle Management
      • Deploying Apica Ascent with OpenShift
    • Boomi RTO Quick Start Guide
      • RTO Dashboarding
      • Alerting on RTO Metrics
      • Alerting on RTO Logs
    • Dashboards & Visualizations
  • DATA SOURCES
    • Data Source Overview
    • API
      • JSON Data source
      • RSS
    • AWS
      • Amazon Athena
      • Amazon CloudWatch ( YAML )
      • Amazon Elasticsearch Service
      • Amazon Redshift
      • MySQL Server (Amazon RDS)
    • NoSQL Data Sources
      • MongoDB
    • OLAP
      • Data Bricks
      • Druid
      • Snowflake
    • SQL Data Sources
      • PostgreSQL
      • Microsoft SQL Server
      • MySQL Server
    • Time Series Databases
      • Prometheus Compatible
      • Elasticsearch
      • InfluxDB
    • Ascent Synthetics
      • Checks
    • Ascent Logs
      • Logs
  • INTEGRATIONS
    • Integrations Overview
      • Generating a secure ingest token
      • Data Ingest Ports
    • List of Integrations
      • Apache Beam
        • Export Metrics to Prometheus
          • Pull Mechanism via Push-Gateway
        • Export Events to Apica Ascent
      • Apica ASM
      • Apica Ascent Observability Data Collector Agent
      • AWS
        • AWS CloudWatch
        • AWS ECS
          • Forwarding AWS ECS logs to Apica Ascent using AWS FireLens
          • ECS prometheus metrics to Apica Ascent
        • AWS S3
      • Azure
        • Azure Databricks
        • Azure Eventhub
        • Azure Event Hubs
      • Docker Compose
      • Docker Swarm logging
      • Docker Syslog log driver
      • F5 Big-Ip System
      • Filebeat
      • Fluent Bit
        • Forwarding Amazon-Linux logs to Apica Ascent using Fluent Bit
        • Fluent Bit installation on Ubuntu
        • Enabling IoT(MQTT) Input (PAAS)
        • IIS Logs on Windows
      • Fluentd
      • FortiNet Firewalls
      • GCP PubSub
      • GCP Cloud Logging
      • IBM QRadar
      • ilert
      • Incident Management
        • Webhooks
      • Jaeger
      • Kafka
      • Kinesis
      • Kubernetes
      • Logstash
      • MQTT
      • Network Packets
      • OpenTelemetry
      • Object store (S3 Compatible)
      • Oracle OCI Infrastructure Audit/Logs
      • Oracle Data Integrator (ODI)
      • OSSEC Variants (OSSEC/WAZUH/ATOMIC)
        • Apica Ascent-OSSEC Agent for Windows
      • Palo Alto Firewall
      • Prometheus
        • Spring Boot
        • Prometheus on Windows
        • Prometheus Remote Write
        • MongoDB Exporter
        • JMX Exporter
      • Rsyslogd
      • Syslog
      • Syslog-ng
      • Splunk Universal Forwarder
      • Splunk Heavy Forwarder
      • SNMP
      • Splunk Forwarding Proxy
      • Vault
        • Audit Vault Logs - AWS
        • Audit Vault Logs - OCI
        • Audit Vault Metrics
    • Apica API DOCS
  • DATA MANAGEMENT
    • Data Management Overview
    • Data Explorer Overview
      • Query Builder
      • Widget
      • Alerts
      • JSON Import
      • Creating Json Schema
        • Visualization
          • Line chart
          • Bar chart
          • Area chart
          • Scatter chart
          • Status chart
          • Counter chart
          • Stat chart
          • Size chart
          • Dense Status chart
          • Honeycomb chart
          • Gauge chart
          • Pie chart
          • Disk chart
          • Table chart
          • Date time chart
      • Time-Series AI/ML
        • Anomaly Detection
        • Averaging
        • Standard Deviation(STD)
      • Data Explorer Dashboard
        • Create a Dashboard
        • Editing Dashboard
          • Dashboard level filters
    • Timestamp handling
      • Timestamp bookmark
    • Large log/events/metrics/traces
  • OBSERVE
    • Monitoring Overview
      • Connecting Prometheus
      • Connecting Amazon Managed Service for Prometheus
      • Windows Redis Monitoring
      • Writing queries
        • Query Snippets
      • Query API
      • Use Apica API to ingest JSON data
    • Distributed Tracing
      • Traces
      • Spans
      • Native support for OTEL Traces
      • Windows .NET Application Tracing
      • Linux+Java Application Tracing
    • Log Management
      • Terminology
      • Explore Logs
      • Topology
      • Apica Ascent Search Cheat Sheet
      • Share Search Results
      • Severity Metrics
      • Log2Metrics
      • Native support for OTEL Logs
      • Reports
        • Accessing Reports results via API
      • Role-Based Access Control (RBAC)
      • Configuring RBAC
    • AI and LLM Observability
      • AI Agent Deployment
      • Ascent AI Agent Monitoring
      • Ascent Quick Start Guide
    • Synthetic Check Monitoring
      • Map View
      • List View
      • Alerting for Check Results
  • Flow
    • Overview
    • Pipeline Management
      • Configuring Pipelines
      • Visualize Pipelines
      • Pipeline Overview Dashboard
      • Forwarding Data
    • OpenTelemetry Ingest
      • OpenTelemetry Logs / Traces
      • OpenTelemetry Metrics
        • Transforming Metrics through Code Rules
    • Vault
      • Certificates
      • Variables
      • Lookups
    • Rules
      • FILTER
      • EXTRACT
      • SIEM and TAG
      • REWRITE
      • CODE
      • FORWARD
        • Rename Attributes
      • STREAM
    • Functions
      • ascent.encode
      • ascent.decode
      • ascent.persist
      • Ascent.variables
      • ascent.crypto
      • Ascent.mask
      • Ascent.net
      • Ascent.text
      • Ascent.time
      • Ascent.lookups
    • List of Forwarders
    • OpenTelemetry Forwarding
      • Metrics
      • Traces
      • Logs
    • Splunk Forwarding
      • Apica UF Proxy App Extension
        • Standalone Instance
        • List of Indexer Instances
        • Indexer Discovery
      • Metric Indexes
      • Non Metric Indexes
      • Syslog Forwarding
    • Real-Time Stream Forwarding
      • AWS Kinesis
      • Azure Eventhub
      • Google Pub/Sub
    • Security Monitor Forwarding
      • Arc Sight
      • RSA New Witness
    • Forwarding to Monitoring Tools
      • Datadog Forwarding
      • New Relic Forwarding
      • Dynatrace Forwarding
      • Elasticsearch Forwarding
      • Coralogix Forwarding
      • Azure Log Analytics Forwarding
    • Object Store Forwarding
      • S3 Compatible
      • Azure Blob Storage
    • Forwarding to Data Warehouse
      • GCP Bigquery
  • Customized Forwarders
    • JS Code Forwarding
  • LAKE
    • Powered by Instastore™
  • FLEET MANAGEMENT
    • Overview
    • Agents
    • Configurations
    • Packages
    • Fleet Repository Management
    • Advanced Search
    • List of Agents
      • Datadog Agent
      • Fluent-bit Agent
      • Grafana Alloy
      • OpenTelemetry Collector
      • OpenTelemetry Kubernetes
      • Prometheus Agent
  • COMMAND LINE INTERFACE
    • apicactl Documentation
  • AUTONOMOUS INSIGHTS
    • Time Series AI-ML
      • Anomaly Detection
      • Averaging
      • Standard Deviation(STD)
      • Forecasting
      • AI-ML on PromQL Query Data Set
      • Statistical Data Description
    • Pattern-Signature (PS)
      • Log PS Explained
        • Unstructured Logs
        • Semi-structured JSON
        • Reduce Logs Based on PS
        • Log PS Use Cases
          • Log Outlier Isolation
          • Log Trending Analysis
          • Simple Log Compare
      • Config PS
        • Config JSON PS
    • ALIVE Log Visualization
      • ALIVE Pattern Signature Summary
      • ALIVE Log Compare
    • Log Explained using Generative AI
      • Configuring Generative AI Access
      • GenAI Example Using Log Explain
    • Alerts
    • Alerts (Simple/Anomaly)
    • Alerts On Logs
    • Rule Packs
    • AI-powered Search
  • PLATFORM DOCS
    • Synthetic Monitoring Overview
      • Getting Started with ASM
        • Achieving 3 Clicks to Issue Resolution via ASM
        • FAQ - Frequently Asked Questions
        • Creating A New Check
          • Creating a New Real Browser Check
      • Explore the Platform
        • API Details
        • Check Types
          • Android Check
          • Command Check
          • Compound Check
          • Browser Check
          • Desktop Application Check
          • AWS Lambda Check
          • DNS Resolver Check
          • DNS Security Check
          • Domain Availability Check
          • Domain Delegation Check
          • Domain Expiration Date Check
          • Hostname Integrity Check
          • iPad Check
          • iPhone Check
          • Ping Check
          • Port Check
          • Postman Check
          • Response Time Check
          • SSL Certificate Expiration Check
          • Scripted Check
        • Dashboards
        • Integrations
          • DynaTrace Integration
          • Google Analytics Integration
          • Akamai Integration
          • Centrify Integration
          • AppDynamics Integration
          • PagerDuty Integration
          • ServiceNow Integration
          • Splunk Integration
        • Metrics
          • Analyze Site
          • Result Values
          • Trends
          • Analyze Metrics
        • Monitoring
          • Integrating ASM Metrics into Grafana Using Apica Panels
            • Understanding the ASM Imported Dashboards
            • Using the Apica Panels Dashboards
          • Understanding ASM Check Host Locations
        • Navigation
          • Manage Menu
        • Reports
        • Use Cases
      • Configurations
        • Configuring Checks
          • Understanding Check Results
            • Understanding ZebraTester Check Results
            • Understanding Browser Check Results
            • Understanding Check Details
          • Editing Checks
            • Editing Browser Checks
            • Editing ZebraTester Checks
          • Using Regular Expressions Within the ASM Platform
          • Understanding the Edit Scenario Page
          • Comparing Selenium IDE Scripts to ASM Scenarios
          • Configuring Apica DNS Check Types
          • Implementing Tags Effectively Within ASM
          • Storing and Retrieving Information Using the ASM Dictionary
        • Configuring Users
          • Configuring SSO Within ASM
        • Configuring Alerts
          • Configuring Webhook Alerts
      • How-To Articles
        • ASM Monitoring Best Practices
        • API Monitoring Guide
        • IT Monitoring Guide
        • Monitor Mission-Critical Applications through the Eyes of Your Users
        • How To Mask Sensitive Data in ASM
        • How to Mask Sensitive Data When Using Postman Checks
        • How to Handle URL Errors in a Check
        • How To Set Up SSO Using Azure AD
        • How to Set Up SSO Using Centrify
        • ASM Scenarios How-To
          • How To Pace a Selenium Script
          • How to Utilize XPath Within a Selenium Script
          • How to Mask Sensitive Information Within an ASM Scenario
          • Handling Elements Which Do Not Appear Consistently
          • How to Handle HTML Windows in ASM Scenarios
    • ZebraTester Scripting
      • ZebraTester Overview
      • Install ZebraTester
        • Download ZebraTester
          • Core ZebraTester V7.5-A Documentation
          • Core ZebraTester V7.0-B Documentation
          • Core ZebraTester V7.0-A Documentation
          • Core ZebraTester V5.5-Z Documentation
          • Core ZebraTester V5.5-F Documentation
        • Download the ZebraTester Recorder Extension
        • Windows Installation
          • ZebraTester on Windows
          • Generate Private CA Root Certificate
          • Windows System Tuning
          • Install a new ZT version on Windows Server
          • Install/Uninstall ZT Windows Installer Silently
        • macOS Installation
          • macOS Preinstallation Instructions
          • Generate Private CA Root Cert (macOS X)
          • System Tuning (macOS)
          • Import a CA Root Certificate to an iOS device
          • Memory Configuration Guidelines for ZebraTester Agents
      • ZebraTester User Guide
        • Menu and Navigation Overview
        • 1. Get a Load Test Session
          • Recording Web Surfing Sessions with ZebraTester
            • Further Hints for Recording Web Surfing Sessions
            • Recording Extension
              • Record Web Session
              • Cookies and Cache
              • Proxy
              • Page Breaks
              • Recording Extension Introduction
              • Troubleshooting
            • Add URL to ZebraTester
            • Page Scanner
          • Next Steps after Recording a Web Surfing Session
        • 2. Scripting the Load Test Session
          • 1. Assertions - HTTP Response Verificaton
          • 2. Correlation - Dynamic Session Parameters
            • 2b. Configuring Variable Rules
            • 2a. Var Finder
          • 3. Parameterization: Input Fields, ADR and Input Files
            • ADR
          • 4. Execution Control - Inner Loops
          • 5. Execution Control - URL Loops
          • 6. Execution Control -User-Defined Transactions And Page Breaks
          • 7. Custom Scripting - Inline Scripts
          • 8. Custom Scripting - Load Test Plug-ins
            • ZebraTester Plug-in Handbooks
          • Modular Scripting Support
        • 3. Recording Session Replay
        • 4. Execute the Load Test
          • Executing a First Load Test
          • Executing Load Test Programs
            • Project Navigator
              • Configuration of the Project Navigator Main Directory
            • Real-Time Load Test Actions
            • Real-Time Error Analysis
            • Acquiring the Load Test Result
            • More Tips for Executing Load Tests
          • Distributed Load Tests
            • Exec Agents
            • Exec Agent Clusters
          • Multiple Client IP Addresses
            • Sending Email And Alerts
            • Using Multiple Client IP Addresses per Load-Releasing System
        • 5. Analyzing Results
          • Detail Results
          • Load Test Result Detail-Statistics and Diagrams
          • Enhanced HTTP Status Codes
          • Error Snapshots
          • Load Curve Diagrams
          • URL Exec Step
          • Comparison Diagrams
            • Analysis Load Test Response Time Comparison
            • Performance Overview
            • Session Failures
        • Programmatic Access to Measured Data
          • Extracting Error Snapshots
          • Extracting Performance Data
        • Web Tools
        • Advanced Topics
          • Execute a JMeter Test Plan in ZebraTester
          • Credentials Manager for ZebraTester
          • Wildcard Edition
          • Execution Plan in ZebraTester
          • Log rotation settings for ZebraTester Processes
          • Modify Session
          • Modular Scripting Support
          • Understanding Pacing
          • Integrating ZebraTester with GIT
            • GitHub Integration Manual V5.4.1
      • ZebraTester FAQ
      • ZebraTester How-to articles
        • How to Combine Multiple ZebraTester Scripts Into One
        • Inline Scripting
        • How to Configure a ZebraTester Script to Fetch Credentials from CyberArk
        • How to Configure a ZebraTester Scenario to Fetch Credentials from CyberArk
        • How to Convert a HAR file into a ZebraTester Script
        • How to Convert a LoadRunner Script to ZebraTester
        • How to Import the ZT Root Certificate to an iOS device
        • How to iterate over JSON objects in ZebraTester using Inline Scripts
        • How to round a number to a certain number of decimal points within a ZebraTester Inline Script
        • How to Use a Custom DNS Host File Within a ZebraTester Script
        • How to Move a ZebraTester Script to an Older Format
        • API Plugin Version
        • Setting up the Memu Player for ZebraTester Recording
        • Inline Script Version
      • Apica Data Repository (ADR) aka Apica Table Server
        • ADR related inline functions available in ZT
        • Apica Data Repository Release Notes
        • REST Endpoint Examples
        • Accessing the ADR with Inline Scripts
      • ZebraTester Plugin Repository
      • Apica YAML
        • Installing and Using the ApicaYAML CLI Tool
        • Understanding ApicaYAML Scripting and Syntax
    • Load Testing Overview
      • Getting Started with ALT
      • Creating / Running a Single Load Test
      • Running Multiple Tests Concurrently
      • Understanding Loadtest Results
    • Test Data Orchestrator (TDO)
      • Technical Guides
        • Hardware / Environment Requirements
        • IP Forwarding Instructions (Linux)
        • Self-Signed Certificate
        • Windows Server Install
        • Linux Server Install
        • User Maintenance
        • LDAP Setup
        • MongoDB Community Server Setup
        • TDX Installation Guide
      • User Documentation
        • End User Guide for TDO
          • Connecting to Orson
          • Coverage Sets and Business Rules
          • Data Assembly
          • Downloading Data
        • User Guide for TDX
          • Connecting to TDX
          • Setting up a Data Profile
          • Extracting Data
          • Analyzing Data Patterns
          • Performing Table Updates
        • API Guide
          • API Structure and Usage
          • Determining Attribute APIs
            • Create Determining Attribute (Range-based)
            • Create Determining Attribute (Value-based)
            • Update Determining Attributes
            • Get Determining Attribute Details
            • Delete a Determining Attribute
          • Coverage Set API’s
            • Create Coverage Set
            • Update Coverage Set
            • Get All Coverage Set Details
            • Get Single Coverage Set Details
            • Lock Coverage Set
            • Unlock Coverage Set
            • Delete Coverage Set
          • Business Rule API’s
            • Create Business Rule
            • Update Business Rule
            • Get Business Rule Details
            • Get All Business Rules
            • Delete Business Rule
          • Workset API's
            • Create Workset
            • Update Workset
            • Get All Worksets
            • Get Workset Details
            • Unlock Workset
            • Clone Workset
            • Delete Workset
          • Data Assembly API's
            • Assemble Data
            • Check Assembly Process
          • Data Movement API's
            • Ingest (Upload) Data Files
            • Download Data Files
              • HTML Download
              • CSV Download
              • Comma Delimited with Sequence Numbers Download
              • Pipe Delimited Download
              • Tab Delimited with Sequence Numbers Download
              • EDI X12 834 Download
              • SQL Lite db Download
              • Alight File Format Download
          • Reporting API's
            • Session Events
            • Rules Events
            • Coverage Events
            • Retrieve Data Block Contents
            • Data Assembly Summary
        • Workflow Guide
      • Release Notes
        • Build 1.0.2.0-20250213-1458
  • IRONdb
    • Getting Started
      • Installation
      • Configuration
      • Cluster Sizing
      • Command Line Options
      • ZFS Guide
    • Administration
      • Activity Tracking
      • Compacting Numeric Rollups
      • Migrating To A New Cluster
      • Monitoring
      • Operations
      • Rebuilding IRONdb Nodes
      • Resizing Clusters
    • API
      • API Specs
      • Data Deletion
      • Data Retrieval
      • Data Submission
      • Rebalance
      • State and Topology
    • Integrations
      • Graphite
      • Prometheus
      • OpenTSDB
    • Tools
      • Grafana Data Source
      • Graphite Plugin
      • IRONdb Relay
      • IRONdb Relay Release Notes
    • Metric Names and Tags
    • Release Notes
    • Archived Release Notes
  • Administration
    • E-Mail Configuration
    • Single Sign-On with SAML
    • Port Management
    • Audit Trail
      • Events Trail
      • Alerts Trail
Powered by GitBook
On this page
  • Changes in 1.5.0
  • Changes in 1.4.0
  • Changes in 1.3.0
  • Changes in 1.2.1
  • Changes in 1.2.0
  • Changes in 1.1.0
  • Changes in 1.0.1
  • Changes in 1.0.0
  • Older Releases

Was this helpful?

Edit on GitHub
Export as PDF
  1. IRONdb

Release Notes

Changes in 1.5.0

2025-05-08

  • Avoid excess CPU usage during replication when a node is in journal-only mode.

  • Fix issue with stalling in journal-only mode due to check tag replication.

  • Remove unneeded checks for flatbuffer availability in journaling. Non-flatbuffer journals were removed long ago.

  • Fix journal-only mode so that it exits when the replication journals are fully drained.

  • Disable the graphite, prometheus, opentsdb, check tag replication, and monitor modules when running in reconstitute or journal-only modes.

  • Remove Pickle support from Graphite module.

  • Reduce error log volume on text reconstitute errors.

  • Fix issue where the opentsdb and prometheus modules were incorrectly reporting that the toplogy was wrong when attempting to load data onto a node that is reconstituting. Nodes will now report that the service is unavailable.

  • Improve libsnowth listener error to include the IP address of the remote side of the socket.

  • Accelerate NNTBS reconstitute by avoiding memory copies.

  • Reduce unnecessary NNTBS data copies during reconstitute to improve speed.

  • Write NNTBS data out in parallel during reconstitute to improve speed.

  • Add parameter to text reconstitute data fetching to allow excluding data older than the provided time.

  • Improve text reconstitute performance by doing fewer surrogate id lookups.

  • Deprecate configuration parameters reconstitute/nntbs@batchsize andreconstitute/nnt@batchsize. Use reconstitute/nntbs@lmdb_commit_batch_size instead.

  • Add new flag, reject_data_older_than, to the <text> config stanza that will disallow text data older than the provided time from being ingested, whether by standard ingestion or reconstitute.

  • Improve NNTBS and histogram rollup reconstitute performance by no longer fetching shards that fall outside of any configured retention window.

  • Inhibit retention-related deletion of NNTBS and histogram rollup shards while a reconstitute is in progress.

  • Add error handling of invalid flatbuffer records found while replaying metric index journals. Optionally these invalid records can be saved to files. The number of replay errors is now tracked in the /state API.

  • Treat MDB_NOTFOUND errors returned from LMDB transaction puts as corruption.

  • Only flush column families in the raw database that finish rolling up. Previously all column families were flushed unconditionally, including those that had not finished rolling up or had long since finished.

  • Fix snowthsurrogatecontrol use-after-free error.

Changes in 1.4.0

2024-11-05

  • Fix use after free bug that could occasionally happen due to a race when fetching raw data.

  • Fix potential memory leak on certain oil/activity data operations.

  • Fix fetch bug where C-style calloc allocations were being mixed with C++-style deletes.

  • Add new paramter to whisper config, end_epoch_time, that takes an epoch timestamp and directs the code to not look in whisper files if the fetch window starts after this time.

  • Fix bug where histogram ingestion data was not being sent properly during rebalance operations.

  • Fix bug where histogram rollup data was not being reconstituted during reconstitute operations.

  • Add get_engine parameter to histogram data retrieval to allow pulling from either rollups or ingestion data.

  • No longer open new LMDB transactions when reading data for merging NNTBS blocks together.

  • Remove all support for legacy, non-sharded histograms.

  • Fix bug where if a raw shard rollup was aborted after being scheduled but before actually starting, multiple rollups could end up triggering at once.

  • Fix rename bug where type was not getting set when failing to send NNTBS data.

  • Add header to the /rename endpoint to X-Snowth-Activity-Data-Mode, which can accept either use_existing or create_new as values.

  • Treat MDB_CORRUPTED, MDB_PAGE_FULL, MDB_TXN_FULL, MDB_BAD_TXN, and ENOMEM as LMDB corruption consistently when checking for errors.

  • When using the /merge/nntbs endpoint to send data to a node, allow either updating the receiving node's activity data using the incoming NNTBS data or leaving it as is and not updating it.

  • Fix bug where activity data was not being updated correctly when inserting NNTBS data.

  • Fix bug where rollups were marked clean after a rollup had been kicked off asynchronously, resulting in a race that could lead to shards being incorrectly considered dirty.

  • Deprecate support for rebalancing data into a cluster with fewer NNTBS periods.

  • The /rename endpoint will now detect when it gets a 500 error from the /merge/nntbs endpoint and will return an error instead of spinning forever.

  • The /merge/nntbs endpoint will no longer crash on detecting corrupt shards; it will offline the shards and return errors.

  • Various small fixes to reduce memory consumption, improve performance, and prevent possible crashes or memory corruption.

Changes in 1.3.0

2024-07-17

  • Fix bug in build_level_index where we were invoking a hook that called pcre_exec with an uninitialized metric length.

  • Reduce spam in error log when trying to fetch raw data for a metric and there isn't any for the requested range.

  • Add new API endpoint, /rename, to allow renaming a metric. This calculates where the new metric will live, sends the data for the metric to the new location, then deletes the old metric. This only works for numeric metrics.

  • Add new API endpoint, /full/canonical/<check uuid>/<canonical metric name> that will allow deleting an exact metric from the system without using tag search.

  • Add ability to skip data after a given time when using the copy sieve in snowth_lmdb_tool.

Changes in 1.2.1

2024-06-04

  • Avoid metric index corruption by using pread(2) in jlog instead of mmap(2).

  • Deprecate max_ingest_age from the graphite module. Require the validation fields instead.

  • Change Prometheus module to convert nan and inf records to null.

  • Add logging when when the snowth_lmdb_tool copy operation successfully completes.

  • Fix bug where a node could crash if we closed a raw shard for delete, then tried to roll up another shard before the delete ran.

  • Fix bug where setting raw shard granularity values above 3w could cause data to get written with incorrect timestamps during rollups.

  • Improve various listener error messages.

  • Add checks for timeouts in the data journal path where they were missing.

  • Improve graphite PUT error messages.

  • Fix NNTBS rollup fetch bug where we could return no value when there was valid data to return.

  • Fix bug where histogram rollup shards were sometimes not being deleted even though they were past the retention window.

Changes in 1.2.0

2024-03-27

NOTE: This release bumps the metric index version from 4 to 5. Upon restart, new indexes will be built and the old ones will be deleted. This process will use a significant amount of memory while the indexes are being rebuilt. It will also cause the first post-update boot to take longer than usual.

  • Update index version from 4 to 5.

  • Automatically clean up old index versions on startup to make sure outdated indexes don't clog the disk.

  • Fix Ubuntu 20.04 specific bug where nodes could crash when trying to clean up status files when rolling up raw shards.

  • Fix issue with level indexes where data was being lost when deleting metrics on levels where the metric has multiple tags.

  • Fix issue where level indexes were incorrectly reporting that levels existed when all underlying metrics had been removed.

  • Add new API endpoints, /compact_indexes and /invalidate_index_cache, that allow forcing compaction and cache invalidation for specific accounts, respectively.

  • Fix rollup bug where raw shards could be prematurely deleted if a rollup was aborted due to corruption.

  • Fix various potential memory corruption issues.

  • Fix issue where jlog journal data could get corrupted.

Changes in 1.1.0

2024-01-02

  • Fix bug where rollups were being flagged "not in progress" and "not dirty" when attempting to schedule a rollup and the rollup is already running.

  • Use activity ranges as part of query cache key. Previously, cached results from queries with a time range could be used to answer queries that had no time range, leading to incorrect results.

  • Fix logic bug where rollups were sometimes flagged as still being in progress after they were completed.

  • Account index WAL can keep growing without bounds due to a local variable value being squashed early.

  • Fix bug where the reconst_in_progress file was not being cleaned up after reconstitute operations, which could block rollups and deletes from running.

  • The raw/rollup and histogram_raw/rollup API endpoints will no longer block if there is a rollup already running. They will also return sensible error messages.

  • Raw shard rollups will not be allowed to run unless all previous rollups have run at least once.

  • Fix bug where deferred rollups could cause the rollup process to lock up until the node is restarted.

  • Add REST endpoint POST/PUT /histogram/<period>/<check_uuid>/<metric_name>?num_records=<x> which can be used with a json payload to directly insert histogram shard metric data for repair purposes.

  • Added configuration file field, //reconstitute/@max_simultaneous_nodes, that will cause the reconstituting node to only hit a predetermined number of peer nodes at once. The default if not specified is "all peers". This setting can be used if a reconstitute places too much load on the rest of the cluster, causing degradation of service.

  • Disallow starting single-shard reconstitutes with merge enabled if the shard exists and is flagged corrupt.

  • Improve NNTBS error messages if unable to open a shard.

  • PromQL - improve error messages on invalid or unsupported range queries.

  • PromQL - fix range nested inside one or more instant functions.

  • Include maintenance mode when pulling lists of raw, histogram, or histogram rollup shards.

  • Use read-copy-update (RCU) for Graphite level indexes and the surrogate database. It allows more queries to run concurrently without affecting ingestion, and vice versa.

  • Defer rollups of raw shards if there is a rollup shard in maintenance that the raw shard would write to.

  • Reject live shard reconstitute requests on NNTBS or histogram rollup shards if there is a raw shard rollup in progress that would feed into them.

  • Fix bug where the system would report that a live reconstitute was not in progress, even when one was running.

  • Allow running single-shard or single-metric reconstitute on non-raw shards, even if the shard extends beyond the current time.

  • The reconstitute GUI no longer apppears when doing online reconstitutes.

  • Fix iteration bug when reconstituting NNTBS shards.

  • Added the merge_all_nodes flag to live reconstitute which causes all available and non-blacklisted write copies to send metric data instead of only the "primary" available node.

  • Added the ability to repair a local database by reconstituting a single metric stream.

  • Fix bug where /fetch would not proxy if the data for a time period was all in the raw database, but the relevant raw shards were offline.

Changes in 1.0.1

2023-09-06

NOTE: This version updates RocksDB (raw database, histogram shards) from version 6 to 7. It is not possible to revert a node to a previous version once this version has been installed.

  • Add a new configuration parameter, //rest/@default_connect_timeout, that allows configuring the connect timeout for inter-node communication. This was previously hardcoded to 3 seconds.

  • Graphite series and series_multi fetches now return 500 when there are no results and at least one node had an issue returning data.

  • Graphite series and series_multi fetches now return 200 with an empty results set on no data rather than a 404.

  • Fix bug on /find proxy calls where activity ranges were being set incorrectly.

  • Add ability to filter using multiple account IDs in the snowthsurrogatecontrol tool by providing the -a flag multiple times.

  • Reduce usage of rollup debug log to avoid log spam.

  • Upgrade RocksDB from version 6.20.3 to version 7.9.2.

Changes in 1.0.0

2023-07-28

  • Prevent ingestion stalls by setting a better eventer mask on socket read errors.

  • Fix bug where surrogate deletes running at the same time as level index compactions can cause crashes.

  • Improve scalability of lock protecting all_surrogates in indexes.

  • Fix logging of old-data ingestion.

  • Don't stop a rollup in progress if new data is received; finish and reroll later.

  • Add ability to filter by account ID in the snowthsurrogatecontrol utility by using the -a flag.

  • Fix full-delete crash on malformed tag query.

  • Rewrite Graphite level-index query cache to remove race on lookup.

  • Remove surrogate2.

  • Some data was hidden post arts compaction, make sure it stays visible.

  • Fix bug where if a fetch to the /raw endpoint exceeded the raw streaming limit (10,000 by default), the fetch would hang.

  • Reduce memory usage during extremely large /raw fetches.

  • Fix bug where extremely large double values generated invalid JSON on /raw data fetches.

  • Handle surrogate duplicates on migration from s2 to s3.

  • Require all nodes for active_count queries.

  • Add back-pressure to raw puts, allows the database to shed load by returning HTTP 503.

Older Releases

PreviousMetric Names and TagsNextArchived Release Notes

Last updated 5 days ago

Was this helpful?

NOTE: This release deprecates legacy histograms. must be configured before upgrading to this release. If this is not done, nodes may not start up after the upgrade.

Add preliminary support for operating IRONdb clusters with SSL/TLS. This allows securing ingestion, querying, and intra-cluster replication. See for details. This feature should be considered alpha.

IMPORTANT NOTE: Any node running 0.23.7 or earlier MUST do a PRIOR to upgrading to 1.0.0. This is due to removal of support for the previous surrogate database format.

For older release notes, see .

Histogram shards
libmtev 2.7.1
TLS Configuration
libmtev 2.5.3
surrogate3 migration
libmtev 2.5.2
Archived Release Notes