Data Flow Pipelines

The pipeline is a series of processes or stages through which data flow systematically and efficiently. It helps to visualize the flow between nodes, rules, and filters applied for the data flow.

Creating a Pipeline

Click on the Explore Option from the top menu and click on Pipelines

Navigating to Pipeline Page

Hover on the Actions and click on Create Pipeline

Actions button to create Pipeline

Enter the Pipeline name to create your pipeline.

Creating a Pipeline

Created pipeline displayed in the pipeline list

Configuring Pipeline

Once the Pipeline is created we will configure the pipeline with Rules based on which the events in Data flow will be determined.

Configure Pipeline

Hover on Add Rule and select the Rule you want to set (CODE, EXTRACT, FILTER, REWRITE, SIEM, STREAM, TAG) and once done clicking on Save button will save the configuration.

Pipeline Configuration View

You can also preview how the pipeline will be executed using the Preview option.

There are three ways to preview:

Use Sample Logs:

Sample Logs, Dropdown helps select the sample logs

Preview based on the Rules set

Dropped Alerts highlighted and displayed as Dropped

On clicking on the logs the expected changes are displayed

Use Data Flows:

After selecting the Time Range, Namespace and Application Preview can be seen on clicking the Preview

Dropped Alerts highlighted and displayed as Dropped

On clicking on the logs the expected changes are displayed

Use GenAI:

Click on Generate Logs

Enter the prompt and click on the Arrow, Sample log will be generated

Click on the Use Logs to use the same logs

Sample logs are added and preview can be seen on clicking Preview Button

Preview based on the rules configured is displayed

On clicking on the logs the expected changes are displayed

Apply Pipeline

Using the Apply Pipeline option we will be able to apply the pipeline to multiple Dataflow.

Apply Pipeline to associate the Namespace and Application to it

Select the time range and then select Namespace and Application (Dataflow) to which you want to apply the pipeline. When a namespace and application have other Pipelines linked to it they will be displayed as well, the new one we are trying to associate will be displayed at the bottom and outlined by Green. User will be able to reorder by dragging.

New pipeline added outlined by green

Oder can be set by dragging the pipelines.

Once desired order has been set Click on Apply to apply it to the Dataflow. Post this the execution will be in the order of the Pipeline set.

Pipeline View:

All the Dataflows linked are displayed under the Pipeline and can be accessed using the Arrow icon.

All the linked Dataflow are displayed under the Pipeline. Along with the stats.

The Stats displays the Events Ingested, Events Processed and Saved Bytes:

• Green Indicates the total Events Ingested.

• Orange Indicates the Events Processed.

• Red Indicates the Saved Bytes.

Pipelines stats is the total amount of Events and Saved Bytes and the data for each of the associated Dataflow is displayed against the individual Dataflows.