> For the complete documentation index, see [llms.txt](https://docs.apica.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.apica.io/flow/venn-and-pipeline-build.md). # Venn and Pipeline build ### 1. What is Venn? Venn answers questions about your logs, metrics, traces, and anomalies — and can build pipeline rules for you from plain-language requests. **How to open Venn:** * **Floating bubble** — click the bubble in the corner of any page (tooltip **"Ask Venn"**). It opens the Venn drawer. * **Drawer** — a panel slides in from the right, titled **Venn**. You can pin it, expand it wider, start a **New chat**, or close it. * **Full page** — click on the Venn menu option from the top navbar for a full-screen chat. When the chat is empty you'll see **"Ask Venn anything"** with a few starter prompts. Type your question and press **Enter** to send (**Shift+Enter** for a new line).

*** ### 2. Building pipeline rules with Venn You do not need to know which rule type to choose. Describe your goal, and Venn determines the best option. **The flow:** 1. **Ask in plain language.** For example, *"Help me cut log volume from the payment app"* or *"Redact user name and IP address from access-logs."* 2. **Venn recommends and drafts.** If multiple namespaces or applications are available, Venn asks you to select them, then retrieves logs from the last hour, analyzes them, and drafts suitable rules. 3. **Review the suggestion card.** The card is headed **"Pipeline suggestion ready:"**. It shows the drafted rules and a short summary. Redaction suggestions also show a **"Redaction preview (before → after)"**. 4. **Click "Review."** The chat drawer closes and the **pipeline builder opens pre-filled** with the suggested rules. The **preview runs against real dataflow logs** so you can see the rules' effect before saving: 1. For **Forward** suggestions, the namespace/application Venn analyzed is **pre-selected** in the preview's Data Flows dropdowns and the preview runs against that flow automatically. 2. For other kinds (filter, rewrite, code, SIEM…), select the namespace and application on the **IN** tab of the **Pipeline Preview** section — the preview runs against whichever dataflow is selected. This selection only provides logs for the preview; it does not associate the pipeline with the dataflow. 5. **Validate with the DIFF tab.** Select the **DIFF** tab in **Pipeline Preview** to check the rule changes, using the left and right arrows to step through the previous and next logs. 6. **Save, then apply.** When you are satisfied with the preview, click **Save**. Then use **Apply Pipeline** to associate the pipeline with the namespace and application. The rules process incoming data only after you apply the pipeline. > **Nothing is saved automatically.** Suggested rules arrive as **draft** rules for you to review. Your own hand-authored rules are always kept — Venn's additions are grouped under headings like *Suggested Filters*, *Suggested Rewrites*, so you can tell them apart. Saving happens only when **you** press Save in the builder. **If a name is already taken:** when a suggested new pipeline's name already exists, Venn renames it and shows a banner (e.g. *A pipeline named "X" already exists, so the suggestion was named "Y". Edit the name above before saving…*). Change the name before saving if you'd like. **Don't want a suggestion?** Click **Dismiss** on the card. You'll be asked to confirm — dismissing discards the suggestion, and you'd need to ask Venn to regenerate it. #### Example: cut log volume from the payment app Ask: *"Help me cut log volume from the payment app."*

If multiple namespaces or applications are available, Venn asks you to select them. After you select a namespace and application, Venn retrieves logs from the last hour, analyzes them, and drafts pipeline rules to filter them.

Choose a suggestion, such as *"Suppress payment logs — keep 5 per minute, drop the rest."*

Venn may ask for a pipeline name. Choose a suggested name or enter your own.

After you provide a name, Venn presents the proposed pipeline and its rules.

Click **Review** to open the pipeline builder with the drafted rules. In the **Pipeline Preview** section, select the namespace and application on the **IN** tab. This selection provides logs for the preview. It does not associate the pipeline with the dataflow.

Select the **DIFF** tab in **Pipeline Preview** to validate the rule changes.

Use the left and right arrows to view the previous and next logs. This view shows which logs the pipeline filters. The following screenshot shows a filtered log.

When you are satisfied with the preview, click **Save**. Then use **Apply Pipeline** to associate the pipeline with the namespace and application. The rules process incoming data only after you apply the pipeline. #### Example: redact user name from access-logs Ask: *"Redact user name from access-logs."*

The suggested pipeline with the redaction review of the logs is shown below.

Click the **Review** button.

Use the **DIFF** view to validate the rule's output.

*** ### 3. The rule kinds Rules are added from the **Add Rule** menu in the builder. Ask Venn for any of these in plain language, or add them yourself.

#### Filter * **What it does:** Excludes whole events, or strips low-value labels off an event, to save storage and processing. * **When to use:** Cut ingest volume and cost by dropping noise. * **Ask Venn:** *"Drop debug-level logs from the checkout service."* #### Rewrite * **What it does:** Masks or replaces sensitive information using pattern matching — mainly PII redaction. * **When to use:** Redact emails, tokens, card numbers, etc. * **⚠️ Caveat:** Redaction is **permanent** — the original value is not recoverable once masked. Check the before → after preview first. * **Ask Venn:** *"Mask all email addresses in the auth logs."* #### Code * **What it does:** Runs a small custom script on each event, for logic a filter or rewrite can't express (parse, transform, enrich). * **When to use:** Advanced, one-off transformations. * **Ask Venn:** *"Write a rule that adds a field marking events over 500ms as slow."* #### Extract * **What it does:** Parses unstructured log text into structured fields using a pattern with named captures — each captured piece becomes its own field. * **When to use:** Turn raw log lines into queryable fields. * **Ask Venn:** *"Extract the status code and latency from my nginx logs."* #### Aggregate * **What it does:** Applies statistical functions (count, sum, average, min, max, median, percentiles, distinct…) over a time window. The result can be added back onto the event or turned into a metric. * **When to use:** Summarize high-volume events into counts or rates. * **Ask Venn:** *"Count errors per service every minute."* #### Forward * **What it does:** Chooses which events (and which fields) are forwarded to downstream destinations, to lower EPS and cost. The original is retained in storage. * **⚠️ Caveat:** Forward rules **cannot be mixed** with other rule types — a forward rule needs its **own pipeline**. * **Ask Venn:** *"Forward only error logs to the SIEM destination."* #### Stream * **What it does:** Redirects incoming logs by changing their Namespace and Application. By default it **copies** the events; if the originals are dropped it effectively **moves** them. * **When to use:** Route a subset of logs to a different namespace/app. * **Ask Venn:** *"Send audit logs to the security namespace."* #### SIEM * **What it does:** Matches known security signatures/patterns and turns matches into a real-time metric. Can optionally add a dashboard widget and an alert. * **⚠️ Note:** Any alert it creates starts **disabled** — enable it when you're ready. * **Ask Venn:** *"Detect failed SSH login attempts."* #### Tag * **What it does:** Identifies interesting events based on evaluation criteria and tracks them in real time. * **When to use:** Flag and follow events matching a condition. * **Ask Venn:** *"Tag every event mentioning 'OutOfMemory'."* > A pipeline with no rules shows: *"No rules have been configured for this pipeline. Select '+ Add Rule' to configure new rule."* *** ### 4. Editing an existing pipeline **Refine a suggestion:** with the builder open, just ask Venn again — *"make the filter also drop info logs"* — and the rules **refresh live** in the builder, replacing the previous AI suggestion. Your own rules stay put. **Structural changes** (without regenerating rule content) — Venn shows a card for each: | You ask | Card label | What happens | | ------------------------------------------ | ------------------- | ------------------------ | | Change rule order | **Reorder rules** | Execution order updated | | Remove rules | **Delete rules** | Named rules removed | | Rename the pipeline / edit its description | **Rename pipeline** | Name/description updated | Click **Review** on the card to apply it. These act on the pipeline **currently open in the builder** — if it isn't open you'll see *"Open the pipeline in the builder to apply this change."* Once applied, the card reads **"Applied to the pipeline."** All rule kinds are preserved — nothing gets regenerated or lost. #### Example: reorder the pipeline rules — swap rule #1 with rule #3

After reordering the rules:

*** ### 5. Deleting rules — you approve, always Deleting rules needs **your explicit approval**. Venn cannot approve a deletion on its own — you confirm it before anything is removed. This is a safety guard so an AI request can never quietly drop rules you meant to keep. #### Example: delete rule #3 from the pipeline