# Pull Mechanism via Push-Gateway

You can use the [Connector](https://github.com/ApicaSystem/hauler-apache-beam-io-connector) provided by Apica Ascent via GitHub to push Apache Beam metrics to Push-Gateway.

## Setting up Push-Gateway via Docker (recommended)

In order to set up push-gateway, just run the provided docker image.

```bash
docker pull prom/pushgateway
docker run -d -p 9091:9091 prom/pushgateway
```

You'll now have an instance of push-gateway running on your machine, you can verify by running the below command.

```bash
docker ps
```

Once the instance is up and running, we can then specify it in our prometheus.yaml config file.

View the sample prometheus.yaml file below.

```yaml
scrape_configs:
  - job_name: 'pushgateaway'
    scheme: http
    static_configs:
    - targets: ['localhost:9091']
```

Great, now you will have Prometheus scraping the metrics from the given PushGateway endpoint.

## Setting up Apache Beam to export the Metrics to Push-Gateway

Now that you have configured the push-gateway and Prometheus, it's time that we start configuring the Apache Beam Pipeline to export the metrics to the Push-Gateway instance.

For this, we will refer to the tests written in the [Connector](https://github.com/ApicaSystem/hauler-apache-beam-io-connector) here.

```
package logiqio

import org.apache.beam.sdk.Pipeline
import org.apache.beam.sdk.transforms.Create
import org.apache.beam.sdk.metrics.Metrics
import org.apache.beam.sdk.transforms.DoFn
import org.apache.beam.sdk.transforms.ParDo
import kotlin.test.Test

class ApplyMetrics : DoFn<LogiqEvent, LogiqEvent>() {
    private var counter = Metrics.counter("Pipeline Metrics", "logiq_events_processed");

    @ProcessElement
    fun processElement() {
        counter.inc()
    }
}

class LibraryTest {
       @Test fun metrics() {
        val pipeline = Pipeline.create()

        val elems = List(1029) {
            LogiqEvent("ns$it", "$it Events occurred", it, "host-$it", "process-$it", "app-$it", "cos$it")
        }

        pipeline
            .apply("Create", Create.of(elems))
            .apply(ParDo.of(ApplyMetrics()))

        val result = pipeline.run()
        val metrics = result.metrics()

        PushGateway("http://localhost:9091/metrics/job/test").write(metrics)
    }
}

```

The `metrics()` method is responsible for sending the metrics to the given push-gateway endpoint. Once the pipeline has been modeled, we are good to view the result. we should be able to access the metrics of the PipelineResult at `PipelineResult.metrics`, now just pass this to the Push-Gateway class with the correct endpoint and call the `write()` method with the metrics.

Hooray, you have successfully pushed your Apache Beam Metrics to Push-Gateway. These metrics will shortly be scraped by Prometheus and you would be able to access them.

You can check your results on Push-gateway Instance and Prometheus Instance.

### Advanced

If you want to apply any transformations other than the default transformers, you can specify the functions with `withCounterTransformer, withDistributionTransformer, withGaugeTransformer` provided by the PushGateway class. This allows you to perform complex operations and achieve granularity within your metrics.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.apica.io/integrations/list-of-integrations/apache-beam/export-metrics-to-prometheus/pull-mechanism-via-push-gateway.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.