Setting Up Network Flow Logs on an On-Prem Netmaker Installation

Overview

Network Flow Logs in Netmaker provide visibility into traffic traversing your overlay network. They allow you to see:

• Which nodes are communicating
• Source and destination IPs
• Source and destination ports
• Protocol and direction
• Byte and packet counts
• Flow start and end timestamps

This helps with:

• Security auditing
• Network troubleshooting
• Access validation
• Visibility into overlay traffic patterns

Note: Flow Logs is a Pro / Enterprise feature and requires an appropriate license.

---

Architecture Summary

Flow Logs on Netmaker rely on the following components:

• Netmaker Server
• Netmaker Exporter
• ClickHouse (for storing flow log records)

At a high level:

Netclient / Devices → Netmaker Exporter → ClickHouse → Netmaker UI

---

Prerequisites

Before enabling Flow Logs, ensure the following are available:

Minimum Infrastructure

Recommended minimum for the monitoring stack:

• 2 GB RAM
• 2 vCPUs

If you are running this in production or collecting logs from many devices, allocate more resources accordingly.

---

Step 1 — Enable the Monitoring Stack

Flow Logs depend on the Netmaker monitoring components.

If Netmaker is already installed, re-run the installer with monitoring enabled:

sudo ./nm-quick.sh -m

If this is a fresh install:

sudo ./nm-quick.sh -p -m

This enables:

• Prometheus
• Grafana
• Netmaker Exporter

Flow Logs will not function correctly unless the exporter and monitoring stack are enabled.

---

Step 2 — Enable the GRPC Endpoint in Caddy

Flow Logs requires the gRPC endpoint to be reachable.

Edit the Caddyfile

Open your Netmaker Caddyfile (typically located at):

/root/netmaker/Caddyfile

Locate the gRPC block and ensure it is enabled:

# GRPC
https://grpc.{$NM_DOMAIN} {
    reverse_proxy netmaker-exporter:50051 {
        transport http {
            versions h2c
        }
    }
}

Reload Caddy

After saving the file, reload Caddy:

docker exec caddy caddy reload --config /etc/caddy/Caddyfile

---

Verify gRPC Connectivity

Once enabled, make sure your gRPC subdomain resolves correctly and is reachable.

Example:

nslookup grpc.yourdomain.com

If the gRPC endpoint is not working, Flow Logs will not ingest traffic properly.

---

Step 3 — Set Up ClickHouse

Netmaker stores Flow Logs in ClickHouse.

You have two options:

---

Option A — Use the Bundled ClickHouse Container (Recommended for Quick Setup)

If you are using the default Netmaker Docker deployment, ClickHouse is already referenced in the override file, but may be commented out.

Edit Docker Compose Override

Open:

/root/netmaker/docker-compose.override.yml

Uncomment:

• the clickhouse service
• the depends_on line under netmaker-exporter
• the ClickHouse volumes

Example:

services:
  netmaker-exporter:
    depends_on:
      - netmaker
      - clickhouse

  clickhouse:
    container_name: clickhouse
    image: clickhouse/clickhouse-server:latest
    restart: unless-stopped
    ports:
      - "8123:8123"
      - "9000:9000"
    volumes:
      - clickhouse_data:/var/lib/clickhouse
      - clickhouse_logs:/var/log/clickhouse-server

volumes:
  clickhouse_data: {}
  clickhouse_logs: {}

For on-prem deployments, this is usually the easiest path for first-time enablement.

---

Option B — Use an External ClickHouse Instance

You may instead use:

• A separate self-hosted ClickHouse server
• A managed ClickHouse service

This is often preferable for:

• production-scale flow retention
• higher availability
• centralised logging architectures

If using external ClickHouse, ensure:

• It is reachable from the Netmaker host
• Firewall rules allow access
• Credentials are configured properly

---

Step 4 — Create the ClickHouse Database

If your ClickHouse instance is fresh, create a dedicated database for Flow Logs.

Connect to ClickHouse

If using the bundled container:

docker exec -it clickhouse clickhouse-client

If external:

clickhouse-client --host <clickhouse-host> --port 9000 --user <user> --password

Create the database

CREATE DATABASE IF NOT EXISTS netmaker_logs ENGINE = Atomic;

Verify:

SHOW DATABASES;

You should see:

netmaker_logs

Using a dedicated database keeps Flow Logs separate from other telemetry or analytics workloads.

---

Step 5 — Configure Netmaker Environment Variables

Edit your Netmaker environment file:

/root/netmaker/netmaker.env

Add or update the following:

CLICKHOUSE_HOST=<your-clickhouse-host>
CLICKHOUSE_PORT=<your-clickhouse-port>
CLICKHOUSE_DB=netmaker_logs
CLICKHOUSE_USER=<username>
CLICKHOUSE_PASS=<password>

Example (Bundled ClickHouse)

CLICKHOUSE_HOST=clickhouse
CLICKHOUSE_PORT=9000
CLICKHOUSE_DB=netmaker_logs
CLICKHOUSE_USER=default
CLICKHOUSE_PASS=yourpassword

Example (External ClickHouse)

CLICKHOUSE_HOST=10.0.20.5
CLICKHOUSE_PORT=9000
CLICKHOUSE_DB=netmaker_logs
CLICKHOUSE_USER=netmaker
CLICKHOUSE_PASS=strongpassword

All five CLICKHOUSE_* values must be set correctly or Flow Logs will not appear.

---

Step 6 — Restart Netmaker Services

After saving the configuration, restart the Netmaker stack:

cd /root/netmaker
docker compose up -d --force-recreate

This recreates containers with the updated ClickHouse configuration.

---

Step 7 — Enable Flow Logs in the Netmaker Dashboard

Once the backend prerequisites are complete:

In the UI

Navigate to:

Settings → Monitoring & Debugging

or directly:

https://<your-dashboard-url>/settings?tab=monitoring-debugging

Then:

1. Toggle Traffic Logs to Enabled
2. Optionally toggle Force Enable

What “Force Enable” Does

If enabled, Flow Logs will be turned on for all registered devices automatically.

This is useful for:

• smaller environments
• troubleshooting sessions
• full network visibility

If disabled, logging can be controlled per device.

---

Step 8 — Optional: Enable Flow Logs Per Device

If you want to log only selected devices:

1. Go to All Devices
2. Select a device
3. Click Edit
4. Toggle Enable Traffic Logs

This is useful if you only want logs from:

• gateways
• relay nodes
• jump hosts
• sensitive workloads

---

What You Will See in Flow Logs

Each log record can include:

• Source IP
• Destination IP
• Source port
• Destination port
• Protocol
• Direction
• Packet count
• Byte count
• Start time
• End time

Netmaker also attempts to enrich logs with:

• Node names
• External client identity
• Known domain names (best-effort)

This makes logs more readable than raw IP-only flow records.

---

Verification Checklist

Once enabled, verify the following:

1) Exporter is running

docker ps | grep netmaker-exporter

2) ClickHouse is reachable

If bundled:

docker exec -it netmaker-exporter sh
nc -vz clickhouse 9000

If external:

nc -vz <clickhouse-host> <clickhouse-port>

3) gRPC endpoint is active

Check your reverse proxy logs and confirm the exporter is reachable on port 50051.

4) Dashboard toggle is visible

If the Traffic Logs toggle is missing, check your license.

---

Troubleshooting

Issue: Flow Logs toggle not visible

Possible cause:

• License does not include Flow Logs

Action:

Verify your Pro / Enterprise license.

---

Issue: Logs are not appearing

Possible causes:

• Incorrect ClickHouse credentials
• Exporter not running
• gRPC endpoint not enabled
• ClickHouse not reachable
• Traffic Logs are not enabled per device

Checks:

docker logs netmaker-exporter
docker ps
docker exec -it clickhouse clickhouse-client

---

Issue: gRPC endpoint returns 502

This usually means the Caddy / reverse proxy cannot reach the exporter.

Common causes:

• exporter is not listening on 50051
• container restart / wrong IP / bad Docker networking
• Disk full, causing service instability

Check:

docker ps
docker logs netmaker-exporter
nc -vz <exporter-host> 50051

---

Issue: ClickHouse is reachable, but no data

Check:

• correct DB name
• correct user credentials
• environment variables loaded
• exporter logs for insert/query errors

---

Recommended Production Best Practices

For production on-prem installs, I’d recommend:

Use external ClickHouse when:

• You expect high log volume
• You want better retention control
• You want separation of concerns
• You want HA/backup strategies

Restrict ClickHouse exposure

Do not leave ClickHouse open publicly without:

• authentication
• firewall restrictions
• TLS / proxying if needed

Log only what you need

If you have large networks, per-device selective logging is often better than full network logging.

Monitor disk usage

Flow Logs can grow quickly depending on traffic volume. Ensure retention and disk planning are considered.

---

Recommended Example Deployment Pattern (On-Prem)

For a more robust on-prem setup:

• Netmaker Server + Exporter on one host / VM
• ClickHouse on a separate VM
• Caddy / reverse proxy exposing dashboard + gRPC
• Internal firewall allowlists between components

This is generally cleaner than running everything on one small node.