# Setting Up Network Flow Logs on an On-Prem Netmaker Installation ### Overview **Network Flow Logs** in Netmaker provide visibility into traffic traversing your overlay network. They allow you to see: * Which nodes are communicating * Source and destination IPs * Source and destination ports * Protocol and direction * Byte and packet counts * Flow start and end timestamps This helps with: * Security auditing * Network troubleshooting * Access validation * Visibility into overlay traffic patterns > **Note:** Flow Logs is a **Pro / Enterprise feature** and requires an appropriate license. *** ## Architecture Summary Flow Logs on Netmaker rely on the following components: * **Netmaker Server** * **Netmaker Exporter** * **ClickHouse** (for storing flow log records) At a high level: **Netclient / Devices → Netmaker Exporter → ClickHouse → Netmaker UI** *** ## Prerequisites Before enabling Flow Logs, ensure the following are available: ### Minimum Infrastructure Recommended minimum for the monitoring stack: * **2 GB RAM** * **2 vCPUs** If you are running this in production or collecting logs from many devices, allocate more resources accordingly. *** ## Step 1 — Enable the Monitoring Stack Flow Logs depend on the Netmaker monitoring components. If Netmaker is already installed, re-run the installer with monitoring enabled: ``` sudo ./nm-quick.sh -m ``` If this is a fresh install: ``` sudo ./nm-quick.sh -p -m ``` This enables: * Prometheus * Grafana * Netmaker Exporter > Flow Logs will not function correctly unless the exporter and monitoring stack are enabled. *** ## Step 2 — Enable the gRPC Endpoint in Caddy Flow Logs requires the **gRPC endpoint** to be reachable. ### Edit the Caddyfile Open your Netmaker Caddyfile (typically located at): ``` /root/netmaker/Caddyfile ``` Locate the gRPC block and ensure it is enabled: ``` # GRPC https://grpc.{$NM_DOMAIN} { reverse_proxy netmaker-exporter:50051 { transport http { versions h2c } } } ``` ### Reload Caddy After saving the file, reload Caddy: ``` docker exec caddy caddy reload --config /etc/caddy/Caddyfile ``` *** ### Verify gRPC Connectivity Once enabled, make sure your gRPC subdomain resolves correctly and is reachable. Example: ``` nslookup grpc.yourdomain.com ``` If the gRPC endpoint is not working, Flow Logs will not ingest traffic properly. *** ## Step 3 — Set Up ClickHouse Netmaker stores Flow Logs in **ClickHouse**. You have **two options**: *** ### Option A — Use the Bundled ClickHouse Container (Recommended for Quick Setup) If you are using the default Netmaker Docker deployment, ClickHouse is already referenced in the override file, but may be commented out. #### Edit Docker Compose Override Open: ``` /root/netmaker/docker-compose.override.yml ``` Uncomment: * the `clickhouse` service * the `depends_on` line under `netmaker-exporter` * the ClickHouse volumes Example: ``` services: netmaker-exporter: depends_on: - netmaker - clickhouse clickhouse: container_name: clickhouse image: clickhouse/clickhouse-server:latest restart: unless-stopped ports: - "8123:8123" - "9000:9000" volumes: - clickhouse_data:/var/lib/clickhouse - clickhouse_logs:/var/log/clickhouse-server volumes: clickhouse_data: {} clickhouse_logs: {} ``` > For on-prem deployments, this is usually the easiest path for first-time enablement. *** ### Option B — Use an External ClickHouse Instance You may instead use: * A separate self-hosted ClickHouse server * A managed ClickHouse service This is often preferable for: * production-scale flow retention * higher availability * centralised logging architectures If using external ClickHouse, ensure: * It is reachable from the Netmaker host * Firewall rules allow access * Credentials are configured properly *** ## Step 4 — Create the ClickHouse Database If your ClickHouse instance is fresh, create a dedicated database for Flow Logs. ### Connect to ClickHouse If using the bundled container: ``` docker exec -it clickhouse clickhouse-client ``` If external: ``` clickhouse-client --host --port 9000 --user --password ``` ### Create the database ``` CREATE DATABASE IF NOT EXISTS netmaker_logs ENGINE = Atomic; ``` Verify: ``` SHOW DATABASES; ``` You should see: ``` netmaker_logs ``` > Using a dedicated database keeps Flow Logs separate from other telemetry or analytics workloads. *** ## Step 5 — Configure Netmaker Environment Variables Edit your Netmaker environment file: ``` /root/netmaker/netmaker.env ``` Add or update the following: ``` CLICKHOUSE_HOST= CLICKHOUSE_PORT= CLICKHOUSE_DB=netmaker_logs CLICKHOUSE_USER= CLICKHOUSE_PASS= ``` #### Example (Bundled ClickHouse) ``` CLICKHOUSE_HOST=clickhouse CLICKHOUSE_PORT=9000 CLICKHOUSE_DB=netmaker_logs CLICKHOUSE_USER=default CLICKHOUSE_PASS=yourpassword ``` #### Example (External ClickHouse) ``` CLICKHOUSE_HOST=10.0.20.5 CLICKHOUSE_PORT=9000 CLICKHOUSE_DB=netmaker_logs CLICKHOUSE_USER=netmaker CLICKHOUSE_PASS=strongpassword ``` > All five `CLICKHOUSE_*` values must be set correctly or Flow Logs will not appear. *** ## Step 6 — Restart Netmaker Services After saving the configuration, restart the Netmaker stack: ``` cd /root/netmaker docker compose up -d --force-recreate ``` This recreates containers with the updated ClickHouse configuration. *** ## Step 7 — Enable Flow Logs in the Netmaker Dashboard Once the backend prerequisites are complete: ### In the UI Navigate to: **Settings → Monitoring & Debugging** or directly: ``` https:///settings?tab=monitoring-debugging ``` Then: 1. Toggle **Traffic Logs** to **Enabled** 2. Optionally toggle **Force Enable** #### What “Force Enable” Does If enabled, Flow Logs will be turned on for **all registered devices** automatically. This is useful for: * smaller environments * troubleshooting sessions * full network visibility If disabled, logging can be controlled per device. *** ## Step 8 — Optional: Enable Flow Logs Per Device If you want to log only selected devices: 1. Go to **All Devices** 2. Select a device 3. Click **Edit** 4. Toggle **Enable Traffic Logs** This is useful if you only want logs from: * gateways * relay nodes * jump hosts * sensitive workloads *** ## What You Will See in Flow Logs Each log record can include: * Source IP * Destination IP * Source port * Destination port * Protocol * Direction * Packet count * Byte count * Start time * End time Netmaker also attempts to enrich logs with: * Node names * External client identity * Known domain names (best-effort) This makes logs more readable than raw IP-only flow records. *** ## Verification Checklist Once enabled, verify the following: ### 1) Exporter is running ``` docker ps | grep netmaker-exporter ``` ### 2) ClickHouse is reachable If bundled: ``` docker exec -it netmaker-exporter sh nc -vz clickhouse 9000 ``` If external: ``` nc -vz ``` ### 3) gRPC endpoint is active Check your reverse proxy logs and confirm the exporter is reachable on port **50051**. ### 4) Dashboard toggle is visible If the **Traffic Logs** toggle is missing, check your license. *** ## Troubleshooting ### Issue: Flow Logs toggle not visible Possible cause: * License does not include Flow Logs #### Action: Verify your Pro / Enterprise license. *** ### Issue: Logs are not appearing Possible causes: * Incorrect ClickHouse credentials * Exporter not running * gRPC endpoint not enabled * ClickHouse not reachable * Traffic Logs are not enabled per device #### Checks: ``` docker logs netmaker-exporter docker ps docker exec -it clickhouse clickhouse-client ``` *** ### Issue: gRPC endpoint returns 502 This usually means the Caddy / reverse proxy cannot reach the exporter. #### Common causes: * exporter is not listening on `50051` * container restart / wrong IP / bad Docker networking * Disk full, causing service instability #### Check: ``` docker ps docker logs netmaker-exporter nc -vz 50051 ``` *** ### Issue: ClickHouse is reachable, but no data Check: * correct DB name * correct user credentials * environment variables loaded * exporter logs for insert/query errors *** ## Recommended Production Best Practices For production on-prem installs, I’d recommend: ### Use external ClickHouse when: * You expect high log volume * You want better retention control * You want separation of concerns * You want HA/backup strategies ### Restrict ClickHouse exposure Do **not** leave ClickHouse open publicly without: * authentication * firewall restrictions * TLS / proxying if needed ### Log only what you need If you have large networks, per-device selective logging is often better than full network logging. ### Monitor disk usage Flow Logs can grow quickly depending on traffic volume. Ensure retention and disk planning are considered. *** ## Recommended Example Deployment Pattern (On-Prem) For a more robust on-prem setup: * **Netmaker Server + Exporter** on one host / VM * **ClickHouse** on a separate VM * **Caddy / reverse proxy** exposing dashboard + gRPC * **Internal firewall allowlists** between components This is generally cleaner than running everything on one small node.