Advanced Options

This section provides in-depth installation and configuration options for Netmaker, including the Netmaker Server, Netmaker UI, MQ Broker, and Reverse Proxy.

System Compatibility

Netmaker requires elevated privileges on the host machine to perform network operations. Netmaker must be able to modify interfaces and set firewall rules using iptables.

Typically, Netmaker is run inside of containers, using Docker or Kubernetes.

Netmaker can be run without containers, but this is not recommended. You must run the Netmaker binary, CoreDNS binary, database, and a web server directly on the host.

Each of these components has its own individual requirements and the management complexity increases exponentially by running outside of containers.

For first-time installs, we recommend the quick install guide. The following documents are meant for more advanced installation environments and are not recommended for most users. However, these documents can be helpful in customizing or troubleshooting your own installation.

Server Configuration Reference

Netmaker sets its configuration in the following order of precedence:

1

Environment Variables

Typically values set in the Docker Compose. This is the most common way of setting server values.

2

Config File

Values set in the config/environments/*.yaml file.

3

Defaults

Default values set on the server if no value is provided in configuration.

In most situations, if you wish to modify a server setting, set it in the netmaker.env file, then run:

• docker kill netmaker

• docker-compose up -d

Variable Description

NM_EMAIL Default: “” Description: Email used for SSL certificates

NM_DOMAIN Default: “” Description: Public IP of machine

SERVER_HOST Default: (Server detects the public IP address of machine) Description: The public IP of the server where the machine is running.

MASTER_KEY Default: “secretkey” Description: The admin master key for accessing the API. Change this in any production installation.

MQ_USERNAME Default: “” Description: The username to set for MQ access

MQ_PASSWORD Default: “” Description: The password to set for MQ access

INSTALL_TYPE Default: ce Description: The installation type to run on the server. “ce” will run community edition. “pro” will run professional edition if you have an on-prem tenant.

LICENSE_KEY Default: “” Description: The license key from your on-prem tenant needed to validate your professional installation

NETMAKER_TENANT_ID Default: “” Description: The ID of your on-prem tenant used to validate your professional installation.

SERVER_IMAGE_TAG Default: Description: The tag used for the server docker image. You can set it to “latest” to get the most up to date version. To stay on a certain version set the tag to the version you would like. ex: v0.24.2. if using a pro image, add “-ee” after the version. ex: v0.24.2-ee.

UI_IMAGE_TAG Default: “” Description: The tag used for the netmaker ui docker image. You can set it to “latest” to get the most up to date version. To stay on a certain version set the tag to the version you would like. ex: v0.24.2.

METRICS_EXPORTER Default: off Description: This is a pro feature that exports metrics to the netmaker server.

PROMETHEUS Default: off Description: This is a pro feature. Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. It is used in our metrics collection.

DNS_MODE Default: on Description: Enables DNS Mode, meaning all nodes will set hosts file for private dns settings.

NETCLIENT_AUTO_UPDATE Default: enabled Description: Enable/Disable auto update of netclient.

API_PORT Default: 8081 Description: The HTTP API port for Netmaker. Used for API calls / communication from front end. If changed, need to change port of BACKEND_URL for netmaker-ui.

EXPORTER_API_PORT Default: 8085 Description: The API port to set for the metrics exporter.

CORS_ALLOWED_ORIGIN Default: “*” Description: The “allowed origin” for API requests. Change to restrict where API requests can come from with comma-separated URLs. ex: https://dashboard.netmaker.domain1.com,https://dashboard.netmaker.domain2.com

DISPLAY_KEYS Default: on Description: Show keys permanently in UI (until deleted) as opposed to 1-time display.

DATABASE Default: sqlite Description: Database to use - sqlite, postgres, or rqlite

SERVER_BROKER_ENDPOINT Default: ws://mq:1883 Description: The address of the mq server. If running from docker compose it will be “mq”. Otherwise, need to input address. If using “host networking”, it will find and detect the IP of the mq container. For EMQX websockets use SERVER_BROKER_ENDPOINT=ws://mq:8083/mqtt

VERBOSITY Default: 1 Description: Logging verbosity level - 1, 2, 3, or 4

REST_BACKEND Default: on Description: Enables the REST backend (API running on API_PORT at SERVER_HTTP_HOST).

DISABLE_REMOTE_IP_CHECK Default: off Description: If turned “on”, Server will not set Host based on remote IP check. This is already overridden if SERVER_HOST is set.

TELEMETRY Default: on Description: Whether or not to send telemetry data to help improve Netmaker. Switch to “off” to opt out of sending telemetry.

ALLOWED_EMAIL_DOMAINS Default: “*” Description: only mentioned domains will be allowed to signup using oauth, by default all domains are allowed

AUTH_PROVIDER Default: “” Description: You can use azure-ad, github, google, oidc

CLIENT_ID Default: “” Description: The client id of your oauth provider.

CLIENT_SECRET Default: “” Description: The client secret of your oauth provider.

FRONTEND_URL Default: “” Description: https://dashboard.

AZURE_TENANT Default: “” Description: only for azure, you may optionally specify the tenant for the OAuth

OIDC_ISSUER Default: “” Description: https://oidc.yourprovider.com/ - URL of oidc provider

JWT_VALIDITY_DURATION Default: 43200 Description: Duration of JWT token validity in seconds

RAC_AUTO_DISABLE Default: false Description: Auto disable a user’s connected clients based on JWT token expiration

CACHING_ENABLED Default: true Description: if turned on data will be cached on to improve performance significantly (IMPORTANT: If HA set to false )

ENDPOINT_DETECTION Default: true Description: if turned on netclient checks if peers are reachable over private/LAN address, and choose that as peer endpoint

SMTP_HOST Default: "" Description: The address of the host SMTP service

SMTP_PORT Default: 587 Description: The port of the SMTP service

EMAIL_SENDER_ADDR Default: "" Description: The email address to send from

EMAIL_SENDER_USER Default: “” Description: The sender’s SMTP user. If empty, EMAIL_SENDER_ADDR would be used

EMAIL_SENDER_PASSWORD Default: "" Description: The password for the email sender

Starting from version v0.90.0, the following parameters are now configured directly from the Netmaker Settings interface:

• Allowed Email Domains

• Dashboard JWT Validity Duration

• Client JWT Validity Duration

• Restrict Simultaneous Network Connections

• Managed DNS

• DNS Base Domain

• LAN Routing

• Netclient Auto Update

• STUN Servers

• Verbosity Level

• Telemetry

• Metrics Port

• Metrics Collection Interval

• Email Configuration Parameters

Compose File - Annotated

All environment variables and options are enabled in this file. It is the equivalent to running the “full install” from the above section. However, all environment variables are included and are set to the default values provided by Netmaker (if the environment variable was left unset, it would not change the installation). Comments are added to each option to show how you might use it to modify your installation.

As of v0.18.0, netmaker now uses a stun server (Session Traversal Utilities for NAT). This provides a tool for communications protocols to detect and traverse NATs that are located in the path between two endpoints. By default, netmaker uses publicly available STUN servers. You are free to set up your own stun servers and use those to augment/replace the public STUN servers. Update the STUN_LIST to list the STUN servers you wish to use. Two resources for installing your own STUN server are:

• https://ourcodeworld.com/articles/read/1175/how-to-create-and-configure-your-own-stun-turn-server-with-coturn-in-ubuntu-18-04

• https://cloudkul.com/blog/how-to-install-turn-stun-server-on-aws-ubuntu-20-04/

There are also some environment variables that have been changed, or removed. Your updated docker-compose and .env files should look like this.

version: "3.4"

services:

  netmaker:
    container_name: netmaker
    image: gravitl/netmaker:$SERVER_IMAGE_TAG
    env_file: ./netmaker.env
    restart: always
    volumes:
      - dnsconfig:/root/config/dnsconfig
      - sqldata:/root/data
    environment:
      # config-dependant vars
      - STUN_LIST=stun1.netmaker.io:3478,stun2.netmaker.io:3478,stun1.l.google.com:19302,stun2.l.google.com:19302
      # The domain/host IP indicating the mq broker address
      - BROKER_ENDPOINT=wss://broker.${NM_DOMAIN}
      # The base domain of netmaker
      - SERVER_NAME=${NM_DOMAIN}
      - SERVER_API_CONN_STRING=api.${NM_DOMAIN}:443
      # Address of the CoreDNS server. Defaults to SERVER_HOST
      - COREDNS_ADDR=${SERVER_HOST}
      # Overrides SERVER_HOST if set. Useful for making HTTP available via different interfaces/networks.
      - SERVER_HTTP_HOST=api.${NM_DOMAIN}

  netmaker-ui:
    container_name: netmaker-ui
    image: gravitl/netmaker-ui:$UI_IMAGE_TAG
    env_file: ./netmaker.env
    environment:
      # config-dependant vars
      # URL where UI will send API requests. Change based on SERVER_HOST, SERVER_HTTP_HOST, and API_PORT
      BACKEND_URL: "https://api.${NM_DOMAIN}"
    depends_on:
      - netmaker
    links:
      - "netmaker:api"
    restart: always

  caddy:
    image: caddy:2.6.2
    container_name: caddy
    env_file: ./netmaker.env
    restart: unless-stopped
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./certs:/root/certs
      - caddy_data:/data
      - caddy_conf:/config
    ports:
      - "80:80"
      - "443:443"

  coredns:
    container_name: coredns
    image: coredns/coredns
    command: -conf /root/dnsconfig/Corefile
    env_file: ./netmaker.env
    depends_on:
      - netmaker
    restart: always
    volumes:
      - dnsconfig:/root/dnsconfig
  mq:
    container_name: mq
    image: eclipse-mosquitto:2.0.15-openssl
    env_file: ./netmaker.env
    depends_on:
      - netmaker
    restart: unless-stopped
    command: [ "/mosquitto/config/wait.sh" ]
    volumes:
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf
      - ./wait.sh:/mosquitto/config/wait.sh
      - mosquitto_logs:/mosquitto/log
      - mosquitto_data:/mosquitto/data

volumes:
  caddy_data: { } # runtime data for caddy
  caddy_conf: { } # configuration file for Caddy
  sqldata: { }
  dnsconfig: { } # storage for coredns
  mosquitto_logs: { } # storage for mqtt logs
  mosquitto_data: { } # storage for mqtt data

The corresponding example .env entries:

# Email used for SSL certificates
NM_EMAIL=

# The base domain of netmaker
NM_DOMAIN=

# Public IP of machine
SERVER_HOST=

# The admin master key for accessing the API. Change this in any production installation.
MASTER_KEY=

# The username to set for MQ access
MQ_USERNAME=

# The password to set for MQ access
MQ_PASSWORD=
INSTALL_TYPE=
NETMAKER_TENANT_ID=
LICENSE_KEY=
SERVER_IMAGE_TAG=
UI_IMAGE_TAG=
NETCLIENT_ENDPOINT_DETECTION="disabled"

# used for HA - identifies this server vs other servers
NODE_ID="netmaker-server-1"
METRICS_EXPORTER="off"
PROMETHEUS="off"

# Enables DNS Mode, meaning all nodes will set hosts file for private dns settings
DNS_MODE="on"

# Enable auto update of netclient ? ENUM:- enabled,disabled | default=enabled
NETCLIENT_AUTO_UPDATE="enabled"

# The HTTP API port for Netmaker. Used for API calls / communication from front end.
# If changed, need to change port of BACKEND_URL for netmaker-ui.
API_PORT="8081"
EXPORTER_API_PORT="8085"

# The "allowed origin" for API requests. Change to restrict where API requests can come from with comma-separated
# URLs. ex:- https://dashboard.netmaker.domain1.com,https://dashboard.netmaker.domain2.com
CORS_ALLOWED_ORIGIN="*"

# Show keys permanently in UI (until deleted) as opposed to 1-time display.
DISPLAY_KEYS="on"

# Database to use - sqlite, postgres, or rqlite
DATABASE="sqlite"

# The address of the mq server. If running from docker compose it will be "mq". Otherwise, need to input address.
# If using "host networking", it will find and detect the IP of the mq container.
SERVER_BROKER_ENDPOINT="ws://mq:1883" # For EMQX websockets use `SERVER_BROKER_ENDPOINT=ws://mq:8083/mqtt`

# The reachable port of STUN on the server
STUN_PORT="3478"

# Logging verbosity level - 1, 2, or 3
VERBOSITY="1"
DEBUG_MODE="off"

# Enables the REST backend (API running on API_PORT at SERVER_HTTP_HOST).
# Change to "off" to turn off.
REST_BACKEND="on"

# If turned "on", Server will not set Host based on remote IP check.
# This is already overridden if SERVER_HOST is set. Turned "off" by default.
DISABLE_REMOTE_IP_CHECK="off"

# Whether or not to send telemetry data to help improve Netmaker. Switch to "off" to opt out of sending telemetry.
TELEMETRY="on"
###

#
# OAuth section
#
###

# "<azure-ad|github|google|oidc>"
AUTH_PROVIDER=

# "<client id of your oauth provider>"
CLIENT_ID=

# "<client secret of your oauth provider>"
CLIENT_SECRET=

# "https://dashboard.<netmaker base domain>"
FRONTEND_URL=

# "<only for azure, you may optionally specify the tenant for the OAuth>"
AZURE_TENANT=

# https://oidc.yourprovider.com - URL of oidc provider
OIDC_ISSUER=

# config for sending emails

# mail server host
SMTP_HOST=smtp.gmail.com

# mail server port
SMTP_PORT=587

# sender email
EMAIL_SENDER_ADDR=

# sender smtp user, if unset sender email will be used
EMAIL_SENDER_USER=

# sender smtp password
EMAIL_SENDER_PASSWORD=

Caddyfile example:

{

    email YOUR_EMAIL
}

# Dashboard
https://dashboard.NETMAKER_BASE_DOMAIN {
    # Apply basic security headers
    header {
            # Enable cross origin access to *.NETMAKER_BASE_DOMAIN
            Access-Control-Allow-Origin *.NETMAKER_BASE_DOMAIN
            # Enable HTTP Strict Transport Security (HSTS)
            Strict-Transport-Security "max-age=31536000;"
            # Enable cross-site filter (XSS) and tell browser to block detected attacks
            X-XSS-Protection "1; mode=block"
            # Disallow the site to be rendered within a frame on a foreign domain (clickjacking protection)
            X-Frame-Options "SAMEORIGIN"
            # Prevent search engines from indexing
            X-Robots-Tag "none"
            # Remove the server name
            -Server
    }

    reverse_proxy http://netmaker-ui
}

# API
https://api.NETMAKER_BASE_DOMAIN {
        reverse_proxy http://netmaker:8081
}

# MQ
wss://broker.NETMAKER_BASE_DOMAIN {
        reverse_proxy ws://mq:8883
}

Available docker-compose files

The default options for docker-compose can be found here: https://github.com/gravitl/netmaker/tree/master/compose

The following is a brief description of each:

• docker-compose.yml — This maintains the most recommended setup at the moment, using the caddy proxy. (https://github.com/gravitl/netmaker/blob/master/compose/docker-compose.yml)

• docker-compose.pro.yml — This is the compose file needed for Netmaker Professional. You will need a licence and tenant id from Netmaker’s licence dashboard. (https://github.com/gravitl/netmaker/blob/master/compose/docker-compose.pro.yml)

Setting a Netmaker server up for emailing

Starting from version v0.90.0, the Email Configuration Parameters are now configured directly from the Settings interface.

Since v0.25.0, the Netmaker server can now send email notifications to users, when relevant actions are effected on the server (e.g., inviting a user to join the server).

To enable emailing, the server administrator must correctly provide the below environment variables (in the netmaker.env file), and restart the server if needed:

• SMTP_HOST

• SMTP_PORT

• EMAIL_SENDER_ADDR

• EMAIL_SENDER_USER

• EMAIL_SENDER_PASSWORD

Refer to the Variable Description section above for more information on what values to assign.

EMQX

Netmaker offers an EMQX option as a broker for your server. The main configuration changes between mosquitto and EMQX occur in the docker-compose.yml, netmaker.env and the Caddyfile.

You can find the EMQX docker-compose file in the netmaker repo: https://github.com/gravitl/netmaker/blob/master/compose/docker-compose-emqx.yml

You should not need to make any changes to the docker-compose-emqx.yml file. Just download this file using the command provided below in the same directory as your netmaker.env file — it will grab information from netmaker.env:

wget https://raw.githubusercontent.com/gravitl/netmaker/master/compose/docker-compose-emqx.yml

In your Caddyfile, change the mq block to:

# MQ
wss://broker.{$NM_DOMAIN} {
    reverse_proxy ws://mq:8083
}

Update netmaker.env:

Replace:

SERVER_BROKER_ENDPOINT="ws://mq:1883"

with:

SERVER_BROKER_ENDPOINT=ws://mq:8083/mqtt

In your docker-compose.yml, update BROKER_ENDPOINT and add broker type entries:

- BROKER_ENDPOINT=wss://broker.${NM_DOMAIN}/mqtt
- BROKER_TYPE=emqx
- EMQX_REST_ENDPOINT=http://mq:18083

If using a professional server, update netmaker-exporter in docker-compose.override.yml:

netmaker-exporter:
    container_name: netmaker-exporter
    image: gravitl/netmaker-exporter:latest
    restart: always
    depends_on:
        - netmaker
    environment:
        SERVER_BROKER_ENDPOINT: "ws://mq:8083/mqtt"
        BROKER_ENDPOINT: "wss://broker.nm.${NM_DOMAIN}/mqtt"
        PROMETHEUS_HOST: "https://prometheus.${NM_DOMAIN}"

Bring up the services:

docker-compose down && docker-compose up -d && docker-compose -f docker-compose-emqx.yml up -d

Your docker logs mq should show listeners similar to:

Listener ssl:default on 0.0.0.0:8883 started.
Listener tcp:default on 0.0.0.0:1883 started.
Listener ws:default on 0.0.0.0:8083 started.
Listener wss:default on 0.0.0.0:8084 started.
Listener http:dashboard on :18083 started.
EMQX 5.0.9 is running now!

You can view your EMQX dashboard at http://<serverip>:18083/. Sign-in credentials are the EMQX_DASHBOARD__DEFAULT_USERNAME and EMQX_DASHBOARD__DEFAULT_PASSWORD located in your netmaker.env file.

Nginx Reverse Proxy Setup with HTTPS

The Swag Proxy (https://github.com/linuxserver/docker-swag) makes it easy to generate a valid SSL certificate for the config below. Here is the documentation: https://docs.linuxserver.io/general/swag

An example config for Netmaker as a subdomain (adapted from swag):

./netmaker.subdomain.conf:

server {
    # Redirect HTTP to HTTPS.
    listen 80;
    server_name *.netmaker.example.org; # Please change to your domain
    return 301 https://$host$request_uri;
    }

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name dashboard.netmaker.example.org; # Please change to your domain
    include /config/nginx/ssl.conf;
    location / {
        proxy_pass http://<NETMAKER_IP>:8082;
        }
    }

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name api.netmaker.example.org; # Please change to your domain
    include /config/nginx/ssl.conf;

    location / {
        proxy_pass http://<NETMAKER_IP>:8081;
        proxy_set_header            Host api.netmaker.example.org; # Please change to your domain
        proxy_pass_request_headers  on;
        }
    }

Nginx Proxy Manager Setup

To use Netmaker with Nginx Proxy Manager, add three proxy hosts, one for each subdomain used by Netmaker. Each subdomain should have SSL enabled and be configured as follows:

api.netmaker.example.com:

• Forward Hostname/IP: netmaker

• Forward Port: 8081

dashboard.netmaker.example.com:

• Forward Hostname/IP: netmaker-ui

• Forward Port: 80

grpc.netmaker.example.com:

• Forward Hostname/IP: netmaker

• Forward Port: 50051

• Custom Locations:

  • Add location /

  • Forward Hostname/IP: netmaker

  • Forward Port: 50051

  • Custom config: grpc_pass netmaker:50051;

A cleaned-up config generated by Nginx Proxy Manager (does not include SSL configuration):

# dashboard.netmaker.example.com
server {
  set $forward_scheme http;
  set $server         "netmaker-ui";
  set $port           80;
  listen 80;
  listen [::]:80;
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name dashboard.netmaker.example.com;
  location / {
    include conf.d/include/proxy.conf;
    # proxy_pass       $forward_scheme://$server:$port$request_uri;
  }
}

# api.netmaker.example.com
server {
  set $forward_scheme http;
  set $server         "netmaker";
  set $port           8081;
  listen 80;
  listen [::]:80;
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name api.netmaker.example.com;
  location / {
    include conf.d/include/proxy.conf;
    # proxy_pass       $forward_scheme://$server:$port$request_uri;
  }
}

# grpc.netmaker.example.com
server {
  set $forward_scheme http;
  set $server         "netmaker";
  set $port           50051;
  listen 80;
  listen [::]:80;
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name grpc.netmaker.example.com;
  location / {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
    proxy_set_header X-Real-IP          $remote_addr;
    proxy_pass       http://netmaker:50051;
    grpc_pass netmaker:50051;
  }
}

Security Settings

It can be useful to secure your web dashboard behind a firewall while keeping the API available for nodes. Below are examples for Caddy and Traefik reverse proxies.

For Caddy

In your /root/Caddyfile, in the Dashboard section where reverse_proxy http://netmaker-ui appears, add above it:

@blocked not remote_ip <ip1> <ip2> <ip3>
respond @blocked "Nope" 403

Replace the placeholders with your whitelist IP ranges.

For Traefik

1. In the labels section, add:

traefik.http.middlewares.nmui-security-1.ipwhitelist.sourcerange=YOUR_IP_CIDR

1. Update the router middlewares line from:

traefik.http.routers.netmaker-ui.middlewares=nmui-security@docker

to:

traefik.http.routers.netmaker-ui.middlewares=nmui-security-1@docker,nmui-security@docker

Replace YOUR_IP_CIDR with the whitelist IP range (can be multiple ranges).

After changes are made for your reverse proxy, run:

docker-compose down && docker-compose up -d

This keeps your dashboard secure while keeping the API available without changing netmaker-ui ports.

Setup Netmaker on an IPv6-only Machine

This is not a guide on adding an overlay network (with IPv6) in Netmaker; see the setup page for that. This guide covers getting Netmaker to run on an IPv6-only machine.

About the install script nm-quick.sh

At the time of writing, the install script nm-quick.sh only supports IPv4. For installation, IPv4 needs to be enabled temporarily.

Add AAAA record for domain name resolution

Netmaker client communicates with the Netmaker server by domain name. Add an AAAA record to resolve the domain name to the server's IPv6 address.

Enable IPv6 support for Docker

1

1. Add/Edit the Docker daemon config at /etc/docker/daemon.json:

{
  "experimental": true,
  "ip6tables": true
}

2

1. Restart the Docker daemon:

sudo systemctl restart docker

3

1. Create a new IPv6 Docker network, for example:

docker network create --ipv6 --subnet 2001:0DB8::/112 ip6net

Here “ip6net” is the network name and “2001:0DB8::/112” is the network range.

Enable IPv6 support for Netmaker

1

1. Edit your docker-compose.yml and add the external network declaration at the bottom:

networks:
  ip6net:
    external: true

2

1. In docker-compose.yml, add the networks field for every service that should attach to the IPv6 network:

networks:
  - ip6net

3

1. Restart Netmaker:

docker-compose down
docker-compose up -d