Erlang Flow Aggregator (erlflow)

erlflow — a high-performance NetFlow v5 aggregator and analyzer designed to transform raw network flow data into structured, easily analyzable metrics with support for tagging and flexible grouping. This tool reduces the number of time series by grouping flows based on logical rules and integrates seamlessly with monitoring systems like Prometheus.

🎯 Key Features

• ✅ NetFlow v5 flow grouping based on customizable rules
• ✅ Dynamic metric tagging with support for static and dynamic labels
• ✅ Flexible filtering by IP, ports, protocols, ToS, and other parameters
• ✅ Automatic scalability — metrics appear automatically as new nodes join the network
• ✅ Prometheus integration — ready-to-use metrics for collection and visualization
• ✅ YAML-based configuration — clear and powerful filtering rules
• ✅ Flow rejection support — ignore irrelevant flows using the reject action

📦 Use Case Example

Scenario: Monitoring SIP Infrastructure

A cluster of SIP servers with the following setup:

• Internal network: 100.127.0.0/24
• External network for clients: 88.127.127.0/24
• Control ports: TCP 5080
• Media ports: UDP 40000–41900

Goal:

• Track traffic between servers in pairs
• Aggregate client traffic per server

Solution with erlflow:

- src_addr:
    match: 100.127.0.0/24
  dst_addr:
    match: 100.127.0.0/24
  proto:
    match: 6
  port:
    match: 5080
  action:
    key_suffix: _sip_srv
    attributes:
      - src_addr
      - dst_addr
    ext_attributes:
      application: SIP
      direction: service-service

Instead of thousands of individual flows, you get 12 stable metrics, such as:

netflow_bytes_sent_sip_srv{src_addr="100.127.0.1",dst_addr="100.127.0.2",application="SIP",direction="service-service",sensor="127.0.0.1"}
netflow_bytes_sent_sip_upstream{dst_addr="88.127.127.1",application="SIP",direction="client-service",sensor="127.0.0.1"}

⚙️ Configuration

Configuration File Format

The default configuration file is config/config.yml. The path can be overridden in sys.config via the config_path parameter.

Rule Structure

Each rule consists of:

1. Filtering conditions — flow parameters and comparison operators
2. Action — how to process matching flows

Available Operators:

• match — matches a value or range
• dismatch — does not match

Supported Flow Parameters:

Important: At least one filtering parameter must be defined in each rule.

Actions (action)

action:
  key_suffix: _my_metric      # Required: suffix for the metric name
  attributes:                 # Flow parameters to use as labels
    - src_addr
    - dst_addr
  ext_attributes:             # Static labels
    application: "SIP"
    direction: "internal"

If attributes are not specified, the following defaults are used: src_addr, dst_addr, proto, port, tos

Ignoring Flows

action: reject

Warning: If multiple rules use the same key_suffix, their attributes and ext_attributes must match for Prometheus compatibility.

📊 Output Metric Format

erlflow generates Prometheus-style metrics:

netflow_bytes_sent_{suffix}{labels}
netflow_packets_sent_{suffix}{labels}

Where:

• {suffix} — the suffix from the rule
• {labels} — labels from attributes and ext_attributes, plus the auto-added sensor label (NetFlow source address)

🚀 Quick Start

1. Install dependencies (Erlang/OTP, rebar3)
2. Clone the repository:

   git clone https://codeberg.org/ttt161/erlflow.git
   cd erlflow

3. Configure rules in config/config.yml
4. Start the application:

   rebar3 shell

5. Send NetFlow v5 packets to the default port (2055)
6. Collect metrics via the Prometheus endpoint

🔧 Advanced Settings

Ephemeral Port Range

Default: 49152-65535. Can be overridden in sys.config:

{ephemeral_range, {49152, 65535}}

Override Configuration Path

{config_path, "/path/to/your/config.yml"}

🤝 Contributing

We welcome issues, pull requests, and improvement suggestions! Project hosted on Codeberg: https://github.com/ttt161/erlflow

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

⚡ Performance

erlflow is built on Erlang/OTP and leverages:

• BEAM VM for parallel flow processing
• Binary pattern matching for fast NetFlow packet parsing
• ETS tables for efficient state storage
• Asynchronous processing for minimal latency

🎯 Summary

erlflow is ideal for:

• monitoring inter-service traffic
• analyzing network interactions
• integrating network monitoring into observability platforms

The tool transforms raw NetFlow into meaningful business metrics, reducing monitoring complexity and simplifying network infrastructure analysis.