IoT Custom Flow

IoT network traffic dataset using the custom flow representation

Dataset DOI: 10.5061/dryad.6q573n6c1

This dataset provides Custom Flow representations derived from raw IoT network traffic traces, capturing detailed behavioral characteristics of IoT communications. Each Custom Flow encapsulates network behavior in a structured, vectorized format that includes flow-level metadata, packet sequence timing, direction, and selected payloads. Flows are uniquely identified by a five-tuple: device IP address, remote IP address, protocol, device port, and remote port, and maintain a fixed one-minute lifetime. The dataset was generated from 60 days of packet capture (PCAP) traces obtained from the publicly available UNSW IoT Traffic Analytics platform, covering 22 consumer IoT device types and containing over 5.9 million custom flow records.

📊 Description of the data and file structure

We provide two compressed archives with different flow variants:

📦 bidirectional.tar.gz (1.78 GB)

Bidirectional Custom Flows capturing both upstream and downstream packets (~6 million flows)

📦 unidirectional.tar.gz (795 MB)

Unidirectional Custom Flows capturing only upstream packets from device perspective (~3.5 million flows)

📁 File Structure

Each archive contains 60 daily Parquet files organized by date:

bidirectional/
├── 16-09-23.parquet
├── 16-09-24.parquet
├── 16-09-25.parquet
│   ...
├── 16-11-20.parquet
├── 16-11-21.parquet
└── 16-11-22.parquet
unidirectional/
├── 16-09-23.parquet
├── 16-09-24.parquet
├── 16-09-25.parquet
│   ...
├── 16-11-20.parquet
├── 16-11-21.parquet
└── 16-11-22.parquet

📐 Custom Flow Structure

Each custom flow record has the following structure:

┌─────────────────────────────────────────────────────────────────────────────────────┐
│                                   Custom Flow                                       │
├─────────────────────────────────────────────────────────────────────────────────────┤
│  Flow Meta  │  P01_Meta  │  P02_Meta  │  ...  │  Pi_Meta  │   B bytes flow payload  │
└──────┬──────┴──────┬─────┴────────────┴───────┴───────────┴───────────┬─────────────┘
       │             │                                                  │
       │             │                                                  │
       │             ▼                                                  │
       │    ┌────────────────────────────────────┐                      │
       │    │        PACKET METADATA             │                      │
       │    ├────────────────────────────────────┤                      │
       │    │ Time Offset │ Pkt Size │ Direction │                      │
       │    └────────────────────────────────────┘                      │
       │                                                                │
       │                                                                ▼
       │    ┌──────────────────────────────────────────────────────────────────────┐
       │    │                          FLOW PAYLOAD                                │
       │    ├──────────────────────────────────────────────────────────────────────┤
       │    │ P_start │ P_end │ P_start │ P2_B001 │ P2_B... │ P2_Bn │ P_end │ Pad  │
       │    └──────────────────────────────────────────────────────────────────────┘
       │
       ▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│                                FLOW METADATA                                        │
├─────────────────────────────────────────────────────────────────────────────────────┤
│ Relative     │  IPv4  │  Remote  │  Device  │ Protocol │  Total    │  Total         │
│ Timestamp    │        │  Port    │  Port    │          │  Bytes    │  Packets       │
└─────────────────────────────────────────────────────────────────────────────────────┘

📋 Data Fields

Each custom flow includes the following fields:

Flow Metadata

  • Device: Device MAC Address
  • FirstSeen: Unix timestamp of the first packet (µs)
  • RemIP: Remote IPv4 address
  • Proto: Transport layer protocol
  • DevPort: Device-side port number
  • RemPort: Remote-side port number
  • TotalFlowSize: Total byte count of the flow
  • PacketCount: Total packet count of the flow

Packet-Level Features (first 10 packets)

  • P00_TO - P10_TO: Time offset from flow's first-seen packet timestamp (µs)
  • P00_PS - P10_PS: Packet size (bytes)
  • P00_D - P10_D: Direction flag (1 = device → remote, 0 = remote → device)
  • C_000 - C_2999: Transport-layer payload bytes with delimiters, -4 for packet start and -8 for packet end

Cite our data