Astro Realtime Web Analytics: Design Notes on Collection and Aggregation

An analytics pipeline design covering accuracy and retention, with bounded browser-event queues, batching, and ClickHouse aggregation.

1. Collection scope and data minimization

These are design notes for an analytics pipeline, not evidence of production throughput or certified legal compliance. First define the data collected, its purpose, retention periods, and access rights. Design the system to avoid collecting sensitive URL queries or unnecessary personally identifying information.

Avoiding cookies or hashing IP addresses does not automatically guarantee full anonymity or compliance with every privacy regulation. IP-derived values do not equal an accurate count of individual visitors because addresses can be shared or change. Actual deployment requires a separate review.

2. Batching and backpressure

• Limit event size and queue length, and specify rejection, sampling, and retry policies when limits are exceeded.
• Use event IDs to identify duplicates and prevent retries from inflating aggregates.
• Tune ClickHouse insert batch size together with delay. Specify when asynchronous inserts are acknowledged and how errors are handled.
• Validate buffer flushing and completion acknowledgment settings against the ClickHouse asynchronous inserts documentation.

3. Dashboard and operational validation

When sending aggregates over WebSockets, design for reconnection, the last received position, and slow clients. Distinguish client-side measurements from server self-measurements, and do not replace unsupported or missing data with zero.

Load tests should record collection success, duplicate and loss rates, p95/p99 latency, and storage cost together. Present throughput or query speed as an outcome only when supported by reproducible measurements that disclose data size, schema, and hardware.