Bot traffic filtering for links: what gets flagged and what slips through

Bot traffic filtering for links decides, request by request, whether a hit on a short link counts as a click.

TrackRev flags a request when its user agent matches the isbot 5.2.1 pattern list, when the user agent is empty, or when a purpose header marks it as a prefetch.

A flagged request still gets its 302 redirect, is stored with is_bot true, and stays out of every report.

That removes Googlebot, Slack and Facebook preview fetchers, curl and python-requests scripts, and Chrome's speculative loads. A scanner with an ordinary Chrome 120 user agent passes, and finding those takes a look at your own click data.

Where non-human clicks come from

The Media Rating Council's invalid traffic addendum (June 2020 update) calls what list-based filtering finds general invalid traffic: bots and spiders, non-browser user agents, pre-fetch and pre-render traffic, and known data-center traffic.

Sophisticated invalid traffic takes advanced analytics and human review. The standard covers ad impressions, and the split carries over to links.

Preview fetchers and email scanners

Slack's robots page says Slackbot-LinkExpanding 1.0 fetches links posted into channels with HTTP Range headers and caches responses across the service for around 30 minutes. Meta documents facebookexternalhit/1.1 as the crawler behind previews on Facebook, Instagram and Messenger.

Mailchimp's bot activity page names anti-spam filters at email providers, Apple's Mail Privacy Protection and link preview generators as tools that often click links or open emails before the recipient does.

Its bot-filtered email click data starts on August 10, 2025, and it hides flagged activity without deleting it, the design TrackRev uses. The newsletter guide and the Slack and Discord guide go deeper on those channels.

How bot traffic filtering for links works in TrackRev

• A missing or blank user-agent header is flagged.
• The user agent is checked against isbot 5.2.1, the version TrackRev's lockfile resolves. It is an open-source list of 209 patterns built from public crawler lists such as monperrus/crawler-user-agents and user-agents.net. It matches names like Googlebot and HeadlessChrome, any string where bot ends a word, and any string containing http, which is how Googlebot, Slackbot-LinkExpanding and facebookexternalhit point to their documentation.
• The headers x-purpose, purpose and sec-purpose are read, and a value containing prefetch flags the request. Chrome sends Sec-Purpose: prefetch for a Speculation Rules prefetch and prefetch;prerender for a prerender, and MDN says a server can treat such requests differently when counting visits.

What a flagged request changes

Why flagged clicks are kept

Google Analytics excludes known bots automatically, using Google research and the International Spiders and Bots List kept by the Interactive Advertising Bureau. The exclusion cannot be switched off, and Google does not report how much it removed.

TrackRev keeps the rows. The clicks API leaves them out unless you pass bots=include, and the CLI has the same switch, so you can measure your own bot share.

The API needs a paid plan and a secret key, and --all stops after 200 pages of up to 500 clicks, so on a large workspace add --link with one link ID.

trackrev clicks --bots --all --json \
  | jq '.clicks | group_by(.is_bot) | map({is_bot: .[0].is_bot, requests: length})'

What a user-agent list cannot see

The isbot README says the package identifies good bots that announce themselves and does not try to recognize malicious bots or programs posing as real users.

TrackRev stores each click's IP address and user agent but sets the flag from the three tests alone.

It checks no IP ranges, runs no timing or rate analysis and does not de-duplicate repeat requests, so a script that requests a link 10,000 times with a Chrome 120 string adds 10,000 counted clicks.

How much bot traffic to expect

For scale, Imperva's 2025 Bad Bot Report put automated traffic at 51% of all web traffic and bad bots at 37%, and its 2026 edition says bots now make up the majority.

Those figures cover web traffic as a whole and say nothing about your links, so measure your flagged share with the command above.

How to spot the bot clicks that got through

The clicks API returns a timestamp, city, device, browser, referrer and visitor ID per click, enough to catch bursts within a second of a send and every link in a message clicked from one city.

The visitor journey, in the dashboard or through trackrev visitors journey, lists pixel events after a click, so a click with no pageview behind it stands out.

Revenue as the second filter

For each paid order, the attribution engine credits that visitor's unflagged clicks inside the 30-day default window under last-touch, first-touch or linear.

A scanner's click carries its own visitor ID, so it cannot take credit for someone else's sale, and credited revenue per channel stays clean even when click counts are not. Revenue attribution shows the join.

Click-based payouts are where phantom clicks cost cash. At $0.25 a click, 10,000 scripted requests earn $2,500 for someone who never sold anything.

TrackRev's affiliate commissions are calculated on payments synced from Stripe, Polar, Lemon Squeezy and Paddle, a refunded or disputed charge takes its commission back out, and a program's hold period (0 to 180 days, 0 by default) delays payout past the refund window.

The affiliate fraud guide covers the attacks that do complete a purchase.

When a header and list filter is not enough

For paid ads, the platform's own filter is the record.

Google Ads says it removes invalid clicks from metrics and billing and credits the next invoice when it finds them later, so the ad platform and a link tool report different click totals.

For attacks on a site, put a bot-management product such as Imperva's Advanced Bot Protection in front of it.

In TrackRev the link tracking redirect sets the flag, first-party tracking covers the server-side capture behind it, and the server-side versus client-side guide explains why a check on the server survives blockers.

Flagging runs on every plan, and pricing lists the rest.