Bot traffic filtering for links: what gets flagged and what slips through

Bot traffic filtering for links is the 3-test check that decides, request by request, whether a hit on a short link counts as a click.

We built TrackRev to flag a request when its user agent matches the isbot 5.2.1 pattern list, when the user agent is empty, or when a purpose header marks it as a prefetch.

A flagged request still gets its 302 redirect, is stored with is_bot true, and stays out of every report.

That removes Googlebot/2.1, the Slack and Facebook preview fetchers, curl/8.4.0 and python-requests/2.31.0 scripts, and Chrome's speculative loads. A scanner with an ordinary Chrome 120 user agent passes, and finding those takes a look at your own click data.

Where do non-human clicks come from?

Non-human clicks are requests to a link made by software instead of a person.

The Media Rating Council's invalid traffic addendum (June 2020 update) calls what list-based filtering finds general invalid traffic: bots and spiders, non-browser user agents, pre-fetch and pre-render traffic, and known data-center traffic.

Sophisticated invalid traffic takes advanced analytics and human review. The standard covers ad impressions, and the split carries over to links. The table below lists 7 sources.

Slack's robots page says Slackbot-LinkExpanding 1.0 fetches links posted into channels with HTTP Range headers and caches responses across the service for around 30 minutes. Meta documents facebookexternalhit/1.1 as the crawler behind previews on Facebook, Instagram and Messenger.

Mailchimp's bot activity page names anti-spam filters at email providers, Apple's Mail Privacy Protection and link preview generators as tools that often click links or open emails before the recipient does.

Its bot-filtered email click data starts on August 10, 2025, and it hides flagged activity without deleting it, the design TrackRev uses. The newsletter guide and the Slack and Discord guide go deeper on those channels.

How does TrackRev filter bot traffic on links?

TrackRev's filter is a 3-part check run on every request to a short link, and any single part flags the request.

• Step 1: a missing or blank user-agent header is flagged.
• Step 2: the user agent is checked against isbot 5.2.1, the version we use because TrackRev's lockfile resolves it. It is an open-source list of 209 patterns built from public crawler lists such as monperrus/crawler-user-agents and user-agents.net. It matches names like Googlebot and HeadlessChrome, any string where bot ends a word, and any string containing http, which is how Googlebot, Slackbot-LinkExpanding and facebookexternalhit point to their documentation.
• Step 3: the headers x-purpose, purpose and sec-purpose are read, and a value containing prefetch flags the request. Chrome sends Sec-Purpose: prefetch for a Speculation Rules prefetch and prefetch;prerender for a prerender, and MDN says a server can treat such requests differently when counting visits.

What does a flagged request change?

Why does TrackRev keep flagged clicks?

Keeping flagged clicks means storing the rows with is_bot true while leaving them out of every report. Google Analytics 4 excludes known bots automatically, using Google research and the International Spiders and Bots List kept by the Interactive Advertising Bureau.

The exclusion cannot be switched off, and Google does not report how much it removed. TrackRev keeps the rows.

The clicks API leaves them out unless you pass bots=include, and the CLI has the same switch, so you can measure your own bot share.

The API needs a paid plan and a secret key, and --all stops after 200 pages of up to 500 clicks, so on a large workspace add --link with one link ID.

trackrev clicks --bots --all --json \
  | jq '.clicks | group_by(.is_bot) | map({is_bot: .[0].is_bot, requests: length})'

What can a user-agent list not see?

A user-agent list is a set of patterns matched against the User-Agent header, so a list catches only bots that announce themselves.

The isbot 5.2.1 README says the package identifies good bots that announce themselves and does not try to recognize malicious bots or programs posing as real users.

We built TrackRev to store each click's IP address and user agent but to set the flag from the 3 tests alone.

The filter checks no IP ranges, runs no timing or rate analysis and does not de-duplicate repeat requests, so a script that requests a link 10,000 times with a Chrome 120 string adds 10,000 counted clicks.

For scale, Imperva's 2025 Bad Bot Report put automated traffic at 51% of all web traffic and bad bots at 37%, and its 2026 edition says bots now make up the majority.

Those figures cover web traffic as a whole and say nothing about your links, so measure your flagged share with the command above.

How do you spot the bot clicks that got through?

A bot click that got through is a request with an ordinary browser string such as Chrome 120, so the click needs a behavioral test.

The clicks API returns 6 fields per click: a timestamp, city, device, browser, referrer and visitor ID. Six fields are enough to catch bursts within 1 second of a send and every link in a message clicked from 1 city.

The visitor journey, in the dashboard or through trackrev visitors journey, lists pixel events after a click, so a click with 0 pageviews behind it stands out.

How much do bots distort a conversion rate?

A conversion rate is signups divided by counted clicks, so every counted bot click lowers the rate.

Why is revenue the second filter?

Revenue is the second filter because only a paid order earns credit. When we credit a paid order, the attribution engine takes that visitor's unflagged clicks inside the 30-day default window under last-touch, first-touch or linear.

A scanner's click carries its own visitor ID, so it cannot take credit for someone else's sale, and credited revenue per channel stays clean even when click counts are not. Revenue attribution shows the join.

Click-based payouts are where phantom clicks cost cash. At $0.25 a click, 10,000 scripted requests earn $2,500 for someone who never sold anything.

TrackRev's affiliate commissions are calculated on payments synced from 6 processors (Stripe, Paddle Billing, Polar, Lemon Squeezy, Creem and Dodo Payments), a refunded or disputed charge takes its commission back out, and a program's hold period (0 to 180 days, 0 by default) delays payout past the refund window.

The affiliate fraud guide covers the attacks that do complete a purchase.

When is a header and list filter not enough?

A header and list filter is not enough for paid ads or for attacks on a site. For paid ads, the platform's own filter is the record.

Google Ads says it removes invalid clicks from metrics and billing and credits the next invoice when it finds them later, so the ad platform and a link tool report different click totals.

For attacks on a site, put a bot-management product such as Imperva's Advanced Bot Protection in front of it.

TrackRev is SaaS affiliate software for indie hackers and small teams, with link tracking built in.

That link tracking redirect sets the flag, first-party tracking covers the server-side capture behind it, and the server-side versus client-side guide explains why a check on the server survives blockers.

Flagging runs on all 3 plans, and pricing lists the rest.