The admin panel of this site has a "Traffic" section: page views, visits, sources, countries, devices, time on page. There is a third-party counter on the site as well, and I never planned to replace it. I wanted something else - raw page views in my own database, next to the resume download stats, so that "which source do the people who open my resume come from" is one query instead of reconciling two external reports.

I decided to record views from the browser rather than from a server middleware, and the main argument was this: bots do not execute JS, so crawlers, scanners and the uptime monitor will stay out of the stats on their own. Three weeks later the table held 151 page views, 89 of them flagged as bots. The argument was wrong. Browser tracking stayed anyway, and below is why, together with the five other traps I walked into.

How it works: the browser sends the path, the server takes the rest itself

A small script is included in the shared layout. It makes one POST when the page is shown, and a second, tiny one when the visitor leaves:

fetch('/analytics/pageview', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
        url: window.location.pathname,
        referrer: document.referrer || '',
        lang: navigator.language || '',
        client_id: clientId(),
    }),
    credentials: 'same-origin',
})
    .then(function (r) { return r.json(); })
    .then(function (d) { pvId = d && d.id ? d.id : null; })
    .catch(function () {});

The server answers with the id of the row it created - it is needed later to write the time on page into that same row. The row lands in page_views: path, language, referrer and its type, IP, country, device, browser, OS, utm tags, session id and browser id.

A rule I kept from the start: from the request body I take only what the browser alone knows - path, referrer, language, utm tags, the id from localStorage. IP, User-Agent, country and session are taken on the server. A body is trivial to forge, a connection is not. The browser id is also format-checked:

private function clientId(Request $request): ?string
{
    $cid = (string) $request->input('client_id', '');

    return preg_match('/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i', $cid)
        ? strtolower($cid)
        : null;
}

Without the check, sooner or later the column holds somebody else's 500-character string. The id itself lives in localStorage, not in the session: a session expires within a couple of hours, and a visitor returning the next day would count as new. In private mode localStorage throws, and there we simply go without a cross-visit id - that is a normal mode of operation, not an error.

Trap 1: bots do execute JS

This is the main thing I learned. Here are the 89 bot rows broken down by parsed User-Agent:

Chrome / Android / mobile / US   29
Chrome / -       / desktop / US  22
Chrome / -       / desktop / RU  20
Chrome / Linux   / desktop / CA   5
Chrome / Windows / desktop / US   4
Chrome / macOS   / desktop / CA   4
Chrome / Android / mobile / CA    3

Not a single curl, not a single python-requests. All of them are Chrome-engine browsers, a third of them mobile, more than half from US addresses. In other words, the JS was executed by search crawlers: Google has been rendering pages in Chrome for about a decade, Yandex does the same, and messenger preview crawlers belong in the same group. Their choice of pages was deliberate, too: 67 hits on the home page and 9 on the latest article - exactly the one that was fresh.

The conclusion is simple and unpleasant: browser tracking filters out scripts, but not crawlers. A User-Agent filter is needed exactly as much as in the server-side version. I already had one written for the resume stats, so I reused the same marker list - otherwise the site would hold two different opinions about who counts as a bot.

And one mistake of my own, in the schema. There is no raw User-Agent column in page_views: only the flag and the parsed fields. I did add a bot_name column in the migration and then forgot to write to it. So I know a row is a bot, but not whose bot, and I reconstruct that from the country and the parsed fields. Doing it again, I would store the raw User-Agent at least for rows with the flag raised: it costs almost no space, and parsing can always be redone - but only on future data.

Trap 2: Chrome opens pages in advance

The browser guesses navigations and loads the page before the click. JS in a prerendered page runs, but nobody has seen the page - and the counter gets a view that never happened. The standard API handles it:

if (document.prerendering) {
    document.addEventListener('prerenderingchange', registerView, { once: true });
} else if (document.visibilityState === 'hidden') {
    document.addEventListener('visibilitychange', function onVisible() {
        if (document.visibilityState !== 'hidden') {
            document.removeEventListener('visibilitychange', onVisible);
            registerView();
        }
    });
} else {
    registerView();
}

Two branches, because the cases differ: document.prerendering is prerendering proper, while a hidden tab also happens when a link is opened in the background with the middle mouse button. The view is registered only at the moment of the actual display.

And right there is the second detail: the time-on-page clock must start at display, not when the script loads. That is why start = Date.now() sits inside registerView. Otherwise a page prerendered five minutes before the click reports five minutes of "reading".

Trap 3: one display, two rows

A repeated call on the same page happens without your help: the back button restores the page from bfcache, and in some scenarios the script fires twice for a single display. Duplicates are caught with a short window:

$duplicate = PageView::where('session_id', $sessionId)
    ->where('url', $url)
    ->where('created_at', '>=', now()->subSeconds(5))
    ->value('id');

if ($duplicate) {
    return response()->json(['id' => $duplicate]);
}

What matters is not only the check but the response: return the existing id, not null. Otherwise the time-on-page beacon goes nowhere and the row stays without a duration.

Trap 4: the endpoint is open, and that is not a detail

CSRF verification had to be dropped for the two tracking routes - otherwise a view is never recorded from a page served out of cache with an expired token. The price is obvious: anyone can call the endpoint with curl and inflate the counter. The minimum I put in place is the request source and a rate limit:

private function fromOwnSite(Request $request, string $host): bool
{
    $source = $request->headers->get('Origin') ?: $request->headers->get('Referer');

    if (! $source) {
        return false;
    }

    $sourceHost = strtolower((string) parse_url($source, PHP_URL_HOST));

    return $sourceHost !== ''
        && preg_replace('/^www\./', '', $sourceHost) === preg_replace('/^www\./', '', strtolower($host));
}

Plus throttle:120,1 on the route. To be honest: this is no defence against deliberate inflation - headers are forged in one line. It is a noise filter: random scanners, other people's bots poking an endpoint found in the page source, and my own mistakes such as a stray request from the console. Good enough for a personal site's stats; a counter that money depends on needs signed requests.

Trap 5: time on page is lost for every third view

The duration is sent when the visitor leaves, and it has to go out as a beacon rather than a normal request - a normal one gets cancelled as the page goes away:

var fd = new FormData();
fd.append('pv_id', pvId);
fd.append('time', elapsed);
navigator.sendBeacon('/analytics/ping', fd);

Listen to visibilitychange and pagehide, not unload: unload does not fire in Safari and breaks bfcache on top of that. Even with a beacon, not everything arrives. In my data the duration is filled in for 35 rows out of 62 - that is, 44% of human views have none at all. Laptop lid closed, tab killed, network gone.

Two consequences for the report. Average the filled values only (whereNotNull('time_on_page')), otherwise the gaps drag it down. And define a bounce by the number of pages in the visit, not by duration: a single-page visit is a bounce, while "zero seconds" usually means a beacon that never arrived rather than an instant exit. Averaged over what did arrive, mine is 22 seconds.

Trap 6: your own service endpoints

/analytics/pageview and /analytics/ping showed up in the top pages. No human has ever opened them - the counter recorded itself. One check fixes it:

if (str_starts_with($url, '/analytics/')) {
    return response()->json(['id' => null]);
}

The lesson is not about tracking but about any home-made metric: on day one, open the report and look for your own service endpoints in it. They will be there.

Aggregates: a ten-minute cache and a structure version in the key

The summary computes about twenty breakdowns, and half of them are COUNT(DISTINCT session_id). That is expensive on every admin page load, so the whole result is cached for ten minutes, with a button to force a recount.

And here is the rake waiting for any cached report. I added new fields to the summary - an hourly heatmap and entry pages. The code was ready, the view was ready, and the page died with Undefined variable $heatmap: the report is passed to the view as an array and unpacked into variables, while the cache returns a structure built by the old version of the method, without the new keys. It lasts exactly until the TTL expires, which is why it looks like a ghost. The cure is a structure version right in the key:

/** Bumped whenever data() gains fields: the old cache then simply is not read */
private const STRUCTURE_VERSION = 'v2';

public function cached(?Carbon $from, ?Carbon $to, bool $fresh = false): array
{
    $key = 'traffic_report_'.self::STRUCTURE_VERSION.'_'
        .md5(($from?->toDateString() ?? 'all').'|'.($to?->toDateString() ?? 'all'));
    ...
}

Change the set of fields, change the constant, and old entries stop being read. More reliable than remembering to put cache:clear into the deploy.

Two other things in the report I also got right only on the second pass. First: the cache holds plain arrays, not Eloquent collections. Aggregate rows are not table records, they do not need a model, and in the cache a model is extra weight and an extra tie to a class that is still going to change. Second: bounces and entry pages are computed by the database, not by PHP:

$singlePage = DB::query()->fromSub(
    $human()->select('session_id')->groupBy('session_id')->havingRaw('COUNT(*) = 1'),
    'single'
)->count();

The temptation to pull sessions into a collection and count them in PHP is strong, but over a year that is tens of thousands of values in memory and in an IN (...) list. A subquery does the same without taking the data out of the database.

Where it landed

Three weeks in: 62 human page views, 49 visits, 38 unique browsers. Sources - 47 direct, 7 internal, 6 from external sites and 2 from search. A scale at which talking about conclusions would be ridiculous - and precisely because of that the mechanics are clearly visible: at these numbers every false row stands out by name, whereas on a site with a million views both prerendering and your own service endpoints would dissolve into noise.

Building such a counter again, the order would be this. The User-Agent bot filter goes in immediately, not after the first summary. The raw User-Agent goes into the database, at least for bot rows. Views are registered on actual display from day one, not once odd half-hour "reading sessions" appear in the report. And the structure version goes into the cache key before the report is first shown to anyone else.