IndexNow in Laravel: two traps nobody warns you about

Normally a search engine finds a new page on its own: it crawls on a schedule, reads the sitemap, compares. On a small site that takes anywhere from a few days to a couple of weeks. IndexNow flips it around: you submit the list of changed URLs yourself instead of waiting for a crawl.

The protocol is almost embarrassingly simple - a single POST with JSON. The difficulty is not in the protocol but in two places where it is easy to shoot yourself in the foot. Those are below.

The key and the verification file

You need a key: any string of 8-128 hexadecimal characters.

php -r 'echo bin2hex(random_bytes(16));'

Now the part that most often explains why nothing works: the same key must be served at https://yourdomain/<key>.txt. That is how the engine checks the domain is yours and not some stranger submitting other people's URLs. The file contains exactly the key, no newlines, no quotes.

The key is not a secret - the protocol publishes it and anyone can read it. So the file lives in the repository, and the value is duplicated in .env:

'indexnow' => [
    'key' => env('INDEXNOW_KEY'),
],

An empty key makes a convenient off switch: no value, no submissions.

The submit command

There are three endpoints: Bing, Yandex and api.indexnow.org. Technically any one of them is enough, since participants share submissions with each other. In practice I send to all three: it costs nothing and the behaviour is more predictable.

$response = Http::timeout(10)->asJson()->post($endpoint, [
    'host'    => parse_url(config('app.url'), PHP_URL_HOST),
    'key'     => $key,
    'urlList' => array_values($urls),
]);

A 200 means accepted, a 202 means accepted while the key is still being verified. Both count as success. Everything else goes to the log but must not throw: a submission that did not go through is no reason to break saving an article.

The command has two modes: explicit URLs as arguments, and --all, which collects every public page - the landing, the blog index and all published posts, in both languages. The second mode exists for the weekly safety net.

Trap one: the view counter

The obvious move is to hook submission onto a model observer:

public function saved(BlogPost $post): void
{
    // do not do this
    Artisan::call('indexnow:ping', ['urls' => [url('/blog/'.$post->slug)]]);
}

And it works right up until the first reader opens the article. My view counter increments on the model itself:

$this->increment('views');

And increment() in Eloquent is a regular update that fires the updated and saved events. Which means every single page view would trigger three HTTP requests to search engines. A hundred views a day is three hundred pointless requests, and a very quick way to stop being listened to.

The fix is to check what actually changed:

$changed = array_diff(array_keys($post->getChanges()), ['views', 'updated_at']);

// only the counter moved, nothing to announce
if (! $post->wasRecentlyCreated && ! $changed) {
    return;
}

// a draft that stayed a draft
if (! $post->status && ! in_array('status', $changed, true)) {
    return;
}

The second check has a subtlety of its own. Unpublishing is submitted on purpose: it is useful for the engine to learn the page now returns a 404. Edits to a draft, on the other hand, should go nowhere.

Trap two: the test suite

The observer went in, the tests stayed green - and the run went from one second to eighteen. Same cause: every test that created a post dutifully called three real search engines and waited for the answers.

One line in phpunit.xml fixes it:

<env name="INDEXNOW_KEY" value=""/>

The empty key disables submission, which we arranged for at the very beginning. Where submission itself needs testing, the key is set inside the test and requests are intercepted with Http::fake().

Keeping the admin panel out of the wait

Three HTTP requests with a 10 second timeout inside a form save means up to half a minute of waiting in the worst case. Setting up a queue just for this is overkill; a worker is one moving part too many on a small site.

defer() fits: the callback runs after the response has already gone to the browser.

defer(function () use ($urls) {
    Artisan::call('indexnow:ping', ['urls' => $urls]);
});

In tests deferred calls need to run immediately or there is nothing to assert against - that is what $this->withoutDefer() is for.

The schedule as a safety net

A one-off submission can fail: the network blinked, the service returned an error. So once a week the full list goes out:

Schedule::command('indexnow:ping --all')
    ->weeklyOn(1, '05:00')
    ->withoutOverlapping();

withoutOverlapping() here is not about long-running tasks. My crontab ended up with two schedule:run entries for the same application - one from the hosting panel, one added outside it. Both fire in the same second, and without a lock every task would run twice.

What to expect

IndexNow does not guarantee indexing and does not affect rankings. It shortens the path: the engine learns about a change immediately instead of whenever it gets around to it. For a blog that updates weekly, that is the difference between an hour and a week.

Google does not support the protocol, so your sitemap is not going anywhere. IndexNow complements it rather than replacing it.