Hey everybody,

Been running a Next.js app on Vercel for a while and a few months ago I started noticing requests in my function logs that my analytics never showed. No GA event, no pageview, nothing. When I dug in, it was GPTBot, PerplexityBot, ClaudeBot, a bunch of others.

What surprised me wasn't that they were crawling. It was that they kept hitting the same pages over and over. That pattern made me think some of these aren't just training crawls. Perplexity-User and ChatGPT-User in particular look like real-time fetches, like someone asked a question and the LLM pulled my page to answer it.

So I built a middleware layer that catches these and logs them: bot name, page path, country, timestamp. About 50 bot signatures covered right now across OpenAI's three bots, Anthropic, Perplexity, Grok, Meta, Amazon, plus agent frameworks like LangChain and CrewAI.

For Vercel specifically, there's also a log drain setup that takes one CLI command and no code changes at all. It just parses your existing Vercel logs in the background.

I've been running it on my own site for a few months. Some things I've seen that I didn't expect:

• Perplexity-User is way more active than I expected on certain pages

• Some pages get zero organic traffic but consistent bot hits from chatGPT-User bot, which tells you which pages LLM users are interested in

• Agent frameworks hitting API endpoints are a totally different category of traffic

Sharing this in case anyone else has been curious about this. Would love to hear if you've noticed similar patterns or if this kind of visibility would actually be useful in your workflow.

A few things I'm genuinely trying to figure out before I build more: Do you actively track this already? Would you want a dashboard for it, or is a weekly digest email more useful? And is the no-code log drain setup interesting or would you just drop the middleware snippet into your app?

If you want to see what it looks like on your own site, drop a comment or email me at [ shilpa@agentarena.site ]. Happy to run a free report for anyone who wants to see their own data.