Custom log drain

Send access logs from any server, CDN, log shipper or app middleware to a Sommet drain URL as JSON, NDJSON or raw log lines.

Use a custom drain when your site is not on Cloudflare or Vercel: Netlify, Fastly, AWS, Fly.io, Render, a VPS with nginx, or your own app. Anything that can send an HTTP POST works.

Get the drain URL

Open Settings → AI bot visits, choose Log drain and select Create a drain URL. The URL looks like this:

https://api.sommet.dev/ai-traffic/drains/<secret-token>

The token in the URL is the only credential. Store the URL as a secret, for example in an environment variable named SOMMET_DRAIN_URL.

Send a test request

curl -X POST "$SOMMET_DRAIN_URL" \
  -H "content-type: application/json" \
  -d '{"userAgent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)"}'

The response is {"ok":true}. The overview then shows one ChatGPT visit for today.

Payload format

Send the logs as the body of a POST. Sommet accepts:

Body	`content-type`
A JSON array of log objects	`application/json`
One JSON log object	`application/json`
Newline-delimited JSON (NDJSON)	`application/x-ndjson`
Raw log lines, one request per line	`text/plain`

For each log object, Sommet reads the first field that exists:

Value	Fields read, in order
User agent (required)	`proxy.userAgent`, `userAgent`, `user_agent`, `ClientRequestUserAgent`, `http_user_agent`, `headers["user-agent"]`
Time (optional)	`proxy.timestamp`, `timestamp`, `EdgeStartTimestamp`, `time`
Request ID (optional)	`requestId`, `RayID`, `request_id`

A raw text line counts when it contains a bot user agent anywhere, for example an nginx or Apache combined log line.
Time can be epoch seconds, milliseconds, nanoseconds or an ISO 8601 date. Without a time, or with a time in the future or older than 30 days, the visit counts today.
With a request ID, Sommet counts a request once even if you send it again within 6 hours.
Only requests from AI bots count. You can send all your logs or filter them first.

Limits

5 MB per request, uncompressed. Do not gzip the body.
20,000 log entries per request. Extra entries are ignored, so split large batches.
A 404 response means the drain does not exist: the source was disconnected or changed to Cloudflare.

Examples

Vector can tail an access log and post raw lines in batches. Sommet finds the user agent in each line, so this works with the default nginx and Apache formats.

[sources.access_log]
type = "file"
include = ["/var/log/nginx/access.log"]

[sinks.sommet]
type = "http"
inputs = ["access_log"]
uri = "${SOMMET_DRAIN_URL}"
method = "post"
encoding.codec = "text"
framing.method = "newline_delimited"
request.headers.content-type = "text/plain"
batch.max_bytes = 1000000

Parse the nginx log, rename the agent field to one Sommet reads, and post JSON batches.

[SERVICE]
    Parsers_File  parsers.conf

[INPUT]
    Name    tail
    Path    /var/log/nginx/access.log
    Parser  nginx

[FILTER]
    Name    modify
    Match   *
    Rename  agent user_agent

[OUTPUT]
    Name    http
    Match   *
    Host    api.sommet.dev
    Port    443
    URI     /ai-traffic/drains/<secret-token>
    Format  json
    tls     On

Send bot requests from your own server. This Express middleware posts one entry per bot request and does not slow down the response.

const AI_BOTS =
  /GPTBot|OAI-SearchBot|ChatGPT-User|ChatGPT Agent|ClaudeBot|Claude-User|Claude-SearchBot|anthropic-ai|PerplexityBot|Perplexity-User|Gemini-Deep-Research|Google-CloudVertexBot|Googlebot|bingbot|Applebot|meta-externalagent|meta-externalfetcher|FacebookBot|MistralAI-User|DuckAssistBot|Amazonbot|Bytespider|CCBot|Timpibot/i;

app.use((req, res, next) => {
  const userAgent = req.get("user-agent") ?? "";
  if (AI_BOTS.test(userAgent)) {
    fetch(process.env.SOMMET_DRAIN_URL, {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({ userAgent, timestamp: Date.now() }),
    }).catch(() => {});
  }
  next();
});

For high traffic, collect entries in memory and send one array every minute.

const AI_BOTS =
  /GPTBot|OAI-SearchBot|ChatGPT-User|ChatGPT Agent|ClaudeBot|Claude-User|Claude-SearchBot|anthropic-ai|PerplexityBot|Perplexity-User|Gemini-Deep-Research|Google-CloudVertexBot|Googlebot|bingbot|Applebot|meta-externalagent|meta-externalfetcher|FacebookBot|MistralAI-User|DuckAssistBot|Amazonbot|Bytespider|CCBot|Timpibot/i;

export default defineEventHandler((event) => {
  const userAgent = getHeader(event, "user-agent") ?? "";
  if (!AI_BOTS.test(userAgent)) return;
  event.waitUntil(
    $fetch(process.env.SOMMET_DRAIN_URL!, {
      method: "POST",
      body: { userAgent, timestamp: Date.now() },
    }).catch(() => {}),
  );
});

In Next.js 16, the file is proxy.ts and the function is proxy. In Next.js 15 and earlier, name them middleware.ts and middleware.

import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";

const AI_BOTS =
  /GPTBot|OAI-SearchBot|ChatGPT-User|ChatGPT Agent|ClaudeBot|Claude-User|Claude-SearchBot|anthropic-ai|PerplexityBot|Perplexity-User|Gemini-Deep-Research|Google-CloudVertexBot|Googlebot|bingbot|Applebot|meta-externalagent|meta-externalfetcher|FacebookBot|MistralAI-User|DuckAssistBot|Amazonbot|Bytespider|CCBot|Timpibot/i;

export function proxy(request: NextRequest, event: NextFetchEvent) {
  const userAgent = request.headers.get("user-agent") ?? "";
  if (AI_BOTS.test(userAgent)) {
    event.waitUntil(
      fetch(process.env.SOMMET_DRAIN_URL!, {
        method: "POST",
        headers: { "content-type": "application/json" },
        body: JSON.stringify({ userAgent, timestamp: Date.now() }),
      }).catch(() => {}),
    );
  }
  return NextResponse.next();
}

If your host or CDN can stream request logs to an HTTPS endpoint, point it at the drain URL. Choose JSON or NDJSON and turn compression off. Check that each entry carries the user agent in one of the fields Sommet reads.

If it cannot, run Vector next to your server, add the Node.js, Nuxt or Next.js middleware to your app, or put the Cloudflare proxy in front of your site.