--- title: Custom log drain description: Send access logs from any server, CDN, log shipper or app middleware to a Sommet drain URL as JSON, NDJSON or raw log lines. --- Use a custom drain when your site is not on Cloudflare or Vercel: Netlify, Fastly, AWS, Fly.io, Render, a VPS with nginx, or your own app. Anything that can send an HTTP `POST` works. ## Get the drain URL Open **Settings → AI bot visits**, choose **Log drain** and select **Create a drain URL**. The URL looks like this: ```txt https://api.sommet.dev/ai-traffic/drains/ ``` The token in the URL is the only credential. Store the URL as a secret, for example in an environment variable named `SOMMET_DRAIN_URL`. ## Send a test request ```bash curl -X POST "$SOMMET_DRAIN_URL" \ -H "content-type: application/json" \ -d '{"userAgent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)"}' ``` The response is `{"ok":true}`. The overview then shows one ChatGPT visit for today. ## Payload format Send the logs as the body of a `POST`. Sommet accepts: | Body | `content-type` | | ----------------------------------- | ---------------------- | | A JSON array of log objects | `application/json` | | One JSON log object | `application/json` | | Newline-delimited JSON (NDJSON) | `application/x-ndjson` | | Raw log lines, one request per line | `text/plain` | For each log object, Sommet reads the first field that exists: | Value | Fields read, in order | | --------------------- | ------------------------------------------------------------------------------------------------------------------ | | User agent (required) | `proxy.userAgent`, `userAgent`, `user_agent`, `ClientRequestUserAgent`, `http_user_agent`, `headers["user-agent"]` | | Time (optional) | `proxy.timestamp`, `timestamp`, `EdgeStartTimestamp`, `time` | | Request ID (optional) | `requestId`, `RayID`, `request_id` | - A raw text line counts when it contains a bot user agent anywhere, for example an nginx or Apache combined log line. - Time can be epoch seconds, milliseconds, nanoseconds or an ISO 8601 date. Without a time, or with a time in the future or older than 30 days, the visit counts today. - With a request ID, Sommet counts a request once even if you send it again within 6 hours. - Only requests from [AI bots](/docs/ai-bots#bots-sommet-counts) count. You can send all your logs or filter them first. ### Limits - 5 MB per request, uncompressed. Do not gzip the body. - 20,000 log entries per request. Extra entries are ignored, so split large batches. - A `404` response means the drain does not exist: the source was disconnected or changed to Cloudflare. ## Examples **Vector** [Vector](https://vector.dev) can tail an access log and post raw lines in batches. Sommet finds the user agent in each line, so this works with the default nginx and Apache formats. ```toml vector.toml [sources.access_log] type = "file" include = ["/var/log/nginx/access.log"] [sinks.sommet] type = "http" inputs = ["access_log"] uri = "${SOMMET_DRAIN_URL}" method = "post" encoding.codec = "text" framing.method = "newline_delimited" request.headers.content-type = "text/plain" batch.max_bytes = 1000000 ``` **Fluent Bit** Parse the nginx log, rename the `agent` field to one Sommet reads, and post JSON batches. ```ini fluent-bit.conf [SERVICE] Parsers_File parsers.conf [INPUT] Name tail Path /var/log/nginx/access.log Parser nginx [FILTER] Name modify Match * Rename agent user_agent [OUTPUT] Name http Match * Host api.sommet.dev Port 443 URI /ai-traffic/drains/ Format json tls On ``` **Node.js** Send bot requests from your own server. This Express middleware posts one entry per bot request and does not slow down the response. ```js server.js const AI_BOTS = /GPTBot|OAI-SearchBot|ChatGPT-User|ChatGPT Agent|ClaudeBot|Claude-User|Claude-SearchBot|anthropic-ai|PerplexityBot|Perplexity-User|Gemini-Deep-Research|Google-CloudVertexBot|Googlebot|bingbot|Applebot|meta-externalagent|meta-externalfetcher|FacebookBot|MistralAI-User|DuckAssistBot|Amazonbot|Bytespider|CCBot|Timpibot/i; app.use((req, res, next) => { const userAgent = req.get("user-agent") ?? ""; if (AI_BOTS.test(userAgent)) { fetch(process.env.SOMMET_DRAIN_URL, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ userAgent, timestamp: Date.now() }), }).catch(() => {}); } next(); }); ``` For high traffic, collect entries in memory and send one array every minute. **Nuxt or Nitro** ```ts server/middleware/ai-bots.ts const AI_BOTS = /GPTBot|OAI-SearchBot|ChatGPT-User|ChatGPT Agent|ClaudeBot|Claude-User|Claude-SearchBot|anthropic-ai|PerplexityBot|Perplexity-User|Gemini-Deep-Research|Google-CloudVertexBot|Googlebot|bingbot|Applebot|meta-externalagent|meta-externalfetcher|FacebookBot|MistralAI-User|DuckAssistBot|Amazonbot|Bytespider|CCBot|Timpibot/i; export default defineEventHandler((event) => { const userAgent = getHeader(event, "user-agent") ?? ""; if (!AI_BOTS.test(userAgent)) return; event.waitUntil( $fetch(process.env.SOMMET_DRAIN_URL!, { method: "POST", body: { userAgent, timestamp: Date.now() }, }).catch(() => {}), ); }); ``` **Next.js** In Next.js 16, the file is `proxy.ts` and the function is `proxy`. In Next.js 15 and earlier, name them `middleware.ts` and `middleware`. ```ts proxy.ts import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server"; const AI_BOTS = /GPTBot|OAI-SearchBot|ChatGPT-User|ChatGPT Agent|ClaudeBot|Claude-User|Claude-SearchBot|anthropic-ai|PerplexityBot|Perplexity-User|Gemini-Deep-Research|Google-CloudVertexBot|Googlebot|bingbot|Applebot|meta-externalagent|meta-externalfetcher|FacebookBot|MistralAI-User|DuckAssistBot|Amazonbot|Bytespider|CCBot|Timpibot/i; export function proxy(request: NextRequest, event: NextFetchEvent) { const userAgent = request.headers.get("user-agent") ?? ""; if (AI_BOTS.test(userAgent)) { event.waitUntil( fetch(process.env.SOMMET_DRAIN_URL!, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ userAgent, timestamp: Date.now() }), }).catch(() => {}), ); } return NextResponse.next(); } ``` ## Other hosts If your host or CDN can stream request logs to an HTTPS endpoint, point it at the drain URL. Choose JSON or NDJSON and turn compression off. Check that each entry carries the user agent in one of the [fields Sommet reads](#payload-format). If it cannot, run [Vector](#examples) next to your server, add the [Node.js, Nuxt or Next.js middleware](#examples) to your app, or put the [Cloudflare proxy](/docs/ai-bots/cloudflare) in front of your site.