[lognorth] × Grok Bot

~ $ man 7 grok-bot

Next gen devops.

A LogNorth alert starts a Grok Bot routine. The bot finds the cause in your logs, ships the fix, and watches LogNorth until the alert clears. You wake up to a closed incident.

The loop: LogNorth alerts, a webhook wakes Grok Bot, it investigates, fixes, opens a pull request, ships, and LogNorth verifies the fix until the alert clears. your app shop-prod ■ in production [lognorth] 01 alert /checkout 8% vs 1% [lognorth] 02 webhook POST to the routine grok bot 03 investigate logs, trace, commits grok bot 04 fix branch + failing test grok bot 05 pull request #412 with the evidence grok bot 06 ship CI, merge, deploy [lognorth] 07 verify 0.9% since the deploy [lognorth] 08 closed alert clears, report sent
  1. 01 [lognorth] alert /checkout 8% vs 1%
  2. 02 [lognorth] webhook POST to the routine
  3. 03 grok bot investigate logs, trace, commits
  4. 04 grok bot fix branch + failing test
  5. 05 grok bot pull request #412 with the evidence
  6. 06 grok bot ship CI, merge, deploy
  7. 07 [lognorth] verify 0.9% since the deploy
  8. 08 [lognorth] closed alert clears, report sent
LogNorth watches and judges. Grok Bot does the work. Neither one needs you.

[01] two ways in

On an alert. Or on your word.

Grok Bot acts on its own when LogNorth pages it. Ask it anything the rest of the time. Same tools, same logs.

autonomous · a webhook woke it
03:12 [shop-prod] Spike on /checkout
      8.1% errors, normally 1.0%

bot   on it. no one asked.
      timeline, logs, trace, commits
      fix → #412 → CI → deploy
      watching /checkout...

03:36 0.9% since the deploy. closed.
on command · you asked
you   why did signups drop after lunch?

bot   /signup: 212 an hour until 12:40,
      then 31. 0 errors. uptime: up.
      The app is fine. The traffic stopped.
      #398 moved the form to /join at 12:38.
      Your newsletter still links /signup.

you   ship a redirect
bot   #413 deployed. /signup is back to 204/h.

[02] one thread

Six sources. One thread.

A dashboard shows you one of these at a time. The bot holds all six, and follows the thread between them: event 48213, commit a3f9c1e.

alerts
critical  /checkout
8.1% errors, normally 1.0%
since 03:00
logs
03:02:11 POST /checkout 500
03:02:12 POST /checkout 500
212× Stripe::TimeoutError
trace · event 48213
POST /checkout        2031ms
├ Cart.load             18ms
└ Stripe.charge  2002ms ✗
code
app/payments/charge.rb
40  def charge(order)
41    Stripe.timeout = 2
github
a3f9c1e tune stripe timeouts
merged 02:58, deployed 03:00
#412 restore the 30s timeout
uptime
▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮
up · 84ms · 99.98%
it answered. it failed.

[03] a night on call

03:12 to 03:36. Nobody woke up.

03:12, your phone buzzes. Find the laptop. Open the dashboard. Scroll. Search. Fix. Wait for CI. Deploy. Stare at the graph. It's 04:30.

grok bot · routine lognorth-oncall · an example run
03:12:04  trigger  webhook from lognorth
                   [shop-prod] Spike on /checkout: 8.1% errors, normally 1.0%
03:12:06  tool     lognorth.endpoint_timeline    → flat at 1% until 03:00, then 8%
03:12:09  tool     lognorth.search_logs          → 212 failures, all Stripe::TimeoutError
03:12:11  tool     lognorth.get_event 48213      → app/payments/charge.rb:41
03:12:14  git      log --since 02:00             → a3f9c1e cut the Stripe timeout to 2s
03:13:40  fix      charge.rb: timeout back to 30s, test for a 3s Stripe answer
03:13:52  pr       #412 Restore the 30s Stripe timeout   evidence: event 48213
03:18:27  ci       passed                        4m 35s
03:18:31  merge    #412 into main
03:21:05  deploy   shop-prod → healthy
03:31:10  tool     lognorth.endpoint_timeline    → 0.9% since 03:21
03:36:12  tool     lognorth.list_alerts          → no alerts
03:36:13  post     /checkout failed 8% from 03:00, normally 1%. a3f9c1e cut the Stripe
                   timeout to 2s. #412 restored 30s, deployed 03:21. 0.9% since.
03:36:13  done     24 min, alert to verified fix. Nobody woke up.
24 minalert to verified fix
0people paged
1run per incident, not 200

The last step is the one other tools skip. The bot does not stop at "deployed". It reads the endpoint in LogNorth until the error rate is back to its own normal, and reverts when it is not.

[04] set it up

Two messages.

Coming soon: an official LogNorth connector for Grok Bot. One click, and it asks for your URL and agent key. Until then, two messages do the same job.

Before you start, give Grok Bot your repository, and the way you deploy: a CLI on its computer, or the connector for your host.

1. connect grok bot to lognorth

message to grok bot
Install this connector: lognorth.com/agents

It reads the setup on that page and asks for two things: your LogNorth URL, and an agent key from Settings > Developer. Paste both. It checks the key and lists your apps.

2. send it the alerts

message to grok bot
Configure LogNorth alerts via webhook

Grok Bot creates a routine with a webhook trigger and gives you its URL and key. In LogNorth, open Settings > Alerts, choose Webhook, and paste them. Click Test notification: a run appears in the routine's history.

optional: say what the routine does

Grok Bot writes the routine's instruction for you. To run the whole loop, replace it with this one:

routine instruction
A LogNorth alert arrived. The body has a title, a message, and a
/lognorth:investigate line with the path or issue.

1. Find the cause with the LogNorth tools: list_alerts, endpoint_timeline
   for when it started, search_logs for what failed, get_event for one
   full trace and error_file:error_line. Check the commits just before.
2. Fix it on a branch. Add a test that fails without the fix.
3. Open a pull request with the evidence: the error rate against normal,
   the event id, and the commit that caused it.
4. When CI passes, merge and deploy.
5. Watch LogNorth for 15 minutes: endpoint_timeline for the path, then
   list_alerts. The alert clears after two clean checks.
6. If the error rate is not back to normal, revert and say so.

Post one message at the end: the cause, the PR, and the numbers after
the deploy. Say when the cause is a guess.

The server must be reachable from the internet, because Grok Bot runs in the cloud. Give the bot its own agent key, so you can revoke it without touching your laptop's.

[05] what lognorth sends

One POST per alert.

webhook body
POST https://api2.cursor.sh/automations/webhook/...
Authorization: Bearer crsr_...

{
  "source":  "lognorth",
  "title":   "[shop-prod] Spike on /checkout",
  "message": "8.1% errors, normally 1.0% ... /lognorth:investigate /checkout",
  "text":    "<title>\n\n<message>",
  "sent_at": "2026-09-24T14:05:12Z"
}

The same webhook works for anything that takes a JSON POST. text is the shape Slack incoming webhooks read.

[06] the leash

You set the leash.

  • LogNorth stays read-only. The bot cannot mute an alert to make it go away. It has to fix the endpoint.
  • The agent key reads logs. It cannot send events or sign in to LogNorth.
  • Merge and deploy are the access you give Grok Bot, not LogNorth. Want a person in the loop? Stop the routine at step 3, and it opens the PR and waits for you.
  • Your logs stay on your server. The bot reads the answers to its questions, nothing more.

Alerts, logs, traces, uptime, code, and GitHub, in one bot's hands.

The devops panacea? Close. It still runs your tests.