When Instagram links die

Here’s the uncomfortable truth about the Trad Directory: every artist and every shop in it is, underneath, an Instagram handle. That’s the source of truth for this entire community — I’ve written before about how much I hate that Meta holds all the cards. And the thing about Instagram handles is that they die. People rename accounts, delete them, get banned, or a freed-up username gets grabbed by a completely different person. A directory full of dead links is worse than no directory.

So I needed a link-rot detector. My first one was a confident liar. My second one is honest, and honesty turned out to be the whole design.

The first checker lied to me

Version one was the obvious thing: a cron job that walks the list, fires a request at instagram.com/<handle>, and records whether it’s broken. Simple. Wrong.

It produced false positives constantly, and the reason is fundamental: Instagram’s public profile page returns a 200 OK for handles that exist and handles that don’t. It’s a client-rendered shell. Half the time it also bounces you through a login wall or a redirect. A plain HTTP check of that page carries zero live-or-dead signal. I was basically flipping a coin and writing the result to the database.

The “fix” wasn’t a fix. It was realizing the entire approach couldn’t work. You cannot check if an Instagram profile exists by loading its webpage.

The one endpoint that tells the truth

There’s exactly one unauthenticated endpoint that gives a real answer — Instagram’s internal web_profile_info JSON API, the thing the site itself calls to hydrate a profile. The catch is you have to look like a browser or it slams the door. The header comment in the new probe is basically the whole Feb-to-August arc written in prose:

// IG's public profile HTML (instagram.com/<handle>/) is a client-rendered
// shell that returns 200 for BOTH real and nonexistent handles, so it carries
// no live/dead signal. The one unauthenticated endpoint that does is the
// internal web_profile_info JSON API — called with browser-like Fetch-Metadata
// headers (x-ig-app-id + Sec-Fetch-* + a same-origin Referer); without them IG
// replies 400 "SecFetch Policy violation".

The weird part: a 400 can mean alive

This is my favorite bug in the whole project. Some Instagram accounts — specifically certain business-category ones — trip a server-side serializer bug where the API returns a 400 with a JSON body like {status: "fail"}… even though the profile absolutely exists. A profile that doesn’t exist returns a clean 404. So a JSON fail envelope isn’t an error to me — it’s proof of life:

if (status === 400) {
  // A JSON fail envelope means IG found the profile but its serializer
  // errored (business-category bug) -> the profile exists -> alive. A
  // non-JSON 400 is the SecFetch request rejection -> unknown.
  if (isJson) {
    const body = await res.json().catch(() => null);
    if (isFailEnvelope(body)) {
      return { result: "alive", statusCode: 400, detail: "profile exists (IG serializer error)" };
    }
    return { result: "unknown", statusCode: 400, detail: "json 400" };
  }
  return { result: "unknown", statusCode: 400, detail: "request rejected (400)" };
}

Reading a 400 as “alive” feels insane until you internalize the rule: a dead handle always 404s. Anything that isn’t a 404 is, at worst, ambiguous — never proof of death.

Confidence, not certainty

That word “ambiguous” is the key to the redesign. One probe is never enough to declare a link dead, because Instagram will throttle you, or a login wall bounces you, and you get noise. So links don’t have a boolean is_broken anymore. They have a status with a confidence state machine behind it:

export type LinkStatus = "unchecked" | "alive" | "suspect" | "dead" | "unknown";

A single dead probe demotes a link to suspect, not dead. It takes three consecutive dead probes to actually call it:

if (probe === "dead") {
  const streak = prevStreak + 1;
  const status = streak >= CONFIRM_DEAD_AFTER ? "dead" : "suspect";
  return { status, fail_streak: streak, /* ... */ };
}

An alive result resets the streak to zero. And crucially, an unknown — a throttle, a login wall, a timeout — preserves the previous verdict and just retries soon. Ambiguity never counts against a link. The recheck cadence keys off status too: alive links get rechecked in ~45 days, suspects in 3, confirmed-dead in 21, unknowns in 6 hours.

Small waves, or you get IP-banned

Instagram rate-limits that endpoint hard, so the cron doesn’t blast the whole list. It probes tiny “waves” with jitter, and it has a circuit breaker:

const WAVE_SIZE = 12;         // handles probed per cron tick (kept under IG's per-IP throttle)
const MIN_DELAY_MS = 3000;
const MAX_DELAY_MS = 6000;
const RATE_LIMIT_ABORT = 3;   // consecutive rate-limit/timeout unknowns -> stop wave

Twelve handles per tick, three-to-six seconds apart, and if it eats three rate-limits in a row it bails and lets the next cron tick pick it back up rather than hammering through a block. It prioritizes never-checked handles first, then the most overdue.

Being honest with the database, too

The nastiest part was cleaning up after my own lies. The first migration seeded the new status column from the old untrustworthy is_broken flag — so every row confidently inherited a fake “alive” verdict and a fake check date it never earned. That’s the exact dishonesty I was trying to kill. So there’s a second migration that does nothing but wipe the slate:

-- Clean-slate reset: no inherited verdicts, no fake check dates. Everything
-- becomes due now so the reliable checker re-verifies it over the next waves.
UPDATE link_check_results
  SET status = 'unchecked',
      fail_streak = 0,
      last_alive_at = NULL,
      checked_at = NULL,
      is_broken = false,
      next_check_at = now();

Same principle carries into new additions. When I hand-add an artist I just looked at their profile, so I know it’s real — but the checker hasn’t probed it, so it doesn’t get to claim it did. New handles start as unchecked, deprioritized for 45 days, verdict blank:

// a hand-added handle was just eyeballed, so the checker shouldn't prioritise it
// — but we don't claim a verdict it never actually probed.
status: "unchecked",
fail_streak: 0,
last_alive_at: null,
checked_at: null,

The whole thing runs through Sentry now too, with real tracing and metrics on every probe, transition, and wave — because when it’s silently deciding links are dead in the background at 3am, I want to know exactly what it saw.

The takeaway

The second checker isn’t smarter than the first, it’s just more honest about what it doesn’t know. dead means “I probed it three times and it 404’d three times.” unknown means “Instagram wouldn’t give me a straight answer, so I’m not going to pretend.” That distinction is the entire feature. A link checker that guesses is worse than useless — it rots your data while telling you it’s fine.

Where do the dead ones surface? In the admin I’d already built months earlier to manage all this — the confirmed-dead handles show up as a status pill and a deep-link right in the data browser, so fixing rot is a click, not a SQL query. That admin started life as a scrappy little form and quietly became a whole CMS, which is a rabbit hole of its own.