A missing metric is not a numeric zero. Prometheus distinguishes a series with a recent value, a series marked stale, and a selector with no usable sample. Alerting then applies its own evaluation and waiting rules. Treating all these mechanisms as one universal grace period produces alerts that miss disappearing targets or report an outage without establishing what actually disappeared.

The synthetic input is value ten at minute zero, stale at minute one, then missing samples. Instant absence starts at minute one. The range at minute two includes the original sample; the left-open range at minute three excludes it. The absence alert is pending at three and four, then firing at five after for two minutes. A present zero-valued metric and a present up=0 series are both different from absence.
These are the article’s explicit synthetic samples, not a universal failed-scrape or discovery-removal timetable. At 3m, the range (0m,3m] excludes the sample at 0m; for: 2m is a separate alert layer. Historical local promtool evidence is unchanged. Live collection, expected membership and notification delivery remain separate checks.

A marker is not a five-minute timer

An instant selector normally uses the newest eligible sample within its lookback period, which defaults to five minutes. A stale marker tells the engine to stop returning that series in instant selections after the marker. It does not require the previous value to remain visible for another five minutes. Historical samples are still stored and can remain relevant to range queries.

Collection circumstances matter. A series no longer exported, a target removed from discovery, and an exporter assigning its own timestamps need not follow identical timing. Explicit timestamps and staleness configuration can alter behavior. Determine which path your exporter uses before predicting disappearance time from the scrape interval alone. A configurable lookback is a query selection rule, not a universal service outage detector.

Choose the absence question deliberately

absent(selector) returns a value of one when the selector has no elements. It returns an empty vector when matching data exists, even if that data’s numeric value is zero. absent_over_time(selector[3m]) asks whether the selected range contains any samples. A real sample from before a stale marker can keep that range-based absence expression empty until the sample leaves the window.

The alert rule’s for duration is another layer: the expression must keep returning the relevant alert label set at successive evaluations before it fires. It does not configure staleness or extend the query’s lookback. Combining a three-minute absence window with a waiting duration delays notification further, with evaluation cadence and collection timing also contributing. Choose that delay from the detection objective.

When it breaks

absent(up{job="api"}) is not a failed-scrape alert. If the configured targets are present with up=0, the selector still has elements, so absence is false. A comparison against zero addresses that case. If a target disappears from discovery altogether, checking only zero can miss it once its series stops appearing. Monitor expected membership separately when that distinction matters.

A broad selector also cannot enumerate missing members. If one replica survives, absent over the whole job remains empty even when several expected replicas are gone. Match a deliberately known identity or compare observed series with an authoritative inventory. A label rename can look like an outage under the old matcher while the process continues working under a new identity.

Check it quickly

Save this as stale.py and run python3 stale.py with an installed promtool on PATH, or set PROMTOOL to its executable path. The recorded earlier execution used Python 3.12.14 and promtool 3.14.0; it was not rerun during this source review. The real query engine evaluates synthetic samples in a temporary directory; no server, listener, container or external service is started.

from pathlib import Path
from tempfile import TemporaryDirectory
import os
import shutil
import subprocess

# Uses a real, installed promtool; no server or Docker runtime is started.
promtool = shutil.which(os.environ.get("PROMTOOL", "promtool"))
if not promtool:
    raise SystemExit("Set PROMTOOL to an installed promtool executable")

tests = r"""evaluation_interval: 1m
rule_files: [rules.yml]
tests:
  - interval: 1m
    input_series:
      - series: "up{job=\"api\",instance=\"gone\"}"
        values: "1 0 0 0 0"
      - series: "service_events_total{job=\"api\",instance=\"gone\"}"
        values: "10 stale _ _ _"
      - series: "service_events_total{job=\"api\",instance=\"zero\"}"
        values: "0 0 0 0 0"
    promql_expr_test:
      - expr: "service_events_total{job=\"api\",instance=\"gone\"}"
        eval_time: 1m
        exp_samples: []
      - expr: "up{job=\"api\",instance=\"gone\"}"
        eval_time: 1m
        exp_samples:
          - labels: "{__name__=\"up\",job=\"api\",instance=\"gone\"}"
            value: 0
      - expr: "absent(service_events_total{job=\"api\",instance=\"gone\"})"
        eval_time: 1m
        exp_samples:
          - labels: "{job=\"api\",instance=\"gone\"}"
            value: 1
      - expr: "absent_over_time(service_events_total{job=\"api\",instance=\"gone\"}[3m])"
        eval_time: 2m
        exp_samples: []
      - expr: "absent_over_time(service_events_total{job=\"api\",instance=\"gone\"}[3m])"
        eval_time: 4m
        exp_samples:
          - labels: "{job=\"api\",instance=\"gone\"}"
            value: 1
      - expr: "absent(service_events_total{job=\"api\",instance=\"zero\"})"
        eval_time: 4m
        exp_samples: []
      - expr: "absent(up{job=\"api\"})"
        eval_time: 4m
        exp_samples: []
      - expr: 'absent_over_time(service_events_total{job="api",instance="gone"}[3m])'
        eval_time: 3m
        exp_samples:
          - labels: '{job="api",instance="gone"}'
            value: 1
      - expr: 'ALERTS{alertname="BusinessMetricMissing",alertstate="pending"}'
        eval_time: 3m
        exp_samples:
          - labels: '{__name__="ALERTS",alertname="BusinessMetricMissing",alertstate="pending",job="api",instance="gone"}'
            value: 1
    alert_rule_test:
      - eval_time: 4m
        alertname: BusinessMetricMissing
        exp_alerts: []
      - eval_time: 5m
        alertname: BusinessMetricMissing
        exp_alerts:
          - exp_labels: {job: api, instance: gone}
            exp_annotations: {}
"""
rules = r"""groups:
  - name: absence
    rules:
      - alert: BusinessMetricMissing
        expr: absent_over_time(service_events_total{job="api",instance="gone"}[3m])
        for: 2m
"""

with TemporaryDirectory(prefix="prometheus-query-check-") as directory:
    root = Path(directory)
    (root / "tests.yml").write_text(tests, encoding="utf-8")
    if rules:
        (root / "rules.yml").write_text(rules, encoding="utf-8")
    result = subprocess.run([promtool, "test", "rules", "tests.yml"],
                            cwd=root, text=True, capture_output=True, timeout=45)
    print(result.stdout, end="")
    if result.returncode:
        print(result.stderr, end="")
        raise SystemExit(result.returncode)

The test prints SUCCESS. At minute one, the marked business series is absent from an instant selection, while its separately supplied up series exists with value zero. Instant absent returns one. At minute two, the three-minute range still contains the original business sample, so absent_over_time returns nothing; at minute three the left-open boundary already excludes the original sample, and at minute four it still returns one.

The continuously exported zero-valued business series is never absent, and the job-wide up selector is not absent either. These assertions distinguish missing values from failing values instead of converting both to zero. The temporary test files are removed on exit. The added alert starts pending at minute three, is not firing at minute four, and fires at minute five after its separate two-minute for duration. Assertions check both pending state and firing timing. Live collection, target removal and notification delivery remain separate integration tests.

Sources