Observability for Geocoding Pipelines: The Metrics That Matter

A geocoding pipeline that "feels fine" can be silently degrading for weeks before someone notices. Match rates drift from 95% to 88% over a quarter; cache hit rate quietly falls because a key normalization regression made cache lookups miss; the API bill creeps up 20% and nobody connects it to the new product feature that doubled query volume. None of these get spotted without metrics.

This post is the practical version: the 8 metrics every geocoding pipeline should emit, the 3 alerts worth a pager page, and the structured logging pattern that makes debugging "why did this row fail?" a 30-second query instead of a 30-minute archaeology dig.

The 8 metrics

Emit these from every geocoding worker. Tag by worker_id, tier, and pipeline_name so you can slice them in your dashboard.

1. Request rate (calls/sec)

The simplest one. Total geocoding API calls per second. Bucket by status code (200, 4xx, 5xx) so a single line graph tells you "we're healthy" or "something broke."

from prometheus_client import Counter

geocode_requests = Counter(
    'geocode_requests_total',
    'Geocoding API calls',
    ['status_code', 'endpoint']
)

# In the hot path
geocode_requests.labels(status_code='200', endpoint='geocode').inc()

What it tells you: traffic shape. Spikes correlate with your product features (CSV uploads, batch jobs, new customer onboarded).

2. Success rate (per-status-code histogram)

Of the requests that completed, what % were 2xx vs 4xx vs 5xx? A 5% 503 rate means the upstream is sick. A 5% 400 rate means your input quality is degrading.

The crucial split: don't lump all errors together. A pipeline at "85% success rate" tells you nothing; "85% success, with 12% 400s and 3% 503s" tells you to fix your input parsing, not retry harder.

3. p50 / p95 / p99 latency

Don't average. Averages lie about distribution shape. A pipeline with p50 = 50ms and p99 = 5000ms looks "fast on average" but is unusable for real-time. The p99 is what your users feel during the worst 1% of requests.

from prometheus_client import Histogram

geocode_latency = Histogram(
    'geocode_latency_seconds',
    'Geocoding call latency',
    buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10],
)

with geocode_latency.time():
    result = call_api(addr)

Why p99 matters more than mean: detailed math is in p99 Latency in Geocoding, but the headline is that for a real-time API, the p99 is what determines your worst-case user experience, and it's almost always 10–20× the median.

4. Cache hit rate

cache_lookups = Counter('geocode_cache_lookups_total', 'Cache lookups', ['result'])

if key in cache:
    cache_lookups.labels(result='hit').inc()
    return cache[key]
cache_lookups.labels(result='miss').inc()

Hit rate over time = hits / (hits + misses). Trends matter more than absolute values. A drop from 90% to 70% means either:

• Cache TTL is too short (entries expiring before re-use)
• Cache key normalization regressed (different keys for the same input)
• Input mix changed (new customer with different addresses)

Catching the regression in metrics > catching it in next month's bill.

5. Match rate

Of successful API calls (200 OK), what % returned at least one result? "Match" rate is different from "success" rate — a 200 with empty results: [] is a successful API call but a failed match.

match_results = Counter(
    'geocode_matches_total',
    'Geocoding match results',
    ['matched']  # 'true' | 'false'
)

result = call_api(addr)
match_results.labels(matched='true' if result else 'false').inc()

Healthy match rates depend on input quality:

• B2B mailing list (after parsing): 90–95%
• Consumer signup forms: 85–92%
• Scanned/OCR'd documents: 60–80%
• Foreign-language addresses outside main markets: variable

Drift in match rate is the canary for upstream data quality issues. A pipeline at 95% that drops to 88% over a month means something changed about your inputs.

6. Confidence score distribution

For matches, the geocoder returns a confidence score. Histogram of scores tells you the *quality* of matches, not just the quantity:

confidence_dist = Histogram(
    'geocode_confidence_score',
    'Distribution of confidence scores',
    buckets=[0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 1.0]
)

if result and 'accuracy_score' in result:
    confidence_dist.observe(result['accuracy_score'])

A pipeline where 80% of matches are >0.9 confidence is healthy. One where matches concentrate at 0.6–0.8 is geocoding badly even though "match rate" looks fine. Worth investigating: maybe a country with weak coverage is dominating, or your fallback ladder is grabbing low-quality results too eagerly.

7. Retry count per call

retry_count = Histogram(
    'geocode_retries',
    'Retries per logical call',
    buckets=[0, 1, 2, 3, 5, 10]
)

attempts = ...   # tracked inside call_with_backoff
retry_count.observe(attempts - 1)   # number of retries (not total attempts)

P50 retries = 0 (first try succeeded). P99 retries climbing toward your max-attempts cap means the provider is unhealthy or your client-side rate limiting is wrong. See Exponential Backoff for the underlying math.

8. Queue depth (batch pipelines)

queue_depth = Gauge('geocode_queue_depth', 'Pending jobs in queue', ['queue_name'])

# Updated periodically by a sidecar
queue_depth.labels(queue_name='geocode').set(get_queue_size())

Trending queue depth tells you whether your worker pool is keeping up. Steady-state queue near 0 = healthy. Steady-state queue at 100K and growing = workers can't keep up; scale out or rate-limit producers.

The 3 alerts worth a pager page

Most metrics are dashboard material — useful for trends, not for waking someone up. Three are alert-worthy:

Alert 1: Success rate <90% over 5 minutes

# Prometheus alert rule
- alert: GeocodingFailing
  expr: |
    sum(rate(geocode_requests_total{status_code!~"2.."}[5m]))
    / sum(rate(geocode_requests_total[5m]))
    > 0.10
  for: 5m
  labels: { severity: page }
  annotations:
    summary: "Geocoding pipeline failing — {{ $value | humanizePercentage }} error rate"

5-minute window prevents false alerts from brief blips. Threshold at 10% errors (90% success). Adjust per your tolerance — for realtime UIs, alert at 5%.

Alert 2: Queue depth growing for >15 minutes

- alert: GeocodingQueueBacklog
  expr: |
    geocode_queue_depth - geocode_queue_depth offset 15m > 1000
  for: 15m
  labels: { severity: page }

Detects "workers can't keep up." A growing queue means total processing rate < input rate. Either scale workers or throttle producers. If neither happens, the queue grows unboundedly.

Alert 3: p99 latency >2 seconds for >10 minutes

- alert: GeocodingSlow
  expr: |
    histogram_quantile(0.99, rate(geocode_latency_seconds_bucket[5m])) > 2
  for: 10m
  labels: { severity: page }

For batch this is "ugh, that batch will take 4× as long" (annoying, not breaking). For realtime this is "users are seeing 2-second waits." Set threshold to your real SLA.

What NOT to alert on:

• Single-row failures. Aggregate.
• Cache hit rate drops. Investigate next morning, don't wake anyone up.
• Confidence score shifts. Same.
• Match rate drops. Same.

The rule: alert on things that need human action *right now*. Everything else is a dashboard.

Structured logging

Without structured logs, "why did this batch lose 47 rows?" is unanswerable. With them, it's select * from logs where batch_id = X and status = 'failed'.

import json
import time

def log(level, **fields):
    record = {
        'ts': time.time(),
        'level': level,
        **fields,
    }
    print(json.dumps(record))   # ship to your log aggregator

def geocode(addr, batch_id, row_index):
    start = time.monotonic()
    try:
        result = call_api(addr)
        log('info',
            event='geocode',
            batch_id=batch_id,
            row_index=row_index,
            status='ok',
            matched=bool(result),
            confidence=result.get('accuracy_score') if result else None,
            duration_ms=int((time.monotonic() - start) * 1000),
        )
        return result
    except Exception as e:
        log('error',
            event='geocode',
            batch_id=batch_id,
            row_index=row_index,
            status='error',
            error_type=type(e).__name__,
            error_msg=str(e),
            duration_ms=int((time.monotonic() - start) * 1000),
        )
        raise

Three things to standardize on:

1. JSON line per event. Easy to ship to anything (Loki, Datadog, ELK).
2. `batch_id` and `row_index` on every log. Lets you reconstruct the per-row history of a batch in one query.
3. `event` field tagged consistently. geocode, cache_lookup, retry, etc. Filter by event when debugging.

What NOT to log:

• The raw address. PII. Log a hash if you need correlation; never the human-readable address.
• The full geocoder response. Big, redundant, and slow to ship.
• Anything sensitive (API keys, customer emails, internal IDs that map to PII).

Three concrete dashboard panels

A minimum-viable geocoding dashboard has three panels:

Panel 1: Throughput + success rate

A stacked area chart of requests/sec by status code (2xx green, 4xx yellow, 5xx red). At a glance: traffic shape and health. If the green band shrinks suddenly, something broke.

Panel 2: Latency percentiles

Three lines on one chart: p50, p95, p99 of geocode_latency_seconds. Healthy: p50 stays flat near 50ms, p95 ~150ms, p99 ~500ms. Alert: p99 spikes to multi-seconds while p50 stays flat (= long tail of slow calls; usually retry storms or a single slow upstream replica).

Panel 3: Quality metrics

Three lines: match rate, cache hit rate, mean confidence score. All trending sideways is healthy. Match rate dropping = input quality regression. Cache hit rate dropping = key normalization broke. Confidence dropping = upstream coverage changed.

SLO targets

Worth writing down so the team has a shared expectation:

| Metric | Target | |---|---| | API success rate | ≥99% | | Match rate (well-formed input) | ≥95% | | Cache hit rate (steady state) | ≥85% | | p50 latency | ≤80ms | | p95 latency | ≤300ms | | p99 latency | ≤1s | | 429 rate | <0.1% | | Retries per call (p99) | ≤2 |

These are achievable on csv2geo's API (and most modern geocoders) with the patterns covered in the rest of this series. If you're far from any of these, the right fix is usually obvious from the metric that's off:

• Low match rate → improve preprocessing
• Low cache hit rate → fix key normalization or eviction
• High p99 → check retry/backoff config
• High 429 rate → tune client-side rate limiting

Cost observability

Often forgotten: track per-day API spend.

spend_per_day = Counter(
    'geocode_spend_dollars_total',
    'Estimated geocoding spend',
    ['tier']   # 'cached' | 'paid'
)

# Charge per call (assume $0.0005 for paid tier)
if cache_hit:
    spend_per_day.labels(tier='cached').inc(0)
else:
    spend_per_day.labels(tier='paid').inc(0.0005)

A daily summary chart of paid calls × cost-per-call shows you the bill before the bill arrives. Sudden 2× spike that doesn't correlate with traffic means cache hit rate dropped — investigate before next month's invoice.

What this all looks like together

A pipeline with proper observability has:

1. Three dashboards: Operations (throughput/health), Quality (match/confidence/cache), Cost (spend by day).
2. Three alerts: error rate, queue backlog, p99 latency.
3. Structured JSON logs with batch_id + row_index on every event.
4. SLO targets written down in the team wiki.

Total implementation: ~50 lines of Python adding the metrics + the alert YAML + the dashboard JSON. Time investment: half a day. Payback: every incident is debugged in minutes instead of hours.

FAQ

What's the minimum useful metric set if I'm just starting?

Three: success rate by HTTP status, p99 latency, and queue backlog. Everything else is icing. If you only have one dashboard panel and one alert, make them error rate by status code and queue depth — those two together catch about 80% of incidents before customers notice.

Should I alert on p99 latency or on success rate?

Both, at different thresholds and severities. Success rate dropping below ~98% over a 5-minute window is page-now: customers are seeing failures. p99 latency spikes are typically a warning unless they cross your SLO (often 2× the p50). Alerting on p99 alone misses outages; alerting only on success rate misses degradation.

Where do I emit metrics from — the API gateway or inside the worker?

Both, with different cardinality. The gateway sees the customer-facing reality (the only thing that matters for SLOs); the worker sees what's actually happening internally (which is what you need to debug). If you can emit from only one place, emit from the gateway and use structured worker logs for the deep dives.

How do I tell a vendor outage from a regression in my own code?

Tag every metric with provider (e.g. overture, here, google). When error rate spikes for one provider but the others are flat, it's a vendor issue and you reach for the fallback. When all providers spike together at the same wall-clock moment, it's almost certainly something you shipped. That single dimension turns the diagnosis into a 30-second triage instead of a 30-minute postmortem.

What's the cheapest way to ship logs if I'm not running ELK?

JSON to stdout, picked up by your platform's log driver, parsed in your existing log aggregator (CloudWatch, GCP Logging, Datadog, or Vector → S3). Structured JSON costs nothing extra to emit but is queryable in every log backend. Skip the heavy log infrastructure until you've outgrown grep on the raw files.

FAQ

What's the minimum useful metric set if I'm just starting?

Three: success rate by HTTP status, p99 latency, and queue backlog. Everything else is icing. If you only have one dashboard panel and one alert, make them error rate by status code and queue depth — those two together catch about 80% of incidents before customers notice.

Should I alert on p99 latency or on success rate?

Both, at different thresholds and severities. Success rate dropping below ~98% over a 5-minute window is page-now: customers are seeing failures. p99 latency spikes are typically a warning unless they cross your SLO (often 2× the p50). Alerting on p99 alone misses outages; alerting only on success rate misses degradation.

Where do I emit metrics from — the API gateway or inside the worker?

Both, with different cardinality. The gateway sees the customer-facing reality (the only thing that matters for SLOs); the worker sees what's actually happening internally (which is what you need to debug). If you can emit from only one place, emit from the gateway and use structured worker logs for the deep dives.

How do I tell a vendor outage from a regression in my own code?

Tag every metric with provider (e.g. overture, here, google). When error rate spikes for one provider but the others are flat, it's a vendor issue and you reach for the fallback. When all providers spike together at the same wall-clock moment, it's almost certainly something you shipped. That single dimension turns the diagnosis into a 30-second triage instead of a 30-minute postmortem.

What's the cheapest way to ship logs if I'm not running ELK?

JSON to stdout, picked up by your platform's log driver, parsed in your existing log aggregator (CloudWatch, GCP Logging, Datadog, or Vector → S3). Structured JSON costs nothing extra to emit but is queryable in every log backend. Skip the heavy log infrastructure until you've outgrown grep on the raw files.

Summary

The pipelines that feel reliable are the ones where the operators see what's happening before customers do. Three rules:

1. Per-status-code metrics, not lumped error rates. The fix depends on which status code.
2. p99 not p50. The worst 1% of calls is what users notice.
3. Structured logs with `batch_id` + `row_index`. Debugging becomes a query, not an investigation.

Pair with proper rate limiting and exponential backoff and you have a pipeline that warns you about problems before they become incidents — and tells you exactly where to look when one happens anyway.