← All articles

By ·

How to Clean Address Data Before Geocoding: Deterministic Rules First, an LLM Only for the Residue

Clean address data before geocoding with rules first: in my 30-address test, normalisation and lookup rules matched 23. Send only the residue to an LLM.

To clean address data before geocoding, run deterministic rules first: Unicode normalisation, trimming, case and punctuation fixes, USPS Publication 28 street-type and unit abbreviations, local aliases, then a parser such as libpostal. Send only the residue the rules cannot handle to a language model, constrained to a JSON schema and validated against reference lists. In an illustrative set of 30 messy addresses I wrote, the rules alone took exact matches from 0 to 23; the remaining 7 were the transliterated, compact and landmark-only cases that an LLM or a person should see.

This article walks through the pipeline step by step, with Python for the rules, a pydantic schema for the LLM's structured output and a validation gate, plus how to measure the result and why it matters for NG911 address data.

Why does address cleaning decide your geocoding match rate?

A geocoder finds an address by matching it against reference data: address points, or road centrelines with address ranges. When the input says "Street" and the reference says "ST", or the input has a non-breaking space or full-width digits, an exact match fails. The geocoder then falls back to a fuzzy match, a street centroid or a postal code centroid, and each fallback moves the point further from the building.

So most match-rate work is making the input look like the reference. That is cheap and predictable to do with rules, and it is where an LLM adds cost and risk without adding much. The model earns its place on the small share of records that have no consistent structure at all.

Pipeline diagram: four grey deterministic steps (clean text, standardise, parse, match) lead to a geocoded result; unmatched records go to a red LLM structured-parse step, then a validation gate that either sends them back to matching or to a human review queue whose fixes become new rules

The model only sees records that fail matching, and nothing it returns reaches the geocoder without passing the gate.

What should the deterministic pass do?

The deterministic pass should turn every variant of an address into the form your reference data uses, in a fixed order, with each step logged. These are the transformations that do most of the work:

ProblemExample inputDeterministic fixResult
Unicode and spacing2200 east colfax ave (full-width digits), trailing non-breaking spaceUnicode NFKC normalisation, collapse whitespace2200 EAST COLFAX AVE
Case and punctuation123 Main Street S.W.Uppercase, strip full stops123 MAIN STREET SW
Street type742 Evergreen TerraceUSPS Publication 28 Appendix C1, last street-type word only742 EVERGREEN TER
Directionals and units45 North Elm Avenue, Apt. #4Directional map, Appendix C2 unit designators45 N ELM AVE APT 4
Region namesCalgary, AlbertaProvince and state lookupCALGARY AB
Local shorthandH# 12 St# 5 G9/3 IsbLocal alias table and patternsHOUSE 12 STREET 5 G-9/3 ISLAMABAD

USPS Publication 28, Appendix C1 lists each street suffix with its common variants and the standard abbreviation, and Appendix C2 does the same for secondary unit designators such as APT and STE. Canadian data follows Canada Post and municipal conventions (Calgary writes numbered avenues without ordinals, as in 17 Avenue SW), so match whatever your reference layer uses and apply it consistently.

One trap catches almost everyone: abbreviating every word that appears in the suffix table. Pub 28 lists MOUNTAIN and VIEW as suffixes, and CENTRE as a variant of CENTER, so "12 Mountain View Drive" becomes 12 MTN VW DR and stops matching. Abbreviate only the word in the street-type position, which is the last name word before any directional or unit.

Here is the core of the rules in Python. It is short on purpose: the tables carry the knowledge, and the code only applies them.

import re
import unicodedata

# A few rows of USPS Publication 28, Appendix C1 (street suffixes) and C2 (units).
# Load the full tables in production.
SUFFIX = {"STREET": "ST", "AVENUE": "AVE", "BOULEVARD": "BLVD", "COURT": "CT",
          "DRIVE": "DR", "TERRACE": "TER", "TRAIL": "TRL", "PARKWAY": "PKWY"}
DIRECTION = {"NORTH": "N", "SOUTH": "S", "EAST": "E", "WEST": "W"}
UNIT = {"APARTMENT": "APT", "SUITE": "STE"}

def clean_text(s: str) -> str:
    s = unicodedata.normalize("NFKC", s)          # full-width digits, NBSP -> ASCII
    s = s.upper().replace(".", "").replace("#", " ")
    s = re.sub(r"\s*,\s*", ", ", s)
    return re.sub(r"\s+", " ", s).strip(" ,")

def standardise_street(street: str) -> str:
    """Abbreviate directionals and units, and only the LAST street-type word."""
    toks = [DIRECTION.get(t, UNIT.get(t, t)) for t in street.split()]
    for i in range(len(toks) - 1, 0, -1):          # skip trailing directionals/units
        if toks[i] in DIRECTION.values() or toks[i] in UNIT.values() or toks[i].isdigit():
            continue
        toks[i] = SUFFIX.get(toks[i], toks[i])
        break
    return " ".join(toks)

PK_RULES = [
    (r"\b(?:H ?NO|HNO|HOUSE NO|H)\s*(\d+)", r"HOUSE \1"),
    (r"\b(?:ST|STREET NO|STREET)\s*(\d+)", r"STREET \1"),
    (r"\bSECTOR\s+", ""),
    (r"\b([A-I])[- ]?(\d{1,2})(?:/(\d))?\b",                  # G9/3, G 9/3 -> G-9/3
     lambda m: f"{m[1]}-{m[2]}" + (f"/{m[3]}" if m[3] else "")),
    (r"\bISB\b", "ISLAMABAD"), (r"\bRWP\b", "RAWALPINDI"),
]

def standardise_pk(s: str) -> str:
    for pattern, repl in PK_RULES:
        s = re.sub(pattern, repl, s)
    return re.sub(r"\s+", " ", s)

standardise_street(clean_text("12 Mountain View Drive"))   # '12 MOUNTAIN VIEW DR'
standardise_pk(clean_text("H# 12 St# 5 G9/3 Isb"))          # 'HOUSE 12 STREET 5 G-9/3 ISLAMABAD'

Note that "ST 34" means street number 34 in a Pakistani address and "ST" is a suffix in a North American one. Route each record to the right rule set first (by city, province or country field), then apply that set only.

For parsing into components, libpostal is the standard open-source option. Its authors describe a parser trained on over 1 billion addresses from every inhabited country, with normalisation in more than 50 languages. It returns labelled parts such as house number, road and city. Its README lists geocoding to latitude and longitude as a non-goal, so its output is the input to your geocoder.

What did the rules catch in my test?

To see how far rules go before an LLM is needed, I wrote 30 messy addresses and their correct forms: 16 North American (Calgary, Toronto, Denver, Plano and others) and 14 Pakistani (Islamabad, Rawalpindi, Lahore, Karachi). I built a 23-address reference list from the correct forms and counted exact token matches after each cumulative step. These are examples I made up to illustrate the failure types, and I tuned the rules on the same set, so real records will match at lower rates.

Step (cumulative)North American (16)Pakistani (14)Total (30)
Raw text000
Trim, Unicode NFKC, uppercase, strip punctuation336
+ Pub 28 street types, directionals, units, region names14317
+ Pakistan rules (H#, St#, sector pattern, city aliases)14923

Left: horizontal bars showing exact matches rising from 0 of 30 for raw text to 6, 17 and 23 of 30 as cleaning steps are added. Right: the seven addresses left over and the reason each one failed

An illustrative set of 30 addresses. Each rule set is cheap, explainable and reusable. The 7 left over fall into a few distinct types.

The seven failures are more useful than the 23 matches, because they show what the residue looks like:

Which addresses should go to an LLM?

An LLM is worth calling for records that fail matching after the rules and that have structure a reader could recover: free-text descriptions, transliterated or mixed-script addresses, shorthand like 45-C that needs context, and fields in an unusual order. Pakistani addresses are a good example. Islamabad uses lettered and numbered sectors with subsectors (G-9/3), housing schemes such as DHA use phases and blocks, and people write the same plot many ways in English, Roman Urdu and Urdu.

The model's job is narrow: read the text and put each piece in a named field. It should never be asked where the address is. It has no access to your address register, so any postcode, plot number or coordinate it supplies is a fluent guess. Landmark addresses ("near X") should come back with the landmark in its own field and a flag, for a person or a gazetteer lookup to resolve.

How do you make LLM address parsing safe?

Make the model's output a typed record that code checks before anything uses it. The controls that matter:

  1. Structured output only. Use your provider's JSON schema or function-calling mode, with the schema generated from a pydantic model. Forbid extra keys, so a stray postcode or latitude fails validation.
  2. Temperature 0 and a fixed prompt version. Same input, same output as far as the provider allows, and a version string in the log so you can rerun.
  3. No location fields in the schema. No coordinates, no postcode. The geocoder derives location from the reference layer.
  4. Field-level validation. Check cities, sectors, schemes and street names against reference lists, and reject any number in the output that does not appear in the input.
  5. Confidence and type flags. Anything below "full, high confidence" goes to review.
  6. Caching and cost control. Key the cache on the cleaned text, so repeated addresses cost one call. Because only the residue is sent, volume is a small share of the file.
  7. Logging. Store input, rule version, prompt version, model identifier, raw output, gate result and reviewer decision for every record.

Here is the schema and the gate. The LLM call is left as a provider-agnostic stub.

import re
from typing import Literal, Optional
from pydantic import BaseModel, ConfigDict, Field, ValidationError, field_validator

class ParsedAddress(BaseModel):
    """What the model may return. No coordinates, no postcode field to fill in."""
    model_config = ConfigDict(extra="forbid")   # reject keys the schema does not define

    house_number: Optional[str] = Field(None, description="As written, e.g. '12' or '45-C'")
    street: Optional[str] = Field(None, description="Street or gali, e.g. 'STREET 5'")
    block: Optional[str] = None
    phase: Optional[str] = None
    sector: Optional[str] = Field(None, description="Islamabad sector, e.g. 'G-9/3'")
    locality: Optional[str] = Field(None, description="Scheme or neighbourhood, e.g. 'DHA'")
    city: str
    landmark: Optional[str] = Field(None, description="Text after 'near', verbatim")
    address_type: Literal["full", "partial", "landmark_only", "not_an_address"]
    confidence: Literal["high", "medium", "low"]

    @field_validator("sector")
    @classmethod
    def sector_format(cls, v):
        if v is not None and not re.fullmatch(r"[A-I]-\d{1,2}(/[1-4])?", v):
            raise ValueError(f"sector {v!r} is not in Islamabad sector format")
        return v

KNOWN_CITIES = {"ISLAMABAD", "RAWALPINDI", "LAHORE", "KARACHI"}
KNOWN_SECTORS = {"G-9/3", "F-11/2", "I-8/4", "E-7", "G-6/1"}   # from your reference layer

def validate(raw_text: str, payload: dict):
    try:
        p = ParsedAddress.model_validate(payload)
    except ValidationError as e:
        return None, [f"schema: {err['loc']} {err['msg']}" for err in e.errors()]
    flags = []
    if p.city.upper() not in KNOWN_CITIES:
        flags.append(f"unknown city {p.city!r}")
    if p.sector and p.sector not in KNOWN_SECTORS:
        flags.append(f"sector {p.sector} not in reference layer")
    in_digits = set(re.findall(r"\d+", raw_text))          # no invented numbers
    for field in ("house_number", "street", "block", "phase", "sector"):
        for d in re.findall(r"\d+", getattr(p, field) or ""):
            if d not in in_digits:
                flags.append(f"{field} contains {d}, which is not in the input")
    if p.address_type != "full" or p.confidence != "high":
        flags.append(f"{p.address_type}/{p.confidence}: send to review")
    return p, flags

def parse_residue(raw_text: str, cache: dict):
    key = raw_text.strip().upper()
    if key not in cache:
        # Provider-agnostic: call your LLM's structured-output or tool-calling API
        # with ParsedAddress.model_json_schema(), temperature=0, a versioned prompt
        # and a few labelled examples. Store the raw response for the log.
        cache[key] = call_llm(prompt_v="addr-v3", text=raw_text,
                              schema=ParsedAddress.model_json_schema())
    return validate(raw_text, cache[key])

I tested the gate with hand-written payloads. A payload with an added postcode and latitude fails on the extra keys, G9-3 fails the sector pattern, and a house number of 112 for an input that says 12 is flagged as a number absent from the input. Records that pass go back to step 4 and are matched like any other; records with flags go to the review queue.

How do you measure whether it worked?

Measure on a hold-out sample with hand-labelled truth, drawn before you write any rules and never used to tune them. Stratify it by source system and by the failure types you care about, then label each record with its correct reference address (or "no unique address") and its true location.

MetricWhat it tells youHow to compute
Match rate by match typeHow many records reach an address point, a range, a street or only a postcodeCount geocoder match codes on the hold-out
False match rateRecords matched confidently to the wrong addressCompare matched address ID to the labelled ID
Positional errorHow far the geocoded point sits from the truthGeodesic distance; report the median and 95th percentile
Review loadPeople-hours the pipeline createsRecords flagged per 1,000 processed
Cost per 1,000 recordsModel spend after cachingCalls made times price, from the log

Report false matches alongside the match rate. A pipeline that raises the match rate by guessing has made the data worse, and an LLM that is allowed to guess will do exactly that. Run the same hold-out through rules only and through rules plus LLM, and keep the LLM step only where it adds correct matches without adding false ones.

How does this relate to NG911 address quality?

NG911 makes address quality a call-routing problem. The Location Validation Function checks a civic address against your GIS address points and road centrelines, so the street names in legacy MSAG, ALI and CAD records need to agree with the GIS. The NG911 GIS data readiness checklist covers the 98% synchronisation target, and check 11 in my NG911 QA checks finds address point street names with no matching centreline.

The pipeline above is a good way to triage legacy records against that standard: rules fix format differences in bulk, an LLM can parse the free-text locations that accumulate in CAD, and the review queue puts every real correction in front of the addressing authority. The model should never write to the authoritative address points or centrelines. Those changes are decisions with legal and routing consequences, and a person makes them.

What should you take from this?

Most address cleaning is deterministic, and it should be done with rules you can read, test and rerun. In my illustrative set, Unicode cleanup, Pub 28 abbreviations and a handful of local patterns took exact matches from 0 to 23 of 30. Use a language model for the residue only, as a parser with a strict schema and no way to supply locations, and measure the whole pipeline on a labelled hold-out, counting false matches. For where else LLMs help in GIS work and where they do not, see Can LLMs do GIS?

If you have an address file that will not geocode, or legacy 911 records to reconcile with your GIS, email me at mahad.imran29@gmail.com.

Frequently asked questions

How do I clean address data before geocoding?

Run deterministic steps first: Unicode normalisation, trimming, uppercase, punctuation removal, then standard abbreviations for street types, directionals and unit designators from USPS Publication 28, plus local aliases. Parse into components with libpostal or rules and match against your reference layer. Only the records that still fail should go to a language model or a person, and every change should be logged.

Can an LLM standardize addresses?

It can parse messy free text into fields, including transliterated or Roman Urdu addresses and landmark descriptions that rules miss. It should not be the first step or the last word. Constrain it to a JSON schema with no coordinate or postcode fields, run it at temperature 0, reject any output containing numbers absent from the input, and validate every field against reference lists before matching.

How can I improve my geocoding match rate?

Standardise the input to the same conventions as the reference data, because most failed matches are spelling and format differences. Abbreviate the street type only in its own position, expand local shorthand, fix Unicode and spacing, and route records with no house number to review. Then measure on a hand-labelled hold-out sample, counting false matches as well as matches.

Does libpostal geocode addresses?

No. libpostal parses and normalises addresses into labelled components such as house number, road and city, using a model its authors say was trained on over 1 billion addresses. Its documentation states that geocoding to latitude and longitude is a non-goal because that needs a database or search index. Use its output as the input to your geocoder.

Should an LLM fill in missing postcodes or coordinates?

No. A language model has no access to your address register, so a postcode or coordinate it produces is a guess that looks like data. Leave those fields out of the output schema entirely, so the model has nowhere to put them, and let the geocoder derive location from the reference layer. Records that cannot be matched without a guess belong in a human review queue.

How do you geocode Pakistani addresses like House 12, Street 5, G-9/3, Islamabad?

Parse them into house number, street, sector or block, scheme and city, then match against a reference layer of plots or sectors. Islamabad sectors follow a letter-number pattern with numbered subsectors, such as G-9/3, which a regular expression can standardise from variants like G9/3. Schemes such as DHA use phase and block. Landmark-only addresses need a person.

Let's talk.

Best way to reach me is email. I respond to most things.