# Reference Lists

Reference lists provide static lookup data that can be used by transformers for
operations like mapping values. They allow you to maintain reusable data sets.

## Using Predefined Reference Lists

Tilores provides predefined reference lists that can be loaded from external
sources. To use a predefined reference list, specify the `external` field with
the format `"filename@version"`:

```json
{
  "referenceLists": [
    {
      "id": "countries",
      "external": "map-country-code@latest"
    }
  ]
}
```

### Available Predefined Lists

| Name | Description |
|------|-------------|
| `map-country-code` | Maps country names to ISO 3166-1 alpha-2 codes |
| `map-country-language` | Maps country codes to ISO 639-3 language codes |

The lists below are consumed by the
[normalize company transformer, version 2](transformer.md#normalize-company-version-2).
The first eight are required: they shape the company name, and wiring only some
of them would produce a name that is quietly worse than the same configuration
wiring all of them, so the transformer refuses to load instead. The rest feed a
reporting output and can be left unset; without one, that output stays empty and
the company name is unaffected.

| Name | | Description |
|------|--|-------------|
| `filter-company-legal-form` | required | Legal forms per country (the detection list) |
| `map-company-legal-form-alias` | required | Every spelling of a form to its canonical value |
| `preprocess-company` | required | Per-country regex rewrites applied before detection |
| `postprocess-company` | required | Per-country regex cleanups applied to the resulting name |
| `map-company-token` | required | Converges spelling variants of ordinary name tokens |
| `filter-company-generic-token` | required | Generic vocabulary, so a name is not reduced to a sector word |
| `map-company-root-separator` | required | Per-country policy for separators inside a name |
| `filter-company-root-word` | required | Words that may be glued back onto the name |
| `filter-company-group-function` | optional | Words marking a company's role inside a group |
| `map-company-group-function-alias` | optional | Spellings of a group function to its canonical value |
| `filter-company-branch` | optional | Branch designators (`Zweigniederlassung`, `branch`, …) |
| `map-city-country-code` | optional | City names to country codes, for location detection |
| `map-country-code` | optional | Country names to ISO codes, for location detection |
| `map-city-token` | optional | Spelling variants of city name parts |
| `map-company-legal-form-elf` | optional | ISO 20275 ELF code per country and legal form |

### Version Format

The version in `"filename@version"` can be:
- `latest` - Always uses the most recent version
- A specific version tag (e.g., `v0-1-0`)
- A specific major tag (e.g., `v0`)

## Overriding External Data

You can extend or override entries from an external reference list by providing
inline rows. The first column is used as the merge key:

```json
{
  "referenceLists": [
    {
      "id": "countries",
      "external": "map-country-code@latest",
      "rows": [
        ["custom country", "xx"],
        ["united states", "usa"]
      ]
    }
  ]
}
```

In this example:
- `"custom country"` is added as a new entry
- `"united states"` overrides the existing mapping from the external list

## Inline Reference Lists

You can also define reference lists entirely inline without using external
sources:

```json
{
  "referenceLists": [
    {
      "id": "status-codes",
      "meta": {
        "header": [
          {"type": "token"},
          {"type": "mapping"}
        ]
      },
      "rows": [
        ["active", "A"],
        ["inactive", "I"],
        ["pending", "P"]
      ]
    }
  ]
}
```

## Reference List Structure

Each reference list has:
- `id` - Unique identifier used to reference the list
- `meta.header` - Column definitions with types (`token`, `mapping`, etc.)
- `rows` - Array of value arrays, one per entry

## Column Types

Reference lists support different column types depending on how the data will be used:

### token

The `token` column type defines the lookup key. When a transformer or matcher looks up
a value in a reference list, it searches the `token` column to find a match. This is
typically the first column in a reference list.

### mapping

The `mapping` column type defines the output value for value mapping operations. When
using the [Map Value transformer](transformer.md#map-value), the input is looked up in
the `token` column and the corresponding `mapping` column value is returned.

**Example:** A country name to country code mapping:

```json
{
  "meta": {
    "header": [
      {"type": "token"},
      {"type": "mapping"}
    ]
  },
  "rows": [
    ["united states", "US"],
    ["germany", "DE"],
    ["france", "FR"]
  ]
}
```

### tokenfrequencyweight

The `tokenfrequencyweight` column type stores numeric weights for tokens. This is used
by the [Weighted Token matcher](matcher.md#weighted-token) to assign importance to
different tokens during matching.

Lower weights indicate more common tokens (less distinctive), while higher weights
indicate rarer tokens (more distinctive for matching).

**Example:** A token weight list for name matching:

```json
{
  "meta": {
    "header": [
      {"type": "token"},
      {"type": "tokenfrequencyweight"}
    ]
  },
  "rows": [
    ["john", 0.7],
    ["smith", 1.0],
    ["doe", 0.5]
  ]
}
```

### constraint

The `constraint` column type scopes a row to a specific context, expressed as an
ISO 3166-1 Alpha-2 country code. A row is only considered when its `constraint`
value matches the country the transformer is currently processing. This allows a
single reference list to hold rules for many countries. It is used by the
[Normalize Address](transformer.md#normalize-address) and
[Normalize Country](transformer.md#normalize-country) transformers.

**Example:** A country specific street-type mapping, where `str` is only
expanded to `Straße` for Germany:

```json
{
  "meta": {
    "header": [
      {"type": "token"},
      {"type": "constraint"},
      {"type": "mapping"}
    ]
  },
  "rows": [
    ["str", "de", "Straße"],
    ["ave", "us", "Avenue"]
  ]
}
```

### regex

The `regex` column type holds a regular expression pattern. The pattern is
compiled and validated when the list is loaded, so an invalid expression is
reported early. It is used by the
[Normalize Address](transformer.md#normalize-address) transformer for its
preprocessing and postprocessing replacement rules, where the matched text is
replaced with the `mapping` value (capture groups such as `$1` are supported).

**Example:** A preprocessing rule that splits a compound street type off the
name word (`Hauptstr.` → `Haupt str.`) for Germany:

```json
{
  "meta": {
    "header": [
      {"type": "regex"},
      {"type": "constraint"},
      {"type": "mapping"}
    ]
  },
  "rows": [
    ["(?i)(\\w+)(str\\.)", "de", "$1 $2"]
  ]
}
```