Reference Lists

Reference lists provide static lookup data that can be used by transformers for operations like mapping values. They allow you to maintain reusable data sets.

Using Predefined Reference Lists

Tilores provides predefined reference lists that can be loaded from external sources. To use a predefined reference list, specify the external field with the format "filename@version":

{
  "referenceLists": [
    {
      "id": "countries",
      "external": "map-country-code@latest"
    }
  ]
}

Available Predefined Lists

The lists below are consumed by the normalize company transformer, version 2. The first eight are required: they shape the company name, and wiring only some of them would produce a name that is quietly worse than the same configuration wiring all of them, so the transformer refuses to load instead. The rest feed a reporting output and can be left unset; without one, that output stays empty and the company name is unaffected.

The lists below are consumed by the normalize name transformer, version 2. The first three are required; without them the transformer does not load. The rest are optional, and leaving one out only turns off what it drives.

Version Format

The version in "filename@version" can be:

• latest - Always uses the most recent version
• A specific version tag (e.g., v0-1-0)
• A specific major tag (e.g., v0)

Overriding External Data

You can extend or override entries from an external reference list by providing inline rows. The first column is used as the merge key:

{
  "referenceLists": [
    {
      "id": "countries",
      "external": "map-country-code@latest",
      "rows": [
        ["custom country", "xx"],
        ["united states", "usa"]
      ]
    }
  ]
}

In this example:

• "custom country" is added as a new entry
• "united states" overrides the existing mapping from the external list

Inline Reference Lists

You can also define reference lists entirely inline without using external sources:

{
  "referenceLists": [
    {
      "id": "status-codes",
      "meta": {
        "header": [
          {"type": "token"},
          {"type": "mapping"}
        ]
      },
      "rows": [
        ["active", "A"],
        ["inactive", "I"],
        ["pending", "P"]
      ]
    }
  ]
}

Reference List Structure

Each reference list has:

• id - Unique identifier used to reference the list
• meta.header - Column definitions with types (token, mapping, etc.)
• rows - Array of value arrays, one per entry

Column Types

Reference lists support different column types depending on how the data will be used:

token

The token column type defines the lookup key. When a transformer or matcher looks up a value in a reference list, it searches the token column to find a match. This is typically the first column in a reference list.

mapping

The mapping column type defines the output value for value mapping operations. When using the Map Value transformer, the input is looked up in the token column and the corresponding mapping column value is returned.

Example: A country name to country code mapping:

{
  "meta": {
    "header": [
      {"type": "token"},
      {"type": "mapping"}
    ]
  },
  "rows": [
    ["united states", "US"],
    ["germany", "DE"],
    ["france", "FR"]
  ]
}

tokenfrequencyweight

The tokenfrequencyweight column type stores numeric weights for tokens. This is used by the Weighted Token matcher to assign importance to different tokens during matching.

Lower weights indicate more common tokens (less distinctive), while higher weights indicate rarer tokens (more distinctive for matching).

Example: A token weight list for name matching:

{
  "meta": {
    "header": [
      {"type": "token"},
      {"type": "tokenfrequencyweight"}
    ]
  },
  "rows": [
    ["john", 0.7],
    ["smith", 1.0],
    ["doe", 0.5]
  ]
}

constraint

The constraint column type scopes a row to a specific context, expressed as an ISO 3166-1 Alpha-2 country code. A row is only considered when its constraint value matches the country the transformer is currently processing. This allows a single reference list to hold rules for many countries. It is used by the Normalize Address and Normalize Country transformers.

Example: A country specific street-type mapping, where str is only expanded to Straße for Germany:

{
  "meta": {
    "header": [
      {"type": "token"},
      {"type": "constraint"},
      {"type": "mapping"}
    ]
  },
  "rows": [
    ["str", "de", "Straße"],
    ["ave", "us", "Avenue"]
  ]
}

regex

The regex column type holds a regular expression pattern. The pattern is compiled and validated when the list is loaded, so an invalid expression is reported early. It is used by the Normalize Address transformer for its preprocessing and postprocessing replacement rules, where the matched text is replaced with the mapping value (capture groups such as $1 are supported).

Example: A preprocessing rule that splits a compound street type off the name word (Hauptstr. → Haupt str.) for Germany:

{
  "meta": {
    "header": [
      {"type": "regex"},
      {"type": "constraint"},
      {"type": "mapping"}
    ]
  },
  "rows": [
    ["(?i)(\\w+)(str\\.)", "de", "$1 $2"]
  ]
}