# Schema Customization

The schema defines the data structure for your inputs and outputs of your
Tilores application. The schema can be fully customized to match your existing
data model.

You define the schema by defining how your custom GraphQL API looks like. For
that reason, if you already know the basics of how to define a GraphQL schema,
then it should be pretty straight forward to define your data model. If you do
not have experience with that, we will show you the basics here and additionally
would like to point you to the
[official GraphQL documentation](https://graphql.org/learn/schema/).

By default all files relevant for customizing your schema, can be found in the
`schema` folder in your Tilores project. Feel free to add your own files as you
need them.

## Record

A record represents a single data point of your entity and is the most important
data structure to store any kind of information in. It can be very simple, e.g.
a single attribute, or very complex, e.g. multiple nested layers of attributes.

You can find the definition for a record in `schema/record.graphqls`.

In GraphQL a data structure that can be used in the request as well as in the
response can have separate type definitions. That is the reason, why you will
find the two types `RecordInput` and `Record` defined. `RecordInput` will be
used when submitting data, while `Record` will be used when querying data. For
most users these will contain the same fields. Later we will show under which
circumstances these types might [differ](#diverging-record-and-recordinput).

For now, let's look at the current definition of the `RecordInput`.

```graphql
input RecordInput {
  """Id provides a unique identifier for this record."""
  id: ID!

  """MyCustomField is an example field that can be changed or dropped."""
  myCustomField: String!
}
```

The `input` keyword tells GraphQL that this type is supposed to be used as input
and cannot be queried. `RecordInput` obviously defines the name of the type as
it must be used in other places. And our `RecordInput` currently has two fields,
`id` and `myCustomField`.

The ID field is the only field that is required by Tilores. It must be unique
among all records and we highly recommend to use random UUIDs (V4) for this
purpose. If this would break your data model, there is the possibility to
[rename or generate](#custom-record-ids) the ID field.

`myCustomField` only serves as a simple example and can easily be replaced with
your own fields. However, be aware, that the default configuration uses this
field for matching and searching and the [rules configuration](rules/index.md)
should be modified when removing this field.

Each field can have a multiline description. This description can be queried
and displayed by GraphQL clients and serves as the documentation for your custom
API - a GraphQL feature called
[introspection](https://graphql.org/learn/introspection/).

### Adding Fields

Let's assume, you have JSON data and it looks something like this:

```json
{
  "id": "ae0965d6-eae4-49ac-9897-55b48d376eda",
  "sourceID": 1234,
  "source": "SOURCE_A",
  "person": {
    "name": "John Smith",
    "address": "Some Street, Somewhere"
  }
}
```

Adjusting the `RecordInput` to have the `sourceID` available is a simple task:

```graphql
input RecordInput {
  id: ID!
  sourceID: Int!
}
```

You could proceed the same way with the source. However, in that case you may
already know that there is a limited amount of possible sources and you may want
to model it as an enumeration.

```graphql
enum Source {
  SOURCE_A
  SOURCE_B
  SOURCE_C
}
```

When running or deploying your project, the relevant enumeration type will be
created for you.

For the `person` field we will have to create a new type:

```graphql
input PersonInput {
  name: String!
  address: String
}
```

Notice, that the `address` definition does not end with an exclamation mark `!`.
This is meant to show how to define optional fields. In that case you can either
omit the address or submit it using a `null` value.

Both, the `source` enumeration and the `PersonInput` can either be defined
directly inside the `schema/record.graphqls` file or in new files. This is
completely up to you.

The final `RecordInput` could look something like this:

```graphql
input RecordInput {
  id: ID!
  sourceID: Int!
  source: Source!
  person: PersonInput!
}
```

Now that the `RecordInput` is defined, it is an easy task to apply the changes
to the `Record`:

```graphql
type Record {
  id: ID!
  sourceID: Int!
  source: Source!
  person: Person!
}

type Person {
  name: String!
  address: String
}
```

Notice, that also the `PersonInput` must be copied!

## SearchParams

The type `SearchParams` lets you define which fields can be used during a search
query. You can find its definition in `schema/search.graphqls.`.

The default type definition is again just for demonstration purposes and looks
like this:

```graphql
input SearchParams {
  myCustomField: String!
}
```

Again it is up to you to decide how a search is supposed to look like. Using the
previously defined model for the record, you might e.g. provide the possibility
to search using the name and the address. In that case you could alter the
`SearchParams` to either use the existing `PersonInput` type, define a new type
`PersonSearchInput` or just provide the relevant fields directly in the
`SearchParams`.

Option 1: reuse existing type:

```graphql
input SearchParams {
  person: PersonInput!
}
```

Option 2: define a new type:

```graphql
input PersonSearchInput {
  name: String!
  address: String
}

input SearchParams {
  person: PersonSearchInput!
}
```

Option 3: provide only relevant fields:

```graphql
input SearchParams {
  personName: String!
  address: String
}
```

Note, that the field names do not have to be equal to the field names as they
were defined in the `PersonInput`. In most cases option two or three are the
preferable, because they can evolve independently from the `RecordInput`, e.g.
the `PersonInput` might introduce a new field `dateOfBirth` which is not
relevant for the search.

Also note, that in order to use the fields defined in the `SearchParams` they
must be used in the [search rules configuration](rules/index.md).

## Default Scalars

GraphQL comes with a list of default types, so called scalars. Additionally a
few more scalars are available.

The default scalars can be found in the
[official documentation](https://graphql.org/learn/schema/#scalar-types).

The following additional scalars are defined by default.

```graphql
scalar Time

scalar Map

scalar Any
```

In some cases you may want to
[define your own scalar types](https://gqlgen.com/reference/scalars/#custom-scalars-with-user-defined-types).

## Advanced Configuration

The following examples are more advanced, because they require small changes or
additions in the generated source code.

### Diverging Record and RecordInput

Defining the structure of `Record` and `RecordInput` independently seems very
annoying at first. However, it provides great possibilities to evolve the
underlying data model from the query model. Maybe you would like to return
certain available data only in an aggregated form or you would like to enrich
data that is stored in another system. There are probably limitless amounts of
reasons, why you might need this.

#### Data Enrichment

Let's assume we have an `InputRecord` that, beside other data, has a
`providerID` field. That data about the provider is stored in an external
database. When querying we want that the user can query the providers name and
contract id. Since these values are subject of change and not relevant for our
matching process, we do not want to store them inside Tilores.

Our schema might look like this:

```graphql
input RecordInput {
  id: ID!
  providerID: Int!
  # further fields
}

type Provider {
  name: String!
  contractID: Int!
}

type Record {
  id: ID!
  provider: Provider!
  # further fields
}
```

When running this schema, the provider data would be empty. More precisely, it
would even fail with an error, because the required `provider` field will always
be empty.

To fix this, first we need to create a custom model, that has the `providerID`
field. Create the file `graph/model/record.go`:

```go
package model

type Record struct {
  ID         string `json:"id"`
  ProviderID int
  // further fields
}
```

Now tell the schema to use that type instead of generating one:

```graphql
type Record @goModel(model:"<your module path>/graph/model.Record") {
  id: ID!
  provider: Provider!
}
```

Regenerate the models and resolvers by running `go generate ./...` on your
project folder.

Since our record does not define a `provider` field, but our schema needs an
implementation for one, the new file `graph/record.resolvers.go` was created for
you and should contain something like this:

```go
func (r *recordResolver) Provider(ctx context.Context, obj *model.Record) (*model.Provider, error) {
	panic(fmt.Errorf("not implemented"))
}
```

All that is left is to actually implement this resolver method. In a real
scenario you would most likely make a database query or a call to another
service. In our example, we are going to create a fake implementation.

```go
func (r *recordResolver) Provider(ctx context.Context, obj *model.Record) (*model.Provider, error) {
	if obj.ProviderID == 1 {
		return &model.Provider{
			Name:       "Provider A",
			ContractID: 1234,
		}, nil
	}
	if obj.ProviderID == 2 {
		return &model.Provider{
			Name:       "Provider B",
			ContractID: 9876,
		}, nil
	}
	return nil, fmt.Errorf("unknown provider: %v", obj.ProviderID)
}
```

#### Requiring New Field With Existing Data

When adding a new required field to the `RecordInput` after you have already
submitted data into your Tilores instance, then that same field should not be
required on the `Record`. If you ignore this rule, then this will result in
errors when trying to query that new field on records that do not have this
field and therefore would return a `null` value.

If you have to mark that field as required on the `Record`, then the alternative
would be to enforce the creation of a resolver for that field and return an
appropriate default value if the value would otherwise be `null`.

Assuming your schema looks like this:

```graphql
input RecordInput {
  id: ID!
  existingField: String!
  # further fields
}

type Record {
  id: ID!
  existingField: String!
  # further fields
}
```

Furthermore assuming you already have data added according to that schema. Now
you want to add another required field to the `RecordInput`:

```graphql
input RecordInput {
  id: ID!
  existingField: String!
  newField: String!
  # further fields
}
```

For the `Record` the simplest solution would be to add `newField: String`
(without an exclamation mark). This would allow `newField`, when queried to
return `null`.

For cases when you must ensure that this value is not nullable, then you can
define the field like this:

```graphql
type Record @goModel(model:"<your module path>/graph/model.Record") {
  id: ID!
  existingField: String!
  newField: String! @goField(forceResolver: true)
  # further fields
}
```

Additionally, you need to create or modify your `Record` model under
`graph/model/record.go` like this:

```go
package model

type Record struct {
  ID            string `json:"id"`
  ExistingField string `json:"existingField"`
  NewField      *string `json:"newField"`
  // further fields
}
```

After you regenerated the code using `go generate ./...`, you should find the
following function under `graph/record.resolvers.go`:

```go
func (r *recordResolver) NewField(ctx context.Context, obj *model.Record) (string, error) {
	panic(fmt.Errorf("not implemented"))
}
```

It should now be trivial to implement a default value for that field
(`value is missing` in our example):

```go
func (r *recordResolver) NewField(ctx context.Context, obj *model.Record) (string, error) {
	if obj.NewField != nil {
		return *obj.NewField, nil
	}
	return "value is missing", nil
}
```

### Custom Record IDs

As mentioned before, each record must have an ID. Depending on your needs, you
may not want to stick with the default implementation. The following three
scenarios show you how to adjust it for your use case.

!!!warning
A record ID must never contain a colon `:`! Colons are used to separate the IDs
and labels of edges. A record with a colon in the ID will be rejected with an
error message.
!!!

#### Scenario 1: Rename ID Field

If your data already contains a unique ID field, but you want to stick to your
existing name, you can easily change that.

First you need to change the field name in `schema/record.graphqls` for both the
`Record` and `RecordInput`:

```graphql
input RecordInput {
  customID: ID!
  # further fields
}

type Record @goModel(model:"my-tilores-project/graph/model.Record") {
  customID: ID!
  # further fields
}
```

We replaced `id` with `customID`.

Then you have to change the code that extracts the ID field. In
`graph/helpers.go` you will find the following function:

```go
func extractRecordID(record *model.RecordInput) string {
	return record.ID
}
```

Change that to the correct field name:

```go
func extractRecordID(record *model.RecordInput) string {
	return record.CustomID
}
```

After regenerating the API code using `go generate ./...`, the compile errors
will be gone and your field will be renamed.

#### Scenario 2: Merged ID

Another scenario might be that you have unique ID, but only within each data
provider. The steps are similar to the first scenario.

Change the record schema:

```graphql
input RecordInput {
  provider: String!
  providerRecordID: String!
  # further fields
}

type Record @goModel(model:"my-tilores-project/graph/model.Record") {
  provider: String!
  providerRecordID: String!
  # further fields
}
```

Change the `extractRecordID` function to:

```go
import "fmt"

func extractRecordID(record *model.RecordInput) string {
	return fmt.Sprintf("%v-%v", record.Provider, record.ProviderRecordID)
}
```

#### Scenario 3: Generated Random IDs

If you have no fields available that can serve as the record ID, your only
possibility left is to generate a random ID. The steps are similar to the first
scenario.

Remove the ID field from the record schema:

```graphql
input RecordInput {
  # further fields
}

type Record @goModel(model:"my-tilores-project/graph/model.Record") {
  # further fields
}
```

Change the `extractRecordID` function to:

```go
import "github.com/google/uuid"

func extractRecordID(_ *model.RecordInput) string {
	return uuid.Must(uuid.NewRandom()).String()
}
```

### Custom Record Insights

The [record insights feature](api.md#record-insights) provides filtering,
statistics and aggregation on the entity records. While it provides a lot of
flexibility during query time, some complex cases cannot be handled by it. Some
users also prefer to centralize typical queries on the server side rather than
providing a complex query for each client. One such example is the creation of
a [golden record](golden-record.md).

#### Golden Record Example Requirements and Challenges

The following example provides an idea of how to create a custom `goldenRecord`
field on the `recordInsights`.

Let's assume that our business uses the following definition of a golden record:

- most frequent name of `SOURCE_A`
- if unavailable, most frequent name of any other source
- business needs to know if the name is from a high quality source (`SOURCE_A`)
- latest submitted address

There are a few challenges with that when using the standard record insights.
While selective name frequency is possible using the `filter` of
`recordInsights` (see example below), the resulting query would be quite
complicated and still requires additional checks on the client side.

```graphql
query {
  entity(input: {id: "123"}) {
    entity {
      sourceARecords: filter(conditions: [{field: "source", equals: "SOURCE_A"}]) {
        name: frequencyDistribution(field: "person.name", top: 1) {
          value
        }
      }
      sourceOtherRecords: filter(conditions: [{field: "source", equals: "SOURCE_A", invert: true}]) {
        name: frequencyDistribution(field: "person.name", top: 1) {
          value
        }
      }
    }
  }
}
```

Where the result might look something like that:

```json
{
  "data": {
    "entity": {
      "entity": {
        "sourceARecords": {
          "name": {
            "value": "John Smith"
          }
        },
        "sourceOtherRecords": {
          "name": {
            "value": "J. Smith"
          }
        }
      }
    }
  }
}
```

Not exactly a simple result and you would still need to apply logic on the
client side to decide that `John Smith` should be used instead of `J. Smith`.

Additionally, since our schema does not contain a timestamp field, there is no
attribute available that allows us to query the latest address. Luckily, Tilores
contains an internal submission timestamp, which is by default not exposed via
the API, but can be accessed with custom code.

#### Golden Record Example Implementation

Let's create the custom integration. Start by creating a new file
`schema/goldenrecord.graphqls` in your Tilores project:

```graphql
extend type RecordInsights {
  goldenRecord: GoldenRecord
}

type GoldenRecord {
  name: String!
  highQualityName: Boolean!
  address: String
}
```

With the `extend type` declaration we can extend the type definition for
`RecordInsights` which was defined in `recordinsights.graphql`. While it would
be possible to simply adjust that file, we recommend not to do so to prevent
issues with future upgrades via the `tilores-cli`.

Regenerate the models and resolvers by running `go generate ./...` on your
project folder. This will create a resolver similar to the following in the file
`graph/goldenrecord.graphqls`.

```go
func (r *recordInsightsResolver) GoldenRecord(ctx context.Context, obj *model.RecordInsights) (*model.GoldenRecord, error) {
	panic(fmt.Errorf("not implemented: GoldenRecord - goldenRecord"))
}
```

From there we can start with the actual implementation. We can mostly make use
of the existing insights features by using the
[tilores-insights](https://github.com/tilotech/tilores-insights) library
directly.

To add the library, add the following code to the `import` section:

```golang
import (
  // omitted lines
  insights "github.com/tilotech/tilores-insights/record"
)
```

Then create a filter for the relevant records:

```golang
  records, err := insights.Filter(obj.APIRecords, []*insights.FilterCondition{
    {
      Path: "source",
      Equals: "SOURCE_A",
    },
  })
  if err != nil {
    return nil, err
  }
  isHighQualityName := true
  if len(records) == 0 {
    records = obj.APIRecords
    isHighQualityName = false
  }
```

With that in place, we're already able to provide the value for the
`highQualityField` (because the `person.name` is a required attribute and must be
populated).

With `records` we can now use the `FrequencyDistribution`:

```golang
  fd, err := insights.FrequencyDistribution(records, "person.name", false, 1, false)
  if err != nil {
    return nil, err
  }
  name := fd[0].Value.(string)
```

Since we have no dedicated timestamp field and record insights cannot access the
records meta data, we need to sort without a helper function and get the
address. Note, that we're not using the `records` as this would only contain a
subset from certain sources, but requirement was for all records.

```golang
  addresses := make([]string, 0, len(obj.APIRecords))
  submitTimestamps := make([]time.Time, 0, len(obj.APIRecords))
  idxs := make([]int, 0, len(obj.APIRecords))
  for _, record := range obj.APIRecords {
    address, err := insights.ExtractString(record, "person.address", false)
    if err != nil {
      return nil, err
    }
    if address == nil || *address == "" {
      continue
    }
    idxs = append(idxs, len(idxs))
    addresses = append(addresses, *address)
    submitTimestamps = append(submitTimestamps, *record.Meta.SubmitTimestamp)
  }
  var address *string
  if len(addresses) > 0 {
    slices.SortFunc(idxs, func(i, j int) int {
      return submitTimestamps[j].Compare(submitTimestamps[i]) // sort reversed
    })
    address = &addresses[idxs[0]]
  }
```

Last step is to return the final result:

```golang
  return &model.GoldenRecord{
    Name: name,
    HighQualityName: isHighQualityName,
    Address: address,
  }, nil
```

After deploying these changes, the query can now be changed to:

```graphql
query {
  entity(input: {id: "123"}) {
    entity {
      recordInsights {
        goldenRecord {
          name
          highQualityName
          address
        }
      }
    }
  }
}
```

Resulting in the following output:

```json
{
  "data": {
    "entity": {
      "entity": {
        "recordInsights": {
          "goldenRecord": {
            "name": "John Smith",
            "highQualityName": true,
            "address": "the latest address"
          }
        }
      }
    }
  }
}
```