Deployment

The Tilores CLI comes with the commands to deploy your customized instance into your own AWS account.

Prerequisites

For deploying we will need that terraform in at least version 1.2 and git in any version is installed.

Verify your terraform and git installation by running

terraform version
git version

Furthermore you need to provide valid AWS credentials into either your credentials file or set them up as environment variables. For security reasons, we do not offer flags for providing the credentials. If you are using a credentials file, you can optionally specify a profile using the --profile flag while running the deploy command.

Deploy

Deploying Tilores is as simple as running:

tilores-cli deploy --region <your-aws-region>

The --region flag is required and must be one of the official AWS region codes, e.g. us-east-1 for North Virginia or eu-west-1 for Ireland. Check supported regions.

This will automatically create all the needed resources in your AWS account. Once the deployment finished, you can request an access token using the outputted token_url and start sending requests using the token_url. For more details, please refer to the authentication documentation.

Customize Deployment

The terraform files for the deployment are stored under deployment/tilores and can be customized to some extend or you can add your own features where needed. For a full list of options, please refer to the documentation in the Terraform registry.

Change Resource Prefix

The resource prefix is a short text that every AWS resource created during the deployment is prefixed with. For most of the resources this is only for convenience, but for some resources that must have a global unique name, this is important (e.g. S3 buckets).

You can change the name in deployment/tilores/main.tf in the local resource_prefix variable. We recommend a value that typically only makes sense for your company, most of the times your company name and a stage, e.g. mycompany-prod.

Performance Considerations

By default Tilores uses SQS to buffer messages between the API and the assembly Lambda function. By default a maximum of 10 concurrent Lambda functions will be invoked. While this is often good enough for development instances, production instances often require a higher throughput, especially during initial loads.

A single Lambda function typically can assemble between 2 and 5 records per second depending on the rule configurations complexity. If you need a throughput of 1,000 records per second, you would therefore need between 200 to 500 concurrent Lambda functions.

To increase the maximum number of concurrent Lambda functions, you can change the tilores module in deployment/tilores/main.tf like this:

module "tilores" {
  # other parameters
  assemble_parallelization_sqs = 200
}

Lambda concurrency might be limited by the AWS service quota for concurrent executions. These are soft limits, that can easily be increased. Depending on your support plan with AWS this may take a few days and should be done before planing a large ingestion.

Shard Count

One of the biggest advantages of Tilores is, that you don't need to worry about scaling. However, the automatic scaling feature for Kinesis streams provided by AWS does not (yet) work well. For that reason, the user currently would need to configure the amount of shards for the raw data and for the entity events.

The amount of raw data shards are important when it comes to submitting records. Each shard handles approx. 10 to 20 submissions per second. However, you don't need to scale for spikes, but for the average. This is because all submitted records stay in the shard for up to 24 hours, before they must have been processed or they are lost. Unless you have batch processes, you most likely want to have them processed within a few seconds or within a few minutes.

The amount of shards for entity events is typically 1/8th of the amount of raw data shards. This depends of course on how you use that stream.

To use 16 raw data shards and 2 entity event shards, you can change the tilores module in deployment/tilores/main.tf like this:

module "tilores" {
  # other parameters
  rawdata_stream_shard_count      = 16
  entity_event_stream_shard_count = 2
}

Example: Realtime Processing with Spikes

Let's say you are expecting on average 100 submissions during your normal business hours (ignoring the idle night time) and have spikes of up to 250 submissions for a few minutes every now and then. Since each shard can handle typically at least 10 submissions per second, you would need 10 shards for the average. However, shards should be provisioned in multiples of 2, so the recommendation would be 16 shards, which would provide you with at least 160 submissions per second (or more than 13.8 million per day). Enough to let the backlog of unprocessed items not to grow to fast during spikes. In that case you would need 2 shards for the entity event stream.

Example: Batch Imports

When your business is more batch oriented, you would most likes want to consider the amount of submissions per day or for one time submissions per batch. Let's say, you typically have 4 batches per day, with an average of 250 thousand submissions per batch. You may want to scale your shards so that all submissions will be processed in 12 hours, so your shard capacity should be at 2 million submissions per day or roughly 23 submissions per second. You can now either use 2 or 4 raw data shards and 1 shard for the entity event stream. In most cases 2 shards should be fine for these amounts of records.

Fix Versions

You may want to prevent automatic upgrades when reapplying your API changes. You can change this in the module tilores in deployment/tilores/main.tf.

For fixing the core version you can modify it like this:

module "tilores" {
  # other parameters
  core_version = "v0-1-0"
}

You can also fix it to a major version, e.g. v0.

Please refer to the modules input parameters for the full list of versions, that can be fixed.

Analytics Module

Tilores is designed for realtime use cases and optimized for querying individual entities. If you want to enable the advanced analytics, you can enable this in the tilores module in deployment/tilores/main.tf:

module "tilores" {
  # other parameters
  enable_analytics = true
  snapshot_query_mode = "WAIT" # optional
}

Optionally, we recommend to set the snapshot query mode to WAIT to ensure, that snapshot creation is guaranteed. Omit this option for development instances to reduce the infrastructure costs.

Review

To let people or agents decide weak links in a review queue, enable the review resources in the tilores module in deployment/tilores/main.tf:

module "tilores" {
  # other parameters
  enable_review = true
}

The rule configuration also needs a review block, see Enable the Review.

Entity Stream API

To let clients read the entity event stream through the API, create the stream and enable the API in the tilores module in deployment/tilores/main.tf:

module "tilores" {
  # other parameters
  create_entity_stream     = true
  enable_entity_stream_api = true
}

Both parameters are required. Keep entity_event_stream_shard_count at 0, because the API can only read an SQS based stream.

Each event is delivered to one consumer only. Do not enable the API if anything else consumes the stream, for example a webhook.

A client needs the privilege tilores/mutation.entityStream. tilores-cli upgrade adds it to available_scopes and to the default client. Add it to all other clients that need it, see Authorization.

Non-local State Files

The terraform state files store the current state of your deployed resources. By default Tilores uses a local state, meaning that it is stored within the project under deployment/tilores/terraform.tfstate. While this is easy to setup, it is not a good practice to do so in environments with multiple developers or ops. Most of the time you want to store the state in a system that is reachable by multiple people. Terraform provides various backends for this purpose. We won't go into details, but would recommend you to store your state in an S3 bucket. You can either add the backend configuration in deployment/tilores/provider.tf or create a new .tf file for that in the same folder.

Access Token Validity Duration

Access token validity duration is the time after which the access token is no longer valid and cannot be used.

The default value is 60 minutes. A different value can be provided in cognito module in deployment/tilores/main.tf by adding access_token_validity = <yourOwnValue> to the list of parameters.

Example: for setting the duration to 90 minutes, modify the cognito module as follows:

module "cognito" {
  # other parameters
  access_token_validity = 90
}

Verify Binaries

Tilores runs binaries that we build in your AWS account. For every release, we publish a checksum file next to each zip file of these binaries, e.g. assemble.zip.sha256 next to assemble.zip. The checksum file contains the SHA-256 hash of the zip file, so that you can check that a file you downloaded or deployed is identical to the published one. Releases from before the introduction of checksum files do not have them.

The files are in the artifacts bucket of your AWS region. Each project has a folder for every version, e.g. v0-85-0:

s3://tilotech-artifacts-<your-aws-region>/tilotech/<project>/<version>/

The version input is the parameter of the module tilores that selects the version folder, see Fix Versions. Its default is v0.

<prefix> is the resource prefix of your deployment. If you deploy with a workspace, it is followed by -<your-env-name>.

Check a File Before Deploying

Download the file and its checksum file, then compare them:

aws s3 cp s3://tilotech-artifacts-<your-aws-region>/tilotech/tilores-core/<version>/assemble.zip .
aws s3 cp s3://tilotech-artifacts-<your-aws-region>/tilotech/tilores-core/<version>/assemble.zip.sha256 .
sha256sum --check assemble.zip.sha256

On macOS, use shasum -a 256 --check assemble.zip.sha256 for the last command. The result assemble.zip: OK means that the file is identical to the published one. For a file of another project, use the project and the file from the table.

Check Deployed Code

AWS Lambda shows the SHA-256 hash of the code of every function as CodeSha256, encoded in base64. The following commands download the checksum file, convert the hash of the function and compare the two. For another function, use its project, file and name from the table:

aws s3 cp s3://tilotech-artifacts-<your-aws-region>/tilotech/tilores-core/<version>/assemble.zip.sha256 .
hash=$(aws lambda get-function --function-name <prefix>-tilores-assemble \
  --query Configuration.CodeSha256 --output text | base64 -d | od -An -tx1 | tr -d ' \n')
grep "^$hash " assemble.zip.sha256

If the code is identical to the published one, grep prints the content of the checksum file, e.g. 2a32... assemble.zip. If grep prints nothing, the deployed code is not the code of this version.

Remove

Removing the created resources from your AWS account is as simple as deploying:

tilores-cli destroy --region <your-aws-region>

Deploy Multiple Instances

Multiple Tilores instances can be deployed from the same project. This can be either in the same AWS account or in different ones, e.g. for production and development environments.

Simply use the flag workspace when deploying your environment as follows

tilores-cli deploy --region <your-aws-region> --workspace <your-env-name>

tilores-cli destroy --region <your-aws-region> --workspace <your-env-name>

All resources are prefixed with the workspace you provide, which allows multiple instances to be deployed into the same AWS account.

Supported Regions

Deployment is currently only possible to the following AWS regions:

• US East (N. Virginia) us-east-1
• US East (Ohio) us-east-2
• US West (Oregon) us-west-2
• Europe (Frankfurt) eu-central-1
• Europe (Ireland) eu-west-1
• Europe (London) eu-west-2
• Europe (Milan) eu-south-1
• Europe (Zurich) eu-central-2
• Asia Pacific (Singapore) ap-southeast-1