How to setup primary keys and indexes for target schema

[gkarugi]

Hey all,

I have been researching on how to add primary keys and indexes to the target schema. Without these indexes, the target becomes very slow if the data amount is huge.

I cannot seem to find any documentation or code sample of such a usecase on the docs, slack and github.

It seems like singer/meltano supports setting this, but currently cannot figure out how.

see primary keys reference
https://github.com/shaped-ai/target-clickhouse/blob/f70e4a7f92162922dae0d7936639c794a837d41e/target_clickhouse/connectors.py#L89

[edgarrmondragon]

Hi @gkarugi! The tap communicates that information to the target via the catalog metadata¹.

The docs are indeed lacking an example for that, but you can use the metadata extra² to set them:

extractors:
- name: your-tap
  metadata:
    some_stream_id:
      table-key-properties: [id]

We do have an issue to add examples like this to the Meltano docs: meltano/meltano#7316

Footnotes

1. https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md#metadata ↩

2. https://docs.meltano.com/concepts/plugins/#metadata-extra ↩