Recommendation for list APIs that are sparsely populated (e.g. Meltano API and PokeAPI)

[aaronsteers]

As we move towards a 1.0 release of the SDK, there's one API results pattern that as of yet, I don't think we support.

That is, we don't have a strong design recommendations for APIs whose list endpoint doesn't return the actual items but instead returns url or ref pointers to the actual detail data.

Related to:

• New silent parent stream or "MultiStream" class proposal for taps #166
• Streamline complementary REST requests #93

Examples

Example 1: PokeAPI

The PokeAPI equivalent of a list API returns results that require additional calls to resolve.

• Doc site: https://pokeapi.co/ - type pokemon into the text box and press Submit to see results
• Direct API endpoint: https://pokeapi.co/api/v2/pokemon

Output:

Details

{
  "count": 1154,
  "next": "https://pokeapi.co/api/v2/pokemon?offset=20&limit=20",
  "previous": null,
  "results": [
    {
      "name": "bulbasaur",
      "url": "https://pokeapi.co/api/v2/pokemon/1/"
    },
    {
      "name": "ivysaur",
      "url": "https://pokeapi.co/api/v2/pokemon/2/"
    },
    {
      "name": "venusaur",
      "url": "https://pokeapi.co/api/v2/pokemon/3/"
    },
    {
      "name": "charmander",
      "url": "https://pokeapi.co/api/v2/pokemon/4/"
    },
    {
      "name": "charmeleon",
      "url": "https://pokeapi.co/api/v2/pokemon/5/"
    },
    {
      "name": "charizard",
      "url": "https://pokeapi.co/api/v2/pokemon/6/"
    },
    {
      "name": "squirtle",
      "url": "https://pokeapi.co/api/v2/pokemon/7/"
    },
    {
      "name": "wartortle",
      "url": "https://pokeapi.co/api/v2/pokemon/8/"
    },
    {
      "name": "blastoise",
      "url": "https://pokeapi.co/api/v2/pokemon/9/"
    },
    {
      "name": "caterpie",
      "url": "https://pokeapi.co/api/v2/pokemon/10/"
    },
    {
      "name": "metapod",
      "url": "https://pokeapi.co/api/v2/pokemon/11/"
    },
    {
      "name": "butterfree",
      "url": "https://pokeapi.co/api/v2/pokemon/12/"
    },
    {
      "name": "weedle",
      "url": "https://pokeapi.co/api/v2/pokemon/13/"
    },
    {
      "name": "kakuna",
      "url": "https://pokeapi.co/api/v2/pokemon/14/"
    },
    {
      "name": "beedrill",
      "url": "https://pokeapi.co/api/v2/pokemon/15/"
    },
    {
      "name": "pidgey",
      "url": "https://pokeapi.co/api/v2/pokemon/16/"
    },
    {
      "name": "pidgeotto",
      "url": "https://pokeapi.co/api/v2/pokemon/17/"
    },
    {
      "name": "pidgeot",
      "url": "https://pokeapi.co/api/v2/pokemon/18/"
    },
    {
      "name": "rattata",
      "url": "https://pokeapi.co/api/v2/pokemon/19/"
    },
    {
      "name": "raticate",
      "url": "https://pokeapi.co/api/v2/pokemon/20/"
    }
  ]
}

The challenge here is that the results come back with just name and url, which is basically a ref to actually get the entity info.

{
  "count": 1154,
  "next": "https://pokeapi.co/api/v2/pokemon?offset=20&limit=20",
  "previous": null,
  "results": [
    {
      "name": "bulbasaur",
      "url": "https://pokeapi.co/api/v2/pokemon/1/"
    },
   ...

If we follow the URL, we'll get the actual data: https://pokeapi.co/api/v2/pokemon/1/

Example 2: Meltano Hub API

Our own Meltano API has a similar pattern, where the index doesn't contain the data on each entity.

• https://hub.meltano.com/meltano/api/v1/plugins/utilities/index

{
  "great_expectations": {
    "logo_url": "https://hub.meltano.com/assets/logos/utilities/great_expectations.png",
    "default_variant": "great-expectations",
    "variants": {
      "great-expectations": {
        "ref": "https://hub.meltano.com/meltano/api/v1/plugins/utilities/great_expectations--great-expectations"
      }
    }
  },
  "sqlfluff": {
    "logo_url": "https://hub.meltano.com/assets/logos/utilities/sqlfluff.png",
    "default_variant": "sqlfluff",
    "variants": {
      "sqlfluff": {
        "ref": "https://hub.meltano.com/meltano/api/v1/plugins/utilities/sqlfluff--sqlfluff"
      }
    }
// ...
}

Possible solutions

The two paths I'm aware of this are:

Option 1 - silent and/or non-utilized parent stream

• Similar design language 'silent' stream option here, but applied to different target use cases: New silent parent stream or "MultiStream" class proposal for taps #166

In this first approach, we would create a 'silent parent' stream such as PokemonIndexStream in the case of the PokeAPI and MeltanoPluginsIndex in the case of the Meltano API. This could then have a child stream of PokemonStream or MeltanoPluginsStream respectively.

The challenge with this approach is having to create dual Stream classes (not a big deal) but then also declaring the parent one to be 'silent'. The design just feels a bit awkward and 'extra', even if the underlying technical components are not that challenging.

This approach should work even if the parent stream is not silent, but it might be confusing to an end user to have an 'index' stream as well as the base entity stream with detailed data. For example, this would create two target tables: plugin_index and plugins, with the first one not offering much value over the second.

Option 2 - parse the inner URL and merge results with list view

• Streamline complementary REST requests #93

In the second approach, we introduce a means of sending complementary requests, so that only a single stream is needed. During post_process() or another method, the developer makes a call to the embedded url and then grafts the results into the base entity description.

@edgarrmondragon - Do you have any thoughts on which of these are 'recommendable' to tap developers? Have you run into this and do you have any best practices?

[edgarrmondragon]

I like option 1 better simply because developers may want access to all the niceties of the Stream class, like pagination, URL interpolation for child streams, etc.

Solely for the purpose of enriching a stream record with results from a different endpoint, I think this approach of "silent" (or dummy, as I've been calling them so far 😅) streams approach might be good enough and is already encountered in the wild: https://meltano.slack.com/archives/C01PKLU5D1R/p1655987043129269

My example implementation (of the PokeAPI, no less!) uses that approach, but it requires declaring the dummy stream with placeholders what are ultimately discarded. It'd be better to offer developers a better way of doing this.

[aaronsteers]

@edgarrmondragon - re:

My example implementation (of the PokeAPI, no less!) uses that approach

Funny! 😆 I didn't know you'd created this, but yes, great sample implementation.

It'd be better to offer developers a better way of doing this.

What do you think of offering developers an alternative base class for something like SilentStream or IndexStream or DummyStream? This class would have all the conveniences of the Stream class, except:

1. Doesn't need a name besides the class name.
2. Doesn't emit records.
3. Doesn't send STATE.
4. Doesn't participate in the Catalog.

Perhaps this could be defined as a MetaClass so it could be combined with other base classes, with the primary parent coming from the SDK or from the developers own custom Stream classes. What do you think?