Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: Add comparison between Trustfall and SQL #558

Open
obi1kenobi opened this issue Feb 28, 2024 · 0 comments
Open

Docs: Add comparison between Trustfall and SQL #558

obi1kenobi opened this issue Feb 28, 2024 · 0 comments
Labels
A-docs Area: documentation for the query language or library APIs

Comments

@obi1kenobi
Copy link
Owner

We should help prospective users decide whether Trustfall is for them or not. This should be in the FAQ and perhaps could make a good blog post topic as well.

TL;DR:

  • If your data is already in a SQL database, use SQL.
  • If you need to query data format or schema that may change over time, Trustfall is really good at that. For example, cargo-semver-checks uses Trustfall to query a JSON format that has breaking changes around once a month on average — but the Trustfall queries don't change at all.
  • If your queries get data from a remote API that is rate-limited or charges per request, Trustfall is really good at that. Its queries are lazily evaluated, so you'll only pay for what you use. This is easier and cheaper than building ETL pipelines, where you'd pay for running all the API calls regardless of whether your queries use the results or not.
  • Trustfall doesn't (yet) natively support datetimes; the currently-recommended workaround is to encode them as string. If this won't work for your workload, use SQL.
  • Trustfall doesn't (yet) natively support ORDER BY, though some schemas may allow setting ordering in your query. The best way to impose a global order on results is to run a query to completion, then sort the results (e.g. by deriving PartialOrd, Ord on your query result struct, then collecting the results iterator and sorting it). If this won't work for your workload, use SQL instead.

In some cases, you may want to use SQL together with Trustfall. For example, it can be useful to combine SQLite with Trustfall to cache the results of complex transformations or expensive API calls. As another example, if your dataset is split between a SQL database and a series of files in S3, Trustfall could be used to run federated (cross-datasource) queries across SQL and S3. In both cases, Trustfall is the query engine and SQL is a storage system that helps run portions of those queries as needed.

@obi1kenobi obi1kenobi added the A-docs Area: documentation for the query language or library APIs label Feb 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-docs Area: documentation for the query language or library APIs
Projects
None yet
Development

No branches or pull requests

1 participant