Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable complex query filters #373

Open
pudo opened this issue Feb 26, 2021 · 1 comment
Open

Enable complex query filters #373

pudo opened this issue Feb 26, 2021 · 1 comment

Comments

@pudo
Copy link
Contributor

pudo commented Feb 26, 2021

Riffing off of #366, we've been discussing a more advanced filter mechanism in entity mappings both for queries and for schema assignment. I'm sketching this out here to get some feedback.

The basic idea is to make it possible to compile nested complex queries. This might have to be incompatible to the existing filter, filter_not rules in mappings, so we would be introducing a new query section (which filter and filter_not would be transparently re-written to). The query syntax itself could riff off of the MongoDB syntax, which manages to encapsulate compound queries in JSON.

The second purpose of this enhancement would be to make schema conditional on column contents. For example, we often see tables where companies and people are mixed, and whether a row describes a person or company is determined by a value like individual/entity in one column.

Here's a sketch of how we could address both issues:

query:
  table: zz_donors
  query:
    $and:
      country: xk
      $or:
        political_party:
          $like: "%conservative%"
        district: ["Southern", "Northern"]
  entities:
    donor:
      schema_query:
        Person:
          donor_type: "individual"
        Company:
          donor_type: ["corporate", "business"]

Especially keen for feedback from @uhhhuh @brrttwrks :)

@uhhhuh
Copy link
Contributor

uhhhuh commented Feb 26, 2021

I like this solution to the problem of long unmanageable mappings. Would it be possible and reasonable to also support subqueries for properties? Happened before (customs declarations among others): substitute some conventional value with the one from a dictionary.
Given two sections from a mapping:

tables:
  - table: proceedings_documents
    alias: pr
  - table: constans
    alias: const
joins:
  - left: pr.proceeding_type
    right: const.public_code
filters:
  const.type: "type"
entities:
  case:
    schema: CourtCase
    properties:
      type:
        column: const.description_EN

tables:
  - table: proceedings_documents
    alias: pr
  - table: constans
    alias: const
joins:
  - left: pr.proceeding_category
    right: const.public_code
filters:
  const.type: "category"
entities:
  case:
    schema: CourtCase
    properties:
      category:
        column: const.description_EN


be able to reduce them to something like

tables:
  - table: proceedings_documents
    alias: pr
  - table: constants
    alias: const
entities:
  case:
    schema: CourtCase
    properties:
      type:
        column: const({public_code: pr.proceeding_type, type: "type"}, {description_EN: 1})
      category:
        column: const({public_code: pr.proceeding_form, type: "category"}, {description_EN: 1})

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Status: 📋 Backlog
Development

No branches or pull requests

3 participants