Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run profile queries after fetching columns #2677

Merged
merged 9 commits into from
Jul 17, 2023
Merged

Conversation

AdityaHegde
Copy link
Collaborator

@AdityaHegde AdityaHegde commented Jun 22, 2023

Checklist

  • Manual verification
  • Unit test coverage
  • E2E test coverage
  • Needs manual QA?

Summary

Issue addressed:

closes #2662

Details:

Steps to Verify

  1. Create a model from any source.
  2. Add select <column1> from <source>
  3. Replace the column1 from column2 by pasting and not typing. This way at a single shot the column is replaced.
  4. Doing this back and forth should not add any 400 errors for profiling queries.

Comment on lines 101 to 114
const ProfilingQueryExtractor =
/v1\/instances\/[a-zA-Z0-9-]+\/queries\/([a-zA-Z0-9-]+)\/tables\/(.+?)"/;
export function isProfilingQuery(
queryHash: string,
name: string,
ignoreProfileColumns = false
) {
const queryExtractorMatch = ProfilingQueryExtractor.exec(queryHash);
if (!queryExtractorMatch) return false;
// TODO: move the query matching to a separate module and reuse in http queue
const [, type, table] = queryExtractorMatch;
return (
table === name && (!ignoreProfileColumns || type !== "columns-profile")
);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is hard to follow. Could it be refactored to contain isTableProfileQuery and isColumnProfileQuery, or something along those lines?

Comment on lines 152 to 161
await queryClient.invalidateQueries({
predicate: (query) => {
// TODO: move the query matching to a separate module and reuse in http queue
const queryExtractorMatch = ProfilingQueryExtractor.exec(query.queryHash);
if (!queryExtractorMatch) return false;
const [, type, table] = queryExtractorMatch;
return table === name && type === "columns-profile";
},
type: "active",
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to isProfilingQuery, this is hard to follow. Could one function contain the logic, and you could call it here?

@@ -113,6 +113,7 @@
{hideRight}
{hideNullPercentage}
{compact}
enableProfiling={!$profileColumns?.isFetching}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it work with $profileColumns?.isSuccess or $profileColumns?.data? Either of those would be more semantically clear, IMO.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enableProfiling has to be false when we are refetching profileColumns. isSuccess is true even while the profileColumns is being refetched.

@ericpgreen2
Copy link
Contributor

I think this looks good. Besides my inline comments, a higher-level thought:

  • It'd be ideal if we didn't have to do two sets of queries: one for the table columns, one for the column profiles. Do we have plans to make this one API call – something like GetColumnProfiles(tableName)?

@AdityaHegde AdityaHegde marked this pull request as ready for review July 3, 2023 15:00
Comment on lines 24 to 27
const TopLevelProfilingQuery: Partial<Record<QueryRequestType, boolean>> = {
[QueryRequestType.TableColumns]: true,
[QueryRequestType.TableRows]: true,
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to include TableCardinality in this list? Then the variable could be TableProfilingQuery (not TopLevelProfilingQuery). It's not obvious to me why TableColumns and TableRows are in this unique group together.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes updating

Copy link
Contributor

@ericpgreen2 ericpgreen2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one question, else looks good 👍

@AdityaHegde AdityaHegde merged commit 8719932 into main Jul 17, 2023
@AdityaHegde AdityaHegde deleted the profiling-ordering branch July 17, 2023 06:01
djbarnwal pushed a commit that referenced this pull request Aug 3, 2023
* Run profile queries after fetching columns

* Fixing prettier

* Moving query matching to a new file

* Fix lint

* Updated E2E

* Fix source profiling

* TopLevelProfiling => TableProfiling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

When changing a model column name, the profiling requests for that column uses the old column name
2 participants