[sc-28901] Validate query logs as they're parsed

[usefulalgorithm]

🤔 Why?

We should validate the query logs as they're parsed, otherwise bugs are harder to find when they get to the lineage parser.

🤓 What?

Validate the parsed query logs before writing them to file.

Tested this against our own Snowflake instance (13,589 queries)

• No validation: 36 secs

• Validated: 1 min 37 secs

🧪 Tested?

Tested against

• BigQuery
• Snowflake (see above)
• Unity Catalog

☑️ Checks

• My PR contains actual code changes, and I have updated the version number in pyproject.toml.

[github-actions]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
12578	11242	89%	85%	🟢

New Files

No new covered files...

Modified Files

File	Coverage	Status
metaphor/common/file_sink.py	98%	🟢
TOTAL	98%	🟢

updated for commit: cca070e by action🐍

[codecov]

Codecov Report

Attention: Patch coverage is 75.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 89.37%. Comparing base (167ccb7) to head (cca070e).

Files with missing lines	Patch %	Lines
metaphor/common/file_sink.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #986      +/-   ##
==========================================
- Coverage   89.38%   89.37%   -0.01%     
==========================================
  Files         191      191              
  Lines       12575    12578       +3     
==========================================
+ Hits        11240    11242       +2     
- Misses       1335     1336       +1     

Flag	Coverage Δ
	89.37% <75.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[elic-eon]

Didn't we validate them when ingestion?

[usefulalgorithm]

Didn't we validate them when ingestion?

No, ingestion does not look at what's in the query log files.

[elic-eon]

Didn't we validate them when ingestion?

No, ingestion does not look at what's in the query log files.

Got it, can we also valid entities in the MCE as well? We encounter the MCE validation error when ingestion many times, if we can catch the error earlier, it is better.

[usefulalgorithm]

Didn't we validate them when ingestion?

No, ingestion does not look at what's in the query log files.

Got it, can we also valid entities in the MCE as well? We encounter the MCE validation error when ingestion many times, if we can catch the error earlier, it is better.

MCEs are partially validated: https://github.com/MetaphorData/connectors/blob/main/metaphor/common/sink.py#L23C1-L28C14

It only validates against the json schema though, so it's kind of limited.