Skip to content

Commit

Permalink
add to_query function
Browse files Browse the repository at this point in the history
  • Loading branch information
buremba committed Aug 28, 2024
1 parent e17adfd commit 4761a9e
Show file tree
Hide file tree
Showing 3 changed files with 62 additions and 13 deletions.
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@

UniverSQL is a Snowflake proxy that allows you to run SQL queries **locally** on Snowflake Iceberg tables and Polaris catalog, using DuckDB.
You can join Snowflake data with your local datasets, **without any need for a running warehouse**.
UniverSQL relies on Snowflake and Polaris for access control and data catalog so it's complementary to your Snowflake workloads.
UniverSQL relies on Snowflake for access control and data catalog so it's complementary to your Snowflake workloads.

> [!WARNING]
> Any SQL client that supports Snowflake, also supports UniverSQL as we implement Snowflake's API to support compatibility. If you run into any issue using an app or client, feel free to [create a discussion](https://github.com/buremba/universql/discussions/categories/quality-testing).
> Universql is not affiliated with Snowflake.

[![Demo](./resources/cli_demo.png)](https://www.youtube.com/watch?v=s1fpSEE-pAc)
Expand Down Expand Up @@ -226,14 +227,14 @@ You have two alternatives:
```
Dynamic tables are the recommended approach **if your native tables have more than 1B+ of rows** so that you can filter/aggregate them before pulling them into your local environment. If your native tables are small enough, consider switching them to Iceberg from Native.

2. (coming soon) You can use `universql.execute` function to run queries directly in Snowflake and return the result as a table. You can join native Snowflake table with your local files as follows:
2. You can use `to_query` function to run queries directly in Snowflake and return the result as a table. You can join native Snowflake table with your local files as follows:

```sql
SELECT * FROM table(universql.execute('select col1 from my_snowflake_table', {'target_lag': '1h'})) t1 join 'local://./my_local_table.csv' as t2 on t1.col1 = t2.col1;
SELECT * FROM table(to_query('select col1 from my_snowflake_table')) where col1 = 2
```

UniverSQL doesn't actually require you to create the `universql.execute` function in your Snowflake database. When you use the proxy server, it will create a query plan to execute your Snowflake query first and map the result as an Arrow table in DuckDB.
This approach is recommended for hybrid execution where you need to query your native Snowflake tables on the fly. UniverSQL has query caching based on the setting `target_lag`, which is saved locally.
UniverSQL doesn't actually require you to create the `to_query` function in your Snowflake database. When you use the proxy server, it will create a query plan to execute your Snowflake query first and map the result as an Arrow table in DuckDB.
This approach is recommended for hybrid execution where you need to query your native Snowflake tables on the fly.

## Only read-only `SELECT` queries can use local warehouse.

Expand Down
51 changes: 48 additions & 3 deletions testing.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,31 @@
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"outputs": [],
"source": [
"import snowflake.connector\n",
"\n",
"con = snowflake.connector.connect(connection_name='jinjat_aws_us')\n",
"cur = con.cursor()\n",
"cur.execute(\"select * from table(to_query('select 1'))\")"
],
"metadata": {
"collapsed": false,
"is_executing": true,
"ExecuteTime": {
"start_time": "2024-08-28T02:39:10.524576Z"
}
},
"id": "3f328a29bd4b799b"
},
{
"cell_type": "code",
"execution_count": 3,
"id": "initial_id",
"metadata": {
"ExecuteTime": {
"end_time": "2024-08-28T00:18:33.458129Z",
"start_time": "2024-08-28T00:18:30.285012Z"
"end_time": "2024-08-28T02:01:07.046307Z",
"start_time": "2024-08-28T02:01:01.735142Z"
}
},
"outputs": [],
Expand All @@ -21,6 +40,32 @@
"cur = con.cursor()"
]
},
{
"cell_type": "code",
"outputs": [
{
"data": {
"text/plain": " 1\n0 1",
"text/html": "<div>\n<style scoped>\n .dataframe tbody tr th:only-of-type {\n vertical-align: middle;\n }\n\n .dataframe tbody tr th {\n vertical-align: top;\n }\n\n .dataframe thead th {\n text-align: right;\n }\n</style>\n<table border=\"1\" class=\"dataframe\">\n <thead>\n <tr style=\"text-align: right;\">\n <th></th>\n <th>1</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <th>0</th>\n <td>1</td>\n </tr>\n </tbody>\n</table>\n</div>"
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"con.cursor().execute(\"select * from table(to_query('select 1'))\").fetch_pandas_all()\n"
],
"metadata": {
"collapsed": false,
"ExecuteTime": {
"end_time": "2024-08-28T02:02:57.660342Z",
"start_time": "2024-08-28T02:02:12.584126Z"
}
},
"id": "d00daefb9f311e9e",
"execution_count": 5
},
{
"cell_type": "code",
"execution_count": 24,
Expand Down
13 changes: 8 additions & 5 deletions tests/sqlglot_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,14 @@

from universql.warehouse.duckdb.duckdb import fix_snowflake_to_duckdb_types

# queries = sqlglot.parse("""
# SET tables = (SHOW TABLES);
#
# select * from tables
# """, read="snowflake")
queries = sqlglot.parse("""
SELECT * FROM TABLE(
TO_QUERY(
'SELECT * FROM IDENTIFIER($table_name)
WHERE deptno = TO_NUMBER(:dno)', dno => '10'
)
);
""", read="snowflake")


# query = sqlglot.parse_one("""
Expand Down

0 comments on commit 4761a9e

Please sign in to comment.