refactor(duckdb): pass temp_directory through as-is to duckdb.connect
#10433
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
NickCrews
force-pushed
the
duckdb-default-cache-dir
branch
from
89bee38
to
0090b0c
Compare
github-actions
bot
added
tests
NickCrews
added a commit
to NickCrews/ibis
that referenced
this pull request
Nov 4, 2024
NickCrews
force-pushed
the
duckdb-default-cache-dir
branch
from
0090b0c
to
1f724d4
Compare
NickCrews
added a commit
to NickCrews/ibis
that referenced
this pull request
Nov 4, 2024
Also, only create parent of the temp dir. duckdb will make dir itself only if it needs. Maybe the user can avoid a temp dir if they are only doing low-memory stuff. fixes ibis-project#10433
NickCrews
force-pushed
the
duckdb-default-cache-dir
branch
from
1f724d4
to
48baa07
Compare
cpcloud
requested changes
Nov 5, 2024
cpcloud
reviewed
Nov 5, 2024
cpcloud
requested changes
Nov 5, 2024
cpcloud
requested changes
Nov 11, 2024
cpcloud
force-pushed
the
duckdb-default-cache-dir
branch
from
9a816e9
to
ac0b651
Compare
…ect` BREAKING CHANGE: Special handling of the `temp_directory` argument passed to Ibis is removed in favor of passing the argument through directly to `duckdb.connect`. Interior nodes of directory trees must be created, e.g., using `Path.mkdir(exists_ok=True, parents=True)`, `mkdir -p`, etc.
cpcloud
force-pushed
the
duckdb-default-cache-dir
branch
from
ac0b651
to
03c8854
Compare
cpcloud
changed the title
temp_directory through as-is to duckdb.connect
cpcloud
approved these changes
Nov 11, 2024
NickCrews
added
the
breaking change
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Let's make the pass-through from ibis to duckdb more transparent, so you get closer to the same behavior as when using
import duckdb; duckdb.connect().I considered also removing the nicety of creating the temp_dir for users, but decided to keep it.
Now, we only create parent. duckdb will make dir itself only if it needs. This means that for low-memory loads,
a dir might never be needed. This better matches duckdb's behavior.
Also, adjust our test to really check for the end desired behavior, not an implementation detail
of if WE create the dir.
Related to #6974
BREAKING CHANGE: Special handling of the
temp_directoryargument passed to Ibis is removed in favor of passing the argument through directly toduckdb.connect. Interior nodes of directory trees must be created, e.g., usingPath.mkdir(exists_ok=True, parents=True),mkdir -p, etc.