Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto Install/Load extensions is causing schema inference discrepancies #124

Open
rahulj51 opened this issue Oct 1, 2024 · 2 comments
Open

Comments

@rahulj51
Copy link

rahulj51 commented Oct 1, 2024

We use Duckdb for inferring Parquet file schemas. The latest update (1.1.1) has changed the default behavior of parquet schema inference if spatial extension is installed. The problem is that the nodejs module automatically installs/loads the extensions. Is there a way to disable/override this behavior? We do not want spatial extension to be installed. I see that the DUCKDB_EXTENSION_AUTOINSTALL_DEFAULT flag is set by default. Is there a way to unset it (maybe using an environment variable) ?

@freakone
Copy link

freakone commented Oct 1, 2024

Additionally, even if we use the SQL to disable the configuration options, spatial extension gets loaded anyways.

This is after we start the DB:

[
  {
    extension_name: 'spatial',
    installed: true,
    loaded: false,
    description: 'Geospatial extension that adds support for working with spatial data and functions'
  }
]

then we set the configuration params and confirm:

[ { "current_setting('autoload_known_extensions')": false } ]
[ { "current_setting('autoinstall_known_extensions')": false } ]

then we read the parquet file:

DESCRIBE select * from read_parquet([${files}],...

and after that:

[
  {
    extension_name: 'spatial',
    installed: true,
    loaded: true,
    description: 'Geospatial extension that adds support for working with spatial data and functions'
  }
]

@carlopi
Copy link
Collaborator

carlopi commented Oct 1, 2024

This is a problem in the DuckDB side, to be fixed there, where the autoloading in the parquet extensions do not properly checks configuration (and the API around that is not helping avoiding problems, to be improved).

I am not aware of any proper workaround in the current version, barred something weird / heavy handed like placing a file that needs admin privileges to be read at ~/.duckdb/extensions/v1.1.1/PLATFORM_NAME/spatial.duckdb_extension (then I would expect LOAD to fail silently).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants