Out of Memory error while spill is enabled

[im-trisha]

I'm trying to analyze open food facts' database, but i'm getting an out of memory exception. The /tmp/duckswap gets created and populated, but i still get the exception. I already raised the issue on duckdb async but i think this could be a duckdb problem

import { Database } from "duckdb-async";


async function saveDataPostgres() {
    const db = await Database.create("/tmp/tmp.db", {
        max_memory: "512MB",
        threads: "4",
        temp_directory: "/tmp/duckswap",
    });
    await db.all(`CREATE TABLE products AS
        SELECT product_name_en, product_name_it, serving_quantity, serving_size, languages, nutriments FROM read_ndjson('/tmp/openfoodfacts-products.jsonl.gz', ignore_errors=True);`);
}

saveDataPostgres()

2024-08-26 14:53:09 node:internal/process/promises:391
2024-08-26 14:53:09     triggerUncaughtException(err, true /* fromPromise */);
2024-08-26 14:53:09     ^
2024-08-26 14:53:09 
2024-08-26 14:53:09 [Error: Out of Memory Error: failed to allocate data of size 256.0 MiB (271.9 MiB/488.2 MiB used)] {
2024-08-26 14:53:09   errno: -1,
2024-08-26 14:53:09   code: 'DUCKDB_NODEJS_ERROR',
2024-08-26 14:53:09   errorType: 'Out of Memory'
2024-08-26 14:53:09 }
2024-08-26 14:53:09 
2024-08-26 14:53:09 Node.js v20.17.0

[im-trisha]

As a sidenote, i am able to work with the database in general. For example, i can run DESCRIBE (SELECT * FROM read_ndjson('openfoodfacts-products.jsonl.gz', ignore_errors=True))