Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

import parquet-wasm from local bundle rather than CDN #457

Open
cornhundred opened this issue Apr 8, 2024 · 8 comments · May be fixed by #461
Open

import parquet-wasm from local bundle rather than CDN #457

cornhundred opened this issue Apr 8, 2024 · 8 comments · May be fixed by #461

Comments

@cornhundred
Copy link

Hi, is it possible to import parquet-wasm using esbuild rather than a CDN? I see that parquet-wasm is a dependency in the package.json

lonboard/package.json

Lines 5 to 20 in c8cf2e1

"dependencies": {
"@anywidget/react": "^0.0.4",
"@babel/runtime": "^7.24.1",
"@deck.gl/core": "^8.9.35",
"@deck.gl/extensions": "^8.9.35",
"@deck.gl/layers": "^8.9.35",
"@deck.gl/react": "^8.9.35",
"@geoarrow/deck.gl-layers": "^0.3.0-beta.16",
"apache-arrow": "^15.0.2",
"maplibre-gl": "^3.6.2",
"parquet-wasm": "0.5.0",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"react-map-gl": "^7.1.7",
"uuid": "^9.0.1"
},

but lonboard is also referencing a CDN here:

lonboard/src/parquet.ts

Lines 2 to 19 in c8cf2e1

import _initParquetWasm, { readParquet } from "parquet-wasm/esm/arrow2";
import * as arrow from "apache-arrow";
// NOTE: this version must be synced exactly with the parquet-wasm version in
// use.
const PARQUET_WASM_VERSION = "0.5.0";
const PARQUET_WASM_CDN_URL = `https://cdn.jsdelivr.net/npm/parquet-wasm@${PARQUET_WASM_VERSION}/esm/arrow2_bg.wasm`;
let WASM_READY: boolean = false;
export async function initParquetWasm() {
if (WASM_READY) {
return;
}
await _initParquetWasm(PARQUET_WASM_CDN_URL);
WASM_READY = true;
}

I ask because I'm trying to test out lonboard on a service that prevents me from making front-end requests to CDNs (I would have to request the URL be added to an allowlist, but I want to see if I can avoid this by bundling the code with esbuild but I was running into errors when I tried to use the bundled version - kylebarron/parquet-wasm#488).
@kylebarron
Copy link
Member

I agree, this change would be nice. But it's unclear to me how to get a reference to the local URL within Jupyter to the current bundle. I asked in the anywidget discord

@cornhundred
Copy link
Author

Thanks @kylebarron, I was able to find a workaround that might be relevant here kylebarron/parquet-wasm#488 (comment)

@kylebarron
Copy link
Member

kylebarron commented Apr 10, 2024 •
edited
Loading

Trevor responded on discord that serializing the wasm on the model is probably the best approach

@kylebarron kylebarron linked a pull request Apr 10, 2024 that will close this issue
Open
@prusswan
Copy link

Until #461 gets merged, the easiest way (imho) is to let user specify a different cdn url (e.g. http://localhost:8888/files/). For now, we need to resort to modifying the js bundle.

@kylebarron
Copy link
Member

What's your use case where you can't use the default CDN?

@prusswan
Copy link

prusswan commented Oct 11, 2024 via email

@kylebarron
Copy link
Member

But offline you won't be able to see the base map; does that not matter for you?

@prusswan
Copy link

But offline you won't be able to see the base map; does that not matter for you?

Usually I have a copy of the tiles for performance/practical reasons - it is faster than getting over the internet, while in other cases I am generating them anyway. If BitmapTileLayer (which already works with locally hosted Jupyter file urls) can be expanded to support/include pmtiles it would be perfect (sounds like it is in the works with #494)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Send WebAssembly binary over Jupyter WebSocket developmentseed/lonboard
3 participants
@prusswan @cornhundred @kylebarron