Skip to content

Commit

Permalink
feat(risingwave): add streaming DDLs (#8239)
Browse files Browse the repository at this point in the history
Risingwave plays a role as a computing engine and storage system in
Streaming ecosystems.
Usually, a streaming workload will include two/three systems like this:
`upstream data source --> Risingwave`
Or,
`upstream data source --> Risingwave --> downstream data sink`

This PR adds streaming support, mainly new relations related to
streaming, for the Risingwave backend.

To be specific, Four types of new relations are introduced:
- `Source`, data sources that encompass a connector connected to an
upstream data system like Kafka. A source works like an extraction or a
placeholder for an upstream system, and although it has columns, it does
not store any data itself.
- `Materialized View`, which is Risingwave's core concept for streaming.
One can create an MV with a query on existing tables or sources. The
data in MV is automatically updated in a real-time way. Users can access
the data with a `select` statement just like accessing a table.
- `Table` with connector, works like a combination of a source and a
normal table. The difference between `Table` with connector and `Source`
is that, once the table is created, it will automatically start to
consume data from upstream systems and update it into Risingwave.
- `Sink`, unlink a `Materialized View` which stores the result of a
query in RW, users can choose to sink the result to a downstream system,
e.g. Redis, and then read the data in the downstream system. Unlike an
MV, User cannot access the data directly from a sink.

Some minor changes:
1. Bump the image version to a new nightly build, as the new image
solves some issues found in the [previous
PR](#7954).
2. Mark one test, `test_semi_join`, as skipped for risingwave backend.
The test sometimes stuck and it seems to be Risingwave's fault. I'll
continue to investigate.
  • Loading branch information
KeXiangWang authored Apr 7, 2024
1 parent ba931da commit 356e459
Show file tree
Hide file tree
Showing 6 changed files with 456 additions and 64 deletions.
2 changes: 1 addition & 1 deletion compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -542,7 +542,7 @@ services:
- impala

risingwave:
image: ghcr.io/risingwavelabs/risingwave:nightly-20240122
image: ghcr.io/risingwavelabs/risingwave:nightly-20240204
command: "standalone --meta-opts=\" \
--advertise-addr 0.0.0.0:5690 \
--backend mem \
Expand Down
Loading

0 comments on commit 356e459

Please sign in to comment.