Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Third-party crates support: proc-macro2, quote, syn, serde and serde_derive #1007

Open
wants to merge 16 commits into
base: rust-next
Choose a base branch
from

Conversation

ojeda
Copy link
Member

@ojeda ojeda commented May 5, 2023

Like #910, but this time based on top of rust-next (instead of rust), and adds serde and serde_derive on top.

I include a draft commit at the end as a way to test that serde and serde_derive works in both the kernel crate and in a kernel module:

[    0.801425] rust_serde: Rust serde sample (init)
[    0.801634] rust_serde:             original = S { a: (), b: false, c: true, d: () }
[    0.802079] rust_serde:           serialized = [2, 0, 1, 0, 1, 1, 0, 3]
[    0.802506] rust_serde:         deserialized = S { a: (), b: false, c: true, d: () }
[    0.802718] rust_serde:   serialized (local) = [2, 0, 1, 42, 1, 43, 0, 3]
[    0.802895] rust_serde: deserialized (local) = S { a: (), b: false, c: true, d: () }
[    0.808954] rust_serde: Rust serde sample (exit)

Important note: it is not yet decided whether third-party crates will be upstreamed or not, or which ones, but this is useful for others to experiment and prototype solutions assuming these crates are available.

Cc @ariel-miculas who wanted to experiment with serde in the kernel: https://rust-for-linux.zulipchat.com/#narrow/stream/291565-Help/topic/Adding.20crates.20to.20out-of-tree.20modules/near/355792607

@@ -60,11 +61,73 @@ alloc-cfgs = \
--cfg no_sync \
--cfg no_thin

proc_macro2-skip_flags := \
--edition=2021 \
-Drust_2018_idioms \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe do --cap-lints allow for all third-party crates? This overrides any -D arguments. It is also what cargo passes for all non-local crates.


serde-flags := \
-Amissing_docs \
--cfg no_fp_fmt_parse
Copy link
Member

@bjorn3 bjorn3 May 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you pass --cfg 'feature="derive"' --extern serde_derive and make the serde target depend on serde_derive, you don't need --extern serde_derive elsewhere, but can instead access the derive macros using use serde::{Serialize, Deserialize};.

@nbdd0121
Copy link
Member

nbdd0121 commented May 6, 2023

I am getting error when rustc-dev component is installed:

error[E0464]: multiple candidates for `dylib` dependency `serde_derive` found
  --> rust/kernel/test_serde.rs:18:5
   |
18 | use serde_derive::{Deserialize, Serialize};
   |     ^^^^^^^^^^^^
   |
   = note: candidate #1: /home/gary/.rustup/toolchains/1.68.2-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libserde_derive-a574e0b043b5a546.so
   = note: candidate #2: /home/gary/Projects/linux/rust/libserde_derive.so

@bjorn3
Copy link
Member

bjorn3 commented May 6, 2023

That can probably be fixed by passing the path of libserde_derive.so to --extern serde_derive. Might be a good idea to do anyway. Cargo does this too.

@ariel-miculas
Copy link

I have the following error with rust-analyzer target:

$ make LLVM=1 rust-analyzer
Traceback (most recent call last):
  File "/home/amiculas/work/linux/./scripts/generate_rust_analyzer.py", line 141, in <module>
    main()
  File "/home/amiculas/work/linux/./scripts/generate_rust_analyzer.py", line 134, in main
    "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src),
  File "/home/amiculas/work/linux/./scripts/generate_rust_analyzer.py", line 107, in generate_crates
    if f"{name}.o" not in open(path.parent / "Makefile").read():
FileNotFoundError: [Errno 2] No such file or directory: 'samples/rust/local_data_format/Makefile'
make[1]: *** [rust/Makefile:504: rust-analyzer] Error 1
make: *** [Makefile:1872: rust-analyzer] Error 2

ojeda added 16 commits May 8, 2023 16:48
This is a subset of the Rust `proc-macro2` crate,
version 1.0.46, licensed under "Apache-2.0 OR MIT", from:

    https://github.com/dtolnay/proc-macro2/raw/1.0.46/src

The files are copied as-is, with no modifications whatsoever
(not even adding the SPDX identifiers).

For copyright details, please see:

    https://github.com/dtolnay/proc-macro2/blob/1.0.46/README.md#license
    https://github.com/dtolnay/proc-macro2/blob/1.0.46/LICENSE-APACHE
    https://github.com/dtolnay/proc-macro2/blob/1.0.46/LICENSE-MIT

The next two patches modify these files as needed for use within
the kernel. This patch split allows reviewers to double-check
the import and to clearly see the differences introduced.

The following script may be used to verify the contents:

    for path in $(cd rust/proc-macro2/ && find . -type f -name '*.rs'); do
        curl --silent --show-error --location \
            https://github.com/dtolnay/proc-macro2/raw/1.0.46/src/$path \
            | diff --unified rust/proc-macro2/$path - && echo $path: OK
    done

Signed-off-by: Miguel Ojeda <[email protected]>
The `proc-macro2` crate depends on the `unicode-ident` crate
to determine whether characters have the XID_Start or XID_Continue
properties according to Unicode Standard Annex torvalds#31.

However, we only need ASCII identifiers in the kernel, thus we can
simplify the check and remove completely that dependency.

Signed-off-by: Miguel Ojeda <[email protected]>
This is a subset of the Rust `quote` crate,
version 1.0.21, licensed under "Apache-2.0 OR MIT", from:

    https://github.com/dtolnay/quote/raw/1.0.21/src

The files are copied as-is, with no modifications whatsoever
(not even adding the SPDX identifiers).

For copyright details, please see:

    https://github.com/dtolnay/quote/blob/1.0.21/README.md#license
    https://github.com/dtolnay/quote/blob/1.0.21/LICENSE-APACHE
    https://github.com/dtolnay/quote/blob/1.0.21/LICENSE-MIT

The next patch modifies these files as needed for use within
the kernel. This patch split allows reviewers to double-check
the import and to clearly see the differences introduced.

The following script may be used to verify the contents:

    for path in $(cd rust/quote/ && find . -type f -name '*.rs'); do
        curl --silent --show-error --location \
            https://github.com/dtolnay/quote/raw/1.0.21/src/$path \
            | diff --unified rust/quote/$path - && echo $path: OK
    done

Signed-off-by: Miguel Ojeda <[email protected]>
This is a subset of the Rust `syn` crate,
version 1.0.104, licensed under "Apache-2.0 OR MIT", from:

    https://github.com/dtolnay/syn/raw/1.0.104/src

The files are copied as-is, with no modifications whatsoever
(not even adding the SPDX identifiers).

For copyright details, please see:

    https://github.com/dtolnay/syn/blob/1.0.104/README.md#license
    https://github.com/dtolnay/syn/blob/1.0.104/LICENSE-APACHE
    https://github.com/dtolnay/syn/blob/1.0.104/LICENSE-MIT

The next two patches modify these files as needed for use within
the kernel. This patch split allows reviewers to double-check
the import and to clearly see the differences introduced.

The following script may be used to verify the contents:

    for path in $(cd rust/syn/ && find . -type f -name '*.rs'); do
        curl --silent --show-error --location \
            https://github.com/dtolnay/syn/raw/1.0.104/src/$path \
            | diff --unified rust/syn/$path - && echo $path: OK
    done

Signed-off-by: Miguel Ojeda <[email protected]>
The `syn` crate depends on the `unicode-ident` crate
to determine whether characters have the XID_Start or XID_Continue
properties according to Unicode Standard Annex torvalds#31.

However, we only need ASCII identifiers in the kernel, thus we can
simplify the check and remove completely that dependency.

Signed-off-by: Miguel Ojeda <[email protected]>
This is a subset of the Rust `serde` crate,
version v1.0.156, licensed under "Apache-2.0 OR MIT", from:

    https://github.com/serde-rs/serde/tree/v1.0.156/serde/src

The files are copied as-is, with no modifications whatsoever
(not even adding the SPDX identifiers).

For copyright details, please see:

    https://github.com/serde-rs/serde/blob/v1.0.156/README.md#license
    https://github.com/serde-rs/serde/blob/v1.0.156/LICENSE-APACHE
    https://github.com/serde-rs/serde/blob/v1.0.156/LICENSE-MIT

The next three patches modify these files as needed for use within
the kernel. This patch split allows reviewers to double-check
the import and to clearly see the differences introduced.

The following script may be used to verify the contents:

    for path in $(cd rust/serde/ && find . -type f -name '*.rs'); do
        curl --silent --show-error --location \
            https://github.com/serde-rs/serde/raw/v1.0.156/serde/src/$path \
            | diff --unified rust/serde/$path - && echo $path: OK
    done

Signed-off-by: Miguel Ojeda <[email protected]>
We do not have formatting for floating point in the kernel,
thus simple compile out all that.

Probably we should name it differently, more similar to `integer128`.

Signed-off-by: Miguel Ojeda <[email protected]>
This is a subset of the Rust `serde_derive` crate,
version v1.0.156, licensed under "Apache-2.0 OR MIT", from:

    https://github.com/serde-rs/serde/tree/v1.0.156/serde_derive/src

The files are copied as-is, with no modifications whatsoever
(not even adding the SPDX identifiers).

For copyright details, please see:

    https://github.com/serde-rs/serde/blob/v1.0.156/README.md#license
    https://github.com/serde-rs/serde/blob/v1.0.156/LICENSE-APACHE
    https://github.com/serde-rs/serde/blob/v1.0.156/LICENSE-MIT

The next patch modifies these files as needed for use within
the kernel. This patch split allows reviewers to double-check
the import and to clearly see the differences introduced.

The following script may be used to verify the contents:

    for path in $(cd rust/serde_derive/ && find . -type f -name '*.rs'); do
        curl --silent --show-error --location \
            https://github.com/serde-rs/serde/raw/v1.0.156/serde_derive/src/$path \
            | diff --unified rust/serde_derive/$path - && echo $path: OK
    done

Signed-off-by: Miguel Ojeda <[email protected]>
…e_derive`

With all the new files in place from the new crates, this patch
adds support for them in the build system.

Note that, if we decide to add third-party crates support to upstream, we would
probably want to do it in a different way. But this is useful for playing around
with these crates, experimenting, etc.

Signed-off-by: Miguel Ojeda <[email protected]>
A trivial example based on `serde`'s `example-format' [1].

It contains both a in-`kernel` data format later used by
the kernel module, as well as a local data format in
the module.

The kernel module gives an output such as:

    [    0.801425] rust_serde: Rust serde sample (init)
    [    0.801634] rust_serde:             original = S { a: (), b: false, c: true, d: () }
    [    0.802079] rust_serde:           serialized = [2, 0, 1, 0, 1, 1, 0, 3]
    [    0.802506] rust_serde:         deserialized = S { a: (), b: false, c: true, d: () }
    [    0.802718] rust_serde:   serialized (local) = [2, 0, 1, 42, 1, 43, 0, 3]
    [    0.802895] rust_serde: deserialized (local) = S { a: (), b: false, c: true, d: () }
    [    0.808954] rust_serde: Rust serde sample (exit)

Note that this is just a quick draft/hack to check the previous
commits work. It is not intended to be merged at all.

Link: https://github.com/serde-rs/example-format [1]
Signed-off-by: Miguel Ojeda <[email protected]>
@ojeda
Copy link
Member Author

ojeda commented May 8, 2023

That should be fixed in commit 5c7548d, but rust-next does not have it yet here.

Let me move rust-next now that v6.4-rc1 is out and rebase this one on top.

@XiangruLiu0
Copy link

Could I please know what is preventing this pull request from being merged into the rust-next branch? Incorporating these third-party crates would significantly enhance our ability to develop complex drivers in Rust. 😊

@ojeda
Copy link
Member Author

ojeda commented Jun 29, 2024

We plan to have support for syn and other selected third-party crates soon in mainline, although in a different form than what is in this PR. What are you planning to use syn for? It could be useful to know. Thanks!

(The PRs here are not used for mainline development; we have some notes in https://rust-for-linux.com/contributing and other pages, in case you have not seen them).

@XiangruLiu0
Copy link

Will the lazy_static crate be supported? Static variables are extremely useful, especially in driver development, due to the prevalence of singleton variables.

@bjorn3
Copy link
Member

bjorn3 commented Jun 29, 2024

The lazy_static crate has been pretty much subsumed by the once_cell crate. Neither is directly usable in the kernel though as for no_std usages they use a spinlock without any yielding rather than the kernel's version of mutexes. If you want you could write your own version of once_cell which uses the kernel mutex type though. By the way lazy_static is not implemented as proc-macro anyway. It uses a decl macro.

@XiangruLiu0
Copy link

Ok I see, thanks 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

5 participants