std: Add a new wasm32-unknown-unknown target #45905

alexcrichton · 2017-11-10T09:01:06Z

This commit adds a new target to the compiler: wasm32-unknown-unknown. This target is a reimagining of what it looks like to generate WebAssembly code from Rust. Instead of using Emscripten which can bring with it a weighty runtime this instead is a target which uses only the LLVM backend for WebAssembly and a "custom linker" for now which will hopefully one day be direct calls to lld.

Notable features of this target include:

There is zero runtime footprint. The target assumes nothing exists other than the wasm32 instruction set.
There is zero toolchain footprint beyond adding the target. No custom linker is needed, rustc contains everything.
Very small wasm modules can be generated directly from Rust code using this target.
Most of the standard library is stubbed out to return an error, but anything related to allocation works (aka HashMap, Vec, etc).
Naturally, any #[no_std] crate should be 100% compatible with this new target.

This target is currently somewhat janky due to how linking works. The "linking" is currently unconditional whole program LTO (aka LLVM is being used as a linker). Naturally that means compiling programs is pretty slow! Eventually though this target should have a linker.

This target is also intended to be quite experimental. I'm hoping that this can act as a catalyst for further experimentation in Rust with WebAssembly. Breaking changes are very likely to land to this target, so it's not recommended to rely on it in any critical capacity yet. We'll let you know when it's "production ready".

Building yourself

First you'll need to configure the build of LLVM and enable this target

$ ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly

Next you'll want to remove any previously compiled LLVM as it needs to be rebuilt with WebAssembly support. You can do that with:

$ rm -rf build

And then you're good to go! A ./x.py build should give you a rustc with the appropriate libstd target.

Test support

Currently testing-wise this target is looking pretty good but isn't complete. I've got almost the entire run-pass test suite working with this target (lots of tests ignored, but many passing as well). The core test suite is still getting LLVM bugs fixed to get that working and will take some time. Relatively simple programs all seem to work though!

In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master. The LLVM 4 WebAssembly backend AFAIK isn't broken per se but is likely missing bug fixes available on LLVM 5. I'm hoping though that we can decouple the LLVM 5 upgrade and adding this wasm target!

But the modules generated are huge!

It's worth nothing that you may not immediately see the "smallest possible wasm module" for the input you feed to rustc. For various reasons it's very difficult to get rid of the final "bloat" in vanilla rustc (again, a real linker should fix all this). For now what you'll have to do is:

cargo install --git https://github.com/alexcrichton/wasm-gc
wasm-gc foo.wasm bar.wasm

And then bar.wasm should be the smallest we can get it!

In any case for now I'd love feedback on this, particularly on the various integration points if you've got better ideas of how to approach them!

alexcrichton · 2017-11-10T09:02:13Z

I'd also like to doubly and triply emphasize how this target is a work in progress. The precise semantics of how it works and how everything gets hooked up will likely change over time, but I feel that it's also beneficial if we can get more testing of this backend and more testing of the integration, and hopefully start seeing some neat projects leveraging these benefits!

tlively · 2017-11-10T14:13:26Z

@alexcrichton I just want to say how exciting it is to see so much progress toward making wasm work well with Rust :) Thanks for doing all of this!

shepmaster · 2017-11-10T14:50:13Z

Let's pick a reviewer...

r? @eddyb

shepmaster · 2017-11-10T14:51:37Z

@alexcrichton you've some tidy failures:

[00:03:55] tidy error: /checkout/src/etc/wasm32-shim.js: incorrect license
[00:03:55] Expected a gate test for the feature 'wasm_import_memory'.
[00:03:55] Hint: create a file named 'feature-gate-wasm_import_memory.rs' in the compile-fail
[00:03:55]       test suite, with its failures due to missing usage of
[00:03:55]       #![feature(wasm_import_memory)].
[00:03:55] Hint: If you already have such a test and don't want to rename it,
[00:03:55]       you can also add a // gate-test-wasm_import_memory line to the test file.
[00:03:55] tidy error: Found 1 features without a gate test.
[00:03:56] some tidy checks failed

eddyb · 2017-11-10T15:09:56Z

LGTM but this is a pretty large change. cc @rust-lang/compiler @rust-lang/libs

nagisa · 2017-11-10T15:10:21Z

src/test/run-pass/allocator/custom.rs

@@ -39,8  38,7 @@ unsafe impl<'a> Alloc for &'a A {
 static GLOBAL: A = A;

 fn main() {
-    env::set_var("FOO", "bar");
-    drop(env::var("FOO"));
+    println!("hello!");


Stray print?

Nah this is just intended to force an allocation somewhere that LLVM can't optimize away

NikVolf · 2017-11-10T15:17:11Z

src/libpanic_unwind/wasm32.rs

+//!
+//! Right now we don't support this, so this is just stubs
+
+#![allow(private_no_mangle_fns)]


is it really needed here?

SimonSapin · 2017-11-10T15:18:56Z

anything related to allocation works (aka HashMap, Vec, etc).

Out of curiosity, how does this work? Is there an allocator runtime?

pepyakin · 2017-11-10T15:22:04Z

@SimonSapin I believe yes.

jasondavies · 2017-11-10T16:08:53Z

This is great!

In general I've only tested this with a local fork that makes use of LLVM 5 rather than our current LLVM 4 on master.

If I simply clone this branch, run ./configure --target=wasm32-unknown-unknown --set llvm.experimental-targets=WebAssembly and ./x.py build, after much waiting I get an error:

Intrinsic has incorrect return type!
i32 (i32)* @llvm.wasm.grow.memory.i32
LLVM ERROR: Broken function found, compilation aborted!
error: Could not compile `dlmalloc`.

~~Does this mean I should be testing with LLVM 5, and if so, can you provide some simple instructions on how to do so?~~

Edit: I managed to build successfully using LLVM 5, using the --llvm-root option when calling ./configure.

konstin · 2017-11-10T21:12:49Z

.gitmodules

+	url = https://github.com/alexcrichton/dlmalloc-rs
+[submodule "src/binaryen"]
+	path = src/binaryen
+	url = https://github.com/WebAssembly/binaryen


There's a small inconsistency here: All the other crates use https://github.com/owner/repo.git instead of https://github.com/owner/repo

alexcrichton · 2017-11-10T22:25:48Z

@SimonSapin indeed there is! Wasm actually has memory allocation instructions which the allocator runtime leverages to implement a runtime allocator.

alexcrichton · 2017-11-10T22:26:39Z

@jasondavies oops sorry! Was pinned to the wrong commit of dlmalloc-rs, I believe it should work now.

ElvishJerricco · 2017-11-11T00:14:37Z

Does https://github.com/WebAssembly/lld/tree/wasm have to be merged upstream before you can just use the WebAssembly port of LLD?

shepmaster · 2017-11-11T00:32:16Z

New tidy errors:

Testing libstd stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:58:15]    Compiling libc v0.2.32
[00:58:15]    Compiling core v0.0.0 (file:///checkout/src/libcore)
[00:58:15]    Compiling std_unicode v0.0.0 (file:///checkout/src/libstd_unicode)
[00:58:15]    Compiling dlmalloc v0.0.0 (file:///checkout/src/rustc/dlmalloc_shim)
[00:58:15] error[E0464]: multiple matching crates for `libc`
[00:58:15]  --> /checkout/src/rustc/dlmalloc_shim/../../dlmalloc/src/linux.rs:1:1
[00:58:15]   |
[00:58:15] 1 | extern crate libc;
[00:58:15]   | ^^^^^^^^^^^^^^^^^^
[00:58:15]   |
[00:58:15]   = note: candidates:
[00:58:15]   = note: path: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-658d35794c10b003.rlib
[00:58:15]   = note: crate name: libc
[00:58:15]   = note: path: /checkout/obj/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-f32a17a3111b01aa.rlib
[00:58:15]   = note: crate name: libc
[00:58:15] 
[00:58:15] error[E0463]: can't find crate for `libc`
[00:58:15]  --> /checkout/src/rustc/dlmalloc_shim/../../dlmalloc/src/linux.rs:1:1
[00:58:15]   |
[00:58:15] 1 | extern crate libc;
[00:58:15]   | ^^^^^^^^^^^^^^^^^^ can't find crate
[00:58:15] 
[00:58:15] error: aborting due to 2 previous errors
[00:58:15] 
[00:58:15] error: Could not compile `dlmalloc`.

alexcrichton · 2017-11-11T11:49:34Z

@ElvishJerricco "just use the WebAssembly port of LLD" is actually quite involved, so I wanted to get something working (aka this PR) before starting on a new venture to discover even more integration and/or LLD bugs. This is the final state, though, to use LLD to link WebAsembly (AFAIK)

killercup · 2017-11-11T21:35:08Z

FYI: If someone wants to play with this branch and WASM in Node, I've pushed some of my experiments to this repo: https://github.com/killercup/wasm-experiments

bors · 2017-11-12T09:50:07Z

☔ The latest upstream changes (presumably #45870) made this pull request unmergeable. Please resolve the merge conflicts.

pepyakin · 2017-11-20T18:41:44Z

Yeah, Wasm (as x86) is just an ISA. fs and stuff like that is just a different level of abstraction.

For me wasm32-unknown-unknown seems more like a bare metal environment (like Arm MCU). “apps” for mcus doesnt require (and usually doesnt have a notion of) fs and networking. So is wasm

Also, as I mentioned before, you can use Rust with wasm32-unknown-emscripten triple with LLVM wasm backend (you just have to build it yourself). It already provides all these (and many other) abstractions!

RReverser · 2017-11-20T19:09:54Z

Please, again, I'm not trying to diminish the importance of this addition, no need to try and prove to me how awesome it is 😄 I'm just saying that applicability will be pretty limited until most of standard library gets bindings to JS side.

badboy · 2017-11-20T19:31:58Z

Let's work together and provide the missing pieces then (either as JavaScript shims or Rust-side implementations of functionality). I don't know how easy it will be to extract this stuff from Emscripten.

Anyway, this is not the place to discuss it.

RReverser · 2017-11-20T20:46:43Z

Anyway, this is not the place to discuss it.

It seemed like the best place (as it's related to the context of PR), but if you have other suggestions, sure.

pepyakin · 2017-11-20T21:00:27Z

My point is that many things from the standart library shouldn't get any bindings to host environment. For example, what means to spawn a process in wasm? What character should be used as a path seperator? How std::net should be implemented?
For me, these questions just doesn't make sense.
Also, I want to emphasize, that Web/JS is a important environment but it is not the only possible one!

Please, again, I'm not trying to diminish the importance of this addition, no need to try and prove to me how awesome it is

I didn't even think of that!
(btw sorry for my english)

RReverser · 2017-11-20T21:37:24Z

@pepyakin At least for Node.js side spawning process, net etc. are pretty easily implemented. Character for path can be taken from host system and so on. I think it makes sense to experiment with reusing Emscripten libraries for this as @badboy suggested.

RReverser · 2017-11-20T21:38:19Z

Also, I want to emphasize, that Web/JS is a important environment but it is not the only possible one!

Well other environments might provide own implementations of these system bindings if they need to.

est31 · 2017-11-21T04:50:38Z

Anyway, this is not the place to discuss it.

A better place would be #44006 . It is a bit underused sadly :/

lilianmoraru · 2017-11-21T07:01:35Z

@alexcrichton Now, the linker has some support for wasm: https://reviews.llvm.org/rL318539

pepyakin · 2017-11-21T07:14:08Z

Thank you @est31 ! I've moved discussion there

vmx · 2017-11-25T20:36:57Z

@alexcrichton You mention LLVM 5. Where is your branch and how do I use it? I'm playing with the wasm target but get a segfault during compilation. Before I dig deeper I'd like to try a more recent LLVM as it might be fixed already.

Dessix · 2017-11-26T00:13:59Z

@RReverser the option of not having the extra libraries and runtimes bolted on enables use of this for embedded systems where including additional support code is unnecessary or even undesirable. Ensuring that remains an option would be an important consideration for at least these scenarios.

RReverser · 2017-11-26T08:11:10Z

@Dessix One absolutely doesn't contradict another, you can always just not include runtime and not use anything that requires it. But anyway, this discussion was moved to another issue, please see above.

alexcrichton · 2017-11-27T00:56:09Z

I've now also opened a PR to start running tests on CI over at #46291

jethrogb · 2017-11-28T21:48:22Z

Since this is basically the same as #37133, I wouldn't have expected this to get merged in this state in light of the comments there and in https://internals.rust-lang.org/t/refactoring-std-for-ultimate-portability/4301. Will you reconsider #37133 for merging if I rebase that?

aturon · 2017-11-29T01:13:59Z

@jethrogb it's definitely worth revisiting, yes. Note that the wasm target is a somewhat special case, since it exists at the liballoc level and will eventually include some of the libstd layer as well (e.g. threads). It's also a specific (highly desired) platform, so the motivation was pretty different.

For #37133, is the motivation primarily to have a kind of "template" for tackling other, similar cases?

re: the PAL, AFAIK not much has happened there, and I think the whole story should be rebooted. In particular, we should take into account (and implement...) the portability lint, which in the long term may allow us to do away with the core/std layering altogether.

In any case, before rebasing, I'd suggest at least an internals thread laying out the motivation and attempting to reach some consensus there. You could ping me, @alexcrichton and @sfackler on the thread to ensure visibility.

aochagavia · 2017-12-03T14:25:07Z

src/libstd/sys/wasm/cmath.rs

+    pub fn cosh(n: f64) -> f64;
+    #[link_name = "Math_expm1"]
+    pub fn expm1(n: f64) -> f64;
+    pub fn fdim(a: f64, b: f64) -> f64;


Looks like you forgot to add a #[link_name = "Math_fdim"] attribute

Unfortunately Math.fdim isn't actually a function, and hopefully none of these will be necessary in the future regardless, it's mostly a bug that they're all here in the first place.

est31 mentioned this pull request Nov 10, 2017

Tracking issue for supporting asm.js and WebAssembly without Fastcomp #44006

Closed

5 tasks

alexcrichton force-pushed the add-wasm-target branch from df2c76e to 7552aa5 Compare November 10, 2017 13:33

rust-highfive assigned eddyb Nov 10, 2017

nagisa reviewed Nov 10, 2017

View reviewed changes

NikVolf reviewed Nov 10, 2017

View reviewed changes

konstin reviewed Nov 10, 2017

View reviewed changes

alexcrichton force-pushed the add-wasm-target branch from 7552aa5 to dde8f95 Compare November 10, 2017 22:24

alexcrichton force-pushed the add-wasm-target branch from dde8f95 to f483780 Compare November 10, 2017 22:26

alexcrichton force-pushed the add-wasm-target branch from f41e64f to f483780 Compare November 10, 2017 22:46

alexcrichton force-pushed the add-wasm-target branch from f483780 to 9bdb989 Compare November 11, 2017 11:48

alexcrichton force-pushed the add-wasm-target branch from 9bdb989 to 3a89d09 Compare November 12, 2017 10:09

OpenGG added a commit to OpenGG/docker-rust-wasm that referenced this pull request Nov 21, 2017

Enable wasm32-unknown-unknown rust-lang/rust#45905

0f56926

alexcrichton mentioned this pull request Nov 27, 2017

ci: Start running wasm32 tests on Travis #46291

Merged

bennyyip mentioned this pull request Dec 2, 2017

add rust-nightly-wasm32-unknown-unknown archlinuxcn/repo#586

Merged

aochagavia reviewed Dec 3, 2017

View reviewed changes

ColinEberhardt mentioned this pull request Dec 11, 2017

Is there long term value in wargo? lord/wargo#22

Closed

arielb1 mentioned this pull request Dec 21, 2017

Bounce out the layout refactor from beta #46925

Merged

bennyyip mentioned this pull request Dec 31, 2017

Rust supports wasm32-unknown-unknown target now dtysky/wasm-usage-guide#1

Closed

its-pointless mentioned this pull request Mar 9, 2018

Request: Rust extras its-pointless/gcc_termux#28

Closed

peerhenry mentioned this pull request Jun 3, 2018

Add support for target wasm32-unknown-unknown ianjsikes/rust-wasm-loader#8

Closed

therealbnut mentioned this pull request May 12, 2019

Unable to compile a simple Swift file (maybe not possible?) emscripten-core/emscripten#2427

Closed

danforbes mentioned this pull request May 31, 2020

error[E0424] building substrate template for first tutorial polkadot-developers/tutorials#45

Closed

da1suk8 mentioned this pull request Sep 16, 2022

Check the behavior when contract tries to open a file and fix it Finschia/cosmwasm#219

Closed

davidtwco mentioned this pull request Sep 20, 2023

target tier policy compliance #116004

Closed

std: Add a new wasm32-unknown-unknown target #45905

std: Add a new wasm32-unknown-unknown target #45905

Conversation

alexcrichton commented Nov 10, 2017

Building yourself

Test support

But the modules generated are huge!

alexcrichton commented Nov 10, 2017

tlively commented Nov 10, 2017

shepmaster commented Nov 10, 2017

shepmaster commented Nov 10, 2017

eddyb commented Nov 10, 2017

nagisa Nov 10, 2017

Choose a reason for hiding this comment

alexcrichton Nov 10, 2017

Choose a reason for hiding this comment

NikVolf Nov 10, 2017

Choose a reason for hiding this comment

SimonSapin commented Nov 10, 2017

pepyakin commented Nov 10, 2017

jasondavies commented Nov 10, 2017 • edited Loading

konstin Nov 10, 2017

Choose a reason for hiding this comment

alexcrichton commented Nov 10, 2017

alexcrichton commented Nov 10, 2017

ElvishJerricco commented Nov 11, 2017

shepmaster commented Nov 11, 2017

alexcrichton commented Nov 11, 2017

killercup commented Nov 11, 2017

bors commented Nov 12, 2017

pepyakin commented Nov 20, 2017 • edited Loading

RReverser commented Nov 20, 2017

badboy commented Nov 20, 2017

RReverser commented Nov 20, 2017

pepyakin commented Nov 20, 2017 • edited Loading

RReverser commented Nov 20, 2017

RReverser commented Nov 20, 2017

est31 commented Nov 21, 2017

lilianmoraru commented Nov 21, 2017

pepyakin commented Nov 21, 2017

vmx commented Nov 25, 2017

Dessix commented Nov 26, 2017

RReverser commented Nov 26, 2017

alexcrichton commented Nov 27, 2017

jethrogb commented Nov 28, 2017 • edited Loading

aturon commented Nov 29, 2017

aochagavia Dec 3, 2017

Choose a reason for hiding this comment

alexcrichton Dec 3, 2017

Choose a reason for hiding this comment

jasondavies commented Nov 10, 2017 •

edited

Loading

pepyakin commented Nov 20, 2017 •

edited

Loading

pepyakin commented Nov 20, 2017 •

edited

Loading

jethrogb commented Nov 28, 2017 •

edited

Loading