Although Wasm execution is a main feature of Substrate, you may want to disable it in some use cases. In this article, I will show you how to build a native-only Substrate chain without wasm.
Warning: Think twice in your design!
This is not a recommended way to build a Substrate chain.
The Parity team is removing native execution: The road to the native runtime free world #62. We are in the opposite direction.
If you want to use some native libraries which do not support no-std inside runtime pallet, you should consider using:
- offchain workers
- runtime_interface
Both of them are in the outer node, not in runtime, so they can use std libraries and not limited by wasm.
Environment Setup
Environment: substrate-node-template | tag: polkadot-v0.9.40 | commit: 700c3a1
Suppose you want to import an std Rust library called rust-bert into your runtime pallet. rust-bert is a machine learning library including large language models (LLMs) such as GPT2.
First, download substrate-node-template
git clone https://github.com/substrate-developer-hub/substrate-node-template
cd substrate-node-template
git checkout polkadot-v0.9.40Build
Add rust-bert as a dependency in pallets/template/Cargo.toml.
You also need to specify getrandom as a dependency. Otherwise it will throw an error error: the wasm32-unknown-unknown target is not supported by default, you may need to enable the "js" feature. For more information see: https://docs.rs/getrandom/#webassembly-support
In runtime pallets, all your dependencies must:
- Support
no-std. stdis not enabled by default. (that’s whatdefault-features = falseaccomplishes)
Otherwise, you will get an error error[E0152]: found duplicate lang item panic_impl when building. The reason is that std is leaking into the runtime code.
You can check this stackoverflow question for more details.
In pallets/template/Cargo.toml:
[dependencies]
rust-bert = { version = "0.21.0", default-features = false, features = ["remote", "download-libtorch"] }
getrandom = { version = "0.2", default-features = false, features = ["js"] }However, rust-bert do no support no-std. Even if you add default-feature = false in Cargo.toml, it will still throw an error error[E0152]: found duplicate lang item panic_impl when running cargo build.
To fix this error, you should skip building wasm code by adding env SKIP_WASM_BUILD=1.
SKIP_WASM_BUILD=1 cargo buildRun
By now, You should successfully build a native-only Substrate target without wasm.
But running the target binary is not easy.
--execution nativespecify the execution strategy to native-first
./target/debug/node-template --dev --execution native
Error: Input("Development wasm not available")Search Development wasm not available in the codebase, you will find that it is thrown by node/src/chain_spec.rs.
pub fn development_config() -> Result<ChainSpec, String> {
let wasm_binary = WASM_BINARY.ok_or_else(|| "Development wasm not available".to_string())?;
// ...
}Since we don’t have wasm, we should remove this check.
let wasm_binary = WASM_BINARY.unwrap_or(&[] as &[u8]);Rebuild and run again, new error occurs:
./target/debug/node-template --dev --execution native
2023-08-31 18:10:07 Substrate Node
2023-08-31 18:10:07 ✌️ version 4.0.0-dev-700c3a186e5
2023-08-31 18:10:07 ❤️ by Substrate DevHub <https://github.com/substrate-developer-hub>, 2017-2023
2023-08-31 18:10:07 📋 Chain specification: Development
2023-08-31 18:10:07 🏷 Node name: evanescent-agreement-4299
2023-08-31 18:10:07 👤 Role: AUTHORITY
2023-08-31 18:10:07 💾 Database: RocksDb at /tmp/substrate8yJbyt/chains/dev/db/full
2023-08-31 18:10:07 ⛓ Native runtime: node-template-100 (node-template-1.tx1.au1)
Error: Service(Client(VersionInvalid("cannot deserialize module: HeapOther(\"I/O Error: UnexpectedEof\")")))
2023-08-31 18:10:08 Cannot create a runtime error=Other("cannot deserialize module: HeapOther(\"I/O Error: UnexpectedEof\")")This error is much harder to debug. I spend a lot of time to find the root cause.
Debug Process
Search online only finds one similar issue: https://github.com/paritytech/substrate/issues/7675. The core developer @bkchr suggests to create a dummy wasm binary to bypass the check.
I cannot find relevent information about constructing a dummy wasm binary. So I decide to debug the code step by step. Finally, I find the root cause is in one system dependency native_executor.rs.
There are 2 executors implementations: WasmExecutor and NativeElseWasmExecutor. When --execution native is specified, NativeElseWasmExecutor will be used.
NativeElseWasmExecutor wrap WasmExecutor as its field.
/// A generic `CodeExecutor` implementation that uses a delegate to determine wasm code equivalence
/// and dispatch to native code when possible, falling back on `WasmExecutor` when not.
pub struct NativeElseWasmExecutor<D: NativeExecutionDispatch> {
/// Native runtime version info.
native_version: NativeVersion,
/// Fallback wasm executor.
wasm:
WasmExecutor<ExtendedHostFunctions<sp_io::SubstrateHostFunctions, D::ExtendHostFunctions>>,
}During substrate node running, even if NativeElseWasmExecutor is used, it will still try to load wasm binary in 2 methods: runtime_version and call.
Let’s look at runtime_version at first:
impl<D: NativeExecutionDispatch> RuntimeVersionOf for NativeElseWasmExecutor<D> {
fn runtime_version(
&self,
ext: &mut dyn Externalities,
runtime_code: &RuntimeCode,
) -> Result<RuntimeVersion> {
Ok(self.native_version.runtime_version.clone()) // <--- Edit: We should return native version
self.wasm.runtime_version(ext, runtime_code) // <--- Original: it will try to load wasm binary
}
}Then, let’s look at call.
It will first check if the native version is compatible with the on-chain version. If it is compatible, it will call native executor. Otherwise, it will call wasm executor.
However, we don’t have valid wasm binary, so we should always call native executor. I just use the code inside this branch if use_native && can_call_with {}.
impl<D: NativeExecutionDispatch + 'static> CodeExecutor for NativeElseWasmExecutor<D> {
type Error = Error;
fn call(
&self,
ext: &mut dyn Externalities,
runtime_code: &RuntimeCode,
method: &str,
data: &[u8],
use_native: bool,
context: CallContext,
) -> (Result<Vec<u8>>, bool) {
// Edit
// Do not check wasm since it is dummy, use native execution directly
let used_native = true;
let mut ext = AssertUnwindSafe(ext);
let result = match with_externalities_safe(&mut **ext, move || D::dispatch(method, data)) {
Ok(Some(value)) => Ok(value),
Ok(None) => Err(Error::MethodNotFound(method.to_owned())),
Err(err) => Err(err),
};
(result, used_native)
// Original
tracing::trace!(
target: "executor",
function = %method,
"Executing function",
);
let on_chain_heap_alloc_strategy = runtime_code
.heap_pages
.map(|h| HeapAllocStrategy::Static { extra_pages: h as _ })
.unwrap_or_else(|| self.wasm.default_onchain_heap_alloc_strategy);
let heap_alloc_strategy = match context {
CallContext::Offchain => self.wasm.default_offchain_heap_alloc_strategy,
CallContext::Onchain => on_chain_heap_alloc_strategy,
};
let mut used_native = false;
let result = self.wasm.with_instance(
runtime_code,
ext,
heap_alloc_strategy,
|_, mut instance, onchain_version, mut ext| {
let onchain_version =
onchain_version.ok_or_else(|| Error::ApiError("Unknown version".into()))?;
let can_call_with =
onchain_version.can_call_with(&self.native_version.runtime_version);
if use_native && can_call_with {
// call native executor
tracing::trace!(
target: "executor",
native = %self.native_version.runtime_version,
chain = %onchain_version,
"Request for native execution succeeded",
);
used_native = true;
Ok(with_externalities_safe(&mut **ext, move || D::dispatch(method, data))?
.ok_or_else(|| Error::MethodNotFound(method.to_owned())))
} else {
// call wasm executor
if !can_call_with {
tracing::trace!(
target: "executor",
native = %self.native_version.runtime_version,
chain = %onchain_version,
"Request for native execution failed",
);
}
with_externalities_safe(&mut **ext, move || instance.call_export(method, data))
}
},
);
(result, used_native)
}Finish!
You can use my substrate forked repo, which has already fixed the issue.
You should replace dependencies url in 3 Cargo.toml files:
git = "https://github.com/paritytech/substrate.git" -> git = "https://github.com/doutv/substrate.git"
Build and run again, it should work now!
SKIP_WASM_BUILD=1 cargo build
./target/debug/node-template --dev --execution nativeWe successfully build a native-only Substrate chain without wasm!
Final code: substrate-node-template forked repo in native-only-polkadot-v0.9.40 branch, https://github.com/doutv/substrate-node-template/tree/native-only-polkadot-v0.9.40
Further Improvement
Modify NativeElseWasmExecutor is not the best solution, you may add an new executor implementation NativeOnlyExecutor.
