Skip to content

[Strings] Add a string-builtins feature, and lift/lower automatically when enabled #7601

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open

[Strings] Add a string-builtins feature, and lift/lower automatically when enabled #7601

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
5db98f4
start
kripken May 15, 2025
36d89d5
work
kripken May 15, 2025
956d45a
docs
kripken May 15, 2025
b72d753
p
kripken May 15, 2025
3b21cf2
format
kripken May 15, 2025
adad0dc
work
kripken May 15, 2025
c636685
work
kripken May 15, 2025
7bf1cf7
finish
kripken May 15, 2025
8ad61b5
Merge remote-tracking branch 'origin/main' into sb
kripken May 15, 2025
48f05f7
test
kripken May 15, 2025
60637f9
test
kripken May 15, 2025
3195ac2
test
kripken May 15, 2025
4dd6747
test
kripken May 16, 2025
3c29e7b
use magic imports [ci skip]
kripken May 16, 2025
8ae39b8
Merge remote-tracking branch 'origin/main' into sb
kripken May 16, 2025
bbd6804
gate the pass behind GC
kripken May 16, 2025
2e82f9a
update test
kripken May 16, 2025
0feeeae
work
kripken May 16, 2025
02d1d23
update exact result in test, likely because of the new pass we run
kripken May 16, 2025
57ac595
so many tests to update
kripken May 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,15 @@ There are a few differences between Binaryen IR and the WebAssembly language:
rare cases (we avoid this overhead in the common case where the `br_if`
value is unused).

* Strings

* When the string builtins feature is enabled (`--enable-string-builtins`),
string operations are optimized. First, string imports are lifted into
stringref operations, before any default optimization passes. Those
stringref operations can then be optimized (e.g., a concat of constants
turns into a concatenated constant). When we are about to finish running
default optimizations, we lower stringref back into string builtins.

As a result, you might notice that round-trip conversions (wasm => Binaryen IR
=> wasm) change code a little in some corner cases.

Expand Down
5 changes: 4 additions & 1 deletion scripts/test/fuzzing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,13 @@
unfuzzable = [
# Float16 is still experimental.
'f16.wast',
# TODO: fuzzer and interpreter support for strings
# TODO: fuzzer and interpreter support for strings, including limitations
# like the fuzzer not handling (ref extern) imports (there is no way
# to create a replacement value)
'strings.wast',
'simplify-locals-strings.wast',
'string-lowering-instructions.wast',
'string-builtins.wast',
# TODO: fuzzer and interpreter support for extern conversions
'extern-conversions.wast',
# ignore DWARF because it is incompatible with multivalue atm
Expand Down
25 changes: 19 additions & 6 deletions src/pass.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,26 +324,39 @@ struct PassRunner {
// warning.
void addIfNoDWARFIssues(std::string passName);

// Adds the default set of optimization passes; this is
// what -O does.
void addDefaultOptimizationPasses();
// By default, we do not know if we are running first in the ordering of
// optimization passes, or last - we could be anywhere.
struct Ordering {
bool first;
bool last;
};
static constexpr Ordering UnknownOrdering = {false, false};

// Adds the default set of optimization passes; this is what -O does.
//
// The ordering indicates our position relative to other default
// optimizations, that is, if ordering.first then we are first.
void addDefaultOptimizationPasses(Ordering ordering = UnknownOrdering);

// Adds the default optimization passes that work on
// individual functions.
void addDefaultFunctionOptimizationPasses();
void
addDefaultFunctionOptimizationPasses(Ordering ordering = UnknownOrdering);

// Adds the default optimization passes that work on
// entire modules as a whole, and make sense to
// run before function passes.
void addDefaultGlobalOptimizationPrePasses();
void
addDefaultGlobalOptimizationPrePasses(Ordering ordering = UnknownOrdering);

// Adds the default optimization passes that work on
// entire modules as a whole, and make sense to
// run after function passes.
// This is run at the very end of the optimization
// process - you can assume no other opts will be run
// afterwards.
void addDefaultGlobalOptimizationPostPasses();
void
addDefaultGlobalOptimizationPostPasses(Ordering ordering = UnknownOrdering);

// Run the passes on the module
void run();
Expand Down
38 changes: 31 additions & 7 deletions src/passes/pass.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -605,13 +605,13 @@ void PassRunner::addIfNoDWARFIssues(std::string passName) {
}
}

void PassRunner::addDefaultOptimizationPasses() {
addDefaultGlobalOptimizationPrePasses();
addDefaultFunctionOptimizationPasses();
addDefaultGlobalOptimizationPostPasses();
void PassRunner::addDefaultOptimizationPasses(Ordering ordering) {
addDefaultGlobalOptimizationPrePasses(ordering);
addDefaultFunctionOptimizationPasses(ordering);
addDefaultGlobalOptimizationPostPasses(ordering);
}

void PassRunner::addDefaultFunctionOptimizationPasses() {
void PassRunner::addDefaultFunctionOptimizationPasses(Ordering ordering) {
// All the additions here are optional if DWARF must be preserved. That is,
// when DWARF is relevant we run fewer optimizations.
// FIXME: support DWARF in all of them.
Expand Down Expand Up @@ -723,7 +723,17 @@ void PassRunner::addDefaultFunctionOptimizationPasses() {
addIfNoDWARFIssues("vacuum"); // just to be safe
}

void PassRunner::addDefaultGlobalOptimizationPrePasses() {
void PassRunner::addDefaultGlobalOptimizationPrePasses(Ordering ordering) {
// If we are optimizing string builtins then we lift at the very start of the
// optimization pipeline, not just at the beginning here, but only when we are
// ordered before other bundles of passes.
//
// We check for GC for symmetry with the lowering pass, see comment in
// addDefaultGlobalOptimizationPostPasses() below.
if (wasm->features.hasStringBuiltins() && wasm->features.hasGC() &&
options.optimizeLevel >= 2 && ordering.first) {
addIfNoDWARFIssues("string-lifting");
}
// Removing duplicate functions is fast and saves work later.
addIfNoDWARFIssues("duplicate-function-elimination");
// Do a global cleanup before anything heavy, as it is fairly fast and can
Expand Down Expand Up @@ -772,7 +782,7 @@ void PassRunner::add(std::string passName, std::optional<std::string> passArg) {
doAdd(std::move(pass));
}

void PassRunner::addDefaultGlobalOptimizationPostPasses() {
void PassRunner::addDefaultGlobalOptimizationPostPasses(Ordering ordering) {
if (options.optimizeLevel >= 2 || options.shrinkLevel >= 1) {
addIfNoDWARFIssues("dae-optimizing");
}
Expand All @@ -794,6 +804,20 @@ void PassRunner::addDefaultGlobalOptimizationPostPasses() {
} else {
addIfNoDWARFIssues("simplify-globals");
}

// Lower away strings at the very very end. We do this before
// remove-unused-module-elements so we don't add unused imports, and also
// before reorder-globals, which will sort the new globals.
//
// Note we also test for GC here, as the pass adds imports that use GC arrays
// (and externref). Those imports may be unused, but they exist until
// remove-unused-module-elements cleans them up, which would cause an error in
// between.
if (wasm->features.hasStringBuiltins() && wasm->features.hasGC() &&
options.optimizeLevel >= 2 && ordering.last) {
addIfNoDWARFIssues("string-lowering-magic-imports");
}

addIfNoDWARFIssues("remove-unused-module-elements");
if (options.optimizeLevel >= 2 && wasm->features.hasStrings()) {
// Gather strings to globals right before reorder-globals, which will then
Expand Down
21 changes: 19 additions & 2 deletions src/tools/optimization-options.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,7 +398,19 @@ struct OptimizationOptions : public ToolOptions {
passRunner.clear();
};

for (auto& pass : passes) {
// Find the first and last default opt passes, so we can tell them they are
// first/last.
Index firstDefault = passes.size();
Index lastDefault = passes.size();
for (Index i = 0; i < passes.size(); i++) {
if (passes[i].name == DEFAULT_OPT_PASSES) {
firstDefault = std::min(firstDefault, i);
lastDefault = i;
}
}

for (Index i = 0; i < passes.size(); i++) {
auto& pass = passes[i];
if (pass.name == DEFAULT_OPT_PASSES) {
// This is something like -O3 or -Oz. We must run this now, in order to
// set the proper opt and shrink levels. To do that, first reset the
Expand All @@ -416,8 +428,13 @@ struct OptimizationOptions : public ToolOptions {
passRunner.options.optimizeLevel = *pass.optimizeLevel;
passRunner.options.shrinkLevel = *pass.shrinkLevel;

// Note the ordering of these default passes.
PassRunner::Ordering ordering;
ordering.first = (i == firstDefault);
ordering.last = (i == lastDefault);
Comment on lines +432 to +434
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively:

Suggested change
PassRunner::Ordering ordering;
ordering.first = (i == firstDefault);
ordering.last = (i == lastDefault);
PassRunner::Ordering ordering{i == firstDefault, i == lastDefault};


// Run our optimizations now with the custom levels.
passRunner.addDefaultOptimizationPasses();
passRunner.addDefaultOptimizationPasses(ordering);
flush();

// Restore the default optimize/shrinkLevels.
Expand Down
2 changes: 2 additions & 0 deletions src/tools/tool-options.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,8 @@ struct ToolOptions : public Options {
.addFeature(FeatureSet::FP16, "float 16 operations")
.addFeature(FeatureSet::CustomDescriptors,
"custom descriptors (RTTs) and exact references")
.addFeature(FeatureSet::StringBuiltins,
"string builtins (imported JS strings)")
.add("--enable-typed-function-references",
"",
"Deprecated compatibility flag",
Expand Down
1 change: 1 addition & 0 deletions src/wasm-binary.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -454,6 +454,7 @@ extern const char* FP16Feature;
extern const char* BulkMemoryOptFeature;
extern const char* CallIndirectOverlongFeature;
extern const char* CustomDescriptorsFeature;
extern const char* StringBuiltinsFeature;

enum Subsection {
NameModule = 0,
Expand Down
7 changes: 6 additions & 1 deletion src/wasm-features.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,11 +55,12 @@ struct FeatureSet {
// it does nothing. Binaryen always accepts LEB call-indirect encodings.
CallIndirectOverlong = 1 << 20,
CustomDescriptors = 1 << 21,
StringBuiltins = 1 << 22,
MVP = None,
// Keep in sync with llvm default features:
// https://github.com/llvm/llvm-project/blob/c7576cb89d6c95f03968076e902d3adfd1996577/clang/lib/Basic/Targets/WebAssembly.cpp#L150-L153
Default = SignExt | MutableGlobals,
All = (1 << 22) - 1,
All = (1 << 23) - 1,
};

static std::string toString(Feature f) {
Expand Down Expand Up @@ -108,6 +109,8 @@ struct FeatureSet {
return "call-indirect-overlong";
case CustomDescriptors:
return "custom-descriptors";
case StringBuiltins:
return "string-builtins";
case MVP:
case Default:
case All:
Expand Down Expand Up @@ -168,6 +171,7 @@ struct FeatureSet {
bool hasCustomDescriptors() const {
return (features & CustomDescriptors) != 0;
}
bool hasStringBuiltins() const { return (features & StringBuiltins) != 0; }
bool hasAll() const { return (features & All) != 0; }

void set(FeatureSet f, bool v = true) {
Expand All @@ -194,6 +198,7 @@ struct FeatureSet {
void setFP16(bool v = true) { set(FP16, v); }
void setBulkMemoryOpt(bool v = true) { set(BulkMemoryOpt, v); }
void setCustomDescriptors(bool v = true) { set(CustomDescriptors, v); }
void setStringBuiltins(bool v = true) { set(StringBuiltins, v); }
void setMVP() { features = MVP; }
void setAll() { features = All; }

Expand Down
2 changes: 2 additions & 0 deletions src/wasm/wasm-binary.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1395,6 +1395,8 @@ void WasmBinaryWriter::writeFeaturesSection() {
return BinaryConsts::CustomSections::CallIndirectOverlongFeature;
case FeatureSet::CustomDescriptors:
return BinaryConsts::CustomSections::CustomDescriptorsFeature;
case FeatureSet::StringBuiltins:
return BinaryConsts::CustomSections::StringBuiltinsFeature;
case FeatureSet::None:
case FeatureSet::Default:
case FeatureSet::All:
Expand Down
1 change: 1 addition & 0 deletions src/wasm/wasm.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ const char* FP16Feature = "fp16";
const char* BulkMemoryOptFeature = "bulk-memory-opt";
const char* CallIndirectOverlongFeature = "call-indirect-overlong";
const char* CustomDescriptorsFeature = "custom-descriptors";
const char* StringBuiltinsFeature = "string-builtins";

} // namespace BinaryConsts::CustomSections

Expand Down
2 changes: 1 addition & 1 deletion test/binaryen.js/kitchen-sink.js.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Features.RelaxedSIMD: 4096
Features.ExtendedConst: 8192
Features.Strings: 16384
Features.MultiMemory: 32768
Features.All: 4194303
Features.All: 8388607
InvalidId: 0
BlockId: 1
IfId: 2
Expand Down
2 changes: 1 addition & 1 deletion test/example/c-api-kitchen-sink.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ BinaryenFeatureMemory64: 2048
BinaryenFeatureRelaxedSIMD: 4096
BinaryenFeatureExtendedConst: 8192
BinaryenFeatureStrings: 16384
BinaryenFeatureAll: 4194303
BinaryenFeatureAll: 8388607
(f32.neg
(f32.const -33.61199951171875)
)
Expand Down
6 changes: 6 additions & 0 deletions test/lit/help/wasm-as.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,12 @@
;; CHECK-NEXT: --disable-custom-descriptors Disable custom descriptors (RTTs) and
;; CHECK-NEXT: exact references
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-string-builtins Enable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-string-builtins Disable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-typed-function-references Deprecated compatibility flag
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-typed-function-references Deprecated compatibility flag
Expand Down
6 changes: 6 additions & 0 deletions test/lit/help/wasm-ctor-eval.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,12 @@
;; CHECK-NEXT: --disable-custom-descriptors Disable custom descriptors (RTTs) and
;; CHECK-NEXT: exact references
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-string-builtins Enable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-string-builtins Disable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-typed-function-references Deprecated compatibility flag
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-typed-function-references Deprecated compatibility flag
Expand Down
6 changes: 6 additions & 0 deletions test/lit/help/wasm-dis.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,12 @@
;; CHECK-NEXT: --disable-custom-descriptors Disable custom descriptors (RTTs) and
;; CHECK-NEXT: exact references
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-string-builtins Enable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-string-builtins Disable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-typed-function-references Deprecated compatibility flag
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-typed-function-references Deprecated compatibility flag
Expand Down
6 changes: 6 additions & 0 deletions test/lit/help/wasm-emscripten-finalize.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,12 @@
;; CHECK-NEXT: --disable-custom-descriptors Disable custom descriptors (RTTs) and
;; CHECK-NEXT: exact references
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-string-builtins Enable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-string-builtins Disable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-typed-function-references Deprecated compatibility flag
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-typed-function-references Deprecated compatibility flag
Expand Down
6 changes: 6 additions & 0 deletions test/lit/help/wasm-merge.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,12 @@
;; CHECK-NEXT: --disable-custom-descriptors Disable custom descriptors (RTTs) and
;; CHECK-NEXT: exact references
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-string-builtins Enable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-string-builtins Disable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-typed-function-references Deprecated compatibility flag
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-typed-function-references Deprecated compatibility flag
Expand Down
6 changes: 6 additions & 0 deletions test/lit/help/wasm-metadce.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -787,6 +787,12 @@
;; CHECK-NEXT: --disable-custom-descriptors Disable custom descriptors
;; CHECK-NEXT: (RTTs) and exact references
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-string-builtins Enable string builtins (imported
;; CHECK-NEXT: JS strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-string-builtins Disable string builtins
;; CHECK-NEXT: (imported JS strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-typed-function-references Deprecated compatibility flag
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-typed-function-references Deprecated compatibility flag
Expand Down
6 changes: 6 additions & 0 deletions test/lit/help/wasm-opt.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -799,6 +799,12 @@
;; CHECK-NEXT: --disable-custom-descriptors Disable custom descriptors
;; CHECK-NEXT: (RTTs) and exact references
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-string-builtins Enable string builtins (imported
;; CHECK-NEXT: JS strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-string-builtins Disable string builtins
;; CHECK-NEXT: (imported JS strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-typed-function-references Deprecated compatibility flag
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-typed-function-references Deprecated compatibility flag
Expand Down
6 changes: 6 additions & 0 deletions test/lit/help/wasm-reduce.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,12 @@
;; CHECK-NEXT: --disable-custom-descriptors Disable custom descriptors (RTTs) and
;; CHECK-NEXT: exact references
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-string-builtins Enable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-string-builtins Disable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-typed-function-references Deprecated compatibility flag
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-typed-function-references Deprecated compatibility flag
Expand Down
6 changes: 6 additions & 0 deletions test/lit/help/wasm-split.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -266,6 +266,12 @@
;; CHECK-NEXT: --disable-custom-descriptors Disable custom descriptors (RTTs) and
;; CHECK-NEXT: exact references
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-string-builtins Enable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-string-builtins Disable string builtins (imported JS
;; CHECK-NEXT: strings)
;; CHECK-NEXT:
;; CHECK-NEXT: --enable-typed-function-references Deprecated compatibility flag
;; CHECK-NEXT:
;; CHECK-NEXT: --disable-typed-function-references Deprecated compatibility flag
Expand Down
Loading
Loading