Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unnecessary memory use in some tests causes false failures on constrained hardware #3328

Open
phoddie opened this issue Nov 24, 2021 · 5 comments
Open

Unnecessary memory use in some tests causes false failures on constrained hardware #3328

phoddie opened this issue Nov 24, 2021 · 5 comments
Labels
valid and awaiting action Issue is valid, but no action taken yet

Comments

@phoddie
Copy link
Contributor

phoddie commented Nov 24, 2021

The XS JavaScript engine is designed to run on resource constrained embedded hardware. It runs EcmaScript 2021 on devices with as little as 16 KB of RAM.

We have typically run Test262 against XS running on a computer where there is no practical limit on the memory available to the tests. We have slowly been working on running Test2652 on embedded devices as well. Our current focus is the ESP32 where the default VM size we test with is 64 KB. Our hope is to eventually run Test262 on devices with less memory, so this is a first step.

As expected, some tests fail because of memory exhaustion. That's probably inevitable for some tests. But, there are tests which would pass if they required less memory. A good example is this test Array.prototoype.copyWithin:

test262/test/built-ins/TypedArray/prototype/copyWithin/coerced-values-end-detached-prototype.js

Lines 34 to 40 in 9f2814f

array.length = 10000; // big arrays are more likely to cause a crash if they are accessed after they are freed
array.fill(7, 0);
ta = new TA(array);
assert.throws(TypeError, function(){
ta.copyWithin(0, 100, {valueOf : detachAndReturnIndex});
}, "should throw TypeError as array is detached");
});

Here the 10,000 element array causes an allocation failure of the TypedArray on ESP32 which prevents the test from completing. Reducing this to 1000 allows the test to pass. The larger array size does not appear to contribute to validating conformance with the standard. It would be beneficial for developers using XS to know that it implements the correct behavior, so we'd like tests like this to pass.

There are several possible ways to resolve this -- changing the test for all execution environments, adding some kind of configuration test, etc. Before diving into that, I wanted to see if test262 is open to addressing issues like these that impact the use of test262 to validate JavaScript on constrained device targets.
@jugglinmike
Copy link
Contributor

Thanks for the report!

For tail-call optimization, Test262 maintains a special value called $MAX_ITERATIONS which authors can use to set an abstract upper boundary on the number of stack frames necessary to prove TCO is taking place.

Test262 could take a similar approach here by offering a value like $LARGE_ARRAY_LENGTH. That would likewise empower authors to write tests without making assumptions about runtime capabilities. It would also reduce the number of arbitrary (or "magic") constants in test test material, and replace them with identifiers that more clearly reflect the intent.

But your point about conformance testing is well-taken. Unlike the TCO tests, the pattern you've identified isn't related to any specific normative text. The circumstances which motivate the use of a "large" array are at least partly implementation-specific. That would make the decision to apply $LARGE_ARRAY_LENGTH a little ambiguous. For that reason, I'd also support simply modifying tests like coerced-values-end-detached-prototype.js to use minimal array sizes.

@rwaldron
Copy link
Contributor

rwaldron commented Dec 4, 2021

@jugglinmike I like this solution.

@jugglinmike
Copy link
Contributor

@rwaldron Thanks! Do you mean introducing $LARGE_ARRAY_LENGTH or removing code intended to place stress on resource limitations?

@rwaldron
Copy link
Contributor

rwaldron commented Dec 6, 2021 •
edited
Loading

Oh, I thought it was two parts of a whole, but I see now that I misinterpreted that. I think we should introduce $LARGE_ARRAY_LENGTH for cases where length limitations are actually being tested (I'm thinking of Array tests that exercise ToIndex(), maybe?); and we should adjust any existing tests where size is not actually relevant to use the smallest size possible to effectively exercise the semantics under test.

@rwaldron rwaldron added the valid and awaiting action Issue is valid, but no action taken yet label Dec 6, 2021
@phoddie
Copy link
Contributor Author

phoddie commented Dec 6, 2021

@jugglinmike - thank you for your detailed response like the idea of parameterizing the array length so the relevant tests can easily be tuned as needed. I'll work on generating a list of tests where would help with XS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
valid and awaiting action Issue is valid, but no action taken yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rwaldron @jugglinmike @phoddie