Open
Description
There are currently a few problems with using test262 efficiently on chakracore:
- ch does not support monotonicNow (see issue Test262 Agent feature request: $262.agent.monotonicNow() #5271)
- I'm unsure if the createRealm method works properly - some tests relating to it fail - though it could be that that's exposing underlying CC issues.
- Some spec violations may happen for different reasons in jitted code vs interpreted code, currently running test262 99% of the time it's all going to be in the interpreter.
- test262Harness seems the easiest way to run the thing https://github.com/bterlson/test262-harness with a couple of provisos:
i. I've had to hack in support for running code as a module (it doesn't support this with CC - because it uses a flag that's only in master)
ii. controlling what tests you run is awkward to do manually - almost feels like it could do with a separate script to manage it, I have an alternative js start file for it offline which allows you to specify the inputs as a JS object instead of command line args; not sure if there's a better way of doing this.
iii. managing the output is also awkward, I've got an alternate reporter file that compares results before and after a change to CC by saving a baseline but it all feels a bit hacky
iv. I note that to solve issue 3 above may take some more fundamental edits/may not be feasible in the framework of test262Harness
Solving points 1 and 2 involve working on CC, for the other points I think we could probably do with a CC specific test262 runner script of some kind that can be stored in the tools directory - perhaps written in python (per #6411), something that can:
- fetch the latest test262
- Optionally Serialise the harness code within test262 as bytecode for fast injection
- prepare all the tests
- run them - including capturing and using dynamic profile info for a re-run so that they can get run with the fulljit
- Store the output AND enable comparison between different versions of the output.