Scripts (and scripting languages)

[ppenzin]

Looks like there are a few scripting languages used for various tasks:

$ find ChakraCore -name \*.bat -o -name \*.cmd -o -name \*.ps1 -o -name \*.py -o -name \*.pl -o -name \*.sh | grep -o "\(cmd\|ps1\|py\|pl\|sh\|bat\)" | sort | uniq -c
   1 bat
  26 cmd
   3 pl
  14 ps1
  13 py
  15 sh

One issue is that a number of those only run on Windows. Also there is probably no good reason to depend on Perl and Python, and given that there are only three Perl scripts it probably makes sense to drop Perl altogether. It also makes sense to pythonise the scripts described in #6410 (same scripts as in #3330).

[rhuanjl]

I agree on simplyfing/consolidating on python if we can for consistency and xplat benefits. I've never properly learnt any of these languages though, so not keen to take this task on myself.

[hez2010]

How about consolidating on PowerShell scripts? This may be easier and also cross-platform.

[zenparsing]

Personally, I would probably prefer Python. It's a pretty typical scripting language for cross-platform C++, and contributors are likely to have to set up already.

[rhuanjl]

Personally, I would probably prefer Python. It's a pretty typical scripting language for cross-platform C++, and contributors are likely to have to set up already.

I've now written a little python, it seems easy enough to pick up - in the long term I agree let's aim at getting to only python scripts - though probably a little way of.

[rhuanjl]

On a related note the test suite is run on macOS and Linux using runtests.py but on windows using an executable called rl.exe.

Ideally rl.exe should also be removed in the long term along with all of these different script types. See also #6445

[ppenzin]

+1 on dropping rl.exe in favor of Python runner - it has been doing a pretty good job on *nix-es

[rhuanjl]

#6583 (if merged) when complete will remove a ps1 script and several cmd scripts in favour of python.

[Taritsyn]

I am also in favor of using PowerShell.

[rhuanjl]

I'm curious what the strengths of PowerShell are for @Taritsyn and @hez2010 to both reccomend it?

Note at the moment the key scripts are:
.cmd files on windows
.py or .sh on linux and macOS

The existing .ps1 scripts are small/rarely used.

[Penguinwizzard]

Just from past experience, the Python scripts in the repo are the ones that have probably received the most care over the past few years. On top of this, Python is much more widely known than Powershell, and it doesn't lend itself as easily as Powershell to taking dependencies on system-specific behavior.

[Taritsyn]

The main advantage of PowerShell is that most developers working with Microsoft stack are familiar with it. Almost all Microsoft's open-source projects use a combination of CMD files, PowerShell and Unix shell scripts to automate the build process (for example, ASP.NET Core and Roslyn).

[rhuanjl]

There's certainly no pressing short term reason to move to a single scripting language, that said it would be nice to solidify and avoid any further diversification.

Replacing the the perl scripts (which run a benchmark suite) with python may be a nice to have just to remove a dev dependency (Perl) but certainly not urgent.

It would broadly be nice to rationalise and tidy up - quite a few scripts aren't used often if at all. Also some tasks are split across multiple scripts for no good reason e.g.g jenkins/buildone.cmd simply checks that it has valid parameters and if so redirects to test/jenkins.buildone.cmd (which in turn uses several other scripts) one may say "if it works why fix it" but it's needlessly complex.

Breaking complex tasks into multiple parts can help maintainability BUT in this case we're talking about parsing some command line arguments then calling msbuild and it goes via maybe 5 different .cmd files.

I note that we're maintaining 2 separate build systems (CMake for linux and macOS and MSVC/msbuild for windows) I don't see that changing any time soon and so for it now it may be appropriate to have scripts to do with the windows build process in .ps1 or .cmd whilst having anything intended to be linux/macOS moving to python if possible.

[ppenzin]

I think we can mostly get rid of the shell wrapper around CMake, or at least strive to get rid of it. We are not using it in CI, for example. At some point we might be able to consolidate the build system as well, but that won't be very soon.