Not compatible with pyenv-win or other Python environments without PY.EXE

[phith0n]

When I setup a new database with Python language on Windows, an exception was raised:

λ codeql database create talkbook --language=python --source-root=d:/pro/talkbook
Initializing database at d:\pro\codeql-repo\talkbook.
Running command [D:\program\codeql\python\tools\autobuild.cmd] in d:\pro\talkbook.
[2020-08-21 02:27:51] [build] Calling py -3 D:\program\codeql\python\tools\get_venv_lib.py
[2020-08-21 02:27:51] [build] Error trying to run get_venv_lib (this is Python 3.7.2)
[2020-08-21 02:27:51] [build]     Traceback (most recent call last):
[2020-08-21 02:27:51] [build]       File <buildtools\install.py> line 62, in lib
[2020-08-21 02:27:51] [build]         self._lib = subprocess.check_output(args)
[2020-08-21 02:27:51] [build]       File "C:\Users\phith0n\.pyenv\pyenv-win\versions\3.7.2\lib\subprocess.py", line 395, in check_output
[2020-08-21 02:27:51] [build]         **kwargs).stdout
[2020-08-21 02:27:51] [build]       File "C:\Users\phith0n\.pyenv\pyenv-win\versions\3.7.2\lib\subprocess.py", line 472, in run
[2020-08-21 02:27:51] [build]         with Popen(*popenargs, **kwargs) as process:
[2020-08-21 02:27:51] [build]       File "C:\Users\phith0n\.pyenv\pyenv-win\versions\3.7.2\lib\subprocess.py", line 775, in __init__
[2020-08-21 02:27:51] [build]         restore_signals, start_new_session)
[2020-08-21 02:27:51] [build]       File "C:\Users\phith0n\.pyenv\pyenv-win\versions\3.7.2\lib\subprocess.py", line 1178, in _execute_child
[2020-08-21 02:27:51] [build]         startupinfo)
[2020-08-21 02:27:51] [build]     FileNotFoundError: [WinError 2] The system cannot find the file specified
[2020-08-21 02:27:51] [build] Calling py -3 -S D:\program\codeql\python\tools\python_tracer.py -v -z all -c d:\pro\codeql-repo\talkbook\working\trap_cache -p d:\pro\codeql-repo\talkbook\working\venv\Lib -R d:\pro\talkbook
[2020-08-21 02:27:51] [build-err] Traceback (most recent call last):
[2020-08-21 02:27:51] [build-err]   File "D:\program\codeql\python\tools\index.py", line 19, in <module>
[2020-08-21 02:27:51] [build-err]     buildtools.index.main()
[2020-08-21 02:27:51] [build-err]   File "D:\program\codeql\python\tools\python3src.zip\buildtools\index.py", line 110, in main
[2020-08-21 02:27:51] [build-err]   File "C:\Users\phith0n\.pyenv\pyenv-win\versions\3.7.2\lib\subprocess.py", line 342, in check_call
[2020-08-21 02:27:51] [build-err]     retcode = call(*popenargs, **kwargs)
[2020-08-21 02:27:51] [build-err]   File "C:\Users\phith0n\.pyenv\pyenv-win\versions\3.7.2\lib\subprocess.py", line 323, in call
[2020-08-21 02:27:51] [build-err]     with Popen(*popenargs, **kwargs) as p:
[2020-08-21 02:27:51] [build-err]   File "C:\Users\phith0n\.pyenv\pyenv-win\versions\3.7.2\lib\subprocess.py", line 775, in __init__
[2020-08-21 02:27:51] [build-err]     restore_signals, start_new_session)
[2020-08-21 02:27:51] [build-err]   File "C:\Users\phith0n\.pyenv\pyenv-win\versions\3.7.2\lib\subprocess.py", line 1178, in _execute_child
[2020-08-21 02:27:51] [build-err]     startupinfo)
[2020-08-21 02:27:51] [build-err] FileNotFoundError: [WinError 2] The system cannot find the file specified
[2020-08-21 02:27:51] [ERROR] Spawned process exited abnormally (code 1; tried to run: [D:\program\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, D:\program\codeql\python\tools\autobuild.cmd])
A fatal error occurred: Exit status 1 from command: [D:\program\codeql\tools\win64\runner.exe, cmd.exe, /C, type, NUL, &&, D:\program\codeql\python\tools\autobuild.cmd]

After review about the reason of this mess, I found that I don't have the py.exe executable program, which is used by python\tools\python3src.zip\buildtools\version.py:

WIN = (sys.platform == 'win32')

def executable(version):
    if WIN:
        return ['py', ('-%s' % version)]
    else:
        return [('python%s' % version)]

I install the Python with pyenv-win, which is a Python version management. Pyenv-win will not install the py.exe for users, maybe it was the cause of the exception.

Otherwise, py.exe is not a necessary thing for Python environment. So I believe there is a much better way to choice a right executable program rather than depending on py.exe.

[hmakholm]

Can @github/codeql-python comment here?

[RasmusWL]

Hi @phith0n, thanks for reporting this 👍 I personally didn't know about pyenv-win before now, but I am also not that familiar with Windows development.

Fixing this is sadly not going to be top priority right now, but we will get to it in the future.

For your personal experiments with codeql, I would recommend either making a proxy script so py -3 calls your real python3 executable, or alternatively unzipping python\tools\python3src.zip changing the code so it works on your platform, and zipping it up again. Not an ideal solution, but it's the best I can do right now 😊

PS. sorry for the long response time, didn't spot this issue when I got back from holidays.

[phith0n]

Thank you @RasmusWL, I have reinstalled Python without pyenv-win already, hope this issue helps others.