The def executable for CGIHTTPRequestHandler is simply wrong on Windows.  The Unix executable bits do not apply.

Yet it is not clear what to use instead.  One could check the extension against PATHEXT, perhaps, but Windows doesn't limit itself to that except when not finding the exactly specified executable name.  Or one could require and borrow the unix #! convention.  As an experiment, since I'm mostly dealing the script files, I tried out a hack that implements two #! lines, the first for Unix and the second for Windows, and only consider something executable if the second line exists.  This fails miserably for .exe files, of course.

Another possibility would be to see if there is an association for the extension, but that rule would permit a Word document to be "executable" because there is a way to open it using MS Word.

Another possibility would be to declare a list of extensions in the server source, like the list of directories from which CGIs are found.

Another possibility would be to simply assume that anything found in the CGI directory is executable.

Another possibility is to require the .cgi extension only to be executable, but then it is hard to know how to run it.

Another possibility is to require two "extensions"... the "real" one for Windows, and then .cgi just before it.  So to make a program executable, it would be renamed from file.ext to file.cgi.ext

But the current technique is clearly insufficient.

I think it's fairly clear what should be considered executable in this context: anything that subprocess.Popen is able to launch (because if is_executable succeeds, this is what will happen to the file). This excludes word files, and also excludes files ending in .cgi; only files ending in .exe are executable.

Martin, that is an interesting viewpoint, and one I considered, but didn't state, because it seems much too restrictive.  Most CGI programs are written in scripting languages, not compiled to .exe.  So it seems the solution should allow for launching at least Perl and Python scripts, as well as .exe.  Whether subprocess.Popen can directly execute it, or whether it needs help from the registry or a #! line to get the execution going is just a matter of tweaking the coding for what gets passed to subprocess.Popen.  Declaring the definition based on what the existing code can already do is self-limiting.

Another possible interpretation of executable might be PATHEXT environment variable, but that is similar to declaring a list in the server source, which I did mention.  One might augment the other.

> Martin, that is an interesting viewpoint, and one I considered, but
> didn't state, because it seems much too restrictive.  Most CGI
> programs are written in scripting languages, not compiled to .exe.
> So it seems the solution should allow for launching at least Perl and
> Python scripts, as well as .exe.

Notice that it does support launching Python scripts. I disagree that
Perl scripts need to be supported. The idea of CGI is really that
"programs" get run by the web server, with the notion of "programs"
clearly deviating from system to system. Window really doesn't support
"scripts" (in the hash-bang sense), and it isn't the function of
http.server to extend Windows here. At best, support for .bat files
might be negotiable, using cmd.exe to launch them (but I personally
would not).

Anybody who wants support for other kinds of scripts on Windows will
have to subclass CGIHTTPRequestHandler (and it might be useful to
simplify subclassing that class).

In any case, the bug as stated ("def executable is simply wrong")
has a clear resolution - make it match what the rest of the code
supports. Anything beyond that is a feature request.

The rest of the code has clearly never had its deficiencies exposed on Windows, simply because executable() has prevented that.  So what the rest of the code "already supports" is basically nothing.  Reasonable Windows support is appropriate to implement as part of the bugfix.

You state that it isn't the function of http.server to extend Windows, however, even MS IIS has extended Windows to provide reasonable web scripting functionality, albeit it its own way, thus convicting the Windows facilities of being insufficient.  Attempting to use http.server to get a web testing environment running so that Python scripts can be tested locally requires some way of using an existing environment (except, of course, for "all new" web sites).  I suppose you would claim that using http.server for a web testing environment is an inappropriate use of http.server, also.  

Yet http.server on Unix appears to provide an adequate web testing environment: yes, some of that is because of Unix's #! feature.  This would certainly not be the first case where more code is required on Windows than Unix to implement reasonable functionality.

My desire for support for Perl is not an attempt to convince Python developers to use Perl instead of Python, but simply a reflection of the practicality of life: There are a lot of Perl CGI scripts used for pieces of Web servers.  Reinventing them in Python may be fun, but can be more time consuming than projects may have the luxury to do.

Your claim that it already supports Python CGI scripts must be tempered by the documentation claim that it provides "altered semantics".  "altered semantics", as best as I can read in the code, is that the query string is passed to the Python script as a command line if it doesn't happen to contain an "=" sign.  This is weird, unlikely to be found in a real web server, and hence somewhat useless for use as a test server also.

http.server has chosen to use subprocess which has chosen to use CreateProcess as its way of executing CGI.  There are other Windows facilities for executing programs, such as ShellExecute, but of course it takes the opposite tack: it can "execute" nearly any file, via registry-based associations.  Neither of these seem to be directly appropriate for use by http.server, the former being too restrictive without enhancements, the latter being too liberal in executing too many file types, although the requirement that CGI scripts live in specific directories may sufficiently rein in that liberality.

However, you have made me think through the process: it seems that an appropriate technique for Windows is to allow for a specific set of file extensions, and permit them to be executed using the registry-based association to do so.  However, for .cgi files, which depend heavily on the Unix #!, emulation of #! seems appropriate (and Windows doesn't seem to have an association for .cgi files either).

Your suggestion of making CGIHTTPRequestHandler easier to subclass is certainly a good one, and is almost imperative to implement to fix this bug in a useful manner without implementing an insufficient set of Windows extensions (for someone's definition of wrong).  There should be a way to sidestep the "altered semantics" for Python scripts (and Python scripts shouldn't have to be a special case, they should work with the general case), without replacing the whole run_cgi() function.  There should be a hook to define the list of executable extensions, and how to run them, and/or a hook to alter the command line passed to subprocess.Popen to achieve same.

So is_executable and is_python both seem to currently be replacable.  What is missing is a hook to implement cmdline creation before calling subprocess.Popen()  (besides the other reported bugs, of course)

> Your suggestion of making CGIHTTPRequestHandler easier to subclass is
> certainly a good one, and is almost imperative to implement to fix
> this bug in a useful manner without implementing an insufficient set
> of Windows extensions (for someone's definition of wrong).

It's indeed the approach I would prefer over the alternatives you
suggested - I particularly dislike Python implementing a strategy
where #! files become considered on Windows (you then immediately
run into subsequent problems, such as /usr/bin/perl being no valid
filename on most Windows installations).

So I maintain that technically, in order to resolve the *reported*
issue (msg121875), it is sufficient to define that executables
on Windows are the files ending with .exe. To recall, the reported
issue is "is simply wrong ... is not clear what to use instead"
(to you as the reporter). My job as a maintainer is to resolve this,
and I will decide to resolve this as suggested. Even the refactoring
to allow substitution of process creation is an independent feature,
but I'm willing to accept patches.

Martin, you are splitting hairs about the "reported problem".  The original message does have a paragraph about the executable bits being wrong.  But the bulk of the message is commenting about the difficulty of figuring out what to replace it with.

So it looks like in spite of the hair splitting, we have iterated to a design of making run_cgi a bit friendlier in this regard.

I find it sufficient to define a method fully extracted from run_cgi as follows:

    def make_cmdline( self, scriptfile, query ):
        cmdline = [scriptfile]
        if self.is_python(scriptfile):
            interp = sys.executable
            if interp.lower().endswith("w.exe"):
                # On Windows, use python.exe, not pythonw.exe
                interp = interp[:-5] + interp[-4:]
            cmdline = [interp, '-u'] + cmdline
        if '=' not in query:
            cmdline.append(query)

This leaves run_cgi with:

            import subprocess
            cmdline = self.make_cmdline( scriptfile, query )
            self.log_message("command: %s", subprocess.list2cmdline(cmdline))


Apologies: I don't know what format of patch is acceptable, but this is a simple cut-n-paste change.  I was sort of holding off until the hg conversion to figure out how to make code submissions, since otherwise I'd have to learn it twice in short order.

I have reimplemented my work-arounds in terms of the above fix, and they function correctly, so this fix would suffice for me, for this issue.  (N.B. I'm sure you've noticed that I have entered a number of issues for http.server; I hope that was the right way to do it, to attempt to separate the issues.)

Hi,
I would like to help on this issue.
Let me know what can be done?

I don’t know if CGI on windows servers is very relevant nowadays.

Hi  Éric,

Then would the easiest way of dealing with this issue, to close it without
any further work?
Possibly adding to the documents it doesn't work on windows..
I can move to the next issue.

Regards,
Mike

On Sun, 31 Oct 2021 at 18:03, Éric Araujo <report@bugs.python.org> wrote:

>
> Éric Araujo <merwok@netwok.org> added the comment:
>
> I don’t know if CGI on windows servers is very relevant nowadays.
>
> ----------
> nosy: +eric.araujo
> versions:  -Python 3.2
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue10483>
> _______________________________________
>

As the original author of this report, getting significant pushback from the then maintainer of the code, I went ahead and implemented what I considered to be a useful set of features to make CGI work on Windows.

I never proposed them as a patch, because they were bundled with a bunch of other things that certainly would have been considered feature requests, and I never figured out the process for submitting patches, which has changed a couple times since the bug was submitted, and I didn't have time to do all of
(1) learn the churning patch mechanisms
(2) separate my code changes into independent chunks
(3) also get my projects completed

The only reason that CGI support on Windows may not be relevant today is that it is so outdated, with no support or encouragement from the maintainers to encourage patches to make it useful, discouraging its use on Windows.  With my private, forked implementation, I have a quite useful Windows testbed for implementing web server code that can run on Linux Apache servers.  If only there were such support built-in!

Glen, which version of Python is your fork using?

Ethan,  3.7.2 at the moment.  I doubt there'd be any issues moving it to 3.11, unless there have been significant changes to the main branch in that area, I just haven't bothered because it is working for me as is.  I forget what version I started with, but I guess 3.6 was when I first reported it?  I know I did have to tweak it a little when I upgraded Python once, and there had been changes to http.server in the meantime.

Should I go ahead and make the changes as per msg122208 ? on my local copy and test?

It’s not clear to me if CGI support in http.server is a relic or still something useful and in wide use.  You can for sure make a pull request and see if a core dev will review and merge it, or you could raise the question on Discuss to see what people think.

I have put the question on Discuss and wait for (any) responses,

> I have put the question on Discuss and wait for (any) responses,

Link to the discussion:
https://discuss.python.org/t/issue-10483-http-server-what-is-executable-on-windows/11856

I searched for open issues which contain "cgi" in their title. I found 43 open issues. The oldest is 101 months ago.

In 10 years, Lib/cgi.py got 43 commits. Only 13 commits in the last 5 years (since 2016-01-01).

It seems like the cgi module is not really maintained anymore. One option is to do nothing: I guess that the most basic features continue to work. Another option is to start deprecating all code related to CGI in the stdlib.

While CGI is not "commonly" used, it seems like it remains popular for specific usages. So I don't think that it's time to deprecate it :-)


> You can for sure make a pull request and see if a core dev will review and merge it

Yep, just do that ;-) So far, nobody proposed a PR to fix this issue.

Hi Victor,

Ok I'll do that.

Regards,
Mike

On Thu, 18 Nov 2021 at 15:36, STINNER Victor <report@bugs.python.org> wrote:

>
> STINNER Victor <vstinner@python.org> added the comment:
>
> I searched for open issues which contain "cgi" in their title. I found 43
> open issues. The oldest is 101 months ago.
>
> In 10 years, Lib/cgi.py got 43 commits. Only 13 commits in the last 5
> years (since 2016-01-01).
>
> It seems like the cgi module is not really maintained anymore. One option
> is to do nothing: I guess that the most basic features continue to work.
> Another option is to start deprecating all code related to CGI in the
> stdlib.
>
> While CGI is not "commonly" used, it seems like it remains popular for
> specific usages. So I don't think that it's time to deprecate it :-)
>
>
> > You can for sure make a pull request and see if a core dev will review
> and merge it
>
> Yep, just do that ;-) So far, nobody proposed a PR to fix this issue.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue10483>
> _______________________________________
>

On 11/18/2021 7:36 AM, STINNER Victor wrote:
> STINNER Victor <vstinner@python.org> added the comment:
>
> I searched for open issues which contain "cgi" in their title. I found 43 open issues. The oldest is 101 months ago.
>
> In 10 years, Lib/cgi.py got 43 commits. Only 13 commits in the last 5 years (since 2016-01-01).
>
> It seems like the cgi module is not really maintained anymore. One option is to do nothing: I guess that the most basic features continue to work.
I got the definite feeling 10 years ago that CGI wasn't being 
maintained, at least for Windows use, and that's why I forked it for 
private use and enhancement.  Nothing in the interim has made me change 
my mind, and your statistics support that.  I got the impression that 
most core developers used Unix-variants for their development, and 
likely most CGI users as well, and that Windows use of CGI just wasn't a 
priority for anyone in the core team.  But it has been useful for me, 
and if a little work were put into supporting it, I suspect it could be 
useful to others as well. But as a lowly non-core user unfamiliar with 
the ever-changing processes for submitting patches, I felt rather 
ignored (which apparently isn't an uncommon issue for users like me), 
but neither did I have the time to invest to learn the submission 
protocols, much less to advance to core development status, so I 
realized that it was partly my fault as well.

> Another option is to start deprecating all code related to CGI in the stdlib.
>
> While CGI is not "commonly" used, it seems like it remains popular for specific usages. So I don't think that it's time to deprecate it :-)

Deprecation would certainly be a disservice to the community, as CGI is 
the only (as far as I know) universal service available on pretty much 
all web server implementations.  There are probably better interfaces to 
user-supplied code on many web servers, but the ones I've heard about 
are not as universal, and not as simple for users to implement.

> I got the impression that most core developers used Unix-variants for their development, and likely most CGI users as well, and that Windows use of CGI just wasn't a priority for anyone in the core team.

Core devs are volunteers who are free to pick what they like the most. There are few core devs who use Python on Windows, whereas they are many Python users on Windows.

If you are able to split your enhancements as small PRs, I may be able to have a look. We do run our test suite on Windows. If each enhancement has a related test valided on Windows, it's ok.


> Deprecation would certainly be a disservice to the community, as CGI is 
the only (as far as I know) universal service available on pretty much 
all web server implementations.

The other question is not if CGI is useful or not, but if it should be maintained inside the stdlib, or on PyPI. Well, it's not exclusive :-) It's possible to have a more complete implementation on PyPI under a different name.

I will work on this next week.