classification
Title: webbrowser module broken with Firefox 36+
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: bazwal, berker.peksag, ezio.melotti, lilydjwg, phd, python-dev, serhiy.storchaka, ssokolow
Priority: normal Keywords: patch

Created on 2015-01-18 04:16 by ssokolow, last changed 2017-04-01 05:47 by serhiy.storchaka.

Files
File name Uploaded Description Edit
webbrowser.py-3.5.patch phd, 2016-10-26 08:43
webbrowser.py-3.5.patch serhiy.storchaka, 2016-10-27 07:19 Regenerated for review review
webbrowser.py-3.5-2.patch serhiy.storchaka, 2016-10-27 08:12 review
Messages (45)
msg234218 - (view) Author: Stephan Sokolow (ssokolow) Date: 2015-01-18 04:16
As of Firefox 36 (currently in beta channel), the -remote option has been removed.

https://www.mozilla.org/en-US/firefox/36.0a2/auroranotes/
https://www.mozilla.org/en-US/firefox/36.0beta/releasenotes/

As such, attempting to open http://www.example.com/ using webbrowser.open() or webbrowser.open_new_tab() results in "File not Found" tabs pointing to these two URLs, respectively:

file:///home/ssokolow/openURL%28http://www.example.com/%29
file:///home/ssokolow/openURL%28http://www.example.com/,new-tab%29

As I happen to have the Dead Snakes PPA set up on Lubuntu 14.04 for use with tox, I was able to confirm this as an issue in Python 2.7, 3.1, 3.2, 3.3, and 3.4.
msg234248 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-01-18 14:32
What is the right way to do it then? It should also remain compatible with Firefox < 36.
msg234256 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-01-18 16:20
Details: https://bugzilla.mozilla.org/show_bug.cgi?id=1080319
msg234284 - (view) Author: Stephan Sokolow (ssokolow) Date: 2015-01-18 22:15
The proper solution is to prefer `start` (Windows), `open` (OSX), or `xdg-open` (everything else... usually but not always present) when present instead of calling the browser directly.

That way, you're using the same "delegate to the desktop's associations system and let the user's preferences control new window vs. new tab" behaviour on all OSes where it's reliably possible.

(Yes, it would guarantee that all open* functions are equivalent, but that's already the norm in a lot of cases... especially with Firefox where one of the guiding design principles is ensuring that the user retains control of their browsing experience.)
msg234285 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-01-18 22:26
That doesn't really answer the question. The webbrowser module is already able to use "xdg-open" (though not "open"), but it has a fallback on various concrete browsers (including Firefox) in case xdg-open doesn't work / doesn't exist.
msg234289 - (view) Author: Stephan Sokolow (ssokolow) Date: 2015-01-18 22:58
Well, then the code which chooses a backend is broken because I have xdg-open and, according to WinPdb, it's using the Mozilla backend.

This was my local workaround:

    if os.name == 'posix' and not platform.mac_ver()[0]:
        with open(os.devnull, 'wb') as nul:
            subprocess.Popen(['xdg-open', request_url], stdout=nul, stderr=nul)
    else:
        webbrowser.open_new_tab(request_url)

(This retrofit branch hasn't yet reached the point where it'll be tested on Windows. I may still add another branch which calls `start` directly if it proves to have the same priority bug on Windows.)
msg234290 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-01-18 23:00
> Well, then the code which chooses a backend is broken because I have
xdg-open and, according to WinPdb, it's using the Mozilla backend.

You have xdg-open under Windows?
msg234291 - (view) Author: Stephan Sokolow (ssokolow) Date: 2015-01-18 23:05
WinPdb = Windowed Pdb, not MS Windows Pdb.

`sudo apt-get install winpdb`
msg234292 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-01-18 23:06
Oops, sorry. The best way forward would be for you to investigate and find out why xdg-open isn't preferred in your setting.
msg234293 - (view) Author: Stephan Sokolow (ssokolow) Date: 2015-01-18 23:09
Noted. I'm not sure what my schedule will be like, but I'll try.

(I may get back to you with an answer later today or I may not within the week.)
msg236529 - (view) Author: Oleg Broytman (phd) * Date: 2015-02-24 19:02
I created the attached module to work with new FF in Python 2.7. You can convert it to proper patch for webbrowser.py.
msg236727 - (view) Author: bazwal (bazwal) Date: 2015-02-27 02:45
I arrived at an almost identical solution to the one given by Oleg. It works for me with FF 28, 35 & 36 - but presumably there are earlier versions where it does not, otherise why have the -remote option at all?

The gist of the Mozilla bug report is that the option has existed for a very long time, but there are now other command-line options that do exactly the same thing. So the only question seems to be whether Python should still support those older versions of FF that actually require it (if any exist).

On the xdg-open issue: the $BROWSER env var has priority over the default list of browsers to try, so that might explain why xdg-open is not being used on some systems (that is how I originally became aware of this problem).
msg236737 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-27 11:44
--new-window and --new-tab are Firefox and SeaMonkey2.x only. [1] So this will break the support of IceApe on Debian Etch (iceape v.1.0.13 [2]).

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Command_Line_Options
[2] http://archive.debian.net/etch/devel/iceape
msg236826 - (view) Author: bazwal (bazwal) Date: 2015-02-27 20:05
> this will break the support of IceApe on Debian Etch

Is Python committed to supporting obsolete Debian releases? The latest oldstable Debian release (6.0 Squeeze) has iceape-2.0.11 [1], which does seem to support the -new-window and -new-tab options (I haven't actually tested this, though - I just looked at the source code [2]).

The iceweasel/iceape products are just rebranded versions of Firefox and Seamonkey. The webbrowser module currently supports them via the generic Mozilla class - so Python should only be concerned about which versions of the common Mozilla codebase it needs to continue support for.

[1] https://packages.debian.org/search?suite=squeeze&searchon=names&keywords=iceape
[2] http://anonscm.debian.org/cgit/pkg-mozilla/iceape.git/tree/mozilla/browser/components/nsBrowserContentHandler.js?h=squeeze#n419
msg236838 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-02-27 21:01
May it is time to drop support of SeaMonkey 1.x and systems released 8 years ago. But then we should drop the support of browsers "netsape", "mozilla", "firebird". And this should be documented somewhere.
msg270554 - (view) Author: Oleg Broytman (phd) * Date: 2016-07-16 10:44
Patch for Python 2.7 to support Firefox >= 36.0, with version test.
msg270555 - (view) Author: Oleg Broytman (phd) * Date: 2016-07-16 10:45
Patch for Python 3.4 and 3.5 to support Firefox >= 36.0, with version test.
msg270556 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-07-16 12:34
Thanks for the patches! Can't you just use a function from the high level subprocess API (like subprocess.run()?
msg270563 - (view) Author: Oleg Broytman (phd) * Date: 2016-07-16 15:24
Not subprocess.run() as I'm aiming to Python 2.7. Perhaps subprocess.check_output(). I'll try it.
msg270565 - (view) Author: Oleg Broytman (phd) * Date: 2016-07-16 15:47
Updated patch for Python 2.7 with subprocess.check_output().
msg270566 - (view) Author: Oleg Broytman (phd) * Date: 2016-07-16 15:47
Updated patch for Python 3.4+ with subprocess.check_output().
msg270567 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-07-16 15:49
I was talking about webbrowser.py-3.4-newfox.patch

3.4 is now in security-fix-only mode so we can't push the patch to the 3.4 branch.
msg270569 - (view) Author: Oleg Broytman (phd) * Date: 2016-07-16 15:53
I am on Debian 8 "jessie" that only has Python 3.4, so I want to patch webbrowser.py in a backward-compatible way.
msg278340 - (view) Author: Oleg Broytman (phd) * Date: 2016-10-09 04:38
Add NewFox class to webbrowser for Py 2.7 (added raise_opts — Mozilla no longer supports autoraise).
msg278341 - (view) Author: Oleg Broytman (phd) * Date: 2016-10-09 04:39
Add NewFox class to webbrowser for Py 3.4 (added raise_opts — Mozilla no longer supports autoraise).
msg278346 - (view) Author: Oleg Broytman (phd) * Date: 2016-10-09 05:40
Remove empty args (backported from 3.4).
msg279381 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-25 10:05
Would not be easier to drop a support of SeaMonkey 1.x and other ancient browsers?
msg279384 - (view) Author: Oleg Broytman (phd) * Date: 2016-10-25 10:16
I don't think Firefox versions <36 are too old to be dropped.
msg279390 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-25 10:46
> I don't think Firefox versions <36 are too old to be dropped.

Just Firefox versions <1.5.

Firefox 1.5 released 11 years ago supported options -new-window and -new-tab [1].

[1] http://website-archive.mozilla.org/www.mozilla.org/firefox_releasenotes/en-US/firefox/releases/1.5.html
msg279392 - (view) Author: Oleg Broytman (phd) * Date: 2016-10-25 11:09
Ah, I see. Ok then, see the new patch.
msg279399 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-25 12:25
Tests are failed with webbrowser.py-3.5-newfox.patch.
msg279400 - (view) Author: Oleg Broytman (phd) * Date: 2016-10-25 12:35
What's the error? Buildbot URL?
msg279403 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-25 13:28
$ ./python -m test.regrtest -vuall test_webbrowser
...
======================================================================
FAIL: test_open (test.test_webbrowser.MozillaCommandTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython-3.5/Lib/test/test_webbrowser.py", line 99, in test_open
    arguments=['openURL({})'.format(URL)])
  File "/home/serhiy/py/cpython-3.5/Lib/test/test_webbrowser.py", line 42, in _test
    self.assertIn(option, popen_args)
AssertionError: '-raise' not found in ['http://www.example.com']
msg279407 - (view) Author: Oleg Broytman (phd) * Date: 2016-10-25 13:45
Oops, yes, mea culpa, sorry. Patch for the test added.
msg279491 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-26 08:16
Could you please provide a patch including all changes in unified format?
msg279492 - (view) Author: Oleg Broytman (phd) * Date: 2016-10-26 08:43
Done.
msg279529 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-27 08:12
The problem is that this class is used for Netscape and old Mozilla browsers. What if rename old Mozilla class to Netscape and use it for old browsers?
msg279541 - (view) Author: Oleg Broytman (phd) * Date: 2016-10-27 15:46
I'd rather rename the new class to something like Firefox. So there will be 3 classes — Netscape, Mozilla and Firefox. Firefox only for firefox executable.
msg279544 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-27 16:22
Currently Netscape is just an alias to Mozilla. Since SeaMonkey is synchronized with Firefox, I suppose it dropped the support of the -remote option at the same time. Debian rebranded Firefox and SeaMonkey at 2006, therefore IceWeasel and IceApe always supported -new-win and -new-tab options. The last IceWeasel version was 38, it didn't supported the -remote option.
msg279545 - (view) Author: Oleg Broytman (phd) * Date: 2016-10-27 16:28
Then I don't have any objections. But I also couldn't test the change — I only use Firefox (and sometimes Chrome).
msg279735 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-10-30 17:22
New changeset dacb52577c1c by Serhiy Storchaka in branch '3.5':
Issue #23262: The webbrowser module now supports Firefox 36+ and derived
https://hg.python.org/cpython/rev/dacb52577c1c

New changeset f1abc92a756a by Serhiy Storchaka in branch '3.6':
- Issue #23262: The webbrowser module now supports Firefox 36+ and derived
https://hg.python.org/cpython/rev/f1abc92a756a

New changeset cade42bd0ee0 by Serhiy Storchaka in branch 'default':
Issue #23262: The webbrowser module now supports Firefox 36+ and derived
https://hg.python.org/cpython/rev/cade42bd0ee0
msg279766 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-10-31 06:43
What to do with 2.7? We can just backport the patch from 3.5, but also can use the version check from earlier Oleg's patches. 2.7 was released long time ago and still can be used on systems with old browsers (however it is very unlikely that Python is updated, but browsers are not).
msg279767 - (view) Author: Oleg Broytman (phd) * Date: 2016-10-31 06:52
Let's me disagree because Python 2.7 is a very special case. One can easily update a browser — they are perfectly compatible. But one cannot just update Python 2.7 because Python 3 is rather a different language that requires a lot of porting effort. So I can imagine Python 2.7 with new browser on the same computer.
msg279989 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-11-03 11:53
I don't understand with what you disagree. I can imagine old Python 2.7 with old browser, old Python 2.7 with new browser, new Python 2.7 with new browser, and even new Python 2.7 with old browser on the same computer (but the latter is very unlikely).

Old Python 2.7 don't work with new browser and we can't do anything with this. We can make new Python 2.7.12 working with new browser. But I'm not sure that we can break the compatibility with old browser that could be installed when Python 2.7 was first installed on the same computer.
msg280130 - (view) Author: Oleg Broytman (phd) * Date: 2016-11-06 04:38
> I'm not sure that we can break the compatibility with old browser

I agree with this.
History
Date User Action Args
2017-04-01 05:47:24serhiy.storchakasetpull_requests: - pull_request903
2017-03-31 16:36:15dstufftsetpull_requests: + pull_request903
2016-12-16 17:09:50serhiy.storchakasetassignee: serhiy.storchaka ->
2016-11-06 04:38:21phdsetmessages: + msg280130
2016-11-03 11:53:20serhiy.storchakasetmessages: + msg279989
2016-10-31 06:52:44phdsetmessages: + msg279767
2016-10-31 06:43:10serhiy.storchakasetmessages: + msg279766
2016-10-30 17:22:35python-devsetnosy: + python-dev
messages: + msg279735
2016-10-30 17:09:52serhiy.storchakasetassignee: serhiy.storchaka
2016-10-27 16:28:18phdsetmessages: + msg279545
2016-10-27 16:22:50serhiy.storchakasetmessages: + msg279544
2016-10-27 15:46:45phdsetmessages: + msg279541
2016-10-27 08:12:47serhiy.storchakasetfiles: + webbrowser.py-3.5-2.patch

messages: + msg279529
2016-10-27 07:19:22serhiy.storchakasetfiles: + webbrowser.py-3.5.patch
2016-10-26 08:43:01phdsetfiles: + webbrowser.py-3.5.patch

messages: + msg279492
versions: - Python 2.7
2016-10-26 08:42:32phdsetfiles: - test_webbrowser.py-3.5-newfox.patch
2016-10-26 08:42:27phdsetfiles: - webbrowser.py-3.5-newfox.patch
2016-10-26 08:42:21phdsetfiles: - webbrowser.py-2.7-newfox.patch
2016-10-26 08:42:13phdsetfiles: - webbrowser.py-3.4-newfox.patch
2016-10-26 08:16:44serhiy.storchakasetmessages: + msg279491
2016-10-25 13:45:01phdsetfiles: + test_webbrowser.py-3.5-newfox.patch

messages: + msg279407
2016-10-25 13:28:57serhiy.storchakasetmessages: + msg279403
2016-10-25 12:35:53phdsetmessages: + msg279400
2016-10-25 12:25:19serhiy.storchakasetmessages: + msg279399
versions: + Python 3.7
2016-10-25 11:09:06phdsetfiles: + webbrowser.py-3.5-newfox.patch

messages: + msg279392
2016-10-25 10:46:00serhiy.storchakasetmessages: + msg279390
2016-10-25 10:16:50phdsetmessages: + msg279384
2016-10-25 10:05:26serhiy.storchakasetmessages: + msg279381
2016-10-09 05:40:01phdsetfiles: + webbrowser.py-2.7-newfox.patch

messages: + msg278346
2016-10-09 05:39:19phdsetfiles: - webbrowser.py-2.7-newfox.patch
2016-10-09 04:39:05phdsetfiles: + webbrowser.py-3.4-newfox.patch

messages: + msg278341
2016-10-09 04:38:44phdsetfiles: + webbrowser.py-2.7-newfox.patch

messages: + msg278340
2016-10-09 04:36:59phdsetfiles: - webbrowser.py-3.4-newfox.patch
2016-10-09 04:36:53phdsetfiles: - webbrowser.py-2.7-newfox.patch
2016-10-09 04:36:47phdsetfiles: - webbrowser.py-3.4-newfox.patch
2016-10-09 04:36:41phdsetfiles: - webbrowser.py-2.7-newfox.patch
2016-10-09 04:36:33phdsetfiles: - new_firefox.py
2016-07-16 15:53:40phdsetmessages: + msg270569
2016-07-16 15:49:21berker.peksagsetmessages: + msg270567
2016-07-16 15:47:31phdsetfiles: + webbrowser.py-3.4-newfox.patch

messages: + msg270566
2016-07-16 15:47:15phdsetfiles: + webbrowser.py-2.7-newfox.patch

messages: + msg270565
2016-07-16 15:24:51phdsetmessages: + msg270563
2016-07-16 12:34:27berker.peksagsetstage: patch review
messages: + msg270556
versions: + Python 3.6, - Python 3.4
2016-07-16 10:47:08pitrousetnosy: - pitrou
2016-07-16 10:45:29phdsetfiles: + webbrowser.py-3.4-newfox.patch

messages: + msg270555
2016-07-16 10:44:57phdsetfiles: + webbrowser.py-2.7-newfox.patch
keywords: + patch
messages: + msg270554
2015-03-02 08:06:38ezio.melottisetnosy: + ezio.melotti
2015-02-27 21:01:33serhiy.storchakasetmessages: + msg236838
2015-02-27 20:05:55bazwalsetmessages: + msg236826
2015-02-27 15:19:10lilydjwgsetnosy: + lilydjwg
2015-02-27 11:44:52serhiy.storchakasetmessages: + msg236737
2015-02-27 03:05:04berker.peksagsetnosy: + berker.peksag
2015-02-27 02:45:14bazwalsetnosy: + bazwal
messages: + msg236727
2015-02-24 19:02:51phdsetfiles: + new_firefox.py
nosy: + phd
messages: + msg236529

2015-01-18 23:09:03ssokolowsetmessages: + msg234293
2015-01-18 23:06:59pitrousetmessages: + msg234292
versions: + Python 3.5, - Python 3.2, Python 3.3
2015-01-18 23:05:35ssokolowsetmessages: + msg234291
2015-01-18 23:00:27pitrousetmessages: + msg234290
2015-01-18 22:58:51ssokolowsetmessages: + msg234289
2015-01-18 22:26:47pitrousetmessages: + msg234285
2015-01-18 22:15:14ssokolowsetmessages: + msg234284
2015-01-18 16:20:21serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg234256
2015-01-18 14:32:21pitrousetnosy: + pitrou
messages: + msg234248
2015-01-18 04:16:56ssokolowcreate