classification
Title: Drop Solaris support
Type: Stage: resolved
Components: Interpreter Core Versions:
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: BTaskaya, Yonatan Goldschmidt, abarbu, alanc, brett3, cgrzemba, gaige, giampaolo.rodola, gwr, ijdt.editor, jcea, jclulow, jgardner100, jm650, jomasoftmarcel, jorge.schrauwen, jstasiak, kulikjak, marcheschi, mariuspana, michael.laspina, mimi.vx, olafbohlen, omnios, otis, pablogsal, pfirmsto, predrag.zecevic.1961, rhettinger, robertdfrench, siepkes, tbalbers, terry.reedy, tropikhajma@gmail.com, vstinner, wiedi, yanchii
Priority: normal Keywords: patch

Created on 2020-10-27 15:59 by vstinner, last changed 2020-11-16 17:05 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
Oracle_Solaris_full_test.txt kulikjak, 2020-11-02 08:18
Oracle_Solaris_detailed_test.txt kulikjak, 2020-11-02 08:18
Pull Requests
URL Status Linked Edit
PR 23002 closed vstinner, 2020-10-27 16:02
Messages (49)
msg379767 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-27 15:59
In past years, very few Python core developers took care of the Solaris support. The latest significant Solaris enhancement was in 2018 with the addition of a new Solaris st_fstype attribute to os.stat(), in Python 3.7, by Jesús Cea Avión (bpo-32659).

Another recent Solaris-related change is bpo-41687: "Fix implementation of sendfile to be compatible with Solaris".

On the Oracle side, there was an article about Python on Solaris in 2019:
"Future of Python on Solaris" by  Darren Moffat 
https://blogs.oracle.com/solaris/future-of-python-on-solaris

I now that Giampaolo Rodola cares of the Solaris support for his psutil project:
https://github.com/giampaolo/psutil/

It's unclear to me if Oracle still plans to maintain Solaris. The latest release was published 2 years ago. The Wikipedia article says: "While Oracle did have a large layoff of Solaris development engineering staff, development continues today of which Solaris 11.4 was released in 2018."
https://en.wikipedia.org/wiki/Solaris_(operating_system)

There is currently 25 open issues which could be closed if we drop the Solaris support.

See also the PEP 11 for the process.

I asked asked 4 years ago (in 2016) if Solaris support should/can be removed:
"OpenIndiana and Solaris support"
https://mail.python.org/archives/list/python-dev@python.org/message/NOT2RORSNX72ZLUHK2UUGBD4GTPNKBUS/

Python has no longer buildbot workers running on Solaris or a Solaris variant (OpenIndiana, Illumos, etc.).

I propose to drop the Solaris support in Python to reduce the maintenance burden.

Python has a few features specific to Solaris:

* st_fstype attribute of os.stat()
* select.devpoll and selectors.DevpollSelector
* time.CLOCK_HIGHRES constant
* stat.S_ISDOOR()
* tarfile: "Solaris extended header" (no need to remove this one)

Solaris has many names and variants:

* SunOS
* SRV4
* Solaris
* Illumos (still active in July 2020 at least)
* OpenSolaris: "On Friday, August 13, 2010, details started to emerge relating to the discontinuation of the OpenSolaris project and the pending release of a new closed-source, proprietary version of Solaris, Solaris 11." says Wikipedia
* OpenIndiana (latest release in 2020)

I'm not sure on how to process? Deprecate all Solaris-specific features in Python 3.10, make sure that Python 3.11 cannot be built on Solaris anymore, and drop the code in Python 3.12? Or drop everything as soon as possible in Python 3.10?

The alternative is to leave the code unchanged and let the code slowly die, as we did for other platforms like HP-UX and IRIX.

See also my notes on platforms supported by Python:
https://pythondev.readthedocs.io/platforms.html

Attached draft PR gives an idea on how much code can be removed if Solaris support is removed. A coase estimation is the removal of 700 lines:

 Doc/c-api/init.rst                           |   3 +-
 Doc/distutils/apiref.rst                     |   1 -
 Doc/library/gettext.rst                      |  10 +-
 Doc/library/os.rst                           |  12 +-
 Doc/library/platform.rst                     |   5 +-
 Doc/library/posix.rst                        |   8 +-
 Doc/library/sys.rst                          |   3 +-
 Doc/library/sysconfig.rst                    |   1 -
 Doc/library/time.rst                         |  13 +-
 Doc/using/unix.rst                           |   7 -
 Include/pyport.h                             |   5 -
 Lib/cgi.py                                   |   4 +-
 Lib/ctypes/test/test_byteswap.py             |   2 +-
 Lib/ctypes/util.py                           | 100 ++--------
 Lib/distutils/command/bdist.py               |   2 +-
 Lib/distutils/tests/support.py               |   2 +-
 Lib/distutils/tests/test_build_ext.py        |  21 --
 Lib/distutils/util.py                        |  11 --
 Lib/gettext.py                               |   3 -
 Lib/logging/handlers.py                      |   2 -
 Lib/platform.py                              |  27 +--
 Lib/sysconfig.py                             |  11 --
 Lib/test/fork_wait.py                        |   4 +-
 Lib/test/subprocessdata/fd_status.py         |   3 -
 Lib/test/test__locale.py                     |  24 ---
 Lib/test/test_asyncio/test_sendfile.py       |   6 -
 Lib/test/test_asyncore.py                    |   2 -
 Lib/test/test_cmd_line.py                    |   2 +-
 Lib/test/test_curses.py                      |   2 +-
 Lib/test/test_fileio.py                      |   2 +-
 Lib/test/test_importlib/import_/test_path.py |   2 +-
 Lib/test/test_locale.py                      |   2 +-
 Lib/test/test_os.py                          |  14 +-
 Lib/test/test_posix.py                       |   7 +-
 Lib/test/test_pty.py                         |   4 -
 Lib/test/test_selectors.py                   |   2 +-
 Lib/test/test_shutil.py                      |   2 +-
 Lib/test/test_socket.py                      |   4 -
 Lib/test/test_strftime.py                    |   3 +-
 Lib/test/test_sys.py                         |   2 +-
 Lib/uuid.py                                  |   5 -
 Modules/Setup                                |   4 -
 Modules/_ctypes/ctypes.h                     |   4 -
 Modules/_cursesmodule.c                      |   2 +-
 Modules/_multiprocessing/multiprocessing.h   |  15 --
 Modules/_posixsubprocess.c                   |  11 --
 Modules/errnomodule.c                        |  23 +--
 Modules/mathmodule.c                         |   2 +-
 Modules/posixmodule.c                        |  65 +-----
 Modules/signalmodule.c                       |   2 +-
 Modules/socketmodule.c                       |   7 -
 Modules/termios.c                            |   8 +-
 Modules/timemodule.c                         |   9 +-
 Python/bootstrap_hash.c                      |  17 +-
 Python/fileutils.c                           |   3 +-
 Python/import.c                              |   3 +-
 Tools/freeze/README                          |   5 -
 Tools/i18n/pygettext.py                      |   3 +-
 Tools/pynche/Main.py                         |   2 -
 Tools/pynche/README                          |   2 +-
 Tools/scripts/nm2def.py                      |   2 +-
 configure                                    | 285 +--------------------------
 configure.ac                                 | 101 +---------
 pyconfig.h.in                                |  19 +-
 setup.py                                     |  20 +-
 65 files changed, 91 insertions(+), 868 deletions(-)
msg379886 - (view) Author: Sebastian Wiedenroth (wiedi) * Date: 2020-10-29 22:13
Please don't.

Even if Oracle doesn't care anymore, the open source illumos community does.
We (illumos) share the same uname "SunOS" and depend on this support.

I have offered to host an illumos buildbot in the past and that offer is still valid.
At that time the testsuite did not pass and I could not figure out the fix yet.
If having this fixed is a requirement to keep illumos supported I will invest more time into this. I can also offer shell access to anyone interested in helping with this.
msg379887 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-29 22:13
I looked at PyPI download statistics of the "six" module over the last 180 days (6 months), grouped by operating system name:

* SunOS: 3,975 downloads
* TOTAL: 590,727,298 downloads

SunOS is 0.0007% of all six downloads.

For comparison, the top 4 is:

* Linux: 91% (535,804,817)
* Windows: 4% (21,053,173)
* <empty system name>: 3% (18,695,360)
* macOS (Darwin): 3% (14,776,993)

Raw data of the top 14:

num_downloads,system_name
535804817,Linux
21053173,Windows
18695360,
14776993,Darwin
347090,FreeBSD
9759,Qnx
5350,Java
3975,SunOS
3178,CYGWIN_NT-10.0-14393
3177,CYGWIN_NT-10.0-17763
2505,CYGWIN_NT-10.0-18363
2263,AIX
2061,NetBSD
1891,OpenBSD
msg379888 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-29 22:14
> We (illumos) share the same uname "SunOS" and depend on this support.

Which Python version do you use?

https://blogs.oracle.com/solaris/future-of-python-on-solaris mentions Python 2.7 and 3.4.
msg379890 - (view) Author: Sebastian Wiedenroth (wiedi) * Date: 2020-10-29 22:22
> Which Python version do you use?

Myself currently mostly python 3.8 via pkgsrc on SmartOS (one of the illumos distributions).

OmniOS ships 3.7: https://github.com/omniosorg/omnios-build/tree/master/build/python37

OpenIndiana ships at least 2.7, 3.5, 3.7.

The illumos build process also depends on python and was updated to support 3.9 two weeks ago: https://www.illumos.org/issues/13217
msg379891 - (view) Author: Ondřej Súkup (mimi.vx) Date: 2020-10-29 22:36
Oracle moved solaris 11.4 to +- rolling release mode, last SRU is 26 from 20.10.2020 with month release cadence.

On another side, almost all informations are behind the Oracle paywall
msg379893 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2020-10-29 22:38
I think Solaris Support is still important and I could invest more time on it.

About package downloads:

1. I routinely download packages in my laptop (Linux) and distribute them to a fleet of servers (Linux and Solaris) via ansible and similar. Many servers in intranets don't install packages directly via PIP.

2. Under OmniOS or SmartOS (Illumos distributions), many people use the system package manager to install Python packages, not "PIP". For instance, http://netbsd.2816.n7.nabble.com/WIP-commit-Upgrade-to-Mailman-2-1-34-td427426.html.

Buildbots: I used to manage a buildbot with OpenIndiana, but it didn't survived the GITHUB migration. I could provide now a SmartOS buildbot.

As a core developer, I do care about Solaris. I fully realize I have been quite inactive since the github migration. My fault. I could devote more time to the task.
msg379894 - (view) Author: Jesús Cea Avión (jcea) * (Python committer) Date: 2020-10-29 22:45
I am using Python 3.8 installed via pkgsrc in SmartOS with some Zones running Python 3.9 self compiled code.

More about the downloads stats:

4% packages in windows: I am pretty sure that most windows programmers install packages via conda-like tools. Surely there is more that 4% of python programmers on windows, even more in machine learning and data science areas, where windows and MacOS X rule (apparently being only 7% of PIP downloads).

Even if the PIP download stats would reflect penetration, Solaris is ahead of NetBSD and OpenBSD together.
msg379895 - (view) Author: Joshua M. Clulow (jclulow) Date: 2020-10-29 22:48
Hi!  I'm a member of the illumos core team, and I'm also pretty keen for us to keep Python support!  Some of our core OS build and packaging tooling is written in Python, and certainly applications like Synapse (Matrix) and Review Board are important to myself.

We currently provide a virtual machine for builds to the Go project that is wired in to their official CI/CD.  I expect we can help out with a similar thing here!

I know some illumos distributions (OmniOS, OpenIndiana, anything with pkgsrc, etc) are maintaining some patches against the stock Python to fix failing tests, etc.  Hopefully we can figure out how to get those things upstreamed and increase our participation there.
msg379897 - (view) Author: Andy Fiddaman (omnios) Date: 2020-10-29 22:59
We at OmniOS (an illumos distribution) are in the process of upgrading to Python 3.9 and working on getting tests clean. We're in pretty good shape with a few local patches and only having to skip a few tests at the moment, and most of those patches should be suitable for integrating generally.

Some of the tests just need adjusting to be aware of the different platform (e.g. illumos does not support multiple SCM_RIGHTS messages in a single packet), some still need investigating to find the root cause, but the list is small.
msg379899 - (view) Author: Andy Fiddaman (omnios) Date: 2020-10-29 23:06
Re: Downloads

OmniOS makes modules like 'six' available through its own IPS packaging system (which is written in Python) so installations won't show up pypi stats.

six is a core module on the platform so it is installed on 1000s of OmniOS machines (and then large numbers of SmartOS, OpenIndiana and tribblix ones too - all using the SunOS tag). The numbers aren't going to be in the same league as other platforms but the downloads stats definitely don't represent the illumos installation base and dependency on Python.
msg379900 - (view) Author: hajma (tropikhajma@gmail.com) Date: 2020-10-29 23:39
Like OmniOS, Oracle Solaris provides the 'six' (and many other) module prepackaged.
You can review a clone of the development repository at
https://github.com/oracle/solaris-userland/tree/master/components/python/six
msg379901 - (view) Author: Peter Firmstone (pfirmsto) Date: 2020-10-30 00:47
No, please continue supporting Illumos, I'm still using it.  Illumos is still relevant on servers.
msg379906 - (view) Author: brett (brett3) Date: 2020-10-30 01:44
Between SmartOS, Illumos, and mainline Solaris, there is ample interest and support for ensuring that SunOS derivatives remain 1st class python platforms.

Platform genocide is both unnecessary and unwarranted.
msg379910 - (view) Author: Robert D. French (robertdfrench) Date: 2020-10-30 03:28
Hello! I just want to lend my voice in favor of maintaining "Solarish" support as well, and offer what help I may for resolving issues.
msg379915 - (view) Author: jason matthews (jm650) Date: 2020-10-30 04:24
Please do not drop support. 

Python is widely used in the "Solaris" community with most people downloading packages from Openindiana/Hipster or Joyent/SmartOS repos. Most "Solaris" users are not using Solaris proper.

The download stats published in this ticket is not a good representation of usage. Similarly most (Free|Open|*)BSD users install from their respective versions of ports.
msg379924 - (view) Author: Tony Albers (tbalbers) Date: 2020-10-30 07:21
No no no, please don't.

Apart from FreeBSD, illumos distros are the only really hard-core UNIX OS'es still freely available, the features taken into account. 
SMF, dtrace and several hypervisor types makes illumos really stand out.

I understand that there are resources that need to be assigned to maintaining Python on illumos/SunOS, but please reach out to their communities, maybe someone can help.
msg379925 - (view) Author: Marius Pana (mariuspana) Date: 2020-10-30 07:28
Please do not drop support for SunOS. 
We are happy to provide some hosting services on SmartOS/Illumos for building.
msg379926 - (view) Author: Jasper Siepkes (siepkes) Date: 2020-10-30 07:33
I'm another Illumos user that has crawled from under a rock ;-) to request not to drop Illumos support.
msg379927 - (view) Author: Carsten Grzemba (cgrzemba) Date: 2020-10-30 07:37
Please continue support for Solaris/IllumOS! For build resources on the most recent Solaris platforms you can contact opencsw.org.
msg379933 - (view) Author: Paolo Marcheschi (marcheschi) Date: 2020-10-30 08:17
Please continue support for Solaris/IllumOS!
This is very important for us.
msg379937 - (view) Author: abarbu (abarbu) Date: 2020-10-30 08:44
Please don't drop Solaris support, we still use it to this day.
msg379938 - (view) Author: Predrag Zečević (predrag.zecevic.1961) Date: 2020-10-30 08:46
Please, keep support!
msg379942 - (view) Author: Jorge Schrauwen (jorge.schrauwen) Date: 2020-10-30 10:21
I'm an illumos user too, mostly SmartOS and some OmniOS.

I'm a heavy user of salt and thus phyton too, slowly prepping to get my stuff moved from 2.7 to 3.8 atm.

As mentioned above parts of the illumos build also use python and are moving to wads 3.9.

As mentioned I rarely use pip at all to install packages, nearly everything I need is available via IPS or pkgsrc depending on the distro.
msg379946 - (view) Author: Olaf Bohlen (olafbohlen) Date: 2020-10-30 10:51
Also, let me join in here. 
OpenIndiana hipster and all other illumos distributions will suffer if you drop the "Solaris" support. Just to get an idea which packages in a nearly default install depend on python:
pkg uninstall: Unable to remove 'runtime/python-35@3.5.9-2020.0.1.1' due to the following packages that depend on it:

  communication/im/pidgin@2.13.0-2020.0.1.4
  desktop/compiz/ccsm@0.8.16-2020.0.1.0
  desktop/time-slider@0.2.101-2020.0.1.116
  desktop/xdg/menu-editor/mozo@1.24.0-2020.0.1.0
  developer/build/onbld@0.5.11-2020.0.1.20038
  developer/clang-80@8.0.1-2020.0.1.0
  developer/ui-designer/glade@3.22.2-2020.0.1.0
  developer/versioning/mercurial-35@5.3.2-2020.0.1.0
  diagnostic/ddu@1.3.1-2020.0.1.40
  gnome/config/gconf@2.32.4-2020.0.1.8
  install/distribution-constructor@0.5.11-2020.0.1.1052
  metapackages/build-essential@1.0-2020.0.1.21
  package/pkg@0.5.11-2020.0.1.5543
  package/pkg/system-repository@0.5.11-2020.0.1.5543
  print/cups/system-config-printer@2.30.0-2020.0.1.3
  print/filter/hplip@3.19.12-2020.0.1.0
  release/os-welcome@1.0-2020.0.1.23
  service/gnome/desktop-cache@0.2.2-2020.0.1.7
  service/network/samba@4.12.5-2020.0.1.0
  system/file-system/zfs@0.5.11-2020.0.1.20038
  system/input-method/imf-startup@0.5.11-2020.0.1.3
  system/install@0.5.11-2020.0.1.1052
  system/network/avahi@0.7-2020.0.1.3
  system/network/avahi/gui@0.7-2020.0.1.3
  system/zones/brand/ipkg@0.5.11-2020.0.1.5543
  text/texinfo@6.7-2020.0.1.1
  web/server/apache-24/module/apache-wsgi-35@4.5.22-2020.0.1.5

Actually the package manager itself, installer, ZFS helpers, cups, the illumos build-system and and and all depend on python.
msg379955 - (view) Author: Juraj Lutter (otis) Date: 2020-10-30 12:40
We, in danubecloud (https://github.com/erigones/esdc-ce) are relying on Python, too. Keep up the good work, folks :-)
msg379956 - (view) Author: Mike La Spina (michael.laspina) Date: 2020-10-30 12:47
I suspect the download stats are not a full indication of usage on “SunOS” based on download stats. The bits are preinstalled. I do use it frequently via Illumos. Please support the platform.

Regards,
Mike
msg379959 - (view) Author: Jan Poctavek (yanchii) Date: 2020-10-30 12:57
I'm speaking officially for Danube Cloud, an advanced project which is open-source virtualization platform similar to Proxmox, XCP-NG, oVirt, Joyent Triton, etc. Our base platform is SmartOS and we have everything written in Python.

If you drop support for "Solaris", it will hurt our project a lot. Python has become heart of many projects and it should not limit itself to support just the mainstream platforms and OSes.

Thank you for everything you've done so far and I hope we can continue together.

Jan Poctavek

Danube Cloud
https://github.com/erigones/esdc-ce
msg379961 - (view) Author: Gaige Paulsen (gaige) Date: 2020-10-30 13:04
Let me add my voice to those who ask to keep Solaris support in Python going forward. We use SmartOS which is an Illumos and thus Solaris derivative.

We use Ansible extensively to automate our SmartOS deployments. We have a couple hundred VMs and 2 part-time volunteers on staff. We are extremely reliant on Ansible, and thus Python.

I have multiple blogs and a couple of django-based web sites that are running on SmartOS and we run a wide variety of tools that use Python as part of their mix.
msg379962 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-30 13:06
It seems like two Python core developers (Ronald Oussoren and Jesús Cea Avión) are volunteer to maintain the Solaris support. Moreover, it seems like some people would like to provide a Solaris server to run a buildbot server. In this case, I would be fine to keep the Solaris code.

But it's not enough, please call for help:
https://mail.python.org/archives/list/python-dev@python.org/message/NYWF4DUPFYZSKUUZ3ZHOSBQCWAF4IIQO/
msg379963 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-30 13:06
> But it's not enough, please call for help:

Please *read* my call for help
msg379991 - (view) Author: cyrus torros (sugondesenuts007) Date: 2020-10-30 18:36
Why?? I use python on solaris in my every day toolchain and at work. 

There is no harm in keeping it supported, it's not hard. 

Please do not drop support.
msg379992 - (view) Author: Jakub Kulik (kulikjak) * Date: 2020-10-30 18:51
Hi, here I am passing on several notes from the Oracle Solaris team:

- Oracle continues to develop and advance Oracle Solaris technology, and have committed to doing so for years to come. We release a new update (so called SRU) every month (https://blogs.oracle.com/solaris/oracle-solaris%3a-update-to-the-continuous-delivery-model).
- Python is essential for Oracle Solaris and actively worked on. In fact, we recently finished migration to Python 3.7 and integrated Python 3.9, which will be released to customers soon.
- Internally, we are performing daily testing for each Python version from 3.7+ and reporting any issues that are found* (fortunately, there are not many of them because Python works very well!).
- Python 3 is running for more than a year now on any machine running Oracle Solaris 11.4.
- Looking at six download statistics is very misleading because we are packaging Python and its modules in Solaris packages. This package is essential (meaning that every Oracle Solaris machine will have it installed). Also, we are caching archives internally, so unless customers download a newer version into a virtual environment, you won't see more than a few downloads from Oracle Solaris (which are then distributed onto thousands of machines running Oracle Solaris).
- We are looking at how to provide build bots running on Oracle Solaris.
- We are trying to be part of the Python community and also put back Solaris specific fixes (we provide patches for many of those issues reported).
- It is certainly much better (hopefully for everyone) to resolve any specific Solaris issues and help with the maintenance within the Python community than do this just internally as part of Oracle Solaris development.

*) everything reported by me are Solaris related issues

Jakub
msg379993 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-10-30 18:57
I think the PR needs to be withdrawn.  Users have spoken.  There is zero need to cut them off.
msg379994 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-30 19:08
Discussions are happening in this thread, including the requirements for supported platforms going forward:

https://mail.python.org/archives/list/python-dev@python.org/thread/VDD7NMEDFXMOP4S74GEYJUHJRJPK2UR3/
msg380006 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-10-30 22:24
Dear Solaris Python fans (about 20 so far here): Here is the situation.  There are 1000s of open issues on this tracker and at least 100s of open cpython PRs, and only 20-30 core developers, mostly volunteers, actively (depending on definition) merging PRs and maybe another 10 triagers helping to manage issues.

You all can help on issues by checking whether reported bugs still exist with current Python (3.8+) on current 'Solaris' (which includes what?).  Searching open issues for 'Solaris' in 'All text' just returned 114 hits.  About half were last touched over two years ago, and sometimes the last touch was inconsequential (a version or nosy change).  I suspect many of the 114 are obsolete.  For example, the last comment, five years ago, on #1471934, says the problem was fixed in Solaris 11.2.  Does this mean the issue should be closed?

The bottleneck for merging PRs is reviewing PRs.  We coredevs cannot do enough reviews ourselves.  But any competent user can  help.  Reviewing has two components.  First, does the patch fix the problem?  Testing this requires a Github account and a local clone and ability to build a test binary.  See devguide.python.org for more.  Second, does the patch meet our standards of code quality.  Solaris-specific patches likely change the C part of cpython, so C competence and understanding of PEP 7 is needed here.
msg380008 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-30 22:30
Jakub: Would you mind to build the master branch of Python, run the test suite and post the tests results? In short:

git clone https://github.com/python/cpython.git
cd cpython
./configure
./python -m test -j0 -r

If possible, try to install dependencies (readline, lzma, etc.).

If there are failures, please copy them into a file, and attach the file to this issue.

It is to have an idea if it would be worth it to add a Python buildbot running on Solaris. If there are too many failures, it would be better to fix first most failures before considering to add a buildbot.
msg380080 - (view) Author: Apostolos Syropoulos (ijdt.editor) Date: 2020-10-31 17:02
I think this is really a bad idea. I am using OpenIndiana in my classrooms and I teach Python! So if your drop Solaris support, it means I have to install a new supported OS on my many systems, to learn and teach many things, etc, This will make my life extremely difficult.
msg380196 - (view) Author: Jakub Kulik (kulikjak) * Date: 2020-11-02 08:18
I ran the test and the results are attached (the first one is a complete test and the other one failed tests in verbose mode).

I checked the failing tests and divided them into several groups:

issues already reported:
  test_asyncio - reported and being solved here: https://bugs.python.org/issue38323
  test_posix - problem with return values, reported here: https://bugs.python.org/issue41839
  test_shutil - sendfile for Solaris was disabled in the library, but not in the test suite, which leads to the issues (I asked about reenabling here: https://bugs.python.org/issue41843; that would solve the issue)

these failures are probably related to our internal network settings (false positives):
  test_ssl, test_urllib, test_urllib2, test_urllib2_localnet

other:
  test_float - locale related *
  test_locale - locale related *
  test_re - most likely locale related *
  test_socket - known issues not yet reported upstream (not ready for acceptable PR)
  test_time - one locale related * and other not yet known
  test_tcl - yet to investigate

*) Locale related failures are due to wchar_t differences between the Linux world and ours. It's something we resolved very recently and have yet to rework it into an upstreamable form and report it.

It would be interesting to see Illumos results as well, because while also Solaris, there might be differences due to almost ten years of spit development.
msg380203 - (view) Author: John Gardner (jgardner100) Date: 2020-11-02 10:02
Don't remove, it is still actively used by so many people. If the maintance overhead of 700 lines is a large burdern, then I'm happy to take on whatever work is required for it.
msg380205 - (view) Author: Andy Fiddaman (omnios) Date: 2020-11-02 10:55
Jakub's results looks very familiar to me, having been working on python 3.9 on illumos.

For OmniOS, we currently skip these tests via --ignorefile:

# wchar_t related failures
*.test_re.ReTests.test_locale_compiled
*.test_re.ReTests.test_locale_caching
# illumos does not support multiple SCM_RIGHTS messages in a packet
*FDPassSeparate*
# These tests fail on illumos and the first consumes a significant
# amount of memory. Investigation required.
test.test_socket.SendfileUsingSendfileTest.testCount
test.test_socket.SendfileUsingSendfileTest.testWithTimeout
test.test_socket.SendfileUsingSendfileTest.testOffset
#
test.test_asyncio.test_sendfile.*
test.test_asyncio.test_subprocess.SubprocessMultiLoopWatcherTests.*


We are also carrying some local patches for the following. Several come from Solaris.

- Stop python detecting and using epoll() on illumos

- scheduling priorities can be < 0
https://github.com/citrus-it/omnios-build/blob/python39/build/python39/patches/mod-posix-sched_priority.patch

- differences in sendfile behaviour (improves the situation but is not complete)
https://github.com/citrus-it/omnios-build/blob/python39/build/python39/patches/mod-posix-sendfile.patch

- Enable sendfile for shutil.copy (mismatch between library and testsuite in terms of whether sendfile() is enabled on illumos)
https://github.com/citrus-it/omnios-build/blob/python39/build/python39/patches/mod-shutil-sendfile.patch

- Fixes for building the socket module (_XOPEN_SOURCE=600)
https://github.com/citrus-it/omnios-build/blob/python39/build/python39/patches/mod-socket-xpg6.patch

- Emulate clock_gettime(CLOCK_THREAD_CPUTIME_ID)
https://github.com/citrus-it/omnios-build/blob/python39/build/python39/patches/mod-time-threadtime.patch

- PTY patch
https://github.com/citrus-it/omnios-build/blob/python39/build/python39/patches/pty.patch

and a few others, not all of which are suitable for upstream.
https://github.com/citrus-it/omnios-build/tree/python39/build/python39/patches


With all of this in place, the headline test stats for OmniOS Python 3.9 are:

401 tests OK.

24 tests skipped:
    test_dbm_gnu test_epoll test_gdb test_idle test_kqueue test_msilib
    test_ossaudiodev test_smtpnet test_socketserver test_startfile
    test_tcl test_timeout test_tix test_tk test_ttk_guionly
    test_ttk_textonly test_turtle test_urllib2net test_urllibnet
    test_winconsoleio test_winreg test_winsound test_xmlrpc_net
    test_zipfile64

Tests result: SUCCESS

and, additionally, the dtrace tests succeed (we test them separately as they require elevated privileges).
msg380210 - (view) Author: Jakub Kulik (kulikjak) * Date: 2020-11-02 11:47
Thanks Andy, good to know we are seeing the same issue!

We tried to fix sendfile differences in Python code before as well, but as you said, it was never 100% solved. Recently we finally fixed it in C and it was accepted (https://bugs.python.org/issue41687) and backported to 3.9 so you might/should be ok without the patch as well.

CLOCK_THREAD_CPUTIME_ID was issue for us as well but before the patch was accepted upstream, Oracle Solaris implemented it so it was no longer necessary. But the issue is still open https://bugs.python.org/issue35455 and knowing that others will use it, I will dust it off and finish it.

Knowing that Oracle and Illumos are facing the same issues, I am much more confident that our patches won't break Illumos. I will start upstream more of them right away.
msg380241 - (view) Author: Gordon Ross (gwr) Date: 2020-11-02 17:45
I can understand the frustrations around dealing with minority platforms, but please remember that the illumos project (www.illumos.org) is basically the inheritor of problems around "Build on Solaris" for 3rd party software like Python.  There are several OS distributions based on illumos that would be impacted by removing the ability to (easily) build Python. I'm confident we can find some people to help maintain build-bots etc. if that's what it takes.  If you're still looking for such help, let me know and I'll ask on developers@list.illumos.org for volunteers.

Thanks!  -GWR
msg380256 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-11-02 23:29
People interested in helping Solaris issues should see my post above, msg380006.

Issue #35455, about thread time on Solaris, with a simple but non-trivial patch, was just closed as fixed after reviews from 2 coredevs and a 3rd person.

I presume Victor will close his proposed PR when there is a buildbot and additional action on issues.  We have gotten new information that supports doing so.
msg380257 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-11-02 23:44
Victor, are you going to close the PR?
msg380455 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-06 16:32
Copy of my email sent to python-dev.

https://mail.python.org/archives/list/python-dev@python.org/message/46UUJ4J5YLCWF2JQMC5L7OIYG6PNQLRL/

Since I created the issue and the PR, and sent this email to
python-dev (one week ago), many Solaris and Solaris-like (ex: Illumos)
users replied that the operating system is definitively alive. I
didn't know that Oracle still ships new Solaris updates every month:
that's a good thing!

But this is not enough to support a platform. We would need proactive
contributors to fix known Solaris issues, but also fix new Solaris
issues (either regressions, or bugs newly discovered). We would also
need a buildbot to run the Python test suite on Solaris (or again, a
Solaris-like OS).

The good news is that Jakub Kulik started to fix some Solaris issues.
I understood that Solaris and Solaris-like operating systems do have
downstream patches on Python to fix a bunch of bugs. It seems like
some people want to push these fixes to Python upstream which is also
a good sign.

The other problem that I wanted to discuss is that fixing Solaris
issues require core devs (who merge PRs) accessing Solaris. If
contributors send patches and some core devs are fine with merging
fixes without being able to test them manually, I'm also fine with
that.

My first intent was to remove support for a definitely dead operating
system, but it seems like I was completely wrong (it's alive!). Thanks
to people starting to fix Solaris issues, I close my PR and I no
longer plan to drop Solaris support. I prefer to leave bpo-42173 open
for now, since people decide to use it as a place to collaborate on
fixing Solaris issues.

Once most tests will pass on the master branch, I also hope that
someone will set up a buildbot *and* fix issues discovered by this
buildbot. Sorry but just setting up a buildbot doesn't solve any
problem, it only increases the maintenance burden for people who
maintain the buildbot fleet. For example, we have two AIX buildbots, I
report bugs on bugs.python.org, but it seems like nobody is available
to fix them...

Overall, I'm quite happy with what is happening with Solaris! More
collaboration, issues being fixed in Python upstream. I just hope that
this work will continue next months. ;-)
msg380456 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-06 16:33
This issue title is "Drop Solaris support". I no longer plan to drop Solaris support. I suggest to open new issues and continue the discussion there.
msg380457 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-06 16:37
Gaige B Paulsen offered to set up a SmartOS (Illumos-derived) buildbot:
https://mail.python.org/archives/list/python-buildbots@python.org/thread/7HGPHD3WZG6V6NVP6EGQO6NHZ3DAPPXN/
msg381122 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-16 17:05
I close the issue. Thanks for the constructive discussion and I hope that more and more downstream patches will be merged in Python upstream ;-)
History
Date User Action Args
2020-11-16 17:05:24vstinnersetstatus: open -> closed
resolution: not a bug
messages: + msg381122

stage: patch review -> resolved
2020-11-06 16:37:59vstinnersetmessages: + msg380457
2020-11-06 16:33:22vstinnersetmessages: + msg380456
2020-11-06 16:32:11vstinnersetmessages: + msg380455
2020-11-05 16:21:06jstasiaksetnosy: + jstasiak
2020-11-02 23:44:51rhettingersetmessages: + msg380257
2020-11-02 23:29:10terry.reedysetmessages: + msg380256
2020-11-02 17:45:19gwrsetnosy: + gwr
messages: + msg380241
2020-11-02 11:47:08kulikjaksetmessages: + msg380210
2020-11-02 10:55:38omniossetmessages: + msg380205
2020-11-02 10:02:44jgardner100setnosy: + jgardner100
messages: + msg380203
2020-11-02 08:18:43kulikjaksetfiles: + Oracle_Solaris_detailed_test.txt
2020-11-02 08:18:33kulikjaksetfiles: + Oracle_Solaris_full_test.txt

messages: + msg380196
2020-11-01 14:56:12jomasoftmarcelsetnosy: + jomasoftmarcel
2020-10-31 17:02:52ijdt.editorsetnosy: + ijdt.editor
messages: + msg380080
2020-10-30 22:30:47vstinnersetmessages: + msg380008
2020-10-30 22:24:14terry.reedysetnosy: + terry.reedy
messages: + msg380006
2020-10-30 19:08:52pablogsalsetmessages: + msg379994
2020-10-30 18:57:38rhettingersetnosy: + rhettinger
messages: + msg379993
2020-10-30 18:51:23kulikjaksetnosy: + kulikjak, - sugondesenuts007
messages: + msg379992
2020-10-30 18:36:59sugondesenuts007setnosy: + sugondesenuts007
messages: + msg379991
2020-10-30 13:06:52vstinnersetmessages: + msg379963
2020-10-30 13:06:27vstinnersetmessages: + msg379962
2020-10-30 13:04:53gaigesetnosy: + gaige
messages: + msg379961
2020-10-30 12:57:35yanchiisetnosy: + yanchii
messages: + msg379959
2020-10-30 12:47:06michael.laspinasetnosy: + michael.laspina
messages: + msg379956
2020-10-30 12:40:33otissetnosy: + otis
messages: + msg379955
2020-10-30 10:51:36olafbohlensetnosy: + olafbohlen
messages: + msg379946
2020-10-30 10:21:51jorge.schrauwensetnosy: + jorge.schrauwen
messages: + msg379942
2020-10-30 08:46:16predrag.zecevic.1961setnosy: + predrag.zecevic.1961

messages: + msg379938
versions: - Python 3.10
2020-10-30 08:44:02abarbusetnosy: + abarbu
messages: + msg379937
2020-10-30 08:17:59marcheschisetnosy: + marcheschi
messages: + msg379933
2020-10-30 07:37:20cgrzembasetnosy: + cgrzemba
messages: + msg379927
2020-10-30 07:33:41siepkessetnosy: + siepkes
messages: + msg379926
2020-10-30 07:28:01mariuspanasetnosy: + mariuspana
messages: + msg379925
2020-10-30 07:21:04tbalberssetnosy: + tbalbers
messages: + msg379924
2020-10-30 04:24:13jm650setnosy: + jm650
messages: + msg379915
2020-10-30 03:28:51robertdfrenchsetnosy: + robertdfrench
messages: + msg379910
2020-10-30 01:44:07brett3setnosy: + brett3
messages: + msg379906
2020-10-30 01:17:54alancsetnosy: + alanc
2020-10-30 00:47:00pfirmstosetnosy: + pfirmsto
messages: + msg379901
2020-10-29 23:39:32tropikhajma@gmail.comsetnosy: + tropikhajma@gmail.com
messages: + msg379900
2020-10-29 23:06:31omniossetmessages: + msg379899
2020-10-29 22:59:14omniossetnosy: + omnios
messages: + msg379897
2020-10-29 22:48:27jclulowsetnosy: + jclulow
messages: + msg379895
2020-10-29 22:45:29jceasetmessages: + msg379894
2020-10-29 22:38:57jceasetmessages: + msg379893
2020-10-29 22:36:38mimi.vxsetnosy: + mimi.vx
messages: + msg379891
2020-10-29 22:22:17wiedisetmessages: + msg379890
2020-10-29 22:14:26vstinnersetmessages: + msg379888
2020-10-29 22:13:29vstinnersetmessages: + msg379887
2020-10-29 22:13:07wiedisetnosy: + wiedi
messages: + msg379886
2020-10-28 23:46:51Yonatan Goldschmidtsetnosy: + Yonatan Goldschmidt
2020-10-27 16:02:24vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request21917
2020-10-27 16:01:47BTaskayasetnosy: + BTaskaya
2020-10-27 15:59:08vstinnercreate