classification
Title: Fatal Python error: _PySys_BeginInit: can't initialize sys module
Type: compile error Stage: resolved
Components: Build Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Tasy, vstinner, xtreak
Priority: normal Keywords:

Created on 2019-01-10 22:35 by Tasy, last changed 2019-01-22 17:43 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11647 merged vstinner, 2019-01-22 15:51
PR 11647 merged vstinner, 2019-01-22 15:51
PR 11647 merged vstinner, 2019-01-22 15:51
PR 11650 open vstinner, 2019-01-22 17:43
PR 11650 open vstinner, 2019-01-22 17:43
PR 11650 open vstinner, 2019-01-22 17:43
Messages (11)
msg333423 - (view) Author: (Tasy) Date: 2019-01-10 22:35
.
.
.
./python -E -S -m sysconfig --generate-posix-vars ;\
	if test $? -ne 0 ; then \
		echo "generate-posix-vars failed" ; \
		rm -f ./pybuilddir.txt ; \
		exit 1 ; \
	fi
Fatal Python error: _PySys_BeginInit: can't initialize sys module

Current thread 0x00002b4e5f9bf400 (most recent call first):
Aborted (core dumped)
generate-posix-vars failed
make[1]: *** [pybuilddir.txt] Error 1
make[1]: Leaving directory `/usr/local/mysoftware/Python-3.7.2/build'
make: *** [profile-opt] Error 2
msg333425 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-10 22:48
> .
> .
> .

Hello. Can you elaborate this part?
msg333427 - (view) Author: (Tasy) Date: 2019-01-10 23:38
Sorry for the confusion. The three dots were for many lines of successful compilation output from make. The rest is the final few lines where the compilation fails.
msg333428 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-10 23:42
Because it's late, I will say it shortly: if you don't elaborate how you get this error, I will simply close the issue. You have to describe what you are trying to do, your OS, etc.

(It works for me!)
msg333429 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-01-10 23:45
Please attach the full build log too along with the command executed.
msg333733 - (view) Author: (Tasy) Date: 2019-01-15 22:03
Configuration Options: 

 ../configure --prefix=$HOME --enable-shared --enable-optimizations --with-system-expat --with-system-ffi --with-ensurepip=yes



Make throws the following warning:


*** WARNING: renaming "_curses_panel" since importing it failed: No module named '_curses'

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_ssl                  _uuid                                    
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd                
time                                                           


Failed to build these modules:
_curses                                                        


Following modules built successfully but were removed because they could not be imported:
_curses_panel                                                  


Could not build the ssl module!
Python requires an OpenSSL 1.0.2 or 1.1 compatible libssl with X509_VERIFY_PARAM_set1_host().
LibreSSL 2.6.4 and earlier do not provide the necessary APIs, https://github.com/libressl-portable/portable/issues/381

There ther is a following error...

0:06:18 load avg: 0.55 [171/416] test_hashlib
*** Error in `./python': corrupted size vs. prev_size: 0x000000000276b7a0 ***
Fatal Python error: Aborted

Current thread 0x00002ba4468c7bc0 (most recent call first):
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/test_hashlib.py", line 904 in _test_pbkdf2_hmac
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/test_hashlib.py", line 935 in test_pbkdf2_hmac_c
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/case.py", line 615 in run
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/case.py", line 663 in __call__
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 122 in run
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 122 in run
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 122 in run
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/support/testresult.py", line 162 in run
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/support/__init__.py", line 1895 in _run_suite
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/support/__init__.py", line 1991 in run_unittest
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/runtest.py", line 178 in test_runner
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/runtest.py", line 182 in runtest_inner
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/runtest.py", line 137 in runtest
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 407 in run_tests_sequential
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 514 in run_tests
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 615 in _main
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 582 in main
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/libregrtest/main.py", line 636 in main
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/regrtest.py", line 46 in _main
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/regrtest.py", line 50 in <module>
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/runpy.py", line 85 in _run_code
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/runpy.py", line 193 in _run_module_as_main
Aborted (core dumped)
make[1]: Leaving directory `/usr/local/data/mySoftware/Python-3.7.2/build'
make build_all_merge_profile
make[1]: Entering directory `/usr/local/data/mySoftware/Python-3.7.2/build'
true
make[1]: Leaving directory `/usr/local/data/mySoftware/Python-3.7.2/build'
# Remove profile generation binary since we are done with it.
make clean
make[1]: Entering directory `/usr/local/data/mySoftware/Python-3.7.2/build'
find .. -depth -name '__pycache__' -exec rm -rf {} ';'
find .. -name '*.py[co]' -exec rm -f {} ';'
find . -name '*.[oa]' -exec rm -f {} ';'
find . -name '*.s[ol]' -exec rm -f {} ';'
find . -name '*.so.[0-9]*.[0-9]*' -exec rm -f {} ';'
find build -name 'fficonfig.h' -exec rm -f {} ';' || true
find build -name '*.py' -exec rm -f {} ';' || true
find build -name '*.py[co]' -exec rm -f {} ';' || true
rm -f pybuilddir.txt
rm -f Lib/lib2to3/*Grammar*.pickle
rm -f Programs/_testembed Programs/_freeze_importlib
find build -type f -a ! -name '*.gc??' -exec rm -f {} ';'
rm -f Include/pydtrace_probes.h
rm -f profile-gen-stamp
make[1]: Leaving directory `/usr/local/data/mySoftware/Python-3.7.2/build'
# This is an expensive target to build and it does not have proper
# makefile dependency information.  So, we create a "stamp" file
# to record its completion and avoid re-running it.
touch profile-run-stamp
Rebuilding with profile guided optimizations:
rm -f profile-clean-stamp
make build_all CFLAGS_NODIST=" -fprofile-use -fprofile-correction" LDFLAGS_NODIST=""
make[1]: Entering directory `/usr/local/data/mySoftware/Python-3.7.2/build'

And then finally : 

 Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/zipimport.o  Modules/faulthandler.o  Modules/_tracemalloc.o Modules/hashtable.o  Modules/symtablemodule.o  Modules/xxsubtype.o Python/frozen.o
gcc -pthread -shared     -Wl,--no-as-needed -o libpython3.so -Wl,-hlibpython3.so libpython3.7m.so
gcc -pthread     -Xlinker -export-dynamic -o python Programs/python.o -L. -lpython3.7m -lpthread -ldl  -lutil   -lm  
gcc -pthread     -Xlinker -export-dynamic -o Programs/_testembed Programs/_testembed.o -L. -lpython3.7m -lpthread -ldl  -lutil   -lm  
LD_LIBRARY_PATH=/usr/local/data/mySoftware/Python-3.7.2/build:$HOME/lib:/usr/local/cuda/lib64:$HOME/lib/cuda/lib64:$HOME/lib/cuda/extras/CUPTI/lib64:/usr/local/data/lib:/usr/local/lib/:/usr/lib/ ./python -E -S -m sysconfig --generate-posix-vars ;\
	if test $? -ne 0 ; then \
		echo "generate-posix-vars failed" ; \
		rm -f ./pybuilddir.txt ; \
		exit 1 ; \
	fi
Fatal Python error: _PySys_BeginInit: can't initialize sys module

Current thread 0x00002adc936069c0 (most recent call first):
Aborted (core dumped)
generate-posix-vars failed
make[1]: *** [pybuilddir.txt] Error 1
make[1]: Leaving directory `/usr/local/data/mySoftware/Python-3.7.2/build'
make: *** [profile-opt] Error 2
msg333853 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-17 12:12
What is your OS (name and version)?

What is your compiler (name and version)?

> Fatal Python error: _PySys_BeginInit: can't initialize sys module

I have no idea why you get this error. You should try to run this function in a debugger like gdb and run the code step by step to see what happens.

"""

0:06:18 load avg: 0.55 [171/416] test_hashlib
*** Error in `./python': corrupted size vs. prev_size: 0x000000000276b7a0 ***
Fatal Python error: Aborted

Current thread 0x00002ba4468c7bc0 (most recent call first):
  File "/usr/local/data/mySoftware/Python-3.7.2/Lib/test/test_hashlib.py", line 904 in _test_pbkdf2_hmac
"""

That's maybe unrelated, but _test_pbkdf2_hmac() is not supposed to crash.
msg333984 - (view) Author: (Tasy) Date: 2019-01-18 15:51
Compiler:
$ gcc --version
gcc (Ubuntu 4.8.4-2ubuntu1~14.04.4) 4.8.4

OS:
$ uname -a
Linux machine 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Home directory is of type nfs in case that is relevant.
I'll try to do the gdb thing over the weekend.
msg333992 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-18 16:55
Hum. Are you aware that PGO with GCC is broken on such old and unsupported Ubuntu version?

My old note about that:

"PGO is broken on Ubuntu 14.04 LTS with GCC 4.8.4-2ubuntu1~14.04: Modules/socketmodule.c:7743:1: internal compiler error: in edge_badness, at ipa-inline.c:895"

https://pyperformance.readthedocs.io/usage.html#compile

See also https://bugs.python.org/issue31963

I suggest you to not use PGO compilation on old Ubuntu. Maybe upgrade to latest Ubuntu LTS?
msg333999 - (view) Author: (Tasy) Date: 2019-01-18 17:52
compiling without optimizations worked.
msg334219 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-22 16:39
New changeset bf4ac2d2fd520c61306b2676db488adab9b5d8c5 by Victor Stinner in branch 'master':
bpo-35713: Rework Python initialization (GH-11647)
https://github.com/python/cpython/commit/bf4ac2d2fd520c61306b2676db488adab9b5d8c5
History
Date User Action Args
2019-01-22 17:44:00vstinnersetpull_requests: + pull_request11437
2019-01-22 17:43:52vstinnersetpull_requests: + pull_request11436
2019-01-22 17:43:44vstinnersetpull_requests: + pull_request11435
2019-01-22 16:39:06vstinnersetmessages: + msg334219
2019-01-22 15:52:05vstinnersetpull_requests: + pull_request11432
2019-01-22 15:51:59vstinnersetpull_requests: + pull_request11431
2019-01-22 15:51:52vstinnersetpull_requests: + pull_request11430
2019-01-18 17:52:46Tasysetstatus: open -> closed
resolution: fixed
messages: + msg333999

stage: resolved
2019-01-18 16:55:55vstinnersetmessages: + msg333992
2019-01-18 15:51:09Tasysetmessages: + msg333984
2019-01-17 12:12:50vstinnersetmessages: + msg333853
2019-01-15 22:03:41Tasysetmessages: + msg333733
2019-01-10 23:45:02xtreaksetnosy: + xtreak
messages: + msg333429
2019-01-10 23:42:00vstinnersetmessages: + msg333428
2019-01-10 23:38:41Tasysetmessages: + msg333427
2019-01-10 22:48:50vstinnersetnosy: + vstinner
messages: + msg333425
2019-01-10 22:35:31Tasycreate