classification
Title: expat symbols should be namespaced in pyexpat again
Type: crash Stage: resolved
Components: Extension Modules Versions: Python 3.4, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Lukas.Vacek, Peter.Kruse, athompson, christian.heimes, fdrake, gregory.p.smith, python-dev
Priority: critical Keywords: patch

Created on 2013-10-07 13:04 by athompson, last changed 2014-05-15 08:01 by Peter.Kruse. This issue is now closed.

Files
File name Uploaded Description Edit
fix_expat_names.patch Lukas.Vacek, 2014-02-04 14:40
with-pyexpatns.txt Peter.Kruse, 2014-03-19 09:39
without-pyexpatns.txt Peter.Kruse, 2014-03-19 09:39
Messages (7)
msg199143 - (view) Author: AThompson (athompson) Date: 2013-10-07 13:04
The issue with confliciting expat versions has poped up again in python 3.
http://bugs.python.org/issue1295808

It looks like the fix for this was removed when upgrading to expat 2.1 in http://hg.python.org/cpython/rev/e4dc8be9a72f      on Sat, 14 Jul 2012 14:12:35 -0700 Gregory P. Smith
--- a/Modules/expat/expat_external.h
/* Namespace external symbols to allow multiple libexpat version to
-   co-exist. */
-#include "pyexpatns.h"

Same backtrace as before.
#0  0x0000003507481321 in __strlen_sse2 () from /lib64/libc.so.6
#1  0x00007facace1b0d0 in PyUnicode_FromString (u=0x0) at Objects/unicodeobject.c:1824
#2  0x00007facaceefddc in PyModule_AddStringConstant (m=0x7fac9c7cf678,
    name=0x7fac9c5ab2e3 "XML_ERROR_UNBOUND_PREFIX", value=0x0) at Python/modsupport.c:554
#3  0x00007fac9c582f31 in PyInit_pyexpat () at /home/hex/Downloads/Python-3.3.2/Modules/pyexpat.c:1870
#4  0x00007facacee8f89 in _PyImport_LoadDynamicModule (name=0x7fac9c7db450, path=0x7fac9cdf9928, fp=0x0)
msg210234 - (view) Author: Lukas Vacek (Lukas.Vacek) Date: 2014-02-04 14:40
attaching patch
msg210240 - (view) Author: Roundup Robot (python-dev) Date: 2014-02-04 15:12
New changeset c242a8f30806 by Benjamin Peterson in branch '3.3':
restore namespacing of pyexpat symbols (closes #19186)
http://hg.python.org/cpython/rev/c242a8f30806

New changeset a2d877fb53f6 by Benjamin Peterson in branch 'default':
merge 3.3 (#19186)
http://hg.python.org/cpython/rev/a2d877fb53f6
msg214069 - (view) Author: Peter Kruse (Peter.Kruse) Date: 2014-03-19 09:39
Hello,

it seems that the solution to this issue causes the failure to 
compile the pyexpat extension in my case.  If I do not include
pyexpatns.h in expat_external.h then the compile succeeeds.
I will attach the output for both cases.  There was no problem
with 3.4.0b3.

My environment:

Python 3.4.0
Red Hat Linux 6.4
GCC 4.8.2

it makes no difference if I use --with-system-expat or without.
msg218169 - (view) Author: Lukas Vacek (Lukas.Vacek) Date: 2014-05-09 15:02
Hi Peter,

Thanks for taking the time to report your issue. I just tried on fresh up-to-date CentOS 6 and I could not reproduce the issue.

However, I think I figured out what went wrong. I can see you are supplying custom -I and -L paths (-I/apps/prod/releases/3.0/include ...) I suspect that when compiling pyexpat.c (in Modules/) the header file expat.h is picked up from this location before the bundled expat.h (in Modules/expat/) is found. But when compiling xmlparse.c (in Modules/expat/) the bundled header file expat.h *is* used this time. So we end up with our bundled expat compiled using namespacing and so defining symbols like PyExpat_XML_SetCommentHandler while pyexpat.c is looking for XML_SetCommentHandler because it is compiled *not* using our bundled expat.h.

I am not sure yet why your custom "-I" comes before "-I/ae/data/soft/opensource/build/python3/master/Python-3.4.0/Modules/expat" as it certainly should not. 

So far I tried building with CPPFLAGS and CFLAGS and in both cases my custom paths come after "-I.../Modules/expat" as it should.

Can you please provide your ./configure and make command lines and relevant environment variables?

Thanks,
Lucas
msg218172 - (view) Author: Lukas Vacek (Lukas.Vacek) Date: 2014-05-09 15:55
Actually CFLAGS do indeed come before any include directories.

I can reproduce your problem:
mkdir /tmp/extra_include
cp /usr/include/expat.h /usr/include/expat_external.h /tmp/extra_include/
make CFLAGS="-I/tmp/extra_include"

You should use CPPFLAGS for extra include directories. Not an issue in CPython.
msg218598 - (view) Author: Peter Kruse (Peter.Kruse) Date: 2014-05-15 08:01
Fascinating, you are right, very good, thanks for your time and looking into this.

Peter
History
Date User Action Args
2014-05-15 08:01:28Peter.Krusesetmessages: + msg218598
2014-05-09 15:55:16Lukas.Vaceksetmessages: + msg218172
2014-05-09 15:02:31Lukas.Vaceksetmessages: + msg218169
2014-03-19 09:39:18Peter.Krusesetfiles: + without-pyexpatns.txt
2014-03-19 09:39:01Peter.Krusesetfiles: + with-pyexpatns.txt
nosy: + Peter.Kruse
messages: + msg214069

2014-02-04 15:12:26python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg210240

resolution: fixed
stage: needs patch -> resolved
2014-02-04 14:40:55Lukas.Vaceksetfiles: + fix_expat_names.patch

nosy: + Lukas.Vacek
messages: + msg210234

keywords: + patch
2013-10-08 08:45:59pitrousetpriority: normal -> critical
2013-10-07 13:45:58pitrousetnosy: + gregory.p.smith
2013-10-07 13:24:39christian.heimessetnosy: + fdrake, christian.heimes

stage: needs patch
2013-10-07 13:04:57athompsoncreate