classification
Title: Makefile is not correctly generated when compiling pyextat with DXML_POOR_ENTROPY=1
Type: compile error Stage: resolved
Components: Build Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ahurst, benjamin.peterson, mgmacias95, miss-islington, ned.deily
Priority: normal Keywords: patch

Created on 2018-11-07 16:47 by mgmacias95, last changed 2019-06-11 02:15 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 12900 closed ahurst, 2019-04-22 01:51
PR 13064 merged ahurst, 2019-05-02 18:03
PR 13960 merged miss-islington, 2019-06-11 01:54
PR 13961 merged miss-islington, 2019-06-11 01:54
Messages (10)
msg329424 - (view) Author: Marta Gómez (mgmacias95) Date: 2018-11-07 16:47
Hello,

I was getting this same error (https://bugs.python.org/issue35139) when compiling pyextat so I added these changes (https://github.com/python/cpython/pull/10291/files) to my Setup.dist. 

After running ./configure, the generated Makefile throws the following error:

$ make
Makefile:264: *** missing separator.  Stop.

I have tried this in both Python 3.7 and 3.6 and I get the error in both (the line numbers are different but the error is the same). I have tested this in both Ubuntu 16/18 and CentOS 7.

Any hints in how could I fix this?

Best regards,
Marta
msg340630 - (view) Author: Aaron Hurst (ahurst) * Date: 2019-04-22 01:56
I can reproduce this issue by uncommenting the pyexpat line in Setup.dist and compiling.

The issue is with -DXML_POOR_ENTROPY=1.  The equals character causes the line to be incorrectly interpreted as a macro definition by makesetup.  This results in an invalid Makefile output.

I've submitted a PR, but a quick work-around is to remove the "=1".  It is not necessary.
msg341279 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-05-02 15:53
Thanks for the report and thanks for the PR.  I am confused here as I cannot reproduce the failure.  Can someone show what is invalid in the Makefile?  Also, if the change really does need to be made to Setup.dist (3.7) or Setup (master -> 3.8), then setup.py should likely be changed to reduce future confusion (although it presumably doesn't make a difference to the compile).  Further, please submit PRs like this against the master branch (unless the problem only exists in a specific released branches); fixes are normally applied to the master branch and the core developer handling the merge will elect to backport to appropriate branches as needed.
msg341292 - (view) Author: Aaron Hurst (ahurst) * Date: 2019-05-02 18:00
Hi Ned,

From a fresh checkout of master on Ubuntu 18.04, I uncomment the pyexpat line in Modules/Setup and run:

cpython$ ./configure
...
cpython$ make
Makefile:273: *** missing separator.  Stop.

Here is the offending section of the resulting Makefile:

269 # === Definitions added by makesetup ===
270 
271 LOCALMODLIBS=                      
272 BASEMODLIBS=
273 pyexpat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DXML_$
274 PYTHONPATH=$(COREPYTHONPATH)
275 COREPYTHONPATH=$(DESTPATH)$(SITEPATH)$(TESTPATH)
msg341293 - (view) Author: Aaron Hurst (ahurst) * Date: 2019-05-02 18:04
Sorry for my misunderstanding of the process, and thanks for explaining.  I resubmitted the PR against the master branch.
msg341297 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-05-02 18:40
Thanks for the updates, Aaron.  The plot thickens!  If I perform the steps on a current macOS system, the result is:

273 pyexpat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DXML_POOR_ENTROPY=1 -DUSE_PYEXPAT_CAPI

which works just fine.  So it seems the underlying problem is actually a difference in the behavior of the makesetup shell script, most likely some difference in one of the utilities it uses, like sed, perhaps a BSD vs GNU difference.

I won't have more time to look into this further today but feel free to do so if you like.  In any case, we should fix the root problem in makesetup rather than trying to work around it elsewhere.
msg341304 - (view) Author: Aaron Hurst (ahurst) * Date: 2019-05-02 19:07
Hi Ned,

Thanks for testing this.  I also observe that macOS compiles "without error"... but it's still broken... and silently.

This is because the pyexpat line isn't being turned into the expected set of source compilation rules, but it is instead being dumped into the variable definition section.  Why is it being interpreted by makesetup as a variable definition?  With the equals character, it matches this pattern:

# Lines can also have the form
#
# <name> = <value>
#
# which defines a Make variable definition inserted into Makefile.in

But it is intended to match this pattern:

# Lines have the following structure:
#
# <module> ... [<sourcefile> ...] [<cpparg> ...] [<library> ...]

For reference, here is the corresponding rule in makesetup:

> *=*)    DEFS="$line$NL$DEFS"; continue;;

I fully support tweaking this pattern to better differentiate when "=" means a variable definition and when "=" is part of a compilation flag, but given that pyexpat is the only such case, my one-line fix makes things consistent again.  For now.
msg345174 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2019-06-11 01:54
New changeset 408a2ef1aceff1f4270c44552fa39ef93d9283e3 by Benjamin Peterson (aaronpaulhurst) in branch 'master':
closes bpo-35184: Fix XML_POOR_ENTROPY option that breaks makesetup parsing of pyexpat line in Setup. (GH-13064)
https://github.com/python/cpython/commit/408a2ef1aceff1f4270c44552fa39ef93d9283e3
msg345175 - (view) Author: miss-islington (miss-islington) Date: 2019-06-11 02:14
New changeset 5b94b857f590db80aab69c31f88dd2a4978f8329 by Miss Islington (bot) in branch '3.8':
closes bpo-35184: Fix XML_POOR_ENTROPY option that breaks makesetup parsing of pyexpat line in Setup. (GH-13064)
https://github.com/python/cpython/commit/5b94b857f590db80aab69c31f88dd2a4978f8329
msg345176 - (view) Author: miss-islington (miss-islington) Date: 2019-06-11 02:15
New changeset 30fd7a476bbd6bb8096c1349698463fa8a3bca18 by Miss Islington (bot) in branch '3.7':
closes bpo-35184: Fix XML_POOR_ENTROPY option that breaks makesetup parsing of pyexpat line in Setup. (GH-13064)
https://github.com/python/cpython/commit/30fd7a476bbd6bb8096c1349698463fa8a3bca18
History
Date User Action Args
2019-06-11 04:19:20benjamin.petersonlinkissue35586 superseder
2019-06-11 02:15:27miss-islingtonsetmessages: + msg345176
2019-06-11 02:14:37miss-islingtonsetnosy: + miss-islington
messages: + msg345175
2019-06-11 01:54:44miss-islingtonsetpull_requests: + pull_request13828
2019-06-11 01:54:37miss-islingtonsetpull_requests: + pull_request13827
2019-06-11 01:54:27benjamin.petersonsetstatus: open -> closed

nosy: + benjamin.peterson
messages: + msg345174

resolution: fixed
stage: patch review -> resolved
2019-05-02 19:07:27ahurstsetmessages: + msg341304
2019-05-02 18:40:18ned.deilysetmessages: + msg341297
2019-05-02 18:04:48ahurstsetmessages: + msg341293
2019-05-02 18:03:32ahurstsetpull_requests: + pull_request12981
2019-05-02 18:00:01ahurstsetmessages: + msg341292
2019-05-02 15:53:55ned.deilysetnosy: + ned.deily

messages: + msg341279
versions: + Python 3.8, - Python 3.6
2019-04-22 01:56:08ahurstsetnosy: + ahurst
messages: + msg340630
2019-04-22 01:51:55ahurstsetkeywords: + patch
stage: patch review
pull_requests: + pull_request12827
2018-11-07 16:47:20mgmacias95create