classification
Title: possible free statically allocated string in compiler when easter egg on
Type: Stage: resolved
Components: Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: barry, serhiy.storchaka, vstinner, xiang.zhang
Priority: normal Keywords: easy, patch

Created on 2018-07-10 11:37 by xiang.zhang, last changed 2018-07-24 08:09 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 8242 closed vstinner, 2018-07-11 09:24
PR 8259 closed vstinner, 2018-07-11 21:22
PR 8262 merged serhiy.storchaka, 2018-07-12 07:15
PR 8423 merged miss-islington, 2018-07-23 20:41
PR 8424 merged miss-islington, 2018-07-23 20:42
Messages (13)
msg321381 - (view) Author: Xiang Zhang (xiang.zhang) * (Python committer) Date: 2018-07-10 11:37
While reviewing PR8222, I found `err_ret->text` is assigned a not malloced string, but it will finally get freed in `err_input`.
msg321382 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-10 11:41
I expected it should be easy to reproduce a crash, but I failed to find an example.
msg321418 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-11 09:24
This issue is related to bpo-34080.
msg321420 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-11 09:33
> I expected it should be easy to reproduce a crash, but I failed to find an example.

I don't understand how parsetok.c is supposed to handle "from __future__ import barry_as_FLUFL":

* Parser/parsetok.c tests (ps->p_flags & CO_FUTURE_BARRY_AS_BDFL)
* Python/future.c uses "if (strcmp(feature, FUTURE_BARRY_AS_BDFL) == 0) { ff->ff_features |= CO_FUTURE_BARRY_AS_BDFL; }"
* Python/pythonrun.c: PARSER_FLAGS() copies CO_FUTURE_BARRY_AS_BDFL flag as PyPARSE_BARRY_AS_BDFL

The only way to use "<>" operator seems to pass explicitly CO_FUTURE_BARRY_AS_BDFL flag to compile() flags, which is not the convenient way to use "from __future__ import barry_as_FLUFL" :-(

Code:
---
import inspect
CO_FUTURE_BARRY_AS_BDFL = 0x40000
flags = CO_FUTURE_BARRY_AS_BDFL
code = "print(1 <> 2)"
try:
    compile(code, "filename", "exec", flags=0)
except SyntaxError:
    print("ok")
else:
    raise Exception("SyntaxError expected")
compile(code, "filename", "exec", flags=flags)
print("ok")
---

Maybe we need a test for the easter egg. Or maybe we should remove it? :-p
msg321421 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-11 09:43
Oh, this easter egg is tested by Lib/test/test_flufl.py.
msg321425 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-11 09:48
Could you add a test in test_flufl.py that will fail with not patched code?

This issue and issue34080 look unrelated to me. They can be fixed independently.
msg321429 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-11 10:13
> Could you add a test in test_flufl.py that will fail with not patched code?

It seems like PyObject_FREE() is never called with the static string "with Barry as BDFL, use '<>' instead of '!='".

When parsetok() goes to code path (1):

                err_ret->text = "with Barry as BDFL, use '<>' "
                                "instead of '!='";

Later, it goes to code path (2) as well:

        if (tok->buf != NULL) {
            ...
            err_ret->text = (char *) PyObject_MALLOC(len + 1);

Hum, I modified my PR to removed *dead code*:

                err_ret->text = "with Barry as BDFL, use '<>' "
                                "instead of '!='";

> This issue and issue34080 look unrelated to me. They can be fixed independently.

In practice, both issues are related and it seems easier to me to fix them both at the same time ;-)
msg321487 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2018-07-11 17:05
AFAICT, that future only works in the REPL.
msg321522 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-12 07:22
PR 8262 fixes the intended setting of error message for the "Barry as BDFL" easter egg.

>>> from __future__ import barry_as_FLUFL
>>> 1 != 2
  File "<stdin>", line 1
    1 != 2
       ^
SyntaxError: with Barry as BDFL, use '<>' instead of '!='

But there is other problem. As was exposed in the private communication with Victor and Barry, this easter egg works only in REPL (or if explicitly pass __future__.CO_FUTURE_BARRY_AS_BDFL to compile()). I'll try to fix this too.
msg321525 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-12 08:03
The problem is that 'from __future__ import' is parsed after converting the sources to the AST. But this flag affects the behavior of the tokenizer, before an AST is created.
msg322256 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-23 20:41
New changeset aba24ff3601ddc86b85e01880a8be596fb799287 by Serhiy Storchaka in branch 'master':
bpo-34084: Fix setting an error message for the "Barry as BDFL" easter egg. (GH-8262)
https://github.com/python/cpython/commit/aba24ff3601ddc86b85e01880a8be596fb799287
msg322270 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-23 22:41
New changeset 220afffb683af1ba9e04c37535cfaa41348e9ebb by Serhiy Storchaka (Miss Islington (bot)) in branch '3.7':
bpo-34084: Fix setting an error message for the "Barry as BDFL" easter egg. (GH-8262) (GH-8423)
https://github.com/python/cpython/commit/220afffb683af1ba9e04c37535cfaa41348e9ebb
msg322271 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-23 22:41
New changeset ec729d5407adafaae566136448ef0551bb29c070 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-34084: Fix setting an error message for the "Barry as BDFL" easter egg. (GH-8262) (GH-8424)
https://github.com/python/cpython/commit/ec729d5407adafaae566136448ef0551bb29c070
History
Date User Action Args
2018-07-24 08:09:32serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-07-23 22:41:51serhiy.storchakasetmessages: + msg322271
2018-07-23 22:41:31serhiy.storchakasetmessages: + msg322270
2018-07-23 20:42:20miss-islingtonsetpull_requests: + pull_request7950
2018-07-23 20:41:20miss-islingtonsetpull_requests: + pull_request7949
2018-07-23 20:41:13serhiy.storchakasetmessages: + msg322256
2018-07-12 08:03:34serhiy.storchakasetmessages: + msg321525
2018-07-12 07:22:31serhiy.storchakasetmessages: + msg321522
2018-07-12 07:15:13serhiy.storchakasetpull_requests: + pull_request7795
2018-07-11 21:22:04vstinnersetpull_requests: + pull_request7792
2018-07-11 17:05:29barrysetmessages: + msg321487
2018-07-11 10:13:52vstinnersetmessages: + msg321429
2018-07-11 09:48:56serhiy.storchakasetmessages: + msg321425
2018-07-11 09:43:55vstinnersetmessages: + msg321421
2018-07-11 09:33:44vstinnersetmessages: + msg321420
2018-07-11 09:24:54vstinnersetmessages: + msg321418
2018-07-11 09:24:29vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request7775
2018-07-11 01:59:39xiang.zhangsetnosy: + vstinner
2018-07-10 11:41:30serhiy.storchakasetnosy: + barry
2018-07-10 11:41:14serhiy.storchakasetmessages: + msg321382
2018-07-10 11:38:12xiang.zhangsettitle: possible free statically allocated string -> possible free statically allocated string in compiler when easter egg on
2018-07-10 11:37:01xiang.zhangcreate