This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Configparser fails when the .cfg file contains inline 'comments'
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: DiddiLeija, lukasz.langa, serhiy.storchaka, terry.reedy, uranusjr, zach.ware
Priority: normal Keywords:

Created on 2021-08-11 20:32 by DiddiLeija, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (10)
msg399415 - (view) Author: Diego Ramirez (DiddiLeija) Date: 2021-08-11 20:32
On the Pip GitHub issue tracker (https://github.com/pypa/pip/issues/10348), a user reported a strange behaviour when using a config file (setup.cfg) on its project.

The config file had a percentage character ("%") inside a commentary. But the module "configparser" failed with traceback:

    configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: "%' in string formatting

We consider that the character was badly recognized as a part of the file, when it was just a part of an inline comment.

Is there any way to fix this bug?
msg399562 - (view) Author: Diego Ramirez (DiddiLeija) Date: 2021-08-13 20:59
Lukasz Langa, I would like to know your opinion, as you are recognized as the "configparser" developer.
msg399580 - (view) Author: Diego Ramirez (DiddiLeija) Date: 2021-08-14 02:08
I would like to give a better explanation of the issue (maybe the previous explanation was unclear). A user reported to Pip that, when he used a "setup.cfg" file, the configparser module crashed (as I said above).

The file contained a percentage character (%) inside a commentary (which is expected to be ignored):

    [flake8]
    ignore =
            WPS323      #  percentage sign '%'

And, because pip uses configparser on those cases, any operation with pip fails.

So, I would like to know if we can do something to fix this bug (I noticed the place where it fails at https://github.com/python/cpython/blob/main/Lib/configparser.py#L442). If I can fill a PR to help, just tell me, OK?
msg401081 - (view) Author: Diego Ramirez (DiddiLeija) Date: 2021-09-05 12:38
Any commentaries about this issue?
msg401207 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-09-07 05:15
Diego, I repaired the defacement, which included removing you as nosy.

Relevant doc: https://docs.python.org/3/library/configparser.html#configparser.BasicInterpolation

To me, the following line implies that % may be used freely in comments.
  gain: 80%%  # use a %% to escape the % sign (% is the only character ...
However, IDLE uses .cfg files and configparser, and when I added #% to my C:/Users/Terry/.idlerc/config-main.cfg, IDLE startup fails with the following, confirming the behavior.

Traceback (most recent call last):
  File "f:\dev\3x\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\__main__.py", line 7, in <module>
    idlelib.pyshell.main()
    ^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\pyshell.py", line 1648, in main
    shell = flist.open_shell()
            ^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\pyshell.py", line 335, in open_shell
    self.pyshell = PyShell(self)
                   ^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\pyshell.py", line 898, in __init__
    OutputWindow.__init__(self, flist, None, None)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\outwin.py", line 79, in __init__
    EditorWindow.__init__(self, *args)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\editor.py", line 215, in __init__
    text['font'] = idleConf.GetFont(self.root, 'main', 'EditorWindow')
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\config.py", line 742, in GetFont
    bold = self.GetOption(configType, section, 'font-bold', default=0,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\config.py", line 229, in GetOption
    return self.userCfg[configType].Get(section, option,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\idlelib\config.py", line 61, in Get
    return self.getboolean(section, option)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\configparser.py", line 829, in getboolean
    return self._get_conv(section, option, self._convert_to_boolean,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\configparser.py", line 809, in _get_conv
    return self._get(section, conv, option, raw=raw, vars=vars,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\configparser.py", line 804, in _get
    return conv(self.get(section, option, **kwargs))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\configparser.py", line 800, in get
    return self._interpolation.before_get(self, section, option, value,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\configparser.py", line 395, in before_get
    self._interpolate_some(parser, option, L, value, section, defaults, 1)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "f:\dev\3x\lib\configparser.py", line 442, in _interpolate_some
    raise InterpolationSyntaxError(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '%'

When changed one of the lines in my file see what a valid interpolation would do,
   font-size = 12  # for %(font)s
I got

"Warning: config.py - IdleConf.GetOption -
 invalid 'int' value for configuration option 'font-size'
 from section 'EditorWindow': '12  # for source code pro'"

configparser interpreted the comment as part of the value, and started IDLE with the default size 10.
The same happened when I used %%: the bad value was '12  # for %'
And again with just '#'.  Values read as string are also disabled by '#'.
"name = Custom Dark # %(parenstyle)s" resulted in the default light theme.
Conclusion: 1. Comments are currently worse than useless. 2. Test suite is deficient.

Diego, good PRs are welcome.  Please sign CLA first.  For this, the problem is comments not being recognized and stripped.  Fix that and % in comments will not be an issue.
msg401209 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-09-07 05:41
Make PR branch from main, which I used for testing. I verified issue on 3.10 and 3.9, so we will backport.
msg401211 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-07 06:01
By default configparser does not support inline comments. "#  percentage sign '%'" is a part of value. If you want to support inline comments you should pass the inline_comment_prefixes argument.

But note that it can break parsing existing valid config files containing such prefixes. For example "background=#ffffff" or "path=C:\Python\bin;D:\Users\Me\bin" if use standard comment prefixes "#" and ";". This is why it is disabled by default.
msg402621 - (view) Author: Diego Ramirez (DiddiLeija) Date: 2021-09-25 13:37
Hi Terry, I didn't see your response. I think this won't be possible, taking in count the comment that Serhiy posted:

> By default configparser does not support inline comments. "#  percentage sign '%'" is a part of value. If you want to support inline comments you should pass the inline_comment_prefixes argument.

Maybe this is a reasonable behavior. What do you think about it?
msg402625 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-09-25 14:24
Please report back on the pip issue to read more of the doc, about inline comments in
https://docs.python.org/3/library/configparser.html#customizing-parser-behaviour
like I should have before writing what I did.
msg402635 - (view) Author: Diego Ramirez (DiddiLeija) Date: 2021-09-25 16:46
Sure, thanks!
History
Date User Action Args
2022-04-11 14:59:48adminsetgithub: 89055
2021-09-25 16:46:34DiddiLeijasetmessages: + msg402635
2021-09-25 14:24:42terry.reedysettitle: Configparser fails when the .cfg file contains comments -> Configparser fails when the .cfg file contains inline 'comments'
2021-09-25 14:24:07terry.reedysetstatus: open -> closed
resolution: not a bug
messages: + msg402625

stage: test needed -> resolved
2021-09-25 13:37:12DiddiLeijasetmessages: + msg402621
2021-09-07 06:01:03serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg401211
2021-09-07 05:41:25terry.reedysettitle: Configparser fails when the file contains $ inside a comment -> Configparser fails when the .cfg file contains comments
stage: needs patch -> test needed
messages: + msg401209
versions: + Python 3.9, Python 3.10
2021-09-07 05:15:36terry.reedysetnosy: + terry.reedy, uranusjr, zach.ware, DiddiLeija
title: [configparser] Module configparser fails when the config file contains a "%" inside a commentary -> Configparser fails when the file contains $ inside a comment
messages: + msg401207

versions: + Python 3.11
stage: needs patch
2021-09-07 05:08:34koobssetnosy: + lukasz.langa, - terry.reedy, paul.moore, ronaldoussoren, tim.golden, ned.deily, zach.ware, koobs, steve.dower
title: [configparser] Module configparser fails when the config file contains a0918076336 "100%" inside a commentary -> [configparser] Module configparser fails when the config file contains a "%" inside a commentary
assignee: terry.reedy ->
versions: - Python 3.6, Python 3.7, Python 3.8, Python 3.9, Python 3.10, Python 3.11
components: + Library (Lib), - Extension Modules, IDLE, Installation, Interpreter Core, macOS, Tests, Tkinter, Windows, XML, IO, FreeBSD, SSL, Subinterpreters
type: enhancement -> behavior
2021-09-07 03:48:23nirawat.pusetversions: + Python 3.6, Python 3.7, Python 3.8, Python 3.10, Python 3.11
nosy: + koobs, terry.reedy, ned.deily, paul.moore, tim.golden, ronaldoussoren, zach.ware, steve.dower, - lukasz.langa, uranusjr, DiddiLeija

assignee: terry.reedy
components: + Extension Modules, IDLE, Installation, Interpreter Core, macOS, Tests, Tkinter, Windows, XML, IO, FreeBSD, SSL, Subinterpreters, - Library (Lib)
title: [configparser] Module configparser fails when the config file contains a "%" inside a commentary -> [configparser] Module configparser fails when the config file contains a0918076336 "100%" inside a commentary
type: behavior -> enhancement
2021-09-05 12:39:39DiddiLeijasettype: crash -> behavior
2021-09-05 12:38:03DiddiLeijasetmessages: + msg401081
2021-08-24 01:40:26DiddiLeijasetnosy: + uranusjr

title: [configparser] Module configparser fails when the file contains a "%" inside a commentary -> [configparser] Module configparser fails when the config file contains a "%" inside a commentary
2021-08-14 02:08:38DiddiLeijasetmessages: + msg399580
title: configparser fails when the file contains a "%" inside a commentary -> [configparser] Module configparser fails when the file contains a "%" inside a commentary
2021-08-13 20:59:58DiddiLeijasetnosy: + lukasz.langa
messages: + msg399562
2021-08-12 03:20:35DiddiLeijasettitle: Percentage character (%) inside a comment is badly recognized when using configparser -> configparser fails when the file contains a "%" inside a commentary
2021-08-11 20:32:21DiddiLeijacreate