classification
Title: ConfigParser calls optionxform twice when assigning dict
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Phil Kang, inada.naoki, lukasz.langa, xtreak
Priority: normal Keywords: patch, patch, patch

Created on 2019-01-27 22:27 by Phil Kang, last changed 2019-03-07 09:30 by inada.naoki.

Pull Requests
URL Status Linked Edit
PR 11760 open xtreak, 2019-02-05 10:06
PR 11760 open xtreak, 2019-02-05 10:06
PR 11760 open xtreak, 2019-02-05 10:06
Messages (5)
msg334439 - (view) Author: Phil Kang (Phil Kang) Date: 2019-01-27 22:27
ConfigParser calls ConfigParser.optionxform twice per each key when assigning a dictionary to a section.

The following code:

    ini = configparser.ConfigParser()
    ini.optionxform = lambda x: '(' + x + ')'
    
    # Bugged
    ini['section A'] = {'key 1': 'value 1', 'key 2': 'value 2'}
    # Not bugged
    ini.add_section('section B')
    ini['section B']['key 3'] = 'value 3'
    ini['section B']['key 4'] = 'value 4'

   inifile = io.StringIO()
   ini.write(inifile)
   print(inifile.getvalue())

...results in an INI file that looks like:

    [section A]
    ((key 1)) = value 1
    ((key 2)) = value 2
    
    [section B]
    (key 3) = value 3
    (key 4) = value 4

Here, optionxform has been called twice on key 1 and key 2, resulting in the double parentheses.

This also breaks conventional mapping access on the ConfigParser:

    print(ini['section A']['key 1'])    # Raises KeyError('key 1')
    print(ini['section A']['(key 1)'])  # OK
    
    # Raises ValueError: too many values to unpack (expected 2)
    for key, value in ini['section A']:
        print(key + ', ' + value)
msg334452 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2019-01-28 08:56
I think it's easy to solve this particular case.
But there may be some other cases.

optionxform must be idempotent?  If so, this is document issue.
msg334470 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python triager) Date: 2019-01-28 14:27
This seems to be a bug with read_dict which is used internally when a dictionary is directly assigned. In read_dict optionxform is called with key [0] to check for duplicate and the transformed value is again passed to self.set which also calls optionxform [1] causing optionxform to be applied twice. A possible fix would be to assign the transformed key to a temporary variable to check for duplicate and then pass the original key to self.set ? My patch gives correct value and no tests fail on master. I can make a PR with test for this if my analysis is correct.

This fixes the below since the key is stored correctly now. 

print(ini['section A']['key 1'])    # OK
print(ini['section A']['(key 1)'])  # Raises KeyError

I think for iterating over the section items [2] need to be used and the reported code can be written as below

for key, value in ini.items('section A'):
    print(key + ', ' + value)

[0] https://github.com/python/cpython/blob/ea446409cd5f1364beafd5e5255da6799993f285/Lib/configparser.py#L748
[1] https://github.com/python/cpython/blob/ea446409cd5f1364beafd5e5255da6799993f285/Lib/configparser.py#L903
[2] https://docs.python.org/3.8/library/configparser.html#configparser.ConfigParser.items

# sample reproducer

import io
import configparser

ini = configparser.ConfigParser()
ini.optionxform = lambda x: '(' + x + ')'
ini.read_dict({'section A': {'key 1': 'value 1'}})

inifile = io.StringIO()
ini.write(inifile)
print(inifile.getvalue())

$ ./python.exe ../backups/bpo35838_1.py
[section A]
((key 1)) = value 1

# Possible patch

$ git diff -w | cat
diff --git a/Lib/configparser.py b/Lib/configparser.py
index 79a991084b..1389f4ac08 100644
--- a/Lib/configparser.py
+++ b/Lib/configparser.py
@@ -745,13 +745,13 @@ class RawConfigParser(MutableMapping):
                     raise
             elements_added.add(section)
             for key, value in keys.items():
-                key = self.optionxform(str(key))
+                option_key = self.optionxform(str(key))
                 if value is not None:
                     value = str(value)
-                if self._strict and (section, key) in elements_added:
-                    raise DuplicateOptionError(section, key, source)
-                elements_added.add((section, key))
-                self.set(section, key, value)
+                if self._strict and (section, option_key) in elements_added:
+                    raise DuplicateOptionError(section, option_key, source)
+                elements_added.add((section, option_key))
+                self.set(section, str(key), value)

     def readfp(self, fp, filename=None):
         """Deprecated, use read_file instead."""


$ ./python.exe ../backups/bpo35838_1.py
[section A]
(key 1) = value 1
msg337282 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2019-03-06 06:30
It seems twice call of `optionxform` is not avoidable when read-and-write workflow.
I'm not against about fixing readdict.
But I don't think configparser supports non-idempotent optionxform.

>>> import configparser
>>> cfg = configparser.ConfigParser()
>>> cfg.optionxform = lambda s: "#"+s
>>> cfg.add_section("sec")
>>> cfg.set("sec", "foo", "1")
>>> cfg["sec2"] = cfg["sec"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/inada-n/work/python/cpython/Lib/configparser.py", line 974, in __setitem__
    self.read_dict({key: value})
  File "/Users/inada-n/work/python/cpython/Lib/configparser.py", line 747, in read_dict
    for key, value in keys.items():
  File "/Users/inada-n/work/python/cpython/Lib/_collections_abc.py", line 744, in __iter__
    yield (key, self._mapping[key])
  File "/Users/inada-n/work/python/cpython/Lib/configparser.py", line 1254, in __getitem__
    raise KeyError(key)
KeyError: '#foo'
msg337375 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2019-03-07 09:30
I sent a mail to python-dev ML.
https://mail.python.org/pipermail/python-dev/2019-March/156613.html
History
Date User Action Args
2019-03-07 09:30:52inada.naokisetkeywords: patch, patch, patch

messages: + msg337375
2019-03-06 06:30:12inada.naokisetkeywords: patch, patch, patch

messages: + msg337282
2019-02-05 10:06:27xtreaksetkeywords: + patch
stage: patch review
pull_requests: + pull_request11709
2019-02-05 10:06:21xtreaksetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11708
2019-02-05 10:06:14xtreaksetkeywords: + patch
stage: (no value)
pull_requests: + pull_request11707
2019-01-28 14:27:28xtreaksetnosy: + xtreak

messages: + msg334470
versions: + Python 3.8
2019-01-28 08:56:57inada.naokisetnosy: + inada.naoki
messages: + msg334452
2019-01-27 22:41:11xtreaksetnosy: + lukasz.langa
2019-01-27 22:27:01Phil Kangcreate