classification
Title: csv writer doesn't escape escapechar
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Aldrian Obaja, Alex Shpilkin, Apathymannequin, Dmitriy Shashkin, avatarofhope, berker.peksag, catalin.iacob, ebreck, eric.araujo, taleinat, xflr6
Priority: normal Keywords: needs review, patch

Created on 2011-05-25 18:27 by ebreck, last changed 2021-07-13 19:26 by xflr6. This issue is now closed.

Files
File name Uploaded Description Edit
pybug.zip ebreck, 2011-05-25 18:27
0eb420ce6567.diff catalin.iacob, 2011-07-06 21:21 review
Pull Requests
URL Status Linked Edit
PR 13710 merged berker.peksag, 2019-05-31 20:57
Repositories containing patches
http://bitbucket.org/cataliniacob/cpython#issue12178
Messages (13)
msg136881 - (view) Author: Eric Breck (ebreck) Date: 2011-05-25 18:27
Consider the attached two files.  A reader and writer with the same dialect parameters (escapechar \ quotechar " doublequote False) read, then write a CSV cell that looks like "C\\".  It's written "C\".  The problem is, when doublequote=False, the escapechar isn't used to escape itself, and the writer writes something that in the same dialect would be understood differently (\" isn't \ then end of string, it's an escaped quotechar within the string).

Execute python err.py first.csv to see.
msg136973 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-05-26 14:59
Thanks for the report.  It would be best if you could attach files as plain text instead of archives.
msg139953 - (view) Author: Catalin Iacob (catalin.iacob) * Date: 2011-07-06 21:20
I looked at this and tried to provide a patch + tests. Please review.

The bug is that a writer can use writerow on some input data but if a reader with the same dialect reads them back they are different from the input ones. This happens when the input data contains escapechar. Contrary to msg136881, this happens regardless whether doublequote is True or False.

The docs say "On reading, the escapechar removes any special meaning from the following character". Therefore, I understand that on writing, escapechar must always be escaped by itself. If that doesn't happen, when reading it back, escapechar alters the thing that follows it instead of counting as escapechar which is precisely what this bug is about.
msg140002 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-07-07 23:31
Thanks for the patch.  The tests look good at first glance.  I can’t comment on the C code, I don’t know C.  Hopefully someone will do it, otherwise if you don’t get feedback in say four weeks you can ask for a review on python-dev.
msg273746 - (view) Author: Wayne Harris (Apathymannequin) Date: 2016-08-27 02:11
Hi, 

I can confirm that this behavior still exists in my current python versions (3.5.2 & 2.7.11). I'm happy to take a look at the code, but considering I made this account specifically to comment on this issue I assume someone else will want to, as well.

If you want to make sure this is still broken, just use any of the docs' reader and writer examples, adding a trailing escape char to the end.
msg273784 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2016-08-27 16:18
The patch looked okay to me, and when applied to the 2.7 source, the new tests pass muster.

I'm not going to pretend I know where this patch should be applied. That's for someone else to pronounce.
msg309815 - (view) Author: Sebastian Bank (xflr6) Date: 2018-01-11 16:27
Hi, is there something we can do to get this going? As the issue breaks round-trip, it currently requires work-arounds like this: https://github.com/cldf/csvw/blob/1324550266c821ef32d1e79c124191e93aefbfa8/csvw/dsv.py#L67-L71
msg349720 - (view) Author: Brad MacGregor (avatarofhope) Date: 2019-08-14 17:35
I needed this patch for a project, so I compiled python with the patch applied, and tested my specific use case, and it worked. Thanks for the patch!
msg352447 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-09-14 19:01
FWIW, though this is arguably fixing a bug, IMO this shouldn't be back-ported to 2.7 or 3.7, but only be in 3.8+, to avoid breaking backwards-compatibility.

That being said, it would be great to get this into 3.8.0!
msg355118 - (view) Author: Aldrian Obaja (Aldrian Obaja) Date: 2019-10-22 06:09
I think this issue needs to be escalated, as this is clearly a bug that makes it troublesome to use csv.reader and csv.writer with escapechar.
msg377206 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-09-20 06:38
New changeset 5c0eed7375fdd791cc5e19ceabfab4170ad44062 by Berker Peksag in branch 'master':
bpo-12178: Fix escaping of escapechar in csv.writer() (GH-13710)
https://github.com/python/cpython/commit/5c0eed7375fdd791cc5e19ceabfab4170ad44062
msg377207 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-09-20 06:40
After a great deal of delay, a fix for this has finally been merged, and will be available in Python 3.10.

Thanks to everyone involved!
msg397440 - (view) Author: Sebastian Bank (xflr6) Date: 2021-07-13 19:26
Thanks Tal.

AFAICT there was an undocumented change in behaviour related to this fix.

Python 3.9 quotes values with escapechar:

```
import csv
import io

kwargs = {'escapechar': '\\'}

value = 'spam\\eggs'

print(value)

with io.StringIO() as buf:
    writer = csv.writer(buf, **kwargs)
    writer.writerow([value])
    line = buf.getvalue()

print(line.strip())

with io.StringIO(line) as buf:
    reader = csv.reader(buf, **kwargs)
    (new_value,), = reader

print(new_value)
spam\eggs
"spam\eggs"
spameggs
```

- quotes escapechar
- fails to double the escapechar (this bug)

Btw, from
https://docs.python.org/3/library/csv.html#csv.QUOTE_MINIMAL

> only quote those fields which contain special characters
> such as delimiter, quotechar or any of the characters in lineterminator.

this seems incorrect because escapechar is not mentioned (but at the same time it says 'such as') and maybe better matching the name 'minimal' (or one might expect 'more' quoting as a better default).

Python 3.10:

https://github.com/python/cpython/blob/5c0eed7375fdd791cc5e19ceabfab4170ad44062/Lib/test/test_csv.py#L207-L208

See also https://github.com/xflr6/csv23/actions/runs/1027687524
History
Date User Action Args
2021-07-13 19:26:06xflr6setmessages: + msg397440
2020-09-20 06:40:59taleinatsetstatus: open -> closed
versions: + Python 3.10, - Python 2.7, Python 3.8, Python 3.9
messages: + msg377207

resolution: fixed
stage: patch review -> resolved
2020-09-20 06:38:14taleinatsetmessages: + msg377206
2019-10-22 06:12:05larrysetnosy: - larry
2019-10-22 06:11:30larrysetversions: + Python 3.8, Python 3.9, - Python 3.5, Python 3.6
2019-10-22 06:09:22Aldrian Obajasetnosy: + Aldrian Obaja
messages: + msg355118
2019-09-14 19:01:26taleinatsetnosy: + taleinat
messages: + msg352447
2019-08-14 17:35:07avatarofhopesetnosy: + avatarofhope
messages: + msg349720
2019-06-02 18:19:01Jeffrey.Kintschersetnosy: - Jeffrey.Kintscher
2019-05-31 20:57:08berker.peksagsetpull_requests: + pull_request13597
2019-05-31 08:45:22Jeffrey.Kintschersetnosy: + Jeffrey.Kintscher
2019-05-09 09:10:10Alex Shpilkinsetnosy: + Alex Shpilkin
2018-01-11 16:27:51xflr6setnosy: + xflr6
messages: + msg309815
2017-12-27 12:12:40Dmitriy Shashkinsetnosy: + Dmitriy Shashkin
2016-08-27 16:45:23berker.peksagsetnosy: + berker.peksag

versions: + Python 3.5, Python 3.6, - Python 3.2, Python 3.3
2016-08-27 16:18:51skip.montanarosetnosy: - skip.montanaro
2016-08-27 16:18:33skip.montanarosetmessages: + msg273784
2016-08-27 02:11:08Apathymannequinsetnosy: + Apathymannequin
messages: + msg273746
2012-07-07 08:40:10catalin.iacobsetnosy: + larry
2011-07-07 23:31:42eric.araujosetkeywords: + needs review

stage: patch review
messages: + msg140002
versions: - Python 3.1
2011-07-06 21:21:25catalin.iacobsetfiles: + 0eb420ce6567.diff
keywords: + patch
2011-07-06 21:20:31catalin.iacobsetnosy: + catalin.iacob

messages: + msg139953
hgrepos: + hgrepo38
2011-05-26 14:59:20eric.araujosetversions: + Python 3.1, Python 3.2, Python 3.3, - Python 2.6
nosy: + eric.araujo, skip.montanaro

messages: + msg136973

components: + Extension Modules, - None
2011-05-25 18:27:10ebreckcreate