I want to manage all the packages that I installed so every time adding package I set "--target" so the package will be downloaded there. Then I wrote the directory in a .pth file which is located in "/Python36/Lib/site-packages" so I could still get accessed to all the packages even though they are not located within "Python36" folder.

However, my current user name of Windows is a Chinese name, which means the customized path I mentioned before has Chinese characters within it, thus the .pth file will be also encoded with 'gbk'. Every time I would like to import these packages will get "UnicodeDecodeError: 'gbk' can't decode byte xxx...".

Fortunately I have found the reason and cracked the problem: python read .pth files without setting any encoding. The code is located in "Python36/Lib/site.py"

def addpackage(sitedir, name, known_paths):
    if known_paths is None:
        known_paths = _init_pathinfo()
        reset = True
    else:
        reset = False
    fullname = os.path.join(sitedir, name)
    try:
        # here should set the second param as encoding='utf-8'
        f = open(fullname, "r")
    except OSError:
        return
    # other codes

And after I doing this, everything goes well.

Can you save your file in gbk encoding? That will be an immediate fix.

I don't know that we can/should change the encoding we read without checking with everyone who writes out .pth files. (+Jason as a start here, but I suspect there are more tools that write them.)

We could add a handler for UnicodeDecodeError that falls back on utf-8? I think that's reasonable.

I'll mark this easy as well, since adding that handler is straightforward. Unless someone knows a reason we shouldn't do that either.

Hello, Valentin Zhao, do you have time to fix it? Or I can create a PR

I'm only aware of one tool that writes .pth files, and that's setuptools, and it always writes ASCII (assuming package names are ASCII), so any encoding handling should be fine there.

> We could add a handler for UnicodeDecodeError that falls back on utf-8?

Yes, reasonable, but maybe we should consider instead _preferring_ UTF-8 and fall back to default encodings. That would be my preference.

Also, I would argue that this is an enhancement request and not a bug - that the prior expectation was that the .pth file is encoded in whatever encoding the system expects by default, and that adding support for a standardized encoding for .pth files is a new feature.

As another aside: Valentin, the technique you're using to manage packages is likely to run into issues with certain packages - in particular any packages that rely on their own `.pth` files to invoke behavior, such as future_fstrings (https://pypi.org/project/future-fstrings/). I learned about this issue in (https://github.com/jaraco/rwt/issues/29), which is why the rwt project adds a `sitecustomize.py` to the target directory that ensures .pth files are run. Just FYI.

I am better just waiting you guys fixing that because it is not urgent.
On Sat, Nov 3, 2018 at 10:12 PM Jason R. Coombs <report@bugs.python.org>
wrote:

>
> Jason R. Coombs <jaraco@jaraco.com> added the comment:
>
> Also, I would argue that this is an enhancement request and not a bug -
> that the prior expectation was that the .pth file is encoded in whatever
> encoding the system expects by default, and that adding support for a
> standardized encoding for .pth files is a new feature.
>
> As another aside: Valentin, the technique you're using to manage packages
> is likely to run into issues with certain packages - in particular any
> packages that rely on their own `.pth` files to invoke behavior, such as
> future_fstrings (https://pypi.org/project/future-fstrings/). I learned
> about this issue in (https://github.com/jaraco/rwt/issues/29), which is
> why the rwt project adds a `sitecustomize.py` to the target directory that
> ensures .pth files are run. Just FYI.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35131>
> _______________________________________
>

I tried to create a PR for it, However, I don't know how to handle the code at https://github.com/python/cpython/blob/d4c76d960b/Lib/site.py#L159

So how to check UnicodeDecodeError when we just open the file, I use readlines() but it may use too many memory than before (I'm not sure it's important in this case).

    try:
        f = open(fullname, "r")
        data = f.readlines()
    except UnicodeDecodeError:
        f = open(fullname, "r", encoding="utf-8")
        data = f.readlines()

The problem you've encountered is that previously the file was assumed to be one encoding and would fail if it was not that encoding... so it was possible to lazy-load the file and process each line.

In the new model, where you need to evaluate the viability of the file in one of two candidate encodings, you'll necessarily need to read the entire file once before processing its contents.

Therefore, I recommend one of these options:

1. Always read the file in binary mode, ascertain the "best" encoding, then rewind the file and wrap it in a TextIOWrapper for that encoding. Presumably this logic is common--perhaps there's already a routine that does just that.
2. In a try/except block, read the entire content, decoded, into another iterable ... and then have the logic below rely on that content. i.e. `f = list(f)`.
3. Always assume UTF-8 instead of the system encoding. This change would be backward incompatible, so probably isn't acceptable without at least an interim release with a deprecation warning.

I recommend a combination of (1) and then (3) in the future. That is:

def determine_best_encoding(f, encodings=('utf-8', sys.getdefaultencoding())):
    """
    Attempt to read and decode all of stream f using the encodings
    and return the first one that succeeds. Rewinds the file.
    """


f = open(..., 'rb)
encoding = determine_best_encoding(f)
if encoding != 'utf-8':
    warnings.warn("Detected pth file with unsupported encoding", DeprecationWarning)
f = io.TextIOWrapper(f, encoding)


Then, in a future version, dropping support for local encodings, all of that code can be replaced with `f = open(..., encoding='utf-8')`.

There is not "find best encoding" code, hence why so much code out there uses chardet. :)

This might also tie into issue #33944 and the idea of rethinking .pth files.

I will fix this issue after we have consensus with the future of .pth file in #33944