classification
Title: Special Characters Don't Work When Using Virtual Environment
Type: behavior Stage: resolved
Components: Unicode, Windows Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, ezio.melotti, glGarg, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords:

Created on 2019-09-10 23:30 by glGarg, last changed 2019-09-13 17:06 by eryksun. This issue is now closed.

Messages (4)
msg351757 - (view) Author: Spandan Garg (glGarg) Date: 2019-09-10 23:30
Recently, I noticed that whenever I execute python from inside a virtual environment, I can't seem to write special characters into the shell. Although, when I exit the virtual environment, this issue no longer occurs. I've confirmed that the versions of python inside and outside the virtual environment are the same.

For example, when I paste the string 'doesn’t' (with the Unicode apostrophe) into the shell without virtual environment, I get the following:
>>> doesn’t     # without virtual environment

When I paste the same string into the shell with a virtual environment active (and the same version of python), I get the following:
>>> doesnt      # with virtual environment

The version of python I am using is python 3.7.4 (though I have reproduced this on python 3.6.5 as well). I am creating the virtual environments using the venv module. Also, I think that this problem only happens on Windows as I couldn’t reproduce it on Linux.

Does anyone know why this may be happening? Please let me know if I should provide any other details.
msg352313 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-09-13 12:53
Thanks Spandan

This doesn't reproduce trivially for me, so I suspect there is some difference in the venv.

Could you run "pip list" in both the venv and with it deactivated and share the output? There are some packages that modify how console input works.
msg352348 - (view) Author: Spandan Garg (glGarg) Date: 2019-09-13 15:52
This is the output of pip list with venv active:
(aladdin_windows_env) C:\Users\t-spga\Documents\workspace\aladdin_windows_env>pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
adal (1.2.2)
antlr4-python3-runtime (4.7.2)
applicationinsights (0.11.9)
argcomplete (1.10.0)
asn1crypto (0.24.0)
azure-batch (7.0.0)
azure-cli (2.0.72)
azure-cli-command-modules-nspkg (2.0.3)
azure-cli-core (2.0.72)
azure-cli-nspkg (3.0.4)
azure-cli-telemetry (1.0.3)
azure-common (1.1.23)
azure-cosmos (3.1.1)
azure-datalake-store (0.0.47)
azure-functions-devops-build (0.0.22)
azure-graphrbac (0.60.0)
azure-keyvault (1.1.0)
azure-kusto-data (0.0.33)
azure-mgmt-advisor (2.0.1)
azure-mgmt-apimanagement (0.1.0)
azure-mgmt-appconfiguration (0.1.0)
azure-mgmt-applicationinsights (0.1.1)
azure-mgmt-authorization (0.52.0)
azure-mgmt-batch (6.0.0)
azure-mgmt-batchai (2.0.0)
azure-mgmt-billing (0.2.0)
azure-mgmt-botservice (0.2.0)
azure-mgmt-cdn (3.1.0)
azure-mgmt-cognitiveservices (5.0.0)
azure-mgmt-compute (6.0.0)
azure-mgmt-consumption (2.0.0)
azure-mgmt-containerinstance (1.5.0)
azure-mgmt-containerregistry (3.0.0rc5)
azure-mgmt-containerservice (5.3.0)
azure-mgmt-cosmosdb (0.7.0)
azure-mgmt-datalake-analytics (0.2.1)
azure-mgmt-datalake-nspkg (3.0.1)
azure-mgmt-datalake-store (0.5.0)
azure-mgmt-datamigration (0.1.0)
azure-mgmt-deploymentmanager (0.1.0)
azure-mgmt-devtestlabs (2.2.0)
azure-mgmt-dns (2.1.0)
azure-mgmt-eventgrid (2.2.0)
azure-mgmt-eventhub (2.6.0)
azure-mgmt-hdinsight (1.1.0)
azure-mgmt-imagebuilder (0.2.1)
azure-mgmt-iotcentral (1.0.0)
azure-mgmt-iothub (0.8.2)
azure-mgmt-iothubprovisioningservices (0.2.0)
azure-mgmt-keyvault (1.1.0)
azure-mgmt-kusto (0.3.0)
azure-mgmt-loganalytics (0.2.0)
azure-mgmt-managedservices (1.0.0)
azure-mgmt-managementgroups (0.2.0)
azure-mgmt-maps (0.1.0)
azure-mgmt-marketplaceordering (0.2.1)
azure-mgmt-media (1.1.1)
azure-mgmt-monitor (0.5.2)
azure-mgmt-msi (0.2.0)
azure-mgmt-netapp (0.5.0)
azure-mgmt-network (4.0.0)
azure-mgmt-nspkg (3.0.2)
azure-mgmt-policyinsights (0.3.1)
azure-mgmt-privatedns (0.1.0)
azure-mgmt-rdbms (1.9.0)
azure-mgmt-recoveryservices (0.4.0)
azure-mgmt-recoveryservicesbackup (0.4.0)
azure-mgmt-redis (6.0.0)
azure-mgmt-relay (0.1.0)
azure-mgmt-reservations (0.3.1)
azure-mgmt-resource (2.2.0)
azure-mgmt-search (2.1.0)
azure-mgmt-security (0.1.0)
azure-mgmt-servicebus (0.6.0)
azure-mgmt-servicefabric (0.2.0)
azure-mgmt-signalr (0.3.0)
azure-mgmt-sql (0.13.0)
azure-mgmt-sqlvirtualmachine (0.4.0)
azure-mgmt-storage (4.0.0)
azure-mgmt-trafficmanager (0.51.0)
azure-mgmt-web (0.42.0)
azure-multiapi-storage (0.2.4)
azure-nspkg (3.0.2)
azure-storage-blob (1.5.0)
azure-storage-common (1.4.2)
bcrypt (3.1.7)
certifi (2019.6.16)
cffi (1.12.3)
chardet (3.0.4)
colorama (0.4.1)
cryptography (2.7)
fabric (2.5.0)
gitdb2 (2.0.5)
GitPython (2.1.11)
humanfriendly (4.18)
idna (2.8)
invoke (1.3.0)
isodate (0.6.0)
javaproperties (0.5.1)
Jinja2 (2.10.1)
jmespath (0.9.4)
jsondiff (1.2.0)
knack (0.6.3)
MarkupSafe (1.1.1)
mock (2.0.0)
msrest (0.6.10)
msrestazure (0.6.1)
oauthlib (3.1.0)
paramiko (2.6.0)
pbr (5.4.3)
pip (9.0.3)
portalocker (1.5.1)
psutil (5.6.3)
pycparser (2.19)
pydocumentdb (2.3.3)
Pygments (2.4.2)
PyJWT (1.7.1)
PyNaCl (1.3.0)
pyOpenSSL (19.0.0)
pypiwin32 (223)
pyreadline (2.1)
python-dateutil (2.8.0)
python-git (2018.2.1)
pytz (2019.1)
pywin32 (224)
PyYAML (5.1.2)
requests (2.22.0)
requests-oauthlib (1.2.0)
ruamel.yaml (0.16.5)
ruamel.yaml.clib (0.1.2)
scp (0.13.2)
Send2Trash (1.5.0)
setuptools (39.0.1)
six (1.12.0)
smmap2 (2.0.5)
sshtunnel (0.1.5)
tabulate (0.8.3)
urllib3 (1.25.3)
vsts (0.1.25)
vsts-cd-manager (1.0.2)
websocket-client (0.56.0)
wheel (0.30.0)
xmltodict (0.12.0)
You are using pip version 9.0.3, however version 19.2.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

This is the output of pip list with venv deactivated:
C:\Users\t-spga\Documents\workspace\aladdin_windows_env>pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pip (9.0.3)
setuptools (39.0.1)
You are using pip version 9.0.3, however version 19.2.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Thanks!

Spandan
msg352370 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-09-13 17:06
> pyreadline (2.1)

The problem is pyreadline. It uses a low-level console read, but it fails to account for the Alt+Numpad sequence of input records that the console uses for non-OEM characters. IIRC, the Unicode character is only sent in the final input record of the sequence.
History
Date User Action Args
2019-09-13 17:06:07eryksunsetstatus: open -> closed

nosy: + eryksun
messages: + msg352370

resolution: third party
stage: resolved
2019-09-13 15:52:52glGargsetmessages: + msg352348
2019-09-13 12:53:24steve.dowersetmessages: + msg352313
2019-09-13 12:51:22steve.dowersetnosy: + paul.moore, tim.golden, zach.ware, steve.dower

components: + Windows
versions: + Python 3.8, Python 3.9
2019-09-10 23:30:52glGargcreate