Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Windows: Use /utf-8 compiler flag #87340

Closed
methane opened this issue Feb 9, 2021 · 6 comments
Closed

Windows: Use /utf-8 compiler flag #87340

methane opened this issue Feb 9, 2021 · 6 comments
Labels
3.9 only security fixes 3.10 only security fixes build The build process and cross-build OS-windows

Comments

@methane
Copy link
Member

methane commented Feb 9, 2021

BPO 43174
Nosy @pfmoore, @tjguk, @methane, @zware, @zooba, @miss-islington
PRs
  • bpo-43174: Windows: Use /utf-8 compiler option. #24498
  • [3.9] bpo-43174: Windows: Use /utf-8 compiler option. (GH-24498) #24513
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-02-12.04:44:41.807>
created_at = <Date 2021-02-09.02:59:02.930>
labels = ['3.10', 'build', '3.9', 'OS-windows']
title = 'Windows: Use /utf-8 compiler flag'
updated_at = <Date 2021-02-12.04:44:41.807>
user = 'https://github.com/methane'

    bugs.python.org fields:

    activity = <Date 2021-02-12.04:44:41.807>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2021-02-12.04:44:41.807>
closer = 'methane'
components = ['Build', 'Windows']
creation = <Date 2021-02-09.02:59:02.930>
creator = 'methane'
dependencies = []
files = []
hgrepos = []
issue_num = 43174
keywords = ['patch']
message_count = 6.0
messages = ['386689', '386744', '386842', '386843', '386846', '386851']
nosy_count = 6.0
nosy_names = ['paul.moore', 'tim.golden', 'methane', 'zach.ware', 'steve.dower', 'miss-islington']
pr_nums = ['24498', '24513']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43174'
versions = ['Python 3.9', 'Python 3.10']
    @methane
    Copy link
    Member Author

    methane commented Feb 9, 2021

    When building Python on Windows Japanese environment, several warnings are shown.

    C:\Users\songo\source\repos\cpython\Modules\_sha3\kcp\KeccakSponge.h(1,1): warning C4819: ファイルは、現在のコード ページ (932) で表示できない文字を含んでいます。データの損失を防ぐために、ファイルを
Unicode 形式で保存してください。 [C:\Users\songo\source\repos\cpython\PCbuild\pythoncore.vcxproj]
C:\Users\songo\source\repos\cpython\Modules\_statisticsmodule.c(1,1): warning C4819: ファイルは、現在のコード ページ (932) で表示できない文字を含んで います。データの損失を防ぐために、ファイルを Unico
de 形式で保存してください。 [C:\Users\songo\source\repos\cpython\PCbuild\pythoncore.vcxproj]
C:\Users\songo\source\repos\cpython\Modules\_zoneinfo.c(1,1): warning C4819: ファイルは、現在のコード ページ (932) で表示できない文字を含んでいます。 データの損失を防ぐために、ファイルを Unicode 形式で保存
してください。 [C:\Users\songo\source\repos\cpython\PCbuild\_zoneinfo.vcxproj]
C:\Users\songo\source\repos\cpython\Modules\_zoneinfo.c(1028,1): warning C4819: ファイルは、現在のコード ページ (932) で表示できない文字を含んでいます。データの損失を防ぐために、ファイルを Unicode 形式
で保存してください。 [C:\Users\songo\source\repos\cpython\PCbuild\_zoneinfo.vcxproj]
C:\Users\songo\source\repos\cpython\Modules\_zoneinfo.c(1970,1): warning C4819: ファイルは、現在のコード ページ (932) で表示できない文字を含んでいます。データの損失を防ぐために、ファイルを Unicode 形式
で保存してください。 [C:\Users\songo\source\repos\cpython\PCbuild\_zoneinfo.vcxproj]
C:\Users\songo\source\repos\cpython\Modules\expat\xmltok.c(1,1): warning C4819: ファイルは、現在のコード ページ (932) で表示できない文字を含んでいます。データの損失を防ぐために、ファイルを Unicode 形式
で保存してください。 [C:\Users\songo\source\repos\cpython\PCbuild\_elementtree.vcxproj]
C:\Users\songo\source\repos\cpython\Modules\expat\xmltok.c(1,1): warning C4819: ファイルは、現在のコード ページ (932) で表示できない文字を含んでいます。データの損失を防ぐために、ファイルを Unicode 形式
で保存してください。 [C:\Users\songo\source\repos\cpython\PCbuild\pyexpat.vcxproj]
C:\Users\songo\source\repos\cpython\Modules\_lzmamodule.c(1,1): warning C4819: ファイルは、現在のコード ページ (932) で表示できない文字を含んでいます 。データの損失を防ぐために、ファイルを Unicode 形式で
保存してください。 [C:\Users\songo\source\repos\cpython\PCbuild\_lzma.vcxproj]

    These warnings are shown because source code is written in UTF-8 but compiler assume it's encoded in current code page.

    I don't know what is the best way to fix the warning, but I can fix it by adding /utf-8 flag.

    diff --git a/PCbuild/_zoneinfo.vcxproj b/PCbuild/_zoneinfo.vcxproj
index 6e6389c377..4602e45ce5 100644
--- a/PCbuild/_zoneinfo.vcxproj
+++ b/PCbuild/_zoneinfo.vcxproj
@@ -91,6 +91,11 @@
   <PropertyGroup>
     <_ProjectFileVersion>10.0.30319.1</_ProjectFileVersion>
   </PropertyGroup>
+  <ItemDefinitionGroup>
+    <ClCompile>
+      <AdditionalOptions>/utf-8 %(AdditionalOptions)</AdditionalOptions>
+    </ClCompile>
+  </ItemDefinitionGroup>
   <ItemGroup>
     <ClCompile Include="..\Modules\_zoneinfo.c" />
   </ItemGroup>

    @methane methane added 3.10 only security fixes build The build process and cross-build labels Feb 9, 2021
    @zooba
    Copy link
    Member

    zooba commented Feb 9, 2021

    The new property should go into PCbuild/pyproject.props, but otherwise seems like an okay change.

    It *will* cause any string literals (const char *) to be UTF-8, but they ought to all be ASCII anyway. It shouldn't affect resource files, because those have a code page override in the source files. (Potentially a "#pragma code_page(1252)" might make more sense for our build, but I'm not sure, and I'm just as happy to got to UTF-8 if it doesn't break anything.)

    @methane
    Copy link
    Member Author

    methane commented Feb 12, 2021

    New changeset fedd86d by Inada Naoki in branch 'master':
    bpo-43174: Windows: Use /utf-8 compiler option. (GH-24498)
    fedd86d

    @methane
    Copy link
    Member Author

    methane commented Feb 12, 2021

    May I backport this for Python 3.9?

    @zooba
    Copy link
    Member

    zooba commented Feb 12, 2021

    I think so, yeah. We shouldn't have any string literals that are non-ASCII, but if we do then it's almost certainly an improvement for them to be UTF-8.

    @methane
    Copy link
    Member Author

    methane commented Feb 12, 2021

    New changeset 68d6bc7 by Miss Islington (bot) in branch '3.9':
    bpo-43174: Windows: Use /utf-8 compiler option. (GH-24498)
    68d6bc7

    @methane methane added the 3.9 only security fixes label Feb 12, 2021
    @methane methane closed this as completed Feb 12, 2021
    @methane methane added the 3.9 only security fixes label Feb 12, 2021
    @methane methane closed this as completed Feb 12, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes 3.10 only security fixes build The build process and cross-build OS-windows
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @methane @zooba @tirkarthi