Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

year 2038 problem in compileall.py #79171

Closed
bmwiedemann mannequin opened this issue Oct 15, 2018 · 13 comments
Closed

year 2038 problem in compileall.py #79171

bmwiedemann mannequin opened this issue Oct 15, 2018 · 13 comments
Labels
3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes build The build process and cross-build

Comments

@bmwiedemann
Copy link
Mannequin

bmwiedemann mannequin commented Oct 15, 2018

BPO 34990
Nosy @jwilk, @serhiy-storchaka, @matrixise, @bmwiedemann, @ammaraskar, @miss-islington, @tirkarthi
PRs
  • WIP: bpo-34990: year 2038 problem in compileall.py #9892
  • bpo-34990: Change pyc headers to use 64-bit timestamps #19651
  • bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int #19708
  • [3.10] bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708) #27928
  • [3.9] bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708) #27929
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-08-24.15:13:34.226>
    created_at = <Date 2018-10-15.11:22:16.253>
    labels = ['build', '3.9', '3.10', '3.11']
    title = 'year 2038 problem in compileall.py'
    updated_at = <Date 2022-02-21.11:46:17.662>
    user = 'https://github.com/bmwiedemann'

    bugs.python.org fields:

    activity = <Date 2022-02-21.11:46:17.662>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-08-24.15:13:34.226>
    closer = 'ammar2'
    components = ['Build']
    creation = <Date 2018-10-15.11:22:16.253>
    creator = 'bmwiedemann'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 34990
    keywords = ['patch']
    message_count = 13.0
    messages = ['327743', '327747', '327748', '327749', '327750', '327751', '327753', '327774', '360270', '367679', '400200', '400214', '400215']
    nosy_count = 7.0
    nosy_names = ['jwilk', 'serhiy.storchaka', 'matrixise', 'bmwiedemann', 'ammar2', 'miss-islington', 'xtreak']
    pr_nums = ['9892', '19651', '19708', '27928', '27929']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'compile error'
    url = 'https://bugs.python.org/issue34990'
    versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    @bmwiedemann
    Copy link
    Mannequin Author

    bmwiedemann mannequin commented Oct 15, 2018

    To reproduce:
    touch -d 2038-01-20 /usr/lib/python3.6/site-packages/six.py
    python3 /usr/lib64/python3.6/compileall.py

    File "/usr/lib64/python3.6/compileall.py", line 198, in compile_path
    legacy=legacy, optimize=optimize)
    File "/usr/lib64/python3.6/compileall.py", line 90, in compile_dir
    legacy, optimize):
    File "/usr/lib64/python3.6/compileall.py", line 138, in compile_file
    mtime)
    struct.error: 'l' format requires -2147483648 <= number <= 2147483647

    It could use either
    64 bit int (requires new .pyc format with different magic number) or
    unsigned 32 bit int (gives us only another 68 years)

    @bmwiedemann bmwiedemann mannequin added 3.7 (EOL) end of life build The build process and cross-build labels Oct 15, 2018
    @matrixise
    Copy link
    Member

    With 3.8a

    Traceback (most recent call last):
      File "/home/stephane/src/github.com/python/cpython/Lib/runpy.py", line 192, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/home/stephane/src/github.com/python/cpython/Lib/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/home/stephane/src/github.com/python/cpython/Lib/compileall.py", line 326, in <module>
        exit_status = int(not main())
      File "/home/stephane/src/github.com/python/cpython/Lib/compileall.py", line 303, in main
        if not compile_file(dest, args.ddir, args.force, args.rx,
      File "/home/stephane/src/github.com/python/cpython/Lib/compileall.py", line 142, in compile_file
        expect = struct.pack('<4sll', importlib.util.MAGIC_NUMBER,
    struct.error: 'l' format requires -2147483648 <= number <= 2147483647

    @matrixise
    Copy link
    Member

    But until 2038, maybe there will be a new format for the .pyc file.

    We should keep this issue and try to fix it for 3.8 or 3.9?

    @bmwiedemann
    Copy link
    Mannequin Author

    bmwiedemann mannequin commented Oct 15, 2018

    It does not need to be fixed tomorrow, but 2037 is too late, because by then there will be a lot of legacy systems around.
    (Un)fortunately many systems live 10+ years now

    @vstinner
    Copy link
    Member

    Timestamp with year >= 2038 are accepted: importlib._bootstrap_external._code_to_timestamp_pyc() uses (int(x) & 0xFFFFFFFF). It's not a bug, but by design. compileall should just do the same. Sorry, I don't know if it's specified somewhere, but I know that it's done on purpose.

    @matrixise
    Copy link
    Member

    So we need to fix compileall.py.

    maybe we could add the label 'easy' to this issue.

    @tirkarthi
    Copy link
    Member

    A reproducer in Python that can be added to test_compileall if needed :

    def test_compile_all_2038(self):
        with open(self.source_path, 'r') as f:
            os.utime(f.name, (2147558400, 2147558400)) # Jan 20, 2038 as touch
        self.assertTrue(compileall.compile_file(pathlib.Path(self.source_path)))

    ./python.exe -m unittest -v test.test_compileall.CompileallTestsWithSourceEpoch.test_compile_all_2038
    test_compile_all_2038 (test.test_compileall.CompileallTestsWithSourceEpoch) ... ERROR

    ======================================================================
    ERROR: test_compile_all_2038 (test.test_compileall.CompileallTestsWithSourceEpoch)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/test_py_compile.py", line 30, in wrapper
        return fxn(*args, **kwargs)
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/test/test_compileall.py", line 114, in test_compile_all_2038
        self.assertTrue(compileall.compile_file(pathlib.Path(self.source_path)))
      File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/compileall.py", line 142, in compile_file
        expect = struct.pack('<4sll', importlib.util.MAGIC_NUMBER,
    struct.error: 'l' format requires -2147483648 <= number <= 2147483647

    Thanks

    @tirkarthi
    Copy link
    Member

    Victor seems there was some discussion about 2038 problem in the original PR but I don't know if it's related to this. Reference : #4575 (comment)

    Thanks

    @bmwiedemann
    Copy link
    Mannequin Author

    bmwiedemann mannequin commented Jan 19, 2020

    ping.
    Another 19th of January passed.

    I'd still like to see progress on this, because this hinders my other y2038 bug discovery work.

    @bmwiedemann bmwiedemann mannequin added 3.8 only security fixes 3.9 only security fixes labels Jan 19, 2020
    @vstinner
    Copy link
    Member

    I would prefer to mimick importlib._bootstrap_external which uses:

    def _pack_uint32(x):
        """Convert a 32-bit integer to little-endian."""
        return (int(x) & 0xFFFFFFFF).to_bytes(4, 'little')

    Using 64-bit timestamp (PR 19651), treat timestamp as unsigned (PR 9892 and PR 19708) have drawback:

    • 64-bit timestamp make .pyc files larger
    • unsigned timestamp no longer support timestamp before 1969 which can cause practical issues

    "& 0xFFFFFFFF" looks dead simple, uses a fixed size of 4 bytes and doesn't have any limitation of year 2038.

    The timestamp doesn't have to be exact. In practice, it sounds very unlikely that two timestamps are equal when compared using (ts1 & 0xFFFFFFFF) == (ts2 & 0xFFFFFFFF). I expect file modification times to be close by a few days, not separated by 2**32 seconds (136 years).

    Use hash based .pyc to avoid any issuse with file modification time: it should make Python more deterministic (more "reproducible").
    https://docs.python.org/dev/reference/import.html#pyc-invalidation

    @serhiy-storchaka
    Copy link
    Member

    New changeset bb21e28 by Ammar Askar in branch 'main':
    bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708)
    bb21e28

    @ammaraskar
    Copy link
    Member

    New changeset 9d3b6b2 by Miss Islington (bot) in branch '3.9':
    [3.9] bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708)
    9d3b6b2

    @ammaraskar
    Copy link
    Member

    New changeset 0af681b by Miss Islington (bot) in branch '3.10':
    [3.10] bpo-34990: Treat the pyc header's mtime in compileall as an unsigned int (GH-19708)
    0af681b

    @ammaraskar ammaraskar added 3.10 only security fixes 3.11 bug and security fixes and removed 3.7 (EOL) end of life 3.8 only security fixes labels Aug 24, 2021
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    bmwiedemann pushed a commit to bmwiedemann/openSUSE that referenced this issue Sep 3, 2022
    https://build.opensuse.org/request/show/1000772
    by user StevenK + dimstar_suse
    - Add patch CVE-2021-28861-double-slash-path.patch:
      * http.server: Fix an open redirection vulnerability in the HTTP server
        when an URI path starts with //. (bsc#1202624, CVE-2021-28861)
    
    - Add bpo34990-2038-problem-compileall.patch making compileall.py
      compliant with year 2038 (bsc#1202666, gh#python/cpython#79171),
      backport of fix to Python 3.8.
    - Add conditional for requiring rpm-build-python, so we should be
      compilable on SLE/Leap.
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes build The build process and cross-build
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants