Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CVE-2018-20852] Cookie domain check returns incorrect results #79302

Closed
bobunderson mannequin opened this issue Oct 31, 2018 · 30 comments
Closed

[CVE-2018-20852] Cookie domain check returns incorrect results #79302

bobunderson mannequin opened this issue Oct 31, 2018 · 30 comments
Assignees
Labels
3.7 (EOL) end of life 3.8 only security fixes release-blocker stdlib Python modules in the Lib dir type-security A security issue

Comments

@bobunderson
Copy link
Mannequin

bobunderson mannequin commented Oct 31, 2018

BPO 35121
Nosy @orsenthil, @vstinner, @larryhastings, @benjaminp, @ned-deily, @ambv, @vadmium, @serhiy-storchaka, @miss-islington, @Windsooon, @tirkarthi, @bobunderson, @ret2libc
PRs
  • bpo-35121: prefix dot in domain for proper subdomain validation #10258
  • [3.6] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) #12260
  • [3.7] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) #12261
  • [3.4] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) #12279
  • [3.5] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) #12281
  • [2.7] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) #13426
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/benjaminp'
    closed_at = <Date 2019-06-15.16:38:00.513>
    created_at = <Date 2018-10-31.06:52:48.946>
    labels = ['type-security', '3.8', '3.7', 'library', 'release-blocker']
    title = '[CVE-2018-20852] Cookie domain check returns incorrect results'
    updated_at = <Date 2019-07-15.09:42:01.659>
    user = 'https://github.com/bobunderson'

    bugs.python.org fields:

    activity = <Date 2019-07-15.09:42:01.659>
    actor = 'vstinner'
    assignee = 'benjamin.peterson'
    closed = True
    closed_date = <Date 2019-06-15.16:38:00.513>
    closer = 'xtreak'
    components = ['Library (Lib)']
    creation = <Date 2018-10-31.06:52:48.946>
    creator = '\xe8\xa5\xbf\xe7\x94\xb0\xe9\x9b\x84\xe6\xb2\xbb'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 35121
    keywords = ['patch', 'security_issue']
    message_count = 30.0
    messages = ['328973', '328975', '328981', '328985', '329176', '329179', '332299', '332576', '332583', '332920', '335386', '337588', '337590', '337592', '337593', '337598', '337600', '337601', '338109', '338114', '338152', '344555', '344556', '344560', '345689', '345700', '345736', '346748', '346749', '347951']
    nosy_count = 13.0
    nosy_names = ['orsenthil', 'vstinner', 'larry', 'benjamin.peterson', 'ned.deily', 'lukasz.langa', 'martin.panter', 'serhiy.storchaka', 'miss-islington', 'Windson Yang', 'xtreak', '\xe8\xa5\xbf\xe7\x94\xb0\xe9\x9b\x84\xe6\xb2\xbb', 'rschiron']
    pr_nums = ['10258', '12260', '12261', '12279', '12281', '13426']
    priority = 'release blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'security'
    url = 'https://bugs.python.org/issue35121'
    versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8']

    @bobunderson
    Copy link
    Mannequin Author

    bobunderson mannequin commented Oct 31, 2018

    http.cookiejar.DefaultPolicy.domain_return_ok returns incorrect results.

    So, HTTP clients send cookies which issued from wrong server.

    policy = http.cookiejar.DefaultCookiePolicy()
    req = urllib.request.Request('https://xxxfoo.co.jp/')
    print(policy.domain_return_ok('foo.co.jp', req)   # should be False, but it returns True

    @bobunderson bobunderson mannequin added type-bug An unexpected behavior, bug, or error 3.7 (EOL) end of life stdlib Python modules in the Lib dir labels Oct 31, 2018
    @tirkarthi
    Copy link
    Member

    The current set of tests are at

    def test_domain_return_ok(self):
    . A simple set of tuple that can be added based on the report as below :

    ("http://barfoo.com", ".foo.com", False)
    ("http://barfoo.com", "foo.com", False) # Fails on master

    The check is done at

    if not (req_host.endswith(domain) or erhn.endswith(domain)):
    . There is no check to add '.' before domain if absent. Hence it performs a substring match with the values req_host = ".barfoo.com" and erhn = ".barfoo.com" and domain = "foo.com" so the condition not (req_host.endswith(domain) or erhn.endswith(domain)) fails and doesn't return False. I would suggest adding a check to make sure domain also starts with '.' similar to req_host and erhn thus fixing the issue. I tried the fix and existing tests along with the reported case works fine.

    diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py
    index 0ba8200f32..da7462701b 100644
    --- a/Lib/http/cookiejar.py
    +++ b/Lib/http/cookiejar.py
    @@ -1173,6 +1173,8 @@ class DefaultCookiePolicy(CookiePolicy):
                 req_host = "."+req_host
             if not erhn.startswith("."):
                 erhn = "."+erhn
    +        if not domain.startswith("."):
    +            domain = "."+domain
             if not (req_host.endswith(domain) or erhn.endswith(domain)):
                 #_debug("   request domain %s does not match cookie domain %s",
                 #       req_host, domain)

    ("http://barfoo.com", ".foo.com", False)
    ("http://barfoo.com", "foo.com", False) # Tests pass with fix

    Also tried the script attached in the report

    $ cat ../backups/bpo35121.py
    import urllib
    from http.cookiejar import DefaultCookiePolicy
    
    policy = DefaultCookiePolicy()
    req = urllib.request.Request('https://xxxfoo.co.jp/')
    print(policy.domain_return_ok('foo.co.jp', req))

    # without fix

    $ ./python.exe ../backups/bpo35121.py
    True

    # With domain fix

    $ ./python.exe ../backups/bpo35121.py
    False

    The check was added in 2004 with commit 2a6ba90 . If my fix is correct I am willing to raise a PR for this with test.

    Hope it helps!

    @bobunderson
    Copy link
    Mannequin Author

    bobunderson mannequin commented Oct 31, 2018

    I think that is desired result. thanks!

    @tirkarthi
    Copy link
    Member

    Thanks for the confirmation. I have created a PR (#10258) with test and added 3.8 as affected version. Please add in if I have missed anything in the PR.

    @tirkarthi tirkarthi added the 3.8 only security fixes label Oct 31, 2018
    @Windsooon
    Copy link
    Mannequin

    Windsooon mannequin commented Nov 3, 2018

    I wonder https://github.com/python/cpython/blob/master/Lib/test/test_http_cookiejar.py#L420

    ("http://foo.bar.com/", "com", True),
    ("http://foo.com/", "com", True),

    are expected behavior?

    @tirkarthi
    Copy link
    Member

    Good catch Windson! I overlooked the tests. There is also a comment that it's liberal in the test function. Since the code was added in 2006 I don't if it's ok broken to fix this or not. I will let the reviewers take a call then. There is also domain_match and user_domain_match that perform more tests.

    This is only a rough check for performance reasons, so it's not too critical as long as it's sufficiently liberal.

    Thanks for your input!

    @tirkarthi
    Copy link
    Member

    Looking further into this the domain validation makes it little more stricter and can have wider implications. For example requests library uses cookiejar to maintain cookies between sessions. One more case is that domain can be empty so only non-empty domains can be prefixed with dot.

    A simple server that sets Cookie with value A=LDJDSFLKSDJLDSF

    import SimpleHTTPServer
    import logging
    
    class MyHTTPRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
        def do_GET(self):
            self.cookieHeader = self.headers.get('Cookie')
            SimpleHTTPServer.SimpleHTTPRequestHandler.do_GET(self)
    
        def end_headers(self):
            self.send_my_headers()
            SimpleHTTPServer.SimpleHTTPRequestHandler.end_headers(self)
    
        def send_my_headers(self):
            self.send_header('Set-Cookie', 'A=LDJDSFLKSDJLDSF')
    
    if __name__ == '__main__':
        SimpleHTTPServer.test(HandlerClass=MyHTTPRequestHandler)

    Add below host entry to /etc/hosts

    127.0.0.1 test.com
    127.0.0.1 1.test.com
    127.0.0.1 footest.com

    Sample script to demonstrate requests behavior change

    import requests
    
    with requests.Session() as s:
        cookies = dict(cookies_are='working')
        m = s.get("http://test.com:8000", cookies=cookies)
        print(m.request.headers)
        m = s.get("http://1.test.com:8000", cookies=cookies)
        print(m.request.headers)
        m = s.get("http://footest.com:8000", cookies=cookies)
        print(m.request.headers)

    Before patch :

    {'User-Agent': 'python-requests/2.11.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/', 'Connection': 'keep-alive', 'Cookie': 'cookies_are=working'}
    {'User-Agent': 'python-requests/2.11.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/
    ', 'Connection': 'keep-alive', 'Cookie': 'A=LDJDSFLKSDJLDSF; cookies_are=working'}
    {'User-Agent': 'python-requests/2.11.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Cookie': 'A=LDJDSFLKSDJLDSF; cookies_are=working'}

    After patch :

    {'User-Agent': 'python-requests/2.11.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/', 'Connection': 'keep-alive', 'Cookie': 'cookies_are=working'}
    {'User-Agent': 'python-requests/2.11.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/
    ', 'Connection': 'keep-alive', 'Cookie': 'A=LDJDSFLKSDJLDSF; cookies_are=working'}
    {'User-Agent': 'python-requests/2.11.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Cookie': 'cookies_are=working'}

    As with my patch since the cookie is set on test.com while making a request to footest.com the cookie is skipped as part of the patch since footest is not a subdomain of test.com but 1.test.com is a subdomain. This is a behavior change to be decided whether worth doing or to document this since in a client with session like requests module connecting to lot of hosts this can potentially pass cookies of test.com to footest.com. A discussion on requests repo on providing the option for user to set a stricter cookie policy : psf/requests#2576

    On testing with curl cookie-jar it seems that the cookies are passed even for the subdomain only when it's set and not as part of top level domain.

    @serhiy-storchaka serhiy-storchaka added release-blocker type-security A security issue and removed type-bug An unexpected behavior, bug, or error labels Dec 24, 2018
    @tirkarthi
    Copy link
    Member

    Also looking at the docs for different frameworks like Flask and Django they recommend setting Domain attribute only for cross-domain cookies.

    From Django docs

    Use domain if you want to set a cross-domain cookie. For example, domain="example.com" will set a cookie that is readable by the domains www.example.com, blog.example.com, etc. Otherwise, a cookie will only be readable by the domain that set it.

    When there is no domain specified then the frameworks seem to set the cookie only for the host only as per RFC 6265. So domain attribute is set all the time and it's just that if the domain attribute is set explicitly by the server with the set_cookie function in the frameworks then the cookiejar has domain_specified set along with dot prefixed for the domain enabling stricter validations. I don't know about the metrics of setting the domain attribute vs not setting it. Checking with a simple Flask app and set_cookie without domain parameter the cookies are passed to suffix domains. With domain passed to set_cookie has dot prefixed and the cookies are not passed to suffix domains.

    I also looked into other implementations

    @tirkarthi
    Copy link
    Member

    I have come across another behavior change between path checks while using the cookie jar implementation available in Python. This is related to incorrect cookie validation but with respect to path so let me know if this needs a separate ticket. I observed the following difference :

    1. Make a request to "/" that sets a cookie with "path=/any"
    2. Make a request to "/any" and the set cookie is passed since the path matches
    3. Make a request to "/anybad" and cookie with "path=/any" is also passed too.

    On using golang stdlib implementation of cookiejar the cookie "path=/any" is not passed when a request to "/anybad" is made. So I checked with RFC 6265 where the path match check is defined in section-5.1.4 . RFC 6265 also obsoletes RFC 2965 upon which cookiejar is based I hope since original implementation of cookiejar is from 2004 and RFC 6265 was standardized later. So I think it's good to enforce RFC 6265 since RFC 2965 is obsolete at least in Python 3.8 unless this is considered as a security issue. I think this is a security issue. The current implementation can potentially cause cookies to be sent to incorrect paths in the same domain that share the same prefix. This is a behavior change with more strict checks but I could see no tests failing with RFC 6265 implementation too.

    RFC 2965 also gives a loose definition of path-match without mentioning about / check in the paths based on which Python implementation is based as a simple prefix match.

    For two strings that represent paths, P1 and P2, P1 path-matches P2
    if P2 is a prefix of P1 (including the case where P1 and P2 string-
    compare equal). Thus, the string /tec/waldo path-matches /tec.

    RFC 6265 path-match definition : https://tools.ietf.org/html/rfc6265#section-5.1.4

    A request-path path-matches a given cookie-path if at least one of
    the following conditions holds:

    o The cookie-path and the request-path are identical.

    o The cookie-path is a prefix of the request-path, and the last
    character of the cookie-path is %x2F ("/").

    o The cookie-path is a prefix of the request-path, and the first
    character of the request-path that is not included in the cookie-
    path is a %x2F ("/") character.

    The current implementation in cookiejar is as below :

    def path_return_ok(self, path, request):
        _debug("- checking cookie path=%s", path)
        req_path = request_path(request)
        if not req_path.startswith(path):
            _debug("  %s does not path-match %s", req_path, path)
            return False
        return True

    Translating the RFC 6265 steps (a literal translation of go implementation) would have something like below and no tests fail on master. So the python implementation goes in line with the RFC not passing cookies of "path=/any" to /anybody

    def path_return_ok(self, path, request):
        req_path = request_path(request)
        if req_path == path:
            return True
        elif req_path.startswith(path) and ((path.endswith("/") or req_path[len(path)] == "/")):
            return True
            
        return False

    The golang implementation is as below which is a literal translation of RFC 6265 steps at https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/net/http/cookiejar/jar.go#L130

    // pathMatch implements "path-match" according to RFC 6265 section 5.1.4.
    func (e *entry) pathMatch(requestPath string) bool {
    if requestPath == e.Path {
    return true
    }
    if strings.HasPrefix(requestPath, e.Path) {
    if e.Path[len(e.Path)-1] == '/' {
    return true // The "/any/" matches "/any/path" case.
    } else if requestPath[len(e.Path)] == '/' {
    return true // The "/any" matches "/any/path" case.
    }
    }
    return false
    }

    Feel free to correct me if I am wrong on the above.

    @tirkarthi
    Copy link
    Member

    I have opened bpo-35647 for path related checks as a separate report.

    @tirkarthi
    Copy link
    Member

    This issue affects 2.7 as well along with 3.4 and 3.5. The initial report was notified to security@python.org . 2.7.16 release candidate dates were announced at https://mail.python.org/pipermail/python-dev/2019-February/156266.html. I have prepared an initial backport of this with tests for 2.7 at 2.7...tirkarthi:bpo-35121-27 . Serhiy has approved the PR for master. I have added notes here and on the PR about the issue and implementation in other languages. It would be helpful if someone can double check my analysis since cookiejar has not received much change over the years.

    If this is a potential candidate for 2.7 release I can help with that once the changes are merged to master. Adding Benjamin Peterson to this issue to take a call on if it needs to be backported to 2.7. If it's planned for a backport then also to decide on priority if this needs to be part of 2.7.16 or later release.

    @ned-deily
    Copy link
    Member

    New changeset ca7fe50 by Ned Deily (Xtreak) in branch 'master':
    bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258)
    ca7fe50

    @ned-deily
    Copy link
    Member

    New changeset e5123d8 by Ned Deily (Miss Islington (bot)) in branch '3.7':
    bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) (GH-12261)
    e5123d8

    @ned-deily
    Copy link
    Member

    New changeset b241af8 by Ned Deily (Miss Islington (bot)) in branch '3.6':
    bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) (GH-12260)
    b241af8

    @ned-deily
    Copy link
    Member

    OK, thanks for the initial report and a big thank you to @XTreak for the PR and following up on things. The PR is now merged to master for 3.8.0 and backported as a security fix for release in 3.7.3 and 3.6.9. Reassigning to Benjamin for a decision on backporting to 2.7.

    @serhiy-storchaka
    Copy link
    Member

    What about 3.4 and 3.5?

    @tirkarthi
    Copy link
    Member

    From my initial tests 3.4 and 3.5 were also affected. 3.4 is going EoL and RC1 is out but there is one another security issue (bpo-36216) fixed last week with a PR open. If the merge window is open and Larry is okay then I can raise backport PRs if needed. There are less changes made to cookiejar and cherry_picker would also work fine as I tried it locally.

    cherry_picker --no-push ca7fe50 3.5
    🐍 🍒 ⛏

    Now backporting 'ca7fe5063593958e5efdf90f068582837f07bd14' into '3.5'
    Switched to a new branch 'backport-ca7fe50-3.5'
    Branch 'backport-ca7fe50-3.5' set up to track remote branch '3.5' from 'upstream'.

    [backport-ca7fe50-3.5 fcb2dd85a0] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258)
    Author: Xtreak <tir.karthi@gmail.com>
    Date: Sun Mar 10 07:39:48 2019 +0530
    3 files changed, 45 insertions(+), 2 deletions(-)
    create mode 100644 Misc/NEWS.d/next/Security/2018-10-31-15-39-17.bpo-35121.EgHv9k.rst

    Finished cherry-pick ca7fe50 into backport-ca7fe50-3.5 😀

    cherry_picker --no-push ca7fe50 3.4
    🐍 🍒 ⛏

    Now backporting 'ca7fe5063593958e5efdf90f068582837f07bd14' into '3.4'
    Switched to a new branch 'backport-ca7fe50-3.4'
    Branch 'backport-ca7fe50-3.4' set up to track remote branch '3.4' from 'upstream'.

    Performing inexact rename detection: 100% (639108/639108), done.
    [backport-ca7fe50-3.4 46ea57d6b3] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258)
    Author: Xtreak <tir.karthi@gmail.com>
    Date: Sun Mar 10 07:39:48 2019 +0530
    3 files changed, 45 insertions(+), 2 deletions(-)
    create mode 100644 Misc/NEWS.d/next/Security/2018-10-31-15-39-17.bpo-35121.EgHv9k.rst

    Finished cherry-pick ca7fe50 into backport-ca7fe50-3.4 😀

    @tirkarthi
    Copy link
    Member

    There are many libraries that use DefaultCookiePolicy and requests library uses it for client where session state needs to be maintained across different requests. Currently, requests doesn't have a documented API to change to cookiejar policy and were not keen on introducing a custom one since this might introduce maintenance burden over keeping it in sync with other changes when made upstream. The team have been informed about this when the issue was created and I also updated the maintainers now about the fix being merged since it's a highly popular library.

    So requests will remain affected when ran on versions where this patch is not available in CPython standard library as of now. A potentially simple workaround in the absence of patch on affected versions is to set DomainStrict in the cookiepolicy that would make sure a literal match against domain is made at [0] . The disadvantage I guess would be that cookie set on example.com would not be shared with subdomain which might break workflow. aio-http was not affected since it uses custom cookiejar policy. scrapy also seems to be not affected since they custom policies. Most of the web frameworks don't recommend setting domain explicitly and set them implicitly so it can be reproduced in the default setup of frameworks and Flask was the one I tested which makes me assume this could be easily exploited.

    [0]

    (self.strict_ns_domain & self.DomainStrictNonDomain) and

    @larryhastings
    Copy link
    Contributor

    New changeset 42ad410 by larryhastings (Xtreak) in branch '3.4':
    [3.4] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) (bpo-12279)
    42ad410

    @larryhastings
    Copy link
    Contributor

    New changeset 4749f1b by larryhastings (Xtreak) in branch '3.5':
    [3.5] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) (bpo-12281)
    4749f1b

    @tirkarthi
    Copy link
    Member

    Larry, I am reopening this since this seems to affects 2.7 and would wait for Benjamin's call on backporting this.

    @tirkarthi tirkarthi reopened this Mar 18, 2019
    @vstinner
    Copy link
    Member

    vstinner commented Jun 4, 2019

    I added this issue to my security website:
    https://python-security.readthedocs.io/vuln/cookie-domain-check.html

    So it's fixed in Python 3.4.10, 3.5.7 and 3.7.3. Right now, 2.7 and 3.6 are vulnerable (but 3.6 branch is fixed).

    @vstinner
    Copy link
    Member

    vstinner commented Jun 4, 2019

    Can someone try to backport the fix to Python 2.7?

    @tirkarthi
    Copy link
    Member

    Can someone try to backport the fix to Python 2.7?

    The backport to 2.7 PR 13426 is open. It would be helpful if someone can review it. I am not sure of the commit review process and who needs to review and approve it since this is assigned to Benjamin Peterson. It's a fairly straightforward backport except that http.cookiejar is cookielib in Python 2.

    @miss-islington
    Copy link
    Contributor

    New changeset 979daae by Miss Islington (bot) (Xtreak) in branch '2.7':
    [2.7] bpo-35121: prefix dot in domain for proper subdomain validation (GH-10258) (GH-13426)
    979daae

    @tirkarthi
    Copy link
    Member

    Closing this as resolved since the fix was merged to all branches. Thank you all.

    @vstinner
    Copy link
    Member

    Again, well done Karthikeyan Singaravelan!

    @ret2libc
    Copy link
    Mannequin

    ret2libc mannequin commented Jun 27, 2019

    Did anybody request a CVE for this issue? I think it deserves one as it is a security issue and it may leak cookies to wrong domains. Does anybody have anything against assigning a CVE to this issue? If not, I would try to get one from MITRE.

    @tirkarthi
    Copy link
    Member

    I also reported it to security@python.org . Please check with them too to see if there is a CVE request already made. Thanks.

    @ret2libc
    Copy link
    Mannequin

    ret2libc mannequin commented Jul 15, 2019

    CVE-2018-20852 has been assigned to this flaw.

    @vstinner vstinner changed the title Cookie domain check returns incorrect results [ CVE-2018-20852] Cookie domain check returns incorrect results Jul 15, 2019
    @vstinner vstinner changed the title [ CVE-2018-20852] Cookie domain check returns incorrect results [CVE-2018-20852] Cookie domain check returns incorrect results Jul 15, 2019
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes release-blocker stdlib Python modules in the Lib dir type-security A security issue
    Projects
    None yet
    Development

    No branches or pull requests

    7 participants