Ah! the User-Agent (or anything which is in unredirected_hdrs) will not be updated if it has already been set once. 
https://hg.python.org/cpython/file/064f6baeb6bd/Lib/urllib/request.py#l1154


>>> headers = dict(request.unredirected_hdrs)
>>> headers
{'User-agent': 'Python-urllib/3.4', 'Host': '127.0.0.1'}
>>> request.headers
{'User-agent': 'Test-Agent', 'Foo': 'cool'}
>>> headers.update(dict((k, v) for k, v in request.headers.items() if k not in headers))
>>> headers
{'User-agent': 'Python-urllib/3.4', 'Host': '127.0.0.1', 'Foo': 'cool'}