On-going discussion at http://tug.org/pipermail/xetex/2017-December/027212.html has brought new element that polyglossia's French module is broken with xetex since TeXLive2016. We had only one problem, we now have two on our hands.

Possibly Sphinx could be default use babel + French, not polyglossia + French, as the former is maintained but apparently less so the latter.

I tested that TeXLive 2015 (fully updated) and test document showing the https://github.com/sphinx-doc/sphinx/issues/3546 problem now compiles fine if using 

latex_elements = {
    'babel': r'\usepackage{babel}',
}

in conf.py file, to override polyglossia which is default for Sphinx with xelatex.