classification
Title: socket.getfqdn() doesn't cope properly with purely DNS-based setups
Type: behavior Stage: needs patch
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7, Python 3.6, Python 3.3, Python 3.4, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: James Shewey, Stijn.Hoop, Thomas.Waldmann, ankitoshniwal, bulb, christian.heimes, devurandom, dfranke, loewis, mcjeff, r.david.murray
Priority: normal Keywords: patch

Created on 2009-01-19 22:29 by dfranke, last changed 2020-06-29 14:51 by bulb.

Files
File name Uploaded Description Edit
python2.7-socket-getfqdn.patch Stijn.Hoop, 2013-04-18 10:34
python5004-test.c Stijn.Hoop, 2013-04-18 14:31 test C program showing various gethost function output on my system
Messages (18)
msg80216 - (view) Author: Daniel Franke (dfranke) Date: 2009-01-19 22:29
On Linux and presumably on other POSIX-like systems, socket.getfqdn()
doesn't work if a system resolves its own FQDN using DNS rather than
/etc/hosts.

My system's FQDN is 'fugue.tank.wellohorld.com'.  My /etc/hosts is empty
except for loopback entries, and /etc/resolv.conf contains the line
'domain tank.wellohorld.com'.  This is sufficient for 'hostname -f' to
do the Right Thing, but socket.getfqdn() simply returns 'fugue':

dfranke@fugue:~/Python-2.6.1$ hostname
fugue
dfranke@fugue:~/Python-2.6.1$ hostname -f
fugue.tank.wellohorld.com
dfranke@fugue:~/Python-2.6.1$ ./python
Python 2.6.1 (r261:67515, Jan 19 2009, 13:56:59)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'fugue'
>>>
dfranke@fugue:~/Python-2.6.1$ echo -e '$a\n172.17.0.120
fugue.tank.wellohorld.com fugue\n.\nwq' | sudo ed /etc/hosts
305
350
dfranke@fugue:~/Python-2.6.1$ ./python
Python 2.6.1 (r261:67515, Jan 19 2009, 13:56:59)
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'fugue.tank.wellohorld.com'
>>>
dfranke@fugue:~/Python-2.6.1$
msg109635 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-08 22:44
Would someone with appropriate knowledge please take a look to see if this is still an issue.
msg109647 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2010-07-09 00:04
I think anybody willing to invest the time could acquire the appropriate knowledge, at least to determine whether it's still an issue (i.e. trying to reproduce it). To fix it, one would then need to read the source code of hostname, and find out what they do differently; strace might be sufficient already.
msg162508 - (view) Author: Ankit Toshniwal (ankitoshniwal) Date: 2012-06-07 23:42
I cannot reproduce this issue. I just tested this on my mac.

atoshniw@prusev-mn:~/Documents/code/python-dev/bin #hostname -f
prusev-mn.helloworld.com
atoshniw@prusev-mn:~/Documents/code/python-dev/bin #python
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'prusev-mn.helloworld.com'
msg173987 - (view) Author: Jeff McNeil (mcjeff) * Date: 2012-10-28 01:32
Gave this a go myself...

$ ./python
Python 3.4.0a0 (default:57a33af85407, Oct 27 2012, 21:26:30) 
[GCC 4.4.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'host.domain.com'
>>> 

$ hostname -f
host.domain.com

$ cat /etc/hosts
127.0.0.1       localhost.localdomain   localhost

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Linux host 3.5.2-x86_64 #1 SMP Wed Aug 15 14:31:07 EDT 2012 x86_64 GNU/Linux

According to strace, both rely on DNS:

recvfrom(3, "Wj\201\200\0\1\0\1\0\5\0\0\00219\003134\003230\003173\7in-a"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("1.2.3.4")}, [16]) = 176

Same behavior on both 2.6 & hg tip.  I think this is a non-issue.
msg187065 - (view) Author: Stijn Hoop (Stijn.Hoop) Date: 2013-04-16 10:35
Still seeing this on Fedora 18 / Python 2.7.3.

I only have loopback in /etc/hosts

[TUE\shoop@pclin281] <~> cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

I search in .campus.tue.nl and .win.tue.nl:

[TUE\shoop@pclin281] <~> grep search /etc/resolv.conf 
search campus.tue.nl. win.tue.nl.

Hostname -f reliably returns .campus.tue.nl as it should

[TUE\shoop@pclin281] <~> hostname -f
pclin281.campus.tue.nl
[TUE\shoop@pclin281] <~> hostname -f   
pclin281.campus.tue.nl

But socket.getfqdn disagrees, even with itself when run multiple times:

[TUE\shoop@pclin281] <~> python
Python 2.7.3 (default, Aug  9 2012, 17:23:57) 
[GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getfqdn()
'pclin281'
>>> socket.getfqdn()
'pclin281.win.tue.nl'
>>> socket.getfqdn()
'pclin281'
>>> socket.getfqdn()
'pclin281.win.tue.nl'
>>> 

Note that pclin281.win.tue.nl is in fact also a valid DNS entry, but not one that I would expect the function to ever return given the search order.
msg187098 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-16 15:44
Note that socket.getfqdn is a wrapper around a couple of socket calls that are just wrappers of OS level socket calls.  If you take a look at socket.py you'll see the definition.  As Martin said earlier, if you (or anyone else) can figure out what hostname does differently and suggest how to patch our getfqdn method to behave similarly, I'm sure the patch will be welcome.

Unfortunately there won't be any good way to write a test for this.
msg187234 - (view) Author: Stijn Hoop (Stijn.Hoop) Date: 2013-04-18 10:14
OK, fair enough.

From reading sources, it appears that hostname is using getaddrinfo(3) on kernelhostname with hints->ai_flags & AI_CANONNAME, while Lib/socket.py simply uses gethostbyaddr(kernelhostname), and falls back on kernelhostname in case of errors.

Unfortunately I am not entirely sure who is "correct" here, as I don't know the intent of socket.getfqdn().

In my case, kernelhostname is set to 'pclin281' e.g. without the dots. I believe this to be correct, but I know that this is already "controversial" as in there exists software that expects an FQDN there, and internet folklore is split about 50/50 about this necessity.

Then, apparently, there is confusion about AI_CANONNAME and what it actually should do. glibc upstream does address lookups but Fedora patches this out. See this recent glibc bug for more pointers:

http://sourceware.org/bugzilla/show_bug.cgi?id=15218

As mentioned in that bug, a lot of software runs on Fedora and works using that definition of AI_CANONNAME.

However, switching Lib/socket.py / getfqdn from gethostbyaddr to getaddrinfo might have more implications than just fixing this case. I can try to write a patch, but is this the right direction?
msg187237 - (view) Author: Stijn Hoop (Stijn.Hoop) Date: 2013-04-18 10:34
Attached is a very lightly tested patch that matches hostname -f behaviour on my system. I suspect this should be OK but it definitely needs more testing than just my system...
msg187238 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-18 11:50
The problem with your patch is that it changes the (effective) meaning of the 'name' parameter.  Before the patch, name can be an IP address.  After the patch, that will fail on Fedora.  (It also fails on my Gentoo system).

It is interesting to note, as well, that the documentation for gethostbyaddr says that it is obsolete and getaddrinfo should be used instead.

Could we use the getaddrinfo call if we don't get an FQDN back from gethostbyaddr?  It doesn't look like that would completely solve your problem, though, given your example output.  Have you figured out why that is happening? 

Alternatively, perhaps we could fall back to gethostbyaddr if we don't get an fqdn from the getaddrinfo call.  

However, given that the documentation actually specifies the algorithm used by getfqdn, I'm not sure if we can make either change in a bugfix version.
msg187253 - (view) Author: Stijn Hoop (Stijn.Hoop) Date: 2013-04-18 14:31
OK, dumping my current findings here, as I'm still not sure what the expected results should be.

First of all, Lib/socket.py calls gethostbyaddr with a name. As in, gethostby _ADDR_ with a name.

This works because Modules/socketmodule.c internally uses setipaddr() to resolve the name to an address. setipaddr() does this using a call to getaddrinfo() with hints.ai_family == AF_UNSPEC and no further flags.

On my system (confirmed using the test program attached) this results in SIX entries, and this is the part that confused me.

Due to virtualization I have a virtual bridge virbr0 configured with an internal IP address 192.168.122.1, as well as my LAN-connected bridge br0 with IP address 131.155.71.8. Both of these addresses are returned in the call to getaddrinfo() (each one 3 times), but NOT ALWAYS IN THE SAME ORDER.

And this is the clue as to why python's socket.getfqdn() does not behave consistently. For 192.168.122.1 does not resolve to anything, hence it will return "pclin281". And 131.155.71.8 will backwards resolve to pclin281.win.tue.nl as the PTR record points to that entry.

Now, again, I'm not entirely sure what to do here. I agree that this is not a simple bugfix. I also think that, apart from the weirdness of getaddrinfo() return order, socket.getfqdn() is doing it's documented job of returning /an/ FQDN for a given host.

But in case of the guaranteed LOCAL canonical hostname, another function is warranted, imho.

Does this make sense?

For the record, output of a given run on my system:

[TUE\shoop@pclin281] <~/tmp> ./test
gai canon result 0: pclin281.campus.tue.nl 192.168.122.1
gai canon result 1: (null) 131.155.71.8
gai result 0: (null) 131.155.71.8
gai result 1: (null) 131.155.71.8
gai result 2: (null) 131.155.71.8
gai result 3: (null) 192.168.122.1
gai result 4: (null) 192.168.122.1
gai result 5: (null) 192.168.122.1
ghbn result 0 h_name: pclin281.campus.tue.nl
ghbn result 0 h_alias: __NONE__
ghbn result 1 h_name: pclin281.campus.tue.nl
ghbn result 1 h_alias: __NONE__
ghbn result 2 h_name: pclin281.campus.tue.nl
ghbn result 2 h_alias: __NONE__
ghbn result 3 h_name: pclin281.campus.tue.nl
ghbn result 3 h_alias: __NONE__
ghbn result 4 h_name: pclin281.campus.tue.nl
ghbn result 4 h_alias: __NONE__
ghbn result 5 h_name: pclin281.campus.tue.nl
ghbn result 5 h_alias: __NONE__
ghbn result 6 h_name: pclin281.campus.tue.nl
ghbn result 6 h_alias: __NONE__
ghbn result 7 h_name: pclin281.campus.tue.nl
ghbn result 7 h_alias: __NONE__
ghbn result 8 h_name: pclin281.campus.tue.nl
ghbn result 8 h_alias: __NONE__
ghbn result 9 h_name: pclin281.campus.tue.nl
ghbn result 9 h_alias: __NONE__
msg187256 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-18 14:52
Yeah, a new function was a thought that had crossed my mind as well.  getfqdnbyname, maybe?  Or gethostnamefqdn?  Then deprecate calling getfqdn without an argument.

I agree that gethostbyaddr accepting a non-IP is weird.  I have no idea why it was implemented that way, much less why it is *used* that way.  It's been that way for a long time, though.
msg187350 - (view) Author: Stijn Hoop (Stijn.Hoop) Date: 2013-04-19 09:09
So after a good nights sleep: does it not make sense to use the canonical hostname iff the name argument is not present / empty? Otherwise, fall back to the documented steps? That way extra API is avoided, and I can't think of a case where you would rather have my weird results vs "the output of hostname -f".
msg187355 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-19 12:33
That is an interesting proposal, yes.  I suppose someone that needs the getaddrinfo semantics for something other than the local host can just call it directly.

Now, do we add the fact that we are doing this to the current alogarithmic documention? :)
msg297435 - (view) Author: James Shewey (James Shewey) Date: 2017-06-30 17:33
According to the man page for gethostbyaddr "The gethostbyname*() and gethostbyaddr*() functions are obsolete. Applications should use getaddrinfo(3) and getnameinfo(3) instead." - so perhaps using the correct API call might be a good start to resolving this issue, but I found that in my case, I needed to chase the problem upstream instead of downstream. On my Red Hat box, the kernel.hostname value with sysctl was incorrect. I had to re-set it with a  sysctl kernel.hostname=hostname.example.com. This overrides /etc/hosts, so I suspect this is probably not an issue on other distros that do not use sysctl.

The moral of the story being garbage in, garbage out.
msg300593 - (view) Author: (devurandom) Date: 2017-08-20 02:22
In my case, /etc/hostname, /proc/sys/kernel/hostname, `uname -n`, `hostname -f` all show the same FQDN, but `python -c 'import socket ; print(socket.getfqdn())'` still prints the short hostname.  /etc/hosts is empty except for localhost.  /etc/nsswitch.conf contains:
hosts:       files mymachines resolve [!UNAVAIL=return] dns myhostname
msg308979 - (view) Author: Thomas Waldmann (Thomas.Waldmann) Date: 2017-12-24 03:58
Embarassing as always to stumble over some stuff and then find a 9y old ticket here, where it is discussed and even (almost) solved.

Our ticket: https://github.com/borgbackup/borg/issues/3471

Fixed getfqdn we use now instead of socket.getfqdn():

https://github.com/borgbackup/borg/pull/3484/commits/9b0d0f3127289fc3bbe620e8adce8038ed594e9f#diff-4b53f84e19a3bb376bf2202371ed269aR188

Note: no "else: name = hostname" at the end, that is a bug in the patch attached to this ticket (hostname is undefined after applying the patch).
msg372583 - (view) Author: Jan Hudec (bulb) Date: 2020-06-29 14:51
Confirming the fixed version linked in previous comment by Thomas Waldmann is correct and matches what `hostname -f` does.
History
Date User Action Args
2020-06-29 14:51:00bulbsetnosy: + bulb

messages: + msg372583
versions: + Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9, Python 3.10
2017-12-24 17:44:19christian.heimessetnosy: + christian.heimes
2017-12-24 03:58:13Thomas.Waldmannsetnosy: + Thomas.Waldmann
messages: + msg308979
2017-08-20 02:22:32devurandomsetmessages: + msg300593
2017-08-20 02:18:17devurandomsetnosy: + devurandom
2017-06-30 17:33:51James Sheweysetnosy: + James Shewey
messages: + msg297435
2014-02-03 18:32:18BreamoreBoysetnosy: - BreamoreBoy
2013-04-19 12:33:06r.david.murraysetmessages: + msg187355
2013-04-19 09:09:27Stijn.Hoopsetmessages: + msg187350
2013-04-18 14:52:45r.david.murraysetmessages: + msg187256
2013-04-18 14:31:32Stijn.Hoopsetfiles: + python5004-test.c

messages: + msg187253
2013-04-18 11:50:04r.david.murraysetmessages: + msg187238
2013-04-18 10:34:17Stijn.Hoopsetfiles: + python2.7-socket-getfqdn.patch
keywords: + patch
messages: + msg187237
2013-04-18 10:14:07Stijn.Hoopsetmessages: + msg187234
2013-04-16 15:44:37r.david.murraysetnosy: + r.david.murray

messages: + msg187098
versions: + Python 3.3, Python 3.4, - Python 3.1, Python 3.2
2013-04-16 10:35:25Stijn.Hoopsetnosy: + Stijn.Hoop
messages: + msg187065
2012-10-28 01:32:48mcjeffsetnosy: + mcjeff
messages: + msg173987
2012-06-07 23:42:54ankitoshniwalsetnosy: + ankitoshniwal
messages: + msg162508
2010-07-09 00:04:39loewissetmessages: + msg109647
2010-07-08 23:40:28pitrousetnosy: + loewis
2010-07-08 22:44:34BreamoreBoysetversions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6
nosy: + BreamoreBoy

messages: + msg109635

components: + Library (Lib), - Extension Modules
stage: needs patch
2009-01-19 22:33:59dfrankesetcomponents: + Extension Modules, - Library (Lib)
2009-01-19 22:29:02dfrankecreate