classification
Title: uuid.getnode() should return the MAC address on Android
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: barry, serhiy.storchaka, xdegaye
Priority: normal Keywords: easy, patch

Created on 2017-12-02 14:29 by xdegaye, last changed 2017-12-07 16:39 by xdegaye. This issue is now closed.

Files
File name Uploaded Description Edit
ip_link.strace xdegaye, 2017-12-06 10:13
ip_link_list.strace xdegaye, 2017-12-06 10:26
archlinux-ip_link.strace xdegaye, 2017-12-07 09:32
archlinux-ip_link_list.strace xdegaye, 2017-12-07 09:35
Pull Requests
URL Status Linked Edit
PR 4696 merged xdegaye, 2017-12-04 11:27
PR 4747 merged python-dev, 2017-12-07 11:59
Messages (19)
msg307432 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-02 14:29
Currently uuid.getnode() returns a random 48-bit number and so the UUIDs are not persistent across time. The reason is that on Android the 'ip link list' command fails.

uuid._ip_getnode() should invoke the 'ip link' command instead.
msg307675 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2017-12-05 19:22
Over in the PR I suggested:

Here's another thought: what if you just added another getter that calls ip link list and placed that after one that calls ip link. Wouldn't that accomplish both goals? Then if ip link fails, we fall back to the old behavior, so nothing changes. It's uglier, but it doesn't special case for the Android platform, and eventually we can decide to remove ip link list altogether.
msg307677 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-12-05 19:26
Why the 'ip link list' command fails on Android at first place? Does Android use its own independent implementation? Or its version is based on the fork of very old version of iproute2 that didn't supported the list command (if there was such version)?
msg307688 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-05 21:28
The result of various 'ip' commands on Android, the last 'ip link list' command is run as root and succeeds (did not think about trying that before):

generic_x86_64:/data/local/tmp/python $ ip link list
request send failed: Permission denied

1|generic_x86_64:/data/local/tmp/python $ ip link help
request send failed: Permission denied

1|generic_x86_64:/data/local/tmp/python $ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000
    link/ether 02:00:00:44:55:66 brd ff:ff:ff:ff:ff:ff
5: hwsim0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ieee802.11/radiotap 12:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
6: sit0: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default 
    link/sit 0.0.0.0 brd 0.0.0.0
8: radio0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 22:d5:92:86:1a:d8 brd ff:ff:ff:ff:ff:ff

generic_x86_64:/data/local/tmp/python # ip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DORMANT group default qlen 1000
    link/ether 02:00:00:44:55:66 brd ff:ff:ff:ff:ff:ff
5: hwsim0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default 
    link/ieee802.11/radiotap 12:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
6: sit0: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default 
    link/sit 0.0.0.0 brd 0.0.0.0
8: radio0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether 22:d5:92:86:1a:d8 brd ff:ff:ff:ff:ff:ff
msg307697 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2017-12-05 22:52
On Dec 5, 2017, at 16:28, Xavier de Gaye <report@bugs.python.org> wrote:
> 
> The result of various 'ip' commands on Android, the last 'ip link list' command is run as root and succeeds (did not think about trying that before):
> 
> generic_x86_64:/data/local/tmp/python $ ip link list
> request send failed: Permission denied
> 
> 1|generic_x86_64:/data/local/tmp/python $ ip link help
> request send failed: Permission denied
> 
> 1|generic_x86_64:/data/local/tmp/python $ ip link
…[output]…

Well, that’s weird!
-B
msg307703 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-12-06 07:02
What if "ip link list" was intentionally prohibited "for security reasons", and "ip link" works just due to oversight? Xavier, could you please inspect the sources of the ip command on Android? Is it the standard iproute2 with additional patches prohibiting the part of the functionality?
msg307704 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-06 07:06
Whatever the change made to fix this issue, it is not possible to add a test case for this change.

So following the suggestion made by Barry in PR 4696, we can add (in another issue) a new keyword parameter to getnode() named 'methods' whose value may be None (the default, meaning try all the known methods) or a tuple containing a subset of the following methods ('unix', 'ifconfig', 'ip', 'arp', 'lanscan', 'netstat',  'random') that would raise an exception if the value cannot be obtained using one of the requested method tried in the requested order. This would also improve the documentation on the methods getnode() is using. Then if we decide to make the change for 'ip link' in the current issue, one can add a test case that would first test for the avaibility of the ip command and if the command exists would fail if getnode(methods=('ip',)) raises an exception.
msg307705 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-06 08:04
You may be right Serhiy. Those tests have been run on the emulator at API 24 (Android 7.0 Nougat, the first API version where SELinux is run in enforced mode) where 'ip link list' fails, but on my device (a Samsung API 21, Android 5.1 Lollipop) running the 'ip link list' (using the termux package installed from google PlayStore) the command is ok.

The Android source of iproute2 can be:
* browsed at: https://android.googlesource.com/platform/external/iproute2/+/master
* cloned from: https://android.googlesource.com/platform/external/iproute2/
They are using iproute2-3.4.0 released in 2012, see README.version.
I checked that the ip/ subdirectory on the Android master branch is unchanged from upstream 3.4.0 at https://www.kernel.org/pub/linux/utils/net/iproute2/.

The Android SELinux policies are at:
https://android.googlesource.com/platform/system/sepolicy/
Does someone know how to read them ?

Maybe we should just close this issue as 'wont fix' then.
msg307706 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-06 08:21
'adb logcat' is a tool that monitors many kind of events on Android. Both ip commands prints a SElinux record on logcat:

Upon the successfull 'ip link' command, logcat prints:
12-06 09:17:24.119  2460  2460 W ip      : type=1400 audit(0.0:8): avc: denied { search } for name="net" dev="vdc" ino=91 scontext=u:r:shell:s0 tcontext=u:object_r:net_data_file:s0 tclass=dir permissive=0

Upon the failed 'ip link' command, logcat prints:
12-06 09:17:42.109  2461  2461 W ip      : type=1400 audit(0.0:9): avc: denied { nlmsg_write } for scontext=u:r:shell:s0 tcontext=u:r:shell:s0 tclass=netlink_route_socket permissive=0
msg307707 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-06 08:24
Oops, the second failed command is 'ip link list' of course.
msg307708 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-06 08:37
It is also possible that rather than an oversight in Android, it is a side effect of SELinux on the implementation of iproute2 if the 'ip link list' command does a little bit more than the 'ip link' command and if this 'little bit more' is prohibited by a SELinux policy. I guess this means diving into the source of iproute2 to confirm that :-(
msg307715 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-06 10:26
Attached ip_link.strace and ip_link_list.strace, the output of strace for the 'ip link' and 'ip link list' commands.

At the end of the process, both commands create an PF_NETLINK socket to receive from the kernel link information through the NETLINK_ROUTE group. The sendto() function fails with EACCES for 'ip link list' and its SELinux avc record relates to this event (the netlink prefix used throughout the documentation [1] is 'nlmsg' and the resource denied in the avc record is 'nlmsg_write'). The SELinux avc record for 'ip link' relates to a failed attempt to open "/data/misc/net/group" (it does not exist).

[1] http://man7.org/linux/man-pages/man7/netlink.7.html
msg307742 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2017-12-06 15:11
On Dec 6, 2017, at 02:06, Xavier de Gaye <report@bugs.python.org> wrote:
> 
> Whatever the change made to fix this issue, it is not possible to add a test case for this change.

Even with say, exception raising mocks for the getters?

> So following the suggestion made by Barry in PR 4696, we can add (in another issue) a new keyword parameter to getnode() named 'methods' whose value may be None (the default, meaning try all the known methods) or a tuple containing a subset of the following methods ('unix', 'ifconfig', 'ip', 'arp', 'lanscan', 'netstat',  'random') that would raise an exception if the value cannot be obtained using one of the requested method tried in the requested order. This would also improve the documentation on the methods getnode() is using. Then if we decide to make the change for 'ip link' in the current issue, one can add a test case that would first test for the avaibility of the ip command and if the command exists would fail if getnode(methods=('ip',)) raises an exception.

I am thinking about this slightly differently.

What if getnode() accepted a `handler` argument and the code was changed to something like this:

1 file changed, 4 insertions(+), 2 deletions(-)
Lib/uuid.py | 6 ++++--

modified   Lib/uuid.py
@@ -656,7 +656,7 @@ def _random_getnode():

 _node = None

-def getnode():
+def getnode(handler=None):
     """Get the hardware address as a 48-bit positive integer.

     The first time this runs, it may launch a separate program, which could
@@ -677,7 +677,9 @@ def getnode():
     for getter in getters + [_random_getnode]:
         try:
             _node = getter()
-        except:
+        except Exception as error:
+            if handler is not None:
+                handler(getter, error)
             continue
         if _node is not None:
             return _node

`handler` could log some diagnostics, reraise the exception, raise StopIteration, etc.  Then we could use that in the test suite too, because we could mock a getter to raise an exception and then pass in a handler that verified the exception was raised with the expected getter.

(Maybe we spell `handler` as `error_handler`.)
msg307743 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2017-12-06 15:15
> Maybe we should just close this issue as 'wont fix' then.

I would be okay with any of these resolutions:

* Close as `wont fix`
* Just call `ip link` (without list)
* Add a new getter such that both `ip link` and `ip link list` are called.

How to handle exceptions in the getters should be addressed in a new issue.
msg307793 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-07 09:35
On archlinux it is easy to know precisely what patches are applied to iproute2 and how it is built (see https://git.archlinux.org/svntogit/packages.git/tree/trunk?h=packages/iproute2).

The attached two files, archlinux-ip_link.strace and archlinux-ip_link_list.strace, contain the output of strace run on the commands 'ip link' and 'ip link list' on archlinux.
* For 'ip link', the sendto() syscall uses RTM_GETLINK to get information about a specific network interface.
* For 'ip link list', this sendto() syscall is preceded by another sendto() syscall using RTM_NEWLINK to *create* information about a specific network interface.

Conclusions:
1) Both commands are not equivalent, this seems to be a bug in iproute2 or its documentation (I did not read the whole iproute2 documentation).
2) By using RTM_NEWLINK, 'ip link list' requests a write-like operation that may be denied by SELinux if there is no policy that allows netlink_route_socket (nlmsg_write). I may be wrong but on Android API 26 it seems that only few processes get that permission: dhcp, clatd, logd, netd, rild, ...
3) From Python perspective it is more robust to call 'ip link' to handle platforms where SELinux is run in enforcing mode.

I will update the PR to do only that change: s/ip link list/ip link/
msg307797 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-07 11:59
New changeset 961dbe0548e26394b7716d41423c61b1e2e58ef7 by xdegaye in branch 'master':
bpo-32199:  The getnode() ip getter now uses 'ip link' instead of 'ip link list' (GH-4696)
https://github.com/python/cpython/commit/961dbe0548e26394b7716d41423c61b1e2e58ef7
msg307801 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-07 12:48
New changeset 03031fbc7d44106d652756462db34eae67de9568 by xdegaye (Miss Islington (bot)) in branch '3.6':
bpo-32199:  The getnode() ip getter now uses 'ip link' instead of 'ip link list' (GH-4696) (#4747)
https://github.com/python/cpython/commit/03031fbc7d44106d652756462db34eae67de9568
msg307815 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2017-12-07 16:26
LGTM, and thanks!
msg307817 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2017-12-07 16:39
Thanks Serhiy and Barry for your comments and reviews :-)
History
Date User Action Args
2017-12-09 15:42:06xdegayelinkissue26865 dependencies
2017-12-07 16:39:39xdegayesetmessages: + msg307817
2017-12-07 16:26:36barrysetmessages: + msg307815
2017-12-07 12:50:06xdegayesetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-12-07 12:48:20xdegayesetmessages: + msg307801
2017-12-07 11:59:42python-devsetpull_requests: + pull_request4650
2017-12-07 11:59:17xdegayesetmessages: + msg307797
2017-12-07 09:35:02xdegayesetfiles: + archlinux-ip_link_list.strace

messages: + msg307793
versions: + Python 3.6
2017-12-07 09:32:19xdegayesetfiles: + archlinux-ip_link.strace
2017-12-06 15:15:44barrysetmessages: + msg307743
2017-12-06 15:11:56barrysetmessages: + msg307742
2017-12-06 10:26:59xdegayesetfiles: + ip_link_list.strace

messages: + msg307715
2017-12-06 10:13:30xdegayesetfiles: + ip_link.strace
2017-12-06 08:37:10xdegayesetmessages: + msg307708
2017-12-06 08:24:02xdegayesetmessages: + msg307707
2017-12-06 08:21:07xdegayesetmessages: + msg307706
2017-12-06 08:04:34xdegayesetmessages: + msg307705
2017-12-06 07:06:23xdegayesetmessages: + msg307704
2017-12-06 07:02:33serhiy.storchakasetmessages: + msg307703
2017-12-05 22:52:19barrysetmessages: + msg307697
2017-12-05 21:28:22xdegayesetmessages: + msg307688
2017-12-05 19:26:48serhiy.storchakasetmessages: + msg307677
2017-12-05 19:22:08barrysetnosy: + barry
messages: + msg307675
2017-12-04 11:27:10xdegayesetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request4609
2017-12-02 18:30:55serhiy.storchakasetkeywords: + easy
nosy: + serhiy.storchaka
2017-12-02 14:29:39xdegayecreate