UUID module build fails on FreeBSD since it supports uuid_create function.

The AIX-specific code was contributed by Michael Felt in issue32399.

Michael, does AIX have uint32_t?  If so, we could happily drop the unsigned32 reference.

On 05/01/2018 13:38, Antoine Pitrou wrote:
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> Michael, does AIX have uint32_t?  If so, we could happily drop the unsigned32 reference.

Yes, AIX has unit32_t.

michael@x071:[/home/michael]grep unsigned32 /usr/include/*.h 
/usr/include/sys/*.h | grep typed
/usr/include/uuid.h:typedef u_int32_t unsigned32;      /* positive 32 
bit integer */
/usr/include/uuid.h:typedef unsigned32 boolean32;

>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue32493>
> _______________________________________
>

Seems to be in inttypes.h:
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_71/com.ibm.aix.files/inttypes.h.htm

Perfect. That solves in the process OpenBSD uuid module build too.

On 08/01/2018 12:47, David Carlier wrote:
> David Carlier <dcarlier@afilias.info> added the comment:
>
> Perfect. That solves in the process OpenBSD uuid module build too.
>
> ----------
p.s. I did not 'invent' unsigned32 - just took the syntax from the man 
page: 
https://www.ibm.com/support/knowledgecenter/en/ssw_aix_71/com.ibm.aix.basetrf2/uuid_create.htm 
- and I just did the grep above to show the type existed AND was 
directly related to unsigned32 - even in AIX include files.
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue32493>
> _______________________________________
>

I guessed that :-) I trusted it worked just fine for you. To be honest I know nearly nothing about AIX specificities :-)

New changeset b4ebaa7099c3413b42a97777581c4ca560fe7540 by Antoine Pitrou (David Carlier) in branch 'master':
bpo-32493: Not only AIX, but FreeBSD has uuid_create support (#5089)
https://github.com/python/cpython/commit/b4ebaa7099c3413b42a97777581c4ca560fe7540

Thank you for the fix David!

> New changeset b4ebaa7099c3413b42a97777581c4ca560fe7540 by Antoine Pitrou (David Carlier) in branch 'master':
> bpo-32493: Not only AIX, but FreeBSD has uuid_create support (#5089)

This change introduced a regression on AMD64 FreeBSD 10.x Shared 3.x:

http://buildbot.python.org/all/#/builders/87/builds/463

======================================================================
FAIL: test_uuid1 (test.test_uuid.TestUUIDWithExtModule)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10/build/Lib/test/test_uuid.py", line 320, in test_uuid1
    equal(u.version, 1)
AssertionError: 14 != 1

Might comes from uuid1 function itself ... e.g. line 704 not setting version "field".

Actually, the libc/libuuid function called should be setting the version/variant values as well.

So, I think the AMD64 FreeBSD 10.x Shared 3.x uuid_generate() function is wrong (if that is what it is using - was it/can it also use the uuid_generate* routines?

FYI:

I notice the AIX include file does not actually define the bits for RFC_4122, etc.., but the include file comments does show that the implementation is taking them into account.

/** The strlen() of a UUID string (does not include terminating null) */
#define UUID_STRLEN 36
/**
 * Constant value indicating an as-yet undetermined UUID for a thing.  Spells \c
 * "\0\0UUID\x10_\x80UNKNOWN" in ASCII.  Zeroes at the top make it very unlikely
 * to collide with a uuid_create()d value, which puts a timestamp in that spot.
 * The \c "\x10" and \c "\x80" are reserved/version/mask bits that make this
 * conform to the format expected by the uuid.h API.
 */
#define UUID_UNKNOWN (uuid_t){0x00005555, 0x4944, 0x105F, 0x80, 0x55, {0x4E, 0x4B, 0x4E, 0x4F, 0x57, 0x4E}}
#define UUID_UNKNOWN_STR       "00005555-4944-105f-8055-4e4b4e4f574e"

I wrote a simple program to examine uuid for myself.

Maybe - if you ran a loop (looking at uuid1().variant and uuid1().version you could see if there is an incorrect version showing up. Rather than python - that would seem to be a library function error.

My very simple program:

cat -n ../x4.py
     1  import _uuid
     2  import uuid
     3  _generate_time_safe = _uuid.generate_time_safe
     4  _has_uuid_generate_time_safe = _uuid.has_uuid_generate_time_safe
     5  uuid_time, _ = _generate_time_safe()
     6
     7  uu1a = uuid.UUID(bytes=uuid_time)
     8  uu1b = uuid.uuid1()
     9  uu4 = uuid.uuid4()
    10
    11  print("a)uuid1 = %x" % uu1a.int)
    12  print("b)uuid1 = %x" % uu1b.int)
    13  print("")
    14  print("a uuid1 = ",uu1a.fields, uu1a.version, uu1a.variant)
    15  print("b uuid1 = ",uu1b.fields, uu1b.version, uu1b.variant)
    16  uu1c = uuid.uuid1()
    17  print("c uuid1 = ",uu1c.fields, uu1c.version, uu1c.variant)
    18  print("")
    19  print("  uuid4 = %x" % uu4.int)
    20  print("  uuid4 = ",uu4.fields, uu4.version, uu4.variant)
    21  print("")
    22  uu4 = uuid.uuid4()
    23  print("  uuid4 = %x" % uu4.int)
    24  print("  uuid4 = ",uu4.fields, uu4.version, uu4.variant)

typo here:

So, I think the AMD64 FreeBSD 10.x Shared 3.x uuid_create() function is wrong (if that is what it is using - was it/can it also use the uuid_generate* routines?

i.e., does AMD FreeBSD use uuid_create() or uuid_generate() - or can it use both?

Could someone with AMD try:
Modules/_uuidmodule.c

 +12  static PyObject *
   +13  py_uuid_generate_time_safe(void)
   +14  {
   +15      uuid_t uuid;
   +16  #ifdef HAVE_UUID_GENERATE_TIME_SAFE
   +17      int res;
   +18
   +19      res = uuid_generate_time_safe(uuid);
   +20      return Py_BuildValue("y#i", (const char *) uuid, sizeof(uuid), res);
   +21  #elif HAVE_UUID_GENERATE_TIME
   +22      uuid_generate_time(uuid);
   +23      return Py_BuildValue("y#O", (const char *) uuid, sizeof(uuid), Py_None);
   +24  #elif HAVE_UUID_CREATE
   +25      uint32_t status;
   +26      uuid_create(&uuid, &status);
   +27      return Py_BuildValue("y#i", (const char *) &uuid, sizeof(uuid), (int) status);
   +28  #else
   +29  #error "no uuid library function available"
   +30  #endif
   +31  }

uuid_create only I think.

Gives same outcome but maybe someone else can confir,

On 1/17/2018 11:16 AM, David CARLIER wrote:
> David CARLIER <devnexen@gmail.com> added the comment:
>
> Might comes from uuid1 function itself ... e.g. line 704 not setting version "field".
IMHO: for uuid1 - the version number should be in the 16 bytes filled by 
uuid_create. The implementation of uuid4() in uuid.py "has to" specify 
the attribute by setting the UUID version attribute to 4, so that when 
an uuid4 instance gets asked it's version it says 4. A random number 
gives no such guarantee.

Taking a hint from the FreeBSD man page for uuid_create() - that says it 
provides a DCE 1.1 RPC implementation - I found this link:
http://pubs.opengroup.org/onlinepubs/9629399/apdxa.htm

Therein:

The timestamp is a 60 bit value. For UUID version 1, this is represented 
by Coordinated Universal Time (UTC) as a count of 100-nanosecond 
intervals since 00:00:00.00, 15 October 1582 (the date of Gregorian 
reform to the Christian calendar).

The version number is multiplexed in the 4 most significant bits of the 
*time_hi_and_version* field. UUID version Field 
<http://pubs.opengroup.org/onlinepubs/9629399/apdxa.htm#tagtcjh_35> 
lists currently defined versions of the UUID.

Again - a random number is not likely to correctly set the 4 most significant bits - so another way is needed to make an instance it is version 4.
IMHO - the output of uuid_create should satisfy the needs of uuid version 1.

>
> ----------
> nosy: +David CARLIER2
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue32493>
> _______________________________________
>

In OpenBSD it s even "worse", the version for each call is random (not a surprise when looking at the source). The question is, do we go back to support only AIX or do we accept somehow wrong version ?

Now that it is visible - maybe OpenBSD will treat it as a bug and get 
the "version bits" to work as their man page 
(https://man.openbsd.org/uuid.3) says it does:

  STANDARDS <https://man.openbsd.org/uuid.3#STANDARDS>

The *uuid_compare*(), *uuid_create*(), *uuid_create_nil*(), 
*uuid_equal*(), *uuid_from_string*(), *uuid_hash*(), *uuid_is_nil*(), 
and *uuid_to_string*() functions are compatible with the DCE 1.1 RPC 
specification.

The FreeBSD man page 
(https://www.freebsd.org/cgi/man.cgi?query=uuid_create) says:

*NAME* <https://www.freebsd.org/cgi/man.cgi?query=uuid_create#end>
      *uuid*_*compare*,*uuid*_*create*,	*uuid*_*create*_*nil*,*uuid*_*equal*,*uuid*_*from*_*string*,
      *uuid*_*hash*,	*uuid*_*is*_*nil*,*uuid*_*to*_*string*  -- DCE 1.1 compliant UUID func-
      tions

Re the DCE 1.1 RPC implementation/description - I found this link:
http://pubs.opengroup.org/onlinepubs/9629399/apdxa.htm

And this comment:

The version number is multiplexed in the 4 most significant bits of the 
*time_hi_and_version* field. UUID version Field 
<http://pubs.opengroup.org/onlinepubs/9629399/apdxa.htm#tagtcjh_35> 
lists currently defined versions of the UUID.

*msb1* 	*msb2* 	*msb3* 	*msb4* 	*Version* 	*Description*
0 	0 	0 	1 	1 	DCE version, as specified herein.
0 	0 	1 	0 	2 	DCE Security version, with embedded POSIX UIDs.

So, maybe either, or both: OpenBSD and FreeBSD fix their implementation. 
Problem solved?!

On 1/23/2018 3:34 PM, David CARLIER wrote:
> David CARLIER <devnexen@gmail.com> added the comment:
>
> In OpenBSD it s even "worse", the version for each call is random (not a surprise when looking at the source). The question is, do we go back to support only AIX or do we accept somehow wrong version ?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue32493>
> _______________________________________
>

Probably. Seems workable under Apple otherwise.

We are still seeing the 3.7 and 3.8 FreeBSD buildbot failures Victor identified in msg310147.  Can someone please take this on and produce a PR?

If it is a bug in OpenBSD how do you propose we "fix" this.

imho, the problem is "outside" python.

however, until it is fixed in openbsd, would you accept an 3error macro that prevents the mod from being built on the specific version of openbsd that fails?

oops, sain openbsd, should be freebsd

This problem is still unresolved and causing buildbot failures.  It really should be fixed before we tag 3.7.0rc1.  Serhiy, could you take a look at it, please?  Thanks!

http://buildbot.python.org/all/#/builders/87/builds/1014/steps/4/logs/stdio

I will be busy next few hours, but will take this issue as soon as I can. However I do not promise the result.

Is AIX big-endian?

On *BSD systems uuid_t is a structure of integers with platform-depending endianess. Thus on little-endian platform UUID should be called with the bytes_le argument. This doesn't fix test on OpenBSD and NetBSD, but at least the result is stable (version=4).

Using bytes_le on Linux breaks tests. Seems uuid_generate_time_safe() always returns bytes in big-endian order.

PR 7098 adds _uuid.little_endian which is true on little-endian platforms using uuid_create(), and false otherwise. Actually there are many ways of solving this problem, the choice of this design was arbitrary. _uuid.generate_time_safe() could return a 3-tuple instead of 2-tuple, or there could be two separate functions: _uuid.generate_time_safe() and _uuid.create().

PR 7099 is an alternate solution. I uses uuid_enc_be() which is not part of the DCE 1.1 RPC specification but exists on all *BSD.


Both solutions gives the same result on *BSD and AIX. I have no idea what solution is appropriate for little-endian non-BSD platform (if there is such platform supporting uuid_create()).

New changeset 17d8830312d82e7de42ab89739b0771f712645ff by Victor Stinner (Serhiy Storchaka) in branch 'master':
bpo-32493: Fix uuid.uuid1() on FreeBSD. (GH-7099)
https://github.com/python/cpython/commit/17d8830312d82e7de42ab89739b0771f712645ff

New changeset 5734f41a9b46b4fd65b6ba90240b108f8a0b7c57 by Miss Islington (bot) in branch '3.7':
bpo-32493: Fix uuid.uuid1() on FreeBSD. (GH-7099)
https://github.com/python/cpython/commit/5734f41a9b46b4fd65b6ba90240b108f8a0b7c57

Thank you Serhiy Storchaka for the fix! If someone cares about AIX, please open a different issue. (AIX has other more important functions that you be fixed, IHMO.)

@vstinner - I know this is "closed", however, I submit a minor PR to fix an error in PR7104. The logic of configure.ac always defines HAVE_UUID_ENC_BE.

I cannot proceed with PR5183 (aka issue28009) until this is repaired.

Thx!

> @vstinner - I know this is "closed", however, I submit a minor PR to fix an error in PR7104. The logic of configure.ac always defines HAVE_UUID_ENC_BE. I cannot proceed with PR5183 (aka issue28009) until this is repaired.

The title of this issue makes it very specific to FreeBSD, so it would be better to create a new one for AIX. What your write for the commit description is not appropriate for a commit, but it's fine for the description of a bug.

Please open a new issue to track uuid changes for AIX. I know that you have a second one for Python <= 3.6.

OK. I know I do not understand this well - when it goes in A, or B.

So, if I understand correctly, like this issue was created after 
uuid_create() was added for AIX and created issues for FreeBSD )that I 
did not know had uuid_create(), I need to create a new issue where I say 
this issue created an issue for AIX?

Seems like a lot of work. I only put it here because it is a very simple 
change directly related to the last PR.

Just verify that is what you want and I'll get to it as I can. In any 
case, I'll pull the PR - as it is specific to AIX.

On 6/8/2018 12:48 AM, STINNER Victor wrote:
> STINNER Victor <vstinner@redhat.com> added the comment:
>
>> @vstinner - I know this is "closed", however, I submit a minor PR to fix an error in PR7104. The logic of configure.ac always defines HAVE_UUID_ENC_BE. I cannot proceed with PR5183 (aka issue28009) until this is repaired.
> The title of this issue makes it very specific to FreeBSD, so it would be better to create a new one for AIX. What your write for the commit description is not appropriate for a commit, but it's fine for the description of a bug.
>
> Please open a new issue to track uuid changes for AIX. I know that you have a second one for Python <= 3.6.
>
> ----------
> resolution: fixed ->
> status: closed -> open
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue32493>
> _______________________________________
>

New changeset 20cd5c6e21559f35feded5b3cfa9069a281d4325 by Ned Deily (Michael Felt) in branch '3.7':
bpo-32493: Correct test for uuid_enc_be availability in configure.ac. (GH-7511)
https://github.com/python/cpython/commit/20cd5c6e21559f35feded5b3cfa9069a281d4325

New changeset ced0adb2638e4c02945bfa82e15595918eef74f1 by Ned Deily in branch 'master':
bpo-32493: Correct test for uuid_enc_be availability in configure.ac. (GH-7511) (GH-7567)
https://github.com/python/cpython/commit/ced0adb2638e4c02945bfa82e15595918eef74f1

As noted on PR 7511 and PR 7567, I have merged Michael's configure.ac fixes for testing for uuid_enc_be availability (independent of platform) that were incorrect in earlier commits for this issue.  Merged for 3.7.0rc1 and master (3.8). Thanks, everyone!

> As noted on PR 7511 and PR 7567, I have merged Michael's configure.ac fixes for testing for uuid_enc_be availability (independent of platform) that were incorrect in earlier commits for this issue.  Merged for 3.7.0rc1 and master (3.8). Thanks, everyone!

Thanks you Ned and Michael. Sorry for the confusion. I was first confused that the fix for master added a lot of code using ctypes. The final fix is the right fix for 3.7 and master ;-)

Michael: if you want to fix uuid on 3.6, I would suggest to open a new issue, because it seems that the fix is much more complex and unrelated to _uuid (module added to Python 3.7).

I was not aware that _uuid was new to python3-3.7. I thought it had been around for a long time. 

Bpo-28009 goes back two years and i was unaware of uuid_create(). 

Would it be easier to split it into 3 issues? One for unixdll, one for netstat, and one for test_uuid?

Michael

Sent from my iPhone

> On 10 Jun 2018, at 13:52, STINNER Victor <report@bugs.python.org> wrote:
> 
> 
> STINNER Victor <vstinner@redhat.com> added the comment:
> 
>> As noted on PR 7511 and PR 7567, I have merged Michael's configure.ac fixes for testing for uuid_enc_be availability (independent of platform) that were incorrect in earlier commits for this issue.  Merged for 3.7.0rc1 and master (3.8). Thanks, everyone!
> 
> Thanks you Ned and Michael. Sorry for the confusion. I was first confused that the fix for master added a lot of code using ctypes. The final fix is the right fix for 3.7 and master ;-)
> 
> Michael: if you want to fix uuid on 3.6, I would suggest to open a new issue, because it seems that the fix is much more complex and unrelated to _uuid (module added to Python 3.7).
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue32493>
> _______________________________________
>

> Would it be easier to split it into 3 issues? One for unixdll, one for netstat, and one for test_uuid?

One issue for AIX should be enough.