The attached example works fine with Python 3.4.2, but fails with Python 3.5.0a1 and 3.5.0a2

I am using:
$ g++ --version
g++ (GCC) 4.9.2 20150212 (Red Hat 4.9.2-6)
$ swig -version

SWIG Version 3.0.5

Compiled with g++ [x86_64-redhat-linux-gnu]

Configured options: +pcre
$ python3 --version
Python 3.5.0a2



The output of trying the makefile with Python 3.5.0a2 is:

$ make
swig -c++ -python -py3 example.i
g++ -g -fPIC -c example.cxx example_wrap.cxx -I/local/python350a2/include/python3.5m -I/local/python350a2/include/python3.5m
In file included from /local/python350a2/include/python3.5m/pyatomic.h:10:0,
                 from /local/python350a2/include/python3.5m/Python.h:53,
                 from example_wrap.cxx:154:
/usr/include/c++/4.9.2/exception:161:34: error: missing binary operator before token "("
 #if (__cplusplus >= 201103L) && (ATOMIC_INT_LOCK_FREE > 1)
                                  ^
In file included from /local/python350a2/include/python3.5m/pyatomic.h:10:0,
                 from /local/python350a2/include/python3.5m/Python.h:53,
                 from example_wrap.cxx:154:
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:40:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic _Bool atomic_bool;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:41:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic char atomic_char;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:42:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic signed char atomic_schar;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:43:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic unsigned char atomic_uchar;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:44:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic short atomic_short;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:45:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic unsigned short atomic_ushort;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:46:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic int atomic_int;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:47:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic unsigned int atomic_uint;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:48:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic long atomic_long;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:49:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic unsigned long atomic_ulong;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:50:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic long long atomic_llong;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:51:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic unsigned long long atomic_ullong;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:52:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __CHAR16_TYPE__ atomic_char16_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:53:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __CHAR32_TYPE__ atomic_char32_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:54:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __WCHAR_TYPE__ atomic_wchar_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:55:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __INT_LEAST8_TYPE__ atomic_int_least8_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:56:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __UINT_LEAST8_TYPE__ atomic_uint_least8_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:57:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __INT_LEAST16_TYPE__ atomic_int_least16_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:58:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __UINT_LEAST16_TYPE__ atomic_uint_least16_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:59:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __INT_LEAST32_TYPE__ atomic_int_least32_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:60:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __UINT_LEAST32_TYPE__ atomic_uint_least32_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:61:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __INT_LEAST64_TYPE__ atomic_int_least64_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:62:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __UINT_LEAST64_TYPE__ atomic_uint_least64_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:63:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __INT_FAST8_TYPE__ atomic_int_fast8_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:64:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __UINT_FAST8_TYPE__ atomic_uint_fast8_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:65:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __INT_FAST16_TYPE__ atomic_int_fast16_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:66:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __UINT_FAST16_TYPE__ atomic_uint_fast16_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:67:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __INT_FAST32_TYPE__ atomic_int_fast32_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:68:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __UINT_FAST32_TYPE__ atomic_uint_fast32_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:69:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __INT_FAST64_TYPE__ atomic_int_fast64_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:70:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __UINT_FAST64_TYPE__ atomic_uint_fast64_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:71:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __INTPTR_TYPE__ atomic_intptr_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:72:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __UINTPTR_TYPE__ atomic_uintptr_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:73:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __SIZE_TYPE__ atomic_size_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:74:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __PTRDIFF_TYPE__ atomic_ptrdiff_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:75:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __INTMAX_TYPE__ atomic_intmax_t;
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:76:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic __UINTMAX_TYPE__ atomic_uintmax_t;        
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:232:9: error: ‘_Atomic’ does not name a type
 typedef _Atomic struct
         ^
/usr/lib/gcc/x86_64-redhat-linux/4.9.2/include/stdatomic.h:239:3: error: ‘atomic_flag’ does not name a type
 } atomic_flag;
   ^
In file included from /local/python350a2/include/python3.5m/Python.h:53:0,
                 from example_wrap.cxx:154:
/local/python350a2/include/python3.5m/pyatomic.h:37:5: error: ‘_Atomic’ does not name a type
     _Atomic void *_value;
     ^
/local/python350a2/include/python3.5m/pyatomic.h:41:5: error: ‘atomic_int’ does not name a type
     atomic_int _value;
     ^
Makefile:4: recipe for target 'example.so' failed
make: *** [example.so] Error 1


The output with Python 3.4.2:
$ make
swig -c++ -python -py3 example.i
g++ -g -fPIC -c example.cxx example_wrap.cxx -I/local/python342m/include/python3.4m -I/local/python342m/include/python3.4m
g++ -g -shared *.o -o _example.so
$ python3.4 a_test.py 
3

_Atomic was added by the issue #22038.

New changeset eb48295e1f8b by Victor Stinner in branch 'default':
Issue #23644, #22038: Move #include <stdatomic.c> inside the extern "C" { ... }
https://hg.python.org/cpython/rev/eb48295e1f8b

Can you please try with the latest development version?

I have:
/local/python_fix/include/python3.5m/pyatomic.h
#ifndef Py_LIMITED_API
#ifndef Py_ATOMIC_H
#define Py_ATOMIC_H

#include "dynamic_annotations.h"

#include "pyconfig.h"

#ifdef __cplusplus
extern "C" {
#endif

#if defined(HAVE_STD_ATOMIC)
#include <stdatomic.h>
#endif

...

and still get:
In file included from /local/python_fix/include/python3.5m/pyatomic.h:14:0,
                 from /local/python_fix/include/python3.5m/Python.h:53,
                 from example_wrap.cxx:154:
/usr/include/c++/4.9.2/exception:161:34: error: missing binary operator before token "("
 #if (__cplusplus >= 201103L) && (ATOMIC_INT_LOCK_FREE > 1)


So the bug still seems to be there.

The problem is more general than Python: #include <stdatomic.h> fails in C++. Try to compiled atomic.cpp with g++, you will get the same error.

The following bug report in the glibc has been closed as WONTFIX:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60932

It suggests to use the following code:

#ifdef __cplusplus
#include <atomic>
using namespace std;
#else
#include <stdatomic.h>
#endif

I don't know if it works.

@haypo, @lbianc will investigate this issue.

If I change the header to:
#ifndef Py_LIMITED_API
#ifndef Py_ATOMIC_H
#define Py_ATOMIC_H

#include "dynamic_annotations.h"

#include "pyconfig.h"

#ifdef __cplusplus
extern "C" {
#endif

#if defined(HAVE_STD_ATOMIC)
#ifdef __cplusplus
#include <atomic>
using namespace std;
#else
#include <stdatomic.h>
#endif
#endif

... (rest same)

AND use -std=c++11
Then the error changes to:
In file included from /local/python_fix/include/python3.5m/Python.h:53:0,
                 from example_wrap.cxx:154:
/local/python_fix/include/python3.5m/pyatomic.h:42:5: error: ‘_Atomic’ does not name a type
     _Atomic void *_value;
     ^


Basically, <atomic> does define std::atomic_int, but it does not define _Atomic, so the line:
_Atomic void *_value;
will also need to be changed.

@haypo

For adding compatibility for atomics based on @Joshua.J.Cogliati change, I propose:

#ifndef Py_LIMITED_API
#ifndef Py_ATOMIC_H
#define Py_ATOMIC_H

#include "dynamic_annotations.h"

#include "pyconfig.h"

#if defined(HAVE_STD_ATOMIC)
#ifdef __cplusplus
#include <atomic>
#define _Atomic(T) atomic<T>
using namespace std;
#else
#include <stdatomic.h>
#endif
#endif

#ifdef __cplusplus
extern "C" {
#endif


/* This is modeled after the atomics interface from C1x, according to
 * the draft at
 * http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1425.pdf.
 * Operations and types are named the same except with a _Py_ prefix
 * and have the same semantics.
 *
 * Beware, the implementations here are deep magic.
 */

#if defined(HAVE_STD_ATOMIC)

typedef enum _Py_memory_order {
    _Py_memory_order_relaxed = memory_order_relaxed,
    _Py_memory_order_acquire = memory_order_acquire,
    _Py_memory_order_release = memory_order_release,
    _Py_memory_order_acq_rel = memory_order_acq_rel,
    _Py_memory_order_seq_cst = memory_order_seq_cst
} _Py_memory_order;

typedef struct _Py_atomic_address {
    _Atomic (void) *_value;
} _Py_atomic_address;

typedef struct _Py_atomic_int {
    atomic_int _value;
} _Py_atomic_int;


... (rest same)

> using namespace std;

Doing this in a header is ugly.

>> using namespace std;

>Doing this in a header is ugly.

If you put it in the:
#ifdef __cplusplus
extern "C" {
#endif

then it would only apply till the end of the pyatomic header, and not to anything that includes it.  Something like:

#ifdef __cplusplus
extern "C" {
#if defined(HAVE_STD_ATOMIC)
using namespace std;
#endif
#endif

But if you are already defining:
#define _Atomic(T) atomic<T>
maybe it should be:
#define _Atomic(T) std::atomic<T>

(I haven't checked any of this code, but they would get around the namespace std being changed in code that includes pyatomic.h problem. )

> #define _Atomic(T) std::atomic<T>

This looks better to me :-) Can you please write a patch?

> #define _Atomic(T) std::atomic<T>
Does not work, since there are definitions like "memory_order_relaxed" that are in std.

I tested the other one, and it works fine:
>#ifdef __cplusplus
>extern "C" {
>#if defined(HAVE_STD_ATOMIC)
>using namespace std;
>#endif
>#endif

@haypo, what do you think?

So use std::memory_order_relaxed, std::memory_order_acquire etc.

Hum, I'm lost with the problem with C++ :-( What is your use case? Do you want to compile CPython with C++? Or compile a third-party extension with C++ and this extension includes "Python.h" which includes "pyatomic.h".

For third-party code, pyatomic.h is only needed by PyThreadState_GET() in pystate.h. Maybe we should hide completly pyatomic.h. Currently, pyatomic.h is not really used if Py_LIMITED_API is defined.

By the way, can you try to compile the extension with Py_LIMITED_API? It should just work.


C++ 11 atomic:
---
#include <atomic>

int main()
{
    std::atomic_uintptr_t current;
    current.store(0, std::memory_order_relaxed);
    return 0;
}
---


C++ 11 atomic used with C functions:
---
#include <atomic>

int main()
{
    std::atomic<uintptr_t> current(0);
    std::atomic_store_explicit(&current, (uintptr_t)0, std::memory_order_relaxed);
    return 0;
}
---

I didn't find how to use std::atomic_store_explicit with std::atomic_uintptr_t, so I used std::atomic<uintptr_t> instead.

"std::atomic_store_explicit(&current, 0, std::memory_order_relaxed);" doesn't work: 0 must be explicitly cast to uintptr_t :-(

I tried to hack pyatomic.h to use std::atomic template: see attached pyatomic_cpp.patch.

But I don't understand: when PyThreadState_GET() is compiled in C++, should it use exactly the same functions of CPython (compiled with C)? It should be strange to to use a std::atomic C++ template to access an atomic C variable.

> For third-party code, pyatomic.h is only needed by PyThreadState_GET() in pystate.h. Maybe we should hide completly pyatomic.h. Currently, pyatomic.h is not really used if Py_LIMITED_API is defined.

pystate_cplusplus.patch: disable completly pyatomic.h on C++, and modify pystate.h: PyThreadState_GET() is just an alias to PyThreadState_Get() with compiled with C++.

IMO pystate_cplusplus.patch makes more sense.

I don't understand the purpose of atomicfix.patch or pyatomic_cpp.patch, PyThreadState_GET() will not work with these patches.

>Hum, I'm lost with the problem with C++ :-( What is your use case? Do >you want to compile CPython with C++? Or compile a third-party >extension with C++ and this extension includes "Python.h" which >includes "pyatomic.h".

My use case is I have C++ code that I want to use from a extension.  The extension includes Python.h, which includes pyatomic.h.  Thank you for looking into this.

@Joshua: Can you please try to compile your extension with Py_LIMITED_API defined? Ex: "#define Py_LIMITED_API 0x03030000" at the top of your C file, or g++ -DPy_LIMITED_API=0x03030000.

And can you also please try to patch Python with pystate_cplusplus.patch?

> @Joshua: Can you please try to compile your extension with  Py_LIMITED_API defined? Ex: "#define Py_LIMITED_API 0x03030000" at the top of your C file, or g++ -DPy_LIMITED_API=0x03030000.

It fails in that case, because SWIG is using functions that are not part of the Py_LIMITED_API.  

> And can you also please try to patch Python with pystate_cplusplus.patch?

With the pystate_cplusplus.patch I was able to compile both min_example.tar.gz and my actual extension.  So I with your patch, it does work.  Thank you.

Once this is fixed, maybe issue 8027 can be fixed as well in 3.5.0.

New changeset cb05b6d7aacd by Victor Stinner in branch 'default':
Issue #23644: Fix issues with C++ when compiling Python extensions
https://hg.python.org/cpython/rev/cb05b6d7aacd

"With the pystate_cplusplus.patch I was able to compile both min_example.tar.gz and my actual extension.  So I with your patch, it does work.  Thank you."

Cool! I applied this simple patch instead of trying to write an ugly glue in pyatomic.h between C and C++.

C and C++ simply look to be incompatible for handling atomic types :-/ The good news is that we don't them atomic types for Python extensions, only to compile Python core.

Thanks for the report. I prefer to detect such annoying issue before a ".0" release (3.5.0). FYI it's a regression introduced by the issue #22038 which started to use <stdatomic.h> (but it's now fixed).

For me compiling 3.5.2 on GCC 5.2.0 fails when I supply --with-cxx-main=g++ with the same "error: ‘_Atomic’ does not name a type" messages.

I'm guessing that --with-cxx-main is probably outdated(?) but at least it worked with 3.5.0

I also got this problem with gcc/g++ version 4.9.3. 

./Include/pyatomic.h:37:5: error: 'atomic_int' does not name a type

I'm using Python 3.6.1. It's not fixed!

> I'm using Python 3.6.1. It's not fixed!

Thanks for the confirmation of the fix ;-)

Ow sorry, I though it was already fixed in the latest stable version of 3.x. Since is issue is from 2015.

Oh, wait, I read your message backward. You said that the fix doesn't work?

Hum, how can I reproduce the issue? What is your use case?

> ./Include/pyatomic.h:37:5: error: 'atomic_int' does not name a type

I don't understand how you got this error. The line 37 of pyatomic.h is only compiled when you build Python itself. At least, since Python 3.5.1 and Python 3.6.0.

Did you try to build Python? Or did you try to build a C extension?

See the following commit of bpo-25150:

commit 6df29ada02d22c43a8d439a70b820cb1ceacca42
Author: Victor Stinner <victor.stinner@gmail.com>
Date:   Fri Sep 18 15:06:34 2015 +0200

    Issue #25150: Hide the private _Py_atomic_xxx symbols from the public
    Python.h header to fix a compilation error with OpenMP. PyThreadState_GET()
    becomes an alias to PyThreadState_Get() to avoid ABI incompatibilies.
    
    It is important that the _PyThreadState_Current variable is always accessed
    with the same implementation of pyatomic.h. Use the PyThreadState_Get()
    function so extension modules will all reuse the same implementation.

Yea I'm building Python 3.6.1 from source, using gcc and --with-cxx-main=correct/location/to/g++ flag. So the warning message appears during ./configure about that it can't find g++.

I didn't try to patch it using the attachments of this issue. Since I was hoping it's fixed in the release itself. I was using gcc/g++ version 4.9.

I don't think that CPython can be built by g++. If you consider that it
should, please open a new issue since it's a different use case.

Interestingly, I get the error with Python 3.6.3 build from source, where I include the path to the g++ binary.

Python builds without the g++ option, but then _Tkinter fails to build - see issue31817

Atomic functions are no longer part of the public C API: code was moved to pycore_atomic.h internal C API. I close the issue.