I am using OSX 10.10.5 (14F27) and Python 3.5. Python 3.5 was installed via brew.
When greenlet is installed inside virtualenv using py3.5 header files are
missing inside /path/to/venv/include/python3.5m. As a result uwsgi is unable to build with asyncio support

~  brew info python3
python3: stable 3.5.0 (bottled), HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python3/3.4.3 (4721 files, 82M)
Poured from bottle
/usr/local/Cellar/python3/3.4.3_2 (5948 files, 98M)
Built from source
/usr/local/Cellar/python3/3.5.0 (3573 files, 61M) *
Poured from bottle
From: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/python3.rb
==> Dependencies
Build: xz , pkg-config 
Required: openssl 
Recommended: readline , sqlite , gdbm , xz 
Optional: homebrew/dupes/tcl-tk


Steps to reproduce
------------------
1. Create a venv

$/tmp  python3.5 -m virtualenv venv_bug

2. Install greenlet
$/tmp  . venv_bug/bin/activate
(venv_bug)$/tmp  pip install greenlet
Collecting greenlet
  Using cached greenlet-0.4.9.tar.gz
Building wheels for collected packages: greenlet
  Running setup.py bdist_wheel for greenlet
  Complete output from command /private/tmp/venv_bug/bin/python3.5 -c "import setuptools;__file__='/private/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-build-sxtdlufe/greenlet/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpf6g8rwt_pip-wheel-:
  running bdist_wheel
  running build
  running build_ext
  building 'greenlet' extension
  creating build
  creating build/temp.macosx-10.10-x86_64-3.5
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/include/python3.5m -c greenlet.c -o build/temp.macosx-10.10-x86_64-3.5/greenlet.o
  creating build/lib.macosx-10.10-x86_64-3.5
  clang -bundle -undefined dynamic_lookup build/temp.macosx-10.10-x86_64-3.5/greenlet.o -o build/lib.macosx-10.10-x86_64-3.5/greenlet.cpython-35m-darwin.so
  installing to build/bdist.macosx-10.10-x86_64/wheel
  running install
  running install_lib
  creating build/bdist.macosx-10.10-x86_64
  creating build/bdist.macosx-10.10-x86_64/wheel
  copying build/lib.macosx-10.10-x86_64-3.5/greenlet.cpython-35m-darwin.so -> build/bdist.macosx-10.10-x86_64/wheel
  running install_headers
  creating build/bdist.macosx-10.10-x86_64/wheel/greenlet-0.4.9.data
  creating build/bdist.macosx-10.10-x86_64/wheel/greenlet-0.4.9.data/headers
  copying greenlet.h -> build/bdist.macosx-10.10-x86_64/wheel/greenlet-0.4.9.data/headers
  running install_egg_info
  running egg_info
  creating greenlet.egg-info
  writing top-level names to greenlet.egg-info/top_level.txt
  writing greenlet.egg-info/PKG-INFO
  writing dependency_links to greenlet.egg-info/dependency_links.txt
  writing manifest file 'greenlet.egg-info/SOURCES.txt'
  warning: manifest_maker: standard file '-c' not found

  reading manifest file 'greenlet.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'greenlet.egg-info/SOURCES.txt'
  Copying greenlet.egg-info to build/bdist.macosx-10.10-x86_64/wheel/greenlet-0.4.9-py3.5.egg-info
  running install_scripts
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-build-sxtdlufe/greenlet/setup.py", line 101, in <module>
      **setuptools_args)
    File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/private/tmp/venv_bug/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 213, in run
      archive_basename = self.get_archive_basename()
    File "/private/tmp/venv_bug/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 161, in get_archive_basename
      impl_tag, abi_tag, plat_tag = self.get_tag()
    File "/private/tmp/venv_bug/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 155, in get_tag
      assert tag == supported_tags[0]
  AssertionError

  ----------------------------------------
  Failed building wheel for greenlet
Failed to build greenlet
Installing collected packages: greenlet
  Running setup.py install for greenlet
Successfully installed greenlet-0.4.9

3. Install uWSGI
(venv_bug)$  /tmp  CFLAGS="-I/tmp/venv_bug/include/python3.5m" UWSGI_PROFILE="asyncio" pip install uwsgi
Collecting uwsgi
  Using cached uwsgi-2.0.11.2.tar.gz
Building wheels for collected packages: uwsgi
  Running setup.py bdist_wheel for uwsgi
  Complete output from command /private/tmp/venv_bug/bin/python3.5 -c "import setuptools;__file__='/private/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-build-bru7wwac/uwsgi/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/tmpwoq08qywpip-wheel-:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  copying uwsgidecorators.py -> build/lib
  installing to build/bdist.macosx-10.10-x86_64/wheel
  running install
  plugins/greenlet/greenlet.c:2:10: fatal error: 'greenlet/greenlet.h' file not found
  #include <greenlet/greenlet.h>
           ^
  1 error generated.

  ----------------------------------------
  Failed building wheel for uwsgi
Failed to build uwsgi
Installing collected packages: uwsgi
  Running setup.py install for uwsgi
    Complete output from command /private/tmp/venv_bug/bin/python3.5 -c "import setuptools, tokenize;__file__='/private/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-build-bru7wwac/uwsgi/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-69gibzr6-record/install-record.txt --single-version-externally-managed --compile --install-headers /private/tmp/venv_bug/bin/../include/site/python3.5/uwsgi:
    running install
    plugins/greenlet/greenlet.c:2:10: fatal error: 'greenlet/greenlet.h' file not found
    #include <greenlet/greenlet.h>
             ^
    1 error generated.

    ----------------------------------------
Command "/private/tmp/venv_bug/bin/python3.5 -c "import setuptools, tokenize;__file__='/private/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-build-bru7wwac/uwsgi/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-69gibzr6-record/install-record.txt --single-version-externally-managed --compile --install-headers /private/tmp/venv_bug/bin/../include/site/python3.5/uwsgi" failed with error code 1 in /private/var/folders/xk/8lhz2_j91_3758cc65stj5z40000gn/T/pip-build-bru7wwac/uwsgi
Also setuptools is at latest version 18.2.

Installing uwsgi site wide succeeds. greenlet directory is present inside `python3.5m`.

(venv_bug)$  /tmp  ls -la /usr/local/include/python3.5m | grep greenlet
drwxr-xr-x   3 root   admin   102 Nov  1 20:40 greenlet

Links:
uWSGI asyncio: https://uwsgi-docs.readthedocs.org/en/latest/asyncio.html#building-uwsgi-with-asyncio-support
Greenlet issue thread: https://github.com/python-greenlet/greenlet/issues/96

Just wanted to clarify that greenlet.h is not missing in virtualenv, but headers seem to be installed in venv/include/site/python3.5, when only venv/include is considered for includes when building extensions. It is not specific to OSX and it is trivial to reproduce on Linux too.

Greenlet isn't part of python, so what bug in CPython are you reporting?  It sounds like you should be reporting this to the greenlet folks.

I have attached the link to greenlet issue as well. The issue seems to be with distutils/setuptools.

I don't see a link to a greenlet issue, and setuptools is not maintained as part of the stdlib.  It sounds like we should close this until and if the greenlet project can point us to a specific problem with the stdlib distutils.

Link to greenlet issue: https://github.com/python-greenlet/greenlet/issues/96

I have created an issue in setuptools repo: https://bitbucket.org/pypa/setuptools/issues/455/unable-to-build-extension-in-python-35.

Thanks for the links.  I'll leave this open for now since it isn't clear where the problem is.  Please update the issue with any additional information you get from setuptools if it looks like a distutils problem, or ping this issue if you get no response there for an extended period.

As David notes, this issue does not document any problems with Python 3.5 itself.  I see at least two issues here.  One, you are using virtualenv, a third-party package, rather than Python's built-in venv.  It appears that the most recent release of virtualenv on PyPI, virtualenv 13.1.2, includes outdated wheels of other third-party products, in particular, of wheel: 0.24, instead of the current 0.26 (https://pypi.python.org/pypi/wheel).  If you update wheel in your virtualenv that should fix the "assert tag == supported_tags[0]" issue.  Suggest you open an issue against virtualenv to ask that they update their vendored packages or perhaps not include them.  Another problem, when using the python.org python3.5, is with the install script of the uwsgi, uwsgiconfig.py.  The script does some machinations to figure out what compiler it is using.  Unfortunately, it seems to get confused by the fact that, with the last several major releases of Xcode, Apple no long ships gcc but does provide a gcc alias that actually invokes clang.  If you override the CC value and explictly set CC=clang, the uwsgi install seems to work correctly with the python.org OS X 3.5.0.  You should probably open an issue with the uwsgi project about that.  I do not have a homebrew installation handy to check, so if there are other problems after updating wheel in your virtualenv (or switching to use builtin venv) and using CC=clang, suggest you ask on a homebrew forum.

I have reproduced this problem on Linux with pyvenv, isn't that part of Python 3.5?

I created a simple gist that can show the problem: https://gist.github.com/snaury/ea167713b1804ffbaf5a (testme.sh builds Python 3.5, creates a venv, install greenlet and tries to compile an extension)

Alexey, thanks for the simplified test case.  I'll look at it further.

Updating wheel to latest version fixed assertion issue but uwsgi installation still fails.

Virtualenv has updated to latest whhel version https://github.com/pypa/virtualenv/blob/develop/virtualenv_support/wheel-0.26.0-py2.py3-none-any.whl. Also, my example used `python -m venv`, this is part of stdlib and using `pyvenv-3.5` produced same result.

Explicitly setting `CC=clang` in OSX didn't help https://gist.github.com/kracekumar/270533354d62be46f77d and also on Ubuntu 14.04 produces same error with gcc and clang.

I have opened an issue in uwsgi https://github.com/unbit/uwsgi/issues/1100.

I've looked at this some more using variations of Alexy's test case and I now think there are at least two problems here.  And both issues have to do with confusion about exactly where a distribution's header files should be installed in venvs (created with the standard library venv) or virtual environments (created with the third-party virtualenv).

One, when building an extension module from a venv, py3 Distutils tries to add the path of the venv include directory to the list of include directories supplied to the C compiler front-end.  However, build_ext assumes that the include files have been installed in venv/include (Lib/distutils/command/build_ext.py#l157) while install.py uses an "install scheme"-specific location to install headers (Lib/distutils/command/install.py#l22); for unix non-user installs, that may be something like venv/include/python3.5m.  This is particularly a problem for a situation like here where the build of an extension module from one distribution, uwsgi, is dependent on the header files for an extension module from a previously installed distribution, greenlet.  So, to resolve this, it seems like build_ext needs to be smarter about the include path for venvs.  There should also be test case(s) to ensure that install_headers and build_ext are using the same paths in all relevant environments, e.g. venv, non-venv, user install, Windows, etc.

But even if that is fixed, there is another issue.  It appears pip uses yet another location for installing distribution header files in virtualenvs and venvs: venv/include/site/python3.5. So, if pip is used to install the distribution supplying the include files, the dependent distribution will still fail to build.  From the deleted comments in a recent commit (https://github.com/pypa/pip/commit/882cd358d1abd5e42df6333dddc42f52ab7d6ff2), it appears the reason why pip does this is that virtualenv creates a symlink to the interpreter's global include directory which means that include files from distributions installing to the virtualenv cannot be written to the normal (non-virtualenv) path.  It also looks like venv does not create such a symlink but depends on Distutils build_ext to supply paths to both the venv include directory and the global include directory to compilers.  I'm not sure what the right solution is here and I certainly may be missing something.  In any case, it must be possible for distributions to find the header files from other distributions regardless how they were installed.

Since these are all packaging issues, I think the PyPA folks need to take this and decide what action(s) are needed where.

Distutils is now deprecated (see PEP 632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools