This is a (hopefully) complete list of patches and modifications required to natively build Python 3.4.2 on an Android device where
patches == listed issues in this bug tracker
modifications (mods) == unsubmitted patches and/or edits made to source

CAVEATS
Build was completed on a tablet running Android 4.4.2 and within the KBOX environment; an app that simulates a Linux filesystem by using a ported version of fakechroot.  Details of the environment can be found here: http://kevinboone.net/kbox2_how_it_works.html. NOTE: During the Python build it was necessary to patch this environment with a modified version of fakechroot; the KBOX version currently available as of 2.21.2015 may not contain the modified fakechroot.

CONFIGURE && POST-CONFIGURE && MAKE
The configure command was run with the following parameters
--enable-shared
--without-ensurepip
--prefix=/usr/python

After running configure. changes were made to pyconfig.h to account for various platform limitations and to Setup to ensure that a) all modules specified were built as shared and b) modules that used symbols from libpython were explicitly linked to said library. See attached pyconfig.h and Setup.

Make fails when the newly built python binary tries to execute and run setup.py. This is resolved by copying the shared library from the build directory to /lib in the KBOX environment.  Otherwise make && make install runs without errors provided patches/mods are applied. I did not run make test or the equivalent for this build.

PATCHES/MODS
The attached patches from the following issues must be applied:
http://bugs.python.org/issue20305
http://bugs.python.org/issue16353 (NOTE: use patch with 'defpath' not 'confstr', which is not supported in Android)
http://bugs.python.org/issue21668

The following modifications to source must be made:
The instructions under 'Code Changes' at this link: https://code.google.com/p/python-for-android/wiki/CrossCompilingPython
The attached patch, contributed by Ryan Gonzales (patch.diff)
The changes listed below by file (apologies for formatting)
        
/* patch for strdup segfault issue in readline.c */
 #define RESTORE_LOCALE(sl) { if (sl) { setlocale(LC_CTYPE, sl); free(sl); } }
 
 char *_locale = setlocale(LC_CTYPE, NULL); 
 char *saved_locale; 
 if(!_locale) 
 _locale = ""; 
 saved_locale = strdup(_locale);
 
 
 char *saved_locale = setlocale(LC_CTYPE, NULL);
    if (saved_locale != NULL) {
        saved_locale = strdup(saved_locale);
        if (!saved_locale)
            Py_FatalError("not enough memory to save locale");
    }
    

/* fix for Lib/ctypes/__init.py__ */
 30 DEFAULT_MODE ==  RTLD_GLOBAL
 
 /* patch for python.c */
/* here line for oldloc is commented out */
     24     int i, res;
     25     /* char *oldloc; */
     26 #ifdef __FreeBSD__

/* here oldloc lines commented out */
     46
     47   /*  oldloc = _PyMem_RawStrdup(setlocale(LC_ALL, NULL));
     48    * if (!oldloc) {
     49    *     fprintf(stderr, "out of memory\n");
     50    *     return 1;
     51    *
     52    * }
     53    */

 /* patch for pythonrun.c 307, --309-310, ++311-313 --1009-1012 ++1013-1014 */
 
    305     return get_codec_name(codeset);
    306 #else
    307    char* m;
    308    PyErr_SetNone(PyExc_NotImplementedError);
    309    /*  return NULL; */
    310    /* char* m; */
    311    m = malloc(6);
    312    strcpy(m, "ascii");
    313    return m;
    314 #endif
    315 }
    
   1002 static int
   1003 initfsencoding(PyInterpreterState *interp)
   1004 {
   1005     PyObject *codec;
   1006
   1007     if (Py_FileSystemDefaultEncoding == NULL)
   1008     {
   1009         /* Py_FileSystemDefaultEncoding = get_locale_encoding();
   1010          * if (Py_FileSystemDefaultEncoding == NULL)
   1011          *   Py_FatalError("Py_Initialize: Unable to get the locale encoding
   1011 ");
   1012          */
   1013            Py_FileSystemDefaultEncoding = malloc(6);
   1014            strcpy(Py_FileSystemDefaultEncoding, "ascii");
   1015         Py_HasFileSystemDefaultEncoding = 0;
   1016         interp->fscodec_initialized = 1;
   1017         return 0;
   1018     }
   1019
   
/* changes to Lib/plat-linux/DLFCN.py */
     76 # Included from bits/dlfcn.h
     77 # Changed for Android
     78 RTLD_LAZY = 1
     79 RTLD_NOW = 0
     80 #RTLD_BINDING_MASK = 0x3
     81 #RTLD_NOLOAD = 0x00004
     82 RTLD_GLOBAL = 2
     83 RTLD_LOCAL = 0
     84 #RTLD_NODELETE = 0x01000
     85
     86 # Also from Android's dlfcn.h
     87 RTLD_DEFAULT ((void*) 0xffffffff)
     88 RTLD_NEXT     ((void*) 0xfffffffe)
     89

The attached g-zipped file contains the first set of patches required to build Python 3.4.2 from source in the environment specified in the origonal post.

Will post the second/final set ASAP

It's not convinient to have patches in a tarball. Please at least open patches one by one, or better: open one issue per patch. You should group similar changes in a single patch: for example all changes related to wcstombs().

You should explain each change.


For example, the following change (os.patch) looks like a bug. You cannot simply rename a function. It completly breaks backward compatibility.

-.. function :: get_shell()
+.. function :: get_shell_executable()


selectmodule.patch:

-        timeout = (int)ceill(dtimeout * 1000.0);
+        timeout = (int)ceil(dtimeout * 1000.0);

I don't understand where ceill() does come from. I cannot find it in the current source code. Maybe the patch was produced in the wrong direction?


patch.diff: please generate unified patches. Your format is ugly, I cannot read it. And the bug tracker failed to create a "review" button.


It's hard to follow instructions to apply patches. Maybe you should use a Mercurial repository, which include all changes?

Apologies for the tarball, but all patches within are related to this "issue"

Removing tarball and will re-post individual, cleaned-up patches, grouped by issue.  

Ryan, can you re-do patch for pythonrun.c? If not. I'l. work it in.

Cyd,

What revision of the Python 3.4 source is this against? The tarball?

I'm not quite sure what to redo that patch against.

Also, shouldn't the code be selectively enabled with macros/configure.ac edits? Or did I miss where someone said this should be in its own branch (which, in my opinion, isn't very clean)?

Ryan,
I manually applied the patch to the Python 3.4.2 source tarball, dowmloaded from the main site to my Android tablet. I'm guessing (re)generating the patch from that source should work.

I've no knowledge of configure syntax, but most of the patches apply only if __ANDROID__ is defined.  This can be enabled with the -mandroid/-mbionic CFLAGS that work with GCC 4.8.x and later.

Additionally, for the Lib/plat-<os> specific changes, I can create Lib/plat-android and move the modified DLFCN.py and others there, provided someone could help with the appropriate configure.ac syntax.

I can/could do it myself, but it would take a *while* as I'd need to learn a fair amount of configure scripting first.

So...

I know a decent amount of configure scripts, although Python's is easily going to be the absolute longest I've ever touched.

Maybe these should be made against the 3.6 Mercurial source? I'm not sure if the Python devs would prefer that.

Also, should the configure script be based on the Android target triple?

Pulling from Mercurial will require taking a couple of huge steps back, as I've ported git to the environment I am using but not Mercurial.  It would also mean getting up to speed *fast* as I know nothing about VCS...including Mercurial.

I'm willing to do this (what's another port after all) but it may be a while before I can get everything set up correctly.

Take a look at https://docs.python.org/devguide/gitdevs.html
"Mercurial for git developers"

Thanks for the suggestion Victor, however I've only ported git to this environment; the most I;ve used it is to run 'git clone' to pull source from a repo. The recommendation from the mobile-sig mailing list is...given my lack of familiarity with VCS..is to attempt to port and work with Mercurial.  If doing so starts to take too much bandwidth I'll fall back to git and the article you linked.

Side Note: I cannot emphasize enough how little I know about how to use git and most of the other tools I've ported to this environment. My goal was to get as many of the 'standard' Linux utilities running in this environment, then start learning them.

There is a github mirror of the hg repository (https://github.com/python/cpython).  Patches produced from it don't automatically get review links in the tracker, but at least they would then be against the most current source (and someone else could regenerate them).  That would at least allow you to skip the "port mercurial" step.

On the other hand, it sounds like Ryan may be volunteering to update the patches to apply to the head of default for you.  (And yes, Ryan, the patches should be against the head of the default branch.)

Now, whether or not we will accept the patches is a different discussion, but I'd certainly like to see more Android support in our tree myself.

Thanks for that suggestion R. David...that would be ideal given the 2.x requirements of Mercurial would mean i'd need to re-port Python before tackling a port of Mercurial.

I'll wait until Ryan chimes in before I start serious work with git, just in case.

That's what I was saying; I'll update the patches myself. I highly doubt Cyd wants to re-port Python again. :0

I'll ask on python-dev on which would be preferred: configure.ac edits or macro tests.

Thanks Ryan.
Should I create a branch off the master at github and patch there or just clone the master and patch?

...or, should I fork the 3.4 branch?  If so, which one...3.4 or origin/3.4?

I don't think that Android support should be added to a stable branche
(2.7 or 3.4), only in the development branch.

You may maintain your own 2.7 or 3.4 fork if you want.

Once this android Python is built, can it:

  - be copied from/to other android devices?

  - run without KBOX?

Ethan,
The binary produced runs in the KBOX environment. It can be copied between devices provided the target device has the KBOX environment installed. 

Victor et al,
I read https://docs.python.org/devguide/devcycle.html#indevbranch a couple of times and, unless I;m missing something the dev branch only contains the 3.5 release.  This may be an obvious question but if I fork the 3.4 branch would Android-related patches be eligible for that branch?

I'm hesitant to clone the dev branch as I;ve worked hard to get the latest stable version (3.4.2?) working on my device; I'm more interested in a) making my efforts available to others and b) learning the Python language than c) constantly working at making the latest and greatest Android-friendly

> c) constantly working at making the latest and greatest Android-friendly

But that is precisely what "Android support", should it be added, means:
It does take constant work (and build slaves) to support a platform.


Unrelatedly, regarding the localeconv changes: 

I cannot find any bugreport for localeconv at

  https://code.google.com/p/android/issues/list .


Before we add workarounds to Python, it would be nice to have
at least a record that the Android developers actually *did*
refuse to add a simple struct lconv (trivial for en_US!).

How about opening an Android issue?

Given Stephan's comments regarding Android support I'm beginning to think that I may have bitten off more than I can chew. Is there an option between 'constant work' and 'zero contribution'?

I am not by any means an Android developer so the following is possibly riddled with miconceptions and errata, but, regarding locale support, I think there's a difference between 'regular' and 'native' development.  Native development involves using the NDK to port various bits of C-written utilities and libraries to Android...like python... and it is where the locale is broken, because of Android's limited libc.

Most of us fall toward the lower end of "constant work" and "zero contribution", honestly, since the majority of us are doing it in spare time and not getting paid for it.  What is needed is a long-term commitment to fix bugs if they show up when new changes are made to CPython.  Which is where a buildbot is fairly critical...and I have no idea how complicated it will be to set up a buildbot for your environment.  I think it should be possible, though, since buildbot and twisted are written in python.

But NDK bugs are reported to the same bug tracker, aren't they?

BTW, PEP 11 now demands a stable buildbot for official platform support
(IMO a very sane policy).

Cyd, if you want to be a CPython/Android resource that's great.

If you don't have time for it, I completely understand.  What I'm hoping for is to take your initial efforts and build from there, as there are others who can take what you've started and run with it.

Make no mistake, your taking the time to share what you've done is greatly appreciated.

I'll try to do as much as I can to get this through. Once I reinstall the NDK (I accidently wiped my hard drive with a bad dd command recently), I'll test this on my old Android phone that still runs 2.3 and would be very happy if someone else can test it something newer (my laptop doesn't seem to like the Android emulator). I'll also try to do this against the development branch or tip or head or whatever it's called.

I'd be very happy if someone could contribute a build slave, because I can't.

I will work on the build slave (note: it will definitely be /work/ so if anyone has the resource and know-how to just do it, I will not be offended ;) .

Stefan,
I wouldn't know if they're reported to the same bug tracker...it's possible they aren't.  Additionally it's possible that the lack of locale support in libc isn't considered a bug.

Ethan,
Given Victor's recommendation of using the dev branch and the fact that my mods were made to a stable download (3.4.2), what would be the recommended way to go about getting my efforts into github so that they can be built upon?  Should I start working with the fork I've already made of cpython (with 3.4.2 as the default branch) or is there a different method I should use?

I'm afraid I know next to nothing about git, so cannot help there.

I would think that it wouldn't be too hard for someone (such as Ryan or myself) to forward port a set of 3.4.2 patches to 3.5 -- so whatever is easiest for you.

How does this sound?
* I'll clone the fork of the 3.4 branch I made in github, build and patch.
* Ryan will (as best as he can) grab said patches, regenerate them for the bug tracker or forward port them to 3.5 (in github? to Mercurial)
* I'll figure out the upgrade/maintain/patch workflow as best I can so i can set it up on my build device and help out with porting later versions of Python.

Sounds great!

Ok, if I don't finish it today, I *will* finish it tomorrow. Or Sunday. Sunday is the latest, though I'm pretty sure I'll be done by Saturday.

Hardest part is likely going to be re-configuring the Android emulator to test it.

Cyd (and anyone else who can), do you think you could test the binaries, too? I'll get them to you somehow. You have a real device; I have an old real device and an emulator.

Side note for Cyd: I'd advise you just use the fork of CPython you have and just push them to GitHub:

$ git remote add origin <url of your GitHub repo with .git at the end>
$ git push --all origin -u

That will push your changes (and every branch of CPython, I have no clue what your branch name is, so this just does everything) to the GitHub mirror.

So...I'm understanding that the game plan will go like this:

- Cyd pushes the changes he made to a GitHub fork of Python (and hopefully posts the URL here ;)
- I clone the repo and try to get the changes working with the dev version
- Somewhere in between now and the end of the world, Ethan will set up a build slave

I'll also help all I can with keeping these changes working with future versions of Python.

Yup, that sounds right!  ;)

> I wouldn't know if they're reported to the same bug tracker...it's possible they aren't.

Well, why don't you try? :)


> Additionally it's possible that the lack of locale support in libc isn't considered a bug.

Their "struct lconv" violates both the C standard and POSIX. It is
100% a bug.  Lack of locale support would be defensible, shipping
a broken localeconv() isn't.

I'm just furious that 20+ OSS projects go out of their way to work
around this trivial bug.  It's a waste of resources.

>Well, why don't you try? :)

Resource optimization.

Fine, I'm also optimizing and the "fix" isn't going into libmpdec.
Android can use the Python version of decimal.

>Fine, I'm also optimizing and the "fix" isn't going into libmpdec.
>Android can use the Python version of decimal.

Er, ok. Not entirely sure what this means, mostly because the handful of spare cycles not allocated to a) getting a fork up, running and patched b) learning git on the fly for a) c) porting spidermonkey (why I ported Python) and d) learning Python, are already dedicated to other projects.  And work.

Ok...so the joys of autoconf configuring bite again. ALL Android devices have /dev/ptmx (adb even assumes it)...should the configure script be modified to skip that check if cross-compiling for Android?

*ALL Android devices have /dev/ptmx (adb even assumes it)...should the *configure script be modified to skip that check if cross-compiling for *Android?

Yes, definitely. See the mods to pyconfig.h here"
https://code.google.com/p/python-for-android/wiki/CrossCompilingPython

Ignore the #define ANDROID 1.  GCC 4.8.x and later add -mandroid and/or -mbionic which defines __ANDROID__; I've been putting android-related changes under that macro.

*Cyd (and anyone else who can), do you think you could test the binaries, *too? I'll get them to you somehow. You have a real device; I have an old *real device and an emulator.

Sure...especially since the KBOX env is needed.

*Side note for Cyd: I'd advise you just use the fork of CPython you have *and just push them to GitHub:

$ git remote add origin <url of your GitHub repo with .git at the end>
$ git push --all origin -u

*That will push your changes (and every branch of CPython, I have no clue *what your branch name is, so this just does everything) to the GitHub *mirror.

I forked the Cpython repo on Github, set 3.4 as the default branch and

$git clone https://github.com/cydhaselton/cpython.git

After modifying files and/or adding dirs, copying files

$git add <modified file>

Hopefully the above was ok.

Off to read up on commits

Cyd:

Just run:

$ git commit -m "Put some message here" # commit the changes
$ git push # push the changes

I'm working on it, but I haven't even gotten it to build (CURSE YOU, AUTOCONF!)

Ryan,
Thanks. First commit in; all fixes for Android's broken mbstowcs, new plat-android and plat-android/DLFCN.py.

What are you trying to build?

Cyd:

I was trying to see if I could build the latest Python for Android (not to get it to run; just to get it to compile) so that your changes would (theoretically) easier to update to the latest Python. That failed. Miserably. Joys of cross-compiling Python.

So...I'm just going to wait for you to finish uploaded the changes and work from there. :)

Ryan,
Sounds good. I think I've got all of the bug tracker patches committed/pushed...now I need to do all of the other edits.  Aiming to be finished by Friday/Saturday.

Ryan,
I believe I've gotten all non-configure dependent commits in. 
How do you want me to handle post-configure edits I;ve made to get Python working (i.e. changes to pyconfig.h, Modules/Setup...)

Cyd:

What exactly do you mean by post-configure edits?

Ryan,
Edits made to files generated or modified by running ./configure.  Example: pyconfig.h.

Cyd:

Don't upload them. Git won't even let you add pyconfig.h; running `git add PC/pyconfig.h` won't say anything, but running `git status` will show that it did not do anything.

I understand; I haven't made the edits to these files but the edits are needed in order for the Android build to work properly.

How do you want me to handle them? I can post the expected values here as a diff, or just as plain text...?

So...you didn't make the changes by hand? Could you just post whatever you passed to configure?

I can...

./configure --prefix=/usr/python --enable-shared --without-ensurepip

...but configure didn't set up pyconfig.h correctly with those options. There are some changes I had to make post-configure to pyconfig.h and others when I initially built python...see this link: https://code.google.com/p/python-for-android/wiki/CrossCompilingPython
specifically the part about manually editing pyconfig.h

Ahh...ok, then please post the changes. I want to burn autotools enough right now; I doubt that redoing those changes would ease my anger. :)

Ryan,
Here are the changes to pyconfig.h.  I left out this
#define ANDROID 1
because GCC 4.8.x and up have -mandroid -mbionic that define __ANDROID__ so I've been using that.

#define HAVE_BROKEN_MBSTOWCS 1
#undef HAVE_DEV_PTMX
#undef HAVE_GETHOSTBYNAME_R
#undef HAVE_MBRTOWC
#define HAVE_NCURSES_H 1  /* This only if you've cross compiled curses */
#undef HAVE_SETLOCALE
#undef HAVE_WCSCOLL
#undef HAVE_WCSFTIME
#undef HAVE_WCSXFRM

The following changes were made to Modules/Setup

*shared*

<snip>

array arraymodule.c     -L. -lpython3.4m # array objects
cmath cmathmodule.c _math.c -L. -lpython3.4m -lm # complex math library functions
math mathmodule.c _math.c -L. -lpython3.4m -lm # math library functions, e.g. sin()
_struct _struct.c       -L. -lpython3.4m # binary structure packing/unpacking
time timemodule.c  -L. -lpython3.4m -lm # time operations and variables
_testcapi _testcapimodule.c    -L. -lpython3.4m # Python C API test module
_random _randommodule.c -L. -lpython3.4m # Random number generator
_elementtree -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI _ele
menttree.c      -L -lpython3.4m # elementtree accelerator
_pickle _pickle.c       -L. -lpython3.4m # pickle accelerator
_datetime _datetimemodule.c     -L. -lpython3.4m # datetime accelerator
_bisect _bisectmodule.c -L. -lpython3.4m # Bisection algorithms


unicodedata unicodedata.c    -L. -lpython3.4m # static Unicode character database

<snip>

#grp grpmodule.c                # grp(3)                                             select selectmodule.c   -lm -L. -lpython3.4m # select(2); not on ancient System      V                                                                                    

# CSV file helper
_csv _csv.c -L. -lpython3.4m

# Socket module helper for socket(2)
_socket socketmodule.c -L. -lpython3.4m

# Socket module helper for SSL support; you must comment out the other               # socket line above, and possibly edit the SSL variable:                             #SSL=/usr/local/ssl                                                                  _ssl _ssl.c \                                                                                -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \                                -L. -lpython3.4m -L$(SSL)/lib -lssl -lcrypto

I'm getting a weird error message building python:

ryan@DevPC-LX:~/stuff/cpydroid$ make
_PYTHON_PROJECT_BASE=/home/ryan/stuff/cpython _PYTHON_HOST_PLATFORM=linux-arm PYTHONPATH=./Lib:./Lib/plat-linux python3 -S -m sysconfig --generate-posix-vars ;\
	if test $? -ne 0 ; then \
		echo "generate-posix-vars failed" ; \
		rm -f ./pybuilddir.txt ; \
		exit 1 ; \
	fi
Could not import runpy module
Traceback (most recent call last):
  File "./Lib/runpy.py", line 15, in <module>
    import importlib.util
  File "./Lib/importlib/util.py", line 6, in <module>
    from ._bootstrap import module_from_spec
ImportError: cannot import name 'module_from_spec'
generate-posix-vars failed
make: *** [pybuilddir.txt] Error 1
ryan@DevPC-LX:~/stuff/cpydroid$ 


I'm trying to build for generic ARM Linux before I attempt updating the changes. Makes sense theoretically.

Then again, when using autotools, NOTHING ever makes sense. I'm probably looking in the wrong direction...

Ryan,
What version of Android and KBOX are you using? Are you cross-compiling?
If not could you post the output of printenv from within KBOX here?

> ImportError: cannot import name 'module_from_spec'

The command line uses the system python3, which is "too old"
and does not have 'module_from_spec' yet.

Try running ...

   python3 -S -m sysconfig --generate-posix-vars

... and then continue with `make`.


We should open an issue for that, it's annoying.

Python really needs some way of separating the host C compiler and the target C compiler.

I'm giving up cross-compiling from my computer and am going to install KBOX on an Android virtual device. It worked for Cyd, it'll work for me. Beats compiling two separate versions of Python (one just to get host version of pgen and _freeze_importlib), passing 20+ arguments to configure segregated in flags and environment variables, etc.

Please bear with me.

Cross compiling worked for a while in 3.4. It broke again
because we don't have a buildbot for that.

You should not need 20 args for ./configure. At least on
Ubuntu the script from #5404 generally works, but indeed
pgen, importlib and the above issue are currently broken
(again).

This is what I'm using:

PYTHON_FOR_BUILD=`realpath ../cpython/python` CC=arm-linux-androideabi-clang CXX=arm-linux-androideabi-clang++ ./configure --host=arm-linux-androideabi --build=x86_64 --disable-ipv6 ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no ac_cv_have_long_long_format=yes

I know I could use config.site, but I just popped all this in a shell script.

On Ubuntu, when I uploaded the script to #5404, I only needed:

  ./configure --prefix=/tmp/arm-install --without-ensurepip --host=arm-linux-gnueabi --build=x86_64 --disable-ipv6


(And config.site of course.)


We just have to fix the pgen etc. issues and someone has to provide
a buildbot, otherwise it's guaranteed to break again.

This would be a build-bot for cross-compiling?  As opposed to an android build-bot (which we'll also need) ?

Technically we could use the Android buildbot to test both
(cross-compiling and Android breakage), but it would be nice
to have a separate buildbot that just tests cross-compiling
with an easier target (e.g. arm-linux-gnueabi).

Ryan,
There are utilities you'll need in the KBOX environment, some of which may not be available at the KBOX download site.  

You'll need to download the make, gcc 4.8.0 and gawk packages from the download site, and vim and/or nano depending on your preference. 

If you get an 'undefined reference to dlopen' error during the build, you'll need the updated libfakechroot; update this ticket and I'll figure out a way to get it to you (I'm not sure if the currently available KBOX includes it or not).

Ugh...

I'll just cross my fingers and hope that the libfakechroot KBOX2 comes with is new enough.

As for a text editor, I'll just probably use something external.

*crosses fingers*

Ok, please bear with me. I'm going to be REALLY busy for a bit; I have a BIG test/exam coming up next week, a music recital shortly afterwards, and lots of stuff/studying in between.

Sorry for any delays!

Dog just died unexpectedly, behind on photograph project, need to start on Python tutorial, have a chat in a different language on Sunday I need to prep for and its patch day tomorrow.

So yeah. Completely understand.

I am slowly attaching the patches I created.

Titles:

- issue_16353.patch - Issue 16353 for tip
- issue_20305_tweaked.patch - Issue 2305 for tip
- rjmatthews62_fixes_tweaked.patch - rjmatthews62@gmail.com's Android fixes (a bit more modularized)
- unused_var.patch - An extension of 2305. Can't figure out how to merge them.
- issue_21668.patch - Issue 21668 for tip
- issue_20306.patch - Guess...
- android_segfault_fix.patch - The actual segfault fix
- kbox_fix.patch - Cyd's KBox fixes

Someone, please test them!

...except for the pyconfig.h changes. Need to do something with autoconf for that.

Ryan,
Has anyone taken you up on the non-KBOX testing?
Do you need me to test the patches within the KBOX environment?

Cyd: if you could do that, it would be great!

Ryan,
From where do you want me to download Python for testing?
I'm assuming I need to test by a) downloading an unpatched 3.4.2, b) applying patches, c) running ./configure && make && make install.. If this is not correct let me know

No; you need the CPython tip. You can use https://github.com/python/cpython. Other than that, it's the configure+make thing.

As I said before, the pyconfig.h changes aren't there yet. Trying to figure out how I want to approach those.

Ryan,
For now I'll make the edits to pyconfig.h manually after downloading the zip from the tip and running configure with necessary params.

I'll do my best to get this done this weekend, but it may have to wait until next.

Hey Ryan,
Just now patching downloaded/unzipped tip and was wondering if there was an order in which patches should be applied.

I ask because i'm getting the following when applying the android_segfault_fix.patch

/bld/python/cpython-3.4/cpython-3.4/Python $ patch < android_segfault_fix.patch
patching file frozenmain.c
Hunk 3 FAILED 53/56.
     }

     setlocale(LC_ALL, "");
+#endif
     for (i = 0; i < argc; i++) {
         argv_copy[i] = Py_DecodeLocale(argv[i], NULL);
         argv_copy2[i] = argv_copy[i];
patching file pylifecycle.c
patch: can't open 'pylifecycle.c': No such file or directory

Thoughts?

UPDATE: I found the file in github, under master, in Python/. It's not in the 3.4 or origin/3.4 branches...aren't we working on those? Or does it not matter?

I thought this was for the tip, i.e. the 3.5 dev...?

But I created the patches in the order that I wrote the descriptions in the comment. So you might want to use that order.

If that fails, I can figure out the revision I was at when I created the patches.

From previous post:
**********************
How does this sound?
* I'll clone the fork of the 3.4 branch I made in github, build and patch.
* Ryan will (as best as he can) grab said patches, regenerate them for the bug tracker or forward port them to 3.5 (in github? to Mercurial)
**********************

For some reason I was thinking that the 3.4 branch I cloned was a dev branch.  I'll start again with a download of 3.5 dev.

issue_20306.patch won't install; attempting to do so yields the following:

patching file configure.ac                                                           Hunk 56 FAILED 4944/4944.
 AC_MSG_RESULT($ENSUREPIP)
 AC_SUBST(ENSUREPIP)                                                                 
+AC_CHECK_MEMBER([struct passwd.pw_gecos],                                           +          [AC_DEFINE(HAVE_PASSWD_GECOS_FIELD, 1, [Define if <pwd.h> defines field passwd.pw_gecos])],                                                                    +          [],                                                                       +          [[#include <pwd.h>]])
+                                                                                     # generate output files
 AC_CONFIG_FILES(Makefile.pre Modules/Setup.config Misc/python.pc Misc/python-config.sh)
 AC_CONFIG_FILES([Modules/ld_so_aix], [chmod +x Modules/ld_so_aix])

For the time being I'll just use --without-ensurepip when configuring.

Latest error:

gcc --sysroot=/usr/gcc-4.9.2/sysroot -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g  -O3 -Wall -Wstrict-prototypes    -Werror=declaration-after-statement   -I. -IInclude -I./Include    -DPy_BUILD_CORE -o Objects/unicodeobject.o Objects/unicodeobject.c
Objects/unicodeobject.c:45:23: fatal error: androidfn.h: No such file or directory
 #include "androidfn.h"
                       ^
compilation terminated.
make: *** [Objects/unicodeobject.o] Error 1

No idea what androidfn.h is, but will troubleshoot in the AM

Ah, crud. androidfn.h was a header I added but forgot to put the patch for. I just attached the patch for that now.

The commit I based the patches on was by Alexander Belopolsky and was named "merge". Committed on February 28.

Thanks Ryan, the patch worked.

Latest error (when building Modules/pwdmodule.c:)

^
./Modules/pwdmodule.c:86:2: error: stray '#' in program
 +#endif
  ^
./Modules/pwdmodule.c:86:3: error: 'endif' undeclared (first use in this function)
 +#endif
   ^
./Modules/pwdmodule.c:86:3: note: each undeclared identifier is reported only once for each function it appears in
./Modules/pwdmodule.c:75:21: error: expected ';' before 'sets'
 #define SETS(i,val) sets(v, i, val)
                     ^
./Modules/pwdmodule.c:87:5: note: in expansion of macro 'SETS'
     SETS(setIndex++, p->pw_dir);
     ^
./Modules/pwdmodule.c: At top level:
./Modules/pwdmodule.c:81:0: error: unterminated #else
 #ifdef HAVE_PASSWD_GECOS_FIELD
 ^
make: *** [Modules/pwdmodule.o] Error 1
/bld/python/cpython-master $

Remove the + at the beginning of line 87.

I re-uploaded issue_20306.patch to fix that issue.

Will do, re: removing the '+'.

Also, attempted to apply the updated patch and got the following:

/bld/python/cpython-master $ patch -p1 < issue_20306\ \(1\).patch
patching file Modules/pwdmodule.c
Possibly reversed hunk 1 at 244
Hunk 1 FAILED 79/79.
     SETS(setIndex++, p->pw_passwd);
     PyStructSequence_SET_ITEM(v, setIndex++, _PyLong_FromUid(p->pw_uid));
     PyStructSequence_SET_ITEM(v, setIndex++, _PyLong_FromGid(p->pw_gid));
+#ifdef HAVE_PASSWD_GECOS_FIELD
     SETS(setIndex++, p->pw_gecos);
+#else
+    SETS(setIndex++, Py_None);
+    Py_INCREF(Py_None);
+#endif
     SETS(setIndex++, p->pw_dir);
     SETS(setIndex++, p->pw_shell);

patching file configure.ac
Hunk 56 FAILED 4944/4944.
 AC_MSG_RESULT($ENSUREPIP)
 AC_SUBST(ENSUREPIP)

+AC_CHECK_MEMBER([struct passwd.pw_gecos],
+          [AC_DEFINE(HAVE_PASSWD_GECOS_FIELD, 1, [Define if <pwd.h> defines field passwd.pw_gecos])],
+          [],
+          [[#include <pwd.h>]])
+
 # generate output files
 AC_CONFIG_FILES(Makefile.pre Modules/Setup.config Misc/python.pc Misc/python-config.sh)
 AC_CONFIG_FILES([Modules/ld_so_aix], [chmod +x Modules/ld_so_aix])
/bld/python/cpython-master $

Update: Removing the random '+' on line 87 of pwdmodule.c allowed the build to continue but it failed again with the following during or soon after linking:

Objects/unicodeobject.o: In function `PyUnicode_EncodeLocale':
/bld/python/cpython-master/Objects/unicodeobject.c:3236: undefined reference to `android_wcstombs'
/bld/python/cpython-master/Objects/unicodeobject.c:3248: undefined reference to `android_wcstombs'
Objects/unicodeobject.o: In function `wcstombs_errorpos':
/bld/python/cpython-master/Objects/unicodeobject.c:3154: undefined reference to `android_wcstombs'
Objects/unicodeobject.o: In function `PyUnicode_DecodeLocaleAndSize':
/bld/python/cpython-master/Objects/unicodeobject.c:3518: undefined reference to `android_mbstowcs'
/bld/python/cpython-master/Objects/unicodeobject.c:3518: undefined reference to `android_mbstowcs'
Objects/complexobject.o: In function `_Py_c_pow':
/bld/python/cpython-master/Objects/complexobject.c:129: undefined reference to `sincos'
Python/fileutils.o: In function `Py_DecodeLocale':
/bld/python/cpython-master/Python/fileutils.c:322: undefined reference to `android_mbstowcs'
Python/fileutils.o: In function `Py_EncodeLocale':
/bld/python/cpython-master/Python/fileutils.c:487: undefined reference to `android_wcstombs'
/bld/python/cpython-master/Python/fileutils.c:489: undefined reference to `android_wcstombs'
Python/fileutils.o: In function `_Py_wfopen':
/bld/python/cpython-master/Python/fileutils.c:1008: undefined reference to `android_wcstombs'
collect2: error: ld returned 1 exit status
make: *** [Programs/_freeze_importlib] Error 1

Maybe the androidfn.h wasn't added to the files above?

FYI, running 'make clean' && make does not resolve the last reported issue.
Trying 'make distclean' && ./configure && make.

Ack...I feel smart. Attached is kind of a "part 2" to the rjmatthews patch. Apply and the errors shall be solved. :)

Thanks Ryan.
(Probably should remove original androidfn.h patch; patch complains with 'file already exists' if I don't delete Include/androidfn.h before applying the latest patch)

Whoops. Updated the patch.

Hello Ryan,
Error from latest patch:

Python/pythonrun.c:44:8: error: conflicting types for 'android_mbstowcs'
 size_t android_mbstowcs(wchar_t *dest, char * in, int maxlen) {
        ^
In file included from Python/pythonrun.c:18:0:
Include/androidfn.h:10:8: note: previous declaration of 'android_mbstowcs' was here
 size_t android_mbstowcs(wchar_t *dest, const char * source, int maxlen);
        ^
Python/pythonrun.c:61:8: error: conflicting types for 'android_wcstombs'
 size_t android_wcstombs(char * dest, wchar_t *source, int maxlen)
        ^
In file included from Python/pythonrun.c:18:0:
Include/androidfn.h:9:8: note: previous declaration of 'android_wcstombs' was here
 size_t android_wcstombs(char * dest, const wchar_t *source, int maxlen);
        ^
Python/pythonrun.c: In function 'android_wcstombs':
Python/pythonrun.c:68:5: error: expected ';' before 'if'
     if (c >= 0xdc800 && c <= 0xdcff)
     ^
Python/pythonrun.c:63:11: warning: variable 'c' set but not used [-Wunused-but-set-variable]
   wchar_t c;
           ^
make: *** [Python/pythonrun.o] Error 1

Perhaps the 'include androidfn.h' should be removed from pythonrun.c? Or the function definition added to the androidfn.h?

(IANACC)

ARGH!

Fixed. Re-apply rjmatthews64_fixes2.patch.

Hmmmm. That patch failed to apply:

Possibly reversed hunk 1 at 1582
Hunk 1 failed 35/35

Maybe it's conflicted with the last one...

Try:

git checkout Python/pythonrun.c
git apply rjmatthews...

On Thu, Apr 9, 2015 at 12:28 PM, Cyd Haselton <report@bugs.python.org>
wrote:

>
> Cyd Haselton added the comment:
>
> Hmmmm. That patch failed to apply:
>
> Possibly reversed hunk 1 at 1582
> Hunk 1 failed 35/35
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23496>
> _______________________________________
>

Done, but got this error:

Python/pythonrun.c: In function 'android_wcstombs':
Python/pythonrun.c:67:5: error: expected ';' before 'if'
     if (c >= 0xdc800 && c <= 0xdcff)
     ^
Python/pythonrun.c:62:11: warning: variable 'c' set but not used [-Wunused-but-set-variable]
   wchar_t c;
           ^
make: *** [Python/pythonrun.o] Error 1

I put a fix in the patch; I don't know why the heck it didn't apply. Just
go to Python/pythonrun.c, line 66 and put a semicolon (;) at the end of the
line.

On Thu, Apr 9, 2015 at 1:51 PM, Cyd Haselton <report@bugs.python.org> wrote:

>
> Cyd Haselton added the comment:
>
> Done, but got this error:
>
> Python/pythonrun.c: In function 'android_wcstombs':
> Python/pythonrun.c:67:5: error: expected ';' before 'if'
>      if (c >= 0xdc800 && c <= 0xdcff)
>      ^
> Python/pythonrun.c:62:11: warning: variable 'c' set but not used
> [-Wunused-but-set-variable]
>    wchar_t c;
>            ^
> make: *** [Python/pythonrun.o] Error 1
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23496>
> _______________________________________
>

Done.
Unfortunately:

./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
        echo "generate-posix-vars failed" ; \
        rm -f ./pybuilddir.txt ; \
        exit 1 ; \
fi
Segmentation fault
generate-posix-vars failed
make: *** [pybuilddir.txt] Error 1
/bld/python/cpython-master $

I'll break out logcat and addr2line...hopefully it's something that was just overlooked

FYI, Here's the addr2line output

LD_LIBRARY_PATH=/bld/python/cpython-master:/data/data/jackpal.androidterm/kbox2/lib ./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
        echo "generate-posix-vars failed" ; \
        rm -f ./pybuilddir.txt ; \
        exit 1 ; \
fi
Segmentation fault
generate-posix-vars failed
make: *** [pybuilddir.txt] Error 1
/bld/python/cpython-master $ addr2line -C -f -e /lib/libpython3.5m.so.1.0 0008f1a4   _PyMem_RawStrdup
/bld/python/cpython-master/Objects/obmalloc.c:358

FYI, I think this is the culprit.
In Python/pylifecycle.c:

static char*
get_codec_name(const char *encoding)
{
    char *name_utf8, *name_str;
    PyObject *codec, *name = NULL;

    codec = _PyCodec_Lookup(encoding);
    if (!codec)
        goto error;

    name = _PyObject_GetAttrId(codec, &PyId_name);
    Py_CLEAR(codec);
    if (!name)
        goto error;

    name_utf8 = _PyUnicode_AsString(name);
    if (name_utf8 == NULL)
        goto error;
    name_str = _PyMem_RawStrdup(name_utf8);
    Py_DECREF(name);
    if (name_str == NULL) {
        PyErr_NoMemory();
        return NULL;
    }
    return name_str;

error:
    Py_XDECREF(codec);
    Py_XDECREF(name);
    return NULL;
}

If I figure out a working patch I'll post it here

I think it was the same issue as before, but, for some reason, it didn't applu or something...? Will look into it tomorrow.

Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>FYI, I think this is the culprit.
>In Python/pylifecycle.c:
>
>static char*
>get_codec_name(const char *encoding)
>{
>    char *name_utf8, *name_str;
>    PyObject *codec, *name = NULL;
>
>    codec = _PyCodec_Lookup(encoding);
>    if (!codec)
>        goto error;
>
>    name = _PyObject_GetAttrId(codec, &PyId_name);
>    Py_CLEAR(codec);
>    if (!name)
>        goto error;
>
>    name_utf8 = _PyUnicode_AsString(name);
>    if (name_utf8 == NULL)
>        goto error;
>    name_str = _PyMem_RawStrdup(name_utf8);
>    Py_DECREF(name);
>    if (name_str == NULL) {
>        PyErr_NoMemory();
>        return NULL;
>    }
>    return name_str;
>
>error:
>    Py_XDECREF(codec);
>    Py_XDECREF(name);
>    return NULL;
>}
>
>If I figure out a working patch I'll post it here
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

I think the android_segfault_fix patch doesn't include main.c and python.c...which use _PyMem_RawStrdup and also need patching...

FYI, even with patches applied to main.c and python.c, the newly-built python binary segfaults in the same location.

I'll tear down and re-do everything (git clone master, patch, configure and make) but I may not get to it until next weekend.

Unfortunately starting from scratch did not solve the issue.  Still getting a segfault when the newly built binary tries to run sysconfig

Can you try to see exactly *which* line of the function is segfaulting?

Ryan,
Here's the output from addr2line

bld/python/cpython-master/cpython $ addr2line -C -f -e /lib/libpython3.5m.so.1.0 0008f42c
_PyMem_RawStrdup
/bld/python/cpython-master/cpython/Objects/obmalloc.c:358

Ok...try going to Python/pylifecycle.c and changing lines 220-230 from:

#elif defined(HAVE_LANGINFO_H) && defined(CODESET)
    char* codeset = nl_langinfo(CODESET);
    if (!codeset || codeset[0] == '\0') {
        PyErr_SetString(PyExc_ValueError, "CODESET is not set or empty");
        return NULL;
    }
    return get_codec_name(codeset);
#elif defined(__ANDROID__)
    char* m = malloc(6);
    strcpy(m, "ascii");
    return m;

to:

#elif defined(__ANDROID__)
    char* m = malloc(6);
    strcpy(m, "ascii");
    return m;
#elif defined(HAVE_LANGINFO_H) && defined(CODESET)
    char* codeset = nl_langinfo(CODESET);
    if (!codeset || codeset[0] == '\0') {
        PyErr_SetString(PyExc_ValueError, "CODESET is not set or empty");
        return NULL;
    }
    return get_codec_name(codeset);

I just swapped the `elif`'s around.

Do you have the time/means to create a quick patch for that?

I ask because even a simple flip like that becomes a major pain when working with nano on a tablet.

If not, I'll start on it. Just thought I"d ask

Here:

diff -r 38f5b3beeb2a Python/pylifecycle.c
--- a/Python/pylifecycle.c Thu Mar 19 15:16:03 2015 -0500
+++ b/Python/pylifecycle.c Sat Apr 18 13:07:36 2015 -0500
@@ -217,6 +217,10 @@
     char codepage[100];
     PyOS_snprintf(codepage, sizeof(codepage), "cp%d", GetACP());
     return get_codec_name(codepage);
+#elif defined(__ANDROID__)
+    char* m = malloc(6);
+    strcpy(m, "ascii");
+    return m;
 #elif defined(HAVE_LANGINFO_H) && defined(CODESET)
     char* codeset = nl_langinfo(CODESET);
     if (!codeset || codeset[0] == '\0') {
@@ -224,10 +228,6 @@
         return NULL;
     }
     return get_codec_name(codeset);
-#elif defined(__ANDROID__)
-    char* m = malloc(6);
-    strcpy(m, "ascii");
-    return m;
 #else
     PyErr_SetNone(PyExc_NotImplementedError);
     return NULL;

On Sat, Apr 18, 2015 at 12:39 PM, Cyd Haselton <report@bugs.python.org>
wrote:

>
> Cyd Haselton added the comment:
>
> Do you have the time/means to create a quick patch for that?
>
> I ask because even a simple flip like that becomes a major pain when
> working with nano on a tablet.
>
> If not, I'll start on it. Just thought I"d ask
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23496>
> _______________________________________
>

Ryan,
Sorry...same problem.

Segmentation fault
generate-posix-vars failed
make: *** [pybuilddir.txt] Error 1
/bld/python/cpython-master/cpython $ addr2line -C -f -e /lib/libpython3.5m.so.1.0 0008f42c
_PyMem_RawStrdup
/bld/python/cpython-master/cpython/Objects/obmalloc.c:358

Ryan, 
Found the missing fix.

In ./Programs/python.c

#ifndef __ANDROID__
    oldloc = _PyMem_RawStrdup(setlocale(LC_ALL, NULL));
    if (!oldloc) {
        fprintf(stderr, "out of memory\n");                                                  return 1;
    }                                                                                #endif

Added the #ifndef __ANDROID__ #endif around lines 46-51

I'm now looking at the following syntax error:

File "./setup.py", line 1950
    elif host_platform.startswith('arm-linux')
                                             ^
SyntaxError: invalid syntax
make: *** [sharedmods] Error 1

On Sun, Apr 19, 2015 at 7:16 PM, Cyd Haselton <report@bugs.python.org>
wrote:

>
> Cyd Haselton added the comment:
>
> Ryan,
> Found the missing fix.
>
> In ./Programs/python.c
>
> #ifndef __ANDROID__
>     oldloc = _PyMem_RawStrdup(setlocale(LC_ALL, NULL));
>     if (!oldloc) {
>         fprintf(stderr, "out of memory\n");
>                   return 1;
>     }
>           #endif
>
> Added the #ifndef __ANDROID__ #endif around lines 46-51
>
>
...that was in android_segfault_fix.patch...

are you sure you're on the same commit as I am?

> I'm now looking at the following syntax error:
>
> File "./setup.py", line 1950
>     elif host_platform.startswith('arm-linux')
>                                              ^
> SyntaxError: invalid syntax
> make: *** [sharedmods] Error 1
>
>
I forgot to put a colon and the end of the line. I'll upload a fixed
"kbox_fixes.patch".

> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23496>
> _______________________________________
>

Patch for python.c that prevents segfault on Android

Ryan,
That fix is in the android_segfault patch, but it's for frozenmain.c not python.c

I cloned from master on Fri/Sat. Will double-check commit tomorrow but I think the problem is with the unpatched python.c

That's the thing; my repo has no python.c!

On April 19, 2015 7:39:19 PM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>Ryan,
>That fix is in the android_segfault patch, but it's for frozenmain.c
>not python.c
>
>I cloned from master on Fri/Sat. Will double-check commit tomorrow but
>I think the problem is with the unpatched python.c
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

Ryan,
There's not a python.c in the ./Programs file?

FYI, I'm on commit c917493dc4ea2c32371da861aca2235f0a08e68e

Nevermind. I was just being stupid. I kept searching the "Python" directory
for references to get_codec_name and _PyMem_RawStrdup and completely missed
the Programs directory. Sorry.

Did the updated kbox_fix.patch work?

Ryan,
Completely forgot to download and apply it, but based on what I see there are a few things that need to be corrected.

1) -python3.4m needs to be changed to -python3.5m...or the appropriate versioning variable.
2) colons at the end of the elif lines in setup.py (1950- )
3) The _crypt and readline module builds in setup.py need additional libraries to work...python3.5m for both and possibly ncurses for readline, I'll test and get back to you.

With #1 and #2 added, the build and install completes successfully, although you obviously can't import readline and _crypt.

Still working at this; I can't get the readline module to work.  It is built successfully but importing it produces the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen failed: could not load library "libreadline.so.6" needed by "readline.cpython-35m-arm-linux-gnueabi.so"; caused by cannot locate symbol "tgetnum" referenced by "libreadline.so.6"...

tgetnum is undefined in libreadline but defined in -libncurses and libtinfo. I've built and re-built readline (--with-curses and -without that option...linking -lncurses, then -lncurses and -ltinfo) in the KBOX environment. I've edited Modules/Setup and setup.py so as to specify the readline library plus a combination of others (-ncurses, -tinfo) to no avail.

Finally got readline to work with this build, but since the fixes are to the readline source tree I'm not sure if they should go here.

With that plus whatever fixes to patches I suggested previously, I think this build works. What would be the next steps for this...running tests?

On a related note, I managed to get pip working with this build...minus some errors with verbose mode.  I first had to make some edits to setup.py and Modules/Setup so that the build would find and make the _ssl and lzip modules. After running make install, I ran 'python -m ensurepip'. I tested it by searching for and installing the BeautifulSoup module.

Ryan,
Let me know what are the next steps for this issue/build...and I'll do my best to get them done.

All,
Just checking in as it has been a while...is there anything I need to do on my end for this port?

I thought you were running the tests...

Or was there something I had to do that I missed?

On May 29, 2015 5:54:18 AM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>All,
>Just checking in as it has been a while...is there anything I need to
>do on my end for this port?
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

On May 29, 2015 10:52:23 AM CDT, Ryan Gonzalez <report@bugs.python.org> wrote:
>
>Ryan Gonzalez added the comment:
>
>I thought you were running the tests...
>

Which tests?

>Or was there something I had to do that I missed?

From one of the earlier messages I posted:

1) -python3.4m needs to be changed to -python3.5m...or the appropriate versioning variable. 
2) colons at the end of the elif lines in setup.py (1950- ) 
3) The _crypt and readline module builds in setup.py need additional libraries to work...python3.5m for both and possibly ncurses for readline, I'll test and get back to you. With #1 and #2 added, the build and install completes successfully, although you obviously can't import readline and _crypt.

*snip*

The messages after that one document how I got readline to work
>
>On May 29, 2015 5:54:18 AM CDT, Cyd Haselton <report@bugs.python.org>
>wrote:
>>
>>Cyd Haselton added the comment:
>>
>>All,
>>Just checking in as it has been a while...is there anything I need to
>>do on my end for this port?
>>
>>----------
>>
>>_______________________________________
>>Python tracker <report@bugs.python.org>
>><http://bugs.python.org/issue23496>
>>_______________________________________
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

#1 and #2 are fixed. I hand-edited the patch file (!!), so here's to hoping it'll work...

Working on #3. You probably should try the tests now (which is obviously the scary part ;).

Fixes for readline and _crypt done.

Will do...if by "try the tests now" you mean "run 'make tests'"...but it may be about a week or two as a) I'm currently in the last stages of a mono port and b) will be working on a PIE gcc for android 5.0 after, so that I have a KBOX environment to work with after verizon updates my tablet.

If I can keep putting off the verizon 5.0 update

On June 4, 2015 4:52:59 PM CDT, Ryan Gonzalez <report@bugs.python.org> wrote:
>
>Ryan Gonzalez added the comment:
>
>#1 and #2 are fixed. I hand-edited the patch file (!!), so here's to
>hoping it'll work...
>
>Working on #3. You probably should try the tests now (which is
>obviously the scary part ;).
>
>----------
>Added file: http://bugs.python.org/file39628/kbox_fix.patch
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

On June 5, 2015 6:22:07 AM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>Will do...if by "try the tests now" you mean "run 'make tests'"

Pretty much.

>...but it may be about a week or two as a) I'm currently in the last stages of
>a mono port and b) will be working on a PIE gcc for android 5.0 after,
>so that I have a KBOX environment to work with after verizon updates my
>tablet.

Try CCTools. Works on Android 5.0. Clang can be a little broken, but with a bit of flag twiddling it works.

>
>If I can keep putting off the verizon 5.0 update
>
>On June 4, 2015 4:52:59 PM CDT, Ryan Gonzalez <report@bugs.python.org>
>wrote:
>>
>>Ryan Gonzalez added the comment:
>>
>>#1 and #2 are fixed. I hand-edited the patch file (!!), so here's to
>>hoping it'll work...
>>
>>Working on #3. You probably should try the tests now (which is
>>obviously the scary part ;).
>>
>>----------
>>Added file: http://bugs.python.org/file39628/kbox_fix.patch
>>
>>_______________________________________
>>Python tracker <report@bugs.python.org>
>><http://bugs.python.org/issue23496>
>>_______________________________________
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

Update:
Now that I;ve finished porting a much-needed gdb to my device, I should have time to tackle patch and testing this weekend.  Will post results when I have them.

UPDATE:
Spent this past weekend fixing the broken on-device compiler.  Will get to tests this weekend

I've run into an error when building python in preparation for running tests (i. e. with --with-pydebug in the ./configure params).

./libpython3.4dm.so: undefined reference to `_PyUnicode_CheckConsistency'

It doesn't seem to be defined anywhere in source that I can see and i'm not sure why.

Google turns up a few hits from the bug-tracker but none have resolutions
http://bugs.python.org/issue13869
http://bugs.python.org/msg151986

Any thoughts?

FYI, Figured out that running ./configure with --with-pydebug does NOT define Py_DEBUG in pyconfig.h.

Defining it in pyconfig.h got me past the "undefined reference to `_PyUnicode_CheckConsistency'" error, but i'm now getting this:

if test $? -ne 0 ; then \
        echo "generate-posix-vars failed" ; \
        rm -f ./pybuilddir.txt ; \
        exit 1 ; \
fi
Debug memory block at address p=0x90b7b0: API ''
    2416312320 bytes originally requested
    The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
        at p-3: 0x00 *** OUCH
        at p-2: 0x00 *** OUCH
        at p-1: 0x00 *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 4 pad bytes at tail=0x9096b7b0 are Segmentation fault
generate-posix-vars failed
make: *** [pybuilddir.txt] Error 1

Will re-check patch application and try again

You compiled with -fPIE and GCC, right? I know the Android Clang seems broken.

On June 29, 2015 7:09:25 AM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>FYI, Figured out that running ./configure with --with-pydebug does NOT
>define Py_DEBUG in pyconfig.h.
>
>Defining it in pyconfig.h got me past the "undefined reference to
>`_PyUnicode_CheckConsistency'" error, but i'm now getting this:
>
>if test $? -ne 0 ; then \
>        echo "generate-posix-vars failed" ; \
>        rm -f ./pybuilddir.txt ; \
>        exit 1 ; \
>fi
>Debug memory block at address p=0x90b7b0: API ''
>    2416312320 bytes originally requested
>    The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
>        at p-3: 0x00 *** OUCH
>        at p-2: 0x00 *** OUCH
>        at p-1: 0x00 *** OUCH
> Because memory is corrupted at the start, the count of bytes requested
>       may be bogus, and checking the trailing pad bytes may segfault.
>    The 4 pad bytes at tail=0x9096b7b0 are Segmentation fault
>generate-posix-vars failed
>make: *** [pybuilddir.txt] Error 1
>
>Will re-check patch application and try again
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

No...haven't upgraded to Android 5.0 yet.  

On June 29, 2015 9:00:01 AM CDT, Ryan Gonzalez <report@bugs.python.org> wrote:
>
>Ryan Gonzalez added the comment:
>
>You compiled with -fPIE and GCC, right? I know the Android Clang seems
>broken.
>
>On June 29, 2015 7:09:25 AM CDT, Cyd Haselton <report@bugs.python.org>
>wrote:
>>
>>Cyd Haselton added the comment:
>>
>>FYI, Figured out that running ./configure with --with-pydebug does NOT
>>define Py_DEBUG in pyconfig.h.
>>
>>Defining it in pyconfig.h got me past the "undefined reference to
>>`_PyUnicode_CheckConsistency'" error, but i'm now getting this:
>>
>>if test $? -ne 0 ; then \
>>        echo "generate-posix-vars failed" ; \
>>        rm -f ./pybuilddir.txt ; \
>>        exit 1 ; \
>>fi
>>Debug memory block at address p=0x90b7b0: API ''
>>    2416312320 bytes originally requested
>>    The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
>>        at p-3: 0x00 *** OUCH
>>        at p-2: 0x00 *** OUCH
>>        at p-1: 0x00 *** OUCH
>> Because memory is corrupted at the start, the count of bytes
>requested
>>       may be bogus, and checking the trailing pad bytes may segfault.
>>    The 4 pad bytes at tail=0x9096b7b0 are Segmentation fault
>>generate-posix-vars failed
>>make: *** [pybuilddir.txt] Error 1
>>
>>Will re-check patch application and try again
>>
>>----------
>>
>>_______________________________________
>>Python tracker <report@bugs.python.org>
>><http://bugs.python.org/issue23496>
>>_______________________________________
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

Your question about -fPIE brings up a question: How should the differences between Android 5 and previous versions be handled in regards to this issue?

Other than mandatory -fPIE, there are changes to Android[s libc that may make  patches for python on Android 4.x incompatible with Android 5.

Thoughts?

On June 29, 2015 9:29:35 AM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>No...haven't upgraded to Android 5.0 yet.  
>
>On June 29, 2015 9:00:01 AM CDT, Ryan Gonzalez <report@bugs.python.org>
>wrote:
>>
>>Ryan Gonzalez added the comment:
>>
>>You compiled with -fPIE and GCC, right? I know the Android Clang seems
>>broken.
>>
>>On June 29, 2015 7:09:25 AM CDT, Cyd Haselton <report@bugs.python.org>
>>wrote:
>>>
>>>Cyd Haselton added the comment:
>>>
>>>FYI, Figured out that running ./configure with --with-pydebug does
>NOT
>>>define Py_DEBUG in pyconfig.h.
>>>
>>>Defining it in pyconfig.h got me past the "undefined reference to
>>>`_PyUnicode_CheckConsistency'" error, but i'm now getting this:
>>>
>>>if test $? -ne 0 ; then \
>>>        echo "generate-posix-vars failed" ; \
>>>        rm -f ./pybuilddir.txt ; \
>>>        exit 1 ; \
>>>fi
>>>Debug memory block at address p=0x90b7b0: API ''
>>>    2416312320 bytes originally requested
>>>    The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
>>>        at p-3: 0x00 *** OUCH
>>>        at p-2: 0x00 *** OUCH
>>>        at p-1: 0x00 *** OUCH
>>> Because memory is corrupted at the start, the count of bytes
>>requested
>>>       may be bogus, and checking the trailing pad bytes may
>segfault.
>>>    The 4 pad bytes at tail=0x9096b7b0 are Segmentation fault
>>>generate-posix-vars failed
>>>make: *** [pybuilddir.txt] Error 1
>>>
>>>Will re-check patch application and try again
>>>
>>>----------
>>>
>>>_______________________________________
>>>Python tracker <report@bugs.python.org>
>>><http://bugs.python.org/issue23496>
>>>_______________________________________
>>
>>----------
>>
>>_______________________________________
>>Python tracker <report@bugs.python.org>
>><http://bugs.python.org/issue23496>
>>_______________________________________
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

On June 30, 2015 8:14:34 AM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>Your question about -fPIE brings up a question: How should the
>differences between Android 5 and previous versions be handled in
>regards to this issue?
>
>Other than mandatory -fPIE, there are changes to Android[s libc that
>may make  patches for python on Android 4.x incompatible with Android
>5.
>

WHAT??? :( I can't find a list anywhere; how do you know?

>Thoughts?

Not really...

>
>On June 29, 2015 9:29:35 AM CDT, Cyd Haselton <report@bugs.python.org>
>wrote:
>>
>>Cyd Haselton added the comment:
>>
>>No...haven't upgraded to Android 5.0 yet.  
>>
>>On June 29, 2015 9:00:01 AM CDT, Ryan Gonzalez
><report@bugs.python.org>
>>wrote:
>>>
>>>Ryan Gonzalez added the comment:
>>>
>>>You compiled with -fPIE and GCC, right? I know the Android Clang
>seems
>>>broken.
>>>
>>>On June 29, 2015 7:09:25 AM CDT, Cyd Haselton
><report@bugs.python.org>
>>>wrote:
>>>>
>>>>Cyd Haselton added the comment:
>>>>
>>>>FYI, Figured out that running ./configure with --with-pydebug does
>>NOT
>>>>define Py_DEBUG in pyconfig.h.
>>>>
>>>>Defining it in pyconfig.h got me past the "undefined reference to
>>>>`_PyUnicode_CheckConsistency'" error, but i'm now getting this:
>>>>
>>>>if test $? -ne 0 ; then \
>>>>        echo "generate-posix-vars failed" ; \
>>>>        rm -f ./pybuilddir.txt ; \
>>>>        exit 1 ; \
>>>>fi
>>>>Debug memory block at address p=0x90b7b0: API ''
>>>>    2416312320 bytes originally requested
>>>>    The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
>>>>        at p-3: 0x00 *** OUCH
>>>>        at p-2: 0x00 *** OUCH
>>>>        at p-1: 0x00 *** OUCH
>>>> Because memory is corrupted at the start, the count of bytes
>>>requested
>>>>       may be bogus, and checking the trailing pad bytes may
>>segfault.
>>>>    The 4 pad bytes at tail=0x9096b7b0 are Segmentation fault
>>>>generate-posix-vars failed
>>>>make: *** [pybuilddir.txt] Error 1
>>>>
>>>>Will re-check patch application and try again
>>>>
>>>>----------
>>>>
>>>>_______________________________________
>>>>Python tracker <report@bugs.python.org>
>>>><http://bugs.python.org/issue23496>
>>>>_______________________________________
>>>
>>>----------
>>>
>>>_______________________________________
>>>Python tracker <report@bugs.python.org>
>>><http://bugs.python.org/issue23496>
>>>_______________________________________
>>
>>----------
>>
>>_______________________________________
>>Python tracker <report@bugs.python.org>
>><http://bugs.python.org/issue23496>
>>_______________________________________
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

On June 30, 2015 9:20:45 AM CDT, Ryan Gonzalez <report@bugs.python.org> wrote:
>
>Ryan Gonzalez added the comment:
>
>On June 30, 2015 8:14:34 AM CDT, Cyd Haselton <report@bugs.python.org>
>wrote:
>>
>>Cyd Haselton added the comment:
>>
>>Your question about -fPIE brings up a question: How should the
>>differences between Android 5 and previous versions be handled in
>>regards to this issue?
>>
>>Other than mandatory -fPIE, there are changes to Android[s libc that
>>may make  patches for python on Android 4.x incompatible with Android
>>5.
>>
>
>WHAT??? :( I can't find a list anywhere; how do you know?
>

I tried building GCC for Android 5 on device and ran into a bunch of errors like 'undefined symbol __swbuf.'  Googling the errors led to this page: https://developer.android.com/ndk/downloads/revision_history.html.

Relevant Part here:

" Removed the following symbols from all architectures: get_malloc_leak_info,free_malloc_leak_info, __srget, __swbuf, __srefill,__swsetup, __sdidinit, __sflags, __sfp, __sinit, __smakebuf,__sflush, __sread, __swrite, __sseek, __sclose, _fwalk,__sglue, __get_thread, __wait4, __futex_wake, __open,__get_tls, __getdents64, and dlmalloc."

>>Thoughts?
>
>Not really...

Would setting up two repos be advisable...one for 5 and one for 4.x and earlier?
>
>>
>>On June 29, 2015 9:29:35 AM CDT, Cyd Haselton <report@bugs.python.org>
>>wrote:
>>>
>>>Cyd Haselton added the comment:
>>>
>>>No...haven't upgraded to Android 5.0 yet.  
>>>
>>>On June 29, 2015 9:00:01 AM CDT, Ryan Gonzalez
>><report@bugs.python.org>
>>>wrote:
>>>>
>>>>Ryan Gonzalez added the comment:
>>>>
>>>>You compiled with -fPIE and GCC, right? I know the Android Clang
>>seems
>>>>broken.
>>>>
>>>>On June 29, 2015 7:09:25 AM CDT, Cyd Haselton
>><report@bugs.python.org>
>>>>wrote:
>>>>>
>>>>>Cyd Haselton added the comment:
>>>>>
>>>>>FYI, Figured out that running ./configure with --with-pydebug does
>>>NOT
>>>>>define Py_DEBUG in pyconfig.h.
>>>>>
>>>>>Defining it in pyconfig.h got me past the "undefined reference to
>>>>>`_PyUnicode_CheckConsistency'" error, but i'm now getting this:
>>>>>
>>>>>if test $? -ne 0 ; then \
>>>>>        echo "generate-posix-vars failed" ; \
>>>>>        rm -f ./pybuilddir.txt ; \
>>>>>        exit 1 ; \
>>>>>fi
>>>>>Debug memory block at address p=0x90b7b0: API ''
>>>>>    2416312320 bytes originally requested
>>>>>    The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
>>>>>        at p-3: 0x00 *** OUCH
>>>>>        at p-2: 0x00 *** OUCH
>>>>>        at p-1: 0x00 *** OUCH
>>>>> Because memory is corrupted at the start, the count of bytes
>>>>requested
>>>>>       may be bogus, and checking the trailing pad bytes may
>>>segfault.
>>>>>    The 4 pad bytes at tail=0x9096b7b0 are Segmentation fault
>>>>>generate-posix-vars failed
>>>>>make: *** [pybuilddir.txt] Error 1
>>>>>
>>>>>Will re-check patch application and try again
>>>>>
>>>>>----------
>>>>>
>>>>>_______________________________________
>>>>>Python tracker <report@bugs.python.org>
>>>>><http://bugs.python.org/issue23496>
>>>>>_______________________________________
>>>>
>>>>----------
>>>>
>>>>_______________________________________
>>>>Python tracker <report@bugs.python.org>
>>>><http://bugs.python.org/issue23496>
>>>>_______________________________________
>>>
>>>----------
>>>
>>>_______________________________________
>>>Python tracker <report@bugs.python.org>
>>><http://bugs.python.org/issue23496>
>>>_______________________________________
>>
>>----------
>>
>>_______________________________________
>>Python tracker <report@bugs.python.org>
>><http://bugs.python.org/issue23496>
>>_______________________________________
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

UPDATE:
Haven't forgotten about this; I'm currently (thanks to Android's new mandatory PIE binaries requirement) rebuilding all of the necessary utilities (openssl, curl, git, etc) so that I can clone and test.

Between the above and a sharp increase in workload at the day job, expect a few weeks delay between now and continued work on this issue.

UPDATE:
Build environment is up and running; cloning repo now.

Build complete. Unfortunately while some of the tests complete successfully, the run ends in a segfault (see attached log)

What hardware architecture are you compiling for? If it's ARM64, and you're not using a trunk version of libffi, that segfault in test_ctypes is to be expected.

I'm compiling for ARM, not ARM64, on an armv7 device.

On July 23, 2015 7:10:35 PM CDT, Russell Keith-Magee <report@bugs.python.org> wrote:
>
>Russell Keith-Magee added the comment:
>
>What hardware architecture are you compiling for? If it's ARM64, and
>you're not using a trunk version of libffi, that segfault in
>test_ctypes is to be expected.
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

Are you using the libffi sources vendored into the Python source tree, or a more recent version? I can verify that libffi v3.2 works on ARMv7 (on iOS, anyway), and there's been plenty of changes to the ARM source tree since the Python version was vendored in.

I assume so; I'm using whatever is pulled frommy fork of the git repo.

I've actually run into a different error that occured when i rebuilt to include readline support; when I run ./python -m test I;m getting "ImportError: no module named _struct found."

Trying to figure that one out first...

So...I have *no* clue why _struct can't be found.

Can you use gdb to get the segfault backtrace like you did before?

On July 25, 2015 6:56:10 AM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>I assume so; I'm using whatever is pulled frommy fork of the git repo.
>
>I've actually run into a different error that occured when i rebuilt to
>include readline support; when I run ./python -m test I;m getting
>"ImportError: no module named _struct found."
>
>Trying to figure that one out first...
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

As soon as I build a PIE gdb I can.

And right now, running ./python -m test throws the ImportError...so I'll
need to resolve that first
On Jul 25, 2015 12:38 PM, "Ryan Gonzalez" <report@bugs.python.org> wrote:

>
> Ryan Gonzalez added the comment:
>
> So...I have *no* clue why _struct can't be found.
>
> Can you use gdb to get the segfault backtrace like you did before?
>
> On July 25, 2015 6:56:10 AM CDT, Cyd Haselton <report@bugs.python.org>
> wrote:
> >
> >Cyd Haselton added the comment:
> >
> >I assume so; I'm using whatever is pulled frommy fork of the git repo.
> >
> >I've actually run into a different error that occured when i rebuilt to
> >include readline support; when I run ./python -m test I;m getting
> >"ImportError: no module named _struct found."
> >
> >Trying to figure that one out first...
> >
> >----------
> >
> >_______________________________________
> >Python tracker <report@bugs.python.org>
> ><http://bugs.python.org/issue23496>
> >_______________________________________
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23496>
> _______________________________________
>

Finally found a hacky fix for the "no module named _struct found"; I copied all Modules/*.cpython-34m.so objects to Lib.

Unfortunately I still get a segfault error when test_ctypes is run:

[ 83/390/11] test_ctypes
Fatal Python error: Segmentation fault

Current thread 0xb6f2bec8 (most recent call first):
  File "/bld/pyt/cpython-android/Lib/ctypes/test/test_as_parameter.py", line 85 in test_callbacks
  File "/bld/pyt/cpython-android/Lib/unittest/case.py", line 577 in run
  File "/bld/pyt/cpython-android/Lib/unittest/case.py", line 625 in __call__
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 122 in run
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 84 in __call__
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 122 in run
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 84 in __call__
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 122 in run
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 84 in __call__
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 122 in run
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 84 in __call__
  File "/bld/pyt/cpython-android/Lib/test/support/__init__.py", line 1668 in run
  File "/bld/pyt/cpython-android/Lib/test/support/__init__.py", line 1769 in _run_suite
  File "/bld/pyt/cpython-android/Lib/test/support/__init__.py", line 1803 in run_unittest
  File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 1279 in test_runner
  File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 1280 in runtest_inner
  File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 978 in runtest
  File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 763 in main
  File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 1564 in main_in_temp_cwd  File "/bld/pyt/cpython-android/Lib/test/__main__.py", line 3 in <module>
  File "/bld/pyt/cpython-android/Lib/runpy.py", line 85 in _run_code
  File "/bld/pyt/cpython-android/Lib/runpy.py", line 170 in _run_module_as_main
Segmentation fault

A nominal bump before starting to re-build a PIE gdb to debug the segfault above

After struggling to get helpful output from gdb it is looking like it will not be possible due to the lack of debugging symbols in the libs on the android device.

Still investigating.

Doesn't Python still have debug symbols? The system ones don't matter too much.

On August 12, 2015 6:21:23 AM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>After struggling to get helpful output from gdb it is looking like it
>will not be possible due to the lack of debugging symbols in the libs
>on the android device.
>
>Still investigating.
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

I thought porting gdb would be the difficult part of getting debug info. I was so wrong.

Here is what I have so far, after lengthy consultation of the gdb manual.  Bear with me...it's basically a copy & paste of gdb session output:

(gdb) run
Starting program: /bld/pyt/cpython-android/python
setpgrp failed in child: No such process
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Program received signal SIGILL, Illegal instruction.
0xb6a63cc8 in ?? ()
(gdb) f
#0  0xb6a63cc8 in ?? ()
(gdb) info f
Stack level 0, frame at 0xbefff618:
 pc = 0xb6a63cc8; saved pc = 0xb6a5feb0
 called by frame at 0xbefff618
 Arglist at 0xbefff618, args:
 Locals at 0xbefff618, Previous frame's sp is 0xbefff618
(gdb) info args
No symbol table info available.
(gdb) info local
No symbol table info available.
(gdb) info source
Current source file is ./Modules/python.c
Compilation directory is /bld/pyt/cpython-android
Located in /bld/pyt/cpython-android/Modules/python.c
Contains 80 lines.
Source language is c.
Compiled with DWARF 2 debugging format.
Does not include preprocessor macro info.

What if you run:

bt

?

On August 15, 2015 7:32:37 PM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>I thought porting gdb would be the difficult part of getting debug
>info. I was so wrong.
>
>Here is what I have so far, after lengthy consultation of the gdb
>manual.  Bear with me...it's basically a copy & paste of gdb session
>output:
>
>(gdb) run
>Starting program: /bld/pyt/cpython-android/python
>setpgrp failed in child: No such process
>warning: Unable to find dynamic linker breakpoint function.
>GDB will be unable to debug shared library initializers
>and track explicitly loaded dynamic code.
>
>Program received signal SIGILL, Illegal instruction.
>0xb6a63cc8 in ?? ()
>(gdb) f
>#0  0xb6a63cc8 in ?? ()
>(gdb) info f
>Stack level 0, frame at 0xbefff618:
> pc = 0xb6a63cc8; saved pc = 0xb6a5feb0
> called by frame at 0xbefff618
> Arglist at 0xbefff618, args:
> Locals at 0xbefff618, Previous frame's sp is 0xbefff618
>(gdb) info args
>No symbol table info available.
>(gdb) info local
>No symbol table info available.
>(gdb) info source
>Current source file is ./Modules/python.c
>Compilation directory is /bld/pyt/cpython-android
>Located in /bld/pyt/cpython-android/Modules/python.c
>Contains 80 lines.
>Source language is c.
>Compiled with DWARF 2 debugging format.
>Does not include preprocessor macro info.
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

Result:

(gdb) bt
#0  0xb6a63cc8 in ?? ()
#1  0xb6a5feb0 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Wait, did you compile this with Clang?

On August 17, 2015 9:24:50 AM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>Result:
>
>(gdb) bt
>#0  0xb6a63cc8 in ?? ()
>#1  0xb6a5feb0 in ?? ()
>Backtrace stopped: previous frame identical to this frame (corrupt
>stack?)
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

No...compiled with GCC 4.9.2

On August 17, 2015 9:39:55 AM CDT, Ryan Gonzalez <report@bugs.python.org> wrote:
>
>Ryan Gonzalez added the comment:
>
>Wait, did you compile this with Clang?
>
>On August 17, 2015 9:24:50 AM CDT, Cyd Haselton
><report@bugs.python.org> wrote:
>>
>>Cyd Haselton added the comment:
>>
>>Result:
>>
>>(gdb) bt
>>#0  0xb6a63cc8 in ?? ()
>>#1  0xb6a5feb0 in ?? ()
>>Backtrace stopped: previous frame identical to this frame (corrupt
>>stack?)
>>
>>----------
>>
>>_______________________________________
>>Python tracker <report@bugs.python.org>
>><http://bugs.python.org/issue23496>
>>_______________________________________
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

A few questions for Russell:

>What hardware architecture are you compiling for? If it's ARM64, and you're not using a trunk version of libffi, that segfault in test_ctypes is to be expected.

Does this mean I can safely ignore the segfault?

>Are you using the libffi sources vendored into the Python source tree, or a more recent version? 

By 'vendored in' do you mean 'sources included in python source tree for building?'

>I can verify that libffi v3.2 works on ARMv7 (on iOS, anyway), and there's been plenty of changes to the ARM source tree since the Python version was vendored in.

Would your recommend downloading and building libffi from sources (on device) and then building python?

I'm asking the above questions because I've hit a fairly significant roadblock; I'm still getting the segfault when test_ctypes is run and I can't seem to get anything useful out of gdb.

>>What hardware architecture are you compiling for? If it's ARM64, and you're not using a trunk version of libffi, that segfault in test_ctypes is to be expected.

> Does this mean I can safely ignore the segfault?

Well, "safely" in the sense that everything except a very recent version of libffi is known to not work on ARM64 - so if it doesn't work, it's not because of something Python is doing wrong, it's a problem with a dependency.

>>Are you using the libffi sources vendored into the Python source tree, or a more recent version? 

>By 'vendored in' do you mean 'sources included in python source tree for building?'

Correct. The libffi source code that is in the Python source tree is quite old, and *definitely* doesn't work on ARM64. I'm not even sure that it works on ARMv7.

>>I can verify that libffi v3.2 works on ARMv7 (on iOS, anyway), and there's been plenty of changes to the ARM source tree since the Python version was vendored in.

> Would your recommend downloading and building libffi from sources (on device) and then building python?

Well, for starters - as I've said before, I'd recommend not compiling on device at all, but that's a separate issue. 

However, regardless of where you're compiling, you can either use an external libffi, or you can do what I've done in the iOS patch - update the Python source tree to include a newer version of libffi. If you update the source in the Python tree, you need to use 2 versions (or a merged version); you need v3.2 sources for ARMv7, and trunk sources for ARM64. This is because libffi v3.2 doesn't work for ARM64, and trunk doesn't even compile for ARMv7. See the iOS patch for details.

> I'm asking the above questions because I've hit a fairly significant roadblock; I'm still getting the segfault when test_ctypes is run and I can't seem to get anything useful out of gdb.

Personally, I've pretty much given up on CPython on Android. Even when I got it working, the performance of the JNI layer is *abysmal*, and severely crippled. If you're planning to actually interact with the device in any way (like, say, put a button on the screen), that's a big problem. 

I'm working on an approach that uses Java natively - think "Jython for Android". I'm still a way off having anything working, though.

Question for Ryan Gonzalez:
Given this information...

On August 20, 2015 8:03:13 PM CDT, Russell Keith-Magee <report@bugs.python.org> wrote:
>
>Russell Keith-Magee added the comment:
>
>>>What hardware architecture are you compiling for? If it's ARM64, and
>you're not using a trunk version of libffi, that segfault in
>test_ctypes is to be expected.
>
>> Does this mean I can safely ignore the segfault?
>
>Well, "safely" in the sense that everything except a very recent
>version of libffi is known to not work on ARM64 - so if it doesn't
>work, it's not because of something Python is doing wrong, it's a
>problem with a dependency.
>

...do you know of a way to run all tests except _ctypes, so that Imcan verify everything else?

And to answer Russell's questions:

>>>Are you using the libffi sources vendored into the Python source
>tree, or a more recent version? 
>
>>By 'vendored in' do you mean 'sources included in python source tree
>for building?'
>
>Correct. The libffi source code that is in the Python source tree is
>quite old, and *definitely* doesn't work on ARM64. I'm not even sure
>that it works on ARMv7.
>
Definitely using the vendored in sources.

>
>> Would your recommend downloading and building libffi from sources (on
>device) and then building python?
>
>Well, for starters - as I've said before, I'd recommend not compiling
>on device at all, but that's a separate issue. 
>
Any particular reason why? The device I'm working with has a port of GCC which I've used to build other utilities that work well...given that I'm working on a tablet.

>However, regardless of where you're compiling, you can either use an
>external libffi, or you can do what I've done in the iOS patch - update
>the Python source tree to include a newer version of libffi. If you
>update the source in the Python tree, you need to use 2 versions (or a
>merged version); you need v3.2 sources for ARMv7, and trunk sources for
>ARM64. This is because libffi v3.2 doesn't work for ARM64, and trunk
>doesn't even compile for ARMv7. See the iOS patch for details.
>
Since I'm not compiling for ARM64...and have zero experience with hacking configure.ac files...would it be okay to include just the 3.2 sources if I note somewhere that this fork does not include ARM64 support?

>> I'm asking the above questions because I've hit a fairly significant
>roadblock; I'm still getting the segfault when test_ctypes is run and I
>can't seem to get anything useful out of gdb.
>
>Personally, I've pretty much given up on CPython on Android. Even when
>I got it working, the performance of the JNI layer is *abysmal*, and
>severely crippled. If you're planning to actually interact with the
>device in any way (like, say, put a button on the screen), that's a big
>problem. 
>
My goal is to build a port that operates on Android in a Linux-like environment; I'm currently using KBOX3.  I've no experience with the Python language so JNI interop is Greek to me.  For what i'm using the port for it functions fairly well.

>I'm working on an approach that uses Java natively - think "Jython for
>Android". I'm still a way off having anything working, though.
>
I'll keep an eye out for it once I've got Python basics under my belt.
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

On August 21, 2015 9:25:10 AM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>Question for Ryan Gonzalez:
>Given this information...
>
>On August 20, 2015 8:03:13 PM CDT, Russell Keith-Magee
><report@bugs.python.org> wrote:
>>
>>Russell Keith-Magee added the comment:
>>
>>>>What hardware architecture are you compiling for? If it's ARM64, and
>>you're not using a trunk version of libffi, that segfault in
>>test_ctypes is to be expected.
>>
>>> Does this mean I can safely ignore the segfault?
>>
>>Well, "safely" in the sense that everything except a very recent
>>version of libffi is known to not work on ARM64 - so if it doesn't
>>work, it's not because of something Python is doing wrong, it's a
>>problem with a dependency.
>>
>
>...do you know of a way to run all tests except _ctypes, so that Imcan
>verify everything else?
>

You can temporarily change the last line of Lib/ctypes/test/__init__.py to:

return args[1]

>And to answer Russell's questions:
>
>>>>Are you using the libffi sources vendored into the Python source
>>tree, or a more recent version? 
>>
>>>By 'vendored in' do you mean 'sources included in python source tree
>>for building?'
>>
>>Correct. The libffi source code that is in the Python source tree is
>>quite old, and *definitely* doesn't work on ARM64. I'm not even sure
>>that it works on ARMv7.
>>
>Definitely using the vendored in sources.
>
>>
>>> Would your recommend downloading and building libffi from sources
>(on
>>device) and then building python?
>>
>>Well, for starters - as I've said before, I'd recommend not compiling
>>on device at all, but that's a separate issue. 
>>
>Any particular reason why? The device I'm working with has a port of
>GCC which I've used to build other utilities that work well...given
>that I'm working on a tablet.
>
>>However, regardless of where you're compiling, you can either use an
>>external libffi, or you can do what I've done in the iOS patch -
>update
>>the Python source tree to include a newer version of libffi. If you
>>update the source in the Python tree, you need to use 2 versions (or a
>>merged version); you need v3.2 sources for ARMv7, and trunk sources
>for
>>ARM64. This is because libffi v3.2 doesn't work for ARM64, and trunk
>>doesn't even compile for ARMv7. See the iOS patch for details.
>>
>Since I'm not compiling for ARM64...and have zero experience with
>hacking configure.ac files...would it be okay to include just the 3.2
>sources if I note somewhere that this fork does not include ARM64
>support?
>
>>> I'm asking the above questions because I've hit a fairly significant
>>roadblock; I'm still getting the segfault when test_ctypes is run and
>I
>>can't seem to get anything useful out of gdb.
>>
>>Personally, I've pretty much given up on CPython on Android. Even when
>>I got it working, the performance of the JNI layer is *abysmal*, and
>>severely crippled. If you're planning to actually interact with the
>>device in any way (like, say, put a button on the screen), that's a
>big
>>problem. 
>>
>My goal is to build a port that operates on Android in a Linux-like
>environment; I'm currently using KBOX3.  I've no experience with the
>Python language so JNI interop is Greek to me.  For what i'm using the
>port for it functions fairly well.
>
>>I'm working on an approach that uses Java natively - think "Jython for
>>Android". I'm still a way off having anything working, though.
>>
>I'll keep an eye out for it once I've got Python basics under my belt.
>>----------
>>
>>_______________________________________
>>Python tracker <report@bugs.python.org>
>><http://bugs.python.org/issue23496>
>>_______________________________________
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

Thanks Ryan.

Running ./python -m test (with the edit to the __init__.py for ctypes) produces the following error:

[151/390/18] test_hash
Fatal Python error: Bus error

Current thread 0xb6f72ec8 (most recent call first):
  File "/bld/pyt/cpython-android/Lib/test/test_hash.py", line 89 in test_unaligned_buffers
  File "/bld/pyt/cpython-android/Lib/unittest/case.py", line 577 in run
  File "/bld/pyt/cpython-android/Lib/unittest/case.py", line 625 in __call__
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 122 in run
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 84 in __call__
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 122 in run
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 84 in __call__
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 122 in run
  File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 84 in __call__
  File "/bld/pyt/cpython-android/Lib/test/support/__init__.py", line 1668 in run
  File "/bld/pyt/cpython-android/Lib/test/support/__init__.py", line 1769 in _run_suite
  File "/bld/pyt/cpython-android/Lib/test/support/__init__.py", line 1803 in run_unittest
  File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 1279 in test_runner
  File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 1280 in runtest_inner
  File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 978 in runtest
  File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 763 in main
  File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 1564 in main_in_temp_cwd  File "/bld/pyt/cpython-android/Lib/test/__main__.py", line 3 in <module>
  File "/bld/pyt/cpython-android/Lib/runpy.py", line 85 in _run_code
  File "/bld/pyt/cpython-android/Lib/runpy.py", line 170 in _run_module_as_main
Bus error

Not sure what a bus error is...off to Google

Bus error is basically unaligned memory access.

...

...do you feel like trying to get a backtrace from GDB again? :) (The last
time likely didn't end well because the machine stack got somehow
corrupted.)

On Fri, Aug 21, 2015 at 2:04 PM, Cyd Haselton <report@bugs.python.org>
wrote:

>
> Cyd Haselton added the comment:
>
> Thanks Ryan.
>
> Running ./python -m test (with the edit to the __init__.py for ctypes)
> produces the following error:
>
> [151/390/18] test_hash
> Fatal Python error: Bus error
>
> Current thread 0xb6f72ec8 (most recent call first):
>   File "/bld/pyt/cpython-android/Lib/test/test_hash.py", line 89 in
> test_unaligned_buffers
>   File "/bld/pyt/cpython-android/Lib/unittest/case.py", line 577 in run
>   File "/bld/pyt/cpython-android/Lib/unittest/case.py", line 625 in
> __call__
>   File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 122 in run
>   File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 84 in
> __call__
>   File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 122 in run
>   File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 84 in
> __call__
>   File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 122 in run
>   File "/bld/pyt/cpython-android/Lib/unittest/suite.py", line 84 in
> __call__
>   File "/bld/pyt/cpython-android/Lib/test/support/__init__.py", line 1668
> in run
>   File "/bld/pyt/cpython-android/Lib/test/support/__init__.py", line 1769
> in _run_suite
>   File "/bld/pyt/cpython-android/Lib/test/support/__init__.py", line 1803
> in run_unittest
>   File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 1279 in
> test_runner
>   File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 1280 in
> runtest_inner
>   File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 978 in runtest
>   File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 763 in main
>   File "/bld/pyt/cpython-android/Lib/test/regrtest.py", line 1564 in
> main_in_temp_cwd  File "/bld/pyt/cpython-android/Lib/test/__main__.py",
> line 3 in <module>
>   File "/bld/pyt/cpython-android/Lib/runpy.py", line 85 in _run_code
>   File "/bld/pyt/cpython-android/Lib/runpy.py", line 170 in
> _run_module_as_main
> Bus error
>
> Not sure what a bus error is...off to Google
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23496>
> _______________________________________
>

Update: 
I found this bug report (https://bugs.python.org/issue23786) and re-compiled python with -mno-unaligned-access as mentioned. test_hash still throws a bus error.

Back to drawing board....suggestions welcome.

I'm assuming gdb still doesn't work?

On August 22, 2015 1:06:41 PM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>Update: 
>I found this bug report (https://bugs.python.org/issue23786) and
>re-compiled python with -mno-unaligned-access as mentioned. test_hash
>still throws a bus error.
>
>Back to drawing board....suggestions welcome.
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

On August 22, 2015 2:47:42 PM CDT, Ryan Gonzalez <report@bugs.python.org> wrote:
>
>Ryan Gonzalez added the comment:
>
>I'm assuming gdb still doesn't work?

It does. I just don't know it well enough to know if the errors are something I'm doing wrong or something the program is doing wrong.

Basically I'm running
gdb
file ./python
set args -m test
set sysroot /path/to/sysroot
run

and examining/posting the results.  Which are confusing to me because, again, don't know it well enough.
>
>On August 22, 2015 1:06:41 PM CDT, Cyd Haselton
><report@bugs.python.org> wrote:
>>
>>Cyd Haselton added the comment:
>>
>>Update: 
>>I found this bug report (https://bugs.python.org/issue23786) and
>>re-compiled python with -mno-unaligned-access as mentioned. test_hash
>>still throws a bus error.
>>
>>Back to drawing board....suggestions welcome.
>>
>>----------
>>
>>_______________________________________
>>Python tracker <report@bugs.python.org>
>><http://bugs.python.org/issue23496>
>>_______________________________________
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

After typing 'run', enter 'bt' and post the results.

On Sat, Aug 22, 2015 at 3:06 PM, Cyd Haselton <report@bugs.python.org>
wrote:

>
> Cyd Haselton added the comment:
>
> On August 22, 2015 2:47:42 PM CDT, Ryan Gonzalez <report@bugs.python.org>
> wrote:
> >
> >Ryan Gonzalez added the comment:
> >
> >I'm assuming gdb still doesn't work?
>
> It does. I just don't know it well enough to know if the errors are
> something I'm doing wrong or something the program is doing wrong.
>
> Basically I'm running
> gdb
> file ./python
> set args -m test
> set sysroot /path/to/sysroot
> run
>
> and examining/posting the results.  Which are confusing to me because,
> again, don't know it well enough.
> >
> >On August 22, 2015 1:06:41 PM CDT, Cyd Haselton
> ><report@bugs.python.org> wrote:
> >>
> >>Cyd Haselton added the comment:
> >>
> >>Update:
> >>I found this bug report (https://bugs.python.org/issue23786) and
> >>re-compiled python with -mno-unaligned-access as mentioned. test_hash
> >>still throws a bus error.
> >>
> >>Back to drawing board....suggestions welcome.
> >>
> >>----------
> >>
> >>_______________________________________
> >>Python tracker <report@bugs.python.org>
> >><http://bugs.python.org/issue23496>
> >>_______________________________________
> >
> >----------
> >
> >_______________________________________
> >Python tracker <report@bugs.python.org>
> ><http://bugs.python.org/issue23496>
> >_______________________________________
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue23496>
> _______________________________________
>

UPDATE:
Still working on the test_hash issue; not making much progress

QUESTION
Does gdb have to be configured with the --with-python switch in order to debug the python binary correctly?

I don't think so. I believe that switch is just to enable writing gdb extensions in Python.

On August 26, 2015 9:34:29 AM CDT, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>UPDATE:
>Still working on the test_hash issue; not making much progress
>
>QUESTION
>Does gdb have to be configured with the --with-python switch in order
>to debug the python binary correctly?
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

Ryan,

Here's the results after the edit to the ctypes test. I also re-compiled gdb with python support, which is why this took so long

(gdb) file ./python

Load new symbol table from "./python"? (y or n) y
Reading symbols from ./python...done.
Traceback (most recent call last):
  File "/bld/pyt/cpython-android/python-gdb.py", line 59, in <module>
    _type_char_ptr = gdb.lookup_type('char').pointer() # char*
AttributeError: 'module' object has no attribute 'lookup_type'

(gdb) set args -m tests
(gdb) set sysroot /usr/gcc-4.9-pie/sysroot
(gdb) run

Starting program: /bld/pyt/cpython-android/python -m tests
setpgrp failed in child: No such process
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Program received signal SIGILL, Illegal instruction.
0xb6a63cc8 in ?? ()
(gdb) bt
Python Exception <class 'ImportError'> No module named 'gdb.frames':
#0  0xb6a63cc8 in ?? ()
#1  0xb6a5feb0 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Still battling this bus error issue.  Recompiling gdb --with-python did not help; I still cannot get any useful info out of it. I suspect it may be due to this error:

setpgrp failed in child: No such process

but I'm not sure.

At this point I'm going to try downloading the Google-patched version...hopefully it is patched such that the setpgrp bug is resolved.

UPDATE: Before downloading/building Google gdb source I ran test_gdb.py, which failed completely.  Details of those tests are attached.

UPDATE: Finslly rebuilding Python from github clone after having to factory reset my tablet for reasons (root access).  Hopefully build and tests will benefit from paving and starting over.

Unfortunately, in spite of the other minor issues that resolved,I am still getting a bus error when the hash test runs.

Before I dive back into trying to get gdb working to debug this, is there any way to skip this test to see of the others run?

GDB debugging isn't going to happen any time soon; all of the gdb tests fail.
I re-compiled with the --with-pydebug option, thinking that would help, but it won't build; I get the following error when the newly built python starts to build packages:

Debug memory block at address p=0xb6003070: API 'a'
    795107700 bytes originally requested
    The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
        at p-3: 0x2f *** OUCH
        at p-2: 0x00 *** OUCH
        at p-1: 0x00 *** OUCH
    Because memory is corrupted at the start, the count of bytes requested
       may be bogus, and checking the trailing pad bytes may segfault.
    The 4 pad bytes at tail=0xe56491e4 are Segmentation fault
generate-posix-vars failed

Additional things I've attemptedto try to resolve the test_hash failure, based on this  bug report (https://mail.python.org/pipermail/python-bugs-list/2015-March/267442.html)
1) compiling with -mno-unaligned-access
2) compiling with FNV as the default hash algorithm.

Neither worked. At this point I am at a loss and am beginning to think there may be a problem with my device.  Any suggestions are welcome.

Are there no binaries?

Pretty sure it isn't your device. I'd think a compiler issue. What are you using, and what version?

You have insane determination! :D

On November 1, 2015 8:34:21 AM CST, Cyd Haselton <report@bugs.python.org> wrote:
>
>Cyd Haselton added the comment:
>
>GDB debugging isn't going to happen any time soon; all of the gdb tests
>fail.
>I re-compiled with the --with-pydebug option, thinking that would help,
>but it won't build; I get the following error when the newly built
>python starts to build packages:
>
>Debug memory block at address p=0xb6003070: API 'a'
>    795107700 bytes originally requested
>    The 3 pad bytes at p-3 are not all FORBIDDENBYTE (0xfb):
>        at p-3: 0x2f *** OUCH
>        at p-2: 0x00 *** OUCH
>        at p-1: 0x00 *** OUCH
> Because memory is corrupted at the start, the count of bytes requested
>       may be bogus, and checking the trailing pad bytes may segfault.
>    The 4 pad bytes at tail=0xe56491e4 are Segmentation fault
>generate-posix-vars failed
>
>Additional things I've attemptedto try to resolve the test_hash
>failure, based on this  bug report
>(https://mail.python.org/pipermail/python-bugs-list/2015-March/267442.html)
>1) compiling with -mno-unaligned-access
>2) compiling with FNV as the default hash algorithm.
>
>Neither worked. At this point I am at a loss and am beginning to think
>there may be a problem with my device.  Any suggestions are welcome.
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue23496>
>_______________________________________

Pretty sure there are binaries, but the python binary can't build its own modules.

I'm using GCC 4.9.2 on Android L...which means it is a PIE and that Python is being built as a PIE

Key note: it looks like the config option --with-hash-algorithm does nothing; python is still built with the siphash hash algorithim even though I specified fnv and, in pyconfig.h:

/* Define hash algorithm for str, bytes and memoryview. SipHash24: 1, FNV: 2,
   externally defined: 0 */
#define Py_HASH_ALGORITHM 2

What's the correct way to build with fnv as the hash algorithm?

Ryan, please sign the PSF Contributor Agreement
https://www.python.org/psf/contrib/
https://www.python.org/psf/contrib/contrib-form/

Also, when replying by email, please delete the previous message, except possibly for quoting a line or two.  When listed on the web page, big quotations are useless noise.

FYI: https://mail.python.org/pipermail/python-dev/2016-April/144320.html

It seems like attached patches are outdated: Python has been fixed to support Android. For example, Python now uses the UTF-8 codec rather than mbstowcs() to decode byte strings. I'm quite sure that there are still corner cases which are not well supported on Android, but I would suggest to open more specific issues in that case.