This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author Matthew Newville
Recipients Matthew Newville, cheryl.sabella, vinay.sajip
Date 2020-01-14.05:54:28
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1578981268.84.0.724553152494.issue39295@roundup.psfhosted.org>
In-reply-to
Content
So, again, I'm trying to understand what the best workaround for this change is.  I asked "can this workaround be improved" twice and got no reply, while getting plenty of responses to questions about the development process.  I take this to mean that the workaround we have is the best available. That's unfortunate.

>> it appears that an old, poorly verified bug report
>
> Why do you say the bug report is poorly verified? The libffi maintainers
> accept it is an issue.
>
> https://github.com/libffi/libffi/issues/33

Well, it is more than six years old. It did not appear that lots of people were saying "yeah me too, please fix!".  Maybe I missed those calls of urgency? 


>> inspired a change that was actually not well tested and so incorrectly 
>> broke valid code without deprecation. Trying to be as polite as possible, >> this appears to indicate a poor testing process, if not a poor 
>> understanding of the actual code in question.

> Well, the original developer of ctypes is no longer involved in maintaining > it. Those of us who try to address ctypes issues are perhaps not as well-
> versed in the code as the original developer and maintainer, but we do our
> best. This of course applies to other areas in CPython, and many other
> projects besides.

I think you are agreeing with me ;).  That worries me.

> The change was accompanied by tests, which have been no doubt found 
> wanting, 

It appears that the change was *intended* to fix a problem with Unions, but had the unintended consequence of not allowing any structures with bitfields. That suggests that the ctypes tests don't include structures with bitfields, These seem sort of common to me, so it appears that testing of ctypes is far less comprehensive than I had imagined.

> but do *you* write software which guarantees no bugs ever in released 
> versions? 

Of course not, and that is not the expectation.  It's a bit alarming to hear Python devs be so defensive and using such straw man arguments.  What is expected is that working code does not break without warning or deprecation and that testing is sort of complete. It is expected that when changes unintentionally break working code that the devs take a step back and say "wait, how did that happen? what are we not testing that our users are expecting to work?". It is also expected that problems are acknowledged and fixed in a timely manner.  

And yes, to the extent possible, we try to do those things. With Py 3.7.6 available and installed with `conda update`, this break was a very urgent problem (library failed to import!) for us and our downstream users. Within 36 hours of the first report of the problem, we had a workaround verified and pushed to PyPI. The response of Python team to the original problem and to the unintended breakage were much slower.  
 

> Using your example above, I will look into what was missed 
> and try to improve the checking. The underlying libffi issue is a real
> one. The change wasn't introduced in a cavalier manner, as you seem to
> be suggesting.

Well, the change did wait six years from the first report and yet did not include a deprecation cycle. If that TypeError had been a Warning for a few releases, you would have no doubt heard questions like "why is this working code going to be deprecated" instead of "why did Python break by library".

> Do you regularly test your code with Python alpha and beta versions? 
> I ask because I may reinstate the check for Python 3.9 after seeing what
> false-positive cases were missed here. Python 3.9 is at alpha 2 level 
> right now. Continued feedback could help to minimise the chances of 
> future surprises.

I have never tested an `alpha` versions, and not used many `beta` version either (code development is not actually my full-time job) -- certainly not in the Python 3 series. I have trusted the Python devels.  This has worked well for many years and Python versions. But that trust, especially concerning ctypes, is diminished (a structure with bitfields was unexpected usage??) and we probably should be testing beta versions regularly.
History
Date User Action Args
2020-01-14 05:54:28Matthew Newvillesetrecipients: + Matthew Newville, vinay.sajip, cheryl.sabella
2020-01-14 05:54:28Matthew Newvillesetmessageid: <1578981268.84.0.724553152494.issue39295@roundup.psfhosted.org>
2020-01-14 05:54:28Matthew Newvillelinkissue39295 messages
2020-01-14 05:54:28Matthew Newvillecreate