This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Python library structure creates hard to read code when using higher order functions
Type: enhancement Stage:
Components: Versions: Python 3.3
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, dwt, eric.araujo, stutzbach, terry.reedy
Priority: normal Keywords:

Created on 2012-01-17 09:48 by dwt, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (9)
msg151435 - (view) Author: Martin Häcker (dwt) Date: 2012-01-17 09:48
Code that uses higher order methods is often the clearest description of what you want to do. However since the higher order methods in python (filter, map, reduce) are free functions and aren't available on collection classes as methods, using them creates really hard to read code.

For example, if I want to get an attribute out of an array of objects and concatenate the results it's two steps, get the attributes out of the object, then concatenate the results.

Without higher order methods its like this. Uses three lines, intermediate variable is quite clear, but hard to compose and hard to abstract out parts of it.

 self.questions = []
 for topic in self.topics:
   self.questions.append(topic.questions)

if I want to do it with higher order functions there's really two ways, do it in one step with reduce:
 self.questions = reduce(lambda memo, topic: memo + topic.questions, self.topics, [])

Or use two steps with the operator module
 self.questions = reduce(operator.add, map(operator.attrgetter('questions'), self.topics), [])

Of these thee first still couples two steps into one lambda, while the second one decoples everything nicely but is a total train wreck to read, as you need to constantly jump back and forth in the line to understand what it does.

Having map and reduce defined on collections would not only make it drastically shorter but also allows you to read it from front to back in one go. (Ok, there is still the caveat that the assignment is in the front instead of at the end, but hey)

 self.questions = self.topics.map(attrgetter('questions')).reduce(add, [])

That would be nicely separated into individual steps that exactly describe what each step is actually doing, is easy to abstract over and actually very succinct.
msg151454 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-01-17 14:01
Did you consider list comprehension?

self.questions = sum((topic.questions for topic in self.topics), [])
msg151457 - (view) Author: Martin Häcker (dwt) Date: 2012-01-17 14:40
Yes - however it has the same problem as the higher order version in the python libraries that to read it you need to constantly jump back and forth in the line.
msg151465 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-01-17 16:20
I think this discussion would be more useful on the python-ideas mailing list.  The request (adding map to sequences) will probably be rejected*, but you have a good chance to get an explanation for this design choice by Guido van Rossum or one of the old-timer core devs.

*I think so because of str.join: people have said that they would prefer a list operation, but it’s always been denied because it would put the burden on any sequency type to implement the method, whereas with a str method then all iterables and iterators are supported for free.
msg151483 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2012-01-17 19:42
If I'm understanding Martin Häcker's code correctly, the list comprehension equivalent is:

self.questions = [topic.questions for topic in self.topics]

The reduce() variants are not only much harder to read, but they will take O(n**2) operations because they create an O(n) list O(n) times.
msg151526 - (view) Author: Martin Häcker (dwt) Date: 2012-01-18 09:31
@stutzbach: I believe you got me wrong, as the example topic.questions is meant to return a list of questions that need concatenating - thus you can't save the second step.
msg151555 - (view) Author: Daniel Stutzbach (stutzbach) (Python committer) Date: 2012-01-18 17:54
Ah - in your first example (the one with 3 lines) did you mean to use .extend instead of .append?
msg151611 - (view) Author: Martin Häcker (dwt) Date: 2012-01-19 08:45
Jup - oh the joys of writing code in a bugtracker :)
msg151713 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-01-21 03:10
I am closing this because map has even less chance of being made a collection method than join. Unlike join, map takes any positive number of iterables as args, not just one. 'Iterables' includes iterators, which intentionally need have no methods other than __iter__ and __next__.

If map were an attribute, it should be an attribute of 'function' (except that there is no one function class).

To abstract attributes, use get/setattr. Untested example: 

def atcat(self, src,  src_at, dst):
  res = []
  for col in getattr(self, src):
    res += getattr(col, src_at)
  setattr(self, dst, res)
History
Date User Action Args
2022-04-11 14:57:25adminsetgithub: 58012
2012-01-21 03:10:55terry.reedysetstatus: open -> closed
versions: + Python 3.3
nosy: + terry.reedy

messages: + msg151713

resolution: rejected
2012-01-19 08:45:07dwtsetmessages: + msg151611
2012-01-18 17:54:32stutzbachsetmessages: + msg151555
2012-01-18 09:31:51dwtsetmessages: + msg151526
2012-01-17 19:42:34stutzbachsetnosy: + stutzbach
messages: + msg151483
2012-01-17 16:20:07eric.araujosetnosy: + eric.araujo
messages: + msg151465
2012-01-17 14:40:38dwtsetmessages: + msg151457
2012-01-17 14:01:38amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg151454
2012-01-17 09:48:36dwtcreate