Check the robots.txt file from mathworld.

--> http://mathworld.wolfram.com/robots.txt

It contains 2 User-Agent: * lines.

From http://www.robotstxt.org/norobots-rfc.txt

"These name tokens are used in User-agent lines in /robots.txt to
identify to which specific robots the record applies. The robot
must obey the first record in /robots.txt that contains a User-
Agent line whose value contains the name token of the robot as a 
substring. The name comparisons are case-insensitive. If no such
record exists, it should obey the first record with a User-agent
line with a "*" value, if present. If no record satisfied either
condition, or no records are present at all, access is unlimited."

But it seems that our robotparser is obeying the 2nd one. the problem
occures because robotparser assumes that no robots.txt will contain two
* user-agent. it should not have two two such line, but in reality many
site may have two.

So i have changed robotparser.py as follow:

    def _add_entry(self, entry):
        if "*" in entry.useragents:
            # the default entry is considered last
            if self.default_entry == None:   # this check is added
                   self.default_entry = entry
        else:
            self.entries.append(entry)

And at the end of parse(self, lines) method

        if state==2:
#            self.entries.append(entry)
            self._add_entry(entry)  # necessary if there is no new line
at end and last User-Agent is *

this looks like a good fix.  i've put it into my own copy.

Thanks for the patch, fixed in r83238.