If the test suite is run with -3, many deprecation warnings are
reported.  Quite a few are generated by code in the tests themselves,
but many are from constructs in the stdlib which are different or no
longer supported in 3.x.

Even aside from the fact that many of these warnings being caused by
code which I think many people would agree is not ideal, I think it is
important for these warnings to be fixed for Python 2.7.  Since the 3.x
porting guidelines recommend that projects first run their test suite
with -3, these warnings are going to be generating extra noise for
developers to filter through when they try porting their code to 3.x. 
They shouldn't need to wade through stdlib warnings to find warnings
about their own code.

Attached is a log of a complete test run with -3.

I don't think it's so important that tests not raises -3 warnings, but
that the stdlib doesn't.

I agree with Benjamin. We shouldn't release 2.7 w/ any of the standard 
library itself generating a Py3kWarning, but that should not apply to the 
test suite.

I have made this a release blocker until we can create separate issues for 
the modules that are emitting the warning.

FWIW I tried to filter out the test-related warnings using the following
command line, but it didn't work (nothing was filtered out):

$ ./python -3 -W 'ignore::DeprecationWarning:test:0' -m test.regrtest

I've fixed a bunch of them: aifc (r75407), test_atexit (r75408), bsddb
(r75409), test_calendar (r75410), StringIO (r75411), socket (r75412),
sndhdr (r75413), test_memoryio (r75415), test_profilehooks (r75417),
test_random (r75419), httplib (r75420), uuid (r75421).

Here is a record of stderr after a new regrtest run.

(interestingly, one of the culprits for py3k warnings is lib2to3)

Making this a deferred blocker so as not to clutter 2.6.4 release.

Many fixes of modules and packages outside Lib/test

Dev notes:
 * for bsddb and dbhash the warning message should include "module" to
be ignored by "test_support.import_module"
 * patch for mailbox is copied from Lib/cgi.py
 * other fixes are trivial

Patch for the compiler package and astgen.py tool.

Basically:
  def __init__(self, (left, right), lineno=None):

==>

  def __init__(self, leftright, lineno=None):

Patch for pprint: when keys are not the same type, "-3" emits warnings.

Partial backport from Py3k.

In issue7092_lib_many_fixes.diff: why did you change some of the warning messages?

In issue7092_pprint.diff: I'm not sure these changes are right, because they could modify existing behaviour. Raymond, what do you think?

Updated "lib_many_fixes" patch:
 * added filterwarning for pprint
 * added fixes for Lib/sunau.py and Lib/unittest/case.py

Ready for review and merge:
 * issue7092_lib_many_fixes_v2.diff
 * issue7092_compiler.diff

One thing: "lambda k: k in selfdata" can be replaced with
"selfdata.__contains__". It may have better performance characteristics
too.

I learned something.

Looks good, thank you.

Both patches have been committed in r77310.

Still some "-3" warnings to silence:

./python -3m "collections" >/dev/null
./python -3c "import Cookie"
./python -3c "import idlelib.rpc"
./python -3c "import logging.handlers"
./python -3c "import multiprocessing"
./python -3c "import shelve"
./python -3c "import trace"

All these modules yield:
"DeprecationWarning: the cPickle module has been removed in Python 3.0"

There's also "bsddb" but it is deprecated.

> All these modules yield:
> "DeprecationWarning: the cPickle module has been removed in Python 3.0"

I think this warning is both annoying (cPickle is a legitimate module to use in 2.x, since pickle is much slower) and useless (2to3 should be able to do the module rename -- Benjamin, does it?). Therefore I suggest to remove this warning.

> I think this warning is both annoying (cPickle is a legitimate module to use in 2.x, since pickle is much slower) and useless (2to3 should be able to do the module rename -- Benjamin, does it?). Therefore I suggest to remove this warning.

You may be right, but keep in mind that cPickle and pickle are only *mostly* compatible with each other.  There are uses of one which will not automatically work if you switch to the other.

+1 to remove this boring cPickle message.

cStringIO do not print such messages.

>Antoine> (2to3 should be able to do the module rename -- Benjamin, does it?).
2to3 will rename cPickle to pickle.

The Big Patch™.

It removes 99% of noisy deprecation messages.

I split the Big Patch™ into pieces.
It should be easier to review and merge.

 - issue7092_syntax_imports.diff
   --> Only syntax and import_module(m, deprecated=True)

 - issue7092_check_warnings.diff
   --> Use test.test_support.check_warnings()

 - issue7092_filterwarnings.diff
   --> Use warnings.filterwarnings()

 - issue7092_json_sqlite.diff
   --> Silence external libraries

 - issue7092_cpickle.diff
   --> Remove cPickle warning

And there's a last patch for "test_index" attached to issue7532.


Then you reach the Zen of Python.
  “Do not speak — unless it improves on silence.”

About issue7092_json_sqlite.diff: if json produces py3k warnings in normal use (does it?), the warnings should be suppressed in json itself, not in regrtest.py.

Patch for JSON and SQLite libraries updated.

issue7092_cpickle.diff was committed to trunk and 2.6.

This can wait.

Hello

I read the diff for revision 77310 and noticed that arguments with parens
(I mean things like ``def spam((x, y))``, forgot the name) were replaced
by grab-all arguments that are unpacked later (``def spam(x_y)``). I was
wondering whether the documentation for these changed functions was still
accurate when describing the arguments they take. It seems none of them
have a docstring, and docs.python.org/dev/ isn’t up to date so I can’t
check right now.

Kind regards

The documentation is still accurate. The translation is literal so there no
shift except where the unpacking exception occurs.

On Sat, Jan 9, 2010 at 20:13, Éric Araujo <report@bugs.python.org> wrote:

>
> Éric Araujo <merwok@netwok.org> added the comment:
>
> Hello
>
> I read the diff for revision 77310 and noticed that arguments with parens
> (I mean things like ``def spam((x, y))``, forgot the name) were replaced
> by grab-all arguments that are unpacked later (``def spam(x_y)``). I was
> wondering whether the documentation for these changed functions was still
> accurate when describing the arguments they take. It seems none of them
> have a docstring, and docs.python.org/dev/ isn’t up to date so I can’t
> check right now.
>
> Kind regards
>
> ----------
> nosy: +Merwok
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue7092>
> _______________________________________
>

I've created issue7723 for sqlite3 module using buffer().
It's not so simple.

Patches updated against trunk.

Re-uploaded "syntax_imports_v2" with patch for test_bsddb.py

Re-uploaded after fixes for #7737.

To remove all "-3" warnings:
 - apply the 4 patches
 - fix SQLite "buffer()" warnings (issue #7223)

json is now fixed in r77755.

Here is my review of issue7092_syntax_imports_v3.diff:

- test_itertools.py: please replace
      [tuple([arg[i] if i < len(arg) else None for arg in args])
       for i in range(max(map(len, args)))]
  by something more readable (nested for loops for example)

- test_mailbox.py: why doesn't test_support.import_module('rfc822')
  specify deprecated=True? maybe the module can be imported normally...

- test_pyclbr.py:   replace self.assertTrue(key in obj) with assertIn()
- test_wsgiref.py:  replace self.assertTrue(key in h.environ)

- test_queue.py: test could be identical to the one in py3k.

Here's mine about issue7092_check_warnings_v3.diff:

1) test_callable should keep testing callable() and the warnings should be caught;
2) in test_bsddb3 the problems should be correct in the module if possible and worth it (the module is deprecated);
3) next to the several '# Silence py3k warnings' it would be nice to have a note about what warning you are exactly silencing;
4) def test_deprecated_builtin_map -> test_deprecated_builtin_map_with_None, otherwise it seems that map is deprecated;
5) in test[_deep]_copy I'm not entirely sure that the tests are equivalent using in (and if they are you should use assertIn);
6) in test_socket I would keep callable, also shouldn't the raise in the next line raise a warning as well?;
7)  the self.assertEqual(`u2`, `d2`) in test_userdict could just use repr() instead;
8) a few tests in test_weakref should use assert[Not]In instead of assertTrue(x [not] in y).

A couple of comments about issue7092_syntax_imports_v3.diff too:
1) in test_copy you remove (k,v), but left the name 'k' even if now it represent the item and not the key;
2) in test_fractions you should probably use self.fail() instead of an assert;
3) on test_ftp you can use two separate lines and remove the ';';
4) in test_pyclbr there's one extra 'f' in the comment in assertHaskey;
5) in test_xml_etree_c you can either leave callable() and catch the warning or use isinstance(x, collections.Callable) instead (there are also a few more places where callable was used too).

Thank you for the review and your comments.
Here, the replies.


>
> Amaury review - msg98491
>
> Here is my review of issue7092_syntax_imports_v3.diff:
>
> - test_itertools.py: please replace
>       [tuple([arg[i] if i < len(arg) else None for arg in args])
>        for i in range(max(map(len, args)))]
>   by something more readable (nested for loops for example)

It is the 3.x syntax.
--> No change

> - test_mailbox.py: why doesn't test_support.import_module('rfc822')
>   specify deprecated=True? maybe the module can be imported normally...

Added the "deprecated=True"

> - test_pyclbr.py:   replace self.assertTrue(key in obj) with assertIn()
> - test_wsgiref.py:  replace self.assertTrue(key in h.environ)
> - test_queue.py: test could be identical to the one in py3k.

Thanks
--> Done

> 
> Ezio review - msg98494
> 
> Here's mine about issue7092_check_warnings_v3.diff:
> 
> 1) test_callable should keep testing callable() and the warnings should be caught;

Done

> 2) in test_bsddb3 the problems should be correct in the module if possible and worth it (the module is deprecated);

Ok. I prepare a separate patch to fix bsddb3

> 3) next to the several '# Silence py3k warnings' it would be nice to have a note about what warning you are exactly silencing;
> 4) def test_deprecated_builtin_map -> test_deprecated_builtin_map_with_None, otherwise it seems that map is deprecated;

Done

> 5) in test[_deep]_copy I'm not entirely sure that the tests are equivalent using in (and if they are you should use assertIn);

Will use assertSetEqual, no need to reinvent the wheel

> 6) in test_socket I would keep callable, also shouldn't the raise in the next line raise a warning as well?;

Ok to keep callable. For the next line, I do not see the warning.

> 7)  the self.assertEqual(`u2`, `d2`) in test_userdict could just use repr() instead;

Done

> 8) a few tests in test_weakref should use assert[Not]In instead of assertTrue(x [not] in y).

Done

> 
> Ezio review - msg98507
> 
> A couple of comments about issue7092_syntax_imports_v3.diff too:
> 1) in test_copy you remove (k,v), but left the name 'k' even if now it represent the item and not the key;

Changed 'k'-> 'pair' (like 3.x)

> 2) in test_fractions you should probably use self.fail() instead of an assert;

The surrounding methods use "assert "
--> No change

> 3) on test_ftp you can use two separate lines and remove the ';';

Done

> 4) in test_pyclbr there's one extra 'f' in the comment in assertHaskey;

It is not an extra f. 'iff' = "if and only if" (not obvious for non-English people)

> 5) in test_xml_etree_c you can either leave callable() and catch the warning or use isinstance(x, collections.Callable) instead (there are also a few more places where callable was used too).

It is done this way in 3.x.
I propose to fix it separately (if a fix is required).



Now I prepare the updated patches.
I follow the proposal of Amaury: create a specific "test_support.silence_py3k_warnings()" context manager.
It replaces the "check_warnings" and "filterwarnings" of the proposed patches.

Each patch can be reviewed and committed separately (except the last 2).

[patch 1] bsddb3.diff – related to Bsddb3
[patch 2] test_support.diff – new helper "silence_py3k_warnings()"
[patch 3] test_exceptions.diff – "test_pep352" and "test_exceptions"
[patch 4] Wd_warnings.diff – fix few "-Wd" warnings
[patch 5] syntax_fixes.diff – fix syntax only (2to3-like)
[patch 6] silence_imports.diff – import deprecated modukes


Patches below require the new helper "silence_py3k_warnings" (patch 2)

[patch 7] py3k_warnings_args.diff – 26 files
[patch 8] py3k_warnings_noargs.diff – 28 files

Warnings for imports of deprecated modules are now silenced in r77841 (patch 6 / silence_imports.diff).

Cleanup test_exceptions:
 - removed "filterwarnings" for "testSlicing"

The patch for bsddb3 looks good, however there's no reason (IMHO) to keep in the tests those "if sys.version_info[0] < 3:" or replace the version with (2,6) since we are already using things like assertIn that are 2.7-only. I changed it only to avoid warnings, and left it unchanged in other places (not related to this issue) and in dbtables.py (to preserve compatibility in the unlikely case that someone wants to try running 2.7's bsddb with older versions of Python).
Attached a new patch.

bsddb is now fixed in r77910.

Proposed patch to filter out some deprecated comparisons in "unittest" module.

test_exceptions and test_pep352 fixed in r77913.

test_unittest fixed in r77911. See also #7837.

In issue7092_test_support:
1) silence_py3k_warning should have a docstring that explain what the function does and what filters should be, possibly including an example;
2) I'd use "if not filters:" instead of "if filters == ():";
3) are there cases where the third arg of filterwarnings is not DeprecationWarning? If not, it's probably better to turn "filters" in a list of regex instead than a sequence of tuples like (regex, DeprecationWarning);
4) maybe silence_py3k_warning could yield an instance of WarningsRecorder, in case someone wants to access the silenced warnings.

Docstring added.

Notes:
 - there's 2 categories of -3 warnings: SyntaxWarning and DeprecationWarning.
 - AFAICT, the WarningsRecorder needs "catch_warnings(record=True)".
   And in such case, *all* warnings are silenced.

Applied the syntax fixes in r77942.

The enhancement proposed on #7849 should be considered before reviewing the last 2 patches.

If #7849 is accepted, these patches will need update.

Silence the last warnings, using check_py3k_warnings.

But we could provide a smaller patch, if #7832 is fixed before.

Checking the pybsddb 4.8.4devel with current trunk I still see this:

"""
Berkeley DB 4.8.26: (December 18, 2009)
bsddb.db.version():   (4, 8, 26)
bsddb.db.__version__: 4.8.4devel1
bsddb.db.cvsid:       $Id: _bsddb.c,v 6408a812f035 2010/03/15 19:15:06 jcea $
py module:            /home/pybsddb/build/lib.solaris-2.10-i86pc-2.7/bsddb3/__init__.pyc
extension module:     /home/pybsddb/build/lib.solaris-2.10-i86pc-2.7/bsddb3/__init__.pyc
python version:       2.7a4+ (trunk, Mar 16 2010, 00:50:35) 
[GCC 4.4.3]
My pid:               5068
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
............................................................................................................................................................................................................................................................................................................................................................................................................................................./usr/local/lib/python2.7/threading.py:575: DeprecationWarning: sys.exc_clear() not supported in 3.x; use except clauses
  self.__exc_clear()
.........................................
----------------------------------------------------------------------
Ran 470 tests in 7.605s

OK

"""

Python 2.6.4 has the same issue too.

> Checking the pybsddb 4.8.4devel with current trunk I still see this:

It is because the "test_early_close" module calls resetwarnings().
It defeats the filter in the "threading" module.

You may change the test case with a context manager:

  with warnings.catch_warnings():
      warnings.resetwarnings()
      # test the warning

I took part of Florent latest patch, changed a few things and committed it in r79023.

Florent you are right, good catch. I can not use context managers because "with" is invalid syntax in python 2.3 and 2.4, that I must support for a while.

I have implemented a context manager manually, following the description in PEP 343. This change is available in pybsddb 4.8.3+. I plan to integrate pybsddb 4.8.3+, see issue8156.

More warnings silenced with r79165.

other py3k warnings silenced with r79187 r79188 r79189

Now the "-Wd" and "-3" flags are active on some buildbots.
And the first failures ...


Need to backport fixes from py3k branch for:
- cgitb "SyntaxWarning: tuple parameter unpacking has been removed in 3.x"
- idlelib "reduce() not supported in 3.x; use functools.reduce()"


And test_multiprocessing is failing on: "sys.exc_clear() not supported in 3.x; use except clauses"

Other py3k warnings:

lib\distutils\util.py:209: DeprecationWarning: apply() not supported in 3.x; use func(*args, **kwargs)
  return apply(os.path.join, paths)

lib\distutils\command\build_ext.py:679: DeprecationWarning: apply() not supported in 3.x; use func(*args, **kwargs)
  return apply(os.path.join, ext_path) + '_d' + so_ext

lib\platform.py:1035: DeprecationWarning: dict.has_key() not supported in 3.x; use the in operator
  if _default_architecture.has_key(sys.platform):

Fixed with r79558, except test_multiprocessing.

Additional fixes: r79576, r79577, r79578 (idlelib package and some test modules)

This issue is over, except for #7723 which is tracked separately.

Barry, is it OK if I backport this to 2.6 before the rc1?

Hi Ezio, what's the status on this issue for 2.6.6rc1?

I am removing this as a release blocker given all of Ezio's great work to get the test suite clean with -3.  I will leave it up to him to actually close the issue once the work is complete.  It no longer needs to block 2.6.6.

I fixed the deprecation warnings on 2.6 in r83519, r83576-r83579, r83620-r83622, r83627-r83632, r83635, r83637-83642, r83652-83656, r83658, r83687.
Now the test suite should pass with and without -3 -Wd with no errors and no warnings (except an import warning on _ctypes and/or _multiprocessing).
This can be closed.