Python C API evolves. For example, the Unicode type has been rewriten for scratch to use a more efficient design. Would it be possible to mark deprecated functions as deprecated, so users will be noticed that the API evolved and Python has better functions?

For example, PyUnicode_GET_SIZE() is deprecated and returns a different value than PyUnicode_GET_LENGTH() if wchar_t size is 16-bit (ex: Windows and AIX).

GCC has an nice __attribute__((deprecated)) to tag functions.

I just commited a change to avoid PyUnicode_GET_SIZE(): this function doesn't handle errors very well, if PyUnicode_AsUnicode() fails, the result is zero. The caller is unable to know that an error occurred.

http://hg.python.org/cpython/rev/28f71af02b69
"""
Don't use deprecated function PyUnicode_GET_SIZE()

Replace it with PyUnicode_GET_LENGTH() or PyUnicode_AsUnicodeAndSize()
"""

Idea looks good to me.

Does the __attribute__ work for macros as well?

Deprecated macros can be replaced by deprecated functions.

MSC has __declspec(deprecated). See http://stackoverflow.com/a/21265197, for example. It's a level 3 (/W3) warning.

http://msdn.microsoft.com/en-us/library/ttcz0bys%28v=vs.100%29.aspx

There is already Py_DEPRECATED in Include/pyport.h:

#if defined(__GNUC__) && ((__GNUC__ >= 4) || \
              (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1))
#define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
#else
#define Py_DEPRECATED(VERSION_UNUSED)
#endif


You can add this before '#else':
#elif defined(_MSC_VER) && _MSC_VER >= 1300
#define Py_DEPRECATED(VERSION_UNUSED) __declspec(deprecated)

Py_DEPRECATED is not used since Python 3.0 (it is still used in 2.7).

Ping myself!

I just cited this GCC attribute in a recent discussion on deprecating macros in the C API:
https://mail.python.org/pipermail/python-dev/2016-September/146537.html

FYI I used Py_DEPRECATED for marking PyUnicode_AsDecodedObject, PyUnicode_AsDecodedUnicode, PyUnicode_AsEncodedObject and PyUnicode_AsEncodedUnicode.

Proposed patch marks most deprecated functions. Code is rewritten for using non-deprecated functions if possible. Unfortunately some deprecated function still are used in the code and can't be easier replaced. They are left not marked.

• PyEval_ReleaseLock() is used in Python/pystate.c. It can't be replaced with PyEval_ReleaseThread() since the latter don't accept NULL.

• Py_UNICODE (currently an alias of wchar_t) is used in a number of deprecated functions and bridges between deprecated and new APIs. Maybe it can be just replaced with wchar_t.

• Macros PyUnicode_GET_SIZE, PyUnicode_GET_DATA_SIZE, PyUnicode_AS_UNICODE, PyUnicode_AS_DATA, functions PyUnicode_AsUnicode and PyUnicode_AsUnicodeAndSize are used in a number of places. They can't be easily replaced with wchar-based functions since they return a borrowed reference to cached representation.

• PyUnicode_FromUnicode, PyUnicode_EncodeDecimal and PyUnicode_TransformDecimalToASCII are used only in Modules/_testcapimodule.c. I think we should write tests for modern APIs and eliminate tests for deprecated APIs. Or temporary silence compiler warning in test functions.

• _PyUnicode_ToLowercase, _PyUnicode_ToUppercase and corresponding public macros Py_UNICODE_TOLOWER and Py_UNICODE_TOUPPER are used in Modules/_sre.c (this is a bug in regex implementation). The problem is that more modern functions _PyUnicode_ToLowerFull and _PyUnicode_ToUpperFull is a private API.

All these cases needs separate issues.

Updated documentation. This part should be backported to all maintained 3.x versions.

GCC supports pragmas to locally control warnings.
https://gcc.gnu.org/onlinedocs/gcc-6.2.0/gcc/Diagnostic-Pragmas.html

Example:
====================================
__attribute__((deprecated)) int some_deprecated_function(void)
{
return 0;
};

void some_function(void)
{
  int x, y;
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
  x = some_deprecated_function();
#pragma GCC diagnostic pop
  y = x + 1;
} 

int main(int argc, char** argv)
{
  return 0;
}

====================================
In the above example, call to some_deprecated_function() does not trigger deprecation warning.

'#pragma GCC diagnostic push' and '#pragma GCC diagnostic pop' are supported since GCC 4.6.0.
https://gcc.gnu.org/gcc-4.6/changes.html

'#pragma GCC diagnostic ignored' is documented in documentation of GCC since 4.2.0.

Clang supposedly supports both '#pragma GCC diagnostic ...' and '#pragma clang diagnostic ...':
http://clang.llvm.org/docs/UsersManual.html#controlling-diagnostics-via-pragmas

_Pragma syntax would be more suitable since it could be used with macros:
https://gcc.gnu.org/onlinedocs/gcc-6.2.0/cpp/Pragmas.html

Then Include/pyport.h (which defines Py_DEPRECATED) could also define:

#if defined(__GNUC__) && ((__GNUC__ >= 5) || (__GNUC__ == 4) && (__GNUC_MINOR__ >= 6))
#define Py_COMPILER_DIAGNOSTIC_PUSH _Pragma("GCC diagnostic push")
#define Py_COMPILER_DIAGNOSTIC_IGNORE_DEPRECATED_DECLARATIONS _Pragma("GCC diagnostic ignored \"-Wdeprecated-declarations\"")
#define Py_COMPILER_DIAGNOSTIC_POP _Pragma("GCC diagnostic pop")
#else
#define Py_COMPILER_DIAGNOSTIC_PUSH
#define Py_COMPILER_DIAGNOSTIC_IGNORE_DEPRECATED_DECLARATIONS
#define Py_COMPILER_DIAGNOSTIC_POP
#endif


These macros would be used in this way:
void some_function(void)
{
  int x, y;
Py_COMPILER_DIAGNOSTIC_PUSH
Py_COMPILER_DIAGNOSTIC_IGNORE_DEPRECATED_DECLARATIONS
  x = some_deprecated_function();
Py_COMPILER_DIAGNOSTIC_POP
  y = x + 1;
}

I am not compiler specialist, but as I do not use GCC (it adds extra run-time environment (support) requirements when not in a GNU environment such as Linux - I am just wondering what effect this has (e.g., no deprecated message) when not using GCC.

Or, is this a "statement of direction" that only GCC (syntax) compilers are (going to be) supported?

(FYI: there are OSS projects that only accept GCC, and those tend to be non-portable (imho). The "solution" in those cases is to build an additional run-time environment. Personally, I consider that non-portable as I do not want the role of having to maintain security patches for a "non-native" runtime environment (i.e., the maintenance of the (native) rte is performed by the OS supplier, not by me).

Proposed patch marks most deprecated functions. Code is rewritten for using non-deprecated functions if possible. Unfortunately some deprecated function still are used in the code and can't be easier replaced. They are left not marked.

Hum, I suggest to first update these functions instead of using compiler tricks to ignore deprecation warnings. Once CPython code base doesn't use deprecated functions anymore, deprecate functions (emit a warning).

All these cases needs separate issues.

I agree :-)

• PyEval_ReleaseLock() is used in Python/pystate.c. It can't be replaced with PyEval_ReleaseThread() since the latter don't accept NULL.

Does this feature (accept NULL) make sense outside CPython core? If not, add a private function exposing the feature.

• Py_UNICODE (currently an alias of wchar_t) is used in a number of deprecated functions and bridges between deprecated and new APIs. Maybe it can be just replaced with wchar_t.

Which functions are bridges?

Would it be possible to split these bridges into two functions: a private function which doesn't emit a warning, and a public deprecated function which calls the private function?

I dislike the idea of replacing Py_UNICODE* with wchar_t*.

• Macros PyUnicode_GET_SIZE, PyUnicode_GET_DATA_SIZE, PyUnicode_AS_UNICODE, PyUnicode_AS_DATA, functions PyUnicode_AsUnicode and PyUnicode_AsUnicodeAndSize are used in a number of places. They can't be easily replaced with wchar-based functions since they return a borrowed reference to cached representation.

We must not used these functions in CPython core, except to develop the "bridges" you mentionned before. Please open a separated issue to discuss how to handle the deprecation of the functions using Py_UNICODE*.

• PyUnicode_FromUnicode, PyUnicode_EncodeDecimal and PyUnicode_TransformDecimalToASCII are used only in Modules/_testcapimodule.c. I think we should write tests for modern APIs and eliminate tests for deprecated APIs. Or temporary silence compiler warning in test functions.

• _PyUnicode_ToLowercase, _PyUnicode_ToUppercase and corresponding public macros Py_UNICODE_TOLOWER and Py_UNICODE_TOUPPER are used in Modules/_sre.c (this is a bug in regex implementation). The problem is that more modern functions _PyUnicode_ToLowerFull and _PyUnicode_ToUpperFull is a private API.

"Py_UCS4 _PyUnicode_ToLowercase(Py_UCS4 ch)" doesn't use Py_UNICODE, what is the issue?

"#define Py_UNICODE_ISDECIMAL(ch) _PyUnicode_IsDecimalDigit(ch)" this macro doesn't use the Py_UNICODE type, so we don't have to deprecate it.

Do you want to deprecate functions like Py_UNICODE_ISDECIMAL(ch)?

Or, is this a "statement of direction" that only GCC (syntax) compilers are (going to be) supported?

This is only the first step. Since this feature is compiler specific, we should add the support for every compiler separately. Arfrever suggested the syntax for supporting this on Microsoft compiler. But I don't know how to silence warnings on this compiler. Other compilers can support GCC syntax. We should check and switch on this.

Which functions are bridges?

All deprecated Py_UNICODE related functions use Py_UNICODE. Most of them are bridges to new APIs. Py_UNICODE also is used in implementation of format codes 'u' and 'Z' of PyArg_Parse*. It also is used in many functions that are bridges to Windows API.

"Py_UCS4 _PyUnicode_ToLowercase(Py_UCS4 ch)" doesn't use Py_UNICODE, what is the issue?

The problem with this function is not related to Py_UNICODE, but that correct Unicode mapping can return multiple characters.

Do you want to deprecate functions like Py_UNICODE_ISDECIMAL(ch)?

No. It works correctly.

About MSVC compiler:

https://msdn.microsoft.com/en-us/library/044swk7y.aspx
https://msdn.microsoft.com/en-us/library/2c8f766e.aspx
https://msdn.microsoft.com/en-us/library/d9x1s805.aspx

So both:

#pragma warning(push)
#pragma warning(disable: 4996)
/* Some code */
#pragma warning(pop)

And:

__pragma(warning(push))
__pragma(warning(disable: 4996))
/* Some code */
__pragma(warning(pop))

Would generally work, but only the second form is suitable for usage inside macros.

Updated proposition of Py_COMPILER_DIAGNOSTIC_* macros:

#if defined(__GNUC__) && ((__GNUC__ >= 5) || (__GNUC__ == 4) && (__GNUC_MINOR__ >= 6))
#define Py_COMPILER_DIAGNOSTIC_PUSH _Pragma("GCC diagnostic push")
#define Py_COMPILER_DIAGNOSTIC_IGNORE_DEPRECATED_DECLARATIONS _Pragma("GCC diagnostic ignored \"-Wdeprecated-declarations\"")
#define Py_COMPILER_DIAGNOSTIC_POP _Pragma("GCC diagnostic pop")
#elif defined(_MSC_VER) && _MSC_VER >= 1300
#define Py_COMPILER_DIAGNOSTIC_PUSH __pragma(warning(push))
#define Py_COMPILER_DIAGNOSTIC_IGNORE_DEPRECATED_DECLARATIONS __pragma(warning(disable: 4996))
#define Py_COMPILER_DIAGNOSTIC_POP __pragma(warning(pop))
#else
#define Py_COMPILER_DIAGNOSTIC_PUSH
#define Py_COMPILER_DIAGNOSTIC_IGNORE_DEPRECATED_DECLARATIONS
#define Py_COMPILER_DIAGNOSTIC_POP
#endif

New changeset 75fe67538905 by Serhiy Storchaka in branch '3.5':
Issue bpo-19569: Suggested more appropriate replacements for deprecated Unicode
https://hg.python.org/cpython/rev/75fe67538905

New changeset f26d3f9a958a by Serhiy Storchaka in branch '3.6':
Issue bpo-19569: Suggested more appropriate replacements for deprecated Unicode
https://hg.python.org/cpython/rev/f26d3f9a958a

New changeset 2ae992bc2def by Serhiy Storchaka in branch 'default':
Issue bpo-19569: Suggested more appropriate replacements for deprecated Unicode
https://hg.python.org/cpython/rev/2ae992bc2def

New changeset f692dafe6797 by Serhiy Storchaka in branch 'default':
Issue bpo-19569: Compiler warnings are now emitted if use most of deprecated
https://hg.python.org/cpython/rev/f692dafe6797

Arfrever, could you please provide your proposition as a patch?

I do not have the possibility to test on Windows. Maybe someone wants to test Arfrever's suggestion.

bpo-33407 added MSVC support for Py_DEPRECATED().

New changeset de4304d by Zackery Spytz in branch 'master':
bpo-19569: Add a macro to suppress deprecation warnings (GH-9004)
de4304d

New changeset 2c6d6c1 by Miss Islington (bot) in branch '3.9':
bpo-19569: Add a macro to suppress deprecation warnings (GH-9004)
2c6d6c1