This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: codec registry and Python embedding problem
Type: Stage:
Components: Interpreter Core Versions:
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: niemeyer Nosy List: gvanrossum, lemburg, loewis, niemeyer
Priority: normal Keywords:

Created on 2003-01-06 12:17 by lemburg, last changed 2022-04-10 16:06 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
python-2.3-codecs.patch niemeyer, 2003-03-01 08:07
Messages (11)
msg13908 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2003-01-06 12:17
Found by Hartmut Ring:

#include <Python.h>
int main() {
  for (int i=0; i<2; i++) {
    Py_Initialize();
    PyRun_SimpleString("a = u'\xe4'.encode('Latin-1')");
    Py_Finalize();
  }
  return 0;
}

First try runs fine, second try gives:

Traceback (most recent call last):
  File "<string>", line 1, in ?
LookupError: no codec search functions registered:
can't find encoding

msg13909 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2003-01-06 12:18
Logged In: YES 
user_id=38388

This is due to a flag used in codecs.c:

/* Flag used for lazy import of the standard encodings
package */
static int import_encodings_called = 0;

The solution is to reset this flag in Py_Finalize().
msg13910 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2003-01-22 10:07
Logged In: YES 
user_id=21627

I think the various global variables maintained in codecs.c
(_PyCodec_SearchCache, import_encodings_called, ...) need to
become part of the interpreter state. Otherwise, a codec
registered in one interpreter will be found in another.

Also, it appears that import_encodings_called can be
eliminated. Instead, a NULL value of _PyCodec_SearchPath
could be used to indicate whether initialization has taken
place. 
msg13911 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2003-01-22 12:12
Logged In: YES 
user_id=38388

Good suggestion. Do you have time to cook up a patch ?
msg13912 - (view) Author: Gustavo Niemeyer (niemeyer) * (Python committer) Date: 2003-02-12 13:13
Logged In: YES 
user_id=7887

As discussed in python-dev, I'll work on that.
msg13913 - (view) Author: Gustavo Niemeyer (niemeyer) * (Python committer) Date: 2003-03-01 08:07
Logged In: YES 
user_id=7887

Here is a proposed solution.

It has been tested with the standard regression tests, and
in an environment with multiple interpreters.
msg13914 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2003-03-01 14:41
Logged In: YES 
user_id=6380

I can't review this beyond "it doesn't break the tests for
me"; I suggest to assign to MAL for review. One note: I got
a compiler warning

  implicit declaration of function `_PyCodecRegistry_Init'

to fix, add

  static int _PyCodecRegistry_Init(void); /* Forward */

somewhere to the top of codecs.c.
msg13915 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2003-03-04 10:07
Logged In: YES 
user_id=38388

Same comments as Guido, plus:

* in the interpreter state struct you should add the new 
  fields to the end, not in the middle

* the finalization phase is moved to a later point in
  the process; this could cause problems when finalizing
  codecs -- I guess we'll just have to see whether we
  get bug reports related to this; perhaps it's better
  to ZAP the dicts before ZAPping builtins etc. ?!

Great work, Gustavo. Please check the patch in with 
the above changes.

Thanks.


msg13916 - (view) Author: Gustavo Niemeyer (niemeyer) * (Python committer) Date: 2003-03-07 17:20
Logged In: YES 
user_id=7887

No problems.

If you don't mind, I'd just to have a confirmation: are you
sure I should move the new fields to the end, and leave a
conditional member in the middle?
msg13917 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2003-03-07 17:36
Logged In: YES 
user_id=38388

Oh, I see what you mean. In that case, it's OK to add 
them just before the optional parts.
msg13918 - (view) Author: Gustavo Niemeyer (niemeyer) * (Python committer) Date: 2003-03-19 00:50
Logged In: YES 
user_id=7887

Commited as:

Include/pystate.h: 2.23
Misc/NEWS: 1.699
Python/codecs.c: 2.21
Python/pystate.c: 2.24
Python/pythonrun.c: 2.182

Thanks for your support!
History
Date User Action Args
2022-04-10 16:06:06adminsetgithub: 37727
2003-01-06 12:17:28lemburgcreate