classification
Title: Extract file-finding and language-handling code from gettext.find
Type: enhancement Stage:
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ajaksu2, chrysn, eric.araujo, gregory.p.smith, jjinux, loewis, tarek
Priority: normal Keywords: patch

Created on 2007-02-01 02:20 by jjinux, last changed 2017-09-27 21:43 by eric.araujo.

Files
File name Uploaded Description Edit
gettext.patch jjinux, 2007-02-08 00:49 Patch showing a possible way to refactor the code
Messages (22)
msg54994 - (view) Author: Shannon -jj Behrens (jjinux) Date: 2007-02-01 02:20
If you distribute your code in a zipped egg, you can't use translation catalogs stored in that egg.  That's because gettext.py only knows how to find the translation catalogs in the actual filesystem.

This wouldn't be so bad if the "find" function didn't serve double duty.  On the one hand, it implements the "find" algorithm to pick the right languages and fallbacks.  On the other hand, it actually resolves the files in the filesystem.  It would be nice if these were separate.  It seems like people in projects like Pylons are stuck copying code from the find function in order to work around this problem.

Perhaps gettext can be updated to know about eggs.  Perhaps setting localedir to something like "egg://..." would enable this functionality.
msg54995 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2007-02-06 19:43
I fail to see the bug. gettext.find behaves as specified; if you want something else, don't use that function.

If you want to load a .mo file from a zip file, you should be able to create a GNUTranslation object directly, from a file-like object.

I don't think that gettext should support eggs directly. If you want it to do things other than loading from file systems, you should generalize that appropriately. One appropriate generalization could be the introduction of a directory-like object, where you can do .exists(relpath), and .open(relpath). However, introduction of directory-like objets is PEP material.

Reclassifying this as a feature request.
msg54996 - (view) Author: Shannon -jj Behrens (jjinux) Date: 2007-02-06 22:28
Sorry, yes, this is a feature request.  The most important thing that I'm requesting is to refactor the code.  That "find" function implements a certain algorithm that has nothing to do with the filesystem.
msg54997 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2007-02-07 06:45
Do you have a proposal on how to do the refactoring? It would be fine to split find into two parts; it would not be acceptable (to me) to teach it egg: URLs.
msg54998 - (view) Author: Shannon -jj Behrens (jjinux) Date: 2007-02-08 00:49
File Added: gettext.patch
msg54999 - (view) Author: Shannon -jj Behrens (jjinux) Date: 2007-02-08 00:52
Ok, I've uploaded a patch with a possible way to refactor the code.  I'm okay if you mark this bug as WONTFIX.  I'm raising it as a concern because I know there are a lot of people out there using Web frameworks such as Pylons, Turbo Gears, etc. that are going to have to duplicate the code in "find" in order to use eggs.  I hate to have see them do that. :-/
msg55000 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2007-02-08 08:27
Ok, I see how this fixes find(). What about translation() (which calls find, then also performs open())? Wouldn't this need to be fixed as well?

Also, what about os.path.join? It might depend on the structure of the storage, as well (suppose you have the .mo data stored in a relational database)?

In principle, I'm willing to fix this. I just want to avoid adding a hack.

For example, it might be sensible to come up with a MOOpener interface, that has an .exists method, taking a list of strings, a .join method, taking a list of strings and returning something that can be passed to the .open method, which would take something that .join returned and returning a file-like object. It might be possible to extend the meaning of the localedir parameter to be either a string (understood as a directory name), or a MOOpener object.

In any case, people *will* have to duplicate find for the moment, as any changes we discuss will only appear in 2.6.
msg55001 - (view) Author: Shannon -jj Behrens (jjinux) Date: 2007-02-08 19:09
> What about translation() (which calls find, then also performs open())? Wouldn't this need to be fixed as well?

Perhaps, but it's probably not as critical.  I.e. if the code gets duplicated to deal with eggs, it's not as big a loss.  I really wish the distutils guys would weigh in on this one.

> Also, what about os.path.join?

I think it's relatively harmless.  One can work around that.  One can't work around os.path.exists.

> In principle, I'm willing to fix this. I just want to avoid adding a hack.

I agree.  The patch isn't smelling too pretty.  Neither of us wants to commit to a virtual filesystem layer which is how I've handled problems like this in the past.  However, doing nothing results in a ton of code being duplicated in any project that uses gettext and zipped eggs.  Should we see if Phillip Eby has any ideas since he's Mr. Eggs?

> For example, it might be sensible to come up with a MOOpener interface,
that has an .exists method, taking a list of strings, a .join method,
taking a list of strings and returning something that can be passed to the
.open method, which would take something that .join returned and returning
a file-like object. It might be possible to extend the meaning of the
localedir parameter to be either a string (understood as a directory name),
or a MOOpener object.

That's a light virtual filesystem layer ;)  If you want that, I have code for that.

> In any case, people *will* have to duplicate find for the moment, as any changes we discuss will only appear in 2.6.

True.  Alas, this may be a moot point.

Thanks for your time.
msg55002 - (view) Author: Shannon -jj Behrens (jjinux) Date: 2007-02-08 19:11
Perhaps instead of committing to a special virtual filesystem layer, perhaps we should just take something that matches some subset of the posix interface.  We can document the functions we need.
msg84641 - (view) Author: Daniel Diniz (ajaksu2) (Python triager) Date: 2009-03-30 20:34
Not really a distutils issue, but fits the current distribution
discussion scope.
msg119392 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-10-22 15:48
The specific problem of gettext+eggs seems to be solved by this project: https://pypi.python.org/pypi/EggTranslations/

Since the egg format is not retained in distutils2, I don’t think any support for it should be added in the stdlib.

FWIW, I like the use of file-like objects to build Translation instances.  It’s easier to specify than a VFS and very versatile.
msg120531 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-11-05 20:45
Barry said in http://mail.python.org/pipermail/python-dev/2009-March/087847.html :

  The class-based API for gettext takes streams, so resource_stream()
  would work just fine.  I think i18n plugins for Python do not
  necessarily need to use the classic gettext API.

The title as I changed it some time ago is thus invalid, so I’m changing it again.  This is still a request for separating the language selection code from the file finding.  Anyone feel free to enter a better title, this one is not very clear.

My first vote was to try to fix the code without writing a VFS, but OTOH maybe a lightweight ABC with mixin methods and a concrete implementation of the full gettext logic may be clear and educational here.  Shannon, if you’re still getting those emails, what do you think?
msg122390 - (view) Author: Shannon -jj Behrens (jjinux) Date: 2010-11-25 17:54
> OTOH maybe a lightweight ABC with mixin methods and a concrete implementation of the full gettext logic may be clear and educational here.  Shannon, if you’re still getting those emails, what do you think?

Yep, that'd be fine.
msg122393 - (view) Author: Shannon -jj Behrens (jjinux) Date: 2010-11-25 17:55
Although, perhaps this bug is going away.  It seems like using zip files for eggs is going out of vogue.
msg122394 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-11-25 17:59
Would you like to propose a patch?
msg122395 - (view) Author: Shannon -jj Behrens (jjinux) Date: 2010-11-25 18:00
I've never managed to get a patch into Python, but I wouldn't mind trying ;)
msg122953 - (view) Author: Shannon -jj Behrens (jjinux) Date: 2010-11-30 23:11
Sorry, I just had a baby on Saturday.  Hence, I'm a bit late getting to this.  It might take me a couple weeks.
msg122964 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-12-01 00:54
Congratulations.  Take the time you’ll need, if this doesn’t go into 3.2 it’ll just be in 3.3.
msg128692 - (view) Author: Shannon -jj Behrens (jjinux) Date: 2011-02-16 20:04
The more I think about this, the more I think we should just drop it.

 * The easiest way around my original problem was to not install the eggs in zipped format.  That's easy.  Just mark the egg as not zip safe.
 * I don't currently need this functionality.
 * No one else is clamoring about it.

I think the increase in complexity just isn't warranted.  If someone is really stuck, they can just hack their own version of gettext.  It's not that big a deal, and it's fairly unlikely anyway.

Sorry for taking so long to respond; have 6 kids really makes keeping up with extracurricular activities difficult!
msg128693 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-02-16 20:34
Fine, let’s close this then.  Don’t hesitate to propose a patch for another bug report or feature request if you get enough free time someday :)
msg303184 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2017-09-27 21:32
FYI - We run into this problem getting things using gettext for i18n within a hermetic zipped up Python application at work.  I'm not reopening it as I have nothing new to propose right now, but figured I'd drop a note that it does seem like a good idea.

Assuming that everything lives on a local filesystem is not a given.
msg303185 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2017-09-27 21:43
Reopening.  Eggs are still not officially blessed, but zipapps are.
History
Date User Action Args
2017-09-27 21:43:24eric.araujosetstatus: closed -> open
versions: + Python 3.7, - Python 3.2
messages: + msg303185

resolution: rejected ->
stage: resolved ->
2017-09-27 21:32:59gregory.p.smithsetnosy: + gregory.p.smith
messages: + msg303184
2011-02-16 20:34:05eric.araujosetstatus: open -> closed
nosy: loewis, ajaksu2, jjinux, tarek, chrysn, eric.araujo
messages: + msg128693

resolution: rejected
stage: patch review -> resolved
2011-02-16 20:04:15jjinuxsetnosy: loewis, ajaksu2, jjinux, tarek, chrysn, eric.araujo
messages: + msg128692
2010-12-01 00:54:51eric.araujosetmessages: + msg122964
2010-11-30 23:11:06jjinuxsetmessages: + msg122953
2010-11-25 18:00:48jjinuxsetmessages: + msg122395
2010-11-25 17:59:17eric.araujosetmessages: + msg122394
2010-11-25 17:55:27jjinuxsetmessages: + msg122393
2010-11-25 17:54:36jjinuxsetmessages: + msg122390
2010-11-05 20:45:09eric.araujosetassignee: tarek ->
messages: + msg120531
title: Support for non-filesystem-based catalogs in gettext -> Extract file-finding and language-handling code from gettext.find
2010-10-22 15:48:06eric.araujosetversions: + Python 3.2, - Python 3.1, Python 2.7
nosy: loewis, ajaksu2, jjinux, tarek, chrysn, eric.araujo
title: gettext.py incompatible with eggs -> Support for non-filesystem-based catalogs in gettext
messages: + msg119392

components: - Distutils
stage: test needed -> patch review
2010-04-09 01:37:49eric.araujosetnosy: + eric.araujo
2009-03-30 20:34:18ajaksu2setassignee: tarek
components: + Distutils
versions: + Python 3.1, Python 2.7
keywords: + patch
nosy: + tarek, ajaksu2

messages: + msg84641
stage: test needed
2008-03-17 19:19:15chrysnsetnosy: + chrysn
2007-02-01 02:20:07jjinuxcreate