Issue64

If you're reporting an issue for setuptools 0.7 or higher, please use BitBucket

Title Subversion 1.6 entries format 'unrecognized'
Priority bug Status resolved
Superseder Nosy List asmodai, cboos, durin42, dw, grant, jaraco, mjpieters, pje, rjollos, srid, zgoda
Assigned To Keywords

Created on 2009-03-27.14:38:52 by mjpieters, last changed 2010-01-04.12:00:37 by rjollos.

Files
File name Uploaded Type Edit Remove
svn_1.6_format.patch jaraco, 2009-04-13.13:07:54 text/x-patch
svn_versioning_2.patch jaraco, 2009-04-24.23:19:42 text/x-patch
svn_versioning_2.patch jaraco, 2009-04-30.20:15:20 text/plain
svn_versioning_3.patch grant, 2009-05-27.16:08:35 text/x-patch
svn_versioning_4.patch grant, 2009-05-29.18:58:46 text/x-patch
Messages
msg423 (view) Author: pje Date: 2009-10-19.19:32:28
setuptools 0.6c10 is released with a fix for this issue.  (In setuptools 0.7,
we'll be moving SVN support to a plugin that can be updated separately.)
msg321 (view) Author: jaraco Date: 2009-07-05.23:46:42
I've created issue79 (http://bugs.python.org/setuptools/issue79) to track the
improvement of setuptools to support command-based entries parsing.
msg319 (view) Author: jaraco Date: 2009-07-05.23:29:51
@dw You make some good points, and I don't disagree that shelling out to an SVN
client would be a valuable approach, and probably even the preferred approach.

Nevertheless, the existing implementation, while having dependency on a specific
file format, does not have any dependency on a subversion client.  I'll describe
some scenarios that come to my mind in which a person may have a subversion
checkout of a project without having the defacto subversion client available.

1) Using Subclipse under Eclipse.  Subclipse may be using JavaHL or SVNKit as
the underlying library for working with the SVN checkout.  JavaHL links to the
reference implementation whereas SVNKit re-implements the same code in Java.  In
either case, there is no svn command that can be run (although it may be
possible to initiate a JavaVM with the proper java class to parse out the SVN
entries info).
2) Windows users typically use TortoiseSVN.  On 64-bit Windows, the reference
implementation of SVN isn't even available, although there is SlikSVN which
makes available a 64-bit command-line client for Windows.  Nevertheless, Windows
users prefer TortoiseSVN, which like JavaHL SVN links to the reference
implementation, but does not provide an svn command.  As far as I can tell,
TortoiseProc.exe doesn't expose any command to retrieve the info for a checkout.
3) In some deployment scenarios, deployment may be performed on a different
machine than that on which development is performed.  For example, I have a
Windows XP box that mounts a share on my development machine for the purpose of
deploying binaries for older versions of Python.  This deployment machine has
Python and setuptools, but has no need for subversion, as no subversion
operations are ever done on that machine.

I point out these scenarios to illustrate only some of the cases that affect my
use of setuptools.  I'm all but certain there are other scenarios where having a
subversion client is not a defacto assumption.

durin42 points out that we might be able to use bindings, but I would argue that
adding compile-time dependencies on setuptools is also a barrier for some
platforms.  Because setuptools is such a pervasive dependency, it would be
unwise to add a compile-time requirement that delays or prohibits the deployment
to the various platforms.  For example, pysvn for Windows 64-bit is not readily
available.  I would hate for setuptools to find itself in the same boat because
of the challenges with compiling it.  If Tigris.org were to release pure-python
bindings that could be borrowed wholesale by setuptools, that would great.

You mention that this approach cost you an hour of your billable time last week,
but you would not have had to spend this time if the patch had been applied and
released sooner.  The patch takes an incremental step toward improving the
approach and providing a more generalizable architecture for determining
entries, including possibly using a shell out to an existing SVN client.  A lot
more than an hour of non-billable work has gone into making setuptools more
compatible while maintaining backward-compatibility and portability.  I think
it's unfair to dismiss this effort simply because it isn't the implementation
that works best for your particular configuration.

That said, your suggestions are very good.  This effort is one of my few
contributions to setuptools.  I have very little visibility into the effort and
am only trying to contribute where I can without making too many waves.

I still contend that to the extent possible, setuptools must support native,
pure-python access to svn info to support the nature of its broad deployment.

If someone would commit the patch as we have it, I'd be happy to move forward
with an additional effort to implement a command-based version... and to query
the community on whether this should be the preferred or fallback approach.
msg318 (view) Author: jaraco Date: 2009-07-05.22:51:29
@durin42 Thanks for the insight.  Certainly this indicates that a more robust
approach is desirable.
msg317 (view) Author: durin42 Date: 2009-07-05.18:02:22
I pay attention to Subversion development semi-closely. You should definitely 
not depend on the existing entries format, as they're planning a complete 
overhaul of the .svn structure for 1.7, I think even to the point of tracking 
entries using sqlite instead of a binary file format.

Shelling out to svn (or using bindings) is probably the Right answer. Just my 
2ยข.
msg316 (view) Author: dw Date: 2009-07-05.16:10:45
@jaraco This approach cost me an hour of billable time last week, and your
suggestion is likely to do the same in future. The .svn layout is internal to
Subversion: it should simply be assumed that without the official client, its
contents cannot be accessed (an approach taken by most sane software).

It is then suggested that depending on the Subversion client would be
"non-trivial in general and creates a hard dependency on a specific
implementation of svn." -- what exactly is this (broken) patch doing, if not
requiring a specific implementation of Subversion? Except instead of depending
on an interface whose interface is contracted never to change, depending on an
internal interface, external software shouldn't even be aware of.

I do not understand the lack of desire to depend on an external binary. Surely
any environment where a package is being built for some program just exported
from source control, the relevant binaries are almost certainly present on the
system (what other sane means could a directory with ".svn" subdirs have ended
up on the machine)

David
msg315 (view) Author: jaraco Date: 2009-07-05.15:48:12
dw: The patch supplied doesn't simply patch the existing functionality.  It
addresses the underlying approach to parsing the svn entries file, making it
more robust and resilient to changes.  It also doesn't explicitly require a
specific version number in the entries file.  It still checks the version, and
logs a warning if it is unaware of the version, but it will continue to function
even if the version is unknown (assuming any format changes are still compatible).

And to re-iterate my message from 2009-04-14, it does not make since to simply
shell out to svn.  While this approach is simple and would be straightforward in
some environments, it is non-trivial in general and creates a hard dependency on
a specific implementation of svn.  Some environments may be using another SVN
client, or may not have an SVN client at all (consider a folder mounted from
another system).  Furthermore, if the svn command is not in the user's path,
this further complicates the use of shelling out to svn (where to find the
binary, which binary to use, etc).

So for optimum compatibility and minimum dependencies, setuptools should
continue to have a pure-python means to determine the SVN info.

Furthermore, the patch as supplied makes it much easier to implement new
approaches for handling SVN entries.  So if one was to implement an approach
that shells out to svn, that approach could be implemented using the constructs
supplied by this patch.

I would consider adding 'svn info' support, but it would be wise to first commit
this tested implementation to address the immediate issue, and then we can
consider a subsequent patch per your suggestion.
msg314 (view) Author: dw Date: 2009-07-04.15:59:38
Rather than patching in support for the new working copy format, why not just
look for the 'svn' binary and call 'svn info --xml'?

This is exactly the kind of use case that command was designed for; it's save
untold man-hours of misery across the globe, at the small cost of a slightly
slower build.

I'm completely against the patches that simply update the existing (broken)
functionality.

Thoughts?
msg294 (view) Author: grant Date: 2009-05-29.18:58:46
Actually, I found another bug in svn_versioning_2.patch: If you do a "setup.py
sdist" on setuptools itself, setuptools/cli.exe and setuptools/gui.exe weren't
being copied.

This looks like a logic error in SVNEntriesText.get_undeleted_records(): it no
longer accepts directory records (which have a length of 2), which means that
sdist no longer recurses into subdirectory. I've attached svn_versioning_4.patch
as a fix to both the problems I ran into.
msg293 (view) Author: grant Date: 2009-05-27.16:08:35
I applied the most recent svn_versioning_2.patch to the dev06 setuptools branch, 
but ran into a recursion executing

easy_install 'svn://svn.eby-sarna.com/svnroot/AddOns#egg=AddOns-dev'

with subversion-1.6.2 installed.

The problem seemed to be that the checked out AddOns had an svn:externals for 
ez_setup, and listing that would recurse because the first entry returned by 
SVNEntriesText.get_undeleted_records() had a path of ''. So, I'm attaching a 
svn_versioning_3.patch that excludes an empty path. I'm really not sure if this 
is generally correct, but it does work for me.
msg291 (view) Author: jaraco Date: 2009-05-22.18:51:11
So can this be applied to the 0.6 branch of setuptools?  It works great for me
in several environments.
msg286 (view) Author: jaraco Date: 2009-04-30.20:15:20
Re-uploading svn_versioning_2.patch, this time as UTF-8 instead of UTF-16.
msg285 (view) Author: asmodai Date: 2009-04-30.16:14:12
The second patch is binary garbage for me, whereas the first works.
msg283 (view) Author: mjpieters Date: 2009-04-25.07:26:02
The patch looks great; this will also allow easy future support for using a svn
command-line client to get the info or implement a 1.7-specific format parser
(new class, use the class in SVNEntries.read if so detected/configured, bob's
your uncle).
msg282 (view) Author: jaraco Date: 2009-04-24.23:19:42
mjpieters: That's a good point.  I didn't realize I'd missed sdist.

I went ahead and implemented your suggestion, attached as this new patch.

This new patch addresses the issue with egg_info and sdist, but also refactors
the svn entries handling in a separate module (setuptools.svn_util).

This is still a patch against the setuptools-0.6 branch.
msg281 (view) Author: mjpieters Date: 2009-04-24.09:45:57
jaraco's patch only fixes the problem in one location, namely egg_info.py.
Unfortunately, the problem also exists sdist.py. There should probably be a svn
utility module instead, that provides svn .entries parsing in one central place.
That would make it easier to add optional external client support as well.
msg278 (view) Author: jaraco Date: 2009-04-14.16:48:10
While it may be reasonable to first attempt to shell out to svn or SubWCRev,
it's not guaranteed that the user will have either of these clients (He could be
using Eclipse, Subclipse, and PyDev, for example).  Because of the core nature
of setuptools, it should have as few dependencies as possible, so it should
fallback to inspection using pure Python.
msg277 (view) Author: cboos Date: 2009-04-14.12:28:11
Please also consider using the svn command line for this (and/or the 
TortoiseSvn's SubWCRev utility, on Windows), as this would guarantee forward 
compatibility.

Trac users have been hitting this problem pretty consistently for the 1.4.x to 
1.5.x upgrade and now for the 1.5.x to 1.6.x upgrade. Let's avoid this for 1.7.x 
and upward.

See also issue4 and issue44 and http://trac.edgewall.org/ticket/8151.
msg276 (view) Author: jaraco Date: 2009-04-13.13:11:33
If there's interest, I'll port the aforementioned patch to the trunk as well.
msg275 (view) Author: jaraco Date: 2009-04-13.13:07:54
I'm enclosing a patch against r71559 of /branches/setuptools-0.6 that refactors
the svn revision parsing code to its own method and handles the svn 1.6.x line.
 It also will handle the 1.7+ versions with a warning, assuming the format is
consistent enough.
msg256 (view) Author: durin42 Date: 2009-03-30.23:25:29
You could make that change, except that svn 1.7 is expected to include a new .svn 
setup to remove the .svn-dirs-everywhere bug. I'd not count on the entries being 
in a similar format (I recall some talk of sqlite, although not sure if that's for 
entries or not).
msg253 (view) Author: mjpieters Date: 2009-03-27.14:38:51
After upgrading to subversion 1.6, the dreaded unrecognized .svn/entries format
pops up again. The entries format version number is 10, and if I add that to
sdist.entries_finder, things appear to work just fine.

Perhaps the test can be updated to parse the integer and use a >= 9 test on it,
and just assume that future formats will continue to work until proven otherwise?
History
Date User Action Args
2010-01-04 12:00:37rjollossetnosy: + rjollos
2009-10-19 19:32:33pjesetstatus: in-progress -> resolved
2009-10-19 19:32:28pjesetnosy: + pje
messages: + msg423
2009-10-10 21:23:34pjesetstatus: chatting -> in-progress
2009-08-31 22:18:50sridsetnosy: + srid
2009-07-15 08:50:40zgodasetnosy: + zgoda
2009-07-05 23:46:42jaracosetmessages: + msg321
2009-07-05 23:29:52jaracosetmessages: + msg319
2009-07-05 22:51:29jaracosetmessages: + msg318
2009-07-05 18:02:23durin42setmessages: + msg317
2009-07-05 16:10:46dwsetmessages: + msg316
2009-07-05 15:48:13jaracosetmessages: + msg315
2009-07-04 15:59:38dwsetnosy: + dw
messages: + msg314
2009-05-29 18:58:46grantsetfiles: + svn_versioning_4.patch
messages: + msg294
2009-05-27 16:08:35grantsetfiles: + svn_versioning_3.patch
nosy: + grant
messages: + msg293
2009-05-22 18:51:11jaracosetmessages: + msg291
2009-04-30 20:15:20jaracosetfiles: + svn_versioning_2.patch
messages: + msg286
2009-04-30 16:14:12asmodaisetnosy: + asmodai
messages: + msg285
2009-04-25 07:26:03mjpieterssetmessages: + msg283
2009-04-24 23:19:43jaracosetfiles: + svn_versioning_2.patch
messages: + msg282
2009-04-24 09:45:58mjpieterssetmessages: + msg281
2009-04-14 16:48:10jaracosetmessages: + msg278
2009-04-14 12:28:11cboossetnosy: + cboos
messages: + msg277
2009-04-13 13:11:33jaracosetmessages: + msg276
2009-04-13 13:07:54jaracosetfiles: + svn_1.6_format.patch
messages: + msg275
2009-04-13 12:12:51jaracosetnosy: + jaraco
2009-03-30 23:25:30durin42setstatus: unread -> chatting
nosy: + durin42
messages: + msg256
2009-03-27 14:38:52mjpieterscreate