Here's a first crack at a bdist_deb.

This patch implements two new distutils commands:

bdist_deb:
  Build Debian packages
  
debianize:
  Create and populate a top-level debian subdirectory.
  (Essentially dh_make for distutils packages.)

There is a slightly detailed README.bdist_deb included
in the patches.

I'm open to suggestions for improvements and bug-fixes.

Logged In: YES 
user_id=81797

I'm just doing a review of this code.  A couple of things:

   There's been some concern expressed about
get_default_maintainer.
   Namely, that if debchange changes it's algorithm, it won't be
   reflected in this code.  It seems like one possible way
around that
   would be to build a directory with a "debian" directory
under it, a
   fake "changelog", and then call debchange to write the
data out, and
   parse it.  Too bad there's not a direct hook into
debchange to get
   that information.

   Can _formatdate, if email doesn't exist, use
rfc822.formatdate()?

   Ditto for _parseaddr?

It looks pretty good.  However, when trying to build a .deb
of my
jotweb2 package, it's failing with:

   [...]
   dh_testdir
   dh_testroot
   dh_installchangelogs-
   dh_installdocs
   cp: cannot stat `doc': No such file or directory
   dh_installdocs: command returned error code 256
   [...]

I'm not sure exactly why.  I do have a "doc" directory in my
main
package directory, but I don't reference to it in my setup.py or
MANIFEST.  Adding it to the MANIFEST doesn't seem to help this.

Sean

Logged In: YES 
user_id=170840

I've a woody with some packages backported from sarge
(debhelper 4.1.90 and dpkg-dev 1.10).

Running python2.3 ./setup.py bdist_deb with some of
my projects, I got the error:

dpkg-buildpackage: unknown option or argument
--check-dirname-level=1
Debian dpkg-buildpackage .

Commenting out the "check-dirname-level" and
"check-dirname-regex" options in the bdist_deb.py file
the script can go on, but it exits with the error:

debian/rules:11: *** first argument to `word' function must
be greater than 0.  Stop.

Hope this helps.

Logged In: YES 
user_id=45814

Thanks for the comments.  Here's a second attempt.

Changes include:

It might work with woody.  (I'd appreciate it if you could
try again, Davide)

'debianize' command renamed to 'dh_make'.

Use debchange to create debian/changelog.  This eliminates
the need to duplicate debchange's logic to deduce the
packagers name and e-mail.

A more complete test script.


Patches are on today's CVS.

Logged In: YES 
user_id=170840

I've tried the 29/10 patch with my home system (dpkg-dev 1.10,
debhelper 4.1.90 and dh-make 0.30) and another "pure woody"
system (dpkg-dev 1.9.21, debhelper 4.0.2 and dh-make 0.30).

I've used your test suite and some of my personal projects
(using both bdist_deb and dh_make); as far as I can tell, it
works very well. :-)

bdist_deb always worked as expected, producing good debian
packages.

I've used dh_make and then I've run "dpkg-buildpackage
-rfakeroot" and I got some errors (but I'm not sure they
depends on your code).  Also your test suite produced some
failures.

Here you can find the output of "dpkg-buildpackage -rfakeroot"
and of your tests suite (both were running on my home system):
  http://erlug.linux.it/~da/tmp/dpkg-buildpackage
  http://erlug.linux.it/~da/tmp/testsuite


Thank you for your great effort!

Logged In: YES 
user_id=45814

Here's try number three!

Changes are mostly fixes for woody.  The README.bdist_deb
(included in the patch set) enumerates the changes in more
detail.

(I've bitten the bullet and installed woody on one of my
machines, so I'm fairly confident it should work now.)

The patches, on todays CVS are, as always, attached below.

====
Note to Davide,

Thank you, again, very much for all the testing.

A couple of the lintian warnings
('prerm-does-not-remove-usr-doc-link' and
'postinst-does-not-set-usr-doc-link') that are listed in
your test suite output are due, I think, to your mixed
system.  Specifically, I suspect you've got the woody
version of lintian (which wants to see symlinks from
/usr/doc/<package> to /usr/share/doc/<package>), but a later
version of debhelper.
(Woody's dh_installdocs (in the debhelper package)
automatically generates those links, later versions don't;
/usr/doc has been phased out.)

Anyhow, I'm not going to fix those warnings (unless it turns
out that you're seeing them on your pure woody system.)


On package with python scripts (in /usr/bin) Woody's lintian
generates an 'unusual-interpreter' warning which I think is
spurious (i.e. it is not a valid warning.)  (Lintian 1.20.17
doesn't recognize things like /usr/bin/python2.1 as a valid
interpreter --- it seems valid enough to me.)

Best Regards,
Jeff

Logged In: YES 
user_id=45814

Another updated patch set:

As suggested by Bastian Kleineidam, when the source
distribution includes it's own debian subdirectory,
bdist_deb (rather than just crapping out) will now skip the
dh_make stage, and just run debuild.

As always, patches on the current CVS version of distutils
are attached below.

Logged In: YES 
user_id=170840

I've created a patch to make your bdist_deb/dh_make commands
compatible with Python 1.5.2; it can be found here:
 
http://erlug.linux.it/~da/tmp/distutils-cvs_debpy152-20041109.patch

Beware that I've tested it for a very short time,
so double check everything (especially the permissions
on file/umasks).

Moreover, the patch changes also some other files
like build_py.py, install_lib.py, bdist_rpm.py, that
were not 1.5.2 compatible - these changes are part of
another patch I've posted today.

HTH.

Logged In: YES 
user_id=45814

Another updated patch set.

The only change is a bug fix: when using a hand-build set of
debian/* files, dh_make will now deduce the correct Debian
revision and build architecture.  (Bug reported by Bastian
Kleineidam.)

Patches, as always are on today's CVS version of distutils.

Logged In: YES 
user_id=872713

I was looking a long time for something like this. I really
hope seeing this in Python 2.5, although I'm not familiar
with Python's development process.

Logged In: YES 
user_id=170840

Sometimes you need to install the same package for many different releases of Python;
unfortunately this will fail, because dpkg refuses to override the dummy man page, present
in both packages.
Would it be a good idea to modify line 453 of dh_make to include pyver in the
name of the manpage file?

Logged In: YES 
user_id=149084

In response to jafo's request for comments on python-dev:

1. This facility should be documented in the Python Module docs.
A section should be introduced in "Distributing Python Modules"
after the 5.2 Creating RPM Packages.  Also, the reference
disutils.command.bdist_deb should be supplied.
The fact that the rest of distutils is 'lightly' documented is no 
excuse :-)

2. The README.bdist_deb should be folded into the doc strings
for the two modules (except the change info should be left behind
in the patch).

3. The existing doc strings in the two modules are not always 
in standard form, and could be more extensive.

4. There are triple quoted block comments in the code.  These 
should be # comment blocks, cf. PEP8 and PEP 257.  You 
don't use doc strings to document variables.

5. deb_util.py probably belongs in distutils/command/

6. I would like it better if the default was to build the source
package, and that would be consistent with bdist_rpm's action.

7. I'm not entirely happy with the overall design: it seems there
are two ways to go.  First, run bdist_deb, which give you a 
package which isn't debian compliant; there's no 
debian/changelog.  Second, you can run dh_make, which will
'debianize' the Python source.  This can only be done once, and
then the recommendation is to use the standard Debian tools
to create the packages.  In this second case, apparently you
can't use bdist_deb, nor do you have a convenient way of
updating when the Python code changes, if I understand the
README.bdist_deb correctly.  If these impressions are correct,
it's a lot of code for relatively little effect, at least at its current
level of development.

8. What about alberanid's last comment?

8. Do we have a commitment from Dairiki to maintain this code
for at least a couple of years?

Logged In: YES 
user_id=21627

I agree with most of kbk's comments, except for 5: all of
the *_util modules are in distutils, why not deb_util? It is
not a command.

Also, dairiki, are you willing to sign the contributor form? see

http://www.python.org/psf/contrib.html

You may also be interested in stdeb, which produces Debian source packages from Python packages. See http://stdeb.python-hosting.com/

This code has a number of outstanding issues including (primarily) whether a contributors agreement will be signed.  It would be nice to have this code in Distutils, but I think this is a show-stopper to getting it in.  If Geoffrey signs a contributor agreement, I'll open it back up and see if I can address the other outstanding issues.

Sean

jafo wrote:
> It would be nice to have this code in Distutils

Absolutely!  I think that a deb packages generator (so policy-compliant as it can be) will be useful to many people.

Unfortunately I'm not an expert about debian packages, but maybe I can do some testing and/or write some code.

Sean,

I am re-opening this issue, to study the inclusion of a bdist_deb
command for 2.7/3.1, since there's some interest in the community.

I have taken the assigment since it was rejected, but let me know if you
want me to assign back to you of course.

Count me in

I've been doing a review of this patch for the last few hours.  There
are a few issues that need to be taken care of in order for it to move
forward, and I'm currently working on them, in this order:

  1. Update the patch to the most recent svn.  The most current patch is
almost 5 years old.  I've ported it forward.
  2. Fix the unittests.  The patch is failing its unittests currently,
and so I'm fixing the unittests, and will be adding tests where there
may not be any coverage.
  3. Implement the suggestions made by alberanid and kbk, adding tests
where necessary.

  I've been chatting with jafo, and am willing to commit to maintaining
this code for the next few years, and am willing to sign the contributor
agreement in order to do so.  I don't particularly want to step on
anyone's toes, so if someone else is also working on this, I'm happy to
hand off what I've already been working on.

Thanks for your efforts. I don't think you are stepping on anyone's toes
when picking up an issue that was unsolved for almost 5 years :-)

Please post patches to this bug for review/comments/help/whatever

stdeb ( http://github.com/astraw/stdeb ) now includes a bdist_deb
distutils command.

I am closing this issue, since stdeb implements this command now.

Also, I'd love to add it in the distutils documentation, but there's one
issue.

Andrew, do you think you could have a plain distutils version of this
command ? 

Setuptools changes the Command and the Distribution classes, and affects
Distutils global behavior, so we can't document its usage in the
Distutils documentation as it is.

As of stdeb 0.5.0, the setuptools dependency has been removed. Now stdeb
requires only plain distutils.

astraw: I've been playing with stdeb.  I think it's a nice implementation of bdist_deb, but it doesn't seem to include the dh_make/debianize functionality of this patch.  Is that the case or have I missed something?

Hello

There has been a number of discussions about bdist_deb, and some code too. I don’t have links handy, unfortunately, but I remember a conclusion that was: Don’t.

 Debian packages are best made by Debian tools, which go to great lengths to comply with Debian Policy, work together, and are frequently updated. On the Distutils(2) site, the most we can do would be a bdist_dsc command that produces a special file which is used by debianizing tools.

 A key thing to remember is that .deb is not just a file format, it’s the embodiment of a Policy (the real value of Debian is its policy, not its package manager—a nice technical side effect), and as such, best handled by Debian tools.

Disclaimers: I’ve not been following packaging discussions for long, these lines are a summary of other people’s thoughts, I’m not a Debian developer or maintainer.

Cheers

There are really two aspects of this patch.  At the moment, I'm less interested in bdist_deb than I am in the 'debianize' (i.e. dh_make) functionality.  I think 'python setup.py debianize' (or whatever) would be a nice way to jump start a debian/ directory from the available metadata.

Why not use Debian’s dh_make?

That's essentially what I want, except I want to feed dh_make some better defaults based on metadata that already lives in setup.py.

Hello

Thanks for clarifying your point.

Note that I had quickly tested dh_make in a near-empty directory before
posting, and the only question I got asked was the type of the package
(library, many libraries, program, etc). But I’ve just realized that I
have two dh_make-related variables set in my environment, plus maybe
config files, so a typical user may have more questions to answer, which
is annoying if the setup.cfg¹ already has them. And to be honest, the
question about the type of the package is not simple if you’re not
fluent with the Policy, so if Distutils2 can answer that thanks to its
knowledge, all the better. I therefore agree with you :)

One thing I’m attached to: The name of the command should respect
existing conventions. Entirely new concepts get dedicated names (e.g.
“check” and “test”), but commands fitting in the areas of sdist or bdist
should say so in their name. How about sdist_debian?

It may be useful to precise something at this point: You may read about
both “debian” and “Debian” packages. “debian” or “.deb” or “debs” are
binary distributions that use the file formats defined by Debian,
whereas “Debian” packages are binary distributions that use these
formats, conform to the Debian policy, are tested on lintian.debian.org
and piuparts.debian.org, and most importantly, are scrutinized for
Debian Free Software Guidelines compliance. That’s why I, for one, never
uses third-party deb packages or apt repos: If it’s going to be
installed system-wide, I want a verified Debian package.

HTH

¹ Not a typo, just an anticipation of PEP 390 ;)

Kind regards

I'm not so sure about sdist_debian for the command I'm thinking about because it doesn't actually build a distribution.  It just creates a 'debian' directory so I think I like just 'debian' as the name of the command.  But thanks for the feedback; time to write some code!

Barry, stdeb does much of what you're describing. (Try the "python setup.py sdist_dsc" command.)

I'm not particularly pleased with the stdeb codebase as it stands, but it does work reasonably well, and I'd like to see progress in this domain, but I'm afraid I don't understand the difference between what it offers and what you're proposing. I guess maybe you're not aware of the "sdist_dsc" distutils command it offers?

-Andrew

Hi Andrew,

Yes, I've played with sdist_dsc, and it's nice in that it takes a non-Debianized source tree and turns it into a source package that can be uploaded.  What I'm looking for is something that can take a non-Debianized source tree and turn it into a native package, essentially by pre-filling a debian/ directory like dh_make does, but including additional metadata available in the setup.py.

I've started taking a crack at this:

https://launchpad.net/setupdebian

Barry, I'm sorry I still don't understand what you think is the essential distinction. A debian source package (.dsc, .orig.tar.gz, and .diff.gz files) is simply made from a (patched) source directory, including the debian/ directory, and an upstream tarball. To build this sdist_dsc must of course make the debian/ directory. Thus, it "takes a non-Debianized source tree and turn it into a native [Debian source] package, essentially by pre-filling a debian/ directory like dh_make does, but including additional metadata available in the setup.py". Then, once it done this, it calls "dpkg-buildpackage -S" to turn this filesystem structure into the actual source package. If your final goal is simply the filesystem structure rather than the Debian source package (.dsc, .orig.tar.gz and .diff.gz), then it seems like it would be easier to refactor stdeb to stop before calling dpkg-buildpackage rather than writing a whole new alternate way of generating the debian/ directory based on the contents of setup.py.

So I'm afraid I still don't understand what you want setupdebian to do that stdeb does not.

Hi Andrew,

I want debian/ to be created in . (aka, cwd, the directory containing setup.py), not in ./deb_dist/package-version

I see. So is copying the debian/ directory into its desired location a possibility for you, either manually or via a new distutils commands that shares 99% of its code with sdist_dsc? It doesn't seem like enough of a difference to start a new project.

Manual moving/copying is what I want to avoid.  I'd be totally happy with not reinventing the wheel if you think this would be easy to add to stdeb!  Specifically: just create debian/ in place and do not build package.

I have a few other minor suggestions, though if you think this isn't the best place to track them, please let me know where to submit them.

* Add setup.py entry_points so you don't have to use --command-packages
* Package: shouldn't change dots to dashes.  E.g. flufl.enum should be called python-flufl.enum
* Update to Standards-Version: 3.8.3

I'm happy to test whatever you come up with.

tree and turns it into a source package that can be uploaded.  What I'm
looking for is
something that can take a non-Debianized source tree and turn it into a
native package,
essentially by pre-filling a debian/ directory like dh_make does, but
including additional
metadata available in the setup.py.
> 
> I've started taking a crack at this:

Barry, what do you mean with “native package”? In Debian parlance, a
native package is a piece of software that isn’t released outside of
Debian and thus has no hyphen in the version number (a hyphen is
normally used to separate the upstream version number from a
debian-packaging-specific number, e.g. “0.2-1”). I don’t know stdeb (I
avoid Setuptools), so I don’t understand the difference you make.

On a related note, I’ve seen in your setupdebian code that you
unconditionally remove some files under the debian directory, but some
of them may be useful, especially for building multiple binary packages
(i.e. separate installable .deb packages) out of one source package,
e.g. to split architecture-independent modules from arch-dependent C
extensions (like mercurial and mercurial-common). More info:
- how to build multiple binary packages: http://wiki.debian.org/PkgSplit
http://lists.debian.org/debian-python/2009/09/msg00091.html
- questioning usefulness of PEP 345 for automation of deb packaging:
http://lists.debian.org/debian-python/2009/11/msg00076.html
- extensions to Distutils that may meet some debian packaging needs:
http://bazaar.launchpad.net/~python-distutils-extra-hackers/python-distutils-extra/debian/files
- rough proposal by myself for support in Distutils2:
http://wiki.python.org/moin/Distutils/Proposals/SplitAndCompoundDistributions

HTH

Eric, you're right, "native" isn't exactly what I was getting at.  I want the debian/ directory created right next to the setup.py, as I've described in some follow ups.

As far as what you see in setupdebian, this is just preliminary experiments using test-driven development.  I had intended to support switches for other types of packaging options, but first things first.  Since Andrew is willing, I'd rather work with him to get some of the features I'd like to see rather than continue with a separate package, so if that proves fruitful, I'll abandon setupdebian.

BTW, have you seen the python-debian package?  It seems to have a lot of useful stuff for manipulating changelogs, control files and such.

Barry, IIUC you want the “debian” command to create a debian source
package, which can be converted to debian binary package(s), checked and
uploaded with standard tools. Perfect.

Andrew, I am uncomfortable with stdeb. (Trying to express respectful
constructive critique, not just picking on your project.) The version in
Debian testing, 0.5.1-1, still requires setuptools (actually only
pkg_resources, which is split into another package in Debian, for space
or politic reasons). The py2dsc script seems to be obsoleted by the
to-be “debian” command. The pypi-install script seems dangerous to me,
because downloading unverified software and putting it into the system
seems a rather bad idead (a.k.a. “sudo ./setup.py install considered
harmful”).

python-debian provides useful support code for generating and reading
debian formats. A quick read show the code would benefit from some 2.6
modernizing (i.e. gain in simplicity and readability by using modern
idioms and stdlib functions). Would using this package in stdeb /
distutils2.command.debian instead of shelling out bring anything?

(Sorry about the badly wrapped paragraph I forgot to remove in my
previous message. And sorry to write so much words instead of code ;)

acute-accent-ly yours

Éric, sorry about missing that accent.  /me needs a bigger font. :)

RE: python-debian modernization.  We'd have to check with the authors to see what version of Python they need to remain compatible with.  If it's 2.6, then I'm all for modernizing.

I'm not a huge fan of shelling out to get the debian directory, but dh_make seems to be what people want to emulate.  I was planning on using python-debian to fill out e.g. debian/control with information that dh_make couldn't easily be convinced to create.

I have moved the recent discussion on stdeb to a thread on the distutils-sig.

We forgot to reopen the bug.

Please don't. I've closed this bug because the bdist_deb command is currently managed by the stdeb project and it's fine like htis. Distutils2 will not have specialized linux bdist_* command like this one, because it is best to keep it in a separate project that can evolve as its own pace, in accordance with debian/ubuntu cycles.