Automating Debian package creation and management with Maven/Ant

October 2, 2013 – 17:09 by Adnan Hodzic

Preface

 

This document was composed in aim to briefly reflect on Debian packaging system (dpkg) and provide information on how Debian packages are automatically created and managed (uploaded) using Maven/Ant. Scope of the document implies that the reader already has basic knowledge of Debian/dpkg and/or Maven/Ant. Even though there are concise theoretical explanation, author tried the “teach by examples” approach, thus you’ll be able to find plethora of code examples.Debian packet creation is more then just a simple hack which consists of putting right files into right directories, there’s also lot of parts of packing process which weren’t explained in depth. I highly advise you read the official Debian New Maintainers’ Guide to get a full understanding on what was tried to be said here.

Since intention of this document is to be as straightforward as possible, for your assistance some parts have been marked as:
  • “Technical²”, providing additional technical insight, isn’t absolutely necessary and can even be skipped: Technical²
  • “Additional info.”, additional notes regarding particular step, should pay attention: Additional information


What is this “black magic” all about?

With our company, before introduction of Debian packages for every service/application we have in production, things were set/configured and therefor done manually with Puppet, yes I did dare to say “manual” and “Puppet” in same sentence :)
Debian is a Linux distribution (one of the oldest and most popular) that has one of the most powerful software packaging control systems, and since our whole infrastructure is based on it, this gives us limitless possibilities. By adding outside “Debian packaging wrapper” (which doesn’t affect your current code in any way), you’re able to add ability for Debian package to be created and deployed every time you run a build on TeamCity!
So, how does this work?
Every time you run a build in your company’s build management system (in our case it’s TeamCity) …
1 - teamcity screenshot - close up

besides your service/package being built, a proper (adheres to rules and policies) Debian package is being made.

2 - package creation

Besides Debian package being built, package is also being uploaded to our repositories. We follow Debian’s strict policy on package quality, thus in order for package to appear in our repository listings, Lintian check is being made.

Technical² Technical²:
Lintian checks package by performing checks on inconsistencies and errors, as well as policy violations. If package has no errors, it is then accepted into our software repositories. More information about this whole process is available in “Package Creation” section.
3 - file monitor
ebuddy-file-monitor in action, performing lintian checks and upload process.

4 - repos
Your application/service is ready for installation/deployment.

Additional information
  • After each build upload, you may run sudo apt-get update && apt-cache policy packagename to check if the particular version is there.
  • If you want to install particular package version which is not a installation candidate, you can do so by: sudo apt-get install packagename=version.number


Anatomy of Debian package created with Maven/Ant

Package creation with Maven

 

Rather then using “regular” method and creating our packages with (dpkg), method explained here is targeted towards automation, cross-platform support and Java developers as packages are built using jdeb plugin. Also be warned that both methods mentioned here, even though they may seem similar or even identical, in reality they can greatly differ. So if you were able to create Debian package with Maven/Ant doesn’t necessarily mean that you’re proficient in creating Debian packages.Since I think best way to explain what is done here is through examples, I’ll heavily rely on using examples. Most simple way to break up the whole process is by dividing it into 2 sections:

  • Adding required files into the ordered directory structure
  • Editing the child .pom file and adjusting it to the changes you made to directory structure

If we take our xms-server as an example:

1 example
Contents of “xms-server/webapp/” | Plain textwe want to concentrate what’s in “..webapp/src/deb/” directory (line: 3-10); if the directory doesn’t exist, first step would be to create a directory named “deb/”.

Files and directories required under deb/ directory

  • control/

This is the most important directory where most of the “magic happens”. It contains various files with values which are later used to manipulate and manage the package. Without this directory package won’t be created.

2 example
Example file structure within deb/control | Plain text

Besides the control directory itself, there’s couple of files which are absolutely required under this same directory. One of them is:

control (../control/control)

Within this file there’s also few lines without which package creation will not happen:

3 example
Contents of “control” file | Plain text

  • Line 1 (Package): Name of the binary package, i.e: this is how the package will be called in our repositories.
  • Line 2 (Section): Distribution section it belongs to.
  • Line 3 (Priority): How important it is for user to have this package installed.
  • Line 4 (Architecture): Architecture for which binary package was built for.
  • Line 5 (Homepage): Upstream author’s URL.
  • Technical²Line 6 (Depends): List of package/s which will need to be installed in order for installation process of our package to take place. If dependencies are not satisfied, during install procedure of our package, Debian package management system (dpkg) will mark its dependency to be installed along with it. Other behaviours of packages relations which might be worth noting are: Pre-Depends/Recommends/Suggests/Conflicts/Breaks/Provides.
  • Line 7 (Version): Upstream version number. If you want your version to be picked up from TeamCity you’ll inject build version to the <properties> element.
  • Line 8 (Maintainer): Name and contact information of person/team who is responsible for maintaining this package.
  • Line 9 (Description): Brief summary of package and its description.
  • Line 10 (Description – extended): Extended description of package and software that’s behind it.

Notes:
In regards to “Package” field. Please add “ebuddy” prefix to every service/application that has been build/developed by eBuddy. Separate name of the package with dashes, (i.e: ebuddy-authservice, ebuddy-xms-web, etc …)

In regards to “Depends” field. Values used in combination with package/s versions are:

Less then: <<
Less then or equal to: <=
Strictly equal to: =
Equal or greater then: =>
Greater then >>
Or: |

Example: Depends: java7-runtime, >= tomcat6

Additional information Additional info.:

This is the very brief explanation of control file and its content, there’s couple of more fields whose explanation might be worth explaining but are not covered by the scope of this document.


deb/control

conffiles

Besides indispensable control file, package also contains conffiles. What are these conffiles?

Most people hates upgrades, because upgrades are prone to breaking stuff. This file is usually used for configuration files, as files marked within this file won’t be overwritten. If anything during the installation, before overwriting configuration files installer will ask you if you want to keep current configuration or not.

5 example
Example contents of “conffiles” | Plain text

preinst

This script will be executed before installation process. Might be worth noting that this script will be run even before package has been even unpacked from .deb/

6 example
Example of “preinst” file | Plain text

postinst

As the name says “postinst” this script will be run after package installation process. This script is usually run after the installer to configure or restart services.

6 example
Example of postinst file | Plain text

There’s dozens of more files which can be found under deb/control directory.

Technical² Debinazing your directory

Instead of creating these files manually, or copying the existing files from other packages/projects. You can “Debianize” your source directory by running “dh_make” command which will create debian/ directory within your source/ directory along with all necessary files. This command will even make the example files marked with .ex (i.e: menu.ex), you’re free to remove all the files you don’t need.

Please note that you’ll be able to run this step on Debian (or other Debian based Linux distribution) running machine and you’ll need to install devscripts and dh-make package (sudo apt-get install devscripts dh-make). Once run, dh_make will use DEBEMAIL and DEBFULLNAME environmental variables, if not set use option to set these variables manually. Command you’ll most likely want to use is:

dh_make -c GPL3 -e adnan@hodzic.org -n -p packagename_1

  • -c GPL3 (copyright for files under debian/ directory)
  • -e adnan@hodzic.org (maintainer email address)
  • -n (native package, will not generate .orig file)
  • -p (force package name and version)

Other directories and files necessary under deb/ directory

Unlike in “regular” Debian package creation process where “dpkg-buildpackage” is being used to make the actual package while reading/relying on debian/rules file. In process of creating package for Maven/Ant, you lay a directory structure which you’d want to be shown in a package.

In case you want a particular file to be present in i.e: /usr/share/$packagename/, you’ll make a directory with same contents within the deb/ directory.

Thus if we look at the structure of xms-server service we’ll see following scenario:

8 example
Directory and file structure within deb/ | Plain text

Additional information Regardless, it’s important to note that all files that are currently necessary in our directory tree are the files that are part of Debian packaging system and files which will not be mapped within the pom.xml with Maven/Ant later on in next step.

Files which are present in these directories are different depending on the package purpose, but files which should always be present are located in /usr/share/doc/$packagename, and every package should at least have files “changelog.gz” and “copyright” present in that directory otherwise package will most likely fail from being uploaded due to lintian errors.

changelog.gz file is used to keep record of changes that were made to the Debian package.

9 example
Example contents of “changelog.gz” | Plain text

copyright file contains information about copyright and licences, of the package itself as well as the upstream source.

10 example
Example contents of “copyright” | Plain text

Maven – Mirroring changes we made to our POM.xml

Making Debian package from Maven point of view is fairly easy, as all it requires you to do is add jdeb plugin to your POM file.

11 example
jdeb plugin | Plain text

12 example
Example of jdeb plugin in xms-server | Plain text

Additional properties you may want to add are following:

13 example
teamcity.build.version | Plain text

Adding package upload support

To enable upload support for our packages you need to add “remote-deploy-deb” profile along with necessary plugins your pom.xml.

14 example
xms-server example remote-deploy-deb profile | Plain text

After you’ve made these changes to your project, you’ve have added Debian packing wrapper, which doesn’t affect your current code in any way and which allows you to create Debian (.deb) packages with Maven.

Build!

Of course, after pushing your changes, you’re able to run the build which will also create Debian package from now on. Or you might want to try building your package locally. Simplest way to do this is by running following command in your projects root directory:

mvn clean install -DskipTests

Upon successful build process, you’ll be able to find your package in specified directory, in xms-server case: ../src/webapp/target/ebuddy-xmsserver_2.23.0-SNAPSHOT_all.deb

Also be aware that your versioning will be different locally and once run on TeamCity, due to TeamCity injected build versions and et cetera.

Technical² Package check/Lintian

In case there were no Lintian errors, package will be uploaded to our Debian package repository from where it’ll be available for installation. If you’re package is not showing up in repositories after build being complete, if you did everything as instructed here, it’s most likely that it failed lintian check. It’s best if you simply contact the author of this article to take care of your package, or if you’re feeling brave proceed with reading :)

Additional information You’ll need Debian based Linux distribution with lintian package installed (sudo apt-get install lintian). Usually lintian is where package build source is being checked, in our case we don’t have that so we’ll want to run lintian against our .deb binary, i.e:

lintian ebuddy-xmsserver_2.23.0-SNAPSHOT_all.deb

Results will be shown as “Warning (W)” or “Errors (E)”, warnings in our case are acceptable and won’t hamper your package from being uploaded. However, error messages will and you’ll want to fix those package problems. Best practice is to Google your error/warning + lintian or simple finding the problem on Lintian Tags website. Also be aware that, running lintian check locally will give you different results then running it on production/dev. machines, simply for a reason of teamcity build/version injections not being applied locally.

Difference between packaging with Maven and Ant

Package creation with Ant

Repeating this same process with Ant has some rather subtle differences with Maven. To illustrate this more clearly, we’ll use our “xms-web” service as an example. First thing we’ll want to do is create our deb/debian directory in:servicename/build/debian
So this is how the file and directory structure looks like for “xms-web”
15 example

xms-web/build/debian directory structure | Plain text

Generally, when it comes to file/directory structure, even though not identical things are greatly similar to Maven’s structure.

16 example
Contents of xms-web/build/debian/control | Plain text

Technical² You can use different values with Package and Version fields. Package: ebuddy-xms-web could also be Package: ${app.name}, even though in packagename, author of this document would always recommend you hardcode the package name field. Also as displayed in example above, instead of hardcoding the Version filed, you can use: Version: ${label}.

build.xml instead of pom.xml

Instead of adding our “debian packaging wrapper” to pom.xml, we’ll do so by adding it to build.xml which is Ant’s build file.
17 example

“Debian packaging wrapper” inside of build.xml | Plain text

Definition: jdeb

 

One particularity that differs from Maven is that you need to download jdeb plugin manually and place into your ..build/ directory, after which you should make sure it’s set in your classpath attribute.

jdeb task definition:

<taskdef name=”jdeb” classname=”org.vafer.jdeb.ant.DebAntTask” classpath=”jdeb-1.0.1.jar”/>

Running build, locally

As previously mentioned, author highly encourages running the local build before pushing the changes. With Ant this is done differently, in your ../build directory run:

ant jdeb.build

Which will result with our newly created .deb file being placed into ../build/debian directory.

Troubleshooting/FAQ

 

How to check the package contents?

 You can do this by running: dpkg -c packagename.deb

18 example
i.e: dpkg -c ebuddy-file-monitor_1.0.10_all.deb | Plain text
Or if you prefer GUI, you can do this with any archive tool. Example below illustrate viewing the same package archive with “Archive Manager” on Debian Jessie.

5 - viewing .deb file with archive manager

Technical² How do I write changes to changelog?

As you might’ve noticed, changelog file has .gz extension at the end of it.How to extract the changelog.gz

gunzip changelog.gz

How to properly compress the changelog

gzip -9 changelog

Does it matter if you name your directory debian or deb?

No, you can name “Debian packaging wrapper” directory whatever you like, but for the sake of consistency we advocate it is highly advised you name it “deb” or “debian”.

Technical² How to add supplementary options to my package, such as manpages?

Add file “ebuddy-package-name.1” to “..debian/usr/share/man/man1” directory.

19 example
Example contents of “ebuddy-package-name.1” | Plain text

After package installation you’ll be able to use “man ebuddy-package-name” to see manual page for that particular package.

Additional information Typically, you can add any option/file which is consistent with directory structure used within your debian/ directory, after adding it you’ll simply have jdeb pick it up.

Reference:

Questions/Feedback/Contact?

If you have any additional questions or comments, or would just like to provide us with feedback, please use the comments section below.

 

Disqus Comments

foolcontrol