End to end P2 Repository Mirror with Package Drone and Docker

by James O'Carroll

Speed up OSGi bundle development with an on-site P2 repository.

One of the challenges with software consulting is your client may have security policies that are well intentioned, but as a side-effect make development onerous. I've faced situations where I've needed to use a client's proprietary VPN solution to access their intellectual assets, but that same VPN will make accessing third-party repositories, software updates, and generally anything from the internet a drag. In the past, I've done development on a Java OSGi project, and the added latency when checking against a dozen Maven and P2 repositories was killing my ability to get things done. I found that package drone was a reasonable solution to set up mirrors of remote P2 repositories, and because I couldn't find a good tutorial on how to set one up, I decided to write one. This setup can work well for a single developer, but it especially pays off if you have multiple developers in one office all working on the same project-having a mirror of your dependencies on the local network can go a long way in cutting down build time and bandwidth.

I also assume that if you're reading this, you know what a repository mirror is and why you might want one, if that isn't the case check out Apache's guide to mirror settings to learn more.

This article assumes that you have a working knowledge of Docker. The example code will do the heavy lifting for you, but explaning how to build and run Docker containers has already been done by better writers, so I'm not going to redo it here. I'm also assuming that you're successfully building an OSGi bundle with Maven, if that's not working for you yet than you don't need to worry about speeding up your build time.

Getting Package Drone up and running

The official Package Drone Dockerfile is in good shape, but is getting a little dated. We need to make light modifications to the Dockerfile to properly support mirrored repositories, so we'll take the opportunity to update the Dockerfile as well:

One key change we're making is to modify the setting drone.web.maxListSize.
It's default value of 10000 is typically fine, but if we mirror a number of P2 repositories, each with hundreds or thousands of individual packages, we can blow through that limit pretty easily. In practice, I've found 20000 to be a good limit.

Get Package Drone up and running in any way you see fit, for the rest of this article I'll assume you're running it with the docker environment I provide here.

Configure Package Drone

Create Package Drone User

Package Drone will create an admin user to initially login, you'll need to look at the docker logs to view it:

docker logs package_drone

You'll see a snippet in the text output that looks like:
=================== Admin>> ===================
User: admin
Password: 0123456789abcdef
=================== <<Admin ===================

Use those credentials to log in to Package Drone's Web UI. If you're using the docker setup I've provided, it'll be available on http://:8083
By design, the Admin user can't do much. Best thing is to just use the admin account to create a new user with whatever credentials you want. After creating the user, be sure to edit it, and assign the ADMIN and MANAGER roles to it. Logout of the admin user, re-login in back as your new user.

Create a Deploy Group

  1. From the Package Drone toolbar, choose Administration->Deploy Keys. Press the "Add Group" button, and provide "Upload" as the Group Name.
  2. You'll also need to create a key to be associated with the group. From the Deploy Groups UI menu, click on the group's ID (it'll be a UUID), and choose "Create key".
    Name the key deploy, and you'll be rewarded with a screen like this:

Your password will be different from mine, note it down because we'll need it later.

You're ready to make your mirror repo!

Creating a P2 Mirror Channel

Package Drone organizes the services it supports in "Channels". Channels can be individually accessed by their URL, and each can individual properties or actions that can be configured. The only tough part here is that Package Drone mostly supports configuration through the UI-you can import and export configurations, but there isn't a great way to programmatically configure things. So, through the UI we go:

From the main menu, create a new channel. Call it P2Mirror and designate it as a Plain channel:

Configuring the Channel's Deploy Keys

Each Channel needs to configured with a Deploy Group to manage who can upload changes to the Channel. Go to the P2Mirror Channel configuration page, select the "Deploy Keys" tab, and use the UI to add the "Upload" Group to the channel:

Mirror the P2 repositories

With the Channel created and capable of receiving files via http upload, we can use eclipse's built-in capability to mirror P2 repositories, and upload the repo's contents to the Channel. A full script to make this easy is available in the repo, but here are the highlights:

Eclipse has a built-in application to mirror the metadata and contents of any P2 repository. All we have to do is call it and provide an output directory to store the contents:

# Download the metadata (content.jar)
/opt/eclipse/eclipse -nosplash -verbose -application org.eclipse.equinox.p2.metadata.repository.mirrorApplication -source "$2" -destination ./repo

# Download the artifacts
/opt/eclipse/eclipse -nosplash -verbose -application org.eclipse.equinox.p2.artifact.repository.mirrorApplication -source "$2" -destination ./repo

After zipping up the captured P2 repo, we can just use curl to upload it to Package Drone:

EPOCH=$(date +%s)
curl -X PUT --data-binary @repo.zip "http://$DEPLOY_USERNAME:$DEPLOY_PASSWORD@$PACKAGE_DRONE_URL/api/v3/upload/plain/channel/$P2_MIRROR_NAME/$1.zip?filesystem:name=$1&filesystem:epoch=$EPOCH"

The HTTP parameters of the curl call are attributes that will be assigned to the uploaded file in the Channel, the format is "namespace:key"=value. These values will become important later as we configure the Package Drone Channel to manage it's own contents. In further detail, we're creating a namespace called "filesystem", and assigning the key "name" to a simple name we define, and the key "epoch" to the current datetime. There is no special connotation to the "filesystem" namespace, it's just a sensible string value that we're choosing.

Run the script replicate_p2_repos.sh, or your version of it to mirror the repos to Package Drone. Be prepared, you're probably going to be downloading multiple gigabytes here, so you can expect this script to run for some time.

Configuring Triggers

One limitation we still have is that if we upload multiple copies of the same mirrored repo to Package Drone, Package Drone will now have multiple copies of the same repo stored. We really only want one copy of each mirror available, so if we run the mirroring script again it would be ideal if Package Drone would automatically delete the old copies of the mirrors. This can be accomplished via the Channel's triggers. By default, Package Drone has a "Channel cleanup" trigger that can be run everytime the contents of the repo has changed. We'll configure the trigger so that if the Channel has two files with the same name, it will delete the older version.

Select the Triggers button and add a "Post operation" trigger, add a "Channel cleanup" trigger.

The UI for the channel triggers might not initially be intuitive-the trigger will have some aggregator values that look like suggestions, but are actually already enabled.
Additionally, it's not enough to write in new aggregator and sorter names. You must press the accompanying button to lock in the setting.

First, remove the default aggregators by selecting the 'x' next to each of their names.
In the trigger's settings, set:

  • Aggregator: filesystem:name
  • Sorter: filesystem:epoch, select sorting order by pressing the "Add ascending" down arrow (it has the longer bars at the bottom).
  • Number of entries: 1
  • Only root artifacts
  • Ignore artifacts which don't have all aggregator fields.
    Configured correctly, the trigger description should read:
    Group all artifacts by: filesystem:name then sort by the values of: filesystem:epoch then delete all but the last 1 entries of each group. Only root artifacts will be processed. Artifacts which are missing an aggregator field will be ignored.

Configuring Maven to reference your mirrored repo

With a working P2 Mirror repository running, all that's left is to let Maven know about it. There's a number of different ways to reference remote repositories and mirrors, but the simplest way is to create or edit your version of settings.xml. settings.xml is typically located at ~/.m2/settings.xml, it may or may not already exist on your development system. In order to let maven know about available an P2 mirror, you simply add an entry like this:

<mirror>
  <id>oxygen-drone</id>
  <mirrorOf>oxygen</mirrorOf>
  <url>http://192.168.1.101:8083/unzip/newestByName/P2Mirror/oxygen.zip</url>
  <layout>p2</layout>
  <mirrorOfLayouts>p2</mirrorOfLayouts>
</mirror>

This assumes your Package Drone instance is available at 192.168.1.101, port 8083, and the channel is still called P2Mirror. It's also assumes in your project's pom or targetfile you have a reference to the mirrored repository marked with an ID "oxygen" - Maven isn't smart enough to check mirrors by url or content, it's relying on you to correctly match up and values.

Note: if you want, you can put an entry like this in settings.xml

# NOT recommended!!!
<profiles>
  <profile>
    <id>development</id>
    <repositories>
  	<repository>
  	  <id>oxygen</id>
  	  <url>http://download.eclipse.org/releases/oxygen/201709271000</url>
  	  <layout>p2</layout>
  	  <releases>
  	    <updatePolicy>daily</updatePolicy>
  	  </releases>
  	</repository>
    </repositories>
  </profile>
</profiles>
<activeProfiles>
  <activeProfile>development</activeProfile>
</activeProfiles>

This can be a quick and visible way of associating a P2 repo with it's mirror. However, adding this in your settings.xml file effectively adds this repo as a package source to ALL projects you're building in the machine, which will force it to do more package resolution than is necessary, and will probably end up giving you package conflicts. So don't do it. Instead, you'll have to go through your project's pom.xml files or Eclipse Target Definition file, and manually set the ID field of all your referenced P2 repositories.

How do I know it's working?

This is easy:mvn clean install
If you have everyting configured correctly, you can watch dependency bundle downloads in the build log, and you'll see them coming from your mirror's local IP address instead of the remote repository.

Wrapping up and sources

The Dockerfile environment and the script to mirror the remote repositories can be found here. Good luck have fun.