Sure, here you go: PyZMQ Paranoid Pirate

Considering ZeroMQ for your application

I sometimes get the opportunity to work with ZeroMQ professionally. ZeroMQ is a very powerful and versatile tool, which is great, but comes with the tradeoff of needing additional development work and testing to ensure your application code is using it correctly. If you're considering using ZeroMQ, you should first ask yourself if one of it's competing technologies will work. If you have sufficient hardware and can use an external piece of hardware or a cloud service to run something like RabbitMQ, ActiveMQ, or MQTT, I generally recommend you do it. These technologies are going to just be easier to develop against, and you'll save a good chunk of development time using them instead. That being said, if your use case involves distributed networking without a dedicated central hub, or specifically requires a very quick messaging rate, pull in ZeroMQ.

ZeroMQ DEALER/ROUTER

If you're reading this, you should already have some knowledge on ZeroMQ works and it's multiple forms of communication. For this article, we're going to focus on the DEALER/ROUTER pattern. The official documentation for ZeroMQ is better than what I can write, so reference these articles for more information on what the DEALER/ROUTER pattern is and how it works:

ZeroMQ asynchronous client/server pattern
Dealer and Router socket explanation

The key thing you'll need to take away for this article is to know that DEALER/ROUTER ZeroMQ sockets allow for reliable bi-directional communication. If your use case doesn't need bidirectional communication, you can use a simpler ZeroMQ pattern-Publish/Subscribe or Request/Reply. These work as advertised, but I often find that typical use cases demand an initial handshake before sending information, and to be alerted if their target fails for some reason. The ZeroMQ developers recognized this need long ago, and describe a solution with DEALER/ROUTER called the Paranoid Pirate

That link is it will describe the primary benefit of the paranoid pirate-keeping a heartbeat in addition to client/worker communications gives you a ZMQ-style way of letting clients and workers introduce themselves to each other, and an established way for these partners to halt/resume communications. You can even get a core Python code sample.

The bad news is that while the ZeroMQ documentation is readable and does a great job describing the ZeroMQ protocol, but doesn't feature many application level examples-there are some snippets that show you how to correctly set up communications, but larger examples are left to higher-level libraries that wrap around ZeroMQ. These higher level libraries like PyZMQ or zmqpp in turn explain their own usage well, but don't really provide an example implementation of any of the higher level patterns like the paranoid pirate. So, let's implement the paranoid pirate in PyZMQ.

A PyZMQ implementation of the ZeroMQ Paranoid Pirate pattern

We'll start with an intentionally simplified version of the Paranoid Pirate-the "full" implementation would include a single client and multiple workers, but to keeps things simple here we'll stick to just a single worker. Additionally, we'll make the example a bit easier to visualize by implementing an RPC-style interface-the client will issue a request to the worker, and the worker will check that request against a set of known procedures it has, and will either return a result or throw an exception. For the most part, the sample code is documented well, so to understand what's going on I'm mostly going to direct you to look at the code. Still, let's look at a few key parts and see what outcome we get when we run the sample.

Let's run it!

I've included a few scripts that should make running this code easier. First, take a look at run_demo_xero_in_docker_env.sh

#!/bin/bash

set -eu
set -o pipefail

docker-compose build && docker-compose up --abort-on-container-exit \
    codedrunk_xero_uniclient \
    codedrunk_xero_uniworker

I've included some dockerfiles that will build the example and run some integration tests to make sure things are working correctly. If everything builds correctly, you should see an output like this:

Building codedrunk_xero_base
Step 1/16 : FROM ubuntu:18.04
 ---> ccc6e87d482b
Step 2/16 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> 05583fe42b89
...
Step 16/16 : RUN (cd demo_xero && pip3 install .)
 ---> Using cache
 ---> f63b97161944

Successfully built f63b97161944
Successfully tagged stackhead/codedrunk_xero_base:latest
Building codedrunk_xero_uniclient
Step 1/2 : FROM stackhead/codedrunk_xero_base
 ---> f63b97161944
Step 2/2 : CMD ["demo_xerouniclient", "tcp://*:5550", "ping", "--count", "20"]
 ---> Running in 7d5aa7ad7cf5
Removing intermediate container 7d5aa7ad7cf5
 ---> ba084294f665

Successfully built ba084294f665
Successfully tagged stackhead/codedrunk_xero_uniclient:latest
Building codedrunk_xero_uniworker
Step 1/2 : FROM stackhead/codedrunk_xero_base
 ---> f63b97161944
Step 2/2 : CMD ["demo_xerouniworker", "tcp://codedrunk_xero_uniclient:5550"]
 ---> Using cache
 ---> 3b83047e3d4b

Successfully built 3b83047e3d4b
Successfully tagged stackhead/codedrunk_xero_uniworker:latest
Starting codedrunk_xero_uniworker   ... done
Recreating codedrunk_xero_uniclient ... done
Attaching to codedrunk_xero_uniworker, codedrunk_xero_uniclient
codedrunk_xero_uniworker    | 03:49:43.094 asyncio      DEBUG    [MainThread  ] Using selector: EpollSelector
codedrunk_xero_uniworker    | 03:49:43.095 asyncio      DEBUG    [MainThread  ] Using selector: EpollSelector
codedrunk_xero_uniclient    | 03:49:43.162 asyncio      DEBUG    [MainThread  ] Using selector: EpollSelector
codedrunk_xero_uniclient    | 03:49:43.162 asyncio      DEBUG    [MainThread  ] Using selector: EpollSelector
codedrunk_xero_uniclient    | 03:49:43.280 xero.uni.uniclient INFO     [uniclientthread] _register_worker
codedrunk_xero_uniclient    | 03:49:43.280 xero.uni.uniclient DEBUG    [uniclientthread] worker.registerWorker for 'b'\x00k\x8bEg'' is connected.
codedrunk_xero_uniclient    | 03:49:43.281 demo_xero.demo_xerouniclient DEBUG    [uniclientthread] partial msg: 'started'
codedrunk_xero_uniclient    | 03:49:43.282 demo_xero.demo_xerouniclient DEBUG    [MainThread  ] rpc reply: 'pong'
...
codedrunk_xero_uniclient    | 03:49:43.292 demo_xero.demo_xerouniclient DEBUG    [uniclientthread] partial msg: 'started'
codedrunk_xero_uniclient    | 03:49:43.292 demo_xero.demo_xerouniclient DEBUG    [MainThread  ] rpc reply: 'pong'
codedrunk_xero_uniclient    | 03:49:43.293 demo_xero.demo_xerouniclient DEBUG    [uniclientthread] partial msg: 'started'
codedrunk_xero_uniclient    | 03:49:43.293 demo_xero.demo_xerouniclient DEBUG    [MainThread  ] rpc reply: 'pong'
codedrunk_xero_uniclient    | Starting client
codedrunk_xero_uniclient    | waiting for worker
codedrunk_xero_uniclient    | Shutting down
codedrunk_xero_uniclient exited with code 0
Aborting on container exit...
Stopping codedrunk_xero_uniworker   ... done

You should find the dockerfiles have installed the example python packages and are running the basic "ping" command as quickly as possible.

If you want to run the samples directly on your dev machine, install both xero and demo_xero via:

pip3 install -e ./xero
pip3 install -e ./demo_xero

In one terminal, run the worker:

./bin/demo_xerouniworker tcp://127.0.0.1:5550

In a second terminal, run any of the commands in demo_xero/run_examples.sh:

#!/bin/bash

set -eu
set -o pipefail

# Simplest example-call RPC with no arguments and get single string as a reply
./bin/demo_xerouniclient tcp://127.0.0.1:5550 ping

# Similar, call RPC with no arguments and ensure the "reply" comes in, but no data returned
./bin/demo_xerouniclient tcp://127.0.0.1:5550 return_none

# Call an RPC that just does an equality comparison. Simple example to demonstrate how to pass arguments to worker RPC
./bin/demo_xerouniclient tcp://127.0.0.1:5550 compare --args '"apple", "orange"'

# Effectively the same thing, but showing how to use named arguments.
./bin/demo_xerouniclient tcp://127.0.0.1:5550 compare --kwargs '"str1":"apple", "str2":"orange"'

# Call an RPC that doesn't actually exist, and make sure the worker is able to handle the problem cleanly while returning
# the actual exception that is raised.
./bin/demo_xerouniclient tcp://127.0.0.1:5550 invalid_request

# The slow_ functions are serviced by an "actor" model. slow_succeed will succeed after the provided timeout,
# slow_fail will fail after the provided timeout.
./bin/demo_xerouniclient tcp://127.0.0.1:5550 slow_succeed --args 5 --timeout 20
./bin/demo_xerouniclient tcp://127.0.0.1:5550 slow_fail --args 5 --timeout 20

A warning about PyZMQ

The PyZMQ project seems to be under pretty active development, which is great. However, it looks like they're working with a fairly challenging dependency environment as they try to incorporate/maintain compatibility with Facebook's Tornado module. I'd say they're doing rather well, but it's still easy to inadvertently create a system that seems to work reliable for tens of seconds, or even minutes, but will later stop dropping messages for seemingly no reason. This can be mitigated by making a particular effort to version match your environment, PyZMQ, and tornado once you have something that works reliably. The PyZMQ team also keeps a good listing of errata/upgrade headaches and their solutions listed in their documentation, so if you have reliability issues with PyZMQ I'd carefully read their explanation of PyZMQ and the event loop. Whenever I've had problems with this sample breaking, it is because the PyZMQ team had to make updates under the hood to manage the different async libraries they're compatible with, and the problem could be fixed by reading the changelog and updating accordingly.

Summary

Thanks for reading, this should give you some ideas on how to use PyZMQ with some other modern python libraries to do robust messaging. This sample hasn't been battle-hardened on a production system, but it does includes some tests that should keep it running on your dev machine. Here's a link to the full implementation again: PyZMQ Paranoid Pirate.

Out of the box, modern Java development can be pretty resource intenstive. On a typical Java development project, I may have to download gigabytes worth of dependencies, source packages, javadocs, and the Eclipse IDE on top in order to just get started. Multiply that by the number of developers you have on your project and add a CI server building and rebuilding the package everytime you push an update, and you've got some real time and bandwidth getting eaten as you develop. Similar to how you can cache dependencies for OSGi bundles with a repsotory manager on your local network, you can use Nexus 3 to set up a Maven repository proxy. Set up correctly, your development machine will check against your local Maven repository to satisfy your build dependencies before reaching out to the internet. Better yet, this local cache will automatically grow to match the dependencies you're attempting to include-you won't need to manually specify what packages you want to save, only the remote repositories that you should potentially proxy.

I'm assuming you have a working knowledge of Docker. A Docker image will be provided for you, but I'm counting on you to know how to deploy it to your server, and how to run it. I'll also be showing you how to let Maven know about the proxy server, but I'll also assume you have a Maven project to build. After all, why would you care about speeding up your Maven builds, if you're not using Maven already?

Setting up Nexus 3

We're going to use Nexus 3 as our proxy, happily the community version of Nexus 3 supports Maven proxy repositories out of the box. Even better, Sonatype keeps a Dockerfile maintained for Nexus3, so we don't have to write one. I've provided a Docker Compose file to match my personal perferences. Get it up and running on your local server. If you're using the provided docker compose file, the Nexus UI will become available on port 8084. You'll also have to give it a little time after starting the container before the UI will actually become available.

Configuring Nexus 3 Proxies

You'll have to sign in before you can do anything further, the default username and password are admin/admin123. This would be a great opportunity to set a new admin password. Once you've logged in, a gear symbol will become available at the top toolbar. Select it, and then select the Repository->Repositories section on the left toolbar.

The default maven-central proxy repository

The default installation of Nexus 3 comes with a number of built-in repositories:

Lucky for us, it looks like one of the default repositories is a proxy for maven central. Click on it to take a closer look:

This tells us all of the settings we'll need to make other proxies.

Make a jcenter proxy repository

Return to the repositories page, and choose the "Create Repository" button on top. When presented with the list of repository recipes, choose "maven2 (proxy)".
You'll be presented with a new page, similar to the maven central configuration page we saw earlier. You'll need to consider these fields:

1. Name: set this to a unique ID, I just name mine to match the name of the remote maven proxy. We'll just call it jcenter.
2. URL: Just to make it easy to track, match this URL against the Name. In this case, we'll set it to http://<server_ip_address>:8084/repository/jcenter/
3. Maven 2->What type of artifacts does this repository store? Maven repositories can store release artifacts, snapshot artifacts, or mixed (both release and shapshot). To the best of my knowledge, there's no reason to not choose mixed, so go with that.
4. Proxy->Remote Storage. This will have to match the location of the remote repository. You can figure this out by looking at your maven build logs, but for jcenter you'll want https://jcenter.bintray.com/

That's it! Press Save and this proxy is ready to go. If you're following along, your page will look like this:

By the way, the default maven central repository is set to only proxy Release packages, if you want to also support snapshot packages you'll have to delete and remake the repository with the updated settings.

Configuring Maven to reference your proxies

With Nexus 3 ready to go, all that's left is to let Maven know about it. You'll typically want to edit Maven's settings.xml file to accomplish this. settings.xml is typically located at ~/.m2/settings.xml, it may or may not already exist on your development system. In order to let maven know your proxied repositories, you simply add an entry like this:

<mirrors>
  <mirror>
    <id>maven-central</id>
    <name>maven-central</name>
    <url>http://192.168.1.101:8084/repository/maven-central/</url>
    <mirrorOf>central</mirrorOf>
  </mirror>
  <mirror>
    <id>jcenter-mirror</id>
    <name>jcenter-mirror</name>
    <url>http://192.168.1.101:8084/repository/jcenter/</url>
    <mirrorOf>jcenter</mirrorOf>
  </mirror>
</mirrors>

This assumes your Nexus 3 instance is available at 192.168.1.101, port 8084. It's also assumes in your project's pom or targetfile you have a reference to the mirrored repositories marked with IDs "central" and "jcenter". Maven isn't smart enough to check mirrors by url or content, it's relying on you to correctly match up and values.

How do I know it's working?

This is easy:mvn clean install
If you have everyting configured correctly, you can watch dependency bundle downloads in the build log, and you'll see them coming from your mirror's local IP address instead of the remote repository.
That's about it-Nexus 3 does a lot of the lifting for you, so this isn't too hard to set up. Good luck have fun.

One of the challenges with software consulting is your client may have security policies that are well intentioned, but as a side-effect make development onerous. I've faced situations where I've needed to use a client's proprietary VPN solution to access their intellectual assets, but that same VPN will make accessing third-party repositories, software updates, and generally anything from the internet a drag. In the past, I've done development on a Java OSGi project, and the added latency when checking against a dozen Maven and P2 repositories was killing my ability to get things done. I found that package drone was a reasonable solution to set up mirrors of remote P2 repositories, and because I couldn't find a good tutorial on how to set one up, I decided to write one. This setup can work well for a single developer, but it especially pays off if you have multiple developers in one office all working on the same project-having a mirror of your dependencies on the local network can go a long way in cutting down build time and bandwidth.

I also assume that if you're reading this, you know what a repository mirror is and why you might want one, if that isn't the case check out Apache's guide to mirror settings to learn more.

This article assumes that you have a working knowledge of Docker. The example code will do the heavy lifting for you, but explaning how to build and run Docker containers has already been done by better writers, so I'm not going to redo it here. I'm also assuming that you're successfully building an OSGi bundle with Maven, if that's not working for you yet than you don't need to worry about speeding up your build time.

Getting Package Drone up and running

The official Package Drone Dockerfile is in good shape, but is getting a little dated. We need to make light modifications to the Dockerfile to properly support mirrored repositories, so we'll take the opportunity to update the Dockerfile as well:

One key change we're making is to modify the setting drone.web.maxListSize.
It's default value of 10000 is typically fine, but if we mirror a number of P2 repositories, each with hundreds or thousands of individual packages, we can blow through that limit pretty easily. In practice, I've found 20000 to be a good limit.

Get Package Drone up and running in any way you see fit, for the rest of this article I'll assume you're running it with the docker environment I provide here.

Configure Package Drone

Create Package Drone User

Package Drone will create an admin user to initially login, you'll need to look at the docker logs to view it:

docker logs package_drone

You'll see a snippet in the text output that looks like:
=================== Admin>> ===================
User: admin
Password: 0123456789abcdef
=================== <<Admin ===================

Use those credentials to log in to Package Drone's Web UI. If you're using the docker setup I've provided, it'll be available on http://:8083
By design, the Admin user can't do much. Best thing is to just use the admin account to create a new user with whatever credentials you want. After creating the user, be sure to edit it, and assign the ADMIN and MANAGER roles to it. Logout of the admin user, re-login in back as your new user.

Create a Deploy Group

1. From the Package Drone toolbar, choose Administration->Deploy Keys. Press the "Add Group" button, and provide "Upload" as the Group Name.
2. You'll also need to create a key to be associated with the group. From the Deploy Groups UI menu, click on the group's ID (it'll be a UUID), and choose "Create key".
   Name the key deploy, and you'll be rewarded with a screen like this:
   

Your password will be different from mine, note it down because we'll need it later.

You're ready to make your mirror repo!

Creating a P2 Mirror Channel

Package Drone organizes the services it supports in "Channels". Channels can be individually accessed by their URL, and each can individual properties or actions that can be configured. The only tough part here is that Package Drone mostly supports configuration through the UI-you can import and export configurations, but there isn't a great way to programmatically configure things. So, through the UI we go:

From the main menu, create a new channel. Call it P2Mirror and designate it as a Plain channel:

Configuring the Channel's Deploy Keys

Each Channel needs to configured with a Deploy Group to manage who can upload changes to the Channel. Go to the P2Mirror Channel configuration page, select the "Deploy Keys" tab, and use the UI to add the "Upload" Group to the channel:

Mirror the P2 repositories

With the Channel created and capable of receiving files via http upload, we can use eclipse's built-in capability to mirror P2 repositories, and upload the repo's contents to the Channel. A full script to make this easy is available in the repo, but here are the highlights:

Eclipse has a built-in application to mirror the metadata and contents of any P2 repository. All we have to do is call it and provide an output directory to store the contents:

# Download the metadata (content.jar)
/opt/eclipse/eclipse -nosplash -verbose -application org.eclipse.equinox.p2.metadata.repository.mirrorApplication -source "$2" -destination ./repo

# Download the artifacts
/opt/eclipse/eclipse -nosplash -verbose -application org.eclipse.equinox.p2.artifact.repository.mirrorApplication -source "$2" -destination ./repo

After zipping up the captured P2 repo, we can just use curl to upload it to Package Drone:

EPOCH=$(date +%s)
curl -X PUT --data-binary @repo.zip "http://$DEPLOY_USERNAME:$DEPLOY_PASSWORD@$PACKAGE_DRONE_URL/api/v3/upload/plain/channel/$P2_MIRROR_NAME/$1.zip?filesystem:name=$1&filesystem:epoch=$EPOCH"

The HTTP parameters of the curl call are attributes that will be assigned to the uploaded file in the Channel, the format is "namespace:key"=value. These values will become important later as we configure the Package Drone Channel to manage it's own contents. In further detail, we're creating a namespace called "filesystem", and assigning the key "name" to a simple name we define, and the key "epoch" to the current datetime. There is no special connotation to the "filesystem" namespace, it's just a sensible string value that we're choosing.

Run the script replicate_p2_repos.sh, or your version of it to mirror the repos to Package Drone. Be prepared, you're probably going to be downloading multiple gigabytes here, so you can expect this script to run for some time.

Configuring Triggers

One limitation we still have is that if we upload multiple copies of the same mirrored repo to Package Drone, Package Drone will now have multiple copies of the same repo stored. We really only want one copy of each mirror available, so if we run the mirroring script again it would be ideal if Package Drone would automatically delete the old copies of the mirrors. This can be accomplished via the Channel's triggers. By default, Package Drone has a "Channel cleanup" trigger that can be run everytime the contents of the repo has changed. We'll configure the trigger so that if the Channel has two files with the same name, it will delete the older version.

Select the Triggers button and add a "Post operation" trigger, add a "Channel cleanup" trigger.

The UI for the channel triggers might not initially be intuitive-the trigger will have some aggregator values that look like suggestions, but are actually already enabled.
Additionally, it's not enough to write in new aggregator and sorter names. You must press the accompanying button to lock in the setting.

First, remove the default aggregators by selecting the 'x' next to each of their names.
In the trigger's settings, set:

• Aggregator: filesystem:name
• Sorter: filesystem:epoch, select sorting order by pressing the "Add ascending" down arrow (it has the longer bars at the bottom).
• Number of entries: 1
• Only root artifacts
• Ignore artifacts which don't have all aggregator fields.
  Configured correctly, the trigger description should read:
  Group all artifacts by: filesystem:name then sort by the values of: filesystem:epoch then delete all but the last 1 entries of each group. Only root artifacts will be processed. Artifacts which are missing an aggregator field will be ignored.
  

Configuring Maven to reference your mirrored repo

With a working P2 Mirror repository running, all that's left is to let Maven know about it. There's a number of different ways to reference remote repositories and mirrors, but the simplest way is to create or edit your version of settings.xml. settings.xml is typically located at ~/.m2/settings.xml, it may or may not already exist on your development system. In order to let maven know about available an P2 mirror, you simply add an entry like this:

<mirror>
  <id>oxygen-drone</id>
  <mirrorOf>oxygen</mirrorOf>
  <url>http://192.168.1.101:8083/unzip/newestByName/P2Mirror/oxygen.zip</url>
  <layout>p2</layout>
  <mirrorOfLayouts>p2</mirrorOfLayouts>
</mirror>

This assumes your Package Drone instance is available at 192.168.1.101, port 8083, and the channel is still called P2Mirror. It's also assumes in your project's pom or targetfile you have a reference to the mirrored repository marked with an ID "oxygen" - Maven isn't smart enough to check mirrors by url or content, it's relying on you to correctly match up and values.

Note: if you want, you can put an entry like this in settings.xml

# NOT recommended!!!
<profiles>
  <profile>
    <id>development</id>
    <repositories>
  	<repository>
  	  <id>oxygen</id>
  	  <url>http://download.eclipse.org/releases/oxygen/201709271000</url>
  	  <layout>p2</layout>
  	  <releases>
  	    <updatePolicy>daily</updatePolicy>
  	  </releases>
  	</repository>
    </repositories>
  </profile>
</profiles>
<activeProfiles>
  <activeProfile>development</activeProfile>
</activeProfiles>

This can be a quick and visible way of associating a P2 repo with it's mirror. However, adding this in your settings.xml file effectively adds this repo as a package source to ALL projects you're building in the machine, which will force it to do more package resolution than is necessary, and will probably end up giving you package conflicts. So don't do it. Instead, you'll have to go through your project's pom.xml files or Eclipse Target Definition file, and manually set the ID field of all your referenced P2 repositories.

How do I know it's working?

This is easy:mvn clean install
If you have everyting configured correctly, you can watch dependency bundle downloads in the build log, and you'll see them coming from your mirror's local IP address instead of the remote repository.

Wrapping up and sources

The Dockerfile environment and the script to mirror the remote repositories can be found here. Good luck have fun.