What's a dropbox?

Chances are, if you have ever connected to a shared computer, you have used some form of a dropbox. File servers, FTP servers, email servers even occasionally web servers can all serve as dropbox servers. If a coworker requests that you save a report to particular location on a server, you and your coworker are using a similar 'dropbox' mechanism and a simple form of messaging. DropboxMQ uses a very specific form of dropbox and a very specific protocol for communicating with other clients through shared dropboxes. Each dropbox is a unique JMS destination.

The Root

The DropboxMQ Dropbox

A Dropbox Message Producer

Dropbox Message Consumers

Message Filenames

The Root

DropboxMQ groups a set of dropboxes into a single directory called the root. More than one root can be configured, but any given DropboxMQ ConnectionFactory can only connect to one root at any given time. Each individual dropbox has its own directory directly in the root directory. The name of dropbox (which is the name of the Queue or Topic) is the name of the dropbox's directory.

.../root/
    queue1/...
    queue2/...
    topic-root/
        topic1/...
        topic2/...

The DropboxMQ Dropbox

Each queue dropbox has the same directory structure shown below. On all mainsteam operating systems and network operating systems, moving a file from one directory to another is an atomic operation (provided both directories are on the same logical disk). DropboxMQ relies on this atomicity to dole out work to message consumers and to resolve contention between message consumers.

.../root/
    queue1/
        incoming/
            working/...
            target/...
            processing/...
            processed/...
            routing/...
            expired/...
            error/...

Topic dropboxes have a slightly different structure to allow for multiple subscribers, but they work with the same basic principal:

.../root/
    topic-root/
        topic1
            .incoming/
                working/...
                target/...
                processing/...
                processed/...
                routing/...
                expired/...
                error/...
            .subscriptions/
                sub1.connection1/
                    working/...
                    target/...
                    processing/...
                    processed/...
                    routing/...
                    expired/...
                    error/...
                sub2.connection2/
                    working/...
                    target/...
                    processing/...
                    processed/...
                    routing/...
                    expired/...
                    error/...

A Dropbox Message Producer

A dropbox message producer creates a unique file in the working directory of the dropbox it is sending to and writes the message contents to the file. While the message file remains in the working directory, the message has not actually been sent and the producer can take as long as it needs to complete the message.

Once the producer is done writing the file and ready to commit the message, the producer moves the file to the target directory. The producer must move (not copy) the file to assure that a consumer doesn't attempt to read a message before it is completely written. Once the message file is in the target directory, the producer has relinquished control of the message and it is consumable.

NOTE: It is very important that producers of any type (shell scripts, etc.) follow the protocol of first writing the message to the working directory, then moving (not copying) the files to the target directory. Even when sending very small messages, it is possible a message consumer could attempt to read a message before it is complete if it is created directly in the target directory.

Dropbox Message Consumers

Several message consumers may be scanning the target at the same time. When a consumer finds a message file, it attempts to move the file to the processing directory. Only one consumer will succeed. The consumers that fail to move the message file (or fail to notice the message file) quietly go back to scanning the target directory for the next message file.

The single consumer that succeeded to move the file then begins the process of consuming the message. If the message is consumed and acknowledged, the consumer moves the message file to the processed directory. If the message is not successfully consumed and is recovered, the consumer moves the message file to the error directory.

Message Filenames

DropboxMQ encodes all message metadata (priority, message identifier, reply-to, etc.) in the message's filename. This allows the contents of the message file to contain just the message body which integrates more easily with dropbox clients that don't use DropboxMQ. Message producers are only required to create message files with unique names.

A message producing client that doesn't use DropboxMQ but wishes to have tighter integration with DropboxMQ, can choose to create message filenames that are compatible with the default DropboxMQ transcoder. By default, DropboxMQ message filenames are broken into several dot (.) separated fields. Trailing dots are required if the remaining fields are undefined. Of the following fields, only the first three (JMS Priority, JMS Message Id and Message Class) are required to permit integration with a DropboxMQ consumer.

Example 1: All fields defined
4.1140429201295000-9262574723.T.1140429211295.corr1283.TestQueue1.XYZType.prop1S=hello
Example 2: Minimum fields defined
4.1140429201295000-9262574723.T

Each field is described below:

Position Name Default

1. JMS Priority Message.DEFAULT_PRIORITY (4)
DropboxMQ supports priority values from -231 to 231-1.

2. JMS Message Id None
If no message identifier can be parsed, the entire message filename is used for the message identifier. If DropboxMQ is used to produce a message, JMS Timestamp is also encoded into the JMS Message Id.

3. Message Class 'B' (BytesMessage)
'M' = Message
'B' = BytesMessage
'S' = StreamMessage
'P' = MapMessage
'O' = ObjectMessage
'T' = TextMessage

4. JMS Expiration No expiration (0)
The date in milliseconds past midnight, January 1, 1970 UTC at which time the message will expire.

5. JMS Correlation Id None, no correlation id

6. JMS Reply To None, no reply-to

7. JMS Type None, no type

8. Properties None, no properties defined
Properties are define in name-value pairs with each pair separated by an ampersand (&). Names and values are separated by an equals sign (=). Names and values are URL encoded to prevent odd characters from causing problems with the parser. The last character of the name determines the property type. Valid types are listed below.
'B' = boolean
'Y' = byte
'H' = short
'I' = int
'L' = long
'F' = float
'D' = double
'S' = String