Configuring SPADE to do loopback transfers

This article explains how to configure a SPADE deployment so that data dropped into the local dropbox is transferred back to the same deployment where it is then placed in the warehouse. This is not so much a production scenario as chance to get used to transfers without needing to establish a second SPADE deployment.

Pre-requisites

It is assumed that the nest-spade-war project has been install and running as outlined here, and that the log output of the JBoss server can be seen in a second terminal.

It is also helpful if you have at least read the Local Warehouse scenario in order to familiarize yourself with the concepts discussed there as they will be re-used here. As in that scenario, the following environmental variables need to be set. They are show here being set to their standard values.

export JBOSS_HOME=${HOME}/server/jboss-as-7.1.1.Final
export SPADE_VERSION=2.2.0
export SPADE_WAR_HOME=${HOME}/nest-spade-war-${SPADE_VERSION}

Configuration

In order to set up SPADE to ship and/or received transfers you will need to include outboundTransfer and inboundTransfer elements, respectively, in the spade.xml file. You will also need to provide an identity for the SPADE deployment. This identity takes the form of an email address in order to be globally unique. While this scenario does not use the identity to access email other scenarios do, therefore you should choose a suitable address before continuing.

For this scenario, the following XML needs to be added to the spade.xml after the existing warehouse element, with local_user@local_mail_host replaced by the address you've selected for this deployment.

    <outboundTransfer>
        <name>Loopback to Self</name>
        <neighbor>local_user@local_mail_host</neighbor>
        <location>localhost:~/spade/receiving/loopback</location>
        <shipper>loopback</shipper>
        <class>gov.lbl.nest.spade.services.impl.LocalhostTransfer</class>
    </outboundTransfer>
    <inboundTransfer>
        <name>Loopback from Self</name>
        <neighbor>local_user@local_mail_host</neighbor>
        <location>~/spade/receiving/loopback</location>
    </inboundTransfer>
    <neighborhood>
        <home>
          <neighbor>local_user@local_mail_host</neighbor>
          <verifyUrl>http://localhost:8080/spade/local/command/verify</verifyUrl>
        </home>
    </neighborhood>

You can then set up this scenario's registration by using the following commands.

mkdir -p ~/spade/registrations/local
cp ${SPADE_WAR_HOME}/src/main/extras/examples/registration.2.xml ~/spade/registrations/local

The difference between this registration and the on used in the Local Warehouse scenario, apart from changes to the local identity and semaphore suffix elements, is the addition of warehouse and outboundTransfers elements. The first of these, which is set to false, means that the file will not be stored in the local warehouse before it is transferred. The second one means that files from this files stream will be transferred using the Loopback to Self outbound transfer that was defined earlier in the spade.xml file.

Execution

The following deploys the SPADE application so that it can read the new registration created in the previous section.

${JBOSS_HOME}/bin/jboss-cli.sh --connect --command="deploy \
    --name=spade.war ${SPADE_WAR_HOME}/target/spade-${SPADE_VERSION}.war"

Now that SPADE is running you can create and submit some test data using the following commands (remembering to use the correct semaphore suffix!).

mkdir -p ~/spade/dropbox/mock
cat > ~/spade/dropbox/mock/mock.2.data << EOF
put some junk in here
EOF
touch ~/spade/dropbox/mock/mock.2.lem

If you do not want you can force SPADE to poll the dropbox by executing the following command.

${SPADE_WAR_HOME}/src/main/bash/localScan

As in other scenarios, you should now see in SPADE's log file the data file being processed and transferred.

As with monitoring the local dropbox, SPADE polls its inbound dropboxes at a set interval, by default 5 minutes. You can force it to poll immediately using the following command.

${SPADE_WAR_HOME}/src/main/bash/inboundScan

Again SPADE's log file should show the inbound file being processed and, this time, being placed in the warehouse. You can check for its presence in the warehouse using the following command.

find ~/spade/warehouse -name "mock.2.*"

The final part of the transfer, after SPADE has successfully received the transfer file, is to send a checksum back to the original SPADE in order to confirm that the file has not been corrupted. If it has been corrupted, the original SPADE will transfer the file again. To force the transfer confirmation to be sent, rather than waiting for the default 10 minutes, you can use the following command.

${SPADE_WAR_HOME}/src/main/bash/sendConfirmations

The SPADE log file should show the cached shipped file being deleted as the transfer is successful. The entire transfer operation is now complete.

Cleanup

Having successfully completed this scenario you should now undeploy the application using the following command.

${JBOSS_HOME}/bin/jboss-cli.sh --connect --command="undeploy spade.war"