Configuring SPADE to transfer data not placed in the dropbox

This article explains how to configure a SPADE deployment so that it can pick up and transfer data that is not placed in the dropbox. This scenario can be used in situations where the data files are managed by a software system other than SPADE, for example an experiment's data acquisition system.

Pre-requisites

It is assumed that the SPADE project has been install and running as outlined here, and that the log output of the JBoss server can be seen in a second terminal.

Moreover, this scenario builds upon the "Loopback" scenario to handle the transfers so you should work through that one first.

Also, as in that scenario, the following environmental variables need to be set. They are show here being set to their standard values.

export WILDFLY_HOME=${HOME}/server/wildfly-9.0.2.Final
export SPADE_VERSION=3.0.1
export SPADE_HOME=${HOME}/nest-spade-war-${SPADE_VERSION}
export SPADE_WAR=${SPADE_HOME}/target/spade-${SPADE_VERSION}.war

Configuration

The simplest configuration of SPADE deployment expects to find data in the same directory as its associated semaphore file, with the same file name except for a simple change in suffix. After SPADE has copied the semaphore file and its data into its cache it deletes these from the drop box. However, there are situations where the data already exists elsewhere and making a copying of it in the dropbox gains you nothing. In those situations you can define a registration for the file stream that does not take its data from the dropbox, but rather takes it from some other location. You can install a registration that does just that by using the following command.

cp ${SPADE_HOME}/src/main/extras/examples/registration.4.xml ~/spade/registrations/local

The difference between this registration and the on used in the "Loopback" scenario, apart from changes to the local identity and semaphore suffix elements, is that the <dataSuffix> element of the registration has been replaced by a <locatorClass> element. That element specifies the class that implements the gov.lbl.nest.spade.registry.DataFileLocator interface and will be used to locate the data when its accompanying semaphore file has been detected by SPADE. In this case the ReadFromSemaphore class simply reads the data's location from the first line of the semaphore file.

Execution

Execution of this scenario is similar to that of the "Loopback" one with the one change being the creation of the data file in a directory other than the dropbox and putting that location, relative to the dropbox, into the semaphore file. The following commands show how to run the whole scenario.

${WILDFLY_HOME}/bin/jboss-cli.sh --connect --command="deploy \
    --name=spade.war ${SPADE_WAR}"
mkdir -p ~/spade/other/directory/scenario/external
cat > ~/spade/other/directory/scenario/external/mock.7.data << EOF
put some junk in here
EOF
mkdir -p ~/spade/dropbox/scenario/external
echo "../../../other/directory/scenario/external/mock.7.data" > ~/spade/dropbox/scenario/external/mock.7.sem
${SPADE_HOME}/src/main/python/spade-cli local_scan
# Wait for the "finisher" for this file to stop, e.g. 5 seconds
sleep 5
${SPADE_HOME}/src/main/python/spade-cli inbound_scan
# Wait for the "finisher" for this file to stop, e.g. 5 seconds
sleep 5
${SPADE_HOME}/src/main/python/spade-cli send_confirmations
find ~/spade/warehouse -name "mock.7.*"
ls -l ~/spade/other/directory/scenario/external/mock.7.data

The last command above is included to show that when a DataFileLocator instance is used to map the semaphore file onto its data then, by default, the data is not deleted by SPADE. This behavior can be modified. (See the Registration reference article for details.)

Cleanup

Having successfully completed this scenario you should now undeploy the application using the following command.

${WILDFLY_HOME}/bin/jboss-cli.sh --connect --command="undeploy spade.war"