This scenario show how a local file can be fed to SPADE which will then send it one to another instance of SPADE.
For any file to be handled by SPADE it must be associated with a registration. This tells SPADE how it should handle any file that is associated with it. The following command creates a file containing the registration that will be used in this scenario.
In order to demonstrate sending a file, you will need a new regiration to handle this behavior. The following commands creates such a registration.
mkdir -p ${HOME}/spade.zero/spade/registrations/local
cat > ${HOME}/spade.zero/spade/registrations/local/loopback.2.xml << EOF
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<registration>
<local_id>loopback.2</local_id>
<drop_box>
<location>
<directory>data/spade/dropbox</directory>
</location>
<pattern>.*.lem</pattern>
<mapping>gov.lbl.nest.spade.tour.LoopbackLocator</mapping>
</drop_box>
<warehouse>false</warehouse>
<outbound_transfer>Loopback to Self</outbound_transfer>
</registration>
EOF
If you compare this to the registration on the previous scenario, you will see a number of important differences. To begin with there are two additional element in the dropbox element.
pattern : this is a regular expression that uses the Java syntax in order to identify files that are semaphore files matching this registration. In the registration above any file in the ~/dropbox directory whose name has a .lem suffix will be matched with this registration.
mapping : this specifies the Java class that will be used to derived the bundle and data file names based on the semaphore file. In this case the LoopbackLocator derives the bundle name by removing the .lem suffix and the data file name by replacing that suffix with .data.
There are also two new elements that follow the dropbox one.
warehouse : this indicates whether the bundle should be stored in the local warehouse. The default value it true so it only needs to be specified when the bundle should not be stored locally.
outbound_transfer : this declares which, if any, outbound transfers will be used for bundles matching this registration. (This will be explained more detail in the next section.)
In the first scenario you did not have to worry about how SPADE was configured as it was set up to use the default setting, but now these need to be modified. Running the first scenario will have produced a default version of the configuration file. The following command will allow you to view it.
cat ${HOME}/spade.zero/spade/spade.xml
In it you can see four elements defined:
assembly : at this moment this simply declares the name of the SPADE instance as it will be seen in reports.
warehouse : this tells SPADE the directory that is the base directory of the local warehouse.
archive : this tells SPADE the directory that it can keep files while the await to be archived.
cache : this declares where SPADE will keep its cache, i.e. working files, along with a minumim amount of free space it need there in order to operate successfully.
In order to support outbound transfers this configuration needs to be updated, which you can do using your favorite editor. What you need to do is place the following element between the assembly and warehouse elements.
<outbound_transfer>
<name>Loopback to Self</name>
<description>Send bundles back to itself use the cp command</description>
<neighbor>SPADE@${DOCKER_CONTAINER_HOSTNAME}</neighbor>
<location>localhost:~/data/external/spade.zero/receiving/loopback</location>
<class>gov.lbl.nest.spade.services.impl.LocalhostTransfer</class>
</outbound_transfer>
This element declares that bundle in the SPADE can be transfered to another SPADE, or in this case back to itself to keep thing easy. The child elements of this element are as follows:
name : this is the name by which this transfer will be referred to in registrations.
description : this describe the purpose of this thransfer.
neighbor : this provides a local name by which the target SPADE be referred. By convention this value takes the form of <instance>@<host>. In this case, as it is its own target, this should be set to the same value as the assembly/name element. This will be the docker container’s and can be found with the following command:
docker exec -it tour_spade hostname
location : this declares a location where the other SPADE will look for the transferred files. The exact format of its value depends on the transfer class that is being used, but normally it is of the form <host>:<path>.
class : this specified the Java class that should be used to make the transfer. In this case the LocalhostTransfer class simply uses the local file system, but there are other transfer classes such as BbcpTransfer, GridFTPTransfer and SCPtransfer. The latter of these will be demonstrated later in this tour.
As this instance of SPADE is now referencing itself as spade.zero@localhost, we need to also update the name element in the assembly element at the top of this file so that everything agrees on which instance of SPADE is being discussed.
With a new registration and a modified configuration file SPADE must be redeployed in order for it to pick them up. This is done with the same command are before.
docker exec -it tour_spade \
cp wars/spade-${SPADE_VERSION}.war \
/opt/wildfly/standalone/deployments/spade.war
As noted in the previous section there need to be a common directory where the sending SPADE will put files and from whic the receiving SPADE will take then. For the case of this scenarion the following command creates such a directory.
mkdir -p ${HOME}/spade/shared/spade.zero/receiving/loopback
As before we can create a simple data file along with its associated semaphore file useing the following commands.
cat > ${HOME}/spade.zero/dropbox/tour.2.data << EOF
This data file should be shipped via the "Loopback to Self" outbound
transfer and then, upon reception at the other end, placed in the
warehouse.
EOF
touch ${HOME}/spade.zero/dropbox/tour.2.lem
Again, getting SPADE to pick up this bundle is a matter of repeating the local_scan command.
docker exec -it tour_spade bash -l -c "spade-cli local_scan"
In the other terminal showing the tour_spade container’s output you should see the bundle progress through the SPADE workflow, with it being shipped to its destination and as before ending with the finished task.
With the following command you can check, if you wish, that the files have been put in the correct receiving area.
find ${HOME}/spade/shared/spade.zero/receiving/loopback -name "*tour.2.*"
As the “other” SPADE have not yet handled this inbound transfer, you check that this transfer has not been verified yet by executing the following command.
docker exec -it tour_spade bash -l -c "spade-cli unverified"
The next scenario show how a SPADE instance handles in inbound transfer.