This scenario show how to set up both shipping and receiving SPADE instances so that they can use scp to transfer their file.
In order to avoid using passwords when doing scp transfers, the first item of business is to set up a ssh key pair on the shipping node that will be used to authenticate the sending SPADE to the receiving one. The following commands create such a pair. (Note This pair is set up without a passphrase as executing up scp transfers using a passphrase is beyond the scope of this tour.)
Note: If hostname -f does ot return the full qualified domain name of the host machine you should set this valu e by hand.
HOST_FQDN=$(hostname -f)
HOST_USER=${USER}
# Execute the next line as a separate command! i.e. with a CR after it
# before executing the `ssh-keygen` line.
TOUR_HOST=$(echo "$(docker exec -i -t tour_spade hostname)" | tr -d '\r' )
ssh-keygen -b 4096 -N '' -C "SCP Transfers from ${TOUR_HOST}" \
-f ${SPADE_ZERO_SECRETS}/${TOUR_HOST}_scp.transfer
The shipping node also need to know where to find these files when doing the transfer, this is done by an appropriate config file. The following commands create just such am appropriate file.
cat >> ${HOME}/spade.zero/spade/ssh_config << EOF
Host spade.zero.scp
IdentityFile /opt/jboss/services/run/secrets/${TOUR_HOST}_scp.transfer
HostName ${HOST_FQDN}
HostKeyAlias spade.zero
User ${HOST_USER}
UserKnownHostsFile /opt/jboss/services/spade/ssh_known_hosts
EOF
chmod 644 ${HOME}/spade.zero/spade/ssh_config
Finally, the shipping node needs to authenticate that it is connected to the correct destination. This is done by putting the receiving nodes signature in the the ssh_known_hosts file. THe following commands do that for this tour.
export EXPRESSION="s/${HOST_FQDN}/spade.zero/g"
ssh-keyscan -t ecdsa-sha2-nistp256 ${HOST_FQDN} | sed -e "${EXPRESSION}" > tmp.txt
cat >> spade.zero/spade/ssh_known_hosts < tmp.txt
rm tmp.txt
In many cases the receiving node may not be the same node as the one one which the receiving SPADE is executing. One reason for this is that may be that the receiving node is a special Data Transfer Node (DTN), another reason maybe that the SPADE node does not support inbound scp connections. In either case, the receiving node and the SPADE node will normally share a disk onto which the received files will be placed.
In this tour, the host node will be acting as the receiving node and the SPADE’s docker container does not support inbound scp connections. These two “nodes” are sharing the following diretory on thie host node:
${HOME}/spade/shared/spade.zero/receiving/
In order for the receiving node to accept connections using the ssh credentials created above the following commands need to be executed.
read PUBLIC_KEY < ${SPADE_ZERO_SECRETS}/${TOUR_HOST}_scp.transfer.pub
mkdir -p ${HOME}/.ssh
chmod 700 ${HOME}/.ssh
echo "${PUBLIC_KEY}" >> ${HOME}/.ssh/authorized_keys
chmod 644 ${HOME}/.ssh/authorized_keys
unset PUBLIC_KEY
Before SPADE uses the ssh connection for transfers it is advisable to run it once by hand to maker sure all the pieces are correctly in place. This can be done by running the following commands on the shipping SPADE’s node.
docker exec -i -t tour_spade scp -F services/spade/ssh_config \
SPADE_VERSION spade.zero.scp:spade/shared/spade.zero/receiving/loopback/
ls -l spade/shared/spade.zero/receiving/loopback/SPADE_VERSION
docker exec -i -t tour_spade \
ls -l /opt/jboss/data/spade/receiving/loopback/
If that works successfully then the following will clean up the now useless file.
rm ${HOME}/spade/shared/spade.zero/receiving/loopback/SPADE_VERSION
We are now ready to configure SPADE to use this transfer channel.
In order to use this new transfer protocol we need to have a registration that uses it. Rather than editing an existing one, which is perfectly acceptable, for the purposes of this tour we’ll create a new one. The following commands create that new regisration.
mkdir -p ${HOME}/spade.zero/spade/registrations/local
cat > ${HOME}/spade.zero/spade/registrations/local/scp_transfer.5.xml << EOF
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<registration>
<local_id>scp.transfer.5</local_id>
<drop_box>
<location>
<directory>data/spade/dropbox</directory>
</location>
<pattern>.*.scp</pattern>
<mapping>gov.lbl.nest.spade.tour.SCPTransferLocator</mapping>
</drop_box>
<warehouse>false</warehouse>
<outbound_transfer>Loopback using SCP</outbound_transfer>
</registration>
EOF
In order to get an outbound transfer to use the scp channel you need to specifiy the SCPTransfer class as the class for the transfer. Therefore the following element should be added to spade.xml between the existing outboundTransfer element and the inboundTransfer one.
<outbound_transfer>
<name>Loopback using SCP</name>
<description>Send bundles back to itself use the scp command</description>
<neighbor>SPADE@${DOCKER_CONTAINER_HOSTNAME}</neighbor>
<location>spade.zero.scp:spade/shared/spade.zero/receiving/loopback</location>
<class>gov.lbl.nest.spade.services.impl.SCPTransfer</class>
</outbound_transfer>
Note All outbound_transfer declarations for a single neighbor may deliver to that same location as the receiving SPADE does not care which transfer protocol was used. However it does mean that for any given registration only one outbound_transfer per neighbor must be declared.
Once the configuration has been updated and the registration exists you will need to redeploy SPADE in order for it to pick up both of these changes. This is done with the same command are before.
docker exec -it tour_spade \
cp wars/spade-${SPADE_VERSION}.war \
/opt/wildfly/standalone/deployments/spade.war
You are now ready to use the scp transfer channel. As before this is a matter of creating data and semaphore files and telling SPADE to handle this bundle. The following commands do just that.
cat > ${HOME}/spade.zero/dropbox/tour.7.data << EOF
This data file should be shipped via the "Loopback using SCP" outbound
transfer.
EOF
cat > ${HOME}/spade.zero/dropbox/tour.7.tmp << EOF
<path_metadata>
<path>/opt/jboss/data/external/spade.zero/tour/local/structure/tour.7.data</path>
</path_metadata>
EOF
mv ${HOME}/spade.zero/dropbox/tour.7.tmp ${HOME}/spade.zero/dropbox/tour.7.scp
docker exec -it tour_spade /bin/bash -l -c "spade-cli local_scan"
docker exec -it tour_spade /bin/bash -l -c "spade-cli inbound_scan"
docker exec -it tour_spade /bin/bash -l -c "spade-cli delivery_scan"
Once again you can check that the file made it into the warehouse with the following command.
find ${HOME}/spade.zero/warehouse -name "*tour.7.*"
And that is has been successfully verified with this
docker exec -it tour_spade /bin/bash -l -c "spade-cli verified"