Customizing the Placement of Files in the Warehouse

Overview

This scenario explains how to configure a SPADE deployment in order to customize the placement of files in the warehouse rather than using the default placement.

Updating the SPADE configuration

The default placement of a file in the warehouse if governed by the Sha1Placement implementation of the the PlacingPolicy interface as this does not require any particular metadata, just the name of the file in which SPADE holds the metadata. However this can be changed by specifying the PlacementPolicy the placer activity should use. (In order to do this you must have first enabled the custom metadata in this scenario.

To get the placing activity to use a different PlacingPolicy you need to explicitly declare this activity and its setting in the spade.xml file. You do that by adding the following to that file using your preferred editor and placing it inside the assembly element underneath the name element.

<activity>
    <name>placer</name>
    <policy>
        <class>gov.lbl.nest.spade.policy.impl.PathPlacement</class>
    </policy>
</activity>

Mapping Mount Points

The PathPlacement policy is designed to mirror the file layout on and “upstream” SPADE. However as not all SPADEs have the same mount points so the policy needs to be able to map the remote mount point to local ones. This is done by a mount-point.properties file in the configuration directory. The following sets this up for this example:

cat > ${HOME}/spade.zero/spade/mount-point.properties << EOF
# Maps original mount points to locations in the warehouse
/opt/jboss/data/external/spade.zero/tour=customized/placement
EOF

You are now ready to redeploy SPADE:

docker exec -it tour_spade \
    cp wars/spade-${SPADE_VERSION}.war \
    /opt/wildfly/standalone/deployments/spade.war

Creating data and semaphore files

The creation and the data and semaphore files follow the same approach as that in the Customizing Metadata scenario, but with the metadata containing a path related to where the data should be placed in the warehouse.

cat > ${HOME}/spade.zero/dropbox/tour.5.data << EOF
This data file should use custom placement
EOF

cat > ${HOME}/spade.zero/dropbox/tour.5.tmp << EOF
<path_metadata>
    <path>/opt/jboss/data/external/spade.zero/tour/local/structure/tour.5.data</path>
</path_metadata>
EOF

mv ${HOME}/spade.zero/dropbox/tour.5.tmp ${HOME}/spade.zero/dropbox/tour.5.sem

mv ${HOME}/spade.zero/dropbox/tour.5.tmp ${HOME}/spade.zero/dropbox/tour.5.sem

docker exec -it tour_spade bash -l -c "spade-cli local_scan"

Seeing the File in the Warehouse

As with the Local Warehousing scenario, you can see where the file are in the warehouse with the following command.

find ${HOME}/spade.zero/warehouse -name "*tour.5.*"

You can see that the mount point has been mapped and the rest of the path below that has been used to complete the placement within the warehouse.

NEXT STEP