Visualizing IBM SAN Volume Controller FlashCopy MappingsDanRumneyDCRJanuary 22, 2009 With the introduction of Multiple-Target FlashCopy and Cascaded FlashCopy to IBMSystem StorageSAN Volume Controller (SVC) , it can become difficult to keep track of
which Virtual Disks are mapped to which, and how FlashCopy Mappings
(FCM) depend upon one another. While submitting svcinfocommands or using
the SVC Console can provide this information in a textual format, there are times when a
diagram provides all the information you need in an easy to understand format. This paper outlines a method for generating diagrams that link VDisks and FlashCopy
Mappings. It also serves as a worked example of automation on the SVC Command Line. This paper
assumes familiarity with SVC and the FlashCopy functionality. SVC FlashCopy Mappings When SVC was introduced in 2003 it included a number of Copy Services: FlashCopy and
Remote Copy. FlashCopy is a Point-In-Time Copy Service, whereby the contents of a
Source Virtual Disk (Source) is copied to a Target Virtual
Disk (Target), such that the Target is an exact copy of the
Source, at that point in time. The relationship between a Source and a Target is called a
FlashCopy Mapping (FCM). The original implementation was such that a Source could only ever
have one active Target. In addition to this, a Target of one FCM could not be the Source of
another FCM. A number of FCMs can be gathered together into a FlashCopy Consistency
Group (FCG) and managed as a single entity, to ensure that all Target VDisks in
the FCG represent the exact same point in time. With the release of SVC 4.2.1, new types of FlashCopy arrangements can be created. A
single VDisk can act as the Source to multiple Targets. In addition to this, a VDisk which is
acting as the Target of one FCM can also act as the Source of a different FCM. For more
details on Copy Services in SVC, see the Redbook SVC 4.2.1 Advanced Copy
ServicesSG24-7574-00. In a complex environment, the interactions between FCMs and VDisks can get quite
involved. Dependencies between FCMs stem from internal data structures within the cluster
rather than the logical connections between VDisks. All of these dependencies can be
discovered by submitting the appropriate svcinfo commands, but the
information is presented in a purely textual way; this provides no insight into the
interaction between cluster objects. shows how this output
looks.Sample output from SVC commands, viewing FCM interactionssvcinfo lsfcmap -delim :
id:name:source_vdisk_id:source_vdisk_name:target_vdisk_id:target_vdisk_name:
group_id:group_name:status:progress:copy_rate:clean_progress:incremental
0:fcmap0:1039:vdisk1039:1040:vdisk1040:::idle_or_copied:100:93:100:on
1:fcmap1:1041:vdisk1041:1042:vdisk1042:::idle_or_copied:100:30:100:off
2:fcmap2:1043:vdisk1043:1044:vdisk1044:::idle_or_copied:100:88:100:on
3:fcmap3:1045:vdisk1045:1046:vdisk1046:::idle_or_copied:100:36:100:off
4:fcmap4:1046:vdisk1046:1047:vdisk1047:::idle_or_copied:100:96:100:on
IBM_2145:cluster_name:admin>
IBM_2145:cluster_name:admin>svcinfo lsfcmapdependentmaps -delim : 2
fc_id:fc_name
1:fcmap1
3:fcmap3]]> The output from the svcinfo command does not lend itself to a swift
overview of the cluster state. We look to the DOT language to generate a graphical
representation of this information. The DOT Language The DOT Language is a language used to describe directed and undirected graphs. Once
written, the DOT can then be processed by an appropriate program, to render the graph on
screen. A directed graph consists of nodes and
edges. A node is a graphical shape which may or may not contain some
text. The nodes are interconnected by lines, called edges. In a directed graph, the edges have
arrows at one or both ends. shows a sample directed graph.
Ellipses a,b, c and
d are all nodes. The DOT language is quite straightforward; for instance, the DOT required to generate
can be seen in . The nodes are
represented by the various letters. In each line in the example, you can see the 'arrow'
notation which indicates a directed edge, joining two nodes.DOT language for generating a directed graphdigraph EXAMPLE
{
a -> c;
b -> c;
c -> d;
d -> a;
} A DOT file only describes the graph. You must pass this file to a
rendering program in order to generate a graphical representation. Once such program is
dot, which is part of the
Graphviz package.
Once installed, the invocation shown in will
generate a GIF file rendering of the sample directed graph described in . Invocation to generate graphic from DOT language (Windows)dot -T gif -o sampleDigraph.gif -K dot -v sampleDiGraph.dot The capabilities of the DOT language lend themselves directly to the challenge of
visualizing FlashCopy Mapping and VDisk relationships. The challenge is to generate the DOT
necessary to generate our required visualization. In order to do this, we can use SVC Command
Line Scripting. SVC Command Line ScriptingThe SVC Command Line Interface is based on a restricted Bash Shell. This provides us with
a double-edged opportunity. On one hand, the Bash Shell means that we have the opportunity to
execute scripts while logged in to the SVC Command Line. On the other hand, the restricted
aspect strongly limits what we can do and results in the need for some imaginative scripting. The Bash Shell includes a number of 'built-in' commands that can be used, such as:
ifwhileforreadechoThe restrictions on the Shell mean that there is no access to scripting standards such as
sed, awk and grep. In addition, IO
cannot be redirected to files. However, command output can be redirected
to other commands, via pipes. contains the script that we'll be discussing for the remainder of
this paper. This script creates a CLI function which, when executed, generates DOT language
which describes the connections between a set of VDisks and FCMs. The CLI function has one
parameter, which is the id of one of the cluster's VDisks. Given a VDisk ID, this command
follows the procedure below.Search for any FCMs which have the provided VDisk as a Source or Target For any FCMs found, identify the counterpart VDisks and repeat step 1 for each of the new VDisksFor all of the FCMs found, search for all dependant FCMsGenerate the DOT language, describing the VDisk and FCM interactions shows example output from this function. When rendered
by dot, the graphic shown in Sample output from scriptmakeFCMapTree 1212
digraph F {
1210 [style=filled,fillcolor=green]
514 [shape=box,height=0.4,width=0.4,fillcolor=green,style=filled]
1210 -> 514
514 -> 1212
515 [shape=box,height=0.4,width=0.4,fillcolor=green,style=filled]
1210 -> 515
515 -> 1213
1212 [style=filled,fillcolor=green]
1213 [style=filled,fillcolor=green]
1211 [style=filled,fillcolor=green]
512 [shape=box,height=0.4,width=0.4,fillcolor=green,style=filled]
1211 -> 512
512 -> 1209
513 [shape=box,height=0.4,width=0.4,fillcolor=green,style=filled]
1211 -> 513
513 -> 1210
1209 [style=filled,fillcolor=green]
}]]> The ellipses represent VDisks. The rectangles represent FlashCopy Mappings. The colours
of the shapes represent the state of the FlashCopy Mappings and VDisks. In one glance, the
interactions between VDisks and FlashCopy Mappings are clear and any issues are immediately
obvious.
In the next section, we will go through this script line by line and explain what is being done. The techniques used in this script can be used in scripts of your own to perform whatever actions you require.
If you prefer, you can skip to to learn how to use the script.
Script AnalysisWhen the script in is executed, it creates a new function called
makeFCMapTree in the active SVC CLI session. The script itself has
no output. Once the function has been created, it is invoked by
providing the function name and a VDisk ID. It generates output in the DOT language which can
be captured and saved for later rendering. In this section, we analysis the script section by
section and explain its function.Lines 1-18The first 18 lines are concerned with setting up variables for use during the main
loop.Lines 1-15 The fifteen first lines simply start the function and set up some variables assigning
colours to FCM and VDisk states. These can be changed to suit your needs, with the
requirement that all 10 states are present and the colours that are selected are part of
the DOT language. shows the colours supported by DOT.Line 16This line outputs the first line of DOT, which indicates that we'll be describing a
directed graph. We call it F for FlashCopy, but the name is entirely arbitrary (within the
constraints of the DOT language) Line 17In this line, we create an array called $possSrcs. This array will be
treated as a FIFO stack. It will contain a list of VDisks which are possibly acting as the Sources
of an FCM. At this point in the execution, we add the VDisk ID that was provided as the
sole parameter to the function. In subsequent passes through the main loop, new VDisk IDs
may be added, as required. Line 18Here, we create an empty array which will keep track of all VDisk IDs which
have been processed. This is to ensure that the script processes each VDisk ID once and
once only. Lines 19-26These lines start the main loop and gather information about the next VDisk ID in the
$possSrcs stack.Line 19This starts the main loop of this script. The loop will execute as long as there are
VDisk IDs in $possSrcs. This loop ends on line 56.Lines 20-21Lines 20-21 perform the pop operation of a stack, removing the
first element from $possSrcs and assigning it to
$currSrc.Lines 22-26 Lines 22-26 show a technique that will be repeated a number of times in this script.
This technique is to run an svcinfo command and execute a series of
commands based on each line of output from the svcinfo command. The technique has the following form:
svcinfo xxxx | while read var1 var2 var3 rest; do
some commands using $var1, $var2, etc
done
This technique will take each line of output from the svcinfo command and
pass it to the read built-in command. The read command works in the
following way: readname The read command takes a line from STDIN and splits it into words
separated according to the Internal Field Separator (IFS). By default, IFS is
set to whitespace. The first word is assigned to the first
name, the second to the second name and so on. Any leftover
words are assigned to the final name with the intervening IFS included.In the technique shown above, the variables $var1, $var2, etc can now be used by commands inside the
while loop. The loop will repeat once per line of output from the
svcinfo command. The variable $rest is needed to
capture any remaining values at the end of the line of output. The command svcinfo lsvdisk -nohdr -filtervalue id=$currSrc will
generate zero or one line of output. (depending on whether the value in
$currSrc is an actual VDisk ID). Line 24 This line acts to 'dereference' the VDisk status. The code $(eval echo
\$$vStatus) will take the value stored in $vStatus and
treat it as the name of a variable and try to find the value stored in that. For instance,
if $vStatus is equal to 'preparing' then this line will look for the
value found in $preparing and, in this case, return 'yellow'. In this
way, we can assign colours to VDisk states.Line 25 This generates lines of DOT which define a node in the graph. In this instance a
VDisk node is generated. The VDisk node is an ellipse, filled in a colour representing the
VDisk's state. Lines 27-30 These two lines generate an array of FCM Target VDisk IDs and mark the current VDisk ID
(held in $currentSrc) as having been processed. Lines 27-29 Lines 27-29 create a new array that holds all of the FCMs which have the current
VDisk (represented by $currSrc) as the Source VDisk. This array is
called $newTgts. These three lines shows a method of generating an
array directly from the output of an svcinfo command. In this
particular instance, each line of svcinfo output generates 3 array
elements: $newTgts[L]FlashCopy Mapping ID$newTgts[L+1]Target VDisk ID$newTgts[L+2] FlashCopy Mapping status where L increases by one per line of
svcinfo output.Line 30 Line 30 keeps track of the fact that we have now processed the VDisk ID(as an
FCM Source... it may appear later as an FCM Target). Lines 31-46These lines process the FCMs that were placed into $newTgts and
generate the appropriate DOT to represent them.Line 31 Lines 31 creates the loop to process $newTgts. It creates an index
variable which is incremented by 3 for each pass (since 3 elements in an array represent one
FC Mapping). Lines 32-35Lines 32 to 34 simply collect the relevant array elements in to clearer variable
names. Line 35 decodes the FC Mapping status into a colour, much like line 24. Lines 36-39Lines 36-39 generate DOT language; lines 36 and 37 creates an FC Mapping node, which
is a square filled with a colour that represents its state. Lines 38 and 39 generate the edges that link the FC Mapping node with its Source and
Target VDisks. Lines 40-42Lines 40-42 look at the FCM Target and determine whether or not it has been processed as
a Source. If it has not, it is added to our list of possible Sources: $possSrcs. The
code in line 41 acts to place this new VDisk ID at the end of the stack. Lines 43-45Lines 43-45 look to see if there are any dependencies between this FCM and other FCMs.
If there are, a new edge is generated to indicate this. Lines 48-55 Lines 48-55 perform a similar task to line 27 and the loop that follows it.Line 48Line 48 generates an array of FCM Source VDisk IDs, which have
$currentSrc as their Target. This array is called
$newSrcs. Instead of generating any DOT in the loop, however, this
loop simply adds the indicated VDisk IDs to the $possSrcs array, if they
have not previously been processed. Lines 56-58 All that remains at this point is to complete the loops, close out the DOT language
with a curly bracket and the function is complete. Using the script There are two simple ways to use the script. It can be executed directly from an
interactive CLI session, or it can be used as part of a batch session.Interactive sessionAdding the makeFCTreeMap function to an interactive session is very
straightforward. Simply copy the full text of the script into the clipboard and then paste
it into the terminal. Once the script has executed, the makeFCTreeMap
function will be available to you for the remainder of that CLI session. Once you've passed a VDisk ID to the function, you will need to copy the output from the CLI
session and place it into a dot file for rendering. Batch session Adding the makeFCTreeMap function to a batch session depends on
your SSH client. Here, we will discuss PuTTY for the Windows
operating system and ssh for Linux or AIX. Whichever operating system you use, the output from the script will be returned to the
STDOUT stream on your local system. You can redirect this output to a dot file, and then
pass it to your rendering application. Required script changes When a script is submitted to the SVC Cluster in this way, STDIN is replaced by the
contents of the script and executed as if it was typed in manually. Once the end of the
file is reached, control returns to the local command line and not
the SVC command line. Since the normal function of the makeFCTreeMap script is to create a
new function in the CLI session (and nothing more), the following modifications are needed
to generate output: Delete lines 1 and 2Delete line 58Replace $1 in line 17 with the ID of the VDisk that you're
interested in.
Once these changes have been made, the resulting script_file should be submitted to the cluster using one of the methods shown in the next subsections.
PuTTY The plink command comes as part of the
PuTTY application. shows the command
to use to submit a script to an SVC cluster. Submitting script to an SVC cluster using plink
plink -l admin -m script_file -i private_key_file cluster_name
script_fileThe file containing the makeFCTreeMap scriptprivate_key_fileAn SSH private key which corresponds to a public key that has been uploaded to
the SVC cluster in questioncluster_nameThe IP address or DNS nameSSH The ssh command comes with most (if not all) *nix operating
systems. shows the invocation required to submit a script to an SVC
cluster using ssh. Submitting script to an SVC cluster using ssh
ssh -i private_key_file -T admin@cluster_name < script_file
script_fileThe file containing the makeFCTreeMap scriptprivate_key_fileAn SSH private key which corresponds to a public key that has been uploaded to
the SVC cluster in questioncluster_nameThe IP address or DNS namePossible Improvements
The script in functions correctly for all possible VDisk IDs, including ones that are not present on the cluster. However, there are some interesting changes that could be made to enhance the script. These
are offered as suggestions and are left to the reader to implement:
Possible improvements to scriptHandle Multiple VDisk IDsThe current version of the makeFCTreeMap only accepts a single VDisk ID. It would be a fairly simple task to change the script to allow the function to accept any number of VDisk IDs. It would be important
to check for duplicate VDisk IDs appearing in the $possSrcs stack.Handle FlashCopy Mapping IDsExpanding this script to support FCM IDs instead of VDisk IDs is the straightforward task of taking the FCM ID, determining the Source VDisk's ID and placing this into the $possSrcs stack and then proceeding
as before. The challenge lies in making the one script support VDisk IDs and FCM IDs.Handle FlashCopy Consistency Group IDsHandling FlashCopy Consistency Groups is the natural combination of handling multiple VDisk IDs and handling FlashCopy Mappings, since an FCG is simply a group of FCMs. The following code will turn an FCG id into an array of
FCM ids: Graph generating script The script below has been formatted so that it will fit onto the page. As a result, line
continuation operators have been used on lines 22, 27, 28, 36 and 48. fc$fcm";
echo "fc$fcm -> $tgt";
if [ "${processed[$tgt]}" != "y" ]; then
possSrcs=(${possSrcs[@]} $tgt);
fi;
svcinfo lsfcmapdependentmaps -nohdr $fcm | while read fcId fcName; do
echo "$fcm -> $fcId [style=dotted]"
done
done;
newSrcs=(`svcinfo lsfcmap -nohdr -filtervalue target_vdisk_id=$currSrc| \
while read id name srcId junk; do echo "$srcId "; done`);
for src in ${newSrcs[@]};
do
if [ "${processed[$src]}" != "y" ]; then
possSrcs=(${possSrcs[@]} $src);
fi;
done;
done;
echo "}";
}]]>Valid colours in the DOT languageThe following are acceptable colours in the DOT language: This list can also be found at Graphviz Color Namesaliceblueantiquewhiteantiquewhite1antiquewhite2antiquewhite3antiquewhite4aquamarineaquamarine1aquamarine2aquamarine3aquamarine4azureazure1azure2azure3azure4beigebisquebisque1bisque2bisque3bisque4blackblanchedalmondblueblue1blue2blue3blue4bluevioletbrownbrown1brown2brown3brown4burlywoodburlywood1burlywood2burlywood3burlywood4cadetbluecadetblue1cadetblue2cadetblue3cadetblue4chartreusechartreuse1chartreuse2chartreuse3chartreuse4chocolatechocolate1chocolate2chocolate3chocolate4coralcoral1coral2coral3coral4cornflowerbluecornsilkcornsilk1cornsilk2cornsilk3cornsilk4crimsoncyancyan1cyan2cyan3cyan4darkgoldenroddarkgoldenrod1darkgoldenrod2darkgoldenrod3darkgoldenrod4darkgreendarkkhakidarkolivegreendarkolivegreen1darkolivegreen2darkolivegreen3darkolivegreen4darkorangedarkorange1darkorange2darkorange3darkorange4darkorchiddarkorchid1darkorchid2darkorchid3darkorchid4darksalmondarkseagreendarkseagreen1darkseagreen2darkseagreen3darkseagreen4darkslatebluedarkslategraydarkslategray1darkslategray2darkslategray3darkslategray4darkslategreydarkturquoisedarkvioletdeeppinkdeeppink1deeppink2deeppink3deeppink4deepskybluedeepskyblue1deepskyblue2deepskyblue3deepskyblue4dimgraydimgreydodgerbluedodgerblue1dodgerblue2dodgerblue3dodgerblue4firebrickfirebrick1firebrick2firebrick3firebrick4floralwhiteforestgreengainsboroghostwhitegoldgold1gold2gold3gold4goldenrodgoldenrod1goldenrod2goldenrod3goldenrod4graygray0gray1gray2gray3gray4gray5gray6gray7gray8gray9gray10gray11gray12gray13gray14gray15gray16gray17gray18gray19gray20gray21gray22gray23gray24gray25gray26gray27gray28gray29gray30gray31gray32gray33gray34gray35gray36gray37gray38gray39gray40gray41gray42gray43gray44gray45gray46gray47gray48gray49gray50gray51gray52gray53gray54gray55gray56gray57gray58gray59gray60gray61gray62gray63gray64gray65gray66gray67gray68gray69gray70gray71gray72gray73gray74gray75gray76gray77gray78gray79gray80gray81gray82gray83gray84gray85gray86gray87gray88gray89gray90gray91gray92gray93gray94gray95gray96gray97gray98gray99gray100greengreen1green2green3green4greenyellowgreygrey0grey1grey2grey3grey4grey5grey6grey7grey8grey9grey10grey11grey12grey13grey14grey15grey16grey17grey18grey19grey20grey21grey22grey23grey24grey25grey26grey27grey28grey29grey30grey31grey32grey33grey34grey35grey36grey37grey38grey39grey40grey41grey42grey43grey44grey45grey46grey47grey48grey49grey50grey51grey52grey53grey54grey55grey56grey57grey58grey59grey60grey61grey62grey63grey64grey65grey66grey67grey68grey69grey70grey71grey72grey73grey74grey75grey76grey77grey78grey79grey80grey81grey82grey83grey84grey85grey86grey87grey88grey89grey90grey91grey92grey93grey94grey95grey96grey97grey98grey99grey100honeydewhoneydew1honeydew2honeydew3honeydew4hotpinkhotpink1hotpink2hotpink3hotpink4indianredindianred1indianred2indianred3indianred4indigoivoryivory1ivory2ivory3ivory4khakikhaki1khaki2khaki3khaki4lavenderlavenderblushlavenderblush1lavenderblush2lavenderblush3lavenderblush4lawngreenlemonchiffonlemonchiffon1lemonchiffon2lemonchiffon3lemonchiffon4lightbluelightblue1lightblue2lightblue3lightblue4lightcorallightcyanlightcyan1lightcyan2lightcyan3lightcyan4lightgoldenrodlightgoldenrod1lightgoldenrod2lightgoldenrod3lightgoldenrod4lightgoldenrodyellowlightgraylightgreylightpinklightpink1lightpink2lightpink3lightpink4lightsalmonlightsalmon1lightsalmon2lightsalmon3lightsalmon4lightseagreenlightskybluelightskyblue1lightskyblue2lightskyblue3lightskyblue4lightslatebluelightslategraylightslategreylightsteelbluelightsteelblue1lightsteelblue2lightsteelblue3lightsteelblue4lightyellowlightyellow1lightyellow2lightyellow3lightyellow4limegreenlinenmagentamagenta1magenta2magenta3magenta4maroonmaroon1maroon2maroon3maroon4mediumaquamarinemediumbluemediumorchidmediumorchid1mediumorchid2mediumorchid3mediumorchid4mediumpurplemediumpurple1mediumpurple2mediumpurple3mediumpurple4mediumseagreenmediumslatebluemediumspringgreenmediumturquoisemediumvioletredmidnightbluemintcreammistyrosemistyrose1mistyrose2mistyrose3mistyrose4moccasinnavajowhitenavajowhite1navajowhite2navajowhite3navajowhite4navynavyblueoldlaceolivedrabolivedrab1olivedrab2olivedrab3olivedrab4orangeorange1orange2orange3orange4orangeredorangered1orangered2orangered3orangered4orchidorchid1orchid2orchid3orchid4palegoldenrodpalegreenpalegreen1palegreen2palegreen3palegreen4paleturquoisepaleturquoise1paleturquoise2paleturquoise3paleturquoise4palevioletredpalevioletred1palevioletred2palevioletred3palevioletred4papayawhippeachpuffpeachpuff1peachpuff2peachpuff3peachpuff4perupinkpink1pink2pink3pink4plumplum1plum2plum3plum4powderbluepurplepurple1purple2purple3purple4redred1red2red3red4rosybrownrosybrown1rosybrown2rosybrown3rosybrown4royalblueroyalblue1royalblue2royalblue3royalblue4saddlebrownsalmonsalmon1salmon2salmon3salmon4sandybrownseagreenseagreen1seagreen2seagreen3seagreen4seashellseashell1seashell2seashell3seashell4siennasienna1sienna2sienna3sienna4skyblueskyblue1skyblue2skyblue3skyblue4slateblueslateblue1slateblue2slateblue3slateblue4slategrayslategray1slategray2slategray3slategray4slategreysnowsnow1snow2snow3snow4springgreenspringgreen1springgreen2springgreen3springgreen4steelbluesteelblue1steelblue2steelblue3steelblue4tantan1tan2tan3tan4thistlethistle1thistle2thistle3thistle4tomatotomato1tomato2tomato3tomato4transparentturquoiseturquoise1turquoise2turquoise3turquoise4violetvioletredvioletred1violetred2violetred3violetred4wheatwheat1wheat2wheat3wheat4whitewhitesmokeyellowyellow1yellow2yellow3yellow4yellowgreen