An ECS node reports a failed disk, and before anyone ships you a replacement you have to prove which disk it is, that it is really dead, and which appliance it sits in. These are the commands that answer those questions on an ECS 3.x appliance — Gen3 EX-series here, though the fabric paths are the same on U-Series and D-Series nodes. None of them change cluster state: they read, they do not repair. Run them as admin over SSH, and note that a full xdoctor sweep takes a few minutes and writes its report locally.
Start cluster-wide, then walk to the node
viprexec "/opt/emc/caspian/fabric/cli/bin/fcli agent disk.disks | grep -e 'health' -e 'operational_status' -e 'mount_status' | sort | uniq -c"
viprexec runs the same command on every node and the sort | uniq -c collapses the output into counts, so instead of reading hundreds of lines you see at a glance whether one disk is unhealthy or thirty are. A cluster-wide count that does not match the single failed disk you were called about is a different, larger problem — and usually not a disk problem at all. Scope the blast radius before you walk to a slot.
From the node with the failed disk: cs_hal, fcli and xdoctor
# every disk on this node with slot, serial and SMART status
cs_hal list disks
# map a disk serial number to its slot and device
cs_hal info <DISK_SERIAL>
# what the fabric thinks of every disk on this node
/opt/emc/caspian/fabric/cli/bin/fcli disks list
# the detailed per-disk state on this node
sudo /opt/emc/caspian/fabric/cli/bin/fcli agent disk.disks
# ECS and xDoctor versions - the first question on any Dell case
sudo xdoctor -x
# the xDoctor health report for this node
sudo xdoctor --hr
If the alert gave you a serial, cs_hal info turns it into a physical position you can walk to. If it gave you a slot or nothing useful, start with cs_hal list disks and work backwards. Note that cs_hal truncates disk serials, so match on the prefix rather than the whole string.
xdoctor -x is worth running first even though it looks trivial: it prints the ECS and xDoctor versions, and those are the first thing Dell asks for. The health report is the separate --hr.
From node 1: service-console, checkdiskio and racadm
# which node is which
getrackinfo
# SMART status across the cluster, filtered to the disks that matter
service-console run Disk_List --target-node 169.254.1.3 --disk-smart 'SUSPECT,FAILED'
# I/O errors on all nodes
bash /opt/emc/xdoctor/tools/ee_scripts/checkdiskio.sh -A
getrackinfo prints the node ID, private address and name, so run it first and pick the address of the node you are chasing. Node 1 is 169.254.1.1; that range is ECS's own internal addressing between nodes, not something you configured.
ee_scripts ships with xDoctor and its contents move between versions — if the path is not there, ls /opt/emc/xdoctor/tools/ee_scripts/ before assuming you mistyped it.
SUSPECT: count before you conclude
SUSPECT is as interesting as FAILED, but read the count before you read the disk. ECS stops writing to a suspect disk and keeps reading from it, so one suspect disk among thirty healthy ones is usually a drive on its way out — the one that wakes you a fortnight from now.
Several suspect disks appearing on the same node at once is usually not a disk problem at all. Dell attributes swarms of suspect disks to connectivity at the node, and they return to Good on their own once that is fixed (Dell KB 000021514 has the state definitions). Count first, then decide whether you are chasing a disk or a cable — and if it looks like a cable, the network link commands are the next stop.
What to collect before you open the Dell support case: serial, service tag, PSNT
# the Dell service tag of the node
sudo -i racadm getsvctag
# the chassis serial, same thing on Gen3 hardware
dmidecode -t system | grep Serial
# the appliance serial (PSNT)
sudo xdoctor --vdc --top | grep PSNT
# the report archive support will ask for anyway
sudo xdoctor --report --archive=<REPORT_ID> -CEW
Support will want the disk serial, the service tag of the node, the PSNT of the appliance and the ECS/xDoctor version pair. The service tag identifies the physical node; the PSNT identifies the appliance it belongs to. They are not the same number and you will be asked for both.
On Gen3 appliances the DMI system serial and racadm getsvctag return the same string, so either command will do. On older Gen1/Gen2 nodes there is no iDRAC to ask and dmidecode returns a chassis serial in a different format instead.
After the replacement arrives
Swapping the drive is not the end of it. ECS keeps the old disk in its inventory and the GUI will keep showing it as removed until you tell the cluster the failed disk is gone — that procedure is in replacing a faulted disk on ECS.
