The drive was swapped, the new one spun up, and the Dell EMC ECS Portal still lists the old faulted disk as removed. fcli agrees: the node reports one unknown disk and never picks the replacement up on its own. Nothing is broken — the fabric is holding a record for a device that no longer exists, and it will hold it until you tell it otherwise. Here is how to read the state, remove the stale entry and register the new drive.
One wording note before the commands: faulted disk is how the Portal describes it. Underneath, fcli never prints that word — it reports UNKNOWN for health and MISSING for operational status, which are the exact strings you will be searching your own output for.
The symptom: one UNKNOWN disk that never clears
admin@ecs-05:~> fcli disks list
AGENT ID HOSTNAME SERVICE DISK TYPE STORAGE NAME FORMAT GOOD BAD SUSPECT UNKNOWN
2a5551e7-bb7a-4068-8227-fd595c022670 ecs-01 object-main 8001GB HDD DATA XFS 30 0 0 0
f7f9b8ff-af2b-4b38-9401-7b7328a5f2f2 ecs-02 object-main 8001GB HDD DATA XFS 30 0 0 0
ae517c39-c16e-4d32-9311-0f18553d1849 ecs-03 object-main 8001GB HDD DATA XFS 30 0 0 0
7435e43a-11d0-44f8-859f-e0a7a1be5891 ecs-04 object-main 8001GB HDD DATA XFS 30 0 0 0
392f7a11-dac4-43aa-a92a-787ff9e12068 ecs-05 object-main 8001GB HDD DATA XFS 30 0 0 1
9b005f9d-5a65-464c-af4e-4d891aa87d4a ecs-06 object-main 8001GB HDD DATA XFS 30 0 0 0
8a0aa1ef-1337-43dc-b60f-c1f47f2564db ecs-07 object-main 8001GB HDD DATA XFS 30 0 0 0
d422ec5b-6050-4f92-9ef5-940cdba96b80 ecs-08 object-main 8001GB HDD DATA XFS 30 0 0 0
One node, one disk in the UNKNOWN column, GOOD still at 30 instead of 31. Note that fcli disks list reports the whole cluster from any node, so this is the fastest single command for spotting which node needs attention.
Why the disk stays MISSING in the fabric inventory
The ECS fabric tracks disks by UUID, not by slot. The UUID is derived when the disk is first provisioned and it belongs to that physical device. Pull the drive and the UUID does not go anywhere — the fabric keeps the record and marks it MISSING, because from its point of view a disk it knows about has stopped answering, which is exactly what a cable fault or a controller problem looks like too. It has no way to tell "gone for good" from "gone for a minute".
The replacement drive arrives with a different UUID, in the same slot. So after a swap the node legitimately holds two records for slot A00: one operative, one missing. Clearing the old one is a deliberate act, and that is by design.
Step 1. Dump the disk detail on the affected node
admin@ecs-05:~> sudo /opt/emc/caspian/fabric/cli/bin/fcli agent disk.disks
This returns JSON with every disk on that node. You are looking for two entries in the same slot: the faulted disk and its replacement. The faulted one is identified by any of mount_status: UNMOUNTED, health: UNKNOWN or operational_status: MISSING. Write down both UUIDs and the slot number — here, slot A00:
{
"disks" : [
{
"capacity_gb" : 8001,
"device" : "/dev/sdc",
"drive_id" : {
"product_id" : "H04728T8CLAR8000",
"serial_number" : "VDHJKVSF",
"vendor_id" : "HITACHI"
},
"endurance" : -1,
"firmware" : "A430",
"format" : "XFS",
"health" : "GOOD",
"mount_path" : "/dae/uuid-b1d53734-16a2-4fcb-aaa4-c2a8084b13b4",
"mount_status" : "MOUNTED",
"operational_status" : "OPERATIVE",
"owner" : "object",
"reason" : "Disk is operative.",
"slot" : "A00",
"storage_class" : "HDD",
"storage_name" : "DATA",
"uuid" : "b1d53734-16a2-4fcb-aaa4-c2a8084b13b4"
},
{
"capacity_gb" : 8001,
"device" : "/dev/sdal",
"drive_id" : {
"product_id" : "H0H72108CLAR8000",
"serial_number" : "7SJ13ENU",
"vendor_id" : "HITACHI"
},
"endurance" : -1,
"firmware" : "P290",
"format" : "XFS",
"health" : "UNKNOWN",
"mount_path" : "/dae/uuid-411b377b-0566-48b0-b3e2-9c90ba929d78",
"mount_status" : "UNMOUNTED",
"operational_status" : "MISSING",
"owner" : "object",
"reason" : "System does not detect the disk. Contact your service provider.",
"slot" : "A00",
"storage_class" : "HDD",
"storage_name" : "DATA",
"uuid" : "411b377b-0566-48b0-b3e2-9c90ba929d78"
},
{
"capacity_gb" : 8001,
"device" : "/dev/sdbc",
"drive_id" : {
"product_id" : "H0H72108CLAR8000",
"serial_number" : "7SJ0VZ5U",
"vendor_id" : "HITACHI"
},
"endurance" : -1,
"firmware" : "P290",
"format" : "XFS",
"health" : "GOOD",
"mount_path" : "/dae/uuid-42c5706e-6e35-44c5-af33-0e0e3577dcc1",
"mount_status" : "MOUNTED",
"operational_status" : "OPERATIVE",
"owner" : "object",
"reason" : "Disk is operative.",
"slot" : "A11",
"storage_class" : "HDD",
"storage_name" : "DATA",
"uuid" : "42c5706e-6e35-44c5-af33-0e0e3577dcc1"
},
],
"etag" : 919185,
"status" : "OK"
}
The output is long — thirty entries on a full node — so filter it instead of scrolling. The faulted disk is the one that fails to be MOUNTED:
sudo /opt/emc/caspian/fabric/cli/bin/fcli agent disk.disks \
| python -c 'import json,sys
for d in json.load(sys.stdin)["disks"]:
if d["mount_status"] != "MOUNTED" or d["health"] != "GOOD":
print(d["slot"], d["uuid"], d["health"], d["operational_status"], d["reason"])'
Note the wording of the reason field on the failed entry: "System does not detect the disk. Contact your service provider." That is the fabric saying the device is not visible on the bus — not that the drive has failed a health check. The distinction matters, and it is the subject of the next section.
Before you remove anything: three checks
--force does what it says, so spend a minute here.
- Confirm the drive is physically out. A
MISSINGentry means "not visible", which also describes a loose carrier, a bad expander port or an HBA that reset. Removing the UUID of a disk that is still installed and merely unreachable discards a record that would have come back on its own. - Check whether more than one disk went quiet at the same moment. A single faulted disk is a drive. Several at once, especially across slots, is almost always connectivity — see the commands that read the network links on every ECS node and the failed-disk checks before touching the fabric.
- Do one node at a time. ECS re-protects the data from the faulted disk across the rest of the cluster; let that settle before you start on the next node.
Step 2. Remove the faulted disk from the fabric
With the UUID of the old device in hand:
admin@ecs-05:~> sudo /opt/emc/caspian/fabric/cli/bin/fcli disks remove --diskid 411b377b-0566-48b0-b3e2-9c90ba929d78 --force
disk 411b377b-0566-48b0-b3e2-9c90ba929d78 removed successfully
The general form is:
sudo /opt/emc/caspian/fabric/cli/bin/fcli disks remove --diskid <old-disk-uuid> --force
Copy and paste the UUID; do not retype it. The two UUIDs in the same slot differ only in the middle and a transposed character here removes the wrong record.
Step 3. Add the new disk
admin@ecs-05:~> sudo /opt/emc/caspian/fabric/cli/bin/fcli agent disk.add --disk b1d53734-16a2-4fcb-aaa4-c2a8084b13b4
{
"etag" : 919188,
"status" : "OK"
}
General form:
sudo /opt/emc/caspian/fabric/cli/bin/fcli agent disk.add --disk <new-disk-uuid>
"status" : "OK" and an incremented etag mean the fabric accepted the change. The disk is formatted and brought into the storage pool from here on without further input.
Step 4. Verify
Run the detail dump again — no entry should be left with UNMOUNTED, UNKNOWN or MISSING:
sudo /opt/emc/caspian/fabric/cli/bin/fcli agent disk.disks | grep -E 'UNMOUNTED|UNKNOWN|MISSING'
Then the cluster view, where the UNKNOWN column drops back to zero and the node returns to its full disk count:
admin@ecs-05:~> fcli disks list
AGENT ID HOSTNAME SERVICE DISK TYPE STORAGE NAME FORMAT GOOD BAD SUSPECT UNKNOWN
2a5551e7-bb7a-4068-8227-fd595c022670 ecs-01 object-main 8001GB HDD DATA XFS 30 0 0 0
f7f9b8ff-af2b-4b38-9401-7b7328a5f2f2 ecs-02 object-main 8001GB HDD DATA XFS 30 0 0 0
ae517c39-c16e-4d32-9311-0f18553d1849 ecs-03 object-main 8001GB HDD DATA XFS 30 0 0 0
7435e43a-11d0-44f8-859f-e0a7a1be5891 ecs-04 object-main 8001GB HDD DATA XFS 30 0 0 0
392f7a11-dac4-43aa-a92a-787ff9e12068 ecs-05 object-main 8001GB HDD DATA XFS 30 0 0 0
9b005f9d-5a65-464c-af4e-4d891aa87d4a ecs-06 object-main 8001GB HDD DATA XFS 30 0 0 0
8a0aa1ef-1337-43dc-b60f-c1f47f2564db ecs-07 object-main 8001GB HDD DATA XFS 30 0 0 0
d422ec5b-6050-4f92-9ef5-940cdba96b80 ecs-08 object-main 8001GB HDD DATA XFS 30 0 0 0
The ECS Portal follows the fabric, not the other way round, so the removed entry disappears from the GUI on its own once fcli is clean. If the Portal still shows it after a few minutes, refresh the disk view rather than repeating the commands — running disk.add twice for the same UUID achieves nothing.
When you should not do this by hand
On a healthy cluster the fabric agent provisions a replacement drive without help; this procedure is for the case where it has not, and the old record is what is in the way. If the node itself is in a bad state — several disks SUSPECT, services restarting, a node that was recently rebooted — fix that first. A disks remove on a node that is still settling removes a record that was about to recover, and you have turned a transient fault into a real disk replacement.
