SSD let me down again
This ship is sinking…
It happened again. Btrfs began complaining about failed checksums on my rootfs and remounted itself as read-only. I rebooted and the BIOS didn’t see my OCZ Agility 2 SSD again. Rebooted a few more times to confirm.
Shut off the power supply completely and then turn it back on and everything was fine again. It’s only time until it fails completely. Until then I’ll keep backing up and waiting for a good SSD deal on Slick Deals. Too bad I missed the 500 GB Samsung 840 SSD for $285. Should have bought it.
I’ll soon learn how exactly a SSD fails when it does fail. No failed AHCI/SCSI operations, just warnings from the file system. I guess this means the NAND memory is failing and the controller has no clue. Interestingly enough, the controller didn’t respond when rebooted and the BIOS enumerated the AHCI ports.
Good thing I have regular backups, just to be safe I explicitly snapshotted my rootfs with:
sudo tar cf - --one-file-system / /boot | bar | lbzip2 -cs > /mnt/massive/backup/sda5-rootfs.2013.05.27.tar.bz2
Health Data?
<sarcasm> As you’d expect, the S.M.A.R.T. data says the drive is 100% ok. Oh yeah, it passes the short, long and conveyance tests with flying colors. I’m glad S.M.A.R.T. helps me confirm my drive is failing. Not. </sarcasm>
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| smartctl 5.43 2012-06-30 r3573 [x86_64-linux-3.9.4-030904-generic] (local build) | |
| Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net | |
| === START OF INFORMATION SECTION === | |
| Model Family: SandForce Driven SSDs | |
| Device Model: OCZ-AGILITY2 | |
| Serial Number: xxxxxxxxx | |
| LU WWN Device Id: x xxx xxxxxxx | |
| Firmware Version: 1.37 | |
| User Capacity: 120,034,123,776 bytes [120 GB] | |
| Sector Size: 512 bytes logical/physical | |
| Device is: In smartctl database [for details use: -P show] | |
| ATA Version is: 8 | |
| ATA Standard is: ATA-8-ACS revision 6 | |
| Local Time is: Tue May 28 00:19:04 2013 PDT | |
| SMART support is: Available - device has SMART capability. | |
| SMART support is: Enabled | |
| === START OF READ SMART DATA SECTION === | |
| SMART overall-health self-assessment test result: PASSED | |
| General SMART Values: | |
| Offline data collection status: (0x02) Offline data collection activity | |
| was completed without error. | |
| Auto Offline Data Collection: Disabled. | |
| Self-test execution status: ( 0) The previous self-test routine completed | |
| without error or no self-test has ever | |
| been run. | |
| Total time to complete Offline | |
| data collection: ( 0) seconds. | |
| Offline data collection | |
| capabilities: (0x7f) SMART execute Offline immediate. | |
| Auto Offline data collection on/off support. | |
| Abort Offline collection upon new | |
| command. | |
| Offline surface scan supported. | |
| Self-test supported. | |
| Conveyance Self-test supported. | |
| Selective Self-test supported. | |
| SMART capabilities: (0x0003) Saves SMART data before entering | |
| power-saving mode. | |
| Supports SMART auto save timer. | |
| Error logging capability: (0x01) Error logging supported. | |
| General Purpose Logging supported. | |
| Short self-test routine | |
| recommended polling time: ( 1) minutes. | |
| Extended self-test routine | |
| recommended polling time: ( 5) minutes. | |
| Conveyance self-test routine | |
| recommended polling time: ( 2) minutes. | |
| SCT capabilities: (0x003d) SCT Status supported. | |
| SCT Error Recovery Control supported. | |
| SCT Feature Control supported. | |
| SCT Data Table supported. | |
| SMART Attributes Data Structure revision number: 10 | |
| Vendor Specific SMART Attributes with Thresholds: | |
| ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE | |
| 1 Raw_Read_Error_Rate 0x000f 116 100 050 Pre-fail Always - 0/135697424 | |
| 5 Retired_Block_Count 0x0033 100 100 003 Pre-fail Always - 0 | |
| 9 Power_On_Hours_and_Msec 0x0032 100 100 000 Old_age Always - 24716h+00m+17.450s | |
| 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 144 | |
| 171 Program_Fail_Count 0x0032 000 000 000 Old_age Always - 2 | |
| 172 Erase_Fail_Count 0x0032 000 000 000 Old_age Always - 1 | |
| 174 Unexpect_Power_Loss_Ct 0x0030 000 000 000 Old_age Offline - 68 | |
| 177 Wear_Range_Delta 0x0000 000 000 --- Old_age Offline - 0 | |
| 181 Program_Fail_Count 0x0032 000 000 000 Old_age Always - 2 | |
| 182 Erase_Fail_Count 0x0032 000 000 000 Old_age Always - 1 | |
| 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 | |
| 194 Temperature_Celsius 0x0022 030 030 000 Old_age Always - 30 (Min/Max 30/30) | |
| 195 ECC_Uncorr_Error_Count 0x001c 116 100 000 Old_age Offline - 0/135697424 | |
| 196 Reallocated_Event_Count 0x0033 100 100 003 Pre-fail Always - 0 | |
| 231 SSD_Life_Left 0x0013 096 096 010 Pre-fail Always - 1 | |
| 233 SandForce_Internal 0x0000 000 000 000 Old_age Offline - 16640 | |
| 234 SandForce_Internal 0x0032 000 000 000 Old_age Always - 21632 | |
| 241 Lifetime_Writes_GiB 0x0032 000 000 000 Old_age Always - 21632 | |
| 242 Lifetime_Reads_GiB 0x0032 000 000 000 Old_age Always - 17728 | |
| SMART Error Log not supported | |
| SMART Self-test Log not supported | |
| SMART Selective self-test log data structure revision number 1 | |
| SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS | |
| 1 0 0 Not_testing | |
| 2 0 0 Not_testing | |
| 3 0 0 Not_testing | |
| 4 0 0 Not_testing | |
| 5 0 0 Not_testing | |
| Selective self-test flags (0x0): | |
| After scanning selected spans, do NOT read-scan remainder of disk. | |
| If Selective self-test is pending on power-up, resume after 0 minute delay. |
Kernel Log
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [67873.504490] btrfs csum failed ino 2814 extent 18735104 csum 2566472073 wanted 1550764569 mirror 0 | |
| [67873.504582] btrfs csum failed ino 2814 extent 18735104 csum 2566472073 wanted 1550764569 mirror 0 | |
| [67875.972459] btrfs csum failed ino 2814 extent 18735104 csum 2566472073 wanted 1550764569 mirror 0 | |
| [67905.682555] btrfs csum failed ino 2814 extent 18735104 csum 2566472073 wanted 1550764569 mirror 0 | |
| [80612.424338] btrfs csum failed ino 114607 extent 5127675904 csum 2566472073 wanted 2346267106 mirror 0 | |
| [80612.424346] btrfs csum failed ino 114607 extent 5127778304 csum 2566472073 wanted 745667138 mirror 0 | |
| [80612.424420] btrfs csum failed ino 114607 extent 5127876608 csum 2566472073 wanted 21024237 mirror 0 | |
| [80612.424428] btrfs csum failed ino 114607 extent 5127991296 csum 2566472073 wanted 3090103677 mirror 0 | |
| [80612.424500] btrfs csum failed ino 114607 extent 5128101888 csum 2566472073 wanted 1136036514 mirror 0 | |
| [80612.424508] btrfs csum failed ino 114607 extent 5128224768 csum 2566472073 wanted 1308107691 mirror 0 | |
| [80612.424579] btrfs csum failed ino 114607 extent 5128347648 csum 2566472073 wanted 1386942802 mirror 0 | |
| [80612.424586] btrfs csum failed ino 114607 extent 5128470528 csum 2566472073 wanted 2262798622 mirror 0 | |
| [80612.424660] btrfs csum failed ino 114607 extent 5128593408 csum 2566472073 wanted 3407177240 mirror 0 | |
| [80612.424667] btrfs csum failed ino 114607 extent 5128716288 csum 2566472073 wanted 670591134 mirror 0 | |
| [80612.424739] btrfs csum failed ino 114607 extent 5128839168 csum 2566472073 wanted 3579358996 mirror 0 | |
| [80612.424745] btrfs csum failed ino 114607 extent 5128962048 csum 2566472073 wanted 181367351 mirror 0 | |
| [80612.424815] btrfs csum failed ino 114607 extent 5129084928 csum 2566472073 wanted 1557767333 mirror 0 | |
| [80612.424819] btrfs csum failed ino 114607 extent 5129207808 csum 2566472073 wanted 3860158340 mirror 0 | |
| [80612.424892] btrfs csum failed ino 114607 extent 5129330688 csum 2566472073 wanted 1635249415 mirror 0 | |
| [80612.424898] btrfs csum failed ino 114607 extent 5129453568 csum 2566472073 wanted 281896671 mirror 0 | |
| [80612.424982] btrfs csum failed ino 114607 extent 5129576448 csum 2566472073 wanted 3249760658 mirror 0 | |
| [80612.424989] btrfs csum failed ino 114607 extent 5129699328 csum 2566472073 wanted 701264376 mirror 0 | |
| [80612.425026] btrfs csum failed ino 114607 extent 5129818112 csum 2566472073 wanted 2088993094 mirror 0 | |
| [80612.425072] btrfs csum failed ino 114607 extent 5129936896 csum 2566472073 wanted 547513300 mirror 0 | |
| [80612.425109] btrfs csum failed ino 114607 extent 5130059776 csum 2566472073 wanted 3635804551 mirror 0 | |
| [80612.425129] btrfs csum failed ino 114607 extent 5130178560 csum 2566472073 wanted 594525707 mirror 0 | |
| [80612.425231] btrfs csum failed ino 114607 extent 5130416128 csum 2566472073 wanted 1300117047 mirror 0 | |
| [80612.425238] btrfs csum failed ino 114607 extent 5130297344 csum 2566472073 wanted 3028195057 mirror 0 | |
| [80612.425314] btrfs csum failed ino 114607 extent 5130539008 csum 2566472073 wanted 2688471965 mirror 0 | |
| [80612.425319] btrfs csum failed ino 114607 extent 5130657792 csum 2566472073 wanted 2332576544 mirror 0 | |
| [80612.425388] btrfs csum failed ino 114607 extent 5130776576 csum 1664377860 wanted 2591896249 mirror 0 | |
| [80612.425393] btrfs csum failed ino 114607 extent 5130895360 csum 4026639418 wanted 2782237159 mirror 0 | |
| [80612.425464] btrfs csum failed ino 114607 extent 5131014144 csum 2566472073 wanted 141599301 mirror 0 | |
| [80612.425469] btrfs csum failed ino 114607 extent 5131128832 csum 2566472073 wanted 4075551073 mirror 0 | |
| [80612.425538] btrfs csum failed ino 114607 extent 5131247616 csum 2566472073 wanted 3579207023 mirror 0 | |
| [80612.425544] btrfs csum failed ino 114607 extent 5131358208 csum 2566472073 wanted 3612524280 mirror 0 | |
| [80612.425549] btrfs csum failed ino 114607 extent 5127675904 csum 2566472073 wanted 2346267106 mirror 0 | |
| [80612.459217] btrfs csum failed ino 114607 extent 5127675904 csum 2566472073 wanted 2346267106 mirror 0 | |
| [80627.747446] btrfs csum failed ino 114607 extent 5127675904 csum 2166476949 wanted 2346267106 mirror 0 | |
| [80627.778744] btrfs csum failed ino 114607 extent 5127675904 csum 293749674 wanted 2346267106 mirror 0 | |
| [80639.627726] btrfs csum failed ino 114607 extent 5127675904 csum 2566472073 wanted 2346267106 mirror 0 | |
| [80639.658077] btrfs csum failed ino 114607 extent 5127675904 csum 2566472073 wanted 2346267106 mirror 0 | |
| [80676.235389] btrfs csum failed ino 175953 extent 5347020800 csum 2566472073 wanted 3181599292 mirror 0 | |
| [80676.235895] btrfs csum failed ino 175951 extent 5347012608 csum 2566472073 wanted 3103485965 mirror 0 | |
| [80676.235908] btrfs csum failed ino 175951 extent 5347012608 csum 2566472073 wanted 3103485965 mirror 0 |
Comments