0

I have lots of IoT devices running linux off an SD card, and for some reason they've all recently stopped working!

To get to the bottom of the issue, I tested these SD cards in Windows laptop.

The laptop could not read the SD cards, meaning:

  • Disk Management hung indefinitely
  • DISKPART hung when I tried to list
  • the volumes CHKDSK reported the disk as RAW formatted

I thought all hope was lost until I set the SD card to read-only using the little switch on the side of the card.

The laptop was then able to fully read the SD cards, meaning:

  • Disk Management worked ok
  • DISKPART showed the volumes ok
  • CHKDSK reported the disk as FAT formatted (i.e. FAT 16)

So my questions now are:

  • What on earth is going on here and why?!
  • How do I recover these SD cards, since reformatting them fails in read only mode and read/write mode hangs everything.
  • Are there tools that can help with this?

Thanks in advance!

Timbo
  • 101
  • 1
  • If you can read the data, I would copy all I could, if it is important to you, and format the SD card(s). Then, like Natsu Kage has pointed out, I would not use them again, unless they are being used for not __important things__. However, a format just might bring them back where they can be used again for pictures, etc.. – vssher Jan 23 '20 at 02:32
  • Hi @vssher unfortunately I can't format an SD card when its physical switch is set to 'lock' (write protect) and when I unlock this switch no SD card reader can read these chips. – Timbo Jan 23 '20 at 22:43
  • @Timbo Have you had any power failures recently? – Natsu Kage Jan 23 '20 at 23:27
  • Yeah these hot devices are prone to unexpected resets – Timbo Jan 24 '20 at 04:02

1 Answers1

1

Sadly this is a common issue with some SD cards and has been answered on this website before:

Flash Storage failure

When a SD card encounters an issue, it will lock the data to prevent data loss. If there is no way to lock the data, it will prevent access until it is physically locked. Basically, your SD card has:

1) Reached the end of its lifespan or was a defective batch. If they are all the same brand and model, this is a big possibility.

2) Something in Linux is breaking the SD cards. This is rare and usually only happens if formatting goes wrong. If Linux recently went through an update on your IoT platforms, this is possible.

3) A power failure or instability is reducing the life of the SD cards? Superuser article on Flash instability Has there been any power cuts lately or instabilities?

To prove if it is Linux, I would try a new brand of SD card and try to reproduce the same conditions before the SD cards failed.

It is possible to recover these SD cards to a functional state. There is multiple articles on this issue: Superuser article on this subject

The issue is with time lost and complexity. I highly suggest not re-using SD cards that suffered such a failure.

Natsu Kage
  • 974
  • 5
  • 12
  • Thanks for your response! – Timbo Jan 23 '20 at 22:40
  • We have this issues on multiple brands of SD card from Transcend, SANDisk and Lexar – Timbo Jan 23 '20 at 22:41
  • Side note, those articles seem to focus on USB sticks, is this the same as SD cards? – Timbo Jan 23 '20 at 23:42
  • Is there an official source somewhere to back up the claim that "When a SD card encounters an issue, it will lock the data to prevent data loss." since this is key to this issue and something I need to prove to my stakeholders – Timbo Jan 23 '20 at 23:44
  • Both use the same basic way of storing data using NAND chips and a controller. https://en.wikipedia.org/wiki/File:Sd_insides.png (SD Card) https://upload.wikimedia.org/wikipedia/commons/thumb/e/e2/Usbkey_internals.jpg/800px-Usbkey_internals.jpg (USB Drive) – Natsu Kage Jan 23 '20 at 23:47
  • @Timbo Hmm, I don't think there's anything official. SD card is a proprietary format, and there's very few official articles on the internals. Even companies such as Sandisk specify it can be locked and no recovery is possible: https://kb.sandisk.com/app/answers/detail/a_id/6971/~/sandisk-microsd-memory-card-is-write-protected-or-locked – Natsu Kage Jan 23 '20 at 23:59
  • Thanks yeah I've been reading the SD card specifications however it's not very clear to me : https://www.sdcard.org/downloads/pls/click.php?p=PartA2_SD_Host_Controller_Simplified_Specification_Ver1.00.jpg&f=PartA2_SD_Host_Controller_Simplified_Specification_Ver1.00.pdf&e=EN_A2100 – Timbo Jan 24 '20 at 04:04
  • Hmm, I'd try reading through the Host controller specifications. It clearly talks about error reporting and the likes. Also, try the SD memory card formatter to see if it helps when the card isn't in locked mode. – Natsu Kage Jan 24 '20 at 04:13
  • Timbo have you read this page : https://www.sdcard.org/downloads/formatter/ (why you should use a SD formatter)... "It is strongly recommended to use the SD Memory Card Formatter to format SD/SDHC/SDXC Cards rather than using formatting tools provided with individual operating systems. In general, formatting tools provided with operating systems can format various storage media including SD/SDHC/SDXC Cards, but it may not be optimized for SD/SDHC/SDXC Cards and it may result in lower performance." – vssher Jan 24 '20 at 16:59