Lily Cohen, the system administrator of Firefish Social, reported a data loss incident that occurred during the reconfiguration of their file configurations.
This resulted in the loss of data, along with a discovered backup system failure that prevented data recovery. The problem originated from a file yaml migration and restructuring. However, it was found that some yaml files were misplaced in a folder that ArgoCD couldn’t see, causing ArgoCD to immediately delete all data in the migrated namespace, including all user data stored in Persistent Volume Claims (PVC).
The situation worsened when Velero, the backup configuration, failed to backup data in PVC as scheduled every 6 hours. This had a devastating impact on the three social media websites under Firefish’s care: firefish.lgbt, musician.social, and outdoors.lgbt. All data from these websites were lost.
Leave a Comment