We saw several storage accounts that hadn’t been updated since 2020 and thought we could delete them to save about $15 per month. We didn’t see anywhere it was used. So on a Monday, we deleted those storage accounts and noticed some Azure Functions stopped working.
A different team member:
Why isn’t our development Function not working? Oh no, our production function is down as well 💣😢
What are the exceptions? (No such host is known. (storageaccount{id}.blob.core.windows.net:443)) hmmm.
Asked in a Teams Channel, and “I deleted the blob storage”, “welp, here are the results of the scream test”.
We can recover the storage accounts with a few button clicks!
After that, the function started working and Metrics started showing executions again. Phew, back in business.
All in all, we lost about an hour, but the impact was minimal. We wrote up our incident review and I decided to share this with you.
We added some Azure Management locks manually on the function and the storage accounts.
I think that Azure should automatically protect the storage account for a function or at least make it obvious that it is for the storage account.
You can find if it is used in “either AzureWebJobsStorage or WEBSITE_CONTENTAZUREFILECONNECTIONSTRING contains the name of your storage account as part of a connection string”
Please consider using Brave and adding me to your BAT payment ledger. Then you won't have to see ads! (when I get to $100 in Google Ads for a payout, I pledge to turn off ads)
Also check out my Resources Page for referrals that would help me.