This may be the most important post you will ever read. If you’re a Zimbra Administrator, please read, and pass this onto your colleagues who use Zimbra. If you’re a CTO or CEO, take time to ask your Zimbra Admin about the subject of this post. This blog post is about backups.
Whether your an Open Source User, Zimbra Desktop User, or Network Edition Customer, you can do backups of your data. There is nothing worse than getting a call from a customer, or a Private Message from a Forum User that says, “I need help. My HD has crashed, and all my backups were on that drive.”
Let me Digress for a moment, and share with you my personal experience with backups. It’s sort of a Legend here at Zimbra. It all begins in 2005, Zimbra was the new kid on the block and I was an inexperienced System Linux admin for Tombstone Unified School District in Arizona. This was a small district with limited funds. When I interviewed for the post, the Superintendent handed me his card, and it had a Hotmail address on it. Right then, I knew this would be quite a difficult job.
One of the first things I did, was investigate E-mail Server platforms, and TCO (Total Cost of Ownership). A quick search on the interwebs (which is a series of tubes, not to be confused with the you tubes), yielded this new thing called a Zimbra. Much like our millions of Users, I downloaded it and tried it. It worked great! We used the Zimbra Open Source Edition (then it was in Beta).
I was so proud of my cool new setup, and our staff were so excited to finally have good e-mail and calendaring. In knew the importance of keeping backups, and archiving. All of you System Admins at Hospitals and Government institutions know what I mean. You get well acquainted with the Laws and Requirements. So, I would stay up until about midnight, and stop Zimbra, and rsync /opt/zimbra into /opt/zimbra/backups. Keep in mind that this was in the early days of Zimbra, and I wasn’t even an employee yet. As a matter of fact, myself and several others, pioneered the Open Source backup procedure.
Everything was great until one night I noticed I was running low on Disk Space on my /opt partition. So, I thought, “If I just remove all my backups, and make a fresh one, that will save me a bunch of space”. So I ran the following command as root:
rm -rf /opt/zimbra backups
Now, back then, we mounted a clamav partition ramdisk for quarantine purposes. The only indication that I had that something was wrong was when I got an error saying that it couldn’t unmount the partition because it was in use. Everything else in /opt/zimbra was gone…including my backups.
As most of you admins know, preserving data is important. We were involved in several litigation matters, and I would later be cited for obstruction of justice for destroying evidence.
When I noticed what had happened, I immediately called Zimbra and talked with MarcMac here at Zimbra. He tried to recover the inodes using Midnight Commander, but it was a total loss.
Lesson Learned. So, from one admin to another, please take time to make sure your backups are not located on the same machine that Zimbra is on. Please! We never want to hear about data loss. Whether an opensource user or network user, I hope you will take a few minutes to consider your backup strategy, and fix any single points in failure.
Learn from my experience.
-John (jholder)
on July 31st, 2008 at 2:00 am
I remember when people used to laugh at redhat when they made rm aliased to rm -i by default which forced people to confirm every deletion. those of us who’ve done this before though me included would never laugh!
on July 31st, 2008 at 6:27 am
Right, so maybe now you do understand why we have been pushing for such a long time for Ubuntu 8.04 (with ISCSI) support
I’m glad we’re nearly there.
on July 31st, 2008 at 7:32 am
There are two types of admin the ones who are making backup’s and those who don’t make backup’s yet.
on July 31st, 2008 at 7:56 am
That’s why after searching around for a post backup function in NE, that I created my own, to tar up the last backup session, and ftp it to another server. At least I now have two copies of the Zimbra backups.
on July 31st, 2008 at 12:39 pm
I remember back in my frist few days of linux in 2001 or 2002.. i was trying to get a new telnet client installed at my home office linux box with my bro and i did rm -rf /etc without thinking… and well i learnt my lesson back then too
on July 31st, 2008 at 1:36 pm
AMEN BROTHER!
on August 1st, 2008 at 6:58 pm
ISCSI is great to prevent loss due to hardware failure but it is NOT going to protect you against John’s tale of woe. I have my script ftp a copy of my current backup archive to a completely different machine in a completely different building for this very reason. All your backup eggs should not be in one box, nor should they be under the command of one OS with one command line. Any iSCSI mount is still subject to a random rm command; it’s a little harder if you actually have to log onto the different box to get at the file.
Sometimes simple really is safer!
And I’ve rm’ed a whole data partition on a central business system too, so I feel your pain — well some of it; at least I had a systape and backup. . .
on August 8th, 2008 at 2:11 am
Just wanted to say “Thanks”, for the reminder. On your suggestion I’m creating monthly tars of full server backups.
on August 12th, 2008 at 3:33 pm
And just a reminder to the new kids:
Spinning magnetic storage is *not* a backup.
If it’s not on a tape you can walk out of the building with and lock up somewhere, it’s not a backup. Period. End of report.
I have backups I can still read on tape that are over 10 years old.
Try that with a hard drive, much less a DVD.
Really: DLT, SDLT, LTO. Anything else, and you’re just kidding yourself.
People will tell you I’m wrong. But I don’t lose data.
on August 16th, 2008 at 2:03 am
Yeah.. Did that to my home directory once. I was using synergy and my cursor went back to my laptop without me realizing it. I typed “rm -R *”. When it didn’t show up I hit enter a couple of times to try to figure out where my cursor was. Doh!
on October 19th, 2008 at 8:03 am
Having been bitten quite hard by losing Zimbra once I really concur with the sentiment here
http://www.zimbra.com/forums/installation/5021-upgrade-4-0-3-just-trashed-my-install-without-backup.html
This was a personal install with a few friendly users, not a work install, but it hurt all the same. My strategy for Zimbra backup is:
1. shut zimbra instance down
2. rsync zimbra to other spindle on same host (real quick)
3. restart zimbra instance
4. on my backup host, rsync (pull) backup with bandwidth throttle (real slow)
I find this reduces the downtime and I always have a backup available on host and remote.
Have fun!
on October 29th, 2008 at 1:35 pm
[…] our earlier advice to take backups frequently, and secure them offsite - thought we’d highlight a few recent […]
on December 1st, 2008 at 1:14 pm
Any change zimbra could integrate with Veritas Netbackup http://www.symantec.com/business/netbackup
and others?