Zimbra

Someone in the forums recently asked about ways to migrate individual accounts from one ZCS instance to another, so thought I’d share the enlightenment with all. Whether you are going from an on-premise install to a hosting provider, want to create handy archives of old employee accounts, or just need to duplicate mailbox contents of a user; the syntax in this article proves remarkably useful, and applies to all editions.

There are a multitude of comparable RFE’s on addressing this need via different approaches. (Bugzilla entries 19630, 29573, 28443 & 30163 to name a few.) Some want graphical tools to browse data and selectively migrate certain things, while others would be happy with a cross LDAP zmmailboxmove.

Depending on your situation, several backup tools can take care of a large portion of your daily needs; and there are ways to do Zimbra-to-Zimbra migrations using the Network Edition’s backup and restore capabilities - however they require admin abilities on both systems. Meanwhile, most of the frequently used open source backup solutions are simply an “all accounts at-once” approach. So what to do when you need to move from your personal setup to a hosting provider? Or if you’re a hosting provider, move a tiny handful of accounts to a separate infrastructure? Before diving into the wiki on user migration for info on Imapsync, REST exports, CURL imports, etc; there’s a handy way to avoid the “one item type at-a-time” transfer methods.

In ZCS 5.0.9+ you can export an entire mailbox with:
/opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL “//?fmt=tgz” > /tmp/account.tgz

Next transfer via rsync, scp, sftp, etc. You’ll also need to create the account on the 2nd server if the desired account doesn’t exist at your destination server yet.

Then import with:
/opt/zimbra/bin/zmmailbox -z -m user@domain.com postRestURL “//?fmt=tgz&resolve=reset” /tmp/account.tgz

The resolve= paramater has several options:

• “skip” ignores duplicates of old items, it’s also the default conflict-resolution.
• “modify” changes old items.
• “reset” will delete the old subfolder (or entire mailbox if /).
• “replace” will delete and re-enter them.

‘Reset’ will be a bit faster on an empty destination mailbox because it skips most dupe checks.

Note: There were some duplication fixes and additional issues (mainly sync related) corrected with the tar formatter in 5.0.10.

Not a Zimbra Admin? Users can get the same zip/tar formatter on REST URL’s by visiting:
http:// server.domain.com/home/user?fmt=zip&query=is:anywhere

The zip format has been around for a long time, but doesn’t contain account & item metadata like the tar formatter automatically does:
http:// server.domain.com/home/user/?fmt=tgz

Infact, this same technique is currently used in Zimbra Desktop’s alpha backup solution.

If this approach doesn’t scale performance wise for your situation, or you simply don’t want to have everyone hit a REST URL for 30GB mailboxes all at the same time, here’s a collection of helpful scripts and other ways to systematically migrate:

Mysqldump & rsync with an interesting blob management technique: Zimbra2Zimbra

Imapsync for mail + postRestURL for contacts, calendar & filters: ZimbraMigrate (Expand the concept for tasks, documents, and briefcase items.)

Another method that could be extrapolated upon for migrations: Per User Mailbox Backup (OE Version - Zimbra :: Wiki)

Most of these solutions aren’t going to respect share permissions, but when pulling an account out of an environment that’s to be expected.

Zimlet spin-offs:
- Mail backup options for end users (.eml)
- Zimlet to save email in a txt file (.txt or html)

The above Zimlets are aimed at making quick self-copies & not for restores, but there are many methods for putting messages back into Zimbra, including tools like zmmailbox addMessge, zmlmtpinject, CURL, etc; for more info checkout these threads: Recover data from store folders & Moving Folders between users

If moving your entire server, I’m a huge fan of the install.sh -s trick when using NE backups to do so isn’t an option.

Part 1 covered Network Edition backup features, today’s snips apply to all editions.

First among the lesser known additions: We recently provided the possibility for a nice performance boost to some environments by adding the ability to turn on batched indexing in ZCS 5.0.3 (you can even fine tune it at the localconfig, COS, and account level). We’re not talking about when you re-index an entire account here, this is a change to the index-as-received model; now new items can sit in a ‘queue’ (really a ‘indexing deferred’ flag on the mail_items table of the pertaining mboxgroup database in MySQL) to run all at once when it reaches the zimbraBatchedIndexingSize threshold, saving you from all the tiny disk thrashing. It might not be immediately apparent that this works better, but you can mention it in the forums and we’ll show you the evidence to the contrary - it proves expecially useful for POP heavy or ZAD archive accounts.


 New & Enhanced Admin Tools

 

Your /opt/zimbra/bin & /opt/zimbra/libexec directories hold a wealth of tools to make your job easier.

zmdumpenv has been around for a long time, but underutilized - it grabs the basics that you should probably provide with every issue to help others understand where you’re coming from.

When you need to send ad-hoc SOAP commands to the server, the powerful zmsoap takes care of authenticating, generating the envelope, sending the request, and writing the response to stdout.

If your server freezes or is busy, running zmdialog can give that ‘my server hung’ support ticket a purpose. With JDK 1.5 it won’t collect a heap dump so you might also run /opt/zimbra/java/bin/jmap -heap:format=b [/opt/zimbra/log/zmmailboxd_java.pid] however zmdiaglog collects a core dump, from which it should be theoretically possible to get a heap dump. Thread dumps when you kill -QUIT/3 [pid] are helpful too. Info on ways to take them (like /opt/zimbra/libexec/zmmailboxdmgr threaddump) plus a handy script, are here.

There’s the MySQL metadata DB, the Lucene index, and the actual blob files on disk in /opt/zimbra/store. If you can click on a folder in the web UI and get results, it means MySQL is up and happy. If you open a message and get the infamous “missing blob for id” error, it means that the message files on disk aren’t where they’re supposed to be (ie: disk crash, not mounted/permissions, or you’ve recently pointed your volumes incorrectly). There’s all sorts of things you can do in that situation - here’s a more detailed list. Even if you don’t have a good grasp of the mailbox database structure yet, you can appreciate the need to easily determine if those blobs are still available (after you’ve moved whatever you can salvage of the store from corrupted drives to a better location). Enter the zmblobchk utility which can determine what files are missing - there’s also plans for a repair mode, for when you finally realize the blobs are gone and want to get rid of those UI error messages. Of course you’ll next consult the Network Edition’s restoreToTime feature or other backup solution you may have.

When you’re absolutely out of room on the disks housing your DB (seriously put in that purchase order for more storage - it’s cheap these days) in a pinch the optimizeMboxgroups.pl script (available in the public cache and added to the upcoming 5.0.10) can help you recover wasted space in your mail_item, appointment, imap_folder, imap_message, open_conversation, pop3_message, revision, and tombstone tables on each mboxgroup. Just note that it temporarily locks each table, and could use considerable IO while they’re being rebuilt. You can certainly use it pro-actively during a maintenance window to reclaim space as well.

There’s also a new wiki page on statistic collection so you can generate nice charts that help you figure out what you might need to tweak.

We could go on and on about the utilities in those folders, so throw up a test environment and experiment sometime - it may just make life easier when you have hundreds of users breathing down your neck. And if you should ever be ’stumbling around in the dark’ you can always enable additional debug logging to shed more light on the situation.

 

Continuing our earlier advice to take backups frequently, and secure them offsite - thought we’d highlight a few recent administrator related things added to ZCS that you might not have noticed.


 Network Edition Backup Enhancements

 

Speaking of backups, there are some new ways to take them in ZCS 5.0.x. With ever larger quota usage, full backups can often take a while to run, and even incrementals which process the redologs may still be one heck of a job when you’re talking thousands or millions of accounts. Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incremental sessions. As a plus it automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time. This allows you to do a point in time restore for any account. Simply divide your total accounts by the number of groups you choose (zimbraBackupAutoGroupedNumGroups is 7 by default) and that’s how many will get a full backup session each night. Newly provisioned accounts, and accounts whose last backup is a specified number of days older are picked first. (zimbraBackupAutoGroupedInterval is defaulted to 1d)

To save space, and therefore store older backups longer or run them more frequently, you can also auto-compress them with the –zip argument. This isn’t new, but it got improved handling of shared blobs in 5.0.5 as well as a -zipStore mode for speed. You can also adjust the buffer & queue capacity of the backup process, as well as additional options like the level of compression, or the number of archives per person via the backup_zip_copier_private_blob_zips localconfig attribute. Of course you lose the hard linking optimization (speed and space) for blobs that are in an earlier full backup already when working from the same disk - so it’s more advantageous for those off-site single-copies (you do make one often right?). However, there are legitimate uses for running it on your normal backups: Fewer files make it easier to copy or rsync later, and prevents you from running out of inodes. You can also easily delete individual backups rather than running zmbackup -del, and therefore keep just a few really old backups around for whatever compliance reasons you may have.

By-the-way, ZCS 5.0.6 added the ability to easily replay redologs from an arbitrary point in time with zmplayredo should you be in a unique situation that needs it. (Say you’ve been taking snapshot backups and but then need to restore, and you’ve also saved all the redologs since the snapshot. Or you take a snapshot, then manually copy redologs from the live system to bring the snapshot copy up to current. This allows you to force replay of them all and not just the uncommitted transactions.)

We’ve always recommended that: “After upgrade, you should run a full backup immediately as changes in this release invalidate all old backups.” It’s still good advice to have a fresh full, but there’s always the infrequent need to get data from an older minor version backup without throwing up a temporary machine. Well, with 5.0.x we aimed to make restores compatible across patch releases (i.e. an older 5.0.x backup, not a prior 4.5.x backup - major version restore is this RFE). There was a bug about zmrestore not handling database schema changes, but that’s fixed in 5.0.5 and later - so backwards compatibility for restore is now theoretically possible. And we’re also looking to put icing on the cake by adding a conversion tool to upgrade backups themselves to allow restore on later ZCS versions.

This may be the most important post you will ever read. If you’re a Zimbra Administrator, please read, and pass this onto your colleagues who use Zimbra. If you’re a CTO or CEO, take time to ask your Zimbra Admin about the subject of this post. This blog post is about backups.

Whether your an Open Source User, Zimbra Desktop User, or Network Edition Customer, you can do backups of your data. There is nothing worse than getting a call from a customer, or a Private Message from a Forum User that says, “I need help. My HD has crashed, and all my backups were on that drive.”

Let me Digress for a moment, and share with you my personal experience with backups. It’s sort of a Legend here at Zimbra. It all begins in 2005, Zimbra was the new kid on the block and I was an inexperienced System Linux admin for Tombstone Unified School District in Arizona. This was a small district with limited funds. When I interviewed for the post, the Superintendent handed me his card, and it had a Hotmail address on it. Right then, I knew this would be quite a difficult job.

One of the first things I did, was investigate E-mail Server platforms, and TCO (Total Cost of Ownership). A quick search on the interwebs (which is a series of tubes, not to be confused with the you tubes), yielded this new thing called a Zimbra. Much like our millions of Users, I downloaded it and tried it. It worked great! We used the Zimbra Open Source Edition (then it was in Beta).

I was so proud of my cool new setup, and our staff were so excited to finally have good e-mail and calendaring. In knew the importance of keeping backups, and archiving. All of you System Admins at Hospitals and Government institutions know what I mean. You get well acquainted with the Laws and Requirements. So, I would stay up until about midnight, and stop Zimbra, and rsync /opt/zimbra into /opt/zimbra/backups. Keep in mind that this was in the early days of Zimbra, and I wasn’t even an employee yet. As a matter of fact, myself and several others, pioneered the Open Source backup procedure.

Everything was great until one night I noticed I was running low on Disk Space on my /opt partition. So, I thought, “If I just remove all my backups, and make a fresh one, that will save me a bunch of space”. So I ran the following command as root:
rm -rf /opt/zimbra backups
Now, back then, we mounted a clamav partition ramdisk for quarantine purposes. The only indication that I had that something was wrong was when I got an error saying that it couldn’t unmount the partition because it was in use. Everything else in /opt/zimbra was gone…including my backups.

As most of you admins know, preserving data is important. We were involved in several litigation matters, and I would later be cited for obstruction of justice for destroying evidence.

When I noticed what had happened, I immediately called Zimbra and talked with MarcMac here at Zimbra. He tried to recover the inodes using Midnight Commander, but it was a total loss.

Lesson Learned. So, from one admin to another, please take time to make sure your backups are not located on the same machine that Zimbra is on. Please! We never want to hear about data loss. Whether an opensource user or network user, I hope you will take a few minutes to consider your backup strategy, and fix any single points in failure.

Learn from my experience.
-John (jholder)

We’ve had some great responses to my blog post yesterday about changing the default mailto links using Firefox. There’s a couple of problems with my method: 1) It’s not really scalable for an admin [ie no auto install] 2) Not really applicable for Safari and Internet Explorer.

(more…)

When you click on a mailto: link, most of the time your Firefox will launch that version of Outlook Express or Apple Mail you never configured. For the novice user, they click next, next, next, next, then send their e-mail…and it never gets delivered. Well using a greasemonkey script that we’ll convert to a firefox extension, we can handle the mailto links.

(more…)

In this Zimbra Administrator’s PowerTip, we’ll discuss how to move your instance of Zimbra to another machine. It applies to all version of Zimbra.

(more…)

Configuring BIND and Zimbra
on the same machine is a hot topic over in the forums. Zimbra checks to make sure that you have the correct MX an A records. In this Administrator’s PowerTip, we’ll discuss how to install and setup BIND and Zimbra on the same machine.

(more…)

The reasons can vary from an emergency restore of user accounts, to changing system architecture. In this Administrator’s PowerTip we will discuss how to export(dump) LDAP data to disk, and re import it.

(more…)

As many of our Administrators know, Zimbra Collaboration Suite logs just about everything. One of the most integral is the mailbox.log located in /opt/zimbra/log .

In this Administrator’s PowerTip, we’ll cover some common entries in the mailbox.log file, and what they mean.
*Note- This is an administrator PowerTip intended for System Administrators of Zimbra Collaboration Suite.

(more…)