Someone in the forums recently asked about ways to migrate individual accounts from one ZCS instance to another, so thought I’d share the enlightenment with all. Whether you are going from an on-premise install to a hosting provider, want to create handy archives of old employee accounts, or just need to duplicate mailbox contents of a user; the syntax in this article proves remarkably useful, and applies to all editions.
There are a multitude of comparable RFE’s on addressing this need via different approaches. (Bugzilla entries 19630, 29573, 28443 & 30163 to name a few.) Some want graphical tools to browse data and selectively migrate certain things, while others would be happy with a cross LDAP zmmailboxmove.
Depending on your situation, several backup tools can take care of a large portion of your daily needs; and there are ways to do Zimbra-to-Zimbra migrations using the Network Edition’s backup and restore capabilities – however they require admin abilities on both systems. Meanwhile, most of the frequently used open source backup solutions are simply an “all accounts at-once” approach. So what to do when you need to move from your personal setup to a hosting provider? Or if you’re a hosting provider, move a tiny handful of accounts to a separate infrastructure? Before diving into the wiki on user migration for info on Imapsync, REST exports, CURL imports, etc; there’s a handy way to avoid the “one item type at-a-time” transfer methods.
In ZCS 5.0.9+ you can export an entire mailbox with: /opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL “//?fmt=tgz” > /tmp/account.tgz
Next transfer via rsync, scp, sftp, etc. You’ll also need to create the account on the 2nd server if the desired account doesn’t exist at your destination server yet.
Then import with: /opt/zimbra/bin/zmmailbox -z -m user@domain.com postRestURL “//?fmt=tgz&resolve=reset” /tmp/account.tgz
The resolve= paramater has several options:
“skip” ignores duplicates of old items, it’s also the default conflict-resolution.
“modify” changes old items.
“reset” will delete the old subfolder (or entire mailbox if /).
“replace” will delete and re-enter them.
‘Reset’ will be a bit faster on an empty destination mailbox because it skips most dupe checks.
Note: There were some duplication fixes and additional issues (mainly sync related) corrected with the tar formatter in 5.0.10.
Not a Zimbra Admin? Users can get the same zip/tar formatter on REST URL’s by visiting: http:// server.domain.com/home/user?fmt=zip&query=is:anywhere
The zip format has been around for a long time, but doesn’t contain account & item metadata like the tar formatter automatically does: http:// server.domain.com/home/user/?fmt=tgz
Infact, this same technique is currently used in Zimbra Desktop’s alpha backup solution.
If this approach doesn’t scale performance wise for your situation, or you simply don’t want to have everyone hit a REST URL for 30GB mailboxes all at the same time, here’s a collection of helpful scripts and other ways to systematically migrate:
Mysqldump & rsync with an interesting blob management technique: Zimbra2Zimbra
Imapsync for mail + postRestURL for contacts, calendar & filters: ZimbraMigrate (Expand the concept for tasks, documents, and briefcase items.)
The above Zimlets are aimed at making quick self-copies & not for restores, but there are many methods for putting messages back into Zimbra, including tools like zmmailbox addMessge, zmlmtpinject, CURL, etc; for more info checkout these threads: Recover data from store folders & Moving Folders between users
If moving your entire server, I’m a huge fan of the install.sh -s trick when using NE backups to do so isn’t an option.
Have another method you’d like to share? Document it in the wiki & note it below, or you can discuss over in the Community Forums.
In 2003, we founded Zimbra because we thought that existing e-mail and calendaring solutions were broken and we knew we could create something much better – that something is the Zimbra Collaboration Suite (ZCS), an Ajax-based collaboration client that integrates email, contacts, shared calendar, instant message, documents with sharing capabilities, advanced search and VoIP into a browser-based interface. We owe the thousands of members of the Zimbra community many thanks for their ongoing contributions – together, we’ve achieved great things!
We’ve given customers “new and improved” mailboxes and we’ve focused on creating the best possible experience for collaboration – streamlining overflowing inboxes, organizing correspondence, and reducing the hassles of managing communication tools on the back-end.
A year ago, we joined the Yahoo! family to extend Zimbra’s reach, share our expertise to one of the top mail services in the world – Yahoo! Mail – and to continue to change the face of how users collaborate at school, work, and home. Yahoo! has given us the resources, including greater computing power, to continue to expand and update ZCS with new features, and support an ever-growing customer base.
We’re proud of our accomplishments over the past year. On the customer front, we welcomed a number of world-class organizations. Stanford, UMass (Dartmouth), UPenn, CalPoly, and Texas A&M are among the institutions that drove well over a million mailboxes sold in the education market this year, driving record growth across all the markets we serve. There are now over 15 million Zimbra mailboxes deployed, serving over 25,000 customers in 82 countries, expanding our world-class reach. That reach has been further expanded through partnerships with Red Hat, Apple, and Ubuntu.
This past year also reflects our ongoing commitment to innovation as demonstrated by the stream of enhancements to the Zimbra product family:
• The launch of ZCS 5.0 extended BlackBerry and Outlook 2007 support plus Web 2.0 IM and task applications;
• Yahoo! Zimbra Desktop – currently in beta – provides users with offline access to a simple, centralized place to manage work, school and personal e-mail without an Internet connection;
• Extensive mobile options give our users anytime, anywhere access to ZCS and extend Zimbra to the broadest range of devices available in the market. Mobile options now include iPhone 2.0, smartphones (Blackberry, iPhone, Treo, etc.) and any Java-enabled mobile device (Nokia E & N Series, Motorola RAZR, ROKR etc.). These build on Yahoo!’s leadership in e-mail and mobile Web services as a key starting point for consumers.
We’ve got tons more in the pipeline. Later this year, we’ll be debuting new products that continue to make collaboration a superior experience, such as the general availability for Yahoo! Zimbra Desktop and exciting cloud services for universities and businesses. Look out for more developments with the Open Mail initiative that debuted at Yahoo! Hack Day last week, along with other cool synergies with Yahoo! Mail and Calendar.
Posted in /etc, Community by Ramesh May on September 12th, 2008
Zimbra is presenting a session on Zimlets at Yahoo’s Open Hack 2008. The session starts at Noon PDT, and we’ll be live blogging here. Stay tuned. Hack Day Blog – Hack Day Twitter – OpenHack Zimlet Wiki
HackDay just kicked off, and we’ve seen all sorts of things go from idea to prototype in just 24 hours. This year has a twist: Not only are our engineers across the globe involved (with many making it an all-nighter) but everyone is invited to participate – even you. Open Hack 2008 takes place September 12-13th where anyone with an idea is encouraged to gather a team up, then spend a day building stuff that they think is cool. HackU (the University Hackdown) is even flying in its top ranks, with a few Zimbra customers among them (including Stanford & Georgia Tech) to join us at Yahoo! HQ in Sunnyvale, CA for rounds of coding; plus camaraderie, food, demos, awards, and good music. From our team KevinH & JohnH are also giving several presentations throughout the event.
Some of the things we’re ran across have given us ideas for Zimlets – so we’re hereby seeking Zimlet beta testers. This will be an ongoing project, meaning that those who volunteer will get continuous access to the latest and greatest Zimlet ideas.
We need volunteers that:
Are using either ZCS (with the ability to deploy Zimlets – so essentially administrators of the respective system) or Zimbra Desktop users.
Are willing to try out different services that some of these may link to, and consciously note how they affect their daily Zimbra experience.
Of course give us feedback about their place in productivity, effectiveness, usefulness, and anything you’d like to see added or extended.
To join in just send me a PM/Email by the end of this week. (We’ll be sure to reward you for your efforts.)
We can’t take everyone, so if you not accepted don’t feel bad – there’s still plenty of cool & useful Zimlets over in the Gallery – plus they’ll soon be making it off engineer workstations and into perforce. We’re even working on a way to make them easier to install in Zimbra Desktop, but you can find current directions here.
Those on the development side will soon see a few community members marked “Zimlet Guru” – if you’ve created a few yourself, and are into helping out others in the Zimlet section of the forums, be sure to drop me a line.
Hey, did you hear that Google released a browser? Yeah, and it’s very cool! We might have been a bit early to call Safari the Browser war winner. Based on WebKit (KHTML), this rendering framework (that Chrome uses) has really stormed the market. If you asked us five months ago who was winning the browser war, we would easily say Firefox, with Safari as a close second. With the introduction of Chrome, a new war has started.
At the start of this century, the war was about “Open-ness” and who could be more open and win the hearts of users. Now it’s a war of speed, and who’s faster. A few blogs and articles have been written, with Mozilla and Google both claiming their JS engine is faster. So who’s right? Both are faster than IE (6, 7, and 8), but in our opinion, what matters is how responsive web applications are. So who will win Zimbra’s Speed trophy?
Zimbra has a testing harness thats in alpha which we will be making available to the public in the future, that measures performance on different actions within Zimbra. This helps us understand what the end user is seeing. People can talk V8 Benchmark, Dromaeo, SunSpider, or what ever they want. What really matters is how applications perform. Our tests are pure UI performance, ie, how fast Zimbra is to the end user.
Considering that one of Zimbra’s strengths is our AJAX web interface, we decided to put Chrome to the test, along with IE, FireFox, and Safari. The control system was: Intel Core2 duo, 2.39Ghz 1.99GB RAM Windows XP
Here’s how it did (lower is faster):
Overall Performance
All Tests
Given that Chrome is built on WebKit, this didn’t come as any particularly huge surprise. In our previous tests, Safari came out the fastest renderer of the Zimbra Web Client. In our tests, Chrome came in as a very close second, and we expect it to get faster.
We want Chrome to work as good as FireFox or Internet Explorer. So, if you find an issue, please report it in the bug report below.
Want more info on the browser war? Check out these links:
Part 1 covered Network Edition backup features, today’s snips apply to all editions.
First among the lesser known additions: We recently provided the possibility for a nice performance boost to some environments by adding the ability to turn on batched indexing in ZCS 5.0.3 (you can even fine tune it at the localconfig, COS, and account level). We’re not talking about when you re-index an entire account here, this is a change to the index-as-received model; now new items can sit in a ‘queue’ (really a ‘indexing deferred’ flag on the mail_items table of the pertaining mboxgroup database in MySQL) to run all at once when it reaches the zimbraBatchedIndexingSize threshold, saving you from all the tiny disk thrashing. It might not be immediately apparent that this works better, but you can mention it in the forums and we’ll show you the evidence to the contrary – it proves expecially useful for POP heavy or ZAD archive accounts.
New & Enhanced Admin Tools
Your /opt/zimbra/bin & /opt/zimbra/libexec directories hold a wealth of tools to make your job easier.
zmdumpenv has been around for a long time, but underutilized – it grabs the basics that you should probably provide with every issue to help others understand where you’re coming from.
When you need to send ad-hoc SOAP commands to the server, the powerful zmsoap takes care of authenticating, generating the envelope, sending the request, and writing the response to stdout.
If your server freezes or is busy, running zmdialog can give that ‘my server hung’ support ticket a purpose. With JDK 1.5 it won’t collect a heap dump so you might also run /opt/zimbra/java/bin/jmap -heap:format=b [/opt/zimbra/log/zmmailboxd_java.pid] however zmdiaglog collects a core dump, from which it should be theoretically possible to get a heap dump. Thread dumps when you kill -QUIT/3 [pid] are helpful too. Info on ways to take them (like /opt/zimbra/libexec/zmmailboxdmgr threaddump) plus a handy script, are here.
There’s the MySQL metadata DB, the Lucene index, and the actual blob files on disk in /opt/zimbra/store. If you can click on a folder in the web UI and get results, it means MySQL is up and happy. If you open a message and get the infamous “missing blob for id” error, it means that the message files on disk aren’t where they’re supposed to be (ie: disk crash, not mounted/permissions, or you’ve recently pointed your volumes incorrectly). There’s all sorts of things you can do in that situation – here’s a more detailed list. Even if you don’t have a good grasp of the mailbox database structure yet, you can appreciate the need to easily determine if those blobs are still available (after you’ve moved whatever you can salvage of the store from corrupted drives to a better location). Enter the zmblobchk utility which can determine what files are missing – there’s also plans for a repair mode, for when you finally realize the blobs are gone and want to get rid of those UI error messages. Of course you’ll next consult the Network Edition’s restoreToTime feature or other backup solution you may have.
When you’re absolutely out of room on the disks housing your DB (seriously put in that purchase order for more storage – it’s cheap these days) in a pinch the optimizeMboxgroups.pl script (available in the public cache and added to the upcoming 5.0.10) can help you recover wasted space in your mail_item, appointment, imap_folder, imap_message, open_conversation, pop3_message, revision, and tombstone tables on each mboxgroup. Just note that it temporarily locks each table, and could use considerable IO while they’re being rebuilt. You can certainly use it pro-actively during a maintenance window to reclaim space as well.
There’s also a new wiki page on statistic collection so you can generate nice charts that help you figure out what you might need to tweak.
We could go on and on about the utilities in those folders, so throw up a test environment and experiment sometime – it may just make life easier when you have hundreds of users breathing down your neck. And if you should ever be ’stumbling around in the dark’ you can always enable additional debug logging to shed more light on the situation.
You can find help for all the above utilities over in the Community Forums or ask us a question on them below.
Infact, this same technique is currently used in Zimbra Desktop’s alpha backup solution.
Hey, did you hear that Google released a browser? Yeah, and it’s very cool! We might have been a bit early to call Safari the Browser war winner. Based on WebKit (KHTML), this rendering framework (that Chrome uses) has really stormed the market. If you asked us five months ago who was winning the browser war, we would easily say Firefox, with Safari as a close second. With the introduction of Chrome, a new war has started. 
