Zimbra

A year an a half ago the (tiny) Yahoo! Calendar team embarked on a mission to build a new Calendar. We were interested in cracking the consumer market where huge potential for growth and innovation lay. The problem was that the 10 year old platform was falling apart and being held together by bungee cord and tape. Innovation on this platform would have been very challenging and forever handicap our efforts going forward.

Zimbra came into the Yahoo! fold and along with it a huge opportunity (large short-term technical challenges as well!) Zimbra’s underlying technology is ripe for customization, and the Yahoo! Calendar Team dove in and came up with some fantastic results.

The all new Yahoo! Calendar Beta is running an a Zimbra back-end which has been embedded into the Yahoo! architecture, with a brand-spanking new front-end composed of JSP enhancements, a new taglib, YUI, and of course AJAX.

 

This is an early beta product where we focused on getting the fundamentals right first. In coming releases expect to see some exciting enhancements. Our Flickr integration is a hint at where we are taking calendaring; its functionality exposes the power of calendars to be a window to discovering interesting events and content, as well as a window to the past.

      

The teams working together from one code repository was an awesome experience, and this cooperation will continue to bring much more innovation across the Yahoo! network. Lots of great calendar code and ideas have come out of this collaboration, look for some in upcoming ZCS releases or check some out in the main branch in perforce.

How can you get on board early? Just visit: switch.calendar.yahoo.com

Checkout some screencasts of the new calendar in action is here, and a video of Scott Dietzen discussing it on All things Digital.

What else are we up to? John Holder is playing host for CalConnect Roundtable XIII this week at one of Yahoo!’s campuses in Santa Carla, CA - we’re collaborating with some big corporations including Sun, Google, Apple, Microsoft, and, Kerio, as well as some major university’s to bring the latest CalDAV & iCalendar specs your way.

 

---

Herbert Wang is a Product Manager on the Zimbra-Yahoo! Calendaring Team.

One of the more nifty features to grace the Zimbra scene recently is the interoperability framework for sharing two-way free/busy information with other server platforms.

Since we released the framework APIs and the reference implementation against Microsoft’s Exchange 2003 (previously covered) there has been a lot of interest from customers and community (interop works with both Open Source and Network Edition).

Argonne National Labs has given excellent feedback culminating in a few enhancements for ZCS 5.0.10. We also recently got a wonderful thank you note from the University of Pennsylvania, who teamed up with the folks over at Sumatra Development to handle some calendar migrations. They were impressed at how well their multi-domain environment behaved, and shared a link to a configuration tip for Exchange 2007. It’s great to see the community enhance, extend, and tweak the open source interop framework.

That type of integration cohesiveness frequently makes Zimbra relevant to organizations in the same way that other open source business application are: often initially at the division level, and then spreading within the enterprise. (Penn breaks their IT into “local support providers” to better serve each school’s specialized needs.) For immense corporations wanting to switch from software such as Exchange, Lotus, Meeting Maker, or other third-parties that interact with our API, picking a new platform can be a massive undertaking - having interop can mean a safe departmental decision.

Admins out there can certainly attest the the headaches involved with maintaining different server infrastructures, but it also works in reverse - that ’stubborn group’ which doesn’t want to switch or the ‘peer organization running different software’ can now seamlessly communicate as one.

Someone in the forums recently asked about ways to migrate individual accounts from one ZCS instance to another, so thought I’d share the enlightenment with all. Whether you are going from an on-premise install to a hosting provider, want to create handy archives of old employee accounts, or just need to duplicate mailbox contents of a user; the syntax in this article proves remarkably useful, and applies to all editions.

There are a multitude of comparable RFE’s on addressing this need via different approaches. (Bugzilla entries 19630, 29573, 28443 & 30163 to name a few.) Some want graphical tools to browse data and selectively migrate certain things, while others would be happy with a cross LDAP zmmailboxmove.

Depending on your situation, several backup tools can take care of a large portion of your daily needs; and there are ways to do Zimbra-to-Zimbra migrations using the Network Edition’s backup and restore capabilities - however they require admin abilities on both systems. Meanwhile, most of the frequently used open source backup solutions are simply an “all accounts at-once” approach. So what to do when you need to move from your personal setup to a hosting provider? Or if you’re a hosting provider, move a tiny handful of accounts to a separate infrastructure? Before diving into the wiki on user migration for info on Imapsync, REST exports, CURL imports, etc; there’s a handy way to avoid the “one item type at-a-time” transfer methods.

In ZCS 5.0.9+ you can export an entire mailbox with:
/opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL “//?fmt=tgz” > /tmp/account.tgz

Next transfer via rsync, scp, sftp, etc. You’ll also need to create the account on the 2nd server if the desired account doesn’t exist at your destination server yet.

Then import with:
/opt/zimbra/bin/zmmailbox -z -m user@domain.com postRestURL “//?fmt=tgz&resolve=reset” /tmp/account.tgz

The resolve= paramater has several options:

• “skip” ignores duplicates of old items, it’s also the default conflict-resolution.
• “modify” changes old items.
• “reset” will delete the old subfolder (or entire mailbox if /).
• “replace” will delete and re-enter them.

‘Reset’ will be a bit faster on an empty destination mailbox because it skips most dupe checks.

Note: There were some duplication fixes and additional issues (mainly sync related) corrected with the tar formatter in 5.0.10.

Not a Zimbra Admin? Users can get the same zip/tar formatter on REST URL’s by visiting:
http:// server.domain.com/home/user?fmt=zip&query=is:anywhere

The zip format has been around for a long time, but doesn’t contain account & item metadata like the tar formatter automatically does:
http:// server.domain.com/home/user/?fmt=tgz

Infact, this same technique is currently used in Zimbra Desktop’s alpha backup solution.

If this approach doesn’t scale performance wise for your situation, or you simply don’t want to have everyone hit a REST URL for 30GB mailboxes all at the same time, here’s a collection of helpful scripts and other ways to systematically migrate:

Mysqldump & rsync with an interesting blob management technique: Zimbra2Zimbra

Imapsync for mail + postRestURL for contacts, calendar & filters: ZimbraMigrate (Expand the concept for tasks, documents, and briefcase items.)

Another method that could be extrapolated upon for migrations: Per User Mailbox Backup (OE Version - Zimbra :: Wiki)

Most of these solutions aren’t going to respect share permissions, but when pulling an account out of an environment that’s to be expected.

Zimlet spin-offs:
- Mail backup options for end users (.eml)
- Zimlet to save email in a txt file (.txt or html)

The above Zimlets are aimed at making quick self-copies & not for restores, but there are many methods for putting messages back into Zimbra, including tools like zmmailbox addMessge, zmlmtpinject, CURL, etc; for more info checkout these threads: Recover data from store folders & Moving Folders between users

If moving your entire server, I’m a huge fan of the install.sh -s trick when using NE backups to do so isn’t an option.

---

Have another method you’d like to share? Document it in the wiki & note it below, or you can discuss over in the Community Forums.

Part 1 covered Network Edition backup features, today’s snips apply to all editions.

First among the lesser known additions: We recently provided the possibility for a nice performance boost to some environments by adding the ability to turn on batched indexing in ZCS 5.0.3 (you can even fine tune it at the localconfig, COS, and account level). We’re not talking about when you re-index an entire account here, this is a change to the index-as-received model; now new items can sit in a ‘queue’ (really a ‘indexing deferred’ flag on the mail_items table of the pertaining mboxgroup database in MySQL) to run all at once when it reaches the zimbraBatchedIndexingSize threshold, saving you from all the tiny disk thrashing. It might not be immediately apparent that this works better, but you can mention it in the forums and we’ll show you the evidence to the contrary - it proves expecially useful for POP heavy or ZAD archive accounts.


 New & Enhanced Admin Tools

 

Your /opt/zimbra/bin & /opt/zimbra/libexec directories hold a wealth of tools to make your job easier.

zmdumpenv has been around for a long time, but underutilized - it grabs the basics that you should probably provide with every issue to help others understand where you’re coming from.

When you need to send ad-hoc SOAP commands to the server, the powerful zmsoap takes care of authenticating, generating the envelope, sending the request, and writing the response to stdout.

If your server freezes or is busy, running zmdialog can give that ‘my server hung’ support ticket a purpose. With JDK 1.5 it won’t collect a heap dump so you might also run /opt/zimbra/java/bin/jmap -heap:format=b [/opt/zimbra/log/zmmailboxd_java.pid] however zmdiaglog collects a core dump, from which it should be theoretically possible to get a heap dump. Thread dumps when you kill -QUIT/3 [pid] are helpful too. Info on ways to take them (like /opt/zimbra/libexec/zmmailboxdmgr threaddump) plus a handy script, are here.

There’s the MySQL metadata DB, the Lucene index, and the actual blob files on disk in /opt/zimbra/store. If you can click on a folder in the web UI and get results, it means MySQL is up and happy. If you open a message and get the infamous “missing blob for id” error, it means that the message files on disk aren’t where they’re supposed to be (ie: disk crash, not mounted/permissions, or you’ve recently pointed your volumes incorrectly). There’s all sorts of things you can do in that situation - here’s a more detailed list. Even if you don’t have a good grasp of the mailbox database structure yet, you can appreciate the need to easily determine if those blobs are still available (after you’ve moved whatever you can salvage of the store from corrupted drives to a better location). Enter the zmblobchk utility which can determine what files are missing - there’s also plans for a repair mode, for when you finally realize the blobs are gone and want to get rid of those UI error messages. Of course you’ll next consult the Network Edition’s restoreToTime feature or other backup solution you may have.

When you’re absolutely out of room on the disks housing your DB (seriously put in that purchase order for more storage - it’s cheap these days) in a pinch the optimizeMboxgroups.pl script (available in the public cache and added to the upcoming 5.0.10) can help you recover wasted space in your mail_item, appointment, imap_folder, imap_message, open_conversation, pop3_message, revision, and tombstone tables on each mboxgroup. Just note that it temporarily locks each table, and could use considerable IO while they’re being rebuilt. You can certainly use it pro-actively during a maintenance window to reclaim space as well.

There’s also a new wiki page on statistic collection so you can generate nice charts that help you figure out what you might need to tweak.

We could go on and on about the utilities in those folders, so throw up a test environment and experiment sometime - it may just make life easier when you have hundreds of users breathing down your neck. And if you should ever be ’stumbling around in the dark’ you can always enable additional debug logging to shed more light on the situation.

 

---

You can find help for all the above utilities over in the Community Forums or ask us a question on them below.

It’s truly amazing how excited people get over Zimbra - so thought we’d share some of the ways people show it.

We were pleasantly surprised a few days ago when a school declared a ‘national Zimbra day’ and sent us pictures of cupcakes (that we can virtually enjoy). Why the baked goods? They recently rolled out ZCS and love it. Since we are also launching 5.0.9 today it’s fitting.

As the summer is the perfect time for schools to do system upgrades when students aren’t around, Thunder04’s organization had a conversion party while they worked on their switchover - with some delicious goodies of course.


The Menlo Park City School District (which serves Menlo Park & Atherton in California) would like to wish you all a “Happy Zimbra Conversion or Upgrade Day” as ZCS 5.0.9 has just been released! (This version includes additional beta builds for Ubuntu 8.04 LTS in both x86 & 64-bit as well.)

 

Continuing our earlier advice to take backups frequently, and secure them offsite - thought we’d highlight a few recent administrator related things added to ZCS that you might not have noticed.


 Network Edition Backup Enhancements

 

Speaking of backups, there are some new ways to take them in ZCS 5.0.x. With ever larger quota usage, full backups can often take a while to run, and even incrementals which process the redologs may still be one heck of a job when you’re talking thousands or millions of accounts. Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incremental sessions. As a plus it automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time. This allows you to do a point in time restore for any account. Simply divide your total accounts by the number of groups you choose (zimbraBackupAutoGroupedNumGroups is 7 by default) and that’s how many will get a full backup session each night. Newly provisioned accounts, and accounts whose last backup is a specified number of days older are picked first. (zimbraBackupAutoGroupedInterval is defaulted to 1d)

To save space, and therefore store older backups longer or run them more frequently, you can also auto-compress them with the –zip argument. This isn’t new, but it got improved handling of shared blobs in 5.0.5 as well as a -zipStore mode for speed. You can also adjust the buffer & queue capacity of the backup process, as well as additional options like the level of compression, or the number of archives per person via the backup_zip_copier_private_blob_zips localconfig attribute. Of course you lose the hard linking optimization (speed and space) for blobs that are in an earlier full backup already when working from the same disk - so it’s more advantageous for those off-site single-copies (you do make one often right?). However, there are legitimate uses for running it on your normal backups: Fewer files make it easier to copy or rsync later, and prevents you from running out of inodes. You can also easily delete individual backups rather than running zmbackup -del, and therefore keep just a few really old backups around for whatever compliance reasons you may have.

By-the-way, ZCS 5.0.6 added the ability to easily replay redologs from an arbitrary point in time with zmplayredo should you be in a unique situation that needs it. (Say you’ve been taking snapshot backups and but then need to restore, and you’ve also saved all the redologs since the snapshot. Or you take a snapshot, then manually copy redologs from the live system to bring the snapshot copy up to current. This allows you to force replay of them all and not just the uncommitted transactions.)

We’ve always recommended that: “After upgrade, you should run a full backup immediately as changes in this release invalidate all old backups.” It’s still good advice to have a fresh full, but there’s always the infrequent need to get data from an older minor version backup without throwing up a temporary machine. Well, with 5.0.x we aimed to make restores compatible across patch releases (i.e. an older 5.0.x backup, not a prior 4.5.x backup - major version restore is this RFE). There was a bug about zmrestore not handling database schema changes, but that’s fixed in 5.0.5 and later - so backwards compatibility for restore is now theoretically possible. And we’re also looking to put icing on the cake by adding a conversion tool to upgrade backups themselves to allow restore on later ZCS versions.

 

---

You can find help on backups via Support Channels, over in the Community Forums, or ask us a question on them below. Stay tuned, as Part 2 will cover tidbits for all editions.

This may be the most important post you will ever read. If you’re a Zimbra Administrator, please read, and pass this onto your colleagues who use Zimbra. If you’re a CTO or CEO, take time to ask your Zimbra Admin about the subject of this post. This blog post is about backups.

Whether your an Open Source User, Zimbra Desktop User, or Network Edition Customer, you can do backups of your data. There is nothing worse than getting a call from a customer, or a Private Message from a Forum User that says, “I need help. My HD has crashed, and all my backups were on that drive.”

Let me Digress for a moment, and share with you my personal experience with backups. It’s sort of a Legend here at Zimbra. It all begins in 2005, Zimbra was the new kid on the block and I was an inexperienced System Linux admin for Tombstone Unified School District in Arizona. This was a small district with limited funds. When I interviewed for the post, the Superintendent handed me his card, and it had a Hotmail address on it. Right then, I knew this would be quite a difficult job.

One of the first things I did, was investigate E-mail Server platforms, and TCO (Total Cost of Ownership). A quick search on the interwebs (which is a series of tubes, not to be confused with the you tubes), yielded this new thing called a Zimbra. Much like our millions of Users, I downloaded it and tried it. It worked great! We used the Zimbra Open Source Edition (then it was in Beta).

I was so proud of my cool new setup, and our staff were so excited to finally have good e-mail and calendaring. In knew the importance of keeping backups, and archiving. All of you System Admins at Hospitals and Government institutions know what I mean. You get well acquainted with the Laws and Requirements. So, I would stay up until about midnight, and stop Zimbra, and rsync /opt/zimbra into /opt/zimbra/backups. Keep in mind that this was in the early days of Zimbra, and I wasn’t even an employee yet. As a matter of fact, myself and several others, pioneered the Open Source backup procedure.

Everything was great until one night I noticed I was running low on Disk Space on my /opt partition. So, I thought, “If I just remove all my backups, and make a fresh one, that will save me a bunch of space”. So I ran the following command as root:
rm -rf /opt/zimbra backups
Now, back then, we mounted a clamav partition ramdisk for quarantine purposes. The only indication that I had that something was wrong was when I got an error saying that it couldn’t unmount the partition because it was in use. Everything else in /opt/zimbra was gone…including my backups.

As most of you admins know, preserving data is important. We were involved in several litigation matters, and I would later be cited for obstruction of justice for destroying evidence.

When I noticed what had happened, I immediately called Zimbra and talked with MarcMac here at Zimbra. He tried to recover the inodes using Midnight Commander, but it was a total loss.

Lesson Learned. So, from one admin to another, please take time to make sure your backups are not located on the same machine that Zimbra is on. Please! We never want to hear about data loss. Whether an opensource user or network user, I hope you will take a few minutes to consider your backup strategy, and fix any single points in failure.

Learn from my experience.
-John (jholder)

Now that the iPhone 2.0 software update is here, here’s a sneak peek of what Zimbra Mobile for iPhone looks like.

First, if you are new to Zimbra Mobile it’s part of Zimbra Collaboration Suite Network Edition and provides over-the-air synchronization of emails, appointments and contacts between a user mailbox and their mobile device. “Push” technology can make sure that updates are delivered to mobile devices as they happen in user’s mailbox on the server. Zimbra Mobile supports a wide range of ActiveSync compatible devices, including Microsoft Window Mobile (PocketPCs and Smartphones), Treo Palm OS series, Nokia E series (and other devices running 3rd party clients like RoadSync). Now, the latest additions to our device list are iPhone and iPod Touch running the iPhone 2.0 software. We’ve been tuning and polishing Zimbra Mobile to be a great match for iPhone.

I know everyone has his or her own favorite email gadget but personally, I’m glad I have an iPhone. With iPhone’s large, bright touch screen, instant “push” delivery, HTML email display, support for all sorts of document/media attachments and meeting invitations plus a photo-enabled address book, what else can I wish for? And the best part is Zimbra Mobile for iPhone takes full advantage of all that!

OK enough words. Let’s see what it will look like in action.

Note: Zimbra Collaboration Suite Network Edition 5.0.7 is required for iPhone sync support.

Initial Sync — Pretty Darn Fast You first setup how the iPhone will sync with a Zimbra Server at your company (or at your service provider): Once the connection is established iPhone will start to download your email, calendar and address book data. iPhone is actually powerful enough to download the different data categories in parallel. You’ll see emails start to show up in the Inbox at the same time contacts are added to the address book. I believe the parallel download is a unique feature that I haven’t seen on other devices. Also many other Zimbra Mobile devices typically only download partial envelope information and some text of each message, and get the rest of the message or attachments when the user asks for them. Given that iPhone has enough memory, it can download up to 200 most recent messages in full MIME format so that everything is available on the phone.

Rendering Messages on the iPhone iPhone can view or play many common attachments, including pictures (inline view), PDFs, various audio/video formats (great for receiving voicemail in your Zimbra mailbox), Word docs, PowerPoint slides, and Excel spreadsheets. Here are some examples of messages. I really like how iPhone displays deeply nested replies.

Appointments, Reminders and Calendar Of course Zimbra Mobile for iPhone is not just about email. As iPhone receives calendar events and meeting invitations from Zimbra server, it will alert users of incoming invitations, and allow users to accept or decline meetings from the device. Accepted meetings are added to the calendar, with alarms to go off at scheduled reminder time. Zimbra Mobile for iPhone supports all types of appointments, recurring events with all kinds of exceptions, as well as participants across multiple timezones. <

Address Book Got a big address book? No problem. One of our power users has 8000+ contacts in his address book. iPhone has enough memory to download and hold it all. Contacts with photos can be synced with the Zimbra address book as well, so go take some pictures of your friends with that iPhone camera.

Zimbra Mobile for iPhone is a new feature in Zimbra Collaboration Suite Network Edition 5.0.7, and it requires the iPhone 2.0 software update from Apple. I’m really excited about the Zimbra Mobile for iPhone user experience. This is a great addition to our existing solutions for iPhone which includes ZCS optimized for the mobile Safari browser and our iSync Connector for the Apple Desktop. To find out more about Zimbra Mobile for iPhone, check out the Zimbra Mobile Forums.

---

JJ Zhuang is lead developer for Zimbra Mobile.

This week, Zimbra held a Zimbra Administration Course at the Yahoo! headquarters in Santa Clara. For those who don’t know, the administration course covers just about everything in Zimbra from A to Z. (more…)

With Zimbra 5.0 we’ve introduced some newer ways to make the user experience faster with the Zimbra Web Client. We’ve talked about Jetty, YUI compression, and Lazy Loading, but now there’s just one burning question: Which browser is fastest? (more…)