Zimbra

Zimbra is presenting a session on Zimlets at Yahoo’s Open Hack 2008. The session starts at Noon PDT, and we’ll be live blogging here. Stay tuned. Hack Day Blog - Hack Day Twitter - OpenHack Zimlet Wiki

HackDay just kicked off, and we’ve seen all sorts of things go from idea to prototype in just 24 hours. This year has a twist: Not only are our engineers across the globe involved (with many making it an all-nighter) but everyone is invited to participate - even you. Open Hack 2008 takes place September 12-13th where anyone with an idea is encouraged to gather a team up, then spend a day building stuff that they think is cool. HackU (the University Hackdown) is even flying in its top ranks, with a few Zimbra customers among them (including Stanford & Georgia Tech) to join us at Yahoo! HQ in Sunnyvale, CA for rounds of coding; plus camaraderie, food, demos, awards, and good music. From our team KevinH & JohnH are also giving several presentations throughout the event.

Some of the things we’re ran across have given us ideas for Zimlets - so we’re hereby seeking Zimlet beta testers. This will be an ongoing project, meaning that those who volunteer will get continuous access to the latest and greatest Zimlet ideas.

We need volunteers that:

• Are using either ZCS (with the ability to deploy Zimlets - so essentially administrators of the respective system) or Zimbra Desktop users.

• Are willing to try out different services that some of these may link to, and consciously note how they affect their daily Zimbra experience.
• Of course give us feedback about their place in productivity, effectiveness, usefulness, and anything you’d like to see added or extended.

To join in just send me a PM/Email by the end of this week. (We’ll be sure to reward you for your efforts.)

We can’t take everyone, so if you not accepted don’t feel bad - there’s still plenty of cool & useful Zimlets over in the Gallery - plus they’ll soon be making it off engineer workstations and into perforce. We’re even working on a way to make them easier to install in Zimbra Desktop, but you can find current directions here.

Those on the development side will soon see a few community members marked “Zimlet Guru” - if you’ve created a few yourself, and are into helping out others in the Zimlet section of the forums, be sure to drop me a line.

Hey, did you hear that Google released a browser? Yeah, and it’s very cool! We might have been a bit early to call Safari the Browser war winner. Based on WebKit (KHTML), this rendering framework (that Chrome uses) has really stormed the market. If you asked us five months ago who was winning the browser war, we would easily say Firefox, with Safari as a close second. With the introduction of Chrome, a new war has started.

At the start of this century, the war was about “Open-ness” and who could be more open and win the hearts of users. Now it’s a war of speed, and who’s faster. A few blogs and articles have been written, with Mozilla and Google both claiming their JS engine is faster. So who’s right? Both are faster than IE (6, 7, and 8), but in our opinion, what matters is how responsive web applications are. So who will win Zimbra’s Speed trophy?

Zimbra has a testing harness thats in alpha which we will be making available to the public in the future, that measures performance on different actions within Zimbra. This helps us understand what the end user is seeing. People can talk V8 Benchmark, Dromaeo, SunSpider, or what ever they want. What really matters is how applications perform. Our tests are pure UI performance, ie, how fast Zimbra is to the end user.

Considering that one of Zimbra’s strengths is our AJAX web interface, we decided to put Chrome to the test, along with IE, FireFox, and Safari. The control system was: Intel Core2 duo, 2.39Ghz 1.99GB RAM Windows XP

Here’s how it did (lower is faster):

Overall Performance

---

All Tests

---

Given that Chrome is built on WebKit, this didn’t come as any particularly huge surprise. In our previous tests, Safari came out the fastest renderer of the Zimbra Web Client. In our tests, Chrome came in as a very close second, and we expect it to get faster.

We want Chrome to work as good as FireFox or Internet Explorer. So, if you find an issue, please report it in the bug report below.

Want more info on the browser war? Check out these links:

Who won the browser war? - http://www.zimbrablog.com/blog/archives/2008/06/and-the-winner-is.html
Safari vs Safari- http://www.zimbrablog.com/blog/archives/2008/06/browser-war-part-3-safari-311-nightlies.html
IE vs IE - http://www.zimbrablog.com/blog/archives/2008/06/browser-war-part-2-ie7-vs-ie8b.html
FF vs FF - http://www.zimbrablog.com/blog/archives/2008/05/round-1-ff2-vs-ff3rc1.html
Support Opera for Zimbra Web Client - http://bugzilla.zimbra.com/show_bug.cgi?id=5932
Hack the Zimbra Web Client to Support Chrome - http://www.zimbra.com/forums/users/21903-googles-chrome-browser.html

Part 1 covered Network Edition backup features, today’s snips apply to all editions.

First among the lesser known additions: We recently provided the possibility for a nice performance boost to some environments by adding the ability to turn on batched indexing in ZCS 5.0.3 (you can even fine tune it at the localconfig, COS, and account level). We’re not talking about when you re-index an entire account here, this is a change to the index-as-received model; now new items can sit in a ‘queue’ (really a ‘indexing deferred’ flag on the mail_items table of the pertaining mboxgroup database in MySQL) to run all at once when it reaches the zimbraBatchedIndexingSize threshold, saving you from all the tiny disk thrashing. It might not be immediately apparent that this works better, but you can mention it in the forums and we’ll show you the evidence to the contrary - it proves expecially useful for POP heavy or ZAD archive accounts.


 New & Enhanced Admin Tools

 

Your /opt/zimbra/bin & /opt/zimbra/libexec directories hold a wealth of tools to make your job easier.

zmdumpenv has been around for a long time, but underutilized - it grabs the basics that you should probably provide with every issue to help others understand where you’re coming from.

When you need to send ad-hoc SOAP commands to the server, the powerful zmsoap takes care of authenticating, generating the envelope, sending the request, and writing the response to stdout.

If your server freezes or is busy, running zmdialog can give that ‘my server hung’ support ticket a purpose. With JDK 1.5 it won’t collect a heap dump so you might also run /opt/zimbra/java/bin/jmap -heap:format=b [/opt/zimbra/log/zmmailboxd_java.pid] however zmdiaglog collects a core dump, from which it should be theoretically possible to get a heap dump. Thread dumps when you kill -QUIT/3 [pid] are helpful too. Info on ways to take them (like /opt/zimbra/libexec/zmmailboxdmgr threaddump) plus a handy script, are here.

There’s the MySQL metadata DB, the Lucene index, and the actual blob files on disk in /opt/zimbra/store. If you can click on a folder in the web UI and get results, it means MySQL is up and happy. If you open a message and get the infamous “missing blob for id” error, it means that the message files on disk aren’t where they’re supposed to be (ie: disk crash, not mounted/permissions, or you’ve recently pointed your volumes incorrectly). There’s all sorts of things you can do in that situation - here’s a more detailed list. Even if you don’t have a good grasp of the mailbox database structure yet, you can appreciate the need to easily determine if those blobs are still available (after you’ve moved whatever you can salvage of the store from corrupted drives to a better location). Enter the zmblobchk utility which can determine what files are missing - there’s also plans for a repair mode, for when you finally realize the blobs are gone and want to get rid of those UI error messages. Of course you’ll next consult the Network Edition’s restoreToTime feature or other backup solution you may have.

When you’re absolutely out of room on the disks housing your DB (seriously put in that purchase order for more storage - it’s cheap these days) in a pinch the optimizeMboxgroups.pl script (available in the public cache and added to the upcoming 5.0.10) can help you recover wasted space in your mail_item, appointment, imap_folder, imap_message, open_conversation, pop3_message, revision, and tombstone tables on each mboxgroup. Just note that it temporarily locks each table, and could use considerable IO while they’re being rebuilt. You can certainly use it pro-actively during a maintenance window to reclaim space as well.

There’s also a new wiki page on statistic collection so you can generate nice charts that help you figure out what you might need to tweak.

We could go on and on about the utilities in those folders, so throw up a test environment and experiment sometime - it may just make life easier when you have hundreds of users breathing down your neck. And if you should ever be ’stumbling around in the dark’ you can always enable additional debug logging to shed more light on the situation.

 

---

You can find help for all the above utilities over in the Community Forums or ask us a question on them below.

It’s truly amazing how excited people get over Zimbra - so thought we’d share some of the ways people show it.

We were pleasantly surprised a few days ago when a school declared a ‘national Zimbra day’ and sent us pictures of cupcakes (that we can virtually enjoy). Why the baked goods? They recently rolled out ZCS and love it. Since we are also launching 5.0.9 today it’s fitting.

As the summer is the perfect time for schools to do system upgrades when students aren’t around, Thunder04’s organization had a conversion party while they worked on their switchover - with some delicious goodies of course.


The Menlo Park City School District (which serves Menlo Park & Atherton in California) would like to wish you all a “Happy Zimbra Conversion or Upgrade Day” as ZCS 5.0.9 has just been released! (This version includes additional beta builds for Ubuntu 8.04 LTS in both x86 & 64-bit as well.)

 

Continuing our earlier advice to take backups frequently, and secure them offsite - thought we’d highlight a few recent administrator related things added to ZCS that you might not have noticed.


 Network Edition Backup Enhancements

 

Speaking of backups, there are some new ways to take them in ZCS 5.0.x. With ever larger quota usage, full backups can often take a while to run, and even incrementals which process the redologs may still be one heck of a job when you’re talking thousands or millions of accounts. Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions - you’re completely backing up a target number of accounts daily rather than running incremental sessions. As a plus it automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time. This allows you to do a point in time restore for any account. Simply divide your total accounts by the number of groups you choose (zimbraBackupAutoGroupedNumGroups is 7 by default) and that’s how many will get a full backup session each night. Newly provisioned accounts, and accounts whose last backup is a specified number of days older are picked first. (zimbraBackupAutoGroupedInterval is defaulted to 1d)

To save space, and therefore store older backups longer or run them more frequently, you can also auto-compress them with the –zip argument. This isn’t new, but it got improved handling of shared blobs in 5.0.5 as well as a -zipStore mode for speed. You can also adjust the buffer & queue capacity of the backup process, as well as additional options like the level of compression, or the number of archives per person via the backup_zip_copier_private_blob_zips localconfig attribute. Of course you lose the hard linking optimization (speed and space) for blobs that are in an earlier full backup already when working from the same disk - so it’s more advantageous for those off-site single-copies (you do make one often right?). However, there are legitimate uses for running it on your normal backups: Fewer files make it easier to copy or rsync later, and prevents you from running out of inodes. You can also easily delete individual backups rather than running zmbackup -del, and therefore keep just a few really old backups around for whatever compliance reasons you may have.

By-the-way, ZCS 5.0.6 added the ability to easily replay redologs from an arbitrary point in time with zmplayredo should you be in a unique situation that needs it. (Say you’ve been taking snapshot backups and but then need to restore, and you’ve also saved all the redologs since the snapshot. Or you take a snapshot, then manually copy redologs from the live system to bring the snapshot copy up to current. This allows you to force replay of them all and not just the uncommitted transactions.)

We’ve always recommended that: “After upgrade, you should run a full backup immediately as changes in this release invalidate all old backups.” It’s still good advice to have a fresh full, but there’s always the infrequent need to get data from an older minor version backup without throwing up a temporary machine. Well, with 5.0.x we aimed to make restores compatible across patch releases (i.e. an older 5.0.x backup, not a prior 4.5.x backup - major version restore is this RFE). There was a bug about zmrestore not handling database schema changes, but that’s fixed in 5.0.5 and later - so backwards compatibility for restore is now theoretically possible. And we’re also looking to put icing on the cake by adding a conversion tool to upgrade backups themselves to allow restore on later ZCS versions.

 

---

You can find help on backups via Support Channels, over in the Community Forums, or ask us a question on them below. Stay tuned, as Part 2 will cover tidbits for all editions.

Linux users have pondered: “Wouldn’t it be nice to just grab Zimbra software via repositories?”

There was just so much positive feedback over Zimbra Desktop Beta 3 (with over 900 new members to the forums last month!) we thought that Ubuntu 32-bit users should be able to grab it easily. (64-bit support is coming soon)

This brings Zimbra Desktop’s easy setup against ZCS, Yahoo! Mail, Gmail, AOL and any other IMAP/POP accounts to the huge Ubuntu community - letting them take mail, contacts, calendars, tasks, documents, and briefcase items offline and sync whatever actions they may take when they’re reconnected.

To install:
sudo synaptic (As you’re read this it’s being added to Applications > Add/Remove.)
Enable the third party packages in settings > repositories.
Reload, search for “Zimbra”, and away you go. (same as a sudo apt-get install zdesktop)

Once downloaded, start via Applications > Internet:



Many thanks to the Canonical Team for showcasing us on their frontpage: ubuntu.com/news/zimbra-desktop

Not using Ubuntu? It’s cross-platform for other Linux variants, as well as Windows and Mac - grab it here.

You may ask: “What about the entirety of the Zimbra Collaboration Suite Server in Ubuntu or other repositories?” We can’t say one way or the other at this point - but think of this as a harmonious step.

 

---

Find help for Zimbra Desktop over in the Community Forums, ask us a question below, or fill out the Ubuntu registration/feedback on it: ubuntu.com/register/zimbra/

This may be the most important post you will ever read. If you’re a Zimbra Administrator, please read, and pass this onto your colleagues who use Zimbra. If you’re a CTO or CEO, take time to ask your Zimbra Admin about the subject of this post. This blog post is about backups.

Whether your an Open Source User, Zimbra Desktop User, or Network Edition Customer, you can do backups of your data. There is nothing worse than getting a call from a customer, or a Private Message from a Forum User that says, “I need help. My HD has crashed, and all my backups were on that drive.”

Let me Digress for a moment, and share with you my personal experience with backups. It’s sort of a Legend here at Zimbra. It all begins in 2005, Zimbra was the new kid on the block and I was an inexperienced System Linux admin for Tombstone Unified School District in Arizona. This was a small district with limited funds. When I interviewed for the post, the Superintendent handed me his card, and it had a Hotmail address on it. Right then, I knew this would be quite a difficult job.

One of the first things I did, was investigate E-mail Server platforms, and TCO (Total Cost of Ownership). A quick search on the interwebs (which is a series of tubes, not to be confused with the you tubes), yielded this new thing called a Zimbra. Much like our millions of Users, I downloaded it and tried it. It worked great! We used the Zimbra Open Source Edition (then it was in Beta).

I was so proud of my cool new setup, and our staff were so excited to finally have good e-mail and calendaring. In knew the importance of keeping backups, and archiving. All of you System Admins at Hospitals and Government institutions know what I mean. You get well acquainted with the Laws and Requirements. So, I would stay up until about midnight, and stop Zimbra, and rsync /opt/zimbra into /opt/zimbra/backups. Keep in mind that this was in the early days of Zimbra, and I wasn’t even an employee yet. As a matter of fact, myself and several others, pioneered the Open Source backup procedure.

Everything was great until one night I noticed I was running low on Disk Space on my /opt partition. So, I thought, “If I just remove all my backups, and make a fresh one, that will save me a bunch of space”. So I ran the following command as root:
rm -rf /opt/zimbra backups
Now, back then, we mounted a clamav partition ramdisk for quarantine purposes. The only indication that I had that something was wrong was when I got an error saying that it couldn’t unmount the partition because it was in use. Everything else in /opt/zimbra was gone…including my backups.

As most of you admins know, preserving data is important. We were involved in several litigation matters, and I would later be cited for obstruction of justice for destroying evidence.

When I noticed what had happened, I immediately called Zimbra and talked with MarcMac here at Zimbra. He tried to recover the inodes using Midnight Commander, but it was a total loss.

Lesson Learned. So, from one admin to another, please take time to make sure your backups are not located on the same machine that Zimbra is on. Please! We never want to hear about data loss. Whether an opensource user or network user, I hope you will take a few minutes to consider your backup strategy, and fix any single points in failure.

Learn from my experience.
-John (jholder)

We’ve aimed to blur the line between a Ajax web-client and a conventional desktop application, and this release is a leap towards reaching that goal. If you’re just joining us here’s the best part: It’s an offline capable client so you can take your data with you whenever you don’t have internet access - then sync any type of interaction that you can do in normal webmail access when you get connected again. So many cool new things I don’t know where to begin - the Zimbra Desktop team has been very busy since Beta 2.


  They’re here, and your tasks, documents, & briefcase items can now follow you wherever you may roam. If you’re already using Zimbra Desktop against a Zimbra Collaboration Suite server these will show up on next edit or item move via delta sync - while a full account sync or reset will pull in prior items. Personally, having briefcase items available offline is a major plus - as offline calendaring using the same AJAX web-client interface has already long since won me over.

Yahoo! Mail users rejoice - There’s now IMAP access through Zimbra Desktop to all free, plus, and business accounts. You didn’t read that wrong. Normally only Plus accounts have POP access, but as a perk when using Zimbra Desktop the mail is synced via IMAP; which is a much better protocol for keeping your mail organized - and yes it’s available to free accounts as well. Hook-up your @yahoo.com account or go grab one of the new @ymail.com and @rocketmail.com addresses. (Note that some apps don’t sync to Yahoo! servers yet so the data is local.)

Mailto: link handler - For Mac and Windows protocol handlers allow you to click on a mailto: link in any browser, and it will bring-up Zimbra Desktop’s composer with a javascript call. If Prism is not already running, it will start the web-app as well with a url call, then pop up compose. We don’t want to be accidentally invasive, so to turn this feature on you’ll have to check the box in global preferences to make it the default mail client on your computer.

Icon badging - To keep you informed, we now display the total number of unread messages across all-inboxes; in the dock icon for Mac and on Windows there’s now a tray icon, which changes to a new mail image if there are unread messages.

Mac & Windows users may just decide to toss out their toasters, because we now have mail & appointment notifications built-in. (Zimbra Toaster still serves as a lightweight new-mail checker with quick flag and delete features. There’s also some community contributed Linux solutions like Zimbra Notify.)

Zimbra Desktop on Windows now takes advantage of the native tray icon bubbles and on Mac of course we use Growl. (You need to install Growl separately which is quite straightforward.) You’ll also need to enable “show pop-up notification” under both Mail and Calendar tabs in preferences, since by default notifications are turned off.

The latest versions of Zimbra Collaboration Suite have also introduced browser title & favicon flashing, mail & account tab highlighting, as well as sound notifications - which have been ported to Beta 3 as well. So there’s no excuse for not noticing a new mail if you’re at your computer. Ok, we can still think of a few excuses - but note that the pop-up notifications are per account settings; so you can have some accounts on and some accounts off if you should need to ‘forget about’ that important meeting

In-case you’ve never tried Zimbra Desktop, or are still using an Alpha, and never tried it out during Beta 1 or when we served-up Beta 2: There’s also easy setup menus for setting up Zimbra Server, Yahoo! Mail, GMail, AOL, or any other IMAP/POP accounts you want to use. For Beta 3 we’ve thrown out JavaMail and wrote a brand-new robust IMAP/POP client-engine from scratch.


To get you up and running when you need it, there’s now an auto-start service. During launch of the Prism web-app a check is run to see if the background service is running - if not, it’s automatically started. This works on all 3 platforms, and proves especially useful on Linux since the service doesn’t automatically start after reboot. (See this forum thread for ways to do that.) There’s also an animated splash screen during launch of Prism so you know it’s working on bringing-up the background process.


Icon menus - On the Mac dock icon and Windows tray icon, we now have right-click menu items to check for updates and shutdown the background service.

Windows minimize to tray - Clicking on the “X” now only minimizes prism window to tray. To quit prism, right click the tray icon and choose “Quit”.


This release makes Zimbra Desktop available to a quarter-billion Yahoo! users with support for 20+ languages. The default theme is a revamped Yahoo! skin to help keep the interface familiar as it spreads to those millions of users. Hope you enjoy, and as we advance upon a GA release: Thanks to the Zimbra Community for all your bug corrections and feature requests so far. The Mozilla team developed a few of these new Prism features from scratch just for us, you can read more about some of them here. But stay tuned, we’re gonna have a closer look under the hood to see how we implemented these features and the inner workings of Prism + Zimbra Desktop in a future blog post.

If it’s not available to you via auto-update yet, you haven’t been building from source, or are even just discovering it for the first time, you can download it here for Mac, Windows, & Linux.
 

---

Have an idea for Zimbra Desktop or just want a tweak built upon these new components? We’re interested in hearing your feedback on it below or over in the Community Forums. A bunch of us are at the O’Reilly Open Source Convention (OSCON) in Portland, Oregon this week - so drop in around booths 415 & 519 if you’re there.

Now that the iPhone 2.0 software update is here, here’s a sneak peek of what Zimbra Mobile for iPhone looks like.

First, if you are new to Zimbra Mobile it’s part of Zimbra Collaboration Suite Network Edition and provides over-the-air synchronization of emails, appointments and contacts between a user mailbox and their mobile device. “Push” technology can make sure that updates are delivered to mobile devices as they happen in user’s mailbox on the server. Zimbra Mobile supports a wide range of ActiveSync compatible devices, including Microsoft Window Mobile (PocketPCs and Smartphones), Treo Palm OS series, Nokia E series (and other devices running 3rd party clients like RoadSync). Now, the latest additions to our device list are iPhone and iPod Touch running the iPhone 2.0 software. We’ve been tuning and polishing Zimbra Mobile to be a great match for iPhone.

I know everyone has his or her own favorite email gadget but personally, I’m glad I have an iPhone. With iPhone’s large, bright touch screen, instant “push” delivery, HTML email display, support for all sorts of document/media attachments and meeting invitations plus a photo-enabled address book, what else can I wish for? And the best part is Zimbra Mobile for iPhone takes full advantage of all that!

OK enough words. Let’s see what it will look like in action.

Note: Zimbra Collaboration Suite Network Edition 5.0.7 is required for iPhone sync support.

Initial Sync — Pretty Darn Fast You first setup how the iPhone will sync with a Zimbra Server at your company (or at your service provider): Once the connection is established iPhone will start to download your email, calendar and address book data. iPhone is actually powerful enough to download the different data categories in parallel. You’ll see emails start to show up in the Inbox at the same time contacts are added to the address book. I believe the parallel download is a unique feature that I haven’t seen on other devices. Also many other Zimbra Mobile devices typically only download partial envelope information and some text of each message, and get the rest of the message or attachments when the user asks for them. Given that iPhone has enough memory, it can download up to 200 most recent messages in full MIME format so that everything is available on the phone.

Rendering Messages on the iPhone iPhone can view or play many common attachments, including pictures (inline view), PDFs, various audio/video formats (great for receiving voicemail in your Zimbra mailbox), Word docs, PowerPoint slides, and Excel spreadsheets. Here are some examples of messages. I really like how iPhone displays deeply nested replies.

Appointments, Reminders and Calendar Of course Zimbra Mobile for iPhone is not just about email. As iPhone receives calendar events and meeting invitations from Zimbra server, it will alert users of incoming invitations, and allow users to accept or decline meetings from the device. Accepted meetings are added to the calendar, with alarms to go off at scheduled reminder time. Zimbra Mobile for iPhone supports all types of appointments, recurring events with all kinds of exceptions, as well as participants across multiple timezones. <

Address Book Got a big address book? No problem. One of our power users has 8000+ contacts in his address book. iPhone has enough memory to download and hold it all. Contacts with photos can be synced with the Zimbra address book as well, so go take some pictures of your friends with that iPhone camera.

Zimbra Mobile for iPhone is a new feature in Zimbra Collaboration Suite Network Edition 5.0.7, and it requires the iPhone 2.0 software update from Apple. I’m really excited about the Zimbra Mobile for iPhone user experience. This is a great addition to our existing solutions for iPhone which includes ZCS optimized for the mobile Safari browser and our iSync Connector for the Apple Desktop. To find out more about Zimbra Mobile for iPhone, check out the Zimbra Mobile Forums.

---

JJ Zhuang is lead developer for Zimbra Mobile.