Zimbra’s open source roots have always been of great importance to both the company and the Zimbra Collaboration Suite (ZCS). When we set out to build a new collaboration system over five years ago, we wanted to bring a fresh perspective to the market, and a big part of that was our commitment to being open source. We understood sharing ideas within the open source community keeps you a one step ahead of competitors by iterating faster to give users what they want. A great example demonstrating how the community has flourished is the Zimlet development program.
Zimlets are simple but powerful extensions of ZCS that connect users’ email, calendar, and contacts with any number of outside services (for a couple of recent examples see Alfresco and Peru and TripIt). Zimlet development growth in the community has been strong and steady, and we are excited to continue supporting the community’s work by providing a place where developers can feature the best of their integrations to share with other Zimbra users. So, today we are launching an updated Zimlet Gallery where you can pick and choose from many handy new ZCS extensions.
At the same time, we also love seeing our Yahoo! friends continue to embrace openness as part of the Yahoo! Open Strategy. In addition to this announcement today, a number of our Yahoo! brethren are extending their platforms to become more open. Today, Yahoo! Mail is introducing applications which enable people to make online payments, access personal photos and more easily send large files directly from their inbox. In addition, My Yahoo! is adding even more third-party applications, driving enhanced personal productivity for users directly from their My Yahoo! start page. You can read more about the Mail and My Yahoo! updates on the Yodel and YDN blogs.
As part of the Zimlet Gallery launch today, we’d like to introduce you to a few new third-party Zimlets, including:
Xythos Zimlet – The Xythos Zimlet allows you to drag and drop email messages and file attachments directly into Xythos’ Enterprise Document Management System. Secure document management is popular in the enterprise and universities; integration in email is key for ubiquitous adoption.
Processmaker Zimlet – The Processmaker Zimlet helps streamline workflows, like time-off requests, all within Zimbra email (see above). This Zimlet is already becoming popular and is being deployed at Access America Transport and Ministerio de Vivienda by our Zimbra Partners.
In addition, Zimbra developers have created a handful of new Zimlets, including:
Place Sticky Notes on Email – The new Sticky Notes Zimlet allows you to attach and tag emails with “notes.” One can leave comments, reminders, additional info about the email and more. And Zimbra’s powerful search can search through emails based on the contents of the tags/notes attached to the email.
Email Highlighter – The Colored Emails Zimlet allows you to apply personally assigned colors to emails from specific senders such as a family member, your boss, etc. You can identify senders by color, but you can also create colored emails through tags, making it easier to prioritize any inbox.
Save Email as Documents – With one click, the Email-2-Doc Zimlet lets you save an important email as a Zimbra Document; it will automatically save any attachments as links in the Document as well. The email can then be edited and shared with others.
It may sound odd offering more Zimbra installation advice since there is a lot on the subject in other blogs, our documents, wiki and Forums. In fact, some quick research surfaced over 1.4 million hits for Zimbra server install on the web and 36,000 on the Zimbra site alone.
But we are also fortunate to have more new Zimbra users than ever, and after helping some trial customers recently, it was a good reminder a few simple tips can help cut through some noise and avoid time-consuming snags once you start the install process. So without further ado here are the top 6 common pre-requisites to consider when preparing for your Zimbra installation:
1. Firewall Servers have firewalls configured once the operating systems are installed for security purposes. Our recommendation is to temporarily disable the firewall on the system during a single and multi-server Zimbra installation. An alternative would be to refer to our installation guide to get a list of ports (see Table 1) used by the application and make sure the ports are open prior to installation.
2. DNS setup All Zimbra configurations store hostnames. We do not have save any IP address information in our configuration. The advantage is this allows an administrator to change IP address (more likely) on the Zimbra system without having to perform any application changes.
This scenario means that all the hostnames to be used in a Zimbra installation have to be defined in DNS. Both A and Mx records for the hostnames and email domains need to be defined and verified prior to beginning your installation.
One other thing to consider is split DNS configuration if you are dealing with servers separated by a firewall.
3. Use of Fully Qualified Hostnames (FQDN) It is crucial to use a Fully Qualified hostname during the Zimbra configuration. For example, you should enter server1.domain.com instead of server1. This avoids incorrect DNS address lookups and ensures that the client would be connecting to the right application.
4. Port Conflicts Standard server configuration comes with support for numerous services like POP, IMAP and HTTP (see Table 1). These services are also installed with the Zimbra Network Edition. Therefore, you want to make sure you disable all these services prior to installation. The Zimbra installation scripts will check for any of these port conflicts and notify you to turn these services off before continuing.
5. Libraries and additional packages Zimbra’s rich feature sets are dependent on additional packages being installed on the system. These packages vary between Linux and Mac Operating system. The Zimbra installation script does perform checks to verify all the dependencies have been met, but going through the System Requirements documentation (available on the Zimbra website) before will save you some time.
STORAGE CALCULATION EXAMPLE
(Based on ‘Mailbox Usage of 200 MB’ and 500 users)
+ User Data: 500 users with 200 MB = 100 GB user data
+ MySQL data: 5% of 100 GB (User Data): 5 GB
+ Zimbra binaries: 10 GB
+ Zimbra logs: 20 GB
+ Zimbra indexes: 25% of 100GB (User Data) = 25 GB
SUBTOTAL: 100 + 5 + 10 + 20 + 25 = 160 GB Backups: 160 % of Subtotal: 160 * 160% = 256 GB for backups TOTAL: 160 + 256 = 416 GB
6. Sizing Storage sizing is important for an excellent performing Zimbra application (see example). If you are doing a Network Edition trial you should contact the Zimbra technical team for sizing information for storage including number of disks, which Raid level to use, and the size of the drives to use. Configuration of the Zimbra store volume is important in satisfying the application IO requirements.
Remember, it’s also a good idea to review the Zimbra Quick Installation Guide where you can find this information and many more good tips.
Do you have a good tip to share? Feel free to add a comment!
Someone in the forums recently asked about ways to migrate individual accounts from one ZCS instance to another, so thought I’d share the enlightenment with all. Whether you are going from an on-premise install to a hosting provider, want to create handy archives of old employee accounts, or just need to duplicate mailbox contents of a user; the syntax in this article proves remarkably useful, and applies to all editions.
There are a multitude of comparable RFE’s on addressing this need via different approaches. (Bugzilla entries 19630, 29573, 28443 & 30163 to name a few.) Some want graphical tools to browse data and selectively migrate certain things, while others would be happy with a cross LDAP zmmailboxmove.
Depending on your situation, several backup tools can take care of a large portion of your daily needs; and there are ways to do Zimbra-to-Zimbra migrations using the Network Edition’s backup and restore capabilities – however they require admin abilities on both systems. Meanwhile, most of the frequently used open source backup solutions are simply an “all accounts at-once” approach. So what to do when you need to move from your personal setup to a hosting provider? Or if you’re a hosting provider, move a tiny handful of accounts to a separate infrastructure? Before diving into the wiki on user migration for info on Imapsync, REST exports, CURL imports, etc; there’s a handy way to avoid the “one item type at-a-time” transfer methods.
In ZCS 5.0.9+ you can export an entire mailbox with: /opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL “//?fmt=tgz” > /tmp/account.tgz
Next transfer via rsync, scp, sftp, etc. You’ll also need to create the account on the 2nd server if the desired account doesn’t exist at your destination server yet.
Then import with: /opt/zimbra/bin/zmmailbox -z -m user@domain.com postRestURL “//?fmt=tgz&resolve=reset” /tmp/account.tgz
The resolve= paramater has several options:
“skip” ignores duplicates of old items, it’s also the default conflict-resolution.
“modify” changes old items.
“reset” will delete the old subfolder (or entire mailbox if /).
“replace” will delete and re-enter them.
‘Reset’ will be a bit faster on an empty destination mailbox because it skips most dupe checks.
Note: There were some duplication fixes and additional issues (mainly sync related) corrected with the tar formatter in 5.0.10.
Not a Zimbra Admin? Users can get the same zip/tar formatter on REST URL’s by visiting: http:// server.domain.com/home/user?fmt=zip&query=is:anywhere
The zip format has been around for a long time, but doesn’t contain account & item metadata like the tar formatter automatically does: http:// server.domain.com/home/user/?fmt=tgz
Infact, this same technique is currently used in Zimbra Desktop’s alpha backup solution.
If this approach doesn’t scale performance wise for your situation, or you simply don’t want to have everyone hit a REST URL for 30GB mailboxes all at the same time, here’s a collection of helpful scripts and other ways to systematically migrate:
Mysqldump & rsync with an interesting blob management technique: Zimbra2Zimbra
Imapsync for mail + postRestURL for contacts, calendar & filters: ZimbraMigrate (Expand the concept for tasks, documents, and briefcase items.)
The above Zimlets are aimed at making quick self-copies & not for restores, but there are many methods for putting messages back into Zimbra, including tools like zmmailbox addMessge, zmlmtpinject, CURL, etc; for more info checkout these threads: Recover data from store folders & Moving Folders between users
If moving your entire server, I’m a huge fan of the install.sh -s trick when using NE backups to do so isn’t an option.
Have another method you’d like to share? Document it in the wiki & note it below, or you can discuss over in the Community Forums.
Part 1 covered Network Edition backup features, today’s snips apply to all editions.
First among the lesser known additions: We recently provided the possibility for a nice performance boost to some environments by adding the ability to turn on batched indexing in ZCS 5.0.3 (you can even fine tune it at the localconfig, COS, and account level). We’re not talking about when you re-index an entire account here, this is a change to the index-as-received model; now new items can sit in a ‘queue’ (really a ‘indexing deferred’ flag on the mail_items table of the pertaining mboxgroup database in MySQL) to run all at once when it reaches the zimbraBatchedIndexingSize threshold, saving you from all the tiny disk thrashing. It might not be immediately apparent that this works better, but you can mention it in the forums and we’ll show you the evidence to the contrary – it proves expecially useful for POP heavy or ZAD archive accounts.
New & Enhanced Admin Tools
Your /opt/zimbra/bin & /opt/zimbra/libexec directories hold a wealth of tools to make your job easier.
zmdumpenv has been around for a long time, but underutilized – it grabs the basics that you should probably provide with every issue to help others understand where you’re coming from.
When you need to send ad-hoc SOAP commands to the server, the powerful zmsoap takes care of authenticating, generating the envelope, sending the request, and writing the response to stdout.
If your server freezes or is busy, running zmdialog can give that ‘my server hung’ support ticket a purpose. With JDK 1.5 it won’t collect a heap dump so you might also run /opt/zimbra/java/bin/jmap -heap:format=b [/opt/zimbra/log/zmmailboxd_java.pid] however zmdiaglog collects a core dump, from which it should be theoretically possible to get a heap dump. Thread dumps when you kill -QUIT/3 [pid] are helpful too. Info on ways to take them (like /opt/zimbra/libexec/zmmailboxdmgr threaddump) plus a handy script, are here.
There’s the MySQL metadata DB, the Lucene index, and the actual blob files on disk in /opt/zimbra/store. If you can click on a folder in the web UI and get results, it means MySQL is up and happy. If you open a message and get the infamous “missing blob for id” error, it means that the message files on disk aren’t where they’re supposed to be (ie: disk crash, not mounted/permissions, or you’ve recently pointed your volumes incorrectly). There’s all sorts of things you can do in that situation – here’s a more detailed list. Even if you don’t have a good grasp of the mailbox database structure yet, you can appreciate the need to easily determine if those blobs are still available (after you’ve moved whatever you can salvage of the store from corrupted drives to a better location). Enter the zmblobchk utility which can determine what files are missing – there’s also plans for a repair mode, for when you finally realize the blobs are gone and want to get rid of those UI error messages. Of course you’ll next consult the Network Edition’s restoreToTime feature or other backup solution you may have.
When you’re absolutely out of room on the disks housing your DB (seriously put in that purchase order for more storage – it’s cheap these days) in a pinch the optimizeMboxgroups.pl script (available in the public cache and added to the upcoming 5.0.10) can help you recover wasted space in your mail_item, appointment, imap_folder, imap_message, open_conversation, pop3_message, revision, and tombstone tables on each mboxgroup. Just note that it temporarily locks each table, and could use considerable IO while they’re being rebuilt. You can certainly use it pro-actively during a maintenance window to reclaim space as well.
There’s also a new wiki page on statistic collection so you can generate nice charts that help you figure out what you might need to tweak.
We could go on and on about the utilities in those folders, so throw up a test environment and experiment sometime – it may just make life easier when you have hundreds of users breathing down your neck. And if you should ever be ’stumbling around in the dark’ you can always enable additional debug logging to shed more light on the situation.
You can find help for all the above utilities over in the Community Forums or ask us a question on them below.
Continuing our earlier advice to take backups frequently, and secure them offsite – thought we’d highlight a few recent administrator related things added to ZCS that you might not have noticed.
Network Edition Backup Enhancements
Speaking of backups, there are some new ways to take them in ZCS 5.0.x. With ever larger quota usage, full backups can often take a while to run, and even incrementals which process the redologs may still be one heck of a job when you’re talking thousands or millions of accounts. Having trouble completing that entire full backup during off-hours? Enter the hybrid auto-grouped mode, which combines the concept of full and incremental backup functions – you’re completely backing up a target number of accounts daily rather than running incremental sessions. As a plus it automatically pulls in the redologs since the last run so you get incremental backups of the remaining accounts; although the incremental accounts captured via the redologs are not listed specifically in the backup account list. Think of auto-grouped mode as a full backup for the scheduled group as well as an incremental (via redologs) for the all other accounts at the same time. This allows you to do a point in time restore for any account. Simply divide your total accounts by the number of groups you choose (zimbraBackupAutoGroupedNumGroups is 7 by default) and that’s how many will get a full backup session each night. Newly provisioned accounts, and accounts whose last backup is a specified number of days older are picked first. (zimbraBackupAutoGroupedInterval is defaulted to 1d)
To save space, and therefore store older backups longer or run them more frequently, you can also auto-compress them with the –zip argument. This isn’t new, but it got improved handling of shared blobs in 5.0.5 as well as a -zipStore mode for speed. You can also adjust the buffer & queue capacity of the backup process, as well as additional options like the level of compression, or the number of archives per person via the backup_zip_copier_private_blob_zips localconfig attribute. Of course you lose the hard linking optimization (speed and space) for blobs that are in an earlier full backup already when working from the same disk – so it’s more advantageous for those off-site single-copies (you do make one often right?). However, there are legitimate uses for running it on your normal backups: Fewer files make it easier to copy or rsync later, and prevents you from running out of inodes. You can also easily delete individual backups rather than running zmbackup -del, and therefore keep just a few really old backups around for whatever compliance reasons you may have.
By-the-way, ZCS 5.0.6 added the ability to easily replay redologs from an arbitrary point in time with zmplayredo should you be in a unique situation that needs it. (Say you’ve been taking snapshot backups and but then need to restore, and you’ve also saved all the redologs since the snapshot. Or you take a snapshot, then manually copy redologs from the live system to bring the snapshot copy up to current. This allows you to force replay of them all and not just the uncommitted transactions.)
We’ve always recommended that: “After upgrade, you should run a full backup immediately as changes in this release invalidate all old backups.” It’s still good advice to have a fresh full, but there’s always the infrequent need to get data from an older minor version backup without throwing up a temporary machine. Well, with 5.0.x we aimed to make restores compatible across patch releases (i.e. an older 5.0.x backup, not a prior 4.5.x backup – major version restore is this RFE). There was a bug about zmrestore not handling database schema changes, but that’s fixed in 5.0.5 and later – so backwards compatibility for restore is now theoretically possible. And we’re also looking to put icing on the cake by adding a conversion tool to upgrade backups themselves to allow restore on later ZCS versions.
You can find help on backups via Support Channels, over in the Community Forums, or ask us a question on them below. Stay tuned, as Part 2 will cover tidbits for all editions.
This may be the most important post you will ever read. If you’re a Zimbra Administrator, please read, and pass this onto your colleagues who use Zimbra. If you’re a CTO or CEO, take time to ask your Zimbra Admin about the subject of this post. This blog post is about backups.
Whether your an Open Source User, Zimbra Desktop User, or Network Edition Customer, you can do backups of your data. There is nothing worse than getting a call from a customer, or a Private Message from a Forum User that says, “I need help. My HD has crashed, and all my backups were on that drive.”
Let me Digress for a moment, and share with you my personal experience with backups. It’s sort of a Legend here at Zimbra. It all begins in 2005, Zimbra was the new kid on the block and I was an inexperienced System Linux admin for Tombstone Unified School District in Arizona. This was a small district with limited funds. When I interviewed for the post, the Superintendent handed me his card, and it had a Hotmail address on it. Right then, I knew this would be quite a difficult job.
One of the first things I did, was investigate E-mail Server platforms, and TCO (Total Cost of Ownership). A quick search on the interwebs (which is a series of tubes, not to be confused with the you tubes), yielded this new thing called a Zimbra. Much like our millions of Users, I downloaded it and tried it. It worked great! We used the Zimbra Open Source Edition (then it was in Beta).
I was so proud of my cool new setup, and our staff were so excited to finally have good e-mail and calendaring. In knew the importance of keeping backups, and archiving. All of you System Admins at Hospitals and Government institutions know what I mean. You get well acquainted with the Laws and Requirements. So, I would stay up until about midnight, and stop Zimbra, and rsync /opt/zimbra into /opt/zimbra/backups. Keep in mind that this was in the early days of Zimbra, and I wasn’t even an employee yet. As a matter of fact, myself and several others, pioneered the Open Source backup procedure.
Everything was great until one night I noticed I was running low on Disk Space on my /opt partition. So, I thought, “If I just remove all my backups, and make a fresh one, that will save me a bunch of space”. So I ran the following command as root: rm -rf /opt/zimbra backups
Now, back then, we mounted a clamav partition ramdisk for quarantine purposes. The only indication that I had that something was wrong was when I got an error saying that it couldn’t unmount the partition because it was in use. Everything else in /opt/zimbra was gone…including my backups.
As most of you admins know, preserving data is important. We were involved in several litigation matters, and I would later be cited for obstruction of justice for destroying evidence.
When I noticed what had happened, I immediately called Zimbra and talked with MarcMac here at Zimbra. He tried to recover the inodes using Midnight Commander, but it was a total loss.
Lesson Learned. So, from one admin to another, please take time to make sure your backups are not located on the same machine that Zimbra is on. Please! We never want to hear about data loss. Whether an opensource user or network user, I hope you will take a few minutes to consider your backup strategy, and fix any single points in failure.
We’ve had some great responses to my blog post yesterday about changing the default mailto links using Firefox. There’s a couple of problems with my method: 1) It’s not really scalable for an admin [ie no auto install] 2) Not really applicable for Safari and Internet Explorer.
When you click on a mailto: link, most of the time your Firefox will launch that version of Outlook Express or Apple Mail you never configured. For the novice user, they click next, next, next, next, then send their e-mail…and it never gets delivered. Well using a greasemonkey script that we’ll convert to a firefox extension, we can handle the mailto links.
This morning, I clicked on a link thinking it would take me to a “Contact us” form, but it was a mailto: link. I was a little annoyed that Apple Mail opened, and I thought: I wonder if there’s a way to change that link to open ZCS?
Well, I found a cool little hack using GreaseMonkey and a site that compiles GreaseMonkey scripts to an xpi extension.
First, some pre-requisites:
You’ll need ZCS 4.5.0 or newer.
This is a Firefox only hack. Sorry IE and Safari users.
This is a GreaseMonkey script for GMail. All we’re going to do is change the URL from GMail to your ZCS server. The script is licensed under GPL, and if you use it, please note that you MUST abide by the coder’s License guidelines. The script was written by Julien Couvreur.
Copy all of the text to your favorite text editor.
Now, in your text editor, you’re going to change a few lines.
And fill in the text areas. Leave the GUID the same, and copy and paste your new script into the script text area, then hit compile.
It will ask you to save the xpi file. Once you save it, Open Firefox, then click file->Open and locate your XPI. Restart firefox, and all of your mailto: links now point to the Zimbra server.
Thanks and credit go to: Julien Couvreur who wrote the script Anthony Lieuallen for the web based GreaseMonkey to Firefox Extension converter cubicgarden for the flickr GreaseMonkey photo.
Gmail™ webmail service is a registered trademark of Google, Inc.
Firefox is a registered trademark of Mozilla, Inc.
Also, these sites and scripts are not hosted or affiliated with Zimbra or Yahoo.
Zimbra’s open source roots have always been of great importance to both the company and the Zimbra Collaboration Suite (ZCS). When we set out to build a new collaboration system over five years ago, we wanted to bring a fresh perspective to the market, and a big part of that was our commitment to being open source. We understood sharing ideas within the open source community keeps you a one step ahead of competitors by iterating faster to give users what they want. A great example demonstrating how the community has flourished is the Zimlet development program.
In addition, Zimbra developers have created a handful of new Zimlets, including:
Infact, this same technique is currently used in Zimbra Desktop’s alpha backup solution.
This may be the most important post you will ever read. If you’re a Zimbra Administrator, please read, and pass this onto your colleagues who use Zimbra. If you’re a CTO or CEO, take time to ask your Zimbra Admin about the subject of this post. This blog post is about backups.
This morning, I clicked on a link thinking it would take me to a “Contact us” form, but it was a mailto: link. I was a little annoyed that Apple Mail opened, and I thought: I wonder if there’s a way to change that link to open ZCS?
