In order to setup an email server you really should have at least 2 MX servers accepting mail in case one is down. It really really sucks when emails are delayed and the sender gets that delayed message.

The simplest self-hosting solution I've found is to setup 2 cheap VPS servers that accept mail via postfix (128mb, maybe 256mb if you want to virus scan at this point, a good idea), and they both can deliver email to an in-house server running roundcube or zarafa or whatever, when the internal server is up and running. As long as the VPSes are up, they hold email until the server inside your local network is up and running.

This means you bypass the very messy "secondary mx" issues, as you only have one internal server to worry about that accepts email arbitrarily from the 2 public server, and you can backup your internal server just like a normal PC.

An alternative is to use some email server from a host/domain registrar/google apps/zoho/etc as secondary MX server. It saves you from having to maintain a second server, and you still get 99% of your messages through your primary server.

I had a similar setup for a while, and it was interesting to see that some messages (spam, mostly) picked the secondary MX by default. Spam servers also seemed to cache the (secondary) MX for much longer than the TTL.

Spammers believe that secondary MXs are set up "in case of emergency", so they must have less restrictions on what goes in (so you don't miss an email) and must be less maintained... meaning that spam goes in more easily.

For anyone wanting to setup a mail server this is the best guide going. I read Part 1 and had to find my own way.

Incidentally, if you are trying to read email or send email with code, having your own email server, working with a desktop client is a very useful starting point. You can downgrade the spam filtering, the SPF record checking, the SSL/TLS sign on until you have something that can read IMAP or POP. This is where the real fun begins as emails come in various multi-part formats, there are lots of character sets to translate to UTF-8 and there is base64 to deal with. If you can reliably read the header of an email and extract something simple like the date then you are doing very well.

This guide is really good as it goes almost whole mile.

It also does very good if not excellent job at illustrating how hard it is to set up, secure, and maintain your own email service. There are lot of places to go from here, including multiple MX'es, backup/recovery, etc.

Nevertheless. I've been using my own email infrastructure for almost ten years years. For last four I'm totally over it. Yes, there are some things to do if you absolutely want to be free to chose your email provider whenever you want to - having clear choice where to go, having backup off service provider, having migration strategy in place, etc.

I feel like any data that is being transferred over internet has a real possibility of being surveilled on. Amazon AWS or any other storage provider has to comply with any three letter agency. Calling this secure sound like calling random numbers as random. What do you guys think?

Email is fundamentally flawed, or better put it is not up to challenges of time. While I am reading this ars articles with great interest and was toying with the idea of rolling my own email server, series size and complexity serve as a best argument of email flawed state. We are fixing and patching and protecting something that fundamentally can't be that well fixed, patched and protected. We need to reinvent everything.

Email is a robust, federated, delay tolerant network. None of the 'solutions' to communications I've seen invented by startups in the last 20 years have captured those properties. Jabber is the closest. Most of the instant messaging clients over the last 20 years have given up enormous value that email brings.

Email servers don't have to be complex. I wrote an SMTP and POP server in a single afternoon in the late 90s. The protocols have lot of optional parts, and if you want to get something quick up and running, there are simple implementations possible. The standard Unix servers are quite complex to administer but that's mostly because they were built for Unix system administrators. I mean, BIND configuration files are also complex, but that doesn't mean we should throw out DNS just because it's not trivially simple to implement.

The only point to reinvent and reimplement everything is if you can make some revolutionary improvement, otherwise, we should just improve what we have.

Nothing is perfect, least of all email. But it did one thing right, and it's a big one: any email user can send to any other email user, regardless of provider or platform. We still lack that capability when it comes to IM, social networks, and most software in general.

> We still lack that capability when it comes to IM

No, we do have this capability: use XMPP.

In fact, XMPP is functionally enough to completely replace email, and does also serve as IM.

Theory versus reality. The theory of federated XMPP is exactly as you said.

The reality is much more of a mess, with multi-device issues -- routing issues -- federation issues, not to mention a ton of non-XMPP messaging services.

And most importantly, no identity system which end users know about. (i.e. people know about email addresses, but Facebook users use XMPP without knowing how to refer to people on other services)

All these problems were applicable to email in the past as well. It took a lot of iteration to get everything working as smoothly as it appears to work now, provided we ignore the massive issue of spam and almost total inability to make it secure.

Email server administration has its challenges, and the tools could definitely get better. But email is one of the few decentralized forms of communication, and all of that fixing, patching, and protecting is the product of a decade of hardening the system so that it can survive. If we're lucky enough to have a decentralized social network get the kind of adoption that email has, it will have to tackle the same problems.

If you want to setup something more complicated than email, setup Diaspora.

Could you be more specific? I suspect that what you consider problems of email there aren't actually problems of email at all, but it's difficult to say.

The way I see it, SMTP protocol and email formats are root of the problem. Yes, they are great in connecting all kinds of devices and making communication easy. But they belong to times when one could write a postcard (text on the outside) and expect it to arrive to destination without being read loudly in every local pub from here to there. In times when we need proper enveloped email, calling postcards emails is inadequate.

Another issue is number of tools and skills we need to implement email server locally, which I suspect is a result of same specifications, protocols and formats.

I am no expert on this (not a real one anyway) but there has to be a point when effort to support a system outweighs convenience?

Most any decent email client allows for GPG encryption so your the source and destination provider can't read your emails. Further, most email is sent straight A to B encrypted (if both providers support it), along the unsecured internet, making anybody between A and B unable to review the email.

As bitJericho already noted, nothing in the protocol or the format prevents encryption--and as a matter of fact, you can use end-to-end encryption with email now, even though it was not originally explicitly designed for that. That is one of the really great strengths of email: It can be adapted and extended in many ways as technology develops.

Other than that, I still don't see what your criticism actually is.

All of that is not to say that the email protocol stack is somehow particularly elegant (it most certainly is not), but most often when you hear people say that we need to reinvent email, they don't actually understand what the problems with email are and just seem to think it's bad because it's old--all while being completely oblivious of the lessons that have been learned with email. Despite its lack of elegance, it does actually incorporate a lot of wisdom, as well as its fair share of stupidity.

But what tends to be overlooked in particular is that for quite a few of the most obvious problems with the usability of email (namely, authentication of senders and spam), those problems are not problems that the people who invented and extended the protocols and formats failed to consider, but that despite a lot of effort and intelligence applied to them, they failed to come up with a solution for. And the reason for that tends to be that those problems are fundamental social problems that just crop up in email just as they do elsewhere, which can not be solved by simply inventing "modern email".

As for how difficult it is to set up your own mail server, I don't really have a clue. I mean, I do it all manually, but that's probably just because I learned how to do it quite a while ago, and now that I know how to do it all manually and I have a working setup anyhow, I don't know what easier ways might be available. But in principle, I don't see why it shouldn't be possible to put together a package of a pre-configured mail setup for standard cases which shouldn't be all that difficult to deploy. I mean, if you look at how many magnitudes more awful the web stack is, and ordinary people still manage to install and use a web browser, completely oblivious to the hilarious complexity of that piece of software.

Email security is a bolt-on and can't be relied on in any way. You cannot run a mail server that forces STARTSSL or TLS because you inevitably have important mail that comes from or goes to a server that can't or won't send it to you encrypted. And PGP only encrypts the body of the message, and not the metadata, the concealment of which in transit is crucial for privacy. I believe there is an RFC for encrypting headers, but it has no traction.

I think the reason some people talk about starting over is because of this. The minimum level of security of email must be allowed to fall back to is "no security."

Exactly. I'd like to see 'new' email protocol built built on non-optional foundations. When we use http over SSL/TLS, browser either shows padlock icon or does not show padlock icon. While implementation has its occasional flaws and bugs, specification does have defined on/off state, not many optional layers in between.

We can make any communication channel secure, by adding layers of encryption, no problem with that. But unless it is defined deep down as a part of the standard, it all ends with no security at all and no means to be sure what happened with the message along the way.

Having headers exposed is one of the reasons why I raised initial concern. It is part of the specification and unlikely to go away by building on top of the standard.

No, "adding layers of encryption" does not equal "security". For encryption to provide real security, you need to solve key distribution, which for all practical purposes is unsolved.

And no, when you claim that HTTPS has "not many optional layers in between" that mostly suggests that you just treat it as a black box rather than that there is no complexity there. The simplicity is in the user interface only, and that is arguably broken because it does not actually reliably reflect actual security.

Really, once again, you mostly make the impression that you don't really understand the problems with email and just think that it's broken because it's old. Anyone with that attitude is likely to just repeat all the same mistakes that we've been through with email, and in the end not solve any of the actual problems.

There is no way in hell I could be accidentally misquoted and misunderstood, three out of three 'arguments' in single post. Yeah, let's pretend that I have no idea what I am talking about -- much easier than acknowledging any kind of flaw in actual foundation of the architecture.

I can't be any more clear than what I already said. If you decide to take an easy road and kill the messenger, no problem with that. I don't have any intention to continue with this discussion indefinitely, especially if it turns out to ad hominem. History teaches that killing the messenger does not make problem go away. You see it or not, decide to ignore it or not... fine by me.

You also might want to look up what an ad hominem is. Hint: Suggesting that someone lacks knowledge about what they are talking about is not. And in particular, it is not logically fallacious to deduce that if someone lacks knowledge that therefore they might be providing unreliable information.

And while you are at it, maybe also look up what "killing the messenger" means and how it is used.

But you are right that the much more fundamental problems that are actually there won't go away if you keep ignoring them.

By the way, the one who claimed that you were not really an expert on this was you, a few posts ago.

Now, if you have any actual specific problems with email that you think a new system would be better able to solve, feel free to explain it, then I can explain more concretely what the problem with your proposal is or how you are missing the actual underlying problem - or possibly accept that you indeed do have a valid point, though I wouldn't be too optimistic about that at the moment.

Well, I would agree that the non-encryption of end-to-end headers with PGP is somewhat of a bug.

Other than that, I think those are all examples of problems that are not actually email problems.

Whether you can force TLS/STARTTLS doesn't really matter. First, you cannot ever force anyone to keep something secret, they can always publish a secret that they know somehow and thus make it not be a secret anymore. You can always offer STARTTLS though, so if the other side wants to keep a secret secret from eavesdroppers, they then can do so. That does not help against People in the Middle, of course, as they can strip out the STARTTLS offer. But being able to "force" STARTTLS doesn't help you either, as that would only force the middleperson to speak TLS to you. If you wanted to force authenticated communication that protects you from MitM, you would first have to set up some authentication mechanism. On a server-by-server basis, you can already do that. If you wanted to do it globally, we would need a reliable global PKI. Such a thing does not exist and is extremely hard to build--and in particular, it's not an "email problem". Reliable authentication is mostly a hard social problem.

As for metadata encryption: Well, that pretty much is impossible by definition, except maybe through mix networks. You cannot simply encrypt the information that needs to be readable in order to route the message to the destination. Even if you deliver a message via TLS to my MX, the mere fact that your mail server looks up my domain MX and then connects to it already tells an eavesdropper all the meta information there is. And if you solve that by instead doing all the routing inside gmail or some other big provider, you obviously haven't solved anything, as now google can see, sell, and be subpoenaed for this information.

Also, "security" is a word without any meaning. You can only be secure against particular attacks/risks, and securing a system against a particular risk often is in conflict with securing it against another risk, so you can only ever achieve one or the other, and it's often a difficult decision which risk to take. In the case of email, you could very easily secure the system against spam, for example: Just have a government agency that licences email server operators and shuts down any that engage in spamming. The side effect: Email is not the slightest bit secure against government censorship anymore. And yet again, there is a social problem at the core, not a technical one.

There are reasons why email is so "insecure". And as I wrote above: It's not that people didn't think about it. It's more likely that those who suggest that starting over could help solving the problems haven't really thought through it. In particular the "security" problems of email are extremely hard problems, and they are hard for social and political reasons, not for technical reasons.

Your argument seems to be that if there isn't perfect security, then it's worthless. I'm arguing for sane minimum standards so Jane Q. Random sitting somewhere on the network between me and my recipient can't just run Snort and read the shit out on their screen. So that the connection can't be easily routed to somewhere else and grab the data. With email as it is today, these things are possible for a reasonably intelligent middle schooler.

Yeah, CA system is shit. but they are better than nothing. I don't trust it won't be compromised by a government NSL, but I trust it enough that I'll pay for ebay crap over SSL at a coffee shop and not worry that the guy next to me is sniffing my credit card number.

Here's what starting over means to me: you don't break something that basically works. If you start adding incompatible changes to email, then it splits the network. Instead create a parallel network with minimum standards attached to it (for instance,) and maybe nobody will use it but it also won't break anything.