Mail transport agents are the software that transfers mail from your local system to remote systems. It is very seldom necessary to mess with or replace your MTA on a modern Linux. Nevertheless, here's a survey to get you started on understanding what the tradeoffs are.
(There are other Unix MTAs besides these, but you are quite unlikely to encounter them on a Linux box.)
BSD sendmail is the grandaddy of Internet MTAs. It has outlasted a few would-be successors. Most Linux distributions now use it and have it preinstalled.
Sendmail has a long-standing reputation for being an administrator's nightmare -- hard to understand, tricky to configure, rife with security holes. As Internet technology and standards have stabilized, however, many of the sendmail options and configurable rules that gave rise to this reputation have ceased to require per-site tweaking (the effective demise of non-TCP/IP network layers like UUCP has helped a lot). Also, recent sendmail versions have an improved configuration system that insulates you from the legendary hideousness of the sendmail.cf configuration file. Most importantly, sendmail now normally comes preconfigured, and you should never need to touch it unless you have unusual requirements (such as needing to route mail over a non-TCP/IP network).
There is a sendmail home page at http://www.sendmail.org. It includes references to extensive documentation of sendmail, should you actually need to wrestle with custom-configuring it.
Other MTAs, if called as `sendmail', may mimic the semantics of sendmail's command-line options. This is convenient for mail user agents, which often assume they are talking to sendmail.
Smail was the first serious attempt to replace sendmail. It has a simpler and much more comprehensible configuration system than sendmail's, and some Linux distributions preinstall it rather than sendmail.
At one time smail's excellent support for mixed TCP/IP and UUCP sites was a major selling point for it, but as UUCP has declined, so has smail. Also, smail is less efficient than sendmail on high-volume connections.
As with sendmail, it is unlikely that you will need to tweak a preinstalled smail configuration.
(Very occasionally you might run across references to an `smail 2.5'. This program has been obsolete for a long time. Don't bother with it.)
The qmail program is a sendmail-compatible MTA designed specifically for high security. The author has a standing reward of $500 for publication of the first verifiable security hole; this reward has gone unclaimed since March 1997.
The qmail home page is at http://pobox.com/~djb/qmail.html.
The exim program is similar to smail3, but with more features. It advertises particular strengths in spam-blocking and support of several virtual hosts (virtual DNS domains) on the same host.
The exim home page is at http://www.exim.org/.