SpamOracle -- a spam classification tool

                        Version 1.6

OVERVIEW:

  SpamOracle is a tool to help detect and filter away "spam"
  (unsolicited commercial e-mail).  It proceeds by statistical analysis
  of the words that appear in the e-mail, comparing the frequencies of
  words with those found in a user-provided corpus of known spam and
  known legitimate e-mail.  The classification algorithm is based on
  Bayes' formula, and is described in Paul Graham's paper, "A plan for
  spam", http://www.paulgraham.com/spam.html.  

  This program is designed to work in conjunction with procmail.
  The result of the analysis is output as an additional message header
  "X-Spam:", followed by "yes", "no" or "unknown", plus additional
  details.  A procmail rule can then test this "X-Spam:" header and
  deliver the e-mail to the appropriate mailbox.

  In addition, SpamOracle also also analyses MIME attachments,
  extracting relevant information such as MIME type, character encoding
  and attached file name, and summarizing them in an additional
  "X-Attachments:" header.  This allows procmail to easily reject
  e-mails containing suspicious attachments, e.g. Windows executables
  which often indicate a virus.


LICENSE:

  This program is distributed under the terms of the GPL (GNU Public License)
  version 2, available from http://www.gnu.org/licenses/gpl.txt.


REQUIREMENTS AND LIMITATIONS:

- To compile: Objective Caml, https://ocaml.org/, version 4.02 or later.

- To use: 

  . Your mail must be delivered to a Unix machine.  You must have
    a shell account on this machine.  This machine must have
    procmail (http://www.procmail.org/) installed.  Your ~/.forward
    file must be set up to run all incoming e-mail through procmail.

  . To provide the corpus of messages from which SpamOracle "learns",
    an archive of at least 1000 of your e-mails is needed.
    The archive must be manually or semi-automatically split into
    known spams and known good messages.  Mis-classified messages
    in the corpus (e.g. spams mistakenly stored among the good messages)
    will decrease the efficiency of the classification.  The archive
    must be in Unix mailbox format, or in "one message per file"
    format (a la MH).  Other formats, such as Emacs' Babyl, are not
    supported.

  . The notion of "word" used by SpamOracle is slanted towards Western
    European languages, i.e. the ISO Latin-1 and Latin-9 character sets.
    Preliminary support for JIS-encoded Japanese is provided.
    Still, SpamOracle will not work well if you receive many
    legitimate e-mails written in other character sets, such as
    Chinese or Korean.

INSTALLATION:

  Edit the Makefile and change the definitions of the following variables
  at the top of the file:
    LANGUAGES    the languages you're interested in besides English
    CPP          how to invoke the C preprocessor
    BINDIR       where to install the executable

  Do "make" in this directory.

  Become superuser if necessary and do "make install".


INITIALIZATION:

  To build the database of word frequencies from the corpus, do:

     rm ~/.spamoracle.db
     spamoracle add -v -good goodmails -spam spammails

  (By default, the database is stored in the file ".spamoracle.db" in
   your home directory.  This can be overriden with the -f option:
     spamoracle -f mydatabase add ... )
     
  This assumes that the good, non-spam messages from the corpus are
  stored in the file "goodmails", and the known spam messages in the
  file "spammails".  You can also fetch corpus messages from several files:

     spamoracle add -v -good goodmails1 ... goodmailsN \
                       -spam spammails1 ... spammailsP

  To check that the database was built correctly, and familiarize
  yourself with the statistical analysis performed by SpamOracle, 
  invoke the "test" mode on the mailboxes that you just used for building
  the corpus:

     spamoracle test goodmails | more
     spamoracle test spammails | more
     
  For each message in the given mailboxes, you'll see a summary like this:

    From: bbo <[email protected]>
    Subject: Check This Out
    Score: 1.00 -- 15
    Details: refid:98 $$$$:98 surfing:98 asp:95 click:93 cable:92
      instantly:90 https:88 internet:87 www:86 U4:85 isn't:14 month:81
      com:75 surf:75 
   Attachments: cset="GB2312" type="application/octet-stream"
      name="Guangwen4.zip" 
 
  The first two lines are just the From: and Subject: fields of the
  original message.

  The "Score:" line summarizes the result of the analysis.
  The first number (between 0.0 and 1.0) is the probability that the
  message is actually spam --- or, equivalently, the degree of similarity
  of the message with the spam messages in the corpus.
  The second number (an integer between 0 and 15) is the number of
  "interesting" words found in the message.  "Interesting" words are
  those that occur at least 5 times in the corpus.
  In the example, we have 15 interesting words (the maximum) and a
  score of 1.00, indicating a spam with high certainty.

  The "Details:" line provides an explanation of the score.  It lists
  the 15 most interesting words found in the message, that is, the 15
  interesting words whose probability of denoting a spam is farthest
  away from the neutral 0.5.  Each word is given with its individual score,
  written as a percentage (between 01 and 99) rather than as a
  probability so as to save space.  Here, we see a number of very
  "spammish" words such as "$$$$" or "click", with probability 0.98
  and 0.93 respectively, and a few "innocent" words such as "isn't"
  (probability 0.14).  The "U4" word with probability 0.85 is actually
  a pseudo-word representing a 4-letter word all in uppercase --
  something spammers are fond of.

  The "Attachments:" line summarizes some information about MIME 
  attachments for this message.  Here, we have one attachment of type
  "application/octect-stream", file name "Guangwen4.zip", and
  character set "GB2312".  The latter is an encoding for Chinese
  and a solid hint that this is a Chinese spam
  (assuming that, like me, you can't read Chinese).

  Normally, when running "spamoracle test goodmails", most messages
  should come out with low score (0.2 or less), and when running
  "spamoracle test spammails", most messages should come out with a
  high score (0.8 or more).  If not, your corpus isn't very good,
  or not well classified into spam and non-spam.  To quickly see
  the outliers, you can reduce the interval of scores for which
  message summaries are displayed, as follows:

          spamoracle test -min 0.2 goodmails | more
                     # Shows only good mails with score >= 0.2
          spamoracle test -max 0.8 spammails | more
                     # Shows only spam mails with score <= 0.8

  Now, for a more challenging test, take a mailbox that contains
  unfiltered e-mails, i.e. a mixture of spam and legitimate e-mails,
  and run it through "spamoracle":

          spamoracle test mymailbox | less

  Marvel at how well the oracle recognizes spam from the rest!
  If the result isn't that marvelous to you, keep in mind that
  certain spams are just too short to be recognized (not enough
  significant words).  Also, perhaps your corpus was too small,
  or not well categorized...


USING SPAMORACLE AND PROCMAIL TO FILTER YOUR INCOMING E-MAIL:

  Once the database is built, you're ready to run incoming e-mails
  through SpamOracle.  The command "spamoracle mark" reads
  one e-mail from standard input, and copies it to standard output,
  with two headers inserted: "X-Spam:" and "X-Attachments:".
  The X-Spam: header is as follows:

    X-Spam: yes; <score>; <details>
  or
    X-Spam: no; <score>; <details>
  or
    X-Spam: unknown; <score>; <details>

  The <score> and <details> are as described for "spamoracle test".
  The "yes" / "no" / "unknown" synthesizes the results of the
  analysis:
     "yes":     score >= 0.8 and at least 5 interesting words found
     "no":      score <= 0.2 and at least 5 interesting words found
     "unknown": otherwise
  The "unknown" case generally occurs for very short messages,
  where not enough interesting words were found.

  The "X-Attachments:" header contains the same information as
  the "Attachments:" output of "spamoracle test", that is, a summary
  of the message attachments.

  To process automatically your incoming e-mail through "spamoracle"
  and act upon the results of the analysis, just insert the following
  "recipes" in the file ~/.procmailrc:

    :0fw
    | /usr/local/bin/spamoracle mark

    :0
    * ^X-Spam: yes;
    spambox

  What these cryptic commands mean is:

  - Run every mail through the "spamoracle mark" command.
    (If spamoracle wasn't installed in /usr/local/bin, adjust the path
    as necessary.)  This adds two headers to the message:
    "X-Spam:" and "X-Attachments:", describing the results of the
    spam analysis and the attachment analysis.

  - If we have an "X-Spam: yes" header, deliver the message to the
    file "spambox" rather than to your regular mailbox.  Presumably,
    you'll read "spambox" once in a while, but less often than your
    regular mailbox.  Daring users can put "/dev/null" instead of "spambox"
    to just throw away the message, but please don't do that until
    you've used SpamOracle for a while and are happy with the results.
    SpamOracle's false positive rate (i.e. legitimate mails
    classified as spam) is low (0.1% on my mail) but not null.
    So, better save the presumed spams somewhere, and scan them
    quickly from time to time.

  If you'd like to enjoy a bit of attachment-based filtering, here are
  some procmail rules for that:

    :0
    * ^X-Attachments:.*name=".*\.(pif|scr|exe|bat)"
    spambox


    :0
    * ^X-Attachments:.*type="audio/(x-wav|x-midi)
    spambox

    :0
    * ^(Content-type:.*|X-Attachments:.*cset="|^Subject:.*=\?)(ks_c|gb2312|iso-2|euc-|big5|windows-1251)
    spambox

  The first rule treats as spam every mail that has a Windows
  executable as attachment.  These mails are typically sent by viruses.
  The second rule does the same with attachments of type x-wav or x-midi.
  I never normally receive music by e-mail, however some popular
  e-mail viruses seem fond of these attachment types.
  The third rule treats as spam every mail that uses character
  encodings corresponding to Korean, Chinese, Japanese, and Cyrillic.
  Since I don't read any of these scripts, why not get rid of the
  messages immediately?


UPDATING THE DATABASE:

  At any time, you can add more known spams or known legitimate
  messages to the database by using the "spamoracle add" command.

  For instance, if you find a spam message that was not classified
  as such, run it through "spamoracle add -spam", so that SpamOracle
  can learn from its mistake.  (Without additional arguments, this
  command will read a single message from standard input and record it
  as spam.)  Under mutt for instance, just highlight the spam message
  and type
      |spamoracle add -spam

  Similarly, if you find a legitimate message while checking your
  spam box, run it through "spamoracle add -good".

  Another option is to collect more known spams or more known good 
  messages into mailbox files, and once in a while do
      spamoracle add -good new_good_mails
  or
      spamoracle add -spam new_spam_mails


TECHNICAL DETAILS:

  SpamOracle's notion of "word" is the following:
  - any run of 3 to 12 letters, single quotes, dashes (-)
  - any run of 3 to 8 digits, dots, commas, dollar, euro and
    percent signs.
  If support for non-English european languages was
  compiled in, letters also include the relevant accented letters 
  for the languages in question.  All words are
  mapped to lowercase, and accented letters are mapped to the corresponding
  non-accented letters.

  In addition, a run of three or more uppercase letters generates a
  pseudo-word "Un" where n is the length of the run.  Similarly,
  a run of three or more non-ASCII characters (code >= 128) generates
  a pseudo-word "Wn" where n is the length of the run.

  For instance, the following text:

      SUMMER in English is written "été" in French ¹²³

  is processed into the following words, assuming French support was selected:

      U5 summer english written ete french W3

  and if French support was not selected:

      U5 summer english written french W3

  For your edification and entertainment, the contents of the database
  can be dumped with the "spamoracle list <regexp>" command, where <regexp>
  is an Emacs-style regexp specifying the words you're interested in, e.g.:

      spamoracle list '.*'     # show all words -- big list!
      spamoracle list 'sex.*'
      spamoracle list 'caml.*'

  It is possible to tweak many of the parameters that govern filtering
  via the configuration file ~/.spamoracle.conf.  The configurable
  parameters are listed and explained in the man page spamoracle.conf (5).
  All parameters have reasonable default values, but you may try
  to tweak them to get better filtering.