Rspamd

Fast, free and open-source spam filtering system.

Rspamd 1.6.0 has been released

2017-06-12 00:00:00 +0200

Today, we release the new major version 1.6.0 of Rspamd. The most significant change in this version is the addition of Milter protocol support in Rspamd. Therefore, Rmilter project is finally turned to the abandoned state and should not be used in new installations. All Rmilter users should consider migration to Rspamd milter support. This release has some incompatible changes so please check the migration guide.

Here is the list of most noticeable changes. The full list is available on the changes page.

Milter protocol support

From Rspamd 1.6, rspamd proxy worker supports milter protocol which is supported by some of the popular MTAs, such as Postfix or Sendmail. The introduction of this feature also finally obsoletes the Rmilter project in honor of the new integration method. Milter support is presented in rspamd_proxy only, however, there are two possibilities to use milter protocol:

  • Proxy mode (for large instances) with a dedicated scan layer
  • Self-scan mode (for small instances)

Here, we describe the simplest self-scan option:

In this mode, rspamd_proxy scans messages itself and talk to MTA directly using Milter protocol. The advantage of this mode is its simplicity. Here is a sample configuration for this mode:

# local.d/worker-proxy.inc
milter = yes; # Enable milter mode
timeout = 120s; # Needed for Milter usually
upstream "local" {
  default = yes; # Self-scan upstreams are always default
  self_scan = yes; # Enable self-scan
}

For more advanced proxy usage, please see the corresponding documentation.

ARC support added

There is full support of ARC signatures and seals for emails scanned in Rspamd 1.6.0. ARC signatures can establish that a specific message has been signed and then forwarded by a number of trusted relays. There is a good overview of the ARC standard here: https://dmarc.org/presentations/ARC-Overview-2016Q2-v03.pdf.

Rspamd ARC module supports both verification and signing for outbound messages. Its configuration is very similar to dkim_signing module.

New statistics model for Redis storage

Rspamd 1.6 includes experimental support for new tokens storage in Redis. In this scheme, it is easier to get data about specific tokens and perform tokens expiration. However, this support is not enabled by default in this release but you can try it as well as Bayes expiration plugin by yourself. In future releases, this model will be the default and you will be able to convert the existing storage to the new scheme without data loss.

New expiration algorithm for internal caches

Rspamd now has an implementation of Least Frequently Used (LFU) algorithm instead of the classic Least Recently Used (LRU) used before. The idea comes from the Redis server where it has been used for a long time. With this algorithm, Rspamd will cache frequently used items for longer time and the overall performance of the caches is expected to be better.

DMARC reports support

DMARC module now supports reports sending (using SMTP smarthost) for specific domains and policies. There are many options available for fine tuning of these reports content, frequency, domains and many other. DMARC reports are intended to provide better feedback for the resources that are using DMARC (e.g. paypal.com) from their recipients. Namely, they can detect some phishing trends and react to them somehow.

Spam trap plugin

New plugin spam trap has been added to Rspamd to simplify spamtraps organization. This plugin allows to learn fuzzy storages or/and Bayes from some honeypots.

URL redirector improvements

There are various changes in url redirector module. Namely, it now expires processing items more aggressively to avoid leftovers. Some of dependency issues have also been resolved. Furthermore, this plugin now has a list of top redirection destinations allowing to deal with some bad URLs exploited by spammers.

Multiple metrics support has been removed from Rspamd

From version 1.6, multiple metrics support is now completely removed from Rspamd. The only valid metric is now default. This feature has never ever been used since version 0.2 of Rspamd, however, it consumed some resources and introduced extra complicatinons to the protocol and configuration.

Hence, this feature has been removed and the new endpoint /checkv2 has been added to the protocol. The legacy /check and /symbols endpoints are still using old protocol definition and they will be used for backward compatibility in future.

Compression support in proxy

Rspamd proxy now supports transport compression when sending messages to the scanning layer.

Various performance improvements

Here is a list of significant performance related changes:

  • Hfilter regular expressions now can use hyperscan
  • DKIM body hash is now cached to improve multiple signatures support
  • Snowball stemmers are also cached for better performance

Miscellaneous

Here is a list of other changes made in this release:

  • Various rules fixes (FORWARDED, URI_COUNT_ODD and others)
  • Bugfixes and other improvements
  • New Lua API functions

IP changed for rspamd.com

2017-05-15 00:00:00 +0200

Synopsis

We have migrated hardware that served https://rspamd.com site and related services including fuzzy storage.

Problem description

All Rspamd users who are using rspamd.com fuzzy storage might see the following messages in the log:

fuzzy_check_timer_callback: got IO timeout with server rspamd.com(5.9.155.182), after 3 retransmits

Normally, Rspamd re-resolves hostnames in this case. However, if there is a single server specified (as enabled by default) there is no resolving on errors. Unfortunately, this bug has been fixed merely in the master branch and is not released in the stable versions yet.

Potential outcome

The quality of filtering might be temporary reduced as fuzzy storage helps to filter certain spam types.

Workaround

You just need to restart Rspamd and it will use the new IP address as intended. We do apologise for any inconveniences caused.

Rspamd 1.5.3 has been released

2017-03-17 00:00:00 +0100

We have released the new stable version of Rspamd today. It includes couple of important fixes and improvements. Here is the list of the most important ones.

Base64 decoding fix

We have found and resolved a serious flaw in the current base64 decoder in Rspamd. It could lead to the corrupted output in case if decoder finds out some non base64 characters, for example, spaces or newlines. This bug could affect statistics, fuzzy checks and couple of other fields in Rspamd. Hence, we recommend to update to 1.5.3 as soon as possible.

Redis history

This release includes experimental feature that allows to save history in Redis. There is the initial WebUI support of this feature, however, it is not now enabled by default. In future, we plan to enable it and to enchance history with a set of new options:

  • displaying of sender and recipient in history table;
  • support of symbols options;
  • clustered history;
  • dynamic load of history rows;
  • compressed history;

All these features are implemented for the backend part (namely, Rspamd controller) but it still requires some major rework of the web interface itself, therefore, this work is postponed till the next version.

Dkim plugin improvements

DKIM signing module now supports the type of private key passed to the module: in addition to PEM format stored in a file, DKIM signing now supports raw keys, base64 encoded keys and PEM keys from raw strings.

DKIM signing now also supports maps for selecting domains to sign.

Other plugins improvements

  • greylist plugin now supports excluding low-scoring messages from greylisting
  • whitelist plugin can now load list of maps
  • ratelimit plugin now excludes greylisted messages
  • metadata exporter uses rule-specific settings for emails
  • metadata exporter can now use non-ASCII characters in reports

Rules update

Here is the list of rules that are fixed or reworked:

  • URI_COUNT_ODD rule now excludes visual URLs which reduces its FP rate
  • RCPT_COUNT* and HAS_X_PRIO* rules are reworked to the normal Rspamd symbols conventions
  • misc.lua has been split to multiple modules that share the common rules

Other bugs fixes

  • imported important fixes for ac-trie module
  • fixed local networks proxying
  • fixed memory corruption in periodic tasks during worker cleanup phase
  • fixed subject rewriting
  • improved zstd lua API to avoid extra reallocation