We have released the new stable version of Rspamd today. It includes couple of important fixes and improvements. Here is the list of the most important ones.
Base64 decoding fix
We have found and resolved a serious flaw in the current base64 decoder in Rspamd. It could lead to the corrupted output in case if decoder finds out some non base64 characters, for example, spaces or newlines. This bug could affect statistics, fuzzy checks and couple of other fields in Rspamd. Hence, we recommend to update to
1.5.3 as soon as possible.
This release includes experimental feature that allows to save history in Redis. There is the initial WebUI support of this feature, however, it is not now enabled by default. In future, we plan to enable it and to enchance history with a set of new options:
- displaying of sender and recipient in history table;
- support of symbols options;
- clustered history;
- dynamic load of history rows;
- compressed history;
All these features are implemented for the backend part (namely, Rspamd controller) but it still requires some major rework of the web interface itself, therefore, this work is postponed till the next version.
Dkim plugin improvements
DKIM signing module now supports the type of private key passed to the module: in addition to PEM format stored in a file, DKIM signing now supports raw keys, base64 encoded keys and PEM keys from raw strings.
DKIM signing now also supports maps for selecting domains to sign.
Other plugins improvements
greylist plugin now supports excluding low-scoring messages from greylisting
whitelist plugin can now load list of maps
ratelimit plugin now excludes greylisted messages
metadata exporter uses rule-specific settings for emails
metadata exporter can now use non-ASCII characters in reports
Here is the list of rules that are fixed or reworked:
URI_COUNT_ODD rule now excludes visual URLs which reduces its FP rate
HAS_X_PRIO* rules are reworked to the normal Rspamd symbols conventions
misc.lua has been splitted to multiple modules that share the common rules
Other bugs fixes
- imported important fixes for
- fixed local networks proxying
- fixed memory corruption in periodic tasks during worker cleanup phase
- fixed subject rewriting
zstd lua API to avoid extra reallocation
We are pleased to announce the new major Rspamd release
1.5 today. This release incudes a lot of major reworks, new cool features and a significant number of bugs being fixed. The update from the previous versions shouldn’t be hard, however, please check the migration document to be sure that the new version will not break the existing configuration.
Here is a list of major changes for this version.
New MIME parser
Rspamd has used the GMime library for a very long time but we decided to switch to from it for several reasons.
The main problem is that Rspamd requires very precise control of MIME parsing as it has to deal with broken messages not for displaying purposes but for extracting data from them. This procedure has some simplifications and some complications comparing to a generic MIME parser, such as GMime: for example, we do not need to support streaming mode but we have to deal with many non-standard messages that are intended to be parsed incorrectly by some adversary side, e.g. spammers. The current architecture is described here: https://gist.github.com/vstakhov/937f253d5935ee4158688932589b1dcc
Through use of the new parser, Rspamd can now deal with the following messages:
- Messages with redundant
Currently, Rspamd always prefers multipart types over plain types and text types unless there is not specific binary type (e.g. if there is
- Messages with broken multipart structures:
- new parts after closing boundary (e.g. attachment in multipart/mixed after the closing part)
- incorrect inheritance
- incorrect multipart type (now Rspamd just ignores the exact
- Filenames that are badly encoded (non-utf8)
Content-Transfer-Encoding (now heuristic based):
8bit when content is Base64
qp when content is 8bit
- Messages with no headers or messages with no body
- Messages with mixed newlines in headers and/or body
Switching from libiconv to libicu
Rspamd has switched charset conversion from
libicu. This allowed to speed up the conversion time since
libicu is much faster (~100MB of text from
0,83s user 0,08s system 98% cpu 0,921 total - iconv
0,36s user 0,07s system 95% cpu 0,450 total - libicu
Furthermore, switching to
libicu allowed for implementation of many useful features:
- heuristic charset detection (NGramms for 1byte charsets);
- visual obfuscation detector (e.g.
- better IDNA processing
- better unicode manipulation
The Web interface has been reworked for better representation and configuration:
- The web interface now supports displaying & aggregating statistics from a cluster of Rspamd machines
- The internal structure of the Web Interface has changed to a set of modules so that new features could be implemented without touching the overall logic
- The throughput graph has been improved and now displays a small pie chart for the specified time range
Lua TCP module rework
Rspamd 1.5 Lua TCP module now supports complex protocols with dialogs and states similar to
AnyEvent module in Perl. For example, it is now possible to set a reaction for each communication stage and perform full SMTP or IMAP dialog.
URL redirector module
URL shorteners and redirectors are part of the modern email ecosystem and they are widely used in many emails, both legitimate and not (e.g. in Spam and Phishing). Rspamd has an old and outdated utility service that is intended to resolve such redirects called
redirector.pl. It is written in Perl and hasn’t been updated for a long time. It has a long dependencies list and performs a lot of unnecessary tasks. In Rspamd 1.5, there is a new lightweight lua redirector module which is intented to resolve URLs redirect in a more efficient and simple way. Dereferenced links are processed by SURBL module and added as tags for other modules. Redis is used for caching. This module is not enabled by default so far, but it can easily be enabled by placing
redirector_hosts_map = "/etc/rspamd/redirectors.inc"; in
The Rmilter headers module provides an easy way to add common headers; support is available for Authentication-Results, SpamAssassin-compatible headers and user-defined headers among others.
DKIM signing module
The DKIM signing module provides a simple policy-based approach to DKIM signing similar to Rmilter. It supports multiple cool features, for example, you can now store your DKIM keys in Redis.
Force actions module
The Force actions module provides a way to force actions for messages based on flexible conditions (an expression consisting of symbols to verify presence/absence of & the already-assigned action of a message), optionally setting SMTP messages & rewritten subjects.
Configuration of this module has been reworked to provide more flexible operation & library functions have been added to provide JSON-formatted general message metadata, e-Mail alerts and more - making this module readily useful for quarantines, logging & alerting.
URLs can now be assigned tags and it is the job of the URL tags plugin to persist these in Redis for a period of time; which could be used to avoid redundant checks.
URL reputation plugin
The URL reputation plugin filters URLs for relevance and assigns dynamic reputation to selected TLDs which is persisted in Redis.
Multimap ‘received’ maps
Now multimap can be used to match information extracted from Received headers (which could be filtered based on their position in the message). It is also possible to use SMTP
HELO messages in maps for this module. There are also new URL filters, SMTP message setup depending on map data and the ability to skip archives checks for certain filetypes or maps.
Changes in RBL module
Support has been added for using hashes in
helo RBLs (so that information which can’t be represented in a DNS record could be queried).
Support for Avira SAVAPI in antivirus module
Rspamd antivirus module now supports AVIRA antivirus. This code has been contributed by Christian Rößner.
Neural net plugin improvements
We have fixed couple of issues in the neural network plugin allowing to have multiple configurations in the cluster. We have also fixed couple of issues with storing and loading of learning vectors especially in errors handling paths. New metatokens have been added to improve neural network classification quality.
Fuzzy matching for images
Rspamd fuzzy hashing now support matching of the images attached to emails checked. To enable this feature, Rspamd should be built with
libgd support (provided by the pre-built packages). However, this feature is not currently enabled by default as it seems to be too aggressive when used in conjunction with large fuzzy storages producing a lot of false positive hits.
There are couple of new rules added to Rspamd 1.5:
OMOGRAPH_URL: detects visually confusable URLs
FROM_NAME_HAS_TITLE: fixed title match
grep-like tool inspired by
exigrep has been added to rspamadm- see
rspamadm grep --help for usage information: this provides a convenient way to produce logically collated logs based on search strings/regular expressions.
There are number of improvements regarding the performance of processing:
- Base64 decoder now has
- Better internal caching of various ‘heavy’ objects
- Switching to a faster hash function
- Enabled link time optimizations for the pre-built packages
- Bundled luajit 2.1 which has significant performance improvements to the provided Debian packages
Stability improvements and bug fixes
We constantly improve the stability of Rspamd and in this version we have fixed number of issues related to the graceful
reload. Historically, this command has very poor support and there were a number of issues related to memory leaks and corruptions that could occur during reload. In this release, we have fixed a lot of such issues, therefore, you can use
reload more safely now. We have also eliminated various issues related to unicode processing, Lua API, signals race conditions and other important problems found by Rspamd users.
The next stable release 1.4.1 of Rspamd is available to download. This release includes various bugfixes and couple of new cool features. The most notable new feature is the Clickhouse plugin.
This plugin is intended to export scan data to the clickhouse column oriented database. This feature allows to perform very deep analysis of data and use advanced statistical tool to examine your mail flows and the efficiency of Rspamd. For example, you can find the most abused domains, the largest spam senders, the attachments statistics, URLs statistics and so on and so forth. The module documentation includes some samples of what you can do with this tool.
It was not very convenient that maps could only contain references to external resources. From the version
1.4.1, you can also embed maps into the configuration to simplify small maps definitions:
map = ["elt1", "elt2" ...]; # Embedded map
map = "/some/file.map"; # External map
Lua modules debugging improvements
You can now specify lua modules in
debug_modules to investigate some concrete module without global debug being enabled
Steve Freegard has added a bunch of new rules useful for the actual spam trends, including such rules as:
- Freemail and disposable emails addresses
- Common Message-ID abuse
- Compromised hosts rules
- Rules for upstream services that have already run spam checks
- Commonly abused patterns in From, To and other headers
- Suspicious subjects
- MIME misusages
Multiple fixes to the ANN module
Neural networks has been fixed to work in a distributed environment. Couple of consistency bugs have been found and eliminated during Redis operations.
There are couple of other bugs and memory leaks that were fixed in this release. Please check the full release notes for details.