1

I'm trying to follow this to get Solr search setup with my Dovecot install:

https://doc.dovecot.org/configuration_manual/fts/solr/#fts-backend-solr

It all seems to work fine - and I can see the core in Solr admin. I'm using Solr 8.9.0, but it seems happy with the config files.

However, one of the commands it says to run to actually index:

doveadm fts rescan -u foo@bar.com

I get:

Fatal: Plugin 'fts_solr' not found from directory /usr/lib/dovecot/modules

Sure enough, I don't see anything in anything by that name:

root@east:/usr/lib/dovecot/modules# ls -lh *fts*
-rw-r--r-- 1 root root 263K Jul  7 17:17 lib20_fts_plugin.so
-rw-r--r-- 1 root root  71K Jul  7 17:17 lib21_fts_squat_plugin.so

What am I missing? Do I need to manually install the plugin as well?

dovecot --version
2.3.7.2 (3c910f64b)

UPDATE: Looking around, I can see dovecot-fts-solr a plugin (https://alpine.pkgs.org/3.15/alpine-main-aarch64/dovecot-fts-solr-2.3.17.1-r0.apk.html ) . I can't seem to get it to install through. Not too sure how to proceed? (I could justr download the .so file and put it into /usr/lib/dovecot/modules , but I'm pretty sure that won't work :))

UPDATE 2: So I managed to find:

apt-get install dovecot-solr

This creates the lib21_fts_solr_plugin.so file. However, when I try and re-index the messages I don't get anything:

doveadm -D fts rescan -u hello@hatlamp.com
Debug: Loading modules from directory: /usr/lib/dovecot/modules
Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so
Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined symbol: expire_set_deinit (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message)
Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message)
Debug: Module loaded: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message)
doveadm(hello@hatlamp.com)<35131><>: Debug: auth-master: userdb lookup(hello@hatlamp.com): Started userdb lookup
doveadm(hello@hatlamp.com)<35131><>: Debug: auth-master: conn unix:/var/run/dovecot//auth-userdb: Connecting
doveadm(hello@hatlamp.com)<35131><>: Debug: auth-master: conn unix:/var/run/dovecot//auth-userdb: Client connected (fd=10)
doveadm(hello@hatlamp.com)<35131><>: Debug: auth-master: userdb lookup(hello@hatlamp.com): auth USER input: hello@hatlamp.com uid=1011 gid=8 home=/home/hatlamp quota_rule=*:storage=0M
doveadm(hello@hatlamp.com)<35131><>: Debug: auth-master: userdb lookup(hello@hatlamp.com): Finished userdb lookup (username=hello@hatlamp.com uid=1011 gid=8 home=/home/hatlamp quota_rule=*:storage=0M)
doveadm(hello@hatlamp.com)<35131><>: Debug: Added userdb setting: plugin/quota_rule=*:storage=0M
doveadm(hello@hatlamp.com): Debug: Effective uid=1011, gid=8, home=/home/hatlamp
doveadm(hello@hatlamp.com): Debug: Namespace : type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:/home/hatlamp/mail/hatlamp.com/hello
doveadm(hello@hatlamp.com): Debug: maildir++: root=/home/hatlamp/mail/hatlamp.com/hello, index=, indexpvt=, control=, inbox=/home/hatlamp/mail/hatlamp.com/hello, alt=
doveadm(hello@hatlamp.com): Debug: auth-master: conn unix:/var/run/dovecot//auth-userdb: Disconnected: Connection closed (fd=10)

Looking at Solr's interface, there are no articles:

enter image description here

Andrew Newby
  • 1,102
  • 2
  • 25
  • 58
  • 1
    Have you configured the URL in dovecot, and have you verified that this URL is accessible? Have you actually tried to access anything from the index after initiating the rescan? Its not super clear from the docs, but *rescan* alone just ensures (e.g. by simply [starting from scratch](https://github.com/dovecot/core/blob/f59f6b660224b9ebd1d5b423a0de0da6ea03dcfb/src/plugins/fts-solr/fts-backend-solr.c#L638)) that the next lookup will not see stale or incomplete data, it does *not* ensure that the index is immediately (re-)created. – anx Dec 17 '22 at 14:26
  • @anx thanks for the reply. I found the issue in the end. I just needed to send an email to the account, and it would scan the folders! So now its indexed and working well :) – Andrew Newby Dec 19 '22 at 06:36
  • AFAIK receiving a message is only sufficient for a mailbox that had been used before (meaning Dovecot already knew what to index). Searching via IMAP should always be sufficient, so I put that into my answer. – anx Dec 19 '22 at 11:43

1 Answers1

1

Use the FTS index, then you will see results in your fts backend.

Synchronising the internal state of full text content submitted to the FTS engine via the doveadm fts reindex command does not necessarily do much by itself. In the case of the solr binding, it simply moves the cursor back to zero.

This removes mails from the index that have already been expunged and makes sure that the next doveadm index will index all the missing mails (if any). Note that currently most FTS backends do not implement this properly, but instead they delete all the FTS indexes. This may change in the future versions. -- from man doveadm-fts, emphasis mine

Only when something (issuing a search, posting new mail, manual doveadm index) triggers updating the index after that command, that is when the data is sent (in your case, to solr).

anx
  • 8,963
  • 5
  • 24
  • 48