Configuring full-text search

Mastodon supports full-text search when Elasticsearch is available. It is strongly recommended to configure this feature.

Mastodon’s full-text search allows logged-in users to find results from:

• public statuses from accounts that opted into appearing in search results
• their own statuses
• their mentions
• their favourites
• their bookmarks
• accounts (display name, usernames and bios)

It deliberately does not allow searching for arbitrary strings in the entire database.

Installing Elasticsearch

Elasticsearch requires a Java runtime. If you don’t have Java already installed, do it now. Assuming you are logged in as root:

apt install openjdk-17-jre-headless

Add the official Elasticsearch repository to apt:

wget -O /usr/share/keyrings/elasticsearch.asc https://artifacts.elastic.co/GPG-KEY-elasticsearch
echo "deb [signed-by=/usr/share/keyrings/elasticsearch.asc] https://artifacts.elastic.co/packages/7.x/apt stable main" > /etc/apt/sources.list.d/elastic-7.x.list

Now you can install Elasticsearch:

apt update
apt install elasticsearch

To start Elasticsearch:

systemctl daemon-reload
systemctl enable --now elasticsearch

Configuring Mastodon

Edit .env.production to add the following variables:

ES_ENABLED=true
ES_HOST=localhost
ES_PORT=9200
ES_PRESET= # single_node_cluster, small_cluster or large_cluster
# ES_USER=
# ES_PASS=

Note: If using TLS, prepend the hostname with https://. For example: https://elastic.example.com.

Choosing the correct preset

The value for ES_PRESET depends on the size of your Elasticsearch and will be used to set the number of shards and replicas for your indices to the best value for your setup:

• single_node_cluster if you only have one node in your Elasticsearch cluster. Indices will be configured without any replica
• small_cluster if you have less than 6 nodes in your cluster. Indices will be configured with 1 replica
• large_cluster if you have 6 or more nodes in your cluster. Indices will be configured with more shards than with the small_cluster setting, to allow them to be distributed over more nodes

If you have multiple Mastodon servers on the same machine, and you are planning to use the same Elasticsearch installation for all of them, make sure that all of them have unique REDIS_NAMESPACE in their configurations, to differentiate the indices. If you need to override the prefix of the Elasticsearch indices, you can set ES_PREFIX directly.

Security

By default, Elasticsearch does not handle any authentication and every request is made with full admin permission. We strongly advise you to configure Elasticsearch security features on your cluster.

To configure it, please refer to the official documentation. It will guide you through:

• Enabling the security features (xpack.security.enabled: true)
• Creating passwords for built-in users

Once done, you can create a custom role for Mastodon to connect.

For example (please adapt this snippet to use your Elastic admin password):

curl -X POST -u elastic:admin_password "localhost:9200/_security/role/mastodon_full_access?pretty" -H 'Content-Type: application/json' -d'
{
  "cluster": ["monitor"],
  "indices": [{
    "names": ["*"],
    "privileges": ["read", "monitor", "write", "manage"]
  }]
}
'

Elasticsearch documentation for role creation

Once the role is created, you can create a user for the Mastodon server to use, and assign it the role.

For example (please adapt this snippet to use your Elastic admin password, and customize your new user mastodon user password):

curl -X POST -u elastic:admin_password "localhost:9200/_security/user/mastodon?pretty" -H 'Content-Type: application/json' -d'
{
  "password" : "l0ng-r4nd0m-p@ssw0rd",
  "roles" : ["mastodon_full_access"]
}
'

Elasticsearch documentation for user creation

Once this is done, you need to configure Mastodon to use the credentials for your newly created user.

In .env.production, adjust your configuration:

ES_USER=mastodon
ES_PASS=l0ng-r4nd0m-p@ssw0rd

You are all set, and your Elasticsearch server should be much more secure!

Populate the indices

After saving the new configuration, restart Mastodon processes for it to take effect:

systemctl restart mastodon-sidekiq
systemctl reload mastodon-web

Now it’s time to create the Elasticsearch indices and fill them with data:

su - mastodon
cd live
RAILS_ENV=production bin/tootctl search deploy

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
# Both values should be the same
-Xms2048m
-Xmx2048m

Search optimization for other languages

Chinese search optimization

The standard analyzer is the default for Elasticsearch, but for some languages like Chinese it may not be the optimal choice. To enhance the search experience, consider installing a language-specific analyzer. Before creating indices in Elasticsearch, be sure to install the following extensions:

• elasticsearch-analysis-ik
• elasticsearch-analysis-stconvert

And then modify Mastodon’s index definition as follows:

diff --git a/app/chewy/accounts_index.rb b/app/chewy/accounts_index.rb
--- a/app/chewy/accounts_index.rb
+++ b/app/chewy/accounts_index.rb
@@ -4,7 +4,7 @@ class AccountsIndex < Chewy::Index
   settings index: { refresh_interval: '5m' }, analysis: {
     analyzer: {
       content: {
-        tokenizer: 'whitespace',
+        tokenizer: 'ik_max_word',
         filter: %w(lowercase asciifolding cjk_width),
       },

diff --git a/app/chewy/statuses_index.rb b/app/chewy/statuses_index.rb
--- a/app/chewy/statuses_index.rb
+++ b/app/chewy/statuses_index.rb
@@ -16,9 +16,17 @@ class StatusesIndex < Chewy::Index
         language: 'possessive_english',
       },
     },
+    char_filter: {
+      tsconvert: {
+        type: 'stconvert',
+        keep_both: false,
+        delimiter: '#',
+        convert_type: 't2s',
+      },
+    },
     analyzer: {
       content: {
-        tokenizer: 'uax_url_email',
+        tokenizer: 'ik_max_word',
         filter: %w(
           english_possessive_stemmer
           lowercase
@@ -27,6 +35,7 @@ class StatusesIndex < Chewy::Index
           english_stop
           english_stemmer
         ),
+        char_filter: %w(tsconvert),
       },
     },
   }
diff --git a/app/chewy/tags_index.rb b/app/chewy/tags_index.rb
--- a/app/chewy/tags_index.rb
+++ b/app/chewy/tags_index.rb
@@ -2,10 +2,19 @@

 class TagsIndex < Chewy::Index
   settings index: { refresh_interval: '15m' }, analysis: {
+    char_filter: {
+      tsconvert: {
+        type: 'stconvert',
+        keep_both: false,
+        delimiter: '#',
+        convert_type: 't2s',
+      },
+    },
     analyzer: {
       content: {
-        tokenizer: 'keyword',
+        tokenizer: 'ik_max_word',
         filter: %w(lowercase asciifolding cjk_width),
+        char_filter: %w(tsconvert),
       },

       edge_ngram: {

Last updated September 13, 2024 · Improve this page