Merge branch 'develop' into issue/2099

CHANGELOG.md

@@ -9,8 +9,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 - Renamed `:await_up_timeout` in `:connections_pool` namespace to `:connect_timeout`, old name is deprecated.
 - Renamed `:timeout` in `pools` namespace to `:recv_timeout`, old name is deprecated.
+- The `discoverable` field in the `User` struct will now add a NOINDEX metatag to profile pages when false.
+- Users with the `discoverable` field set to false will not show up in searches.
 - Minimum lifetime for ephmeral activities changed to 10 minutes and made configurable (`:min_lifetime` option).
-
 ### Removed
 
 - **Breaking:** `Pleroma.Workers.Cron.StatsWorker` setting from Oban `:crontab` (moved to a simpler implementation).
@@ -19,12 +20,27 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Removed `:managed_config` option. In practice, it was accidentally removed with 2.0.0 release when frontends were
 switched to a new configuration mechanism, however it was not officially removed until now.
 
-## unreleased-patch - ???
+## [2.1.2] - 2020-09-17
+
+### Security
+
+- Fix most MRF rules either crashing or not being applied to objects passed into the Common Pipeline (ChatMessage, Question, Answer, Audio, Event).
 
 ### Fixed
 
-- Welcome Chat messages preventing user registration with MRF Simple Policy applied to the local instance
+- Welcome Chat messages preventing user registration with MRF Simple Policy applied to the local instance.
-- Mastodon API: the public timeline returning an error when the `reply_visibility` parameter is set to `self` for an unauthenticated user
+- Mastodon API: the public timeline returning an error when the `reply_visibility` parameter is set to `self` for an unauthenticated user.
+- Mastodon Streaming API: Handler crashes on authentication failures, resulting in error logs.
+- Mastodon Streaming API: Error logs on client pings.
+- Rich media: Log spam on failures. Now the error is only logged once per attempt.
+
+### Changed
+
+- Rich Media: A HEAD request is now done to the url, to ensure it has the appropriate content type and size before proceeding with a GET.
+
+### Upgrade notes
+
+1. Restart Pleroma
 
 ## [2.1.1] - 2020-09-08
 
@@ -41,6 +57,12 @@ switched to a new configuration mechanism, however it was not officially removed
 ### Added
 - Rich media failure tracking (along with `:failure_backoff` option).
 
+<details>
+  <summary>Admin API Changes</summary>
+
+- Add `PATCH /api/pleroma/admin/instance_document/:document_name` to modify the Terms of Service and Instance Panel HTML pages via Admin API
+</details>
+
 ### Fixed
 - Default HTTP adapter not respecting pool setting, leading to possible OOM.
 - Fixed uploading webp images when the Exiftool Upload Filter is enabled by skipping them

docs/API/admin_api.md

@@ -1334,3 +1334,166 @@ Loads json generated from `config/descriptions.exs`.
 { }
 
 ```
+
+## GET /api/pleroma/admin/users/:nickname/chats
+
+### List a user's chats
+
+- Params: None
+
+- Response:
+
+```json
+[
+   {
+      "sender": {
+        "id": "someflakeid",
+        "username": "somenick",
+        ...
+      },
+      "receiver": {
+        "id": "someflakeid",
+        "username": "somenick",
+        ...
+      },
+      "id" : "1",
+      "unread" : 2,
+      "last_message" : {...}, // The last message in that chat
+      "updated_at": "2020-04-21T15:11:46.000Z"
+   }
+]
+```
+
+## GET /api/pleroma/admin/chats/:chat_id
+
+### View a single chat
+
+- Params: None
+
+- Response:
+
+```json
+{
+  "sender": {
+    "id": "someflakeid",
+    "username": "somenick",
+    ...
+  },
+  "receiver": {
+    "id": "someflakeid",
+    "username": "somenick",
+    ...
+  },
+  "id" : "1",
+  "unread" : 2,
+  "last_message" : {...}, // The last message in that chat
+  "updated_at": "2020-04-21T15:11:46.000Z"
+}
+```
+
+## GET /api/pleroma/admin/chats/:chat_id/messages
+
+### List the messages in a chat
+
+- Params: `max_id`, `min_id`
+
+- Response:
+
+```json
+[
+  {
+    "account_id": "someflakeid",
+    "chat_id": "1",
+    "content": "Check this out :firefox:",
+    "created_at": "2020-04-21T15:11:46.000Z",
+    "emojis": [
+      {
+        "shortcode": "firefox",
+        "static_url": "https://dontbulling.me/emoji/Firefox.gif",
+        "url": "https://dontbulling.me/emoji/Firefox.gif",
+        "visible_in_picker": false
+      }
+    ],
+    "id": "13",
+    "unread": true
+  },
+  {
+    "account_id": "someflakeid",
+    "chat_id": "1",
+    "content": "Whats' up?",
+    "created_at": "2020-04-21T15:06:45.000Z",
+    "emojis": [],
+    "id": "12",
+    "unread": false
+  }
+]
+```
+
+## DELETE /api/pleroma/admin/chats/:chat_id/messages/:message_id
+
+### Delete a single message
+
+- Params: None
+
+- Response:
+
+```json
+{
+  "account_id": "someflakeid",
+  "chat_id": "1",
+  "content": "Check this out :firefox:",
+  "created_at": "2020-04-21T15:11:46.000Z",
+  "emojis": [
+    {
+      "shortcode": "firefox",
+      "static_url": "https://dontbulling.me/emoji/Firefox.gif",
+      "url": "https://dontbulling.me/emoji/Firefox.gif",
+      "visible_in_picker": false
+    }
+  ],
+  "id": "13",
+  "unread": false
+}
+```
+
+## `GET /api/pleroma/admin/instance_document/:document_name`
+
+### Get an instance document
+
+- Authentication: required
+
+- Response:
+
+Returns the content of the document
+
+```html
+<h1>Instance panel</h1>
+```
+
+## `PATCH /api/pleroma/admin/instance_document/:document_name`
+- Params:
+  - `file` (the file to be uploaded, using multipart form data.)
+
+### Update an instance document
+
+- Authentication: required
+
+- Response:
+
+``` json
+{
+  "url": "https://example.com/instance/panel.html"
+}
+```
+
+## `DELETE /api/pleroma/admin/instance_document/:document_name`
+
+### Delete an instance document
+
+- Response:
+
+``` json
+{
+  "url": "https://example.com/instance/panel.html"
+}
+```

lib/mix/tasks/pleroma/config.ex

@@ -32,7 +32,8 @@ defmodule Mix.Tasks.Pleroma.Config do
 
   @spec migrate_to_db(Path.t() | nil) :: any()
   def migrate_to_db(file_path \\ nil) do
-    if Pleroma.Config.get([:configurable_from_database]) do
+    with true <- Pleroma.Config.get([:configurable_from_database]),
+         :ok <- Pleroma.Config.DeprecationWarnings.warn() do
       config_file =
         if file_path do
           file_path
@@ -46,7 +47,8 @@ defmodule Mix.Tasks.Pleroma.Config do
 
       do_migrate_to_db(config_file)
     else
-      migration_error()
+      :error -> deprecation_error()
+      _ -> migration_error()
     end
   end
 
@@ -120,6 +122,10 @@ defmodule Mix.Tasks.Pleroma.Config do
     )
   end
 
+  defp deprecation_error do
+    shell_error("Migration is not allowed until all deprecation warnings have been resolved.")
+  end
+
   if Code.ensure_loaded?(Config.Reader) do
     defp config_header, do: "import Config\r\n\r\n"
     defp read_file(config_file), do: Config.Reader.read_imports!(config_file)

lib/pleroma/chat.ex

@@ -6,7 +6,9 @@ defmodule Pleroma.Chat do
   use Ecto.Schema
 
   import Ecto.Changeset
+  import Ecto.Query
 
+  alias Pleroma.Chat
   alias Pleroma.Repo
   alias Pleroma.User
 
@@ -69,4 +71,12 @@ defmodule Pleroma.Chat do
       conflict_target: [:user_id, :recipient]
     )
   end
+
+  @spec for_user_query(FlakeId.Ecto.CompatType.t()) :: Ecto.Query.t()
+  def for_user_query(user_id) do
+    from(c in Chat,
+      where: c.user_id == ^user_id,
+      order_by: [desc: c.updated_at]
+    )
+  end
 end

lib/pleroma/config/deprecation_warnings.ex

@@ -26,6 +26,10 @@ defmodule Pleroma.Config.DeprecationWarnings do
       !!!DEPRECATION WARNING!!!
       You are using the old configuration mechanism for the hellthread filter. Please check config.md.
       """)
+
+      :error
+    else
+      :ok
     end
   end
 
@@ -47,17 +51,26 @@ defmodule Pleroma.Config.DeprecationWarnings do
 
       config :pleroma, :mrf_user_allowlist, #{inspect(rewritten, pretty: true)}
       """)
+
+      :error
+    else
+      :ok
     end
   end
 
   def warn do
-    check_hellthread_threshold()
+    with :ok <- check_hellthread_threshold(),
-    mrf_user_allowlist()
+         :ok <- mrf_user_allowlist(),
-    check_old_mrf_config()
+         :ok <- check_old_mrf_config(),
-    check_media_proxy_whitelist_config()
+         :ok <- check_media_proxy_whitelist_config(),
-    check_welcome_message_config()
+         :ok <- check_welcome_message_config(),
-    check_gun_pool_options()
+         :ok <- check_gun_pool_options(),
-    check_activity_expiration_config()
+         :ok <- check_activity_expiration_config() do
+      :ok
+    else
+      _ ->
+        :error
+    end
   end
 
   def check_welcome_message_config do
@@ -74,6 +87,10 @@ defmodule Pleroma.Config.DeprecationWarnings do
       \n* `config :pleroma, :instance, welcome_user_nickname` is now `config :pleroma, :welcome, :direct_message, :sender_nickname`
       \n* `config :pleroma, :instance, welcome_message` is now `config :pleroma, :welcome, :direct_message, :message`
       """)
+
+      :error
+    else
+      :ok
     end
   end
 
@@ -101,8 +118,11 @@ defmodule Pleroma.Config.DeprecationWarnings do
           end
       end)
 
-    if warning != "" do
+    if warning == "" do
+      :ok
+    else
       Logger.warn(warning_preface <> warning)
+      :error
     end
   end
 
@@ -115,6 +135,10 @@ defmodule Pleroma.Config.DeprecationWarnings do
       !!!DEPRECATION WARNING!!!
       Your config is using old format (only domain) for MediaProxy whitelist option. Setting should work for now, but you are advised to change format to scheme with port to prevent possible issues later.
       """)
+
+      :error
+    else
+      :ok
     end
   end
 
@@ -157,6 +181,9 @@ defmodule Pleroma.Config.DeprecationWarnings do
       Logger.warn(Enum.join([warning_preface | pool_warnings]))
 
       Config.put(:pools, updated_config)
+      :error
+    else
+      :ok
     end
   end
 

lib/pleroma/moderation_log.ex

@@ -320,6 +320,19 @@ defmodule Pleroma.ModerationLog do
     |> insert_log_entry_with_message()
   end
 
+  @spec insert_log(%{actor: User, action: String.t(), subject_id: String.t()}) ::
+          {:ok, ModerationLog} | {:error, any}
+  def insert_log(%{actor: %User{} = actor, action: "chat_message_delete", subject_id: subject_id}) do
+    %ModerationLog{
+      data: %{
+        "actor" => %{"nickname" => actor.nickname},
+        "action" => "chat_message_delete",
+        "subject_id" => subject_id
+      }
+    }
+    |> insert_log_entry_with_message()
+  end
+
   @spec insert_log_entry_with_message(ModerationLog) :: {:ok, ModerationLog} | {:error, any}
   defp insert_log_entry_with_message(entry) do
     entry.data["message"]
@@ -627,6 +640,17 @@ defmodule Pleroma.ModerationLog do
     "@#{actor_nickname} updated users: #{users_to_nicknames_string(subjects)}"
   end
 
+  @spec get_log_entry_message(ModerationLog) :: String.t()
+  def get_log_entry_message(%ModerationLog{
+        data: %{
+          "actor" => %{"nickname" => actor_nickname},
+          "action" => "chat_message_delete",
+          "subject_id" => subject_id
+        }
+      }) do
+    "@#{actor_nickname} deleted chat message ##{subject_id}"
+  end
+
   defp nicknames_to_string(nicknames) do
     nicknames
     |> Enum.map(&"@#{&1}")

lib/pleroma/user/search.ex

@@ -52,6 +52,7 @@ defmodule Pleroma.User.Search do
     |> base_query(following)
     |> filter_blocked_user(for_user)
     |> filter_invisible_users()
+    |> filter_discoverable_users()
     |> filter_internal_users()
     |> filter_blocked_domains(for_user)
     |> fts_search(query_string)
@@ -122,6 +123,10 @@ defmodule Pleroma.User.Search do
     from(q in query, where: q.invisible == false)
   end
 
+  defp filter_discoverable_users(query) do
+    from(q in query, where: q.discoverable == true)
+  end
+
   defp filter_internal_users(query) do
     from(q in query, where: q.actor_type != "Application")
   end

lib/pleroma/web/activity_pub/mrf.ex

@@ -5,16 +5,34 @@
 defmodule Pleroma.Web.ActivityPub.MRF do
   @callback filter(Map.t()) :: {:ok | :reject, Map.t()}
 
-  def filter(policies, %{} = object) do
+  def filter(policies, %{} = message) do
     policies
-    |> Enum.reduce({:ok, object}, fn
+    |> Enum.reduce({:ok, message}, fn
-      policy, {:ok, object} -> policy.filter(object)
+      policy, {:ok, message} -> policy.filter(message)
       _, error -> error
     end)
   end
 
   def filter(%{} = object), do: get_policies() |> filter(object)
 
+  def pipeline_filter(%{} = message, meta) do
+    object = meta[:object_data]
+    ap_id = message["object"]
+
+    if object && ap_id do
+      with {:ok, message} <- filter(Map.put(message, "object", object)) do
+        meta = Keyword.put(meta, :object_data, message["object"])
+        {:ok, Map.put(message, "object", ap_id), meta}
+      else
+        {err, message} -> {err, message, meta}
+      end
+    else
+      {err, message} = filter(message)
+
+      {err, message, meta}
+    end
+  end
+
   def get_policies do
     Pleroma.Config.get([:mrf, :policies], []) |> get_policies()
   end

lib/pleroma/web/activity_pub/mrf/keyword_policy.ex

@@ -20,9 +20,17 @@ defmodule Pleroma.Web.ActivityPub.MRF.KeywordPolicy do
     String.match?(string, pattern)
   end
 
-  defp check_reject(%{"object" => %{"content" => content, "summary" => summary}} = message) do
+  defp object_payload(%{} = object) do
+    [object["content"], object["summary"], object["name"]]
+    |> Enum.filter(& &1)
+    |> Enum.join("\n")
+  end
+
+  defp check_reject(%{"object" => %{} = object} = message) do
+    payload = object_payload(object)
+
     if Enum.any?(Pleroma.Config.get([:mrf_keyword, :reject]), fn pattern ->
-         string_matches?(content, pattern) or string_matches?(summary, pattern)
+         string_matches?(payload, pattern)
        end) do
       {:reject, "[KeywordPolicy] Matches with rejected keyword"}
     else
@@ -30,12 +38,12 @@ defmodule Pleroma.Web.ActivityPub.MRF.KeywordPolicy do
     end
   end
 
-  defp check_ftl_removal(
+  defp check_ftl_removal(%{"to" => to, "object" => %{} = object} = message) do
-         %{"to" => to, "object" => %{"content" => content, "summary" => summary}} = message
+    payload = object_payload(object)
-       ) do
+
     if Pleroma.Constants.as_public() in to and
          Enum.any?(Pleroma.Config.get([:mrf_keyword, :federated_timeline_removal]), fn pattern ->
-           string_matches?(content, pattern) or string_matches?(summary, pattern)
+           string_matches?(payload, pattern)
          end) do
       to = List.delete(to, Pleroma.Constants.as_public())
       cc = [Pleroma.Constants.as_public() | message["cc"] || []]
@@ -51,35 +59,24 @@ defmodule Pleroma.Web.ActivityPub.MRF.KeywordPolicy do
     end
   end
 
-  defp check_replace(%{"object" => %{"content" => content, "summary" => summary}} = message) do
+  defp check_replace(%{"object" => %{} = object} = message) do
-    content =
+    object =
-      if is_binary(content) do
+      ["content", "name", "summary"]
-        content
+      |> Enum.filter(fn field -> Map.has_key?(object, field) && object[field] end)
-      else
+      |> Enum.reduce(object, fn field, object ->
-        ""
+        data =
-      end
+          Enum.reduce(
+            Pleroma.Config.get([:mrf_keyword, :replace]),
+            object[field],
+            fn {pat, repl}, acc -> String.replace(acc, pat, repl) end
+          )
 
-    summary =
+        Map.put(object, field, data)
-      if is_binary(summary) do
+      end)
-        summary
-      else
-        ""
-      end
 
-    {content, summary} =
+    message = Map.put(message, "object", object)
-      Enum.reduce(
-        Pleroma.Config.get([:mrf_keyword, :replace]),
-        {content, summary},
-        fn {pattern, replacement}, {content_acc, summary_acc} ->
-          {String.replace(content_acc, pattern, replacement),
-           String.replace(summary_acc, pattern, replacement)}
-        end
-      )
 
-    {:ok,
+    {:ok, message}
-     message
-     |> put_in(["object", "content"], content)
-     |> put_in(["object", "summary"], summary)}
   end
 
   @impl true

lib/pleroma/web/activity_pub/mrf/subchain_policy.ex

@@ -28,8 +28,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.SubchainPolicy do
         }"
       )
 
-      subchain
+      MRF.filter(subchain, message)
-      |> MRF.filter(message)
     else
       _e -> {:ok, message}
     end

lib/pleroma/web/activity_pub/pipeline.ex

@@ -26,13 +26,17 @@ defmodule Pleroma.Web.ActivityPub.Pipeline do
 
       {:error, e} ->
         {:error, e}
+
+      {:reject, e} ->
+        {:reject, e}
     end
   end
 
   def do_common_pipeline(object, meta) do
     with {_, {:ok, validated_object, meta}} <-
            {:validate_object, ObjectValidator.validate(object, meta)},
-         {_, {:ok, mrfd_object}} <- {:mrf_object, MRF.filter(validated_object)},
+         {_, {:ok, mrfd_object, meta}} <-
+           {:mrf_object, MRF.pipeline_filter(validated_object, meta)},
          {_, {:ok, activity, meta}} <-
            {:persist_object, ActivityPub.persist(mrfd_object, meta)},
          {_, {:ok, activity, meta}} <-
@@ -40,7 +44,7 @@ defmodule Pleroma.Web.ActivityPub.Pipeline do
          {_, {:ok, _}} <- {:federation, maybe_federate(activity, meta)} do
       {:ok, activity, meta}
     else
-      {:mrf_object, {:reject, _}} -> {:ok, nil, meta}
+      {:mrf_object, {:reject, message, _}} -> {:reject, message}
       e -> {:error, e}
     end
   end

lib/pleroma/web/admin_api/controllers/admin_api_controller.ex

@@ -23,8 +23,6 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
   alias Pleroma.Web.Endpoint
   alias Pleroma.Web.Router
 
-  require Logger
-
   @users_page_size 50
 
   plug(
@@ -68,6 +66,12 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
     when action in [:list_user_statuses, :list_instance_statuses]
   )
 
+  plug(
+    OAuthScopesPlug,
+    %{scopes: ["read:chats"], admin: true}
+    when action in [:list_user_chats]
+  )
+
   plug(
     OAuthScopesPlug,
     %{scopes: ["read"], admin: true}
@@ -256,6 +260,20 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
     end
   end
 
+  def list_user_chats(%{assigns: %{user: admin}} = conn, %{"nickname" => nickname} = _params) do
+    with %User{id: user_id} <- User.get_cached_by_nickname_or_id(nickname, for: admin) do
+      chats =
+        Pleroma.Chat.for_user_query(user_id)
+        |> Pleroma.Repo.all()
+
+      conn
+      |> put_view(AdminAPI.ChatView)
+      |> render("index.json", chats: chats)
+    else
+      _ -> {:error, :not_found}
+    end
+  end
+
   def user_toggle_activation(%{assigns: %{user: admin}} = conn, %{"nickname" => nickname}) do
     user = User.get_cached_by_nickname(nickname)
 

lib/pleroma/web/admin_api/controllers/chat_controller.ex

@@ -0,0 +1,85 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.AdminAPI.ChatController do
+  use Pleroma.Web, :controller
+
+  alias Pleroma.Activity
+  alias Pleroma.Chat
+  alias Pleroma.Chat.MessageReference
+  alias Pleroma.ModerationLog
+  alias Pleroma.Pagination
+  alias Pleroma.Plugs.OAuthScopesPlug
+  alias Pleroma.Web.AdminAPI
+  alias Pleroma.Web.CommonAPI
+  alias Pleroma.Web.PleromaAPI.Chat.MessageReferenceView
+
+  require Logger
+
+  plug(Pleroma.Web.ApiSpec.CastAndValidate)
+
+  plug(
+    OAuthScopesPlug,
+    %{scopes: ["read:chats"], admin: true} when action in [:show, :messages]
+  )
+
+  plug(
+    OAuthScopesPlug,
+    %{scopes: ["write:chats"], admin: true} when action in [:delete_message]
+  )
+
+  action_fallback(Pleroma.Web.AdminAPI.FallbackController)
+
+  defdelegate open_api_operation(action), to: Pleroma.Web.ApiSpec.Admin.ChatOperation
+
+  def delete_message(%{assigns: %{user: user}} = conn, %{
+        message_id: message_id,
+        id: chat_id
+      }) do
+    with %MessageReference{object: %{data: %{"id" => object_ap_id}}} = cm_ref <-
+           MessageReference.get_by_id(message_id),
+         ^chat_id <- to_string(cm_ref.chat_id),
+         %Activity{id: activity_id} <- Activity.get_create_by_object_ap_id(object_ap_id),
+         {:ok, _} <- CommonAPI.delete(activity_id, user) do
+      ModerationLog.insert_log(%{
+        action: "chat_message_delete",
+        actor: user,
+        subject_id: message_id
+      })
+
+      conn
+      |> put_view(MessageReferenceView)
+      |> render("show.json", chat_message_reference: cm_ref)
+    else
+      _e ->
+        {:error, :could_not_delete}
+    end
+  end
+
+  def messages(conn, %{id: id} = params) do
+    with %Chat{} = chat <- Chat.get_by_id(id) do
+      cm_refs =
+        chat
+        |> MessageReference.for_chat_query()
+        |> Pagination.fetch_paginated(params)
+
+      conn
+      |> put_view(MessageReferenceView)
+      |> render("index.json", chat_message_references: cm_refs)
+    else
+      _ ->
+        conn
+        |> put_status(:not_found)
+        |> json(%{error: "not found"})
+    end
+  end
+
+  def show(conn, %{id: id}) do
+    with %Chat{} = chat <- Chat.get_by_id(id) do
+      conn
+      |> put_view(AdminAPI.ChatView)
+      |> render("show.json", chat: chat)
+    end
+  end
+end

lib/pleroma/web/admin_api/controllers/instance_document_controller.ex

@@ -0,0 +1,41 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.AdminAPI.InstanceDocumentController do
+  use Pleroma.Web, :controller
+
+  alias Pleroma.Plugs.InstanceStatic
+  alias Pleroma.Plugs.OAuthScopesPlug
+  alias Pleroma.Web.InstanceDocument
+
+  plug(Pleroma.Web.ApiSpec.CastAndValidate)
+
+  action_fallback(Pleroma.Web.AdminAPI.FallbackController)
+
+  defdelegate open_api_operation(action), to: Pleroma.Web.ApiSpec.Admin.InstanceDocumentOperation
+
+  plug(OAuthScopesPlug, %{scopes: ["read"], admin: true} when action == :show)
+  plug(OAuthScopesPlug, %{scopes: ["write"], admin: true} when action in [:update, :delete])
+
+  def show(conn, %{name: document_name}) do
+    with {:ok, url} <- InstanceDocument.get(document_name),
+         {:ok, content} <- File.read(InstanceStatic.file_path(url)) do
+      conn
+      |> put_resp_content_type("text/html")
+      |> send_resp(200, content)
+    end
+  end
+
+  def update(%{body_params: %{file: file}} = conn, %{name: document_name}) do
+    with {:ok, url} <- InstanceDocument.put(document_name, file.path) do
+      json(conn, %{"url" => url})
+    end
+  end
+
+  def delete(conn, %{name: document_name}) do
+    with :ok <- InstanceDocument.delete(document_name) do
+      json(conn, %{})
+    end
+  end
+end

lib/pleroma/web/admin_api/views/chat_view.ex

@@ -0,0 +1,30 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.AdminAPI.ChatView do
+  use Pleroma.Web, :view
+
+  alias Pleroma.Chat
+  alias Pleroma.User
+  alias Pleroma.Web.MastodonAPI
+  alias Pleroma.Web.PleromaAPI
+
+  def render("index.json", %{chats: chats} = opts) do
+    render_many(chats, __MODULE__, "show.json", Map.delete(opts, :chats))
+  end
+
+  def render("show.json", %{chat: %Chat{user_id: user_id}} = opts) do
+    user = User.get_by_id(user_id)
+    sender = MastodonAPI.AccountView.render("show.json", user: user, skip_visibility_check: true)
+
+    serialized_chat = PleromaAPI.ChatView.render("show.json", opts)
+
+    serialized_chat
+    |> Map.put(:sender, sender)
+    |> Map.put(:receiver, serialized_chat[:account])
+    |> Map.delete(:account)
+  end
+
+  def render(view, opts), do: PleromaAPI.ChatView.render(view, opts)
+end

lib/pleroma/web/admin_api/views/status_view.ex

@@ -8,6 +8,7 @@ defmodule Pleroma.Web.AdminAPI.StatusView do
   require Pleroma.Constants
 
   alias Pleroma.Web.AdminAPI
+  alias Pleroma.Web.CommonAPI
   alias Pleroma.Web.MastodonAPI
 
   defdelegate merge_account_views(user), to: AdminAPI.AccountView
@@ -17,7 +18,7 @@ defmodule Pleroma.Web.AdminAPI.StatusView do
   end
 
   def render("show.json", %{activity: %{data: %{"object" => _object}} = activity} = opts) do
-    user = MastodonAPI.StatusView.get_user(activity.data["actor"])
+    user = CommonAPI.get_user(activity.data["actor"])
 
     MastodonAPI.StatusView.render("show.json", opts)
     |> Map.merge(%{account: merge_account_views(user)})

lib/pleroma/web/api_spec/operations/admin/chat_operation.ex

@@ -0,0 +1,96 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ApiSpec.Admin.ChatOperation do
+  alias OpenApiSpex.Operation
+  alias Pleroma.Web.ApiSpec.Schemas.Chat
+  alias Pleroma.Web.ApiSpec.Schemas.ChatMessage
+
+  import Pleroma.Web.ApiSpec.Helpers
+
+  def open_api_operation(action) do
+    operation = String.to_existing_atom("#{action}_operation")
+    apply(__MODULE__, operation, [])
+  end
+
+  def delete_message_operation do
+    %Operation{
+      tags: ["admin", "chat"],
+      summary: "Delete an individual chat message",
+      operationId: "AdminAPI.ChatController.delete_message",
+      parameters: [
+        Operation.parameter(:id, :path, :string, "The ID of the Chat"),
+        Operation.parameter(:message_id, :path, :string, "The ID of the message")
+      ],
+      responses: %{
+        200 =>
+          Operation.response(
+            "The deleted ChatMessage",
+            "application/json",
+            ChatMessage
+          )
+      },
+      security: [
+        %{
+          "oAuth" => ["write:chats"]
+        }
+      ]
+    }
+  end
+
+  def messages_operation do
+    %Operation{
+      tags: ["admin", "chat"],
+      summary: "Get the most recent messages of the chat",
+      operationId: "AdminAPI.ChatController.messages",
+      parameters:
+        [Operation.parameter(:id, :path, :string, "The ID of the Chat")] ++
+          pagination_params(),
+      responses: %{
+        200 =>
+          Operation.response(
+            "The messages in the chat",
+            "application/json",
+            Pleroma.Web.ApiSpec.ChatOperation.chat_messages_response()
+          )
+      },
+      security: [
+        %{
+          "oAuth" => ["read:chats"]
+        }
+      ]
+    }
+  end
+
+  def show_operation do
+    %Operation{
+      tags: ["chat"],
+      summary: "Create a chat",
+      operationId: "AdminAPI.ChatController.show",
+      parameters: [
+        Operation.parameter(
+          :id,
+          :path,
+          :string,
+          "The id of the chat",
+          required: true,
+          example: "1234"
+        )
+      ],
+      responses: %{
+        200 =>
+          Operation.response(
+            "The existing chat",
+            "application/json",
+            Chat
+          )
+      },
+      security: [
+        %{
+          "oAuth" => ["read"]
+        }
+      ]
+    }
+  end
+end

lib/pleroma/web/api_spec/operations/admin/instance_document_operation.ex

@@ -0,0 +1,115 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ApiSpec.Admin.InstanceDocumentOperation do
+  alias OpenApiSpex.Operation
+  alias OpenApiSpex.Schema
+  alias Pleroma.Web.ApiSpec.Helpers
+  alias Pleroma.Web.ApiSpec.Schemas.ApiError
+
+  def open_api_operation(action) do
+    operation = String.to_existing_atom("#{action}_operation")
+    apply(__MODULE__, operation, [])
+  end
+
+  def show_operation do
+    %Operation{
+      tags: ["Admin", "InstanceDocument"],
+      summary: "Get the instance document",
+      operationId: "AdminAPI.InstanceDocumentController.show",
+      security: [%{"oAuth" => ["read"]}],
+      parameters: [
+        Operation.parameter(:name, :path, %Schema{type: :string}, "The document name",
+          required: true
+        )
+        | Helpers.admin_api_params()
+      ],
+      responses: %{
+        200 => document_content(),
+        400 => Operation.response("Bad Request", "application/json", ApiError),
+        403 => Operation.response("Forbidden", "application/json", ApiError),
+        404 => Operation.response("Not Found", "application/json", ApiError)
+      }
+    }
+  end
+
+  def update_operation do
+    %Operation{
+      tags: ["Admin", "InstanceDocument"],
+      summary: "Update the instance document",
+      operationId: "AdminAPI.InstanceDocumentController.update",
+      security: [%{"oAuth" => ["write"]}],
+      requestBody: Helpers.request_body("Parameters", update_request()),
+      parameters: [
+        Operation.parameter(:name, :path, %Schema{type: :string}, "The document name",
+          required: true
+        )
+        | Helpers.admin_api_params()
+      ],
+      responses: %{
+        200 => Operation.response("InstanceDocument", "application/json", instance_document()),
+        400 => Operation.response("Bad Request", "application/json", ApiError),
+        403 => Operation.response("Forbidden", "application/json", ApiError),
+        404 => Operation.response("Not Found", "application/json", ApiError)
+      }
+    }
+  end
+
+  defp update_request do
+    %Schema{
+      title: "UpdateRequest",
+      description: "POST body for uploading the file",
+      type: :object,
+      required: [:file],
+      properties: %{
+        file: %Schema{
+          type: :string,
+          format: :binary,
+          description: "The file to be uploaded, using multipart form data."
+        }
+      }
+    }
+  end
+
+  def delete_operation do
+    %Operation{
+      tags: ["Admin", "InstanceDocument"],
+      summary: "Get the instance document",
+      operationId: "AdminAPI.InstanceDocumentController.delete",
+      security: [%{"oAuth" => ["write"]}],
+      parameters: [
+        Operation.parameter(:name, :path, %Schema{type: :string}, "The document name",
+          required: true
+        )
+        | Helpers.admin_api_params()
+      ],
+      responses: %{
+        200 => Operation.response("InstanceDocument", "application/json", instance_document()),
+        400 => Operation.response("Bad Request", "application/json", ApiError),
+        403 => Operation.response("Forbidden", "application/json", ApiError),
+        404 => Operation.response("Not Found", "application/json", ApiError)
+      }
+    }
+  end
+
+  defp instance_document do
+    %Schema{
+      title: "InstanceDocument",
+      type: :object,
+      properties: %{
+        url: %Schema{type: :string}
+      },
+      example: %{
+        "url" => "https://example.com/static/terms-of-service.html"
+      }
+    }
+  end
+
+  defp document_content do
+    Operation.response("InstanceDocumentContent", "text/html", %Schema{
+      type: :string,
+      example: "<h1>Instance panel</h1>"
+    })
+  end
+end

lib/pleroma/web/api_spec/operations/chat_operation.ex

@@ -184,7 +184,8 @@ defmodule Pleroma.Web.ApiSpec.ChatOperation do
             "application/json",
             ChatMessage
           ),
-        400 => Operation.response("Bad Request", "application/json", ApiError)
+        400 => Operation.response("Bad Request", "application/json", ApiError),
+        422 => Operation.response("MRF Rejection", "application/json", ApiError)
       },
       security: [
         %{

lib/pleroma/web/api_spec/operations/status_operation.ex

@@ -55,7 +55,7 @@ defmodule Pleroma.Web.ApiSpec.StatusOperation do
             "application/json",
             %Schema{oneOf: [Status, ScheduledStatus]}
           ),
-        422 => Operation.response("Bad Request", "application/json", ApiError)
+        422 => Operation.response("Bad Request / MRF Rejection", "application/json", ApiError)
       }
     }
   end

lib/pleroma/web/common_api/common_api.ex

@@ -48,6 +48,9 @@ defmodule Pleroma.Web.CommonAPI do
               local: true
             )} do
       {:ok, activity}
+    else
+      {:common_pipeline, {:reject, _} = e} -> e
+      e -> e
     end
   end
 
@@ -550,4 +553,21 @@ defmodule Pleroma.Web.CommonAPI do
   def show_reblogs(%User{} = user, %User{} = target) do
     UserRelationship.delete_reblog_mute(user, target)
   end
+
+  def get_user(ap_id, fake_record_fallback \\ true) do
+    cond do
+      user = User.get_cached_by_ap_id(ap_id) ->
+        user
+
+      user = User.get_by_guessed_nickname(ap_id) ->
+        user
+
+      fake_record_fallback ->
+        # TODO: refactor (fake records is never a good idea)
+        User.error_user(ap_id)
+
+      true ->
+        nil
+    end
+  end
 end

lib/pleroma/web/instance_document.ex

@@ -0,0 +1,62 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.InstanceDocument do
+  alias Pleroma.Config
+  alias Pleroma.Web.Endpoint
+
+  @instance_documents %{
+    "terms-of-service" => "/static/terms-of-service.html",
+    "instance-panel" => "/instance/panel.html"
+  }
+
+  @spec get(String.t()) :: {:ok, String.t()} | {:error, atom()}
+  def get(document_name) do
+    case Map.fetch(@instance_documents, document_name) do
+      {:ok, path} -> {:ok, path}
+      _ -> {:error, :not_found}
+    end
+  end
+
+  @spec put(String.t(), String.t()) :: {:ok, String.t()} | {:error, atom()}
+  def put(document_name, origin_path) do
+    with {_, {:ok, destination_path}} <-
+           {:instance_document, Map.fetch(@instance_documents, document_name)},
+         :ok <- put_file(origin_path, destination_path) do
+      {:ok, Path.join(Endpoint.url(), destination_path)}
+    else
+      {:instance_document, :error} -> {:error, :not_found}
+      error -> error
+    end
+  end
+
+  @spec delete(String.t()) :: :ok | {:error, atom()}
+  def delete(document_name) do
+    with {_, {:ok, path}} <- {:instance_document, Map.fetch(@instance_documents, document_name)},
+         instance_static_dir_path <- instance_static_dir(path),
+         :ok <- File.rm(instance_static_dir_path) do
+      :ok
+    else
+      {:instance_document, :error} -> {:error, :not_found}
+      {:error, :enoent} -> {:error, :not_found}
+      error -> error
+    end
+  end
+
+  defp put_file(origin_path, destination_path) do
+    with destination <- instance_static_dir(destination_path),
+         {_, :ok} <- {:mkdir_p, File.mkdir_p(Path.dirname(destination))},
+         {_, {:ok, _}} <- {:copy, File.copy(origin_path, destination)} do
+      :ok
+    else
+      {error, _} -> {:error, error}
+    end
+  end
+
+  defp instance_static_dir(filename) do
+    [:instance, :static_dir]
+    |> Config.get!()
+    |> Path.join(filename)
+  end
+end

lib/pleroma/web/mastodon_api/views/status_view.ex

@@ -55,23 +55,6 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
     end)
   end
 
-  def get_user(ap_id, fake_record_fallback \\ true) do
-    cond do
-      user = User.get_cached_by_ap_id(ap_id) ->
-        user
-
-      user = User.get_by_guessed_nickname(ap_id) ->
-        user
-
-      fake_record_fallback ->
-        # TODO: refactor (fake records is never a good idea)
-        User.error_user(ap_id)
-
-      true ->
-        nil
-    end
-  end
-
   defp get_context_id(%{data: %{"context_id" => context_id}}) when not is_nil(context_id),
     do: context_id
 
@@ -119,7 +102,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
           # Note: unresolved users are filtered out
           actors =
             (activities ++ parent_activities)
-            |> Enum.map(&get_user(&1.data["actor"], false))
+            |> Enum.map(&CommonAPI.get_user(&1.data["actor"], false))
             |> Enum.filter(& &1)
 
           UserRelationship.view_relationships_option(reading_user, actors, subset: :source_mutes)
@@ -138,7 +121,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
         "show.json",
         %{activity: %{data: %{"type" => "Announce", "object" => _object}} = activity} = opts
       ) do
-    user = get_user(activity.data["actor"])
+    user = CommonAPI.get_user(activity.data["actor"])
     created_at = Utils.to_masto_date(activity.data["published"])
     activity_object = Object.normalize(activity)
 
@@ -211,7 +194,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
   def render("show.json", %{activity: %{data: %{"object" => _object}} = activity} = opts) do
     object = Object.normalize(activity)
 
-    user = get_user(activity.data["actor"])
+    user = CommonAPI.get_user(activity.data["actor"])
     user_follower_address = user.follower_address
 
     like_count = object.data["like_count"] || 0
@@ -265,7 +248,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
 
     reply_to = get_reply_to(activity, opts)
 
-    reply_to_user = reply_to && get_user(reply_to.data["actor"])
+    reply_to_user = reply_to && CommonAPI.get_user(reply_to.data["actor"])
 
     content =
       object

lib/pleroma/web/metadata/restrict_indexing.ex

@@ -10,7 +10,9 @@ defmodule Pleroma.Web.Metadata.Providers.RestrictIndexing do
   """
 
   @impl true
-  def build_tags(%{user: %{local: false}}) do
+  def build_tags(%{user: %{local: true, discoverable: true}}), do: []
+
+  def build_tags(_) do
     [
       {:meta,
        [
@@ -19,7 +21,4 @@ defmodule Pleroma.Web.Metadata.Providers.RestrictIndexing do
        ], []}
     ]
   end
-
-  @impl true
-  def build_tags(%{user: %{local: true}}), do: []
 end

lib/pleroma/web/pleroma_api/controllers/chat_controller.ex

@@ -90,6 +90,16 @@ defmodule Pleroma.Web.PleromaAPI.ChatController do
       conn
       |> put_view(MessageReferenceView)
       |> render("show.json", chat_message_reference: cm_ref)
+    else
+      {:reject, message} ->
+        conn
+        |> put_status(:unprocessable_entity)
+        |> json(%{error: message})
+
+      {:error, message} ->
+        conn
+        |> put_status(:bad_request)
+        |> json(%{error: message})
     end
   end
 
@@ -146,11 +156,8 @@ defmodule Pleroma.Web.PleromaAPI.ChatController do
     blocked_ap_ids = User.blocked_users_ap_ids(user)
 
     chats =
-      from(c in Chat,
+      Chat.for_user_query(user_id)
-        where: c.user_id == ^user_id,
+      |> where([c], c.recipient not in ^blocked_ap_ids)
-        where: c.recipient not in ^blocked_ap_ids,
-        order_by: [desc: c.updated_at]
-      )
       |> Repo.all()
 
     conn

lib/pleroma/web/pleroma_api/views/scrobble_view.ex

@@ -10,14 +10,14 @@ defmodule Pleroma.Web.PleromaAPI.ScrobbleView do
   alias Pleroma.Activity
   alias Pleroma.HTML
   alias Pleroma.Object
+  alias Pleroma.Web.CommonAPI
   alias Pleroma.Web.CommonAPI.Utils
   alias Pleroma.Web.MastodonAPI.AccountView
-  alias Pleroma.Web.MastodonAPI.StatusView
 
   def render("show.json", %{activity: %Activity{data: %{"type" => "Listen"}} = activity} = opts) do
     object = Object.normalize(activity)
 
-    user = StatusView.get_user(activity.data["actor"])
+    user = CommonAPI.get_user(activity.data["actor"])
     created_at = Utils.to_masto_date(activity.data["published"])
 
     %{

lib/pleroma/web/rich_media/parser.ex

@@ -20,36 +20,61 @@ defmodule Pleroma.Web.RichMedia.Parser do
       with {:ok, data} <- get_cached_or_parse(url),
            {:ok, _} <- set_ttl_based_on_image(data, url) do
         {:ok, data}
-      else
-        {:error, {:invalid_metadata, data}} = e ->
-          Logger.debug(fn -> "Incomplete or invalid metadata for #{url}: #{inspect(data)}" end)
-          e
-
-        error ->
-          Logger.error(fn -> "Rich media error for #{url}: #{inspect(error)}" end)
-          error
       end
     end
 
     defp get_cached_or_parse(url) do
-      case Cachex.fetch!(:rich_media_cache, url, fn _ -> {:commit, parse_url(url)} end) do
+      case Cachex.fetch(:rich_media_cache, url, fn ->
-        {:ok, _data} = res ->
+             case parse_url(url) do
-          res
+               {:ok, _} = res ->
+                 {:commit, res}
 
-        {:error, :body_too_large} = e ->
+               {:error, reason} = e ->
-          e
+                 # Unfortunately we have to log errors here, instead of doing that
+                 # along with ttl setting at the bottom. Otherwise we can get log spam
+                 # if more than one process was waiting for the rich media card
+                 # while it was generated. Ideally we would set ttl here as well,
+                 # so we don't override it number_of_waiters_on_generation
+                 # times, but one, obviously, can't set ttl for not-yet-created entry
+                 # and Cachex doesn't support returning ttl from the fetch callback.
+                 log_error(url, reason)
+                 {:commit, e}
+             end
+           end) do
+        {action, res} when action in [:commit, :ok] ->
+          case res do
+            {:ok, _data} = res ->
+              res
 
-        {:error, {:content_type, _}} = e ->
+            {:error, reason} = e ->
-          e
+              if action == :commit, do: set_error_ttl(url, reason)
+              e
+          end
 
-        # The TTL is not set for the errors above, since they are unlikely to change
+        {:error, e} ->
-        # with time
+          {:error, {:cachex_error, e}}
-        {:error, _} = e ->
-          ttl = Pleroma.Config.get([:rich_media, :failure_backoff], 60_000)
-          Cachex.expire(:rich_media_cache, url, ttl)
-          e
       end
     end
+
+    defp set_error_ttl(_url, :body_too_large), do: :ok
+    defp set_error_ttl(_url, {:content_type, _}), do: :ok
+
+    # The TTL is not set for the errors above, since they are unlikely to change
+    # with time
+
+    defp set_error_ttl(url, _reason) do
+      ttl = Pleroma.Config.get([:rich_media, :failure_backoff], 60_000)
+      Cachex.expire(:rich_media_cache, url, ttl)
+      :ok
+    end
+
+    defp log_error(url, {:invalid_metadata, data}) do
+      Logger.debug(fn -> "Incomplete or invalid metadata for #{url}: #{inspect(data)}" end)
+    end
+
+    defp log_error(url, reason) do
+      Logger.warn(fn -> "Rich media error for #{url}: #{inspect(reason)}" end)
+    end
   end
 
   @doc """

lib/pleroma/web/router.ex

@@ -178,9 +178,14 @@ defmodule Pleroma.Web.Router do
     get("/users", AdminAPIController, :list_users)
     get("/users/:nickname", AdminAPIController, :user_show)
     get("/users/:nickname/statuses", AdminAPIController, :list_user_statuses)
+    get("/users/:nickname/chats", AdminAPIController, :list_user_chats)
 
     get("/instances/:instance/statuses", AdminAPIController, :list_instance_statuses)
 
+    get("/instance_document/:name", InstanceDocumentController, :show)
+    patch("/instance_document/:name", InstanceDocumentController, :update)
+    delete("/instance_document/:name", InstanceDocumentController, :delete)
+
     patch("/users/confirm_email", AdminAPIController, :confirm_email)
     patch("/users/resend_confirmation_email", AdminAPIController, :resend_confirmation_email)
 
@@ -214,6 +219,10 @@ defmodule Pleroma.Web.Router do
     get("/media_proxy_caches", MediaProxyCacheController, :index)
     post("/media_proxy_caches/delete", MediaProxyCacheController, :delete)
     post("/media_proxy_caches/purge", MediaProxyCacheController, :purge)
+
+    get("/chats/:id", ChatController, :show)
+    get("/chats/:id/messages", ChatController, :messages)
+    delete("/chats/:id/messages/:message_id", ChatController, :delete_message)
   end
 
   scope "/api/pleroma/emoji", Pleroma.Web.PleromaAPI do

priv/static/index.html

@@ -1 +1 @@
-<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1,user-scalable=no"><title>Pleroma</title><!--server-generated-meta--><link rel=icon type=image/png href=/favicon.png><link href=/static/css/app.77b1644622e3bae24b6b.css rel=stylesheet><link href=/static/fontello.1599568314856.css rel=stylesheet></head><body class=hidden><noscript>To use Pleroma, please enable JavaScript.</noscript><div id=app></div><script type=text/javascript src=/static/js/vendors~app.90c4af83c1ae68f4cd95.js></script><script type=text/javascript src=/static/js/app.55d173dc5e39519aa518.js></script></body></html>
+<!DOCTYPE html><html lang=en><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1,user-scalable=no"><title>Pleroma</title><!--server-generated-meta--><link rel=icon type=image/png href=/favicon.png><link href=/static/css/app.77b1644622e3bae24b6b.css rel=stylesheet><link href=/static/fontello.1600365488745.css rel=stylesheet></head><body class=hidden><noscript>To use Pleroma, please enable JavaScript.</noscript><div id=app></div><script type=text/javascript src=/static/js/vendors~app.90c4af83c1ae68f4cd95.js></script><script type=text/javascript src=/static/js/app.826c44232e0a76bbd9ba.js></script></body></html>

priv/static/static/font/fontello.1600365488745.svg

@@ -92,6 +92,8 @@
 
 <glyph glyph-name="block" unicode="&#xe82a;" d="M732 359q0 90-48 164l-421-420q76-50 166-50 62 0 118 25t96 65 65 97 24 119z m-557-167l421 421q-75 50-167 50-83 0-153-40t-110-111-41-153q0-91 50-167z m682 167q0-88-34-168t-91-137-137-92-166-34-167 34-137 92-91 137-34 168 34 167 91 137 137 91 167 34 166-34 137-91 91-137 34-167z" horiz-adv-x="857.1" />
 
+<glyph glyph-name="megaphone" unicode="&#xe82b;" d="M929 500q29 0 50-21t21-51-21-50-50-21v-214q0-29-22-50t-50-22q-233 194-453 212-32-10-51-36t-17-57 22-51q-11-19-13-37t4-32 19-31 26-28 35-28q-17-32-63-46t-94-7-73 31q-4 13-17 49t-18 53-12 50-9 56 2 55 12 62h-68q-36 0-63 26t-26 63v107q0 37 26 63t63 26h268q243 0 500 215 29 0 50-22t22-50v-214z m-72-337v532q-220-168-428-191v-151q210-23 428-190z" horiz-adv-x="1000" />
+
 <glyph glyph-name="spin3" unicode="&#xe832;" d="M494 857c-266 0-483-210-494-472-1-19 13-20 13-20l84 0c16 0 19 10 19 18 10 199 176 358 378 358 107 0 205-45 273-118l-58-57c-11-12-11-27 5-31l247-50c21-5 46 11 37 44l-58 227c-2 9-16 22-29 13l-65-60c-89 91-214 148-352 148z m409-508c-16 0-19-10-19-18-10-199-176-358-377-358-108 0-205 45-274 118l59 57c10 12 10 27-5 31l-248 50c-21 5-46-11-37-44l58-227c2-9 16-22 30-13l64 60c89-91 214-148 353-148 265 0 482 210 493 473 1 18-13 19-13 19l-84 0z" horiz-adv-x="1000" />
 
 <glyph glyph-name="spin4" unicode="&#xe834;" d="M498 857c-114 0-228-39-320-116l0 0c173 140 428 130 588-31 134-134 164-332 89-495-10-29-5-50 12-68 21-20 61-23 84 0 3 3 12 15 15 24 71 180 33 393-112 539-99 98-228 147-356 147z m-409-274c-14 0-29-5-39-16-3-3-13-15-15-24-71-180-34-393 112-539 185-185 479-195 676-31l0 0c-173-140-428-130-589 31-134 134-163 333-89 495 11 29 6 50-12 68-11 11-27 17-44 16z" horiz-adv-x="1001" />

priv/static/static/fontello.1600365488745.css

@@ -1,11 +1,11 @@
 @font-face {
 	font-family: "Icons";
-	src: url("./font/fontello.1599568314856.eot");
+	src: url("./font/fontello.1600365488745.eot");
-	src: url("./font/fontello.1599568314856.eot") format("embedded-opentype"),
+	src: url("./font/fontello.1600365488745.eot") format("embedded-opentype"),
-		url("./font/fontello.1599568314856.woff2") format("woff2"),
+		url("./font/fontello.1600365488745.woff2") format("woff2"),
-		url("./font/fontello.1599568314856.woff") format("woff"),
+		url("./font/fontello.1600365488745.woff") format("woff"),
-		url("./font/fontello.1599568314856.ttf") format("truetype"),
+		url("./font/fontello.1600365488745.ttf") format("truetype"),
-		url("./font/fontello.1599568314856.svg") format("svg");
+		url("./font/fontello.1600365488745.svg") format("svg");
 	font-weight: normal;
 	font-style: normal;
 }
@@ -156,3 +156,5 @@
 .icon-doc::before { content: "\e829"; }
 
 .icon-block::before { content: "\e82a"; }
+
+.icon-megaphone::before { content: "\e82b"; }

priv/static/static/fontello.json

@@ -405,6 +405,12 @@
       "css": "block",
       "code": 59434,
       "src": "fontawesome"
+    },
+    {
+      "uid": "3e674995cacc2b09692c096ea7eb6165",
+      "css": "megaphone",
+      "code": 59435,
+      "src": "fontawesome"
     }
   ]
 }

priv/static/sw-pleroma.js

@@ -1,4 +1,4 @@
-var serviceWorkerOption = {"assets":["/static/fontello.1599568314856.css","/static/font/fontello.1599568314856.eot","/static/font/fontello.1599568314856.svg","/static/font/fontello.1599568314856.ttf","/static/font/fontello.1599568314856.woff","/static/font/fontello.1599568314856.woff2","/static/img/nsfw.74818f9.png","/static/css/app.77b1644622e3bae24b6b.css","/static/js/app.55d173dc5e39519aa518.js","/static/js/vendors~app.90c4af83c1ae68f4cd95.js","/static/css/2.0778a6a864a1307a6c41.css","/static/js/2.c92f4803ff24726cea58.js","/static/css/3.b2603a50868c68a1c192.css","/static/js/3.7d21accf4e5bd07e3ebf.js","/static/js/4.5719922a4e807145346d.js","/static/js/5.cf05c5ddbdbac890ae35.js","/static/js/6.ecfd3302a692de148391.js","/static/js/7.dd44c3d58fb9dced093d.js","/static/js/8.636322a87bb10a1754f8.js","/static/js/9.6010dbcce7b4d7c05a18.js","/static/js/10.46fbbdfaf0d4800f349b.js","/static/js/11.708cc2513c53879a92cc.js","/static/js/12.b3bf0bc313861d6ec36b.js","/static/js/13.adb8a942514d735722c4.js","/static/js/14.d015d9b2ea16407e389c.js","/static/js/15.19866e6a366ccf982284.js","/static/js/16.38a984effd54736f6a2c.js","/static/js/17.9c25507194320db2e85b.js","/static/js/18.94946caca48930c224c7.js","/static/js/19.233c81ac2c28d55e9f13.js","/static/js/20.818c38d27369c3a4d677.js","/static/js/21.ce4cda179d888ca6bc2a.js","/static/js/22.2ea93c6cc569ef0256ab.js","/static/js/23.a57a7845cc20fafd06d1.js","/static/js/24.35eb55a657b5485f8491.js","/static/js/25.5a9efe20e3ae1352e6d2.js","/static/js/26.cf13231d524e5ca3b3e6.js","/static/js/27.fca8d4f6e444bd14f376.js","/static/js/28.e0f9f164e0bfd890dc61.js","/static/js/29.0b69359f0fe5c0785746.js","/static/js/30.fce58be0b52ca3e32fa4.js"]};
+var serviceWorkerOption = {"assets":["/static/fontello.1600365488745.css","/static/font/fontello.1600365488745.eot","/static/font/fontello.1600365488745.svg","/static/font/fontello.1600365488745.ttf","/static/font/fontello.1600365488745.woff","/static/font/fontello.1600365488745.woff2","/static/img/nsfw.74818f9.png","/static/css/app.77b1644622e3bae24b6b.css","/static/js/app.826c44232e0a76bbd9ba.js","/static/js/vendors~app.90c4af83c1ae68f4cd95.js","/static/css/2.0778a6a864a1307a6c41.css","/static/js/2.e852a6b4b3bba752b838.js","/static/css/3.b2603a50868c68a1c192.css","/static/js/3.7d21accf4e5bd07e3ebf.js","/static/js/4.5719922a4e807145346d.js","/static/js/5.cf05c5ddbdbac890ae35.js","/static/js/6.ecfd3302a692de148391.js","/static/js/7.dd44c3d58fb9dced093d.js","/static/js/8.636322a87bb10a1754f8.js","/static/js/9.6010dbcce7b4d7c05a18.js","/static/js/10.46fbbdfaf0d4800f349b.js","/static/js/11.708cc2513c53879a92cc.js","/static/js/12.b3bf0bc313861d6ec36b.js","/static/js/13.adb8a942514d735722c4.js","/static/js/14.d015d9b2ea16407e389c.js","/static/js/15.19866e6a366ccf982284.js","/static/js/16.38a984effd54736f6a2c.js","/static/js/17.9c25507194320db2e85b.js","/static/js/18.94946caca48930c224c7.js","/static/js/19.233c81ac2c28d55e9f13.js","/static/js/20.818c38d27369c3a4d677.js","/static/js/21.ce4cda179d888ca6bc2a.js","/static/js/22.2ea93c6cc569ef0256ab.js","/static/js/23.a57a7845cc20fafd06d1.js","/static/js/24.35eb55a657b5485f8491.js","/static/js/25.5a9efe20e3ae1352e6d2.js","/static/js/26.cf13231d524e5ca3b3e6.js","/static/js/27.fca8d4f6e444bd14f376.js","/static/js/28.e0f9f164e0bfd890dc61.js","/static/js/29.0b69359f0fe5c0785746.js","/static/js/30.fce58be0b52ca3e32fa4.js"]};
         
         !function(t){var e={};function n(r){if(e[r])return e[r].exports;var o=e[r]={i:r,l:!1,exports:{}};return t[r].call(o.exports,o,o.exports,n),o.l=!0,o.exports}n.m=t,n.c=e,n.d=function(t,e,r){n.o(t,e)||Object.defineProperty(t,e,{enumerable:!0,get:r})},n.r=function(t){"undefined"!=typeof Symbol&&Symbol.toStringTag&&Object.defineProperty(t,Symbol.toStringTag,{value:"Module"}),Object.defineProperty(t,"__esModule",{value:!0})},n.t=function(t,e){if(1&e&&(t=n(t)),8&e)return t;if(4&e&&"object"==typeof t&&t&&t.__esModule)return t;var r=Object.create(null);if(n.r(r),Object.defineProperty(r,"default",{enumerable:!0,value:t}),2&e&&"string"!=typeof t)for(var o in t)n.d(r,o,function(e){return t[e]}.bind(null,o));return r},n.n=function(t){var e=t&&t.__esModule?function(){return t.default}:function(){return t};return n.d(e,"a",e),e},n.o=function(t,e){return Object.prototype.hasOwnProperty.call(t,e)},n.p="/",n(n.s=196)}([function(t,e){var n=Array.isArray;t.exports=n},function(t,e,n){t.exports=n(53)},function(t,e,n){var r=n(32),o="object"==typeof self&&self&&self.Object===Object&&self,i=r||o||Function("return this")();t.exports=i},function(t,e,n){var r=n(97),o=n(100);t.exports=function(t,e){var n=o(t,e);return r(n)?n:void 0}},function(t,e,n){var r=n(8),o=n(64),i=n(65),a="[object Null]",s="[object Undefined]",c=r?r.toStringTag:void 0;t.exports=function(t){return null==t?void 0===t?s:a:c&&c in Object(t)?o(t):i(t)}},function(t,e){t.exports=function(t){return null!=t&&"object"==typeof t}},function(t,e){var n;n=function(){return this}();try{n=n||new Function("return this")()}catch(t){"object"==typeof window&&(n=window)}t.exports=n},function(t,e,n){var r=n(31),o=n(20);t.exports=function(t){return null!=t&&o(t.length)&&!r(t)}},function(t,e,n){var r=n(2).Symbol;t.exports=r},function(t,e){t.exports=function(t){var e=typeof t;return null!=t&&("object"==e||"function"==e)}},function(t,e,n){var r=n(4),o=n(5),i="[object Symbol]";t.exports=function(t){return"symbol"==typeof t||o(t)&&r(t)==i}},function(t,e,n){var r=n(72),o=n(78),i=n(7);t.exports=function(t){return i(t)?r(t):o(t)}},function(t,e,n){var r=n(87),o=n(88),i=n(89),a=n(90),s=n(91);function c(t){var e=-1,n=null==t?0:t.length;for(this.clear();++e<n;){var r=t[e];this.set(r[0],r[1])}}c.prototype.clear=r,c.prototype.delete=o,c.prototype.get=i,c.prototype.has=a,c.prototype.set=s,t.exports=c},function(t,e,n){var r=n(24);t.exports=function(t,e){for(var n=t.length;n--;)if(r(t[n][0],e))return n;return-1}},function(t,e,n){var r=n(3)(Object,"create");t.exports=r},function(t,e,n){var r=n(109);t.exports=function(t,e){var n=t.__data__;return r(e)?n["string"==typeof e?"string":"hash"]:n.map}},function(t,e,n){var r=n(10),o=1/0;t.exports=function(t){if("string"==typeof t||r(t))return t;var e=t+"";return"0"==e&&1/t==-o?"-0":e}},function(t,e){t.exports=function(t){return t}},function(t,e,n){var r=n(42),o=n(164),i=n(37),a=n(0);t.exports=function(t,e){return(a(t)?r:o)(t,i(e,3))}},function(t,e){t.exports=function(t){return t.webpackPolyfill||(t.deprecate=function(){},t.paths=[],t.children||(t.children=[]),Object.defineProperty(t,"loaded",{enumerable:!0,get:function(){return t.l}}),Object.defineProperty(t,"id",{enumerable:!0,get:function(){return t.i}}),t.webpackPolyfill=1),t}},function(t,e){var n=9007199254740991;t.exports=function(t){return"number"==typeof t&&t>-1&&t%1==0&&t<=n}},function(t,e){t.exports=function(t,e){for(var n=-1,r=null==t?0:t.length,o=Array(r);++n<r;)o[n]=e(t[n],n,t);return o}},function(t,e,n){var r=n(74),o=n(5),i=Object.prototype,a=i.hasOwnProperty,s=i.propertyIsEnumerable,c=r(function(){return arguments}())?r:function(t){return o(t)&&a.call(t,"callee")&&!s.call(t,"callee")};t.exports=c},function(t,e){var n=9007199254740991,r=/^(?:0|[1-9]\d*)$/;t.exports=function(t,e){var o=typeof t;return!!(e=null==e?n:e)&&("number"==o||"symbol"!=o&&r.test(t))&&t>-1&&t%1==0&&t<e}},function(t,e){t.exports=function(t,e){return t===e||t!=t&&e!=e}},function(t,e,n){var r=n(3)(n(2),"Map");t.exports=r},function(t,e,n){var r=n(101),o=n(108),i=n(110),a=n(111),s=n(112);function c(t){var e=-1,n=null==t?0:t.length;for(this.clear();++e<n;){var r=t[e];this.set(r[0],r[1])}}c.prototype.clear=r,c.prototype.delete=o,c.prototype.get=i,c.prototype.has=a,c.prototype.set=s,t.exports=c},function(t,e,n){var r=n(0),o=n(10),i=/\.|\[(?:[^[\]]*|(["'])(?:(?!\1)[^\\]|\\.)*?\1)\]/,a=/^\w*$/;t.exports=function(t,e){if(r(t))return!1;var n=typeof t;return!("number"!=n&&"symbol"!=n&&"boolean"!=n&&null!=t&&!o(t))||a.test(t)||!i.test(t)||null!=e&&t in Object(e)}},function(t,e){
 /*!

test/fixtures/custom_instance_panel.html

@@ -0,0 +1 @@
+<h2>Custom instance panel</h2>

test/fixtures/tesla_mock/admin@mastdon.example.org.json

@@ -1,20 +1,24 @@
 {
-  "@context": ["https://www.w3.org/ns/activitystreams", "https://w3id.org/security/v1", {
+  "@context": [
-    "manuallyApprovesFollowers": "as:manuallyApprovesFollowers",
+    "https://www.w3.org/ns/activitystreams",
-    "sensitive": "as:sensitive",
+    "https://w3id.org/security/v1",
-    "movedTo": "as:movedTo",
+    {
-    "Hashtag": "as:Hashtag",
+      "manuallyApprovesFollowers": "as:manuallyApprovesFollowers",
-    "ostatus": "http://ostatus.org#",
+      "sensitive": "as:sensitive",
-    "atomUri": "ostatus:atomUri",
+      "movedTo": "as:movedTo",
-    "inReplyToAtomUri": "ostatus:inReplyToAtomUri",
+      "Hashtag": "as:Hashtag",
-    "conversation": "ostatus:conversation",
+      "ostatus": "http://ostatus.org#",
-    "toot": "http://joinmastodon.org/ns#",
+      "atomUri": "ostatus:atomUri",
-    "Emoji": "toot:Emoji",
+      "inReplyToAtomUri": "ostatus:inReplyToAtomUri",
-    "alsoKnownAs": {
+      "conversation": "ostatus:conversation",
-      "@id": "as:alsoKnownAs",
+      "toot": "http://joinmastodon.org/ns#",
-      "@type": "@id"
+      "Emoji": "toot:Emoji",
+      "alsoKnownAs": {
+        "@id": "as:alsoKnownAs",
+        "@type": "@id"
+      }
     }
-  }],
+  ],
   "id": "http://mastodon.example.org/users/admin",
   "type": "Person",
   "following": "http://mastodon.example.org/users/admin/following",
@@ -23,6 +27,7 @@
   "outbox": "http://mastodon.example.org/users/admin/outbox",
   "preferredUsername": "admin",
   "name": null,
+  "discoverable": "true",
   "summary": "\u003cp\u003e\u003c/p\u003e",
   "url": "http://mastodon.example.org/@admin",
   "manuallyApprovesFollowers": false,
@@ -34,7 +39,8 @@
     "owner": "http://mastodon.example.org/users/admin",
     "publicKeyPem": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtc4Tir+3ADhSNF6VKrtW\nOU32T01w7V0yshmQei38YyiVwVvFu8XOP6ACchkdxbJ+C9mZud8qWaRJKVbFTMUG\nNX4+6Q+FobyuKrwN7CEwhDALZtaN2IPbaPd6uG1B7QhWorrY+yFa8f2TBM3BxnUy\nI4T+bMIZIEYG7KtljCBoQXuTQmGtuffO0UwJksidg2ffCF5Q+K//JfQagJ3UzrR+\nZXbKMJdAw4bCVJYs4Z5EhHYBwQWiXCyMGTd7BGlmMkY6Av7ZqHKC/owp3/0EWDNz\nNqF09Wcpr3y3e8nA10X40MJqp/wR+1xtxp+YGbq/Cj5hZGBG7etFOmIpVBrDOhry\nBwIDAQAB\n-----END PUBLIC KEY-----\n"
   },
-  "attachment": [{
+  "attachment": [
+    {
       "type": "PropertyValue",
       "name": "foo",
       "value": "bar"
@@ -58,5 +64,7 @@
     "mediaType": "image/png",
     "url": "https://cdn.niu.moe/accounts/headers/000/033/323/original/850b3448fa5fd477.png"
   },
-  "alsoKnownAs": ["http://example.org/users/foo"]
+  "alsoKnownAs": [
-}
+    "http://example.org/users/foo"
+  ]
+}

test/fixtures/tesla_mock/https___osada.macgirvin.com_channel_mike.json

@@ -8,6 +8,7 @@
     "preferredUsername": "mike",
     "name": "Mike Macgirvin (Osada)",
     "updated": "2018-08-29T03:09:11Z",
+    "discoverable": "true",
     "icon": {
         "type": "Image",
         "mediaType": "image/jpeg",
@@ -51,4 +52,4 @@
         "created": "2018-10-17T07:16:28Z",
         "signatureValue": "WbfFVIPImkd3yNu6brz0CvZaeV242rwAbH0vy8DM4vfnXCxLr5Uv/Wj9gwP+tbooTxGaahAKBeqlGkQp8RLEo37LATrKMRLA/0V6DeeV+C5ORWR9B4WxyWiD3s/9Wf+KesFMtktNLAcMZ5PfnOS/xNYerhnpkp/gWPxtkglmLIWJv+w18A5zZ01JCxsO4QljHbhYaEUPHUfQ97abrkLECeam+FThVwdO6BFCtbjoNXHfzjpSZL/oKyBpi5/fpnqMqOLOQPs5WgBBZJvjEYYkQcoPTyxYI5NGpNbzIjGHPQNuACnOelH16A7L+q4swLWDIaEFeXQ2/5bmqVKZDZZ6usNP4QyTVszwd8jqo27qcDTNibXDUTsTdKpNQvM/3UncBuzuzmUV3FczhtGshIU1/pRVZiQycpVqPlGLvXhP/yZCe+1siyqDd+3uMaS2vkHTObSl5r+VYof+c+TcjrZXHSWnQTg8/X3zkoBWosrQ93VZcwjzMxQoARYv6rphbOoTz7RPmGAXYUt3/PDWkqDlmQDwCpLNNkJo1EidyefZBdD9HXQpCBO0ZU0NHb0JmPvg/+zU0krxlv70bm3RHA/maBETVjroIWzt7EwQEg5pL2hVnvSBG+1wF3BtRVe77etkPOHxLnYYIcAMLlVKCcgDd89DPIziQyruvkx1busHI08="
     }
-}
+}

test/support/factory.ex

@@ -31,6 +31,7 @@ defmodule Pleroma.Factory do
       nickname: sequence(:nickname, &"nick#{&1}"),
       password_hash: Pbkdf2.hash_pwd_salt("test"),
       bio: sequence(:bio, &"Tester Number #{&1}"),
+      discoverable: true,
       last_digest_emailed_at: NaiveDateTime.utc_now(),
       last_refreshed_at: NaiveDateTime.utc_now(),
       notification_settings: %Pleroma.User.NotificationSetting{},

test/tasks/config_test.exs

@@ -40,6 +40,19 @@ defmodule Mix.Tasks.Pleroma.ConfigTest do
       on_exit(fn -> Application.put_env(:quack, :level, initial) end)
     end
 
+    @tag capture_log: true
+    test "config migration refused when deprecated settings are found" do
+      clear_config([:media_proxy, :whitelist], ["domain_without_scheme.com"])
+      assert Repo.all(ConfigDB) == []
+
+      Mix.Tasks.Pleroma.Config.migrate_to_db("test/fixtures/config/temp.secret.exs")
+
+      assert_received {:mix_shell, :error, [message]}
+
+      assert message =~
+               "Migration is not allowed until all deprecation warnings have been resolved."
+    end
+
     test "filtered settings are migrated to db" do
       assert Repo.all(ConfigDB) == []
 

test/user_search_test.exs

@@ -25,6 +25,14 @@ defmodule Pleroma.UserSearchTest do
       assert found_user.id == user.id
     end
 
+    test "excludes users when discoverable is false" do
+      insert(:user, %{nickname: "john 3000", discoverable: false})
+      insert(:user, %{nickname: "john 3001"})
+
+      users = User.search("john")
+      assert Enum.count(users) == 1
+    end
+
     test "excludes service actors from results" do
       insert(:user, actor_type: "Application", nickname: "user1")
       service = insert(:user, actor_type: "Service", nickname: "user2")

test/web/activity_pub/pipeline_test.exs

@@ -26,7 +26,7 @@ defmodule Pleroma.Web.ActivityPub.PipelineTest do
         {
           Pleroma.Web.ActivityPub.MRF,
           [],
-          [filter: fn o -> {:ok, o} end]
+          [pipeline_filter: fn o, m -> {:ok, o, m} end]
         },
         {
           Pleroma.Web.ActivityPub.ActivityPub,
@@ -51,7 +51,7 @@ defmodule Pleroma.Web.ActivityPub.PipelineTest do
                  Pleroma.Web.ActivityPub.Pipeline.common_pipeline(activity, meta)
 
         assert_called(Pleroma.Web.ActivityPub.ObjectValidator.validate(activity, meta))
-        assert_called(Pleroma.Web.ActivityPub.MRF.filter(activity))
+        assert_called(Pleroma.Web.ActivityPub.MRF.pipeline_filter(activity, meta))
         assert_called(Pleroma.Web.ActivityPub.ActivityPub.persist(activity, meta))
         assert_called(Pleroma.Web.ActivityPub.SideEffects.handle(activity, meta))
         refute called(Pleroma.Web.Federator.publish(activity))
@@ -68,7 +68,7 @@ defmodule Pleroma.Web.ActivityPub.PipelineTest do
         {
           Pleroma.Web.ActivityPub.MRF,
           [],
-          [filter: fn o -> {:ok, o} end]
+          [pipeline_filter: fn o, m -> {:ok, o, m} end]
         },
         {
           Pleroma.Web.ActivityPub.ActivityPub,
@@ -93,7 +93,7 @@ defmodule Pleroma.Web.ActivityPub.PipelineTest do
                  Pleroma.Web.ActivityPub.Pipeline.common_pipeline(activity, meta)
 
         assert_called(Pleroma.Web.ActivityPub.ObjectValidator.validate(activity, meta))
-        assert_called(Pleroma.Web.ActivityPub.MRF.filter(activity))
+        assert_called(Pleroma.Web.ActivityPub.MRF.pipeline_filter(activity, meta))
         assert_called(Pleroma.Web.ActivityPub.ActivityPub.persist(activity, meta))
         assert_called(Pleroma.Web.ActivityPub.SideEffects.handle(activity, meta))
         assert_called(Pleroma.Web.Federator.publish(activity))
@@ -109,7 +109,7 @@ defmodule Pleroma.Web.ActivityPub.PipelineTest do
         {
           Pleroma.Web.ActivityPub.MRF,
           [],
-          [filter: fn o -> {:ok, o} end]
+          [pipeline_filter: fn o, m -> {:ok, o, m} end]
         },
         {
           Pleroma.Web.ActivityPub.ActivityPub,
@@ -131,7 +131,7 @@ defmodule Pleroma.Web.ActivityPub.PipelineTest do
                  Pleroma.Web.ActivityPub.Pipeline.common_pipeline(activity, meta)
 
         assert_called(Pleroma.Web.ActivityPub.ObjectValidator.validate(activity, meta))
-        assert_called(Pleroma.Web.ActivityPub.MRF.filter(activity))
+        assert_called(Pleroma.Web.ActivityPub.MRF.pipeline_filter(activity, meta))
         assert_called(Pleroma.Web.ActivityPub.ActivityPub.persist(activity, meta))
         assert_called(Pleroma.Web.ActivityPub.SideEffects.handle(activity, meta))
       end
@@ -148,7 +148,7 @@ defmodule Pleroma.Web.ActivityPub.PipelineTest do
         {
           Pleroma.Web.ActivityPub.MRF,
           [],
-          [filter: fn o -> {:ok, o} end]
+          [pipeline_filter: fn o, m -> {:ok, o, m} end]
         },
         {
           Pleroma.Web.ActivityPub.ActivityPub,
@@ -170,7 +170,7 @@ defmodule Pleroma.Web.ActivityPub.PipelineTest do
                  Pleroma.Web.ActivityPub.Pipeline.common_pipeline(activity, meta)
 
         assert_called(Pleroma.Web.ActivityPub.ObjectValidator.validate(activity, meta))
-        assert_called(Pleroma.Web.ActivityPub.MRF.filter(activity))
+        assert_called(Pleroma.Web.ActivityPub.MRF.pipeline_filter(activity, meta))
         assert_called(Pleroma.Web.ActivityPub.ActivityPub.persist(activity, meta))
         assert_called(Pleroma.Web.ActivityPub.SideEffects.handle(activity, meta))
       end

test/web/admin_api/controllers/admin_api_controller_test.exs

@@ -1510,6 +1510,56 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIControllerTest do
     end
   end
 
+  describe "GET /api/pleroma/admin/users/:nickname/chats" do
+    setup do
+      user = insert(:user)
+      recipients = insert_list(3, :user)
+
+      Enum.each(recipients, fn recipient ->
+        CommonAPI.post_chat_message(user, recipient, "yo")
+      end)
+
+      %{user: user}
+    end
+
+    test "renders user's chats", %{conn: conn, user: user} do
+      conn = get(conn, "/api/pleroma/admin/users/#{user.nickname}/chats")
+
+      assert json_response(conn, 200) |> length() == 3
+    end
+  end
+
+  describe "GET /api/pleroma/admin/users/:nickname/chats unauthorized" do
+    setup do
+      user = insert(:user)
+      recipient = insert(:user)
+      CommonAPI.post_chat_message(user, recipient, "yo")
+      %{conn: conn} = oauth_access(["read:chats"])
+      %{conn: conn, user: user}
+    end
+
+    test "returns 403", %{conn: conn, user: user} do
+      conn
+      |> get("/api/pleroma/admin/users/#{user.nickname}/chats")
+      |> json_response(403)
+    end
+  end
+
+  describe "GET /api/pleroma/admin/users/:nickname/chats unauthenticated" do
+    setup do
+      user = insert(:user)
+      recipient = insert(:user)
+      CommonAPI.post_chat_message(user, recipient, "yo")
+      %{conn: build_conn(), user: user}
+    end
+
+    test "returns 403", %{conn: conn, user: user} do
+      conn
+      |> get("/api/pleroma/admin/users/#{user.nickname}/chats")
+      |> json_response(403)
+    end
+  end
+
   describe "GET /api/pleroma/admin/moderation_log" do
     setup do
       moderator = insert(:user, is_moderator: true)

test/web/admin_api/controllers/chat_controller_test.exs

@@ -0,0 +1,219 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.AdminAPI.ChatControllerTest do
+  use Pleroma.Web.ConnCase
+
+  import Pleroma.Factory
+
+  alias Pleroma.Chat
+  alias Pleroma.Chat.MessageReference
+  alias Pleroma.Config
+  alias Pleroma.ModerationLog
+  alias Pleroma.Object
+  alias Pleroma.Repo
+  alias Pleroma.Web.CommonAPI
+
+  defp admin_setup do
+    admin = insert(:user, is_admin: true)
+    token = insert(:oauth_admin_token, user: admin)
+
+    conn =
+      build_conn()
+      |> assign(:user, admin)
+      |> assign(:token, token)
+
+    {:ok, %{admin: admin, token: token, conn: conn}}
+  end
+
+  describe "DELETE /api/pleroma/admin/chats/:id/messages/:message_id" do
+    setup do: admin_setup()
+
+    test "it deletes a message from the chat", %{conn: conn, admin: admin} do
+      user = insert(:user)
+      recipient = insert(:user)
+
+      {:ok, message} =
+        CommonAPI.post_chat_message(user, recipient, "Hello darkness my old friend")
+
+      object = Object.normalize(message, false)
+
+      chat = Chat.get(user.id, recipient.ap_id)
+      recipient_chat = Chat.get(recipient.id, user.ap_id)
+
+      cm_ref = MessageReference.for_chat_and_object(chat, object)
+      recipient_cm_ref = MessageReference.for_chat_and_object(recipient_chat, object)
+
+      result =
+        conn
+        |> put_req_header("content-type", "application/json")
+        |> delete("/api/pleroma/admin/chats/#{chat.id}/messages/#{cm_ref.id}")
+        |> json_response_and_validate_schema(200)
+
+      log_entry = Repo.one(ModerationLog)
+
+      assert ModerationLog.get_log_entry_message(log_entry) ==
+               "@#{admin.nickname} deleted chat message ##{cm_ref.id}"
+
+      assert result["id"] == cm_ref.id
+      refute MessageReference.get_by_id(cm_ref.id)
+      refute MessageReference.get_by_id(recipient_cm_ref.id)
+      assert %{data: %{"type" => "Tombstone"}} = Object.get_by_id(object.id)
+    end
+  end
+
+  describe "GET /api/pleroma/admin/chats/:id/messages" do
+    setup do: admin_setup()
+
+    test "it paginates", %{conn: conn} do
+      user = insert(:user)
+      recipient = insert(:user)
+
+      Enum.each(1..30, fn _ ->
+        {:ok, _} = CommonAPI.post_chat_message(user, recipient, "hey")
+      end)
+
+      chat = Chat.get(user.id, recipient.ap_id)
+
+      result =
+        conn
+        |> get("/api/pleroma/admin/chats/#{chat.id}/messages")
+        |> json_response_and_validate_schema(200)
+
+      assert length(result) == 20
+
+      result =
+        conn
+        |> get("/api/pleroma/admin/chats/#{chat.id}/messages?max_id=#{List.last(result)["id"]}")
+        |> json_response_and_validate_schema(200)
+
+      assert length(result) == 10
+    end
+
+    test "it returns the messages for a given chat", %{conn: conn} do
+      user = insert(:user)
+      other_user = insert(:user)
+      third_user = insert(:user)
+
+      {:ok, _} = CommonAPI.post_chat_message(user, other_user, "hey")
+      {:ok, _} = CommonAPI.post_chat_message(user, third_user, "hey")
+      {:ok, _} = CommonAPI.post_chat_message(user, other_user, "how are you?")
+      {:ok, _} = CommonAPI.post_chat_message(other_user, user, "fine, how about you?")
+
+      chat = Chat.get(user.id, other_user.ap_id)
+
+      result =
+        conn
+        |> get("/api/pleroma/admin/chats/#{chat.id}/messages")
+        |> json_response_and_validate_schema(200)
+
+      result
+      |> Enum.each(fn message ->
+        assert message["chat_id"] == chat.id |> to_string()
+      end)
+
+      assert length(result) == 3
+    end
+  end
+
+  describe "GET /api/pleroma/admin/chats/:id" do
+    setup do: admin_setup()
+
+    test "it returns a chat", %{conn: conn} do
+      user = insert(:user)
+      other_user = insert(:user)
+
+      {:ok, chat} = Chat.get_or_create(user.id, other_user.ap_id)
+
+      result =
+        conn
+        |> get("/api/pleroma/admin/chats/#{chat.id}")
+        |> json_response_and_validate_schema(200)
+
+      assert result["id"] == to_string(chat.id)
+      assert %{} = result["sender"]
+      assert %{} = result["receiver"]
+      refute result["account"]
+    end
+  end
+
+  describe "unauthorized chat moderation" do
+    setup do
+      user = insert(:user)
+      recipient = insert(:user)
+
+      {:ok, message} = CommonAPI.post_chat_message(user, recipient, "Yo")
+      object = Object.normalize(message, false)
+      chat = Chat.get(user.id, recipient.ap_id)
+      cm_ref = MessageReference.for_chat_and_object(chat, object)
+
+      %{conn: conn} = oauth_access(["read:chats", "write:chats"])
+      %{conn: conn, chat: chat, cm_ref: cm_ref}
+    end
+
+    test "DELETE /api/pleroma/admin/chats/:id/messages/:message_id", %{
+      conn: conn,
+      chat: chat,
+      cm_ref: cm_ref
+    } do
+      conn
+      |> put_req_header("content-type", "application/json")
+      |> delete("/api/pleroma/admin/chats/#{chat.id}/messages/#{cm_ref.id}")
+      |> json_response(403)
+
+      assert MessageReference.get_by_id(cm_ref.id) == cm_ref
+    end
+
+    test "GET /api/pleroma/admin/chats/:id/messages", %{conn: conn, chat: chat} do
+      conn
+      |> get("/api/pleroma/admin/chats/#{chat.id}/messages")
+      |> json_response(403)
+    end
+
+    test "GET /api/pleroma/admin/chats/:id", %{conn: conn, chat: chat} do
+      conn
+      |> get("/api/pleroma/admin/chats/#{chat.id}")
+      |> json_response(403)
+    end
+  end
+
+  describe "unauthenticated chat moderation" do
+    setup do
+      user = insert(:user)
+      recipient = insert(:user)
+
+      {:ok, message} = CommonAPI.post_chat_message(user, recipient, "Yo")
+      object = Object.normalize(message, false)
+      chat = Chat.get(user.id, recipient.ap_id)
+      cm_ref = MessageReference.for_chat_and_object(chat, object)
+
+      %{conn: build_conn(), chat: chat, cm_ref: cm_ref}
+    end
+
+    test "DELETE /api/pleroma/admin/chats/:id/messages/:message_id", %{
+      conn: conn,
+      chat: chat,
+      cm_ref: cm_ref
+    } do
+      conn
+      |> put_req_header("content-type", "application/json")
+      |> delete("/api/pleroma/admin/chats/#{chat.id}/messages/#{cm_ref.id}")
+      |> json_response(403)
+
+      assert MessageReference.get_by_id(cm_ref.id) == cm_ref
+    end
+
+    test "GET /api/pleroma/admin/chats/:id/messages", %{conn: conn, chat: chat} do
+      conn
+      |> get("/api/pleroma/admin/chats/#{chat.id}/messages")
+      |> json_response(403)
+    end
+
+    test "GET /api/pleroma/admin/chats/:id", %{conn: conn, chat: chat} do
+      conn
+      |> get("/api/pleroma/admin/chats/#{chat.id}")
+      |> json_response(403)
+    end
+  end
+end

test/web/admin_api/controllers/instance_document_controller_test.exs

@@ -0,0 +1,106 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2020 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.AdminAPI.InstanceDocumentControllerTest do
+  use Pleroma.Web.ConnCase, async: true
+  import Pleroma.Factory
+  alias Pleroma.Config
+
+  @dir "test/tmp/instance_static"
+  @default_instance_panel ~s(<p>Welcome to <a href="https://pleroma.social" target="_blank">Pleroma!</a></p>)
+
+  setup do
+    File.mkdir_p!(@dir)
+    on_exit(fn -> File.rm_rf(@dir) end)
+  end
+
+  setup do: clear_config([:instance, :static_dir], @dir)
+
+  setup do
+    admin = insert(:user, is_admin: true)
+    token = insert(:oauth_admin_token, user: admin)
+
+    conn =
+      build_conn()
+      |> assign(:user, admin)
+      |> assign(:token, token)
+
+    {:ok, %{admin: admin, token: token, conn: conn}}
+  end
+
+  describe "GET /api/pleroma/admin/instance_document/:name" do
+    test "return the instance document url", %{conn: conn} do
+      conn = get(conn, "/api/pleroma/admin/instance_document/instance-panel")
+
+      assert content = html_response(conn, 200)
+      assert String.contains?(content, @default_instance_panel)
+    end
+
+    test "it returns 403 if requested by a non-admin" do
+      non_admin_user = insert(:user)
+      token = insert(:oauth_token, user: non_admin_user)
+
+      conn =
+        build_conn()
+        |> assign(:user, non_admin_user)
+        |> assign(:token, token)
+        |> get("/api/pleroma/admin/instance_document/instance-panel")
+
+      assert json_response(conn, :forbidden)
+    end
+
+    test "it returns 404 if the instance document with the given name doesn't exist", %{
+      conn: conn
+    } do
+      conn = get(conn, "/api/pleroma/admin/instance_document/1234")
+
+      assert json_response_and_validate_schema(conn, 404)
+    end
+  end
+
+  describe "PATCH /api/pleroma/admin/instance_document/:name" do
+    test "uploads the instance document", %{conn: conn} do
+      image = %Plug.Upload{
+        content_type: "text/html",
+        path: Path.absname("test/fixtures/custom_instance_panel.html"),
+        filename: "custom_instance_panel.html"
+      }
+
+      conn =
+        conn
+        |> put_req_header("content-type", "multipart/form-data")
+        |> patch("/api/pleroma/admin/instance_document/instance-panel", %{
+          "file" => image
+        })
+
+      assert %{"url" => url} = json_response_and_validate_schema(conn, 200)
+      index = get(build_conn(), url)
+      assert html_response(index, 200) == "<h2>Custom instance panel</h2>"
+    end
+  end
+
+  describe "DELETE /api/pleroma/admin/instance_document/:name" do
+    test "deletes the instance document", %{conn: conn} do
+      File.mkdir!(@dir <> "/instance/")
+      File.write!(@dir <> "/instance/panel.html", "Custom instance panel")
+
+      conn_resp =
+        conn
+        |> get("/api/pleroma/admin/instance_document/instance-panel")
+
+      assert html_response(conn_resp, 200) == "Custom instance panel"
+
+      conn
+      |> delete("/api/pleroma/admin/instance_document/instance-panel")
+      |> json_response_and_validate_schema(200)
+
+      conn_resp =
+        conn
+        |> get("/api/pleroma/admin/instance_document/instance-panel")
+
+      assert content = html_response(conn_resp, 200)
+      assert String.contains?(content, @default_instance_panel)
+    end
+  end
+end

test/web/admin_api/search_test.exs

@@ -177,5 +177,14 @@ defmodule Pleroma.Web.AdminAPI.SearchTest do
       assert total == 3
       assert count == 1
     end
+
+    test "it returns non-discoverable users" do
+      insert(:user)
+      insert(:user, discoverable: false)
+
+      {:ok, _results, total} = Search.user()
+
+      assert total == 2
+    end
   end
 end

test/web/common_api/common_api_test.exs

@@ -217,6 +217,17 @@ defmodule Pleroma.Web.CommonAPITest do
 
       assert message == :content_too_long
     end
+
+    test "it reject messages via MRF" do
+      clear_config([:mrf_keyword, :reject], ["GNO"])
+      clear_config([:mrf, :policies], [Pleroma.Web.ActivityPub.MRF.KeywordPolicy])
+
+      author = insert(:user)
+      recipient = insert(:user)
+
+      assert {:reject, "[KeywordPolicy] Matches with rejected keyword"} ==
+               CommonAPI.post_chat_message(author, recipient, "GNO/Linux")
+    end
   end
 
   describe "unblocking" do
@@ -1193,4 +1204,24 @@ defmodule Pleroma.Web.CommonAPITest do
       assert Visibility.get_visibility(activity) == "private"
     end
   end
+
+  describe "get_user/1" do
+    test "gets user by ap_id" do
+      user = insert(:user)
+      assert CommonAPI.get_user(user.ap_id) == user
+    end
+
+    test "gets user by guessed nickname" do
+      user = insert(:user, ap_id: "", nickname: "mario@mushroom.kingdom")
+      assert CommonAPI.get_user("https://mushroom.kingdom/users/mario") == user
+    end
+
+    test "fallback" do
+      assert %User{
+               name: "",
+               ap_id: "",
+               nickname: "erroruser@example.com"
+             } = CommonAPI.get_user("")
+    end
+  end
 end

test/web/mastodon_api/views/account_view_test.exs

@@ -68,7 +68,7 @@ defmodule Pleroma.Web.MastodonAPI.AccountViewTest do
         sensitive: false,
         pleroma: %{
           actor_type: "Person",
-          discoverable: false
+          discoverable: true
         },
         fields: []
       },
@@ -166,7 +166,7 @@ defmodule Pleroma.Web.MastodonAPI.AccountViewTest do
         sensitive: false,
         pleroma: %{
           actor_type: "Service",
-          discoverable: false
+          discoverable: true
         },
         fields: []
       },

test/web/metadata/metadata_test.exs

@@ -16,7 +16,14 @@ defmodule Pleroma.Web.MetadataTest do
     end
 
     test "for local user" do
-      user = insert(:user)
+      user = insert(:user, discoverable: false)
+
+      assert Pleroma.Web.Metadata.build_tags(%{user: user}) =~
+               "<meta content=\"noindex, noarchive\" name=\"robots\">"
+    end
+
+    test "for local user set to discoverable" do
+      user = insert(:user, discoverable: true)
 
       refute Pleroma.Web.Metadata.build_tags(%{user: user}) =~
                "<meta content=\"noindex, noarchive\" name=\"robots\">"
@@ -24,11 +31,19 @@ defmodule Pleroma.Web.MetadataTest do
   end
 
   describe "no metadata for private instances" do
-    test "for local user" do
+    test "for local user set to discoverable" do
       clear_config([:instance, :public], false)
-      user = insert(:user, bio: "This is my secret fedi account bio")
+      user = insert(:user, bio: "This is my secret fedi account bio", discoverable: true)
 
       assert "" = Pleroma.Web.Metadata.build_tags(%{user: user})
     end
+
+    test "search exclusion metadata is included" do
+      clear_config([:instance, :public], false)
+      user = insert(:user, bio: "This is my secret fedi account bio", discoverable: false)
+
+      assert ~s(<meta content="noindex, noarchive" name="robots">) ==
+               Pleroma.Web.Metadata.build_tags(%{user: user})
+    end
   end
 end

test/web/metadata/restrict_indexing_test.exs

@@ -14,8 +14,14 @@ defmodule Pleroma.Web.Metadata.Providers.RestrictIndexingTest do
 
     test "for local user" do
       assert Pleroma.Web.Metadata.Providers.RestrictIndexing.build_tags(%{
-               user: %Pleroma.User{local: true}
+               user: %Pleroma.User{local: true, discoverable: true}
              }) == []
     end
+
+    test "for local user when discoverable is false" do
+      assert Pleroma.Web.Metadata.Providers.RestrictIndexing.build_tags(%{
+               user: %Pleroma.User{local: true, discoverable: false}
+             }) == [{:meta, [name: "robots", content: "noindex, noarchive"], []}]
+    end
   end
 end

test/web/pleroma_api/controllers/chat_controller_test.exs

@@ -100,7 +100,7 @@ defmodule Pleroma.Web.PleromaAPI.ChatControllerTest do
         |> post("/api/v1/pleroma/chats/#{chat.id}/messages")
         |> json_response_and_validate_schema(400)
 
-      assert result
+      assert %{"error" => "no_content"} == result
     end
 
     test "it works with an attachment", %{conn: conn, user: user} do
@@ -126,6 +126,23 @@ defmodule Pleroma.Web.PleromaAPI.ChatControllerTest do
 
       assert result["attachment"]
     end
+
+    test "gets MRF reason when rejected", %{conn: conn, user: user} do
+      clear_config([:mrf_keyword, :reject], ["GNO"])
+      clear_config([:mrf, :policies], [Pleroma.Web.ActivityPub.MRF.KeywordPolicy])
+
+      other_user = insert(:user)
+
+      {:ok, chat} = Chat.get_or_create(user.id, other_user.ap_id)
+
+      result =
+        conn
+        |> put_req_header("content-type", "application/json")
+        |> post("/api/v1/pleroma/chats/#{chat.id}/messages", %{"content" => "GNO/Linux"})
+        |> json_response_and_validate_schema(422)
+
+      assert %{"error" => "[KeywordPolicy] Matches with rejected keyword"} == result
+    end
   end
 
   describe "DELETE /api/v1/pleroma/chats/:id/messages/:message_id" do