Merge branch 'feature/mrf-mediaproxy-warm' into 'develop'

MRF: add mediaproxy warming policy

See merge request pleroma/pleroma!1342

CHANGELOG.md

@@ -3,6 +3,10 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## [Unreleased]
+### Added
+- MRF: Support for priming the mediaproxy cache (`Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`)
+
 ## [1.0.0] - 2019-06-29
 ### Security
 - Mastodon API: Fix display names not being sanitized

docs/config.md

@@ -98,6 +98,7 @@ config :pleroma, Pleroma.Emails.Mailer,
   * `Pleroma.Web.ActivityPub.MRF.RejectNonPublic`: Drops posts with non-public visibility settings (See ``:mrf_rejectnonpublic`` section)
   * `Pleroma.Web.ActivityPub.MRF.EnsureRePrepended`: Rewrites posts to ensure that replies to posts with subjects do not have an identical subject and instead begin with re:.
   * `Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy`: Rejects posts from likely spambots by rejecting posts from new users that contain links.
+  * `Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`: Crawls attachments using their MediaProxy URLs so that the MediaProxy cache is primed.
 * `public`: Makes the client API in authentificated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network.
 * `quarantined_instances`: List of ActivityPub instances where private(DMs, followers-only) activities will not be send.
 * `managed_config`: Whenether the config for pleroma-fe is configured in this config or in ``static/config.json``

lib/pleroma/web/activity_pub/mrf/mediaproxy_warming_policy.ex

@@ -0,0 +1,56 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy do
+  @moduledoc "Preloads any attachments in the MediaProxy cache by prefetching them"
+  @behaviour Pleroma.Web.ActivityPub.MRF
+
+  alias Pleroma.HTTP
+  alias Pleroma.Web.MediaProxy
+
+  require Logger
+
+  @hackney_options [
+    pool: :media,
+    recv_timeout: 10_000
+  ]
+
+  def perform(:prefetch, url) do
+    Logger.info("Prefetching #{inspect(url)}")
+
+    url
+    |> MediaProxy.url()
+    |> HTTP.get([], adapter: @hackney_options)
+  end
+
+  def perform(:preload, %{"object" => %{"attachment" => attachments}} = _message) do
+    Enum.each(attachments, fn
+      %{"url" => url} when is_list(url) ->
+        url
+        |> Enum.each(fn
+          %{"href" => href} ->
+            PleromaJobQueue.enqueue(:background, __MODULE__, [:prefetch, href])
+
+          x ->
+            Logger.debug("Unhandled attachment URL object #{inspect(x)}")
+        end)
+
+      x ->
+        Logger.debug("Unhandled attachment #{inspect(x)}")
+    end)
+  end
+
+  @impl true
+  def filter(
+        %{"type" => "Create", "object" => %{"attachment" => attachments} = _object} = message
+      )
+      when is_list(attachments) and length(attachments) > 0 do
+    PleromaJobQueue.enqueue(:background, __MODULE__, [:preload, message])
+
+    {:ok, message}
+  end
+
+  @impl true
+  def filter(message), do: {:ok, message}
+end

test/web/activity_pub/mrf/mediaproxy_warming_policy_test.exs

@@ -0,0 +1,45 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicyTest do
+  use Pleroma.DataCase
+
+  alias Pleroma.HTTP
+  alias Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy
+
+  import Mock
+
+  @message %{
+    "type" => "Create",
+    "object" => %{
+      "type" => "Note",
+      "content" => "content",
+      "attachment" => [
+        %{"url" => [%{"href" => "http://example.com/image.jpg"}]}
+      ]
+    }
+  }
+
+  test "it prefetches media proxy URIs" do
+    with_mock HTTP, get: fn _, _, _ -> {:ok, []} end do
+      MediaProxyWarmingPolicy.filter(@message)
+      assert called(HTTP.get(:_, :_, :_))
+    end
+  end
+
+  test "it does nothing when no attachments are present" do
+    object =
+      @message["object"]
+      |> Map.delete("attachment")
+
+    message =
+      @message
+      |> Map.put("object", object)
+
+    with_mock HTTP, get: fn _, _, _ -> {:ok, []} end do
+      MediaProxyWarmingPolicy.filter(message)
+      refute called(HTTP.get(:_, :_, :_))
+    end
+  end
+end