Merge remote-tracking branch 'origin/develop' into sixohsix/pleroma-post_expiration

.dockerignore

@@ -0,0 +1,12 @@
+.*
+*.md
+AGPL-3
+CC-BY-SA-4.0
+COPYING
+*file
+elixir_buildpack.config
+docs/
+test/
+
+# Required to get version
+!.git

CHANGELOG.md

@@ -4,27 +4,33 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
 ## [Unreleased]
-### Added
+### Security
-- Expiring/ephemeral activites. All activities can have expires_on value set, which controls when they should be deleted automatically.
+- OStatus: eliminate the possibility of a protocol downgrade attack.
-- Mastodon API: in post_status, the expires_in parameter lets you set the number of minutes until an activity expires. It must be at least one hour.
+- OStatus: prevent following locked accounts, bypassing the approval process.
-- Mastodon API: all status JSON responses contain a `pleroma.expires_on` item which states when an activity will expire. The value is only shown to the user who created the activity. To everyone else it's empty.
-- Configuration: `ActivityExpiration.enabled` controls whether expired activites will get deleted at the appropriate time. Enabled by default.
 
 ### Changed
 - **Breaking:** Configuration: A setting to explicitly disable the mailer was added, defaulting to true, if you are using a mailer add `config :pleroma, Pleroma.Emails.Mailer, enabled: true` to your config
+- **Breaking:** Configuration: `/media/` is now removed when `base_url` is configured, append `/media/` to your `base_url` config to keep the old behaviour if desired
 - Configuration: OpenGraph and TwitterCard providers enabled by default
 - Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
 - Federation: Return 403 errors when trying to request pages from a user's follower/following collections if they have `hide_followers`/`hide_follows` set
 - NodeInfo: Return `skipThreadContainment` in `metadata` for the `skip_thread_containment` option
+- NodeInfo: Return `mailerEnabled` in `metadata`
 - Mastodon API: Unsubscribe followers when they unfollow a user
 - AdminAPI: Add "godmode" while fetching user statuses (i.e. admin can see private statuses)
+- Improve digest email template
 
 ### Fixed
 - Not being able to pin unlisted posts
+- Objects being re-embedded to activities after being updated (e.g faved/reposted). Running 'mix pleroma.database prune_objects' again is advised.
+- Favorites timeline doing database-intensive queries
 - Metadata rendering errors resulting in the entire page being inaccessible
+- `federation_incoming_replies_max_depth` option being ignored in certain cases
 - Federation/MediaProxy not working with instances that have wrong certificate order
 - Mastodon API: Handling of search timeouts (`/api/v1/search` and `/api/v2/search`)
 - Mastodon API: Embedded relationships not being properly rendered in the Account entity of Status entity
+- Mastodon API: follower/following counters not being nullified, when `hide_follows`/`hide_followers` is set
+- Mastodon API: `muted` in the Status entity, using author's account to determine if the tread was muted
 - Mastodon API: Add `account_id`, `type`, `offset`, and `limit` to search API (`/api/v1/search` and `/api/v2/search`)
 - Mastodon API, streaming: Fix filtering of notifications based on blocks/mutes/thread mutes
 - ActivityPub C2S: follower/following collection pages being inaccessible even when authentifucated if `hide_followers`/ `hide_follows` was set
@@ -32,11 +38,29 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Rich Media: Parser failing when no TTL can be found by image TTL setters
 - Rich Media: The crawled URL is now spliced into the rich media data.
 - ActivityPub S2S: sharedInbox usage has been mostly aligned with the rules in the AP specification.
+- ActivityPub S2S: remote user deletions now work the same as local user deletions.
+- ActivityPub S2S: POST requests are now signed with `(request-target)` pseudo-header.
+- Not being able to access the Mastodon FE login page on private instances
+- Invalid SemVer version generation, when the current branch does not have commits ahead of tag/checked out on a tag
+- Pleroma.Upload base_url was not automatically whitelisted by MediaProxy. Now your custom CDN or file hosting will be accessed directly as expected.
+- Report email not being sent to admins when the reporter is a remote user
+- MRF: ensure that subdomain_match calls are case-insensitive
+- Reverse Proxy limiting `max_body_length` was incorrectly defined and only checked `Content-Length` headers which may not be sufficient in some circumstances
+- MRF: fix use of unserializable keyword lists in describe() implementations
+- ActivityPub: Deactivated user deletion
 
 ### Added
+- Expiring/ephemeral activites. All activities can have expires_on value set, which controls when they should be deleted automatically.
+- Mastodon API: in post_status, the expires_in parameter lets you set the number of minutes until an activity expires. It must be at least one hour.
+- Mastodon API: all status JSON responses contain a `pleroma.expires_on` item which states when an activity will expire. The value is only shown to the user who created the activity. To everyone else it's empty.
+- Configuration: `ActivityExpiration.enabled` controls whether expired activites will get deleted at the appropriate time. Enabled by default.
+- Conversations: Add Pleroma-specific conversation endpoints and status posting extensions. Run the `bump_all_conversations` task again to create the necessary data.
+- **Breaking:** MRF describe API, which adds support for exposing configuration information about MRF policies to NodeInfo.
+  Custom modules will need to be updated by adding, at the very least, `def describe, do: {:ok, %{}}` to the MRF policy modules.
 - MRF: Support for priming the mediaproxy cache (`Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`)
 - MRF: Support for excluding specific domains from Transparency.
 - MRF: Support for filtering posts based on who they mention (`Pleroma.Web.ActivityPub.MRF.MentionPolicy`)
+- MRF: Support for filtering posts based on ActivityStreams vocabulary (`Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`)
 - MRF (Simple Policy): Support for wildcard domains.
 - Support for wildcard domains in user domain blocks setting.
 - Configuration: `quarantined_instances` support wildcard domains.
@@ -47,21 +71,29 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Mastodon API: Add support for categories for custom emojis by reusing the group feature. <https://github.com/tootsuite/mastodon/pull/11196>
 - Mastodon API: Add support for muting/unmuting notifications
 - Mastodon API: Add support for the `blocked_by` attribute in the relationship API (`GET /api/v1/accounts/relationships`). <https://github.com/tootsuite/mastodon/pull/10373>
+- Mastodon API: Add support for the `domain_blocking` attribute in the relationship API (`GET /api/v1/accounts/relationships`).
 - Mastodon API: Add `pleroma.deactivated` to the Account entity
 - Mastodon API: added `/auth/password` endpoint for password reset with rate limit.
 - Mastodon API: /api/v1/accounts/:id/statuses now supports nicknames or user id
+- Mastodon API: Improve support for the user profile custom fields
 - Admin API: Return users' tags when querying reports
 - Admin API: Return avatar and display name when querying users
 - Admin API: Allow querying user by ID
 - Admin API: Added support for `tuples`.
+- Admin API: Added endpoints to run mix tasks pleroma.config migrate_to_db & pleroma.config migrate_from_db
 - Added synchronization of following/followers counters for external users
 - Configuration: `enabled` option for `Pleroma.Emails.Mailer`, defaulting to `false`.
 - Configuration: Pleroma.Plugs.RateLimiter `bucket_name`, `params` options.
+- Configuration: `user_bio_length` and `user_name_length` options.
 - Addressable lists
 - Twitter API: added rate limit for `/api/account/password_reset` endpoint.
 - ActivityPub: Add an internal service actor for fetching ActivityPub objects.
 - ActivityPub: Optional signing of ActivityPub object fetches.
 - Admin API: Endpoint for fetching latest user's statuses
+- Pleroma API: Add `/api/v1/pleroma/accounts/confirmation_resend?email=<email>` for resending account confirmation.
+- Relays: Added a task to list relay subscriptions.
+- Mix Tasks: `mix pleroma.database fix_likes_collections`
+- Federation: Remove `likes` from objects.
 
 ### Changed
 - Configuration: Filter.AnonymizeFilename added ability to retain file extension with custom text
@@ -69,6 +101,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - RichMedia: parsers and their order are configured in `rich_media` config.
 - RichMedia: add the rich media ttl based on image expiration time.
 
+### Removed
+- Emoji: Remove longfox emojis.
+- Remove `Reply-To` header from report emails for admins.
+- ActivityPub: The `accept_blocks` configuration setting.
+
 ## [1.0.1] - 2019-07-14
 ### Security
 - OStatus: fix an object spoofing vulnerability.
@@ -79,6 +116,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Rich media: Do not crawl private IP ranges
 
 ### Added
+- Digest email for inactive users
 - Add a generic settings store for frontends / clients to use.
 - Explicit addressing option for posting.
 - Optional SSH access mode. (Needs `erlang-ssh` package on some distributions).
@@ -105,6 +143,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Configuration: `notify_email` option
 - Configuration: Media proxy `whitelist` option
 - Configuration: `report_uri` option
+- Configuration: `email_notifications` option
 - Configuration: `limit_to_local_content` option
 - Pleroma API: User subscriptions
 - Pleroma API: Healthcheck endpoint

Dockerfile

@@ -0,0 +1,39 @@
+FROM rinpatch/elixir:1.9.0-rc.0-alpine as build
+
+COPY . .
+
+ENV MIX_ENV=prod
+
+RUN apk add git gcc g++ musl-dev make &&\
+	echo "import Mix.Config" > config/prod.secret.exs &&\
+	mix local.hex --force &&\
+	mix local.rebar --force &&\
+	mix deps.get --only prod &&\
+	mkdir release &&\
+	mix release --path release
+
+FROM alpine:latest
+
+ARG HOME=/opt/pleroma
+ARG DATA=/var/lib/pleroma
+
+RUN echo "http://nl.alpinelinux.org/alpine/latest-stable/community" >> /etc/apk/repositories &&\
+	apk update &&\
+	apk add ncurses postgresql-client &&\
+	adduser --system --shell /bin/false --home ${HOME} pleroma &&\
+	mkdir -p ${DATA}/uploads &&\
+	mkdir -p ${DATA}/static &&\
+	chown -R pleroma ${DATA} &&\
+	mkdir -p /etc/pleroma &&\
+	chown -R pleroma /etc/pleroma
+
+USER pleroma
+
+COPY --from=build --chown=pleroma:0 /release ${HOME}
+
+COPY ./config/docker.exs /etc/pleroma/config.exs
+COPY ./docker-entrypoint.sh ${HOME}
+
+EXPOSE 4000
+
+ENTRYPOINT ["/opt/pleroma/docker-entrypoint.sh"]

README.md

@@ -21,7 +21,7 @@ If you want to run your own server, feel free to contact us at @lain@pleroma.soy
 Currently Pleroma is not packaged by any OS/Distros, but feel free to reach out to us at [#pleroma-dev on freenode](https://webchat.freenode.net/?channels=%23pleroma-dev) or via matrix at <https://matrix.heldscal.la/#/room/#freenode_#pleroma-dev:matrix.org> for assistance. If you want to change default options in your Pleroma package, please **discuss it with us first**.
 
 ### Docker
-While we don’t provide docker files, other people have written very good ones. Take a look at <https://github.com/angristan/docker-pleroma> or <https://github.com/sn0w/pleroma-docker>.
+While we don’t provide docker files, other people have written very good ones. Take a look at <https://github.com/angristan/docker-pleroma> or <https://glitch.sh/sn0w/pleroma-docker>.
 
 ### Dependencies
 

config/config.exs

@@ -253,6 +253,12 @@ config :pleroma, :instance,
   skip_thread_containment: true,
   limit_to_local_content: :unauthenticated,
   dynamic_configuration: false,
+  user_bio_length: 5000,
+  user_name_length: 100,
+  max_account_fields: 10,
+  max_remote_account_fields: 20,
+  account_field_name_length: 512,
+  account_field_value_length: 512,
   external_user_synchronization: true
 
 config :pleroma, :markup,
@@ -302,7 +308,6 @@ config :pleroma, :assets,
   default_mascot: :pleroma_fox_tan
 
 config :pleroma, :activitypub,
-  accept_blocks: true,
   unfollow_blocked: true,
   outgoing_blocks: true,
   follow_handshake_timeout: 500,
@@ -337,6 +342,10 @@ config :pleroma, :mrf_keyword,
 
 config :pleroma, :mrf_subchain, match_actor: %{}
 
+config :pleroma, :mrf_vocabulary,
+  accept: [],
+  reject: []
+
 config :pleroma, :rich_media,
   enabled: true,
   ignore_hosts: [],
@@ -508,6 +517,17 @@ config :pleroma, :auth, oauth_consumer_strategies: oauth_consumer_strategies
 
 config :pleroma, Pleroma.Emails.Mailer, adapter: Swoosh.Adapters.Sendmail, enabled: false
 
+config :pleroma, Pleroma.Emails.UserEmail,
+  logo: nil,
+  styling: %{
+    link_color: "#d8a070",
+    background_color: "#2C3645",
+    content_background_color: "#1B2635",
+    header_color: "#d8a070",
+    text_color: "#b9b9ba",
+    text_muted_color: "#b9b9ba"
+  }
+
 config :prometheus, Pleroma.Web.Endpoint.MetricsExporter, path: "/api/pleroma/app_metrics"
 
 config :pleroma, Pleroma.ScheduledActivity,
@@ -515,6 +535,14 @@ config :pleroma, Pleroma.ScheduledActivity,
   total_user_limit: 300,
   enabled: true
 
+config :pleroma, :email_notifications,
+  digest: %{
+    active: false,
+    schedule: "0 0 * * 0",
+    interval: 7,
+    inactivity_threshold: 7
+  }
+
 config :pleroma, :oauth2,
   token_expires_in: 600,
   issue_new_refresh_token: true,
@@ -535,7 +563,9 @@ config :pleroma, :rate_limit,
   relation_id_action: {60_000, 2},
   statuses_actions: {10_000, 15},
   status_id_action: {60_000, 3},
-  password_reset: {1_800_000, 5}
+  password_reset: {1_800_000, 5},
+  account_confirmation_resend: {8_640_000, 5},
+  ap_routes: {60_000, 15}
 
 config :pleroma, Pleroma.ActivityExpiration, enabled: true
 

config/docker.exs

@@ -0,0 +1,68 @@
+import Config
+
+config :pleroma, Pleroma.Web.Endpoint,
+  url: [host: System.get_env("DOMAIN", "localhost"), scheme: "https", port: 443],
+  http: [ip: {0, 0, 0, 0}, port: 4000]
+
+config :pleroma, :instance,
+  name: System.get_env("INSTANCE_NAME", "Pleroma"),
+  email: System.get_env("ADMIN_EMAIL"),
+  notify_email: System.get_env("NOTIFY_EMAIL"),
+  limit: 5000,
+  registrations_open: false,
+  dynamic_configuration: true
+
+config :pleroma, Pleroma.Repo,
+  adapter: Ecto.Adapters.Postgres,
+  username: System.get_env("DB_USER", "pleroma"),
+  password: System.fetch_env!("DB_PASS"),
+  database: System.get_env("DB_NAME", "pleroma"),
+  hostname: System.get_env("DB_HOST", "db"),
+  pool_size: 10
+
+# Configure web push notifications
+config :web_push_encryption, :vapid_details, subject: "mailto:#{System.get_env("NOTIFY_EMAIL")}"
+
+config :pleroma, :database, rum_enabled: false
+config :pleroma, :instance, static_dir: "/var/lib/pleroma/static"
+config :pleroma, Pleroma.Uploaders.Local, uploads: "/var/lib/pleroma/uploads"
+
+# We can't store the secrets in this file, since this is baked into the docker image
+if not File.exists?("/var/lib/pleroma/secret.exs") do
+  secret = :crypto.strong_rand_bytes(64) |> Base.encode64() |> binary_part(0, 64)
+  signing_salt = :crypto.strong_rand_bytes(8) |> Base.encode64() |> binary_part(0, 8)
+  {web_push_public_key, web_push_private_key} = :crypto.generate_key(:ecdh, :prime256v1)
+
+  secret_file =
+    EEx.eval_string(
+      """
+      import Config
+
+      config :pleroma, Pleroma.Web.Endpoint,
+        secret_key_base: "<%= secret %>",
+        signing_salt: "<%= signing_salt %>"
+
+      config :web_push_encryption, :vapid_details,
+        public_key: "<%= web_push_public_key %>",
+        private_key: "<%= web_push_private_key %>"
+      """,
+      secret: secret,
+      signing_salt: signing_salt,
+      web_push_public_key: Base.url_encode64(web_push_public_key, padding: false),
+      web_push_private_key: Base.url_encode64(web_push_private_key, padding: false)
+    )
+
+  File.write("/var/lib/pleroma/secret.exs", secret_file)
+end
+
+import_config("/var/lib/pleroma/secret.exs")
+
+# For additional user config
+if File.exists?("/var/lib/pleroma/config.exs"),
+  do: import_config("/var/lib/pleroma/config.exs"),
+  else:
+    File.write("/var/lib/pleroma/config.exs", """
+    import Config
+
+    # For additional configuration outside of environmental variables
+    """)

config/emoji.txt

@@ -1,30 +1,2 @@
 firefox, /emoji/Firefox.gif, Gif,Fun
 blank, /emoji/blank.png, Fun
-f_00b, /emoji/f_00b.png
-f_00b11b, /emoji/f_00b11b.png
-f_00b33b, /emoji/f_00b33b.png
-f_00h, /emoji/f_00h.png
-f_00t, /emoji/f_00t.png
-f_01b, /emoji/f_01b.png
-f_03b, /emoji/f_03b.png
-f_10b, /emoji/f_10b.png
-f_11b, /emoji/f_11b.png
-f_11b00b, /emoji/f_11b00b.png
-f_11b22b, /emoji/f_11b22b.png
-f_11h, /emoji/f_11h.png
-f_11t, /emoji/f_11t.png
-f_12b, /emoji/f_12b.png
-f_21b, /emoji/f_21b.png
-f_22b, /emoji/f_22b.png
-f_22b11b, /emoji/f_22b11b.png
-f_22b33b, /emoji/f_22b33b.png
-f_22h, /emoji/f_22h.png
-f_22t, /emoji/f_22t.png
-f_23b, /emoji/f_23b.png
-f_30b, /emoji/f_30b.png
-f_32b, /emoji/f_32b.png
-f_33b, /emoji/f_33b.png
-f_33b00b, /emoji/f_33b00b.png
-f_33b22b, /emoji/f_33b22b.png
-f_33h, /emoji/f_33h.png
-f_33t, /emoji/f_33t.png

config/test.exs

@@ -29,7 +29,8 @@ config :pleroma, :instance,
   email: "admin@example.com",
   notify_email: "noreply@example.com",
   skip_thread_containment: false,
-  federating: false
+  federating: false,
+  external_user_synchronization: false
 
 config :pleroma, :activitypub, sign_object_fetches: false
 
@@ -70,7 +71,8 @@ config :pleroma, Pleroma.ScheduledActivity,
 config :pleroma, :rate_limit,
   search: [{1000, 30}, {1000, 30}],
   app_account_creation: {10_000, 5},
-  password_reset: {1000, 30}
+  password_reset: {1000, 30},
+  ap_routes: nil
 
 config :pleroma, :http_security, report_uri: "https://endpoint.com"
 
@@ -80,6 +82,8 @@ rum_enabled = System.get_env("RUM_ENABLED") == "true"
 config :pleroma, :database, rum_enabled: rum_enabled
 IO.puts("RUM enabled: #{rum_enabled}")
 
+config :joken, default_signer: "yU8uHKq+yyAkZ11Hx//jcdacWc8yQ1bxAAGrplzB0Zwwjkp35v0RK9SO8WTPr6QZ"
+
 config :pleroma, Pleroma.ReverseProxy.Client, Pleroma.ReverseProxy.ClientMock
 
 if File.exists?("./config/test.secret.exs") do

docker-entrypoint.sh

@@ -0,0 +1,14 @@
+#!/bin/ash
+
+set -e
+
+echo "-- Waiting for database..."
+while ! pg_isready -U ${DB_USER:-pleroma} -d postgres://${DB_HOST:-db}:5432/${DB_NAME:-pleroma} -t 1; do
+    sleep 1s
+done
+
+echo "-- Running migrations..."
+$HOME/bin/pleroma_ctl migrate
+
+echo "-- Starting!"
+exec $HOME/bin/pleroma start

docs/api/admin_api.md

@@ -575,6 +575,29 @@ Note: Available `:permission_group` is currently moderator and admin. 404 is ret
     - 404 Not Found `"Not found"`
   - On success: 200 OK `{}`
 
+
+## `/api/pleroma/admin/config/migrate_to_db`
+### Run mix task pleroma.config migrate_to_db
+Copy settings on key `:pleroma` to DB.
+- Method `GET`
+- Params: none
+- Response:
+
+```json
+{}
+```
+
+## `/api/pleroma/admin/config/migrate_from_db`
+### Run mix task pleroma.config migrate_from_db
+Copy all settings from DB to `config/prod.exported_from_db.secret.exs` with deletion from DB.
+- Method `GET`
+- Params: none
+- Response:
+
+```json
+{}
+```
+
 ## `/api/pleroma/admin/config`
 ### List config settings
 List config settings only works with `:pleroma => :instance => :dynamic_configuration` setting to `true`.
@@ -604,6 +627,9 @@ Tuples can be passed as `{"tuple": ["first_val", Pleroma.Module, []]}`.
 Keywords can be passed as lists with 2 child tuples, e.g.
 `[{"tuple": ["first_val", Pleroma.Module]}, {"tuple": ["second_val", true]}]`.
 
+If value contains list of settings `[subkey: val1, subkey2: val2, subkey3: val3]`, it's possible to remove only subkeys instead of all settings passing `subkeys` parameter. E.g.:
+{"group": "pleroma", "key": "some_key", "delete": "true", "subkeys": [":subkey", ":subkey3"]}.
+
 Compile time settings (need instance reboot):
 - all settings by this keys:
   - `:hackney_pools`
@@ -622,6 +648,7 @@ Compile time settings (need instance reboot):
     - `key` (string or string with leading `:` for atoms)
     - `value` (string, [], {} or {"tuple": []})
     - `delete` = true (optional, if parameter must be deleted)
+    - `subkeys` [(string with leading `:` for atoms)] (optional, works only if `delete=true` parameter is passed, otherwise will be ignored)
   ]
 
 - Request (example):

docs/api/differences_in_mastoapi_responses.md

@@ -60,12 +60,19 @@ Has these additional fields under the `pleroma` object:
 - `show_role`: boolean, nullable, true when the user wants his role (e.g admin, moderator) to be shown
 - `no_rich_text` - boolean, nullable, true when html tags are stripped from all statuses requested from the API
 
+## Conversations
+
+Has an additional field under the `pleroma` object:
+
+- `recipients`: The list of the recipients of this Conversation. These will be addressed when replying to this conversation.
+
 ## Account Search
 
 Behavior has changed:
 
 - `/api/v1/accounts/search`: Does not require authentication
 
+
 ## Notifications
 
 Has these additional fields under the `pleroma` object:
@@ -81,6 +88,7 @@ Additional parameters can be added to the JSON body/Form data:
 - `to`: A list of nicknames (like `lain@soykaf.club` or `lain` on the local server) that will be used to determine who is going to be addressed by this post. Using this will disable the implicit addressing by mentioned names in the `status` body, only the people in the `to` list will be addressed. The normal rules for for post visibility are not affected by this and will still apply.
 - `visibility`: string, besides standard MastoAPI values (`direct`, `private`, `unlisted` or `public`) it can be used to address a List by setting it to `list:LIST_ID`.
 - `expires_on`: datetime (iso8601), sets when the posted activity should expire. When a posted activity expires it will be deleted from the server, and a delete request for it will be federated.
+- `in_reply_to_conversation_id`: Will reply to a given conversation, addressing only the people who are part of the recipient set of that conversation. Sets the visibility to `direct`.
 
 ## PATCH `/api/v1/update_credentials`
 

docs/api/pleroma_api.md

@@ -245,6 +245,14 @@ See [Admin-API](Admin-API.md)
 - PATCH `/api/v1/pleroma/accounts/update_banner`: Set/clear user banner image
 - PATCH `/api/v1/pleroma/accounts/update_background`: Set/clear user background image
 
+## `/api/v1/pleroma/accounts/confirmation_resend`
+### Resend confirmation email
+* Method `POST`
+* Params:
+    * `email`: email of that needs to be verified
+* Authentication: not required
+* Response: 204 No Content 
+
 ## `/api/v1/pleroma/mascot`
 ### Gets user mascot image
 * Method `GET`
@@ -311,3 +319,38 @@ See [Admin-API](Admin-API.md)
   "healthy": true # Instance state
 }
 ```
+
+# Pleroma Conversations
+
+Pleroma Conversations have the same general structure that Mastodon Conversations have. The behavior differs in the following ways when using these endpoints:
+
+1. Pleroma Conversations never add or remove recipients, unless explicitly changed by the user. 
+2. Pleroma Conversations statuses can be requested by Conversation id.
+3. Pleroma Conversations can be replied to.
+
+Conversations have the additional field "recipients" under the "pleroma" key. This holds a list of all the accounts that will receive a message in this conversation.
+
+The status posting endpoint takes an additional parameter, `in_reply_to_conversation_id`, which, when set, will set the visiblity to direct and address only the people who are the recipients of that Conversation.
+
+
+## `GET /api/v1/pleroma/conversations/:id/statuses`
+### Timeline for a given conversation
+* Method `GET`
+* Authentication: required
+* Params: Like other timelines
+* Response: JSON, statuses (200 - healthy, 503 unhealthy).
+
+## `GET /api/v1/pleroma/conversations/:id`
+### The conversation with the given ID.
+* Method `GET`
+* Authentication: required
+* Params: None
+* Response: JSON, statuses (200 - healthy, 503 unhealthy).
+
+## `PATCH /api/v1/pleroma/conversations/:id`
+### Update a conversation. Used to change the set of recipients.
+* Method `PATCH`
+* Authentication: required
+* Params:
+    * `recipients`: A list of ids of users that should receive posts to this conversation. This will replace the current list of recipients, so submit the full list. The owner of owner of the conversation will always be part of the set of recipients, though.
+* Response: JSON, statuses (200 - healthy, 503 unhealthy)

docs/config.md

@@ -18,6 +18,7 @@ Note: `strip_exif` has been replaced by `Pleroma.Upload.Filter.Mogrify`.
 
 ## Pleroma.Uploaders.S3
 * `bucket`: S3 bucket name
+* `bucket_namespace`: S3 bucket namespace
 * `public_endpoint`: S3 endpoint that the user finally accesses(ex. "https://s3.dualstack.ap-northeast-1.amazonaws.com")
 * `truncated_namespace`: If you use S3 compatible service such as Digital Ocean Spaces or CDN, set folder name or "" etc.
 For example, when using CDN to S3 virtual host format, set "".
@@ -25,7 +26,7 @@ At this time, write CNAME to CDN in public_endpoint.
 
 ## Pleroma.Upload.Filter.Mogrify
 
-* `args`: List of actions for the `mogrify` command like `"strip"` or `["strip", "auto-orient", {"impode", "1"}]`.
+* `args`: List of actions for the `mogrify` command like `"strip"` or `["strip", "auto-orient", {"implode", "1"}]`.
 
 ## Pleroma.Upload.Filter.Dedupe
 
@@ -102,6 +103,7 @@ config :pleroma, Pleroma.Emails.Mailer,
   * `Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy`: Rejects posts from likely spambots by rejecting posts from new users that contain links.
   * `Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`: Crawls attachments using their MediaProxy URLs so that the MediaProxy cache is primed.
   * `Pleroma.Web.ActivityPub.MRF.MentionPolicy`: Drops posts mentioning configurable users. (see `:mrf_mention` section)
+  * `Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`: Restricts activities to a configured set of vocabulary. (see `:mrf_vocabulary` section)
 * `public`: Makes the client API in authentificated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network.
 * `quarantined_instances`: List of ActivityPub instances where private(DMs, followers-only) activities will not be send.
 * `managed_config`: Whenether the config for pleroma-fe is configured in this config or in ``static/config.json``
@@ -125,9 +127,15 @@ config :pleroma, Pleroma.Emails.Mailer,
 * `safe_dm_mentions`: If set to true, only mentions at the beginning of a post will be used to address people in direct messages. This is to prevent accidental mentioning of people when talking about them (e.g. "@friend hey i really don't like @enemy"). Default: `false`.
 * `healthcheck`: If set to true, system data will be shown on ``/api/pleroma/healthcheck``.
 * `remote_post_retention_days`: The default amount of days to retain remote posts when pruning the database.
+* `user_bio_length`: A user bio maximum length (default: `5000`)
+* `user_name_length`: A user name maximum length (default: `100`)
 * `skip_thread_containment`: Skip filter out broken threads. The default is `false`.
 * `limit_to_local_content`: Limit unauthenticated users to search for local statutes and users only. Possible values: `:unauthenticated`, `:all` and `false`. The default is `:unauthenticated`.
 * `dynamic_configuration`: Allow transferring configuration to DB with the subsequent customization from Admin api.
+* `max_account_fields`: The maximum number of custom fields in the user profile (default: `10`)
+* `max_remote_account_fields`: The maximum number of custom fields in the remote user profile (default: `20`)
+* `account_field_name_length`: An account field name maximum length (default: `512`)
+* `account_field_value_length`: An account field value maximum length (default: `512`)
 * `external_user_synchronization`: Enabling following/followers counters synchronization for external users.
 
 
@@ -275,6 +283,10 @@ config :pleroma, :mrf_subchain,
 ## :mrf_mention
 * `actors`: A list of actors, for which to drop any posts mentioning.
 
+## :mrf_vocabulary
+* `accept`: A list of ActivityStreams terms to accept.  If empty, all supported messages are accepted.
+* `reject`: A list of ActivityStreams terms to reject.  If empty, no messages are rejected.
+
 ## :media_proxy
 * `enabled`: Enables proxying of remote media to the instance’s proxy
 * `base_url`: The base URL to access a user-uploaded file. Useful when you want to proxy the media files via another host/CDN fronts.
@@ -328,7 +340,6 @@ config :pleroma, Pleroma.Web.Endpoint,
 This will make Pleroma listen on `127.0.0.1` port `8080` and generate urls starting with `https://example.com:2020`
 
 ## :activitypub
-* ``accept_blocks``: Whether to accept incoming block activities from other instances
 * ``unfollow_blocked``: Whether blocks result in people getting unfollowed
 * ``outgoing_blocks``: Whether to federate blocks to other instances
 * ``deny_follow_blocked``: Whether to disallow following an account that has blocked the user in question
@@ -540,6 +551,23 @@ Authentication / authorization settings.
 * `oauth_consumer_template`: OAuth consumer mode authentication form template. By default it's `consumer.html` which corresponds to `lib/pleroma/web/templates/o_auth/o_auth/consumer.html.eex`.
 * `oauth_consumer_strategies`: the list of enabled OAuth consumer strategies; by default it's set by `OAUTH_CONSUMER_STRATEGIES` environment variable. Each entry in this space-delimited string should be of format `<strategy>` or `<strategy>:<dependency>` (e.g. `twitter` or `keycloak:ueberauth_keycloak_strategy` in case dependency is named differently than `ueberauth_<strategy>`).
 
+## :email_notifications
+
+Email notifications settings.
+
+  - digest - emails of "what you've missed" for users who have been
+    inactive for a while.
+    - active: globally enable or disable digest emails
+    - schedule: When to send digest email, in [crontab format](https://en.wikipedia.org/wiki/Cron).
+      "0 0 * * 0" is the default, meaning "once a week at midnight on Sunday morning"
+    - interval: Minimum interval between digest emails to one user
+    - inactivity_threshold: Minimum user inactivity threshold
+
+## Pleroma.Emails.UserEmail
+
+- `:logo` - a path to a custom logo. Set it to `nil` to use the default Pleroma logo.
+- `:styling` - a map with color settings for email templates.
+
 ## OAuth consumer mode
 
 OAuth consumer mode allows sign in / sign up via external OAuth providers (e.g. Twitter, Facebook, Google, Microsoft, etc.).

docs/config/howto_mediaproxy.md

@@ -1,8 +1,8 @@
 # How to activate mediaproxy
 ## Explanation
 
-Without the `mediaproxy` function, Pleroma don't store any remote content like pictures, video etc. locally. So every time you open Pleroma, the content is loaded from the source server, from where the post is coming. This can result in slowly loading content or/and increased bandwidth usage on the source server.
+Without the `mediaproxy` function, Pleroma doesn't store any remote content like pictures, video etc. locally. So every time you open Pleroma, the content is loaded from the source server, from where the post is coming. This can result in slowly loading content or/and increased bandwidth usage on the source server.
-With the `mediaproxy` function you can use the cache ability of nginx, to cache these content, so user can access it faster, cause it's loaded from your server.
+With the `mediaproxy` function you can use nginx to cache this content, so users can access it faster, because it's loaded from your server.
 
 ## Activate it
 

lib/mix/tasks/pleroma/benchmark.ex

@@ -26,4 +26,48 @@ defmodule Mix.Tasks.Pleroma.Benchmark do
       end
     })
   end
+
+  def run(["render_timeline", nickname]) do
+    start_pleroma()
+    user = Pleroma.User.get_by_nickname(nickname)
+
+    activities =
+      %{}
+      |> Map.put("type", ["Create", "Announce"])
+      |> Map.put("blocking_user", user)
+      |> Map.put("muting_user", user)
+      |> Map.put("user", user)
+      |> Map.put("limit", 80)
+      |> Pleroma.Web.ActivityPub.ActivityPub.fetch_public_activities()
+      |> Enum.reverse()
+
+    inputs = %{
+      "One activity" => Enum.take_random(activities, 1),
+      "Ten activities" => Enum.take_random(activities, 10),
+      "Twenty activities" => Enum.take_random(activities, 20),
+      "Forty activities" => Enum.take_random(activities, 40),
+      "Eighty activities" => Enum.take_random(activities, 80)
+    }
+
+    Benchee.run(
+      %{
+        "Parallel rendering" => fn activities ->
+          Pleroma.Web.MastodonAPI.StatusView.render("index.json", %{
+            activities: activities,
+            for: user,
+            as: :activity
+          })
+        end,
+        "Standart rendering" => fn activities ->
+          Pleroma.Web.MastodonAPI.StatusView.render("index.json", %{
+            activities: activities,
+            for: user,
+            as: :activity,
+            parallel: false
+          })
+        end
+      },
+      inputs: inputs
+    )
+  end
 end

lib/mix/tasks/pleroma/config.ex

@@ -15,7 +15,7 @@ defmodule Mix.Tasks.Pleroma.Config do
 
       mix pleroma.config migrate_to_db
 
-  ## Transfers config from DB to file.
+  ## Transfers config from DB to file `config/env.exported_from_db.secret.exs`
 
       mix pleroma.config migrate_from_db ENV
   """

lib/mix/tasks/pleroma/database.ex

@@ -8,6 +8,7 @@ defmodule Mix.Tasks.Pleroma.Database do
   alias Pleroma.Repo
   alias Pleroma.User
   require Logger
+  require Pleroma.Constants
   import Mix.Pleroma
   use Mix.Task
 
@@ -35,6 +36,10 @@ defmodule Mix.Tasks.Pleroma.Database do
   ## Remove duplicated items from following and update followers count for all users
 
       mix pleroma.database update_users_following_followers_counts
+
+  ## Fix the pre-existing "likes" collections for all objects
+
+      mix pleroma.database fix_likes_collections
   """
   def run(["remove_embedded_objects" | args]) do
     {options, [], []} =
@@ -99,10 +104,15 @@ defmodule Mix.Tasks.Pleroma.Database do
       NaiveDateTime.utc_now()
       |> NaiveDateTime.add(-(deadline * 86_400))
 
-    public = "https://www.w3.org/ns/activitystreams#Public"
-
     from(o in Object,
-      where: fragment("?->'to' \\? ? OR ?->'cc' \\? ?", o.data, ^public, o.data, ^public),
+      where:
+        fragment(
+          "?->'to' \\? ? OR ?->'cc' \\? ?",
+          o.data,
+          ^Pleroma.Constants.as_public(),
+          o.data,
+          ^Pleroma.Constants.as_public()
+        ),
       where: o.inserted_at < ^time_deadline,
       where:
         fragment("split_part(?->>'actor', '/', 3) != ?", o.data, ^Pleroma.Web.Endpoint.host())
@@ -119,4 +129,36 @@ defmodule Mix.Tasks.Pleroma.Database do
       )
     end
   end
+
+  def run(["fix_likes_collections"]) do
+    import Ecto.Query
+
+    start_pleroma()
+
+    from(object in Object,
+      where: fragment("(?)->>'likes' is not null", object.data),
+      select: %{id: object.id, likes: fragment("(?)->>'likes'", object.data)}
+    )
+    |> Pleroma.RepoStreamer.chunk_stream(100)
+    |> Stream.each(fn objects ->
+      ids =
+        objects
+        |> Enum.filter(fn object -> object.likes |> Jason.decode!() |> is_map() end)
+        |> Enum.map(& &1.id)
+
+      Object
+      |> where([object], object.id in ^ids)
+      |> update([object],
+        set: [
+          data:
+            fragment(
+              "jsonb_set(?, '{likes}', '[]'::jsonb, true)",
+              object.data
+            )
+        ]
+      )
+      |> Repo.update_all([], timeout: :infinity)
+    end)
+    |> Stream.run()
+  end
 end

lib/mix/tasks/pleroma/digest.ex

@@ -0,0 +1,41 @@
+defmodule Mix.Tasks.Pleroma.Digest do
+  use Mix.Task
+
+  @shortdoc "Manages digest emails"
+  @moduledoc """
+  Manages digest emails
+
+  ## Send digest email since given date (user registration date by default)
+  ignoring user activity status.
+
+  ``mix pleroma.digest test <nickname> <since_date>``
+
+  Example: ``mix pleroma.digest test donaldtheduck 2019-05-20``
+  """
+  def run(["test", nickname | opts]) do
+    Mix.Pleroma.start_pleroma()
+
+    user = Pleroma.User.get_by_nickname(nickname)
+
+    last_digest_emailed_at =
+      with [date] <- opts,
+           {:ok, datetime} <- Timex.parse(date, "{YYYY}-{0M}-{0D}") do
+        datetime
+      else
+        _ -> user.inserted_at
+      end
+
+    patched_user = %{user | last_digest_emailed_at: last_digest_emailed_at}
+
+    with %Swoosh.Email{} = email <- Pleroma.Emails.UserEmail.digest_email(patched_user) do
+      {:ok, _} = Pleroma.Emails.Mailer.deliver(email)
+
+      Mix.shell().info("Digest email have been sent to #{nickname} (#{user.email})")
+    else
+      _ ->
+        Mix.shell().info(
+          "Cound't find any mentions for #{nickname} since #{last_digest_emailed_at}"
+        )
+    end
+  end
+end

lib/mix/tasks/pleroma/instance.ex

@@ -183,6 +183,7 @@ defmodule Mix.Tasks.Pleroma.Instance do
         )
 
       secret = :crypto.strong_rand_bytes(64) |> Base.encode64() |> binary_part(0, 64)
+      jwt_secret = :crypto.strong_rand_bytes(64) |> Base.encode64() |> binary_part(0, 64)
       signing_salt = :crypto.strong_rand_bytes(8) |> Base.encode64() |> binary_part(0, 8)
       {web_push_public_key, web_push_private_key} = :crypto.generate_key(:ecdh, :prime256v1)
       template_dir = Application.app_dir(:pleroma, "priv") <> "/templates"
@@ -200,6 +201,7 @@ defmodule Mix.Tasks.Pleroma.Instance do
           dbuser: dbuser,
           dbpass: dbpass,
           secret: secret,
+          jwt_secret: jwt_secret,
           signing_salt: signing_salt,
           web_push_public_key: Base.url_encode64(web_push_public_key, padding: false),
           web_push_private_key: Base.url_encode64(web_push_private_key, padding: false),

lib/mix/tasks/pleroma/relay.ex

@@ -5,6 +5,7 @@
 defmodule Mix.Tasks.Pleroma.Relay do
   use Mix.Task
   import Mix.Pleroma
+  alias Pleroma.User
   alias Pleroma.Web.ActivityPub.Relay
 
   @shortdoc "Manages remote relays"
@@ -22,6 +23,10 @@ defmodule Mix.Tasks.Pleroma.Relay do
   ``mix pleroma.relay unfollow <relay_url>``
 
   Example: ``mix pleroma.relay unfollow https://example.org/relay``
+
+  ## List relay subscriptions
+
+  ``mix pleroma.relay list``
   """
   def run(["follow", target]) do
     start_pleroma()
@@ -44,4 +49,17 @@ defmodule Mix.Tasks.Pleroma.Relay do
       {:error, e} -> shell_error("Error while following #{target}: #{inspect(e)}")
     end
   end
+
+  def run(["list"]) do
+    start_pleroma()
+
+    with %User{following: following} = _user <- Relay.get_actor() do
+      following
+      |> Enum.map(fn entry -> URI.parse(entry).host end)
+      |> Enum.uniq()
+      |> Enum.each(&shell_info(&1))
+    else
+      e -> shell_error("Error while fetching relay subscription list: #{inspect(e)}")
+    end
+  end
 end

lib/mix/tasks/pleroma/user.ex

@@ -31,8 +31,8 @@ defmodule Mix.Tasks.Pleroma.User do
       mix pleroma.user invite [OPTION...]
 
     Options:
-    - `--expires_at DATE` - last day on which token is active (e.g. "2019-04-05")
+    - `--expires-at DATE` - last day on which token is active (e.g. "2019-04-05")
-    - `--max_use NUMBER` - maximum numbers of token uses
+    - `--max-use NUMBER` - maximum numbers of token uses
 
   ## List generated invites
 

lib/pleroma/activity.ex

@@ -99,6 +99,7 @@ defmodule Pleroma.Activity do
     from([a] in query,
       left_join: tm in ThreadMute,
       on: tm.user_id == ^user.id and tm.context == fragment("?->>'context'", a.data),
+      as: :thread_mute,
       select: %Activity{a | thread_muted?: not is_nil(tm.id)}
     )
   end
@@ -227,6 +228,29 @@ defmodule Pleroma.Activity do
 
   def get_create_by_object_ap_id(_), do: nil
 
+  def create_by_object_ap_id_with_object(ap_ids) when is_list(ap_ids) do
+    from(
+      activity in Activity,
+      where:
+        fragment(
+          "coalesce((?)->'object'->>'id', (?)->>'object') = ANY(?)",
+          activity.data,
+          activity.data,
+          ^ap_ids
+        ),
+      where: fragment("(?)->>'type' = 'Create'", activity.data),
+      inner_join: o in Object,
+      on:
+        fragment(
+          "(?->>'id') = COALESCE(?->'object'->>'id', ?->>'object')",
+          o.data,
+          activity.data,
+          activity.data
+        ),
+      preload: [object: o]
+    )
+  end
+
   def create_by_object_ap_id_with_object(ap_id) when is_binary(ap_id) do
     from(
       activity in Activity,
@@ -266,8 +290,8 @@ defmodule Pleroma.Activity do
 
   defp get_in_reply_to_activity_from_object(_), do: nil
 
-  def get_in_reply_to_activity(%Activity{data: %{"object" => object}}) do
+  def get_in_reply_to_activity(%Activity{} = activity) do
-    get_in_reply_to_activity_from_object(Object.normalize(object))
+    get_in_reply_to_activity_from_object(Object.normalize(activity))
   end
 
   def normalize(obj) when is_map(obj), do: get_by_ap_id_with_object(obj["id"])

lib/pleroma/activity/search.ex

@@ -9,6 +9,8 @@ defmodule Pleroma.Activity.Search do
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.Visibility
 
+  require Pleroma.Constants
+
   import Ecto.Query
 
   def search(user, search_query, options \\ []) do
@@ -39,7 +41,7 @@ defmodule Pleroma.Activity.Search do
   defp restrict_public(q) do
     from([a, o] in q,
       where: fragment("?->>'type' = 'Create'", a.data),
-      where: "https://www.w3.org/ns/activitystreams#Public" in a.recipients
+      where: ^Pleroma.Constants.as_public() in a.recipients
     )
   end
 

lib/pleroma/application.ex

@@ -3,11 +3,14 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Application do
+  import Cachex.Spec
   use Application
 
   @name Mix.Project.config()[:name]
   @version Mix.Project.config()[:version]
   @repository Mix.Project.config()[:source_url]
+  @env Mix.env()
+
   def name, do: @name
   def version, do: @version
   def named_version, do: @name <> " " <> @version
@@ -21,120 +24,25 @@ defmodule Pleroma.Application do
   # See http://elixir-lang.org/docs/stable/elixir/Application.html
   # for more information on OTP Applications
   def start(_type, _args) do
-    import Cachex.Spec
-
     Pleroma.Config.DeprecationWarnings.warn()
     setup_instrumenters()
 
     # Define workers and child supervisors to be supervised
     children =
       [
-        # Start the Ecto repository
+        Pleroma.Repo,
-        %{id: Pleroma.Repo, start: {Pleroma.Repo, :start_link, []}, type: :supervisor},
+        Pleroma.Config.TransferTask,
-        %{id: Pleroma.Config.TransferTask, start: {Pleroma.Config.TransferTask, :start_link, []}},
+        Pleroma.Emoji,
-        %{id: Pleroma.Emoji, start: {Pleroma.Emoji, :start_link, []}},
+        Pleroma.Captcha,
-        %{id: Pleroma.Captcha, start: {Pleroma.Captcha, :start_link, []}},
+        Pleroma.FlakeId,
-        %{
+        Pleroma.ScheduledActivityWorker,
-          id: :cachex_used_captcha_cache,
+        Pleroma.ActiviyExpirationWorker
-          start:
-            {Cachex, :start_link,
-             [
-               :used_captcha_cache,
-               [
-                 ttl_interval:
-                   :timer.seconds(Pleroma.Config.get!([Pleroma.Captcha, :seconds_valid]))
-               ]
-             ]}
-        },
-        %{
-          id: :cachex_user,
-          start:
-            {Cachex, :start_link,
-             [
-               :user_cache,
-               [
-                 default_ttl: 25_000,
-                 ttl_interval: 1000,
-                 limit: 2500
-               ]
-             ]}
-        },
-        %{
-          id: :cachex_object,
-          start:
-            {Cachex, :start_link,
-             [
-               :object_cache,
-               [
-                 default_ttl: 25_000,
-                 ttl_interval: 1000,
-                 limit: 2500
-               ]
-             ]}
-        },
-        %{
-          id: :cachex_rich_media,
-          start:
-            {Cachex, :start_link,
-             [
-               :rich_media_cache,
-               [
-                 default_ttl: :timer.minutes(120),
-                 limit: 5000
-               ]
-             ]}
-        },
-        %{
-          id: :cachex_scrubber,
-          start:
-            {Cachex, :start_link,
-             [
-               :scrubber_cache,
-               [
-                 limit: 2500
-               ]
-             ]}
-        },
-        %{
-          id: :cachex_idem,
-          start:
-            {Cachex, :start_link,
-             [
-               :idempotency_cache,
-               [
-                 expiration:
-                   expiration(
-                     default: :timer.seconds(6 * 60 * 60),
-                     interval: :timer.seconds(60)
-                   ),
-                 limit: 2500
-               ]
-             ]}
-        },
-        %{id: Pleroma.FlakeId, start: {Pleroma.FlakeId, :start_link, []}},
-        %{
-          id: Pleroma.ScheduledActivityWorker,
-          start: {Pleroma.ScheduledActivityWorker, :start_link, []}
-        },
-        %{
-          id: Pleroma.ActivityExpirationWorker,
-          start: {Pleroma.ActivityExpirationWorker, :start_link, []}
-        }
       ] ++
+        cachex_children() ++
         hackney_pool_children() ++
         [
-          %{
+          Pleroma.Web.Federator.RetryQueue,
-            id: Pleroma.Web.Federator.RetryQueue,
+          Pleroma.Stats,
-            start: {Pleroma.Web.Federator.RetryQueue, :start_link, []}
-          },
-          %{
-            id: Pleroma.Web.OAuth.Token.CleanWorker,
-            start: {Pleroma.Web.OAuth.Token.CleanWorker, :start_link, []}
-          },
-          %{
-            id: Pleroma.Stats,
-            start: {Pleroma.Stats, :start_link, []}
-          },
           %{
             id: :web_push_init,
             start: {Task, :start_link, [&Pleroma.Web.Push.init/0]},
@@ -151,22 +59,20 @@ defmodule Pleroma.Application do
             restart: :temporary
           }
         ] ++
-        streamer_child() ++
+        oauth_cleanup_child(oauth_cleanup_enabled?()) ++
-        chat_child() ++
+        streamer_child(@env) ++
+        chat_child(@env, chat_enabled?()) ++
         [
-          # Start the endpoint when the application starts
+          Pleroma.Web.Endpoint,
-          %{
+          Pleroma.Gopher.Server
-            id: Pleroma.Web.Endpoint,
-            start: {Pleroma.Web.Endpoint, :start_link, []},
-            type: :supervisor
-          },
-          %{id: Pleroma.Gopher.Server, start: {Pleroma.Gopher.Server, :start_link, []}}
         ]
 
     # See http://elixir-lang.org/docs/stable/elixir/Supervisor.html
     # for other strategies and supported options
     opts = [strategy: :one_for_one, name: Pleroma.Supervisor]
-    Supervisor.start_link(children, opts)
+    result = Supervisor.start_link(children, opts)
+    :ok = after_supervisor_start()
+    result
   end
 
   defp setup_instrumenters do
@@ -203,32 +109,71 @@ defmodule Pleroma.Application do
       end
   end
 
-  if Pleroma.Config.get(:env) == :test do
+  defp cachex_children do
-    defp streamer_child, do: []
+    [
-    defp chat_child, do: []
+      build_cachex("used_captcha", ttl_interval: seconds_valid_interval()),
-  else
+      build_cachex("user", default_ttl: 25_000, ttl_interval: 1000, limit: 2500),
-    defp streamer_child do
+      build_cachex("object", default_ttl: 25_000, ttl_interval: 1000, limit: 2500),
-      [%{id: Pleroma.Web.Streamer, start: {Pleroma.Web.Streamer, :start_link, []}}]
+      build_cachex("rich_media", default_ttl: :timer.minutes(120), limit: 5000),
+      build_cachex("scrubber", limit: 2500),
+      build_cachex("idempotency", expiration: idempotency_expiration(), limit: 2500)
+    ]
   end
 
-    defp chat_child do
+  defp idempotency_expiration,
-      if Pleroma.Config.get([:chat, :enabled]) do
+    do: expiration(default: :timer.seconds(6 * 60 * 60), interval: :timer.seconds(60))
-        [
+
-          %{
+  defp seconds_valid_interval,
-            id: Pleroma.Web.ChatChannel.ChatChannelState,
+    do: :timer.seconds(Pleroma.Config.get!([Pleroma.Captcha, :seconds_valid]))
-            start: {Pleroma.Web.ChatChannel.ChatChannelState, :start_link, []}
+
+  defp build_cachex(type, opts),
+    do: %{
+      id: String.to_atom("cachex_" <> type),
+      start: {Cachex, :start_link, [String.to_atom(type <> "_cache"), opts]},
+      type: :worker
     }
-        ]
+
-      else
+  defp chat_enabled?, do: Pleroma.Config.get([:chat, :enabled])
-        []
+
-      end
+  defp oauth_cleanup_enabled?,
+    do: Pleroma.Config.get([:oauth2, :clean_expired_tokens], false)
+
+  defp streamer_child(:test), do: []
+
+  defp streamer_child(_) do
+    [Pleroma.Web.Streamer]
   end
+
+  defp oauth_cleanup_child(true),
+    do: [Pleroma.Web.OAuth.Token.CleanWorker]
+
+  defp oauth_cleanup_child(_), do: []
+
+  defp chat_child(:test, _), do: []
+
+  defp chat_child(_env, true) do
+    [Pleroma.Web.ChatChannel.ChatChannelState]
   end
 
+  defp chat_child(_, _), do: []
+
   defp hackney_pool_children do
     for pool <- enabled_hackney_pools() do
       options = Pleroma.Config.get([:hackney_pools, pool])
       :hackney_pool.child_spec(pool, options)
     end
   end
+
+  defp after_supervisor_start do
+    with digest_config <- Application.get_env(:pleroma, :email_notifications)[:digest],
+         true <- digest_config[:active] do
+      PleromaJobQueue.schedule(
+        digest_config[:schedule],
+        :digest_emails,
+        Pleroma.DigestEmailWorker
+      )
+    end
+
+    :ok
+  end
 end

lib/pleroma/captcha/captcha.ex

@@ -12,7 +12,7 @@ defmodule Pleroma.Captcha do
   use GenServer
 
   @doc false
-  def start_link do
+  def start_link(_) do
     GenServer.start_link(__MODULE__, [], name: __MODULE__)
   end
 

lib/pleroma/config/transfer_task.ex

@@ -6,7 +6,7 @@ defmodule Pleroma.Config.TransferTask do
   use Task
   alias Pleroma.Web.AdminAPI.Config
 
-  def start_link do
+  def start_link(_) do
     load_and_update_env()
     if Pleroma.Config.get(:env) == :test, do: Ecto.Adapters.SQL.Sandbox.checkin(Pleroma.Repo)
     :ignore

lib/pleroma/constants.ex

@@ -0,0 +1,9 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Constants do
+  use Const
+
+  const(as_public, do: "https://www.w3.org/ns/activitystreams#Public")
+end

lib/pleroma/conversation.ex

@@ -4,6 +4,7 @@
 
 defmodule Pleroma.Conversation do
   alias Pleroma.Conversation.Participation
+  alias Pleroma.Conversation.Participation.RecipientShip
   alias Pleroma.Repo
   alias Pleroma.User
   use Ecto.Schema
@@ -39,6 +40,15 @@ defmodule Pleroma.Conversation do
     Repo.get_by(__MODULE__, ap_id: ap_id)
   end
 
+  def maybe_create_recipientships(participation, activity) do
+    participation = Repo.preload(participation, :recipients)
+
+    if participation.recipients |> Enum.empty?() do
+      recipients = User.get_all_by_ap_id(activity.recipients)
+      RecipientShip.create(recipients, participation)
+    end
+  end
+
   @doc """
   This will
   1. Create a conversation if there isn't one already
@@ -60,6 +70,7 @@ defmodule Pleroma.Conversation do
           {:ok, participation} =
             Participation.create_for_user_and_conversation(user, conversation, opts)
 
+          maybe_create_recipientships(participation, activity)
           participation
         end)
 

lib/pleroma/conversation/participation.ex

@@ -5,6 +5,7 @@
 defmodule Pleroma.Conversation.Participation do
   use Ecto.Schema
   alias Pleroma.Conversation
+  alias Pleroma.Conversation.Participation.RecipientShip
   alias Pleroma.Repo
   alias Pleroma.User
   alias Pleroma.Web.ActivityPub.ActivityPub
@@ -17,6 +18,9 @@ defmodule Pleroma.Conversation.Participation do
     field(:read, :boolean, default: false)
     field(:last_activity_id, Pleroma.FlakeId, virtual: true)
 
+    has_many(:recipient_ships, RecipientShip)
+    has_many(:recipients, through: [:recipient_ships, :user])
+
     timestamps()
   end
 
@@ -65,6 +69,14 @@ defmodule Pleroma.Conversation.Participation do
     |> Pleroma.Pagination.fetch_paginated(params)
   end
 
+  def for_user_and_conversation(user, conversation) do
+    from(p in __MODULE__,
+      where: p.user_id == ^user.id,
+      where: p.conversation_id == ^conversation.id
+    )
+    |> Repo.one()
+  end
+
   def for_user_with_last_activity_id(user, params \\ %{}) do
     for_user(user, params)
     |> Enum.map(fn participation ->
@@ -81,4 +93,46 @@ defmodule Pleroma.Conversation.Participation do
     end)
     |> Enum.filter(& &1.last_activity_id)
   end
+
+  def get(_, _ \\ [])
+  def get(nil, _), do: nil
+
+  def get(id, params) do
+    query =
+      if preload = params[:preload] do
+        from(p in __MODULE__,
+          preload: ^preload
+        )
+      else
+        __MODULE__
+      end
+
+    Repo.get(query, id)
+  end
+
+  def set_recipients(participation, user_ids) do
+    user_ids =
+      [participation.user_id | user_ids]
+      |> Enum.uniq()
+
+    Repo.transaction(fn ->
+      query =
+        from(r in RecipientShip,
+          where: r.participation_id == ^participation.id
+        )
+
+      Repo.delete_all(query)
+
+      users =
+        from(u in User,
+          where: u.id in ^user_ids
+        )
+        |> Repo.all()
+
+      RecipientShip.create(users, participation)
+      :ok
+    end)
+
+    {:ok, Repo.preload(participation, :recipients, force: true)}
+  end
 end

lib/pleroma/conversation/participation_recipient_ship.ex

@@ -0,0 +1,34 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Conversation.Participation.RecipientShip do
+  use Ecto.Schema
+
+  alias Pleroma.Conversation.Participation
+  alias Pleroma.Repo
+  alias Pleroma.User
+
+  import Ecto.Changeset
+
+  schema "conversation_participation_recipient_ships" do
+    belongs_to(:user, User, type: Pleroma.FlakeId)
+    belongs_to(:participation, Participation)
+  end
+
+  def creation_cng(struct, params) do
+    struct
+    |> cast(params, [:user_id, :participation_id])
+    |> validate_required([:user_id, :participation_id])
+  end
+
+  def create(%User{} = user, participation), do: create([user], participation)
+
+  def create(users, participation) do
+    Enum.each(users, fn user ->
+      %__MODULE__{}
+      |> creation_cng(%{user_id: user.id, participation_id: participation.id})
+      |> Repo.insert!()
+    end)
+  end
+end

lib/pleroma/digest_email_worker.ex

@@ -0,0 +1,39 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.DigestEmailWorker do
+  import Ecto.Query
+
+  @queue_name :digest_emails
+
+  def perform do
+    config = Pleroma.Config.get([:email_notifications, :digest])
+    negative_interval = -Map.fetch!(config, :interval)
+    inactivity_threshold = Map.fetch!(config, :inactivity_threshold)
+    inactive_users_query = Pleroma.User.list_inactive_users_query(inactivity_threshold)
+
+    now = NaiveDateTime.truncate(NaiveDateTime.utc_now(), :second)
+
+    from(u in inactive_users_query,
+      where: fragment(~s(? #> '{"email_notifications","digest"}' @> 'true'), u.info),
+      where: u.last_digest_emailed_at < datetime_add(^now, ^negative_interval, "day"),
+      select: u
+    )
+    |> Pleroma.Repo.all()
+    |> Enum.each(&PleromaJobQueue.enqueue(@queue_name, __MODULE__, [&1]))
+  end
+
+  @doc """
+  Send digest email to the given user.
+  Updates `last_digest_emailed_at` field for the user and returns the updated user.
+  """
+  @spec perform(Pleroma.User.t()) :: Pleroma.User.t()
+  def perform(user) do
+    with %Swoosh.Email{} = email <- Pleroma.Emails.UserEmail.digest_email(user) do
+      Pleroma.Emails.Mailer.deliver_async(email)
+    end
+
+    Pleroma.User.touch_last_digest_emailed_at(user)
+  end
+end

lib/pleroma/emails/admin_email.ex

@@ -63,7 +63,6 @@ defmodule Pleroma.Emails.AdminEmail do
     new()
     |> to({to.name, to.email})
     |> from({instance_name(), instance_notify_email()})
-    |> reply_to({reporter.name, reporter.email})
     |> subject("#{instance_name()} Report")
     |> html_body(html_body)
   end

lib/pleroma/emails/user_email.ex

@@ -5,23 +5,23 @@
 defmodule Pleroma.Emails.UserEmail do
   @moduledoc "User emails"
 
-  import Swoosh.Email
+  use Phoenix.Swoosh, view: Pleroma.Web.EmailView, layout: {Pleroma.Web.LayoutView, :email}
 
+  alias Pleroma.Config
+  alias Pleroma.User
   alias Pleroma.Web.Endpoint
   alias Pleroma.Web.Router
 
-  defp instance_config, do: Pleroma.Config.get(:instance)
+  defp instance_name, do: Config.get([:instance, :name])
-
-  defp instance_name, do: instance_config()[:name]
 
   defp sender do
-    email = Keyword.get(instance_config(), :notify_email, instance_config()[:email])
+    email = Config.get([:instance, :notify_email]) || Config.get([:instance, :email])
     {instance_name(), email}
   end
 
   defp recipient(email, nil), do: email
   defp recipient(email, name), do: {name, email}
-  defp recipient(%Pleroma.User{} = user), do: recipient(user.email, user.name)
+  defp recipient(%User{} = user), do: recipient(user.email, user.name)
 
   def password_reset_email(user, token) when is_binary(token) do
     password_reset_url = Router.Helpers.reset_password_url(Endpoint, :reset, token)
@@ -87,4 +87,92 @@ defmodule Pleroma.Emails.UserEmail do
     |> subject("#{instance_name()} account confirmation")
     |> html_body(html_body)
   end
+
+  @doc """
+  Email used in digest email notifications
+  Includes Mentions and New Followers data
+  If there are no mentions (even when new followers exist), the function will return nil
+  """
+  @spec digest_email(User.t()) :: Swoosh.Email.t() | nil
+  def digest_email(user) do
+    notifications = Pleroma.Notification.for_user_since(user, user.last_digest_emailed_at)
+
+    mentions =
+      notifications
+      |> Enum.filter(&(&1.activity.data["type"] == "Create"))
+      |> Enum.map(fn notification ->
+        object = Pleroma.Object.normalize(notification.activity)
+        object = update_in(object.data["content"], &format_links/1)
+
+        %{
+          data: notification,
+          object: object,
+          from: User.get_by_ap_id(notification.activity.actor)
+        }
+      end)
+
+    followers =
+      notifications
+      |> Enum.filter(&(&1.activity.data["type"] == "Follow"))
+      |> Enum.map(fn notification ->
+        %{
+          data: notification,
+          object: Pleroma.Object.normalize(notification.activity),
+          from: User.get_by_ap_id(notification.activity.actor)
+        }
+      end)
+
+    unless Enum.empty?(mentions) do
+      styling = Config.get([__MODULE__, :styling])
+      logo = Config.get([__MODULE__, :logo])
+
+      html_data = %{
+        instance: instance_name(),
+        user: user,
+        mentions: mentions,
+        followers: followers,
+        unsubscribe_link: unsubscribe_url(user, "digest"),
+        styling: styling
+      }
+
+      logo_path =
+        if is_nil(logo) do
+          Path.join(:code.priv_dir(:pleroma), "static/static/logo.png")
+        else
+          Path.join(Config.get([:instance, :static_dir]), logo)
+        end
+
+      new()
+      |> to(recipient(user))
+      |> from(sender())
+      |> subject("Your digest from #{instance_name()}")
+      |> put_layout(false)
+      |> render_body("digest.html", html_data)
+      |> attachment(Swoosh.Attachment.new(logo_path, filename: "logo.png", type: :inline))
+    end
+  end
+
+  defp format_links(str) do
+    re = ~r/<a.+href=['"].*>/iU
+    %{link_color: color} = Config.get([__MODULE__, :styling])
+
+    Regex.replace(re, str, fn link ->
+      String.replace(link, "<a", "<a style=\"color: #{color};text-decoration: none;\"")
+    end)
+  end
+
+  @doc """
+  Generate unsubscribe link for given user and notifications type.
+  The link contains JWT token with the data, and subscription can be modified without
+  authorization.
+  """
+  @spec unsubscribe_url(User.t(), String.t()) :: String.t()
+  def unsubscribe_url(user, notifications_type) do
+    token =
+      %{"sub" => user.id, "act" => %{"unsubscribe" => notifications_type}, "exp" => false}
+      |> Pleroma.JWT.generate_and_sign!()
+      |> Base.encode64()
+
+    Router.Helpers.subscription_url(Endpoint, :unsubscribe, token)
+  end
 end

lib/pleroma/emoji.ex

@@ -24,7 +24,7 @@ defmodule Pleroma.Emoji do
   @ets_options [:ordered_set, :protected, :named_table, {:read_concurrency, true}]
 
   @doc false
-  def start_link do
+  def start_link(_) do
     GenServer.start_link(__MODULE__, [], name: __MODULE__)
   end
 

lib/pleroma/flake_id.ex

@@ -66,6 +66,16 @@ defmodule Pleroma.FlakeId do
   @spec get :: binary
   def get, do: to_string(:gen_server.call(:flake, :get))
 
+  # checks that ID is is valid FlakeID
+  #
+  @spec is_flake_id?(String.t()) :: boolean
+  def is_flake_id?(id), do: is_flake_id?(String.to_charlist(id), true)
+  defp is_flake_id?([c | cs], true) when c >= ?0 and c <= ?9, do: is_flake_id?(cs, true)
+  defp is_flake_id?([c | cs], true) when c >= ?A and c <= ?Z, do: is_flake_id?(cs, true)
+  defp is_flake_id?([c | cs], true) when c >= ?a and c <= ?z, do: is_flake_id?(cs, true)
+  defp is_flake_id?([], true), do: true
+  defp is_flake_id?(_, _), do: false
+
   # -- Ecto.Type API
   @impl Ecto.Type
   def type, do: :uuid
@@ -88,7 +98,7 @@ defmodule Pleroma.FlakeId do
   def autogenerate, do: get()
 
   # -- GenServer API
-  def start_link do
+  def start_link(_) do
     :gen_server.start_link({:local, :flake}, __MODULE__, [], [])
   end
 

lib/pleroma/gopher/server.ex

@@ -6,7 +6,7 @@ defmodule Pleroma.Gopher.Server do
   use GenServer
   require Logger
 
-  def start_link do
+  def start_link(_) do
     config = Pleroma.Config.get(:gopher, [])
     ip = Keyword.get(config, :ip, {0, 0, 0, 0})
     port = Keyword.get(config, :port, 1234)

lib/pleroma/html.ex

@@ -203,6 +203,8 @@ defmodule Pleroma.HTML.Scrubber.Default do
   Meta.allow_tag_with_these_attributes("p", [])
   Meta.allow_tag_with_these_attributes("pre", [])
   Meta.allow_tag_with_these_attributes("strong", [])
+  Meta.allow_tag_with_these_attributes("sub", [])
+  Meta.allow_tag_with_these_attributes("sup", [])
   Meta.allow_tag_with_these_attributes("u", [])
   Meta.allow_tag_with_these_attributes("ul", [])
 
@@ -280,3 +282,31 @@ defmodule Pleroma.HTML.Transform.MediaProxy do
   def scrub({_tag, children}), do: children
   def scrub(text), do: text
 end
+
+defmodule Pleroma.HTML.Scrubber.LinksOnly do
+  @moduledoc """
+  An HTML scrubbing policy which limits to links only.
+  """
+
+  @valid_schemes Pleroma.Config.get([:uri_schemes, :valid_schemes], [])
+
+  require HtmlSanitizeEx.Scrubber.Meta
+  alias HtmlSanitizeEx.Scrubber.Meta
+
+  Meta.remove_cdata_sections_before_scrub()
+  Meta.strip_comments()
+
+  # links
+  Meta.allow_tag_with_uri_attributes("a", ["href"], @valid_schemes)
+
+  Meta.allow_tag_with_this_attribute_values("a", "rel", [
+    "tag",
+    "nofollow",
+    "noopener",
+    "noreferrer",
+    "me"
+  ])
+
+  Meta.allow_tag_with_these_attributes("a", ["name", "title"])
+  Meta.strip_everything_not_covered()
+end

lib/pleroma/http/connection.ex

@@ -11,6 +11,7 @@ defmodule Pleroma.HTTP.Connection do
     connect_timeout: 10_000,
     recv_timeout: 20_000,
     follow_redirect: true,
+    force_redirect: true,
     pool: :federation
   ]
   @adapter Application.get_env(:tesla, :adapter)

lib/pleroma/jwt.ex

@@ -0,0 +1,9 @@
+defmodule Pleroma.JWT do
+  use Joken.Config
+
+  @impl true
+  def token_config do
+    default_claims(skip: [:aud])
+    |> add_claim("aud", &Pleroma.Web.Endpoint.url/0, &(&1 == Pleroma.Web.Endpoint.url()))
+  end
+end

lib/pleroma/notification.ex

@@ -18,6 +18,8 @@ defmodule Pleroma.Notification do
   import Ecto.Query
   import Ecto.Changeset
 
+  @type t :: %__MODULE__{}
+
   schema "notifications" do
     field(:seen, :boolean, default: false)
     belongs_to(:user, User, type: Pleroma.FlakeId)
@@ -31,7 +33,7 @@ defmodule Pleroma.Notification do
     |> cast(attrs, [:seen])
   end
 
-  def for_user_query(user, opts) do
+  def for_user_query(user, opts \\ []) do
     query =
       Notification
       |> where(user_id: ^user.id)
@@ -75,6 +77,25 @@ defmodule Pleroma.Notification do
     |> Pagination.fetch_paginated(opts)
   end
 
+  @doc """
+  Returns notifications for user received since given date.
+
+  ## Examples
+
+      iex> Pleroma.Notification.for_user_since(%Pleroma.User{}, ~N[2019-04-13 11:22:33])
+      [%Pleroma.Notification{}, %Pleroma.Notification{}]
+
+      iex> Pleroma.Notification.for_user_since(%Pleroma.User{}, ~N[2019-04-15 11:22:33])
+      []
+  """
+  @spec for_user_since(Pleroma.User.t(), NaiveDateTime.t()) :: [t()]
+  def for_user_since(user, date) do
+    from(n in for_user_query(user),
+      where: n.updated_at > ^date
+    )
+    |> Repo.all()
+  end
+
   def set_read_up_to(%{id: user_id} = _user, id) do
     query =
       from(
@@ -82,7 +103,10 @@ defmodule Pleroma.Notification do
         where: n.user_id == ^user_id,
         where: n.id <= ^id,
         update: [
-          set: [seen: true]
+          set: [
+            seen: true,
+            updated_at: ^NaiveDateTime.utc_now()
+          ]
         ]
       )
 

lib/pleroma/object/fetcher.ex

@@ -114,7 +114,7 @@ defmodule Pleroma.Object.Fetcher do
     end
   end
 
-  def fetch_and_contain_remote_object_from_id(id) do
+  def fetch_and_contain_remote_object_from_id(id) when is_binary(id) do
     Logger.info("Fetching object #{id} via AP")
 
     date =
@@ -141,4 +141,9 @@ defmodule Pleroma.Object.Fetcher do
         {:error, e}
     end
   end
+
+  def fetch_and_contain_remote_object_from_id(%{"id" => id}),
+    do: fetch_and_contain_remote_object_from_id(id)
+
+  def fetch_and_contain_remote_object_from_id(_id), do: {:error, "id must be a string"}
 end

lib/pleroma/plugs/set_format_plug.ex

@@ -0,0 +1,24 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Plugs.SetFormatPlug do
+  import Plug.Conn, only: [assign: 3, fetch_query_params: 1]
+
+  def init(_), do: nil
+
+  def call(conn, _) do
+    case get_format(conn) do
+      nil -> conn
+      format -> assign(conn, :format, format)
+    end
+  end
+
+  defp get_format(conn) do
+    conn.private[:phoenix_format] ||
+      case fetch_query_params(conn) do
+        %{query_params: %{"_format" => format}} -> format
+        _ -> nil
+      end
+  end
+end

lib/pleroma/reverse_proxy/reverse_proxy.ex

@@ -109,7 +109,11 @@ defmodule Pleroma.ReverseProxy do
       end
 
     with {:ok, code, headers, client} <- request(method, url, req_headers, hackney_opts),
-         :ok <- header_length_constraint(headers, Keyword.get(opts, :max_body_length)) do
+         :ok <-
+           header_length_constraint(
+             headers,
+             Keyword.get(opts, :max_body_length, @max_body_length)
+           ) do
       response(conn, client, url, code, headers, opts)
     else
       {:ok, code, headers} ->
@@ -200,7 +204,11 @@ defmodule Pleroma.ReverseProxy do
          {:ok, data} <- client().stream_body(client),
          {:ok, duration} <- increase_read_duration(duration),
          sent_so_far = sent_so_far + byte_size(data),
-         :ok <- body_size_constraint(sent_so_far, Keyword.get(opts, :max_body_size)),
+         :ok <-
+           body_size_constraint(
+             sent_so_far,
+             Keyword.get(opts, :max_body_length, @max_body_length)
+           ),
          {:ok, conn} <- chunk(conn, data) do
       chunk_reply(conn, client, opts, sent_so_far, duration)
     else

lib/pleroma/scheduled_activity_worker.ex

@@ -16,7 +16,7 @@ defmodule Pleroma.ScheduledActivityWorker do
 
   @schedule_interval :timer.minutes(1)
 
-  def start_link do
+  def start_link(_) do
     GenServer.start_link(__MODULE__, nil)
   end
 

lib/pleroma/signature.ex

@@ -15,7 +15,7 @@ defmodule Pleroma.Signature do
       |> Map.put(:fragment, nil)
 
     uri =
-      if String.ends_with?(uri.path, "/publickey") do
+      if not is_nil(uri.path) and String.ends_with?(uri.path, "/publickey") do
         Map.put(uri, :path, String.replace(uri.path, "/publickey", ""))
       else
         uri

lib/pleroma/stats.ex

@@ -7,31 +7,56 @@ defmodule Pleroma.Stats do
   alias Pleroma.Repo
   alias Pleroma.User
 
-  def start_link do
+  use GenServer
-    agent = Agent.start_link(fn -> {[], %{}} end, name: __MODULE__)
+
-    spawn(fn -> schedule_update() end)
+  @interval 1000 * 60 * 60
-    agent
+
+  def start_link(_) do
+    GenServer.start_link(__MODULE__, initial_data(), name: __MODULE__)
+  end
+
+  def force_update do
+    GenServer.call(__MODULE__, :force_update)
   end
 
   def get_stats do
-    Agent.get(__MODULE__, fn {_, stats} -> stats end)
+    %{stats: stats} = GenServer.call(__MODULE__, :get_state)
+
+    stats
   end
 
   def get_peers do
-    Agent.get(__MODULE__, fn {peers, _} -> peers end)
+    %{peers: peers} = GenServer.call(__MODULE__, :get_state)
+
+    peers
   end
 
-  def schedule_update do
+  def init(args) do
-    spawn(fn ->
+    Process.send(self(), :run_update, [])
-      # 1 hour
+    {:ok, args}
-      Process.sleep(1000 * 60 * 60)
-      schedule_update()
-    end)
-
-    update_stats()
   end
 
-  def update_stats do
+  def handle_call(:force_update, _from, _state) do
+    new_stats = get_stat_data()
+    {:reply, new_stats, new_stats}
+  end
+
+  def handle_call(:get_state, _from, state) do
+    {:reply, state, state}
+  end
+
+  def handle_info(:run_update, _state) do
+    new_stats = get_stat_data()
+
+    Process.send_after(self(), :run_update, @interval)
+    {:noreply, new_stats}
+  end
+
+  defp initial_data do
+    %{peers: [], stats: %{}}
+  end
+
+  defp get_stat_data do
     peers =
       from(
         u in User,
@@ -52,8 +77,9 @@ defmodule Pleroma.Stats do
 
     user_count = Repo.aggregate(User.Query.build(%{local: true, active: true}), :count, :id)
 
-    Agent.update(__MODULE__, fn _ ->
+    %{
-      {peers, %{domain_count: domain_count, status_count: status_count, user_count: user_count}}
+      peers: peers,
-    end)
+      stats: %{domain_count: domain_count, status_count: status_count, user_count: user_count}
+    }
   end
 end

lib/pleroma/upload.ex

@@ -228,7 +228,14 @@ defmodule Pleroma.Upload do
           ""
         end
 
-    [base_url, "media", path]
+    prefix =
+      if is_nil(Pleroma.Config.get([__MODULE__, :base_url])) do
+        "media"
+      else
+        ""
+      end
+
+    [base_url, prefix, path]
     |> Path.join()
   end
 

lib/pleroma/uploaders/local.ex

@@ -11,7 +11,7 @@ defmodule Pleroma.Uploaders.Local do
 
   def put_file(upload) do
     {local_path, file} =
-      case Enum.reverse(String.split(upload.path, "/", trim: true)) do
+      case Enum.reverse(Path.split(upload.path)) do
         [file] ->
           {upload_path(), file}
 
@@ -23,7 +23,7 @@ defmodule Pleroma.Uploaders.Local do
 
     result_file = Path.join(local_path, file)
 
-    unless File.exists?(result_file) do
+    if not File.exists?(result_file) do
       File.cp!(upload.tempfile, result_file)
     end
 

lib/pleroma/uploaders/mdii.ex

@@ -3,6 +3,8 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Uploaders.MDII do
+  @moduledoc "Represents uploader for https://github.com/hakaba-hitoyo/minimal-digital-image-infrastructure"
+
   alias Pleroma.Config
   alias Pleroma.HTTP
 

lib/pleroma/uploaders/s3.ex

@@ -6,10 +6,12 @@ defmodule Pleroma.Uploaders.S3 do
   @behaviour Pleroma.Uploaders.Uploader
   require Logger
 
+  alias Pleroma.Config
+
   # The file name is re-encoded with S3's constraints here to comply with previous
   # links with less strict filenames
   def get_file(file) do
-    config = Pleroma.Config.get([__MODULE__])
+    config = Config.get([__MODULE__])
     bucket = Keyword.fetch!(config, :bucket)
 
     bucket_with_namespace =
@@ -34,15 +36,15 @@ defmodule Pleroma.Uploaders.S3 do
   end
 
   def put_file(%Pleroma.Upload{} = upload) do
-    config = Pleroma.Config.get([__MODULE__])
+    config = Config.get([__MODULE__])
     bucket = Keyword.get(config, :bucket)
 
-    {:ok, file_data} = File.read(upload.tempfile)
-
     s3_name = strict_encode(upload.path)
 
     op =
-      ExAws.S3.put_object(bucket, s3_name, file_data, [
+      upload.tempfile
+      |> ExAws.S3.Upload.stream_file()
+      |> ExAws.S3.upload(bucket, s3_name, [
         {:acl, :public_read},
         {:content_type, upload.content_type}
       ])

lib/pleroma/user.ex

@@ -21,6 +21,7 @@ defmodule Pleroma.User do
   alias Pleroma.Web
   alias Pleroma.Web.ActivityPub.ActivityPub
   alias Pleroma.Web.ActivityPub.Utils
+  alias Pleroma.Web.CommonAPI
   alias Pleroma.Web.CommonAPI.Utils, as: CommonUtils
   alias Pleroma.Web.OAuth
   alias Pleroma.Web.OStatus
@@ -57,6 +58,7 @@ defmodule Pleroma.User do
     field(:search_type, :integer, virtual: true)
     field(:tags, {:array, :string}, default: [])
     field(:last_refreshed_at, :naive_datetime_usec)
+    field(:last_digest_emailed_at, :naive_datetime)
     has_many(:notifications, Notification)
     has_many(:registrations, Registration)
     embeds_one(:info, User.Info)
@@ -114,7 +116,9 @@ defmodule Pleroma.User do
 
   def user_info(%User{} = user, args \\ %{}) do
     following_count =
-      if args[:following_count], do: args[:following_count], else: following_count(user)
+      if args[:following_count],
+        do: args[:following_count],
+        else: user.info.following_count || following_count(user)
 
     follower_count =
       if args[:follower_count], do: args[:follower_count], else: user.info.follower_count
@@ -129,6 +133,28 @@ defmodule Pleroma.User do
     |> Map.put(:follower_count, follower_count)
   end
 
+  def follow_state(%User{} = user, %User{} = target) do
+    follow_activity = Utils.fetch_latest_follow(user, target)
+
+    if follow_activity,
+      do: follow_activity.data["state"],
+      # Ideally this would be nil, but then Cachex does not commit the value
+      else: false
+  end
+
+  def get_cached_follow_state(user, target) do
+    key = "follow_state:#{user.ap_id}|#{target.ap_id}"
+    Cachex.fetch!(:user_cache, key, fn _ -> {:commit, follow_state(user, target)} end)
+  end
+
+  def set_follow_state_cache(user_ap_id, target_ap_id, state) do
+    Cachex.put(
+      :user_cache,
+      "follow_state:#{user_ap_id}|#{target_ap_id}",
+      state
+    )
+  end
+
   def set_info_cache(user, args) do
     Cachex.put(:user_cache, "user_info:#{user.id}", user_info(user, args))
   end
@@ -149,10 +175,10 @@ defmodule Pleroma.User do
   end
 
   def remote_user_creation(params) do
-    params =
+    bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
-      params
+    name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
-      |> Map.put(:info, params[:info] || %{})
 
+    params = Map.put(params, :info, params[:info] || %{})
     info_cng = User.Info.remote_user_creation(%User.Info{}, params[:info])
 
     changes =
@@ -161,8 +187,8 @@ defmodule Pleroma.User do
       |> validate_required([:name, :ap_id])
       |> unique_constraint(:nickname)
       |> validate_format(:nickname, @email_regex)
-      |> validate_length(:bio, max: 5000)
+      |> validate_length(:bio, max: bio_limit)
-      |> validate_length(:name, max: 100)
+      |> validate_length(:name, max: name_limit)
       |> put_change(:local, false)
       |> put_embed(:info, info_cng)
 
@@ -185,22 +211,23 @@ defmodule Pleroma.User do
   end
 
   def update_changeset(struct, params \\ %{}) do
+    bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
+    name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
+
     struct
     |> cast(params, [:bio, :name, :avatar, :following])
     |> unique_constraint(:nickname)
     |> validate_format(:nickname, local_nickname_regex())
-    |> validate_length(:bio, max: 5000)
+    |> validate_length(:bio, max: bio_limit)
-    |> validate_length(:name, min: 1, max: 100)
+    |> validate_length(:name, min: 1, max: name_limit)
   end
 
-  def upgrade_changeset(struct, params \\ %{}) do
+  def upgrade_changeset(struct, params \\ %{}, remote? \\ false) do
-    params =
+    bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
-      params
+    name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
-      |> Map.put(:last_refreshed_at, NaiveDateTime.utc_now())
 
-    info_cng =
+    params = Map.put(params, :last_refreshed_at, NaiveDateTime.utc_now())
-      struct.info
+    info_cng = User.Info.user_upgrade(struct.info, params[:info], remote?)
-      |> User.Info.user_upgrade(params[:info])
 
     struct
     |> cast(params, [
@@ -213,8 +240,8 @@ defmodule Pleroma.User do
     ])
     |> unique_constraint(:nickname)
     |> validate_format(:nickname, local_nickname_regex())
-    |> validate_length(:bio, max: 5000)
+    |> validate_length(:bio, max: bio_limit)
-    |> validate_length(:name, max: 100)
+    |> validate_length(:name, max: name_limit)
     |> put_embed(:info, info_cng)
   end
 
@@ -226,6 +253,7 @@ defmodule Pleroma.User do
     |> put_password_hash
   end
 
+  @spec reset_password(User.t(), map) :: {:ok, User.t()} | {:error, Ecto.Changeset.t()}
   def reset_password(%User{id: user_id} = user, data) do
     multi =
       Multi.new()
@@ -240,6 +268,9 @@ defmodule Pleroma.User do
   end
 
   def register_changeset(struct, params \\ %{}, opts \\ []) do
+    bio_limit = Pleroma.Config.get([:instance, :user_bio_length], 5000)
+    name_limit = Pleroma.Config.get([:instance, :user_name_length], 100)
+
     need_confirmation? =
       if is_nil(opts[:need_confirmation]) do
         Pleroma.Config.get([:instance, :account_activation_required])
@@ -260,8 +291,8 @@ defmodule Pleroma.User do
       |> validate_exclusion(:nickname, Pleroma.Config.get([User, :restricted_nicknames]))
       |> validate_format(:nickname, local_nickname_regex())
       |> validate_format(:email, @email_regex)
-      |> validate_length(:bio, max: 1000)
+      |> validate_length(:bio, max: bio_limit)
-      |> validate_length(:name, min: 1, max: 100)
+      |> validate_length(:name, min: 1, max: name_limit)
       |> put_change(:info, info_change)
 
     changeset =
@@ -330,6 +361,7 @@ defmodule Pleroma.User do
 
   def needs_update?(_), do: true
 
+  @spec maybe_direct_follow(User.t(), User.t()) :: {:ok, User.t()} | {:error, String.t()}
   def maybe_direct_follow(%User{} = follower, %User{local: true, info: %{locked: true}}) do
     {:ok, follower}
   end
@@ -404,6 +436,8 @@ defmodule Pleroma.User do
 
         {1, [follower]} = Repo.update_all(q, [])
 
+        follower = maybe_update_following_count(follower)
+
         {:ok, _} = update_follower_count(followed)
 
         set_cache(follower)
@@ -423,6 +457,8 @@ defmodule Pleroma.User do
 
       {1, [follower]} = Repo.update_all(q, [])
 
+      follower = maybe_update_following_count(follower)
+
       {:ok, followed} = update_follower_count(followed)
 
       set_cache(follower)
@@ -450,6 +486,13 @@ defmodule Pleroma.User do
     Repo.get_by(User, ap_id: ap_id)
   end
 
+  def get_all_by_ap_id(ap_ids) do
+    from(u in __MODULE__,
+      where: u.ap_id in ^ap_ids
+    )
+    |> Repo.all()
+  end
+
   # This is mostly an SPC migration fix. This guesses the user nickname by taking the last part
   # of the ap_id and the domain and tries to get that user
   def get_by_guessed_nickname(ap_id) do
@@ -471,7 +514,7 @@ defmodule Pleroma.User do
   end
 
   def update_and_set_cache(changeset) do
-    with {:ok, user} <- Repo.update(changeset) do
+    with {:ok, user} <- Repo.update(changeset, stale_error_field: :id) do
       set_cache(user)
     else
       e -> e
@@ -707,7 +750,36 @@ defmodule Pleroma.User do
     |> update_and_set_cache()
   end
 
+  @spec maybe_fetch_follow_information(User.t()) :: User.t()
+  def maybe_fetch_follow_information(user) do
+    with {:ok, user} <- fetch_follow_information(user) do
+      user
+    else
+      e ->
+        Logger.error("Follower/Following counter update for #{user.ap_id} failed.\n#{inspect(e)}")
+
+        user
+    end
+  end
+
+  def fetch_follow_information(user) do
+    with {:ok, info} <- ActivityPub.fetch_follow_information_for_user(user) do
+      info_cng = User.Info.follow_information_update(user.info, info)
+
+      changeset =
+        user
+        |> change()
+        |> put_embed(:info, info_cng)
+
+      update_and_set_cache(changeset)
+    else
+      {:error, _} = e -> e
+      e -> {:error, e}
+    end
+  end
+
   def update_follower_count(%User{} = user) do
+    if user.local or !Pleroma.Config.get([:instance, :external_user_synchronization]) do
       follower_count_query =
         User.Query.build(%{followers: user, deactivated: false})
         |> select([u], %{count: count(u.id)})
@@ -731,7 +803,21 @@ defmodule Pleroma.User do
         {1, [user]} -> set_cache(user)
         _ -> {:error, user}
       end
+    else
+      {:ok, maybe_fetch_follow_information(user)}
     end
+  end
+
+  @spec maybe_update_following_count(User.t()) :: User.t()
+  def maybe_update_following_count(%User{local: false} = user) do
+    if Pleroma.Config.get([:instance, :external_user_synchronization]) do
+      maybe_fetch_follow_information(user)
+    else
+      user
+    end
+  end
+
+  def maybe_update_following_count(user), do: user
 
   def remove_duplicated_following(%User{following: following} = user) do
     uniq_following = Enum.uniq(following)
@@ -831,6 +917,13 @@ defmodule Pleroma.User do
         blocker
       end
 
+    # clear any requested follows as well
+    blocked =
+      case CommonAPI.reject_follow_request(blocked, blocker) do
+        {:ok, %User{} = updated_blocked} -> updated_blocked
+        nil -> blocked
+      end
+
     blocker =
       if subscribed_to?(blocked, blocker) do
         {:ok, blocker} = unsubscribe(blocked, blocker)
@@ -882,19 +975,26 @@ defmodule Pleroma.User do
   def muted_notifications?(user, %{ap_id: ap_id}),
     do: Enum.member?(user.info.muted_notifications, ap_id)
 
-  def blocks?(%User{info: info} = _user, %{ap_id: ap_id}) do
+  def blocks?(%User{} = user, %User{} = target) do
-    blocks = info.blocks
+    blocks_ap_id?(user, target) || blocks_domain?(user, target)
-
-    domain_blocks = Pleroma.Web.ActivityPub.MRF.subdomains_regex(info.domain_blocks)
-
-    %{host: host} = URI.parse(ap_id)
-
-    Enum.member?(blocks, ap_id) ||
-      Pleroma.Web.ActivityPub.MRF.subdomain_match?(domain_blocks, host)
   end
 
   def blocks?(nil, _), do: false
 
+  def blocks_ap_id?(%User{} = user, %User{} = target) do
+    Enum.member?(user.info.blocks, target.ap_id)
+  end
+
+  def blocks_ap_id?(_, _), do: false
+
+  def blocks_domain?(%User{} = user, %User{} = target) do
+    domain_blocks = Pleroma.Web.ActivityPub.MRF.subdomains_regex(user.info.domain_blocks)
+    %{host: host} = URI.parse(target.ap_id)
+    Pleroma.Web.ActivityPub.MRF.subdomain_match?(domain_blocks, host)
+  end
+
+  def blocks_domain?(_, _), do: false
+
   def subscribed_to?(user, %{ap_id: ap_id}) do
     with %User{} = target <- get_cached_by_ap_id(ap_id) do
       Enum.member?(target.info.subscribers, user.ap_id)
@@ -1363,6 +1463,80 @@ defmodule Pleroma.User do
     target.ap_id not in user.info.muted_reblogs
   end
 
+  @doc """
+  The function returns a query to get users with no activity for given interval of days.
+  Inactive users are those who didn't read any notification, or had any activity where
+  the user is the activity's actor, during `inactivity_threshold` days.
+  Deactivated users will not appear in this list.
+
+  ## Examples
+
+      iex> Pleroma.User.list_inactive_users()
+      %Ecto.Query{}
+  """
+  @spec list_inactive_users_query(integer()) :: Ecto.Query.t()
+  def list_inactive_users_query(inactivity_threshold \\ 7) do
+    negative_inactivity_threshold = -inactivity_threshold
+    now = NaiveDateTime.truncate(NaiveDateTime.utc_now(), :second)
+    # Subqueries are not supported in `where` clauses, join gets too complicated.
+    has_read_notifications =
+      from(n in Pleroma.Notification,
+        where: n.seen == true,
+        group_by: n.id,
+        having: max(n.updated_at) > datetime_add(^now, ^negative_inactivity_threshold, "day"),
+        select: n.user_id
+      )
+      |> Pleroma.Repo.all()
+
+    from(u in Pleroma.User,
+      left_join: a in Pleroma.Activity,
+      on: u.ap_id == a.actor,
+      where: not is_nil(u.nickname),
+      where: fragment("not (?->'deactivated' @> 'true')", u.info),
+      where: u.id not in ^has_read_notifications,
+      group_by: u.id,
+      having:
+        max(a.inserted_at) < datetime_add(^now, ^negative_inactivity_threshold, "day") or
+          is_nil(max(a.inserted_at))
+    )
+  end
+
+  @doc """
+  Enable or disable email notifications for user
+
+  ## Examples
+
+      iex> Pleroma.User.switch_email_notifications(Pleroma.User{info: %{email_notifications: %{"digest" => false}}}, "digest", true)
+      Pleroma.User{info: %{email_notifications: %{"digest" => true}}}
+
+      iex> Pleroma.User.switch_email_notifications(Pleroma.User{info: %{email_notifications: %{"digest" => true}}}, "digest", false)
+      Pleroma.User{info: %{email_notifications: %{"digest" => false}}}
+  """
+  @spec switch_email_notifications(t(), String.t(), boolean()) ::
+          {:ok, t()} | {:error, Ecto.Changeset.t()}
+  def switch_email_notifications(user, type, status) do
+    info = Pleroma.User.Info.update_email_notifications(user.info, %{type => status})
+
+    change(user)
+    |> put_embed(:info, info)
+    |> update_and_set_cache()
+  end
+
+  @doc """
+  Set `last_digest_emailed_at` value for the user to current time
+  """
+  @spec touch_last_digest_emailed_at(t()) :: t()
+  def touch_last_digest_emailed_at(user) do
+    now = NaiveDateTime.truncate(NaiveDateTime.utc_now(), :second)
+
+    {:ok, updated_user} =
+      user
+      |> change(%{last_digest_emailed_at: now})
+      |> update_and_set_cache()
+
+    updated_user
+  end
+
   @spec toggle_confirmation(User.t()) :: {:ok, User.t()} | {:error, Changeset.t()}
   def toggle_confirmation(%User{} = user) do
     need_confirmation? = !user.info.confirmation_pending

lib/pleroma/user/info.ex

@@ -16,6 +16,8 @@ defmodule Pleroma.User.Info do
     field(:source_data, :map, default: %{})
     field(:note_count, :integer, default: 0)
     field(:follower_count, :integer, default: 0)
+    # Should be filled in only for remote users
+    field(:following_count, :integer, default: nil)
     field(:locked, :boolean, default: false)
     field(:confirmation_pending, :boolean, default: false)
     field(:confirmation_token, :string, default: nil)
@@ -43,9 +45,12 @@ defmodule Pleroma.User.Info do
     field(:hide_follows, :boolean, default: false)
     field(:hide_favorites, :boolean, default: true)
     field(:pinned_activities, {:array, :string}, default: [])
+    field(:email_notifications, :map, default: %{"digest" => false})
     field(:mascot, :map, default: nil)
     field(:emoji, {:array, :map}, default: [])
     field(:pleroma_settings_store, :map, default: %{})
+    field(:fields, {:array, :map}, default: [])
+    field(:raw_fields, {:array, :map}, default: [])
 
     field(:notification_settings, :map,
       default: %{
@@ -93,6 +98,30 @@ defmodule Pleroma.User.Info do
     |> validate_required([:notification_settings])
   end
 
+  @doc """
+  Update email notifications in the given User.Info struct.
+
+  Examples:
+
+      iex> update_email_notifications(%Pleroma.User.Info{email_notifications: %{"digest" => false}}, %{"digest" => true})
+      %Pleroma.User.Info{email_notifications: %{"digest" => true}}
+
+  """
+  @spec update_email_notifications(t(), map()) :: Ecto.Changeset.t()
+  def update_email_notifications(info, settings) do
+    email_notifications =
+      info.email_notifications
+      |> Map.merge(settings)
+      |> Map.take(["digest"])
+
+    params = %{email_notifications: email_notifications}
+    fields = [:email_notifications]
+
+    info
+    |> cast(params, fields)
+    |> validate_required(fields)
+  end
+
   def add_to_note_count(info, number) do
     set_note_count(info, info.note_count + number)
   end
@@ -223,19 +252,31 @@ defmodule Pleroma.User.Info do
       :uri,
       :hub,
       :topic,
-      :salmon
+      :salmon,
+      :hide_followers,
+      :hide_follows,
+      :follower_count,
+      :fields,
+      :following_count
     ])
+    |> validate_fields(true)
   end
 
-  def user_upgrade(info, params) do
+  def user_upgrade(info, params, remote? \\ false) do
     info
     |> cast(params, [
       :ap_enabled,
       :source_data,
       :banner,
       :locked,
-      :magic_key
+      :magic_key,
+      :follower_count,
+      :following_count,
+      :hide_follows,
+      :fields,
+      :hide_followers
     ])
+    |> validate_fields(remote?)
   end
 
   def profile_update(info, params) do
@@ -251,10 +292,40 @@ defmodule Pleroma.User.Info do
       :background,
       :show_role,
       :skip_thread_containment,
+      :fields,
+      :raw_fields,
       :pleroma_settings_store
     ])
+    |> validate_fields()
   end
 
+  def validate_fields(changeset, remote? \\ false) do
+    limit_name = if remote?, do: :max_remote_account_fields, else: :max_account_fields
+    limit = Pleroma.Config.get([:instance, limit_name], 0)
+
+    changeset
+    |> validate_length(:fields, max: limit)
+    |> validate_change(:fields, fn :fields, fields ->
+      if Enum.all?(fields, &valid_field?/1) do
+        []
+      else
+        [fields: "invalid"]
+      end
+    end)
+  end
+
+  defp valid_field?(%{"name" => name, "value" => value}) do
+    name_limit = Pleroma.Config.get([:instance, :account_field_name_length], 255)
+    value_limit = Pleroma.Config.get([:instance, :account_field_value_length], 255)
+
+    is_binary(name) &&
+      is_binary(value) &&
+      String.length(name) <= name_limit &&
+      String.length(value) <= value_limit
+  end
+
+  defp valid_field?(_), do: false
+
   @spec confirmation_changeset(Info.t(), keyword()) :: Changeset.t()
   def confirmation_changeset(info, opts) do
     need_confirmation? = Keyword.get(opts, :need_confirmation)
@@ -348,4 +419,27 @@ defmodule Pleroma.User.Info do
 
     cast(info, params, [:muted_reblogs])
   end
+
+  # ``fields`` is an array of mastodon profile field, containing ``{"name": "…", "value": "…"}``.
+  # For example: [{"name": "Pronoun", "value": "she/her"}, …]
+  def fields(%{fields: [], source_data: %{"attachment" => attachment}}) do
+    limit = Pleroma.Config.get([:instance, :max_remote_account_fields], 0)
+
+    attachment
+    |> Enum.filter(fn %{"type" => t} -> t == "PropertyValue" end)
+    |> Enum.map(fn fields -> Map.take(fields, ["name", "value"]) end)
+    |> Enum.take(limit)
+  end
+
+  def fields(%{fields: fields}), do: fields
+
+  def follow_information_update(info, params) do
+    info
+    |> cast(params, [
+      :hide_followers,
+      :hide_follows,
+      :follower_count,
+      :following_count
+    ])
+  end
 end

lib/pleroma/user/search.ex

@@ -44,7 +44,7 @@ defmodule Pleroma.User.Search do
     query_string = String.trim_leading(query_string, "@")
 
     with [name, domain] <- String.split(query_string, "@"),
-         formatted_domain <- String.replace(domain, ~r/[!-\-|@|[-`|{-~|\/|:]+/, "") do
+         formatted_domain <- String.replace(domain, ~r/[!-\-|@|[-`|{-~|\/|:|\s]+/, "") do
       name <> "@" <> to_string(:idna.encode(formatted_domain))
     else
       _ -> query_string

lib/pleroma/web/activity_pub/activity_pub.ex

@@ -23,6 +23,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   import Pleroma.Web.ActivityPub.Visibility
 
   require Logger
+  require Pleroma.Constants
 
   # For Announce activities, we filter the recipients based on following status for any actors
   # that match actual users.  See issue #164 for more information about why this is necessary.
@@ -64,12 +65,12 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     if not is_nil(actor) do
       with user <- User.get_cached_by_ap_id(actor),
            false <- user.info.deactivated do
-        :ok
+        true
       else
-        _e -> :reject
+        _e -> false
       end
     else
-      :ok
+      true
     end
   end
 
@@ -118,10 +119,10 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   def increase_poll_votes_if_vote(_create_data), do: :noop
 
-  def insert(map, local \\ true, fake \\ false) when is_map(map) do
+  def insert(map, local \\ true, fake \\ false, bypass_actor_check \\ false) when is_map(map) do
     with nil <- Activity.normalize(map),
          map <- lazy_put_activity_defaults(map, fake),
-         :ok <- check_actor_is_active(map["actor"]),
+         true <- bypass_actor_check || check_actor_is_active(map["actor"]),
          {_, true} <- {:remote_limit_error, check_remote_limit(map)},
          {:ok, map} <- MRF.filter(map),
          {recipients, _, _} = get_recipients(map),
@@ -207,8 +208,6 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   def stream_out_participations(_, _), do: :noop
 
   def stream_out(activity) do
-    public = "https://www.w3.org/ns/activitystreams#Public"
-
     if activity.data["type"] in ["Create", "Announce", "Delete"] do
       object = Object.normalize(activity)
       # Do not stream out poll replies
@@ -216,7 +215,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
         Pleroma.Web.Streamer.stream("user", activity)
         Pleroma.Web.Streamer.stream("list", activity)
 
-        if Enum.member?(activity.data["to"], public) do
+        if get_visibility(activity) == "public" do
           Pleroma.Web.Streamer.stream("public", activity)
 
           if activity.local do
@@ -238,12 +237,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
             end
           end
         else
-          # TODO: Write test, replace with visibility test
+          if get_visibility(activity) == "direct",
-          if !Enum.member?(activity.data["cc"] || [], public) &&
-               !Enum.member?(
-                 activity.data["to"],
-                 User.get_cached_by_ap_id(activity.data["actor"]).follower_address
-               ),
             do: Pleroma.Web.Streamer.stream("direct", activity)
         end
       end
@@ -273,6 +267,9 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     else
       {:fake, true, activity} ->
         {:ok, activity}
+
+      {:error, message} ->
+        {:error, message}
     end
   end
 
@@ -391,7 +388,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   def follow(follower, followed, activity_id \\ nil, local \\ true) do
     with data <- make_follow_data(follower, followed, activity_id),
          {:ok, activity} <- insert(data, local),
-         :ok <- maybe_federate(activity) do
+         :ok <- maybe_federate(activity),
+         _ <- User.set_follow_state_cache(follower.ap_id, followed.ap_id, activity.data["state"]) do
       {:ok, activity}
     end
   end
@@ -413,7 +411,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
            "actor" => ap_id,
            "object" => %{"type" => "Person", "id" => ap_id}
          },
-         {:ok, activity} <- insert(data, true, true),
+         {:ok, activity} <- insert(data, true, true, true),
          :ok <- maybe_federate(activity) do
       {:ok, user}
     end
@@ -514,13 +512,15 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   end
 
   defp fetch_activities_for_context_query(context, opts) do
-    public = ["https://www.w3.org/ns/activitystreams#Public"]
+    public = [Pleroma.Constants.as_public()]
 
     recipients =
       if opts["user"], do: [opts["user"].ap_id | opts["user"].following] ++ public, else: public
 
     from(activity in Activity)
     |> maybe_preload_objects(opts)
+    |> maybe_preload_bookmarks(opts)
+    |> maybe_set_thread_muted_field(opts)
     |> restrict_blocked(opts)
     |> restrict_recipients(recipients, opts["user"])
     |> where(
@@ -534,6 +534,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
       )
     )
     |> exclude_poll_votes(opts)
+    |> exclude_id(opts)
     |> order_by([activity], desc: activity.id)
   end
 
@@ -555,7 +556,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
   end
 
   def fetch_public_activities(opts \\ %{}) do
-    q = fetch_activities_query(["https://www.w3.org/ns/activitystreams#Public"], opts)
+    q = fetch_activities_query([Pleroma.Constants.as_public()], opts)
 
     q
     |> restrict_unlisted()
@@ -626,6 +627,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     params =
       params
       |> Map.put("type", ["Create", "Announce"])
+      |> Map.put("user", reading_user)
       |> Map.put("actor_id", user.ap_id)
       |> Map.put("whole_db", true)
       |> Map.put("pinned_activity_ids", user.info.pinned_activities)
@@ -646,10 +648,9 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   defp user_activities_recipients(%{"reading_user" => reading_user}) do
     if reading_user do
-      ["https://www.w3.org/ns/activitystreams#Public"] ++
+      [Pleroma.Constants.as_public()] ++ [reading_user.ap_id | reading_user.following]
-        [reading_user.ap_id | reading_user.following]
     else
-      ["https://www.w3.org/ns/activitystreams#Public"]
+      [Pleroma.Constants.as_public()]
     end
   end
 
@@ -753,8 +754,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   defp restrict_favorited_by(query, %{"favorited_by" => ap_id}) do
     from(
-      activity in query,
+      [_activity, object] in query,
-      where: fragment(~s(? <@ (? #> '{"object","likes"}'\)), ^ap_id, activity.data)
+      where: fragment("(?)->'likes' \\? (?)", object.data, ^ap_id)
     )
   end
 
@@ -790,14 +791,20 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   defp restrict_muted(query, %{"with_muted" => val}) when val in [true, "true", "1"], do: query
 
-  defp restrict_muted(query, %{"muting_user" => %User{info: info}}) do
+  defp restrict_muted(query, %{"muting_user" => %User{info: info}} = opts) do
     mutes = info.mutes
 
-    from(
+    query =
-      activity in query,
+      from([activity] in query,
         where: fragment("not (? = ANY(?))", activity.actor, ^mutes),
         where: fragment("not (?->'to' \\?| ?)", activity.data, ^mutes)
       )
+
+    unless opts["skip_preload"] do
+      from([thread_mute: tm] in query, where: is_nil(tm))
+    else
+      query
+    end
   end
 
   defp restrict_muted(query, _), do: query
@@ -834,7 +841,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
         fragment(
           "not (coalesce(?->'cc', '{}'::jsonb) \\?| ?)",
           activity.data,
-          ^["https://www.w3.org/ns/activitystreams#Public"]
+          ^[Pleroma.Constants.as_public()]
         )
     )
   end
@@ -874,6 +881,12 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     end
   end
 
+  defp exclude_id(query, %{"exclude_id" => id}) when is_binary(id) do
+    from(activity in query, where: activity.id != ^id)
+  end
+
+  defp exclude_id(query, _), do: query
+
   defp maybe_preload_objects(query, %{"skip_preload" => true}), do: query
 
   defp maybe_preload_objects(query, _) do
@@ -892,7 +905,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   defp maybe_set_thread_muted_field(query, opts) do
     query
-    |> Activity.with_set_thread_muted_field(opts["user"])
+    |> Activity.with_set_thread_muted_field(opts["muting_user"] || opts["user"])
   end
 
   defp maybe_order(query, %{order: :desc}) do
@@ -971,7 +984,7 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
       where:
         fragment("? && ?", activity.recipients, ^recipients) or
           (fragment("? && ?", activity.recipients, ^recipients_with_public) and
-             "https://www.w3.org/ns/activitystreams#Public" in activity.recipients)
+             ^Pleroma.Constants.as_public() in activity.recipients)
     )
   end
 
@@ -1010,16 +1023,23 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
           "url" => [%{"href" => data["image"]["url"]}]
         }
 
+    fields =
+      data
+      |> Map.get("attachment", [])
+      |> Enum.filter(fn %{"type" => t} -> t == "PropertyValue" end)
+      |> Enum.map(fn fields -> Map.take(fields, ["name", "value"]) end)
+
     locked = data["manuallyApprovesFollowers"] || false
     data = Transmogrifier.maybe_fix_user_object(data)
 
     user_data = %{
       ap_id: data["id"],
       info: %{
-        "ap_enabled" => true,
+        ap_enabled: true,
-        "source_data" => data,
+        source_data: data,
-        "banner" => banner,
+        banner: banner,
-        "locked" => locked
+        fields: fields,
+        locked: locked
       },
       avatar: avatar,
       name: data["name"],
@@ -1043,6 +1063,71 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
     {:ok, user_data}
   end
 
+  def fetch_follow_information_for_user(user) do
+    with {:ok, following_data} <-
+           Fetcher.fetch_and_contain_remote_object_from_id(user.following_address),
+         following_count when is_integer(following_count) <- following_data["totalItems"],
+         {:ok, hide_follows} <- collection_private(following_data),
+         {:ok, followers_data} <-
+           Fetcher.fetch_and_contain_remote_object_from_id(user.follower_address),
+         followers_count when is_integer(followers_count) <- followers_data["totalItems"],
+         {:ok, hide_followers} <- collection_private(followers_data) do
+      {:ok,
+       %{
+         hide_follows: hide_follows,
+         follower_count: followers_count,
+         following_count: following_count,
+         hide_followers: hide_followers
+       }}
+    else
+      {:error, _} = e ->
+        e
+
+      e ->
+        {:error, e}
+    end
+  end
+
+  defp maybe_update_follow_information(data) do
+    with {:enabled, true} <-
+           {:enabled, Pleroma.Config.get([:instance, :external_user_synchronization])},
+         {:ok, info} <- fetch_follow_information_for_user(data) do
+      info = Map.merge(data.info, info)
+      Map.put(data, :info, info)
+    else
+      {:enabled, false} ->
+        data
+
+      e ->
+        Logger.error(
+          "Follower/Following counter update for #{data.ap_id} failed.\n" <> inspect(e)
+        )
+
+        data
+    end
+  end
+
+  defp collection_private(data) do
+    if is_map(data["first"]) and
+         data["first"]["type"] in ["CollectionPage", "OrderedCollectionPage"] do
+      {:ok, false}
+    else
+      with {:ok, %{"type" => type}} when type in ["CollectionPage", "OrderedCollectionPage"] <-
+             Fetcher.fetch_and_contain_remote_object_from_id(data["first"]) do
+        {:ok, false}
+      else
+        {:error, {:ok, %{status: code}}} when code in [401, 403] ->
+          {:ok, true}
+
+        {:error, _} = e ->
+          e
+
+        e ->
+          {:error, e}
+      end
+    end
+  end
+
   def user_data_from_user_object(data) do
     with {:ok, data} <- MRF.filter(data),
          {:ok, data} <- object_to_user_data(data) do
@@ -1054,7 +1139,8 @@ defmodule Pleroma.Web.ActivityPub.ActivityPub do
 
   def fetch_and_prepare_user_from_ap_id(ap_id) do
     with {:ok, data} <- Fetcher.fetch_and_contain_remote_object_from_id(ap_id),
-         {:ok, data} <- user_data_from_user_object(data) do
+         {:ok, data} <- user_data_from_user_object(data),
+         data <- maybe_update_follow_information(data) do
       {:ok, data}
     else
       e -> Logger.error("Could not decode user at fetch #{ap_id}, #{inspect(e)}")

lib/pleroma/web/activity_pub/mrf.ex

@@ -28,11 +28,43 @@ defmodule Pleroma.Web.ActivityPub.MRF do
 
   @spec subdomains_regex([String.t()]) :: [Regex.t()]
   def subdomains_regex(domains) when is_list(domains) do
-    for domain <- domains, do: ~r(^#{String.replace(domain, "*.", "(.*\\.)*")}$)
+    for domain <- domains, do: ~r(^#{String.replace(domain, "*.", "(.*\\.)*")}$)i
   end
 
   @spec subdomain_match?([Regex.t()], String.t()) :: boolean()
   def subdomain_match?(domains, host) do
     Enum.any?(domains, fn domain -> Regex.match?(domain, host) end)
   end
+
+  @callback describe() :: {:ok | :error, Map.t()}
+
+  def describe(policies) do
+    {:ok, policy_configs} =
+      policies
+      |> Enum.reduce({:ok, %{}}, fn
+        policy, {:ok, data} ->
+          {:ok, policy_data} = policy.describe()
+          {:ok, Map.merge(data, policy_data)}
+
+        _, error ->
+          error
+      end)
+
+    mrf_policies =
+      get_policies()
+      |> Enum.map(fn policy -> to_string(policy) |> String.split(".") |> List.last() end)
+
+    exclusions = Pleroma.Config.get([:instance, :mrf_transparency_exclusions])
+
+    base =
+      %{
+        mrf_policies: mrf_policies,
+        exclusions: length(exclusions) > 0
+      }
+      |> Map.merge(policy_configs)
+
+    {:ok, base}
+  end
+
+  def describe, do: get_policies() |> describe()
 end

lib/pleroma/web/activity_pub/mrf/anti_followbot_policy.ex

@@ -62,4 +62,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.AntiFollowbotPolicy do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/anti_link_spam_policy.ex

@@ -5,6 +5,8 @@
 defmodule Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy do
   alias Pleroma.User
 
+  @behaviour Pleroma.Web.ActivityPub.MRF
+
   require Logger
 
   # has the user successfully posted before?
@@ -22,6 +24,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy do
 
   defp contains_links?(_), do: false
 
+  @impl true
   def filter(%{"type" => "Create", "actor" => actor, "object" => object} = message) do
     with {:ok, %User{} = u} <- User.get_or_fetch_by_ap_id(actor),
          {:contains_links, true} <- {:contains_links, contains_links?(object)},
@@ -45,4 +48,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.AntiLinkSpamPolicy do
 
   # in all other cases, pass through
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/drop_policy.ex

@@ -12,4 +12,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.DropPolicy do
     Logger.info("REJECTING #{inspect(object)}")
     {:reject, object}
   end
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/ensure_re_prepended.ex

@@ -39,4 +39,6 @@ defmodule Pleroma.Web.ActivityPub.MRF.EnsureRePrepended do
   end
 
   def filter(object), do: {:ok, object}
+
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/hellthread_policy.ex

@@ -4,6 +4,9 @@
 
 defmodule Pleroma.Web.ActivityPub.MRF.HellthreadPolicy do
   alias Pleroma.User
+
+  require Pleroma.Constants
+
   @moduledoc "Block messages with too much mentions (configurable)"
 
   @behaviour Pleroma.Web.ActivityPub.MRF
@@ -19,12 +22,12 @@ defmodule Pleroma.Web.ActivityPub.MRF.HellthreadPolicy do
         when follower_collection? and recipients > threshold ->
           message
           |> Map.put("to", [follower_collection])
-          |> Map.put("cc", ["https://www.w3.org/ns/activitystreams#Public"])
+          |> Map.put("cc", [Pleroma.Constants.as_public()])
 
         {:public, recipients} when recipients > threshold ->
           message
           |> Map.put("to", [])
-          |> Map.put("cc", ["https://www.w3.org/ns/activitystreams#Public"])
+          |> Map.put("cc", [Pleroma.Constants.as_public()])
 
         _ ->
           message
@@ -51,10 +54,10 @@ defmodule Pleroma.Web.ActivityPub.MRF.HellthreadPolicy do
     recipients = (message["to"] || []) ++ (message["cc"] || [])
     follower_collection = User.get_cached_by_ap_id(message["actor"]).follower_address
 
-    if Enum.member?(recipients, "https://www.w3.org/ns/activitystreams#Public") do
+    if Enum.member?(recipients, Pleroma.Constants.as_public()) do
       recipients =
         recipients
-        |> List.delete("https://www.w3.org/ns/activitystreams#Public")
+        |> List.delete(Pleroma.Constants.as_public())
         |> List.delete(follower_collection)
 
       {:public, length(recipients)}
@@ -87,4 +90,8 @@ defmodule Pleroma.Web.ActivityPub.MRF.HellthreadPolicy do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe,
+    do: {:ok, %{mrf_hellthread: Pleroma.Config.get(:mrf_hellthread) |> Enum.into(%{})}}
 end

lib/pleroma/web/activity_pub/mrf/keyword_policy.ex

@@ -3,6 +3,8 @@
 # SPDX-License-Identifier: AGPL-3.0-only
 
 defmodule Pleroma.Web.ActivityPub.MRF.KeywordPolicy do
+  require Pleroma.Constants
+
   @moduledoc "Reject or Word-Replace messages with a keyword or regex"
 
   @behaviour Pleroma.Web.ActivityPub.MRF
@@ -31,12 +33,12 @@ defmodule Pleroma.Web.ActivityPub.MRF.KeywordPolicy do
   defp check_ftl_removal(
          %{"to" => to, "object" => %{"content" => content, "summary" => summary}} = message
        ) do
-    if "https://www.w3.org/ns/activitystreams#Public" in to and
+    if Pleroma.Constants.as_public() in to and
          Enum.any?(Pleroma.Config.get([:mrf_keyword, :federated_timeline_removal]), fn pattern ->
            string_matches?(content, pattern) or string_matches?(summary, pattern)
          end) do
-      to = List.delete(to, "https://www.w3.org/ns/activitystreams#Public")
+      to = List.delete(to, Pleroma.Constants.as_public())
-      cc = ["https://www.w3.org/ns/activitystreams#Public" | message["cc"] || []]
+      cc = [Pleroma.Constants.as_public() | message["cc"] || []]
 
       message =
         message
@@ -94,4 +96,36 @@ defmodule Pleroma.Web.ActivityPub.MRF.KeywordPolicy do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe do
+    # This horror is needed to convert regex sigils to strings
+    mrf_keyword =
+      Pleroma.Config.get(:mrf_keyword, [])
+      |> Enum.map(fn {key, value} ->
+        {key,
+         Enum.map(value, fn
+           {pattern, replacement} ->
+             %{
+               "pattern" =>
+                 if not is_binary(pattern) do
+                   inspect(pattern)
+                 else
+                   pattern
+                 end,
+               "replacement" => replacement
+             }
+
+           pattern ->
+             if not is_binary(pattern) do
+               inspect(pattern)
+             else
+               pattern
+             end
+         end)}
+      end)
+      |> Enum.into(%{})
+
+    {:ok, %{mrf_keyword: mrf_keyword}}
+  end
 end

lib/pleroma/web/activity_pub/mrf/mediaproxy_warming_policy.ex

@@ -53,4 +53,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/mention_policy.ex

@@ -21,4 +21,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.MentionPolicy do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/no_placeholder_text_policy.ex

@@ -19,4 +19,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.NoPlaceholderTextPolicy do
 
   @impl true
   def filter(object), do: {:ok, object}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/noop_policy.ex

@@ -10,4 +10,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.NoOpPolicy do
   def filter(object) do
     {:ok, object}
   end
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/normalize_markup.ex

@@ -21,4 +21,6 @@ defmodule Pleroma.Web.ActivityPub.MRF.NormalizeMarkup do
   end
 
   def filter(object), do: {:ok, object}
+
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/reject_non_public.ex

@@ -10,7 +10,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.RejectNonPublic do
 
   @behaviour Pleroma.Web.ActivityPub.MRF
 
-  @public "https://www.w3.org/ns/activitystreams#Public"
+  require Pleroma.Constants
 
   @impl true
   def filter(%{"type" => "Create"} = object) do
@@ -19,8 +19,8 @@ defmodule Pleroma.Web.ActivityPub.MRF.RejectNonPublic do
     # Determine visibility
     visibility =
       cond do
-        @public in object["to"] -> "public"
+        Pleroma.Constants.as_public() in object["to"] -> "public"
-        @public in object["cc"] -> "unlisted"
+        Pleroma.Constants.as_public() in object["cc"] -> "unlisted"
         user.follower_address in object["to"] -> "followers"
         true -> "direct"
       end
@@ -44,4 +44,8 @@ defmodule Pleroma.Web.ActivityPub.MRF.RejectNonPublic do
 
   @impl true
   def filter(object), do: {:ok, object}
+
+  @impl true
+  def describe,
+    do: {:ok, %{mrf_rejectnonpublic: Pleroma.Config.get(:mrf_rejectnonpublic) |> Enum.into(%{})}}
 end

lib/pleroma/web/activity_pub/mrf/simple_policy.ex

@@ -8,6 +8,8 @@ defmodule Pleroma.Web.ActivityPub.MRF.SimplePolicy do
   @moduledoc "Filter activities depending on their origin instance"
   @behaviour MRF
 
+  require Pleroma.Constants
+
   defp check_accept(%{host: actor_host} = _actor_info, object) do
     accepts =
       Pleroma.Config.get([:mrf_simple, :accept])
@@ -89,14 +91,10 @@ defmodule Pleroma.Web.ActivityPub.MRF.SimplePolicy do
     object =
       with true <- MRF.subdomain_match?(timeline_removal, actor_host),
            user <- User.get_cached_by_ap_id(object["actor"]),
-           true <- "https://www.w3.org/ns/activitystreams#Public" in object["to"] do
+           true <- Pleroma.Constants.as_public() in object["to"] do
-        to =
+        to = List.delete(object["to"], Pleroma.Constants.as_public()) ++ [user.follower_address]
-          List.delete(object["to"], "https://www.w3.org/ns/activitystreams#Public") ++
-            [user.follower_address]
 
-        cc =
+        cc = List.delete(object["cc"], user.follower_address) ++ [Pleroma.Constants.as_public()]
-          List.delete(object["cc"], user.follower_address) ++
-            ["https://www.w3.org/ns/activitystreams#Public"]
 
         object
         |> Map.put("to", to)
@@ -179,4 +177,16 @@ defmodule Pleroma.Web.ActivityPub.MRF.SimplePolicy do
   end
 
   def filter(object), do: {:ok, object}
+
+  @impl true
+  def describe do
+    exclusions = Pleroma.Config.get([:instance, :mrf_transparency_exclusions])
+
+    mrf_simple =
+      Pleroma.Config.get(:mrf_simple)
+      |> Enum.map(fn {k, v} -> {k, Enum.reject(v, fn v -> v in exclusions end)} end)
+      |> Enum.into(%{})
+
+    {:ok, %{mrf_simple: mrf_simple}}
+  end
 end

lib/pleroma/web/activity_pub/mrf/subchain_policy.ex

@@ -37,4 +37,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.SubchainPolicy do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/tag_policy.ex

@@ -19,7 +19,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.TagPolicy do
      - `mrf_tag:disable-any-subscription`: Reject any follow requests
   """
 
-  @public "https://www.w3.org/ns/activitystreams#Public"
+  require Pleroma.Constants
 
   defp get_tags(%User{tags: tags}) when is_list(tags), do: tags
   defp get_tags(_), do: []
@@ -70,9 +70,9 @@ defmodule Pleroma.Web.ActivityPub.MRF.TagPolicy do
        ) do
     user = User.get_cached_by_ap_id(actor)
 
-    if Enum.member?(to, @public) do
+    if Enum.member?(to, Pleroma.Constants.as_public()) do
-      to = List.delete(to, @public) ++ [user.follower_address]
+      to = List.delete(to, Pleroma.Constants.as_public()) ++ [user.follower_address]
-      cc = List.delete(cc, user.follower_address) ++ [@public]
+      cc = List.delete(cc, user.follower_address) ++ [Pleroma.Constants.as_public()]
 
       object =
         object
@@ -103,9 +103,10 @@ defmodule Pleroma.Web.ActivityPub.MRF.TagPolicy do
        ) do
     user = User.get_cached_by_ap_id(actor)
 
-    if Enum.member?(to, @public) or Enum.member?(cc, @public) do
+    if Enum.member?(to, Pleroma.Constants.as_public()) or
-      to = List.delete(to, @public) ++ [user.follower_address]
+         Enum.member?(cc, Pleroma.Constants.as_public()) do
-      cc = List.delete(cc, @public)
+      to = List.delete(to, Pleroma.Constants.as_public()) ++ [user.follower_address]
+      cc = List.delete(cc, Pleroma.Constants.as_public())
 
       object =
         object
@@ -164,4 +165,7 @@ defmodule Pleroma.Web.ActivityPub.MRF.TagPolicy do
 
   @impl true
   def filter(message), do: {:ok, message}
+
+  @impl true
+  def describe, do: {:ok, %{}}
 end

lib/pleroma/web/activity_pub/mrf/user_allowlist_policy.ex

@@ -32,4 +32,13 @@ defmodule Pleroma.Web.ActivityPub.MRF.UserAllowListPolicy do
   end
 
   def filter(object), do: {:ok, object}
+
+  @impl true
+  def describe do
+    mrf_user_allowlist =
+      Config.get([:mrf_user_allowlist], [])
+      |> Enum.into(%{}, fn {k, v} -> {k, length(v)} end)
+
+    {:ok, %{mrf_user_allowlist: mrf_user_allowlist}}
+  end
 end

lib/pleroma/web/activity_pub/mrf/vocabulary_policy.ex

@@ -0,0 +1,37 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Pleroma.Web.ActivityPub.MRF.VocabularyPolicy do
+  @moduledoc "Filter messages which belong to certain activity vocabularies"
+
+  @behaviour Pleroma.Web.ActivityPub.MRF
+
+  def filter(%{"type" => "Undo", "object" => child_message} = message) do
+    with {:ok, _} <- filter(child_message) do
+      {:ok, message}
+    else
+      {:reject, nil} ->
+        {:reject, nil}
+    end
+  end
+
+  def filter(%{"type" => message_type} = message) do
+    with accepted_vocabulary <- Pleroma.Config.get([:mrf_vocabulary, :accept]),
+         rejected_vocabulary <- Pleroma.Config.get([:mrf_vocabulary, :reject]),
+         true <-
+           length(accepted_vocabulary) == 0 || Enum.member?(accepted_vocabulary, message_type),
+         false <-
+           length(rejected_vocabulary) > 0 && Enum.member?(rejected_vocabulary, message_type),
+         {:ok, _} <- filter(message["object"]) do
+      {:ok, message}
+    else
+      _ -> {:reject, nil}
+    end
+  end
+
+  def filter(message), do: {:ok, message}
+
+  def describe,
+    do: {:ok, %{mrf_vocabulary: Pleroma.Config.get(:mrf_vocabulary) |> Enum.into(%{})}}
+end

lib/pleroma/web/activity_pub/publisher.ex

@@ -11,6 +11,8 @@ defmodule Pleroma.Web.ActivityPub.Publisher do
   alias Pleroma.Web.ActivityPub.Relay
   alias Pleroma.Web.ActivityPub.Transmogrifier
 
+  require Pleroma.Constants
+
   import Pleroma.Web.ActivityPub.Visibility
 
   @behaviour Pleroma.Web.Federator.Publisher
@@ -44,7 +46,7 @@ defmodule Pleroma.Web.ActivityPub.Publisher do
   """
   def publish_one(%{inbox: inbox, json: json, actor: %User{} = actor, id: id} = params) do
     Logger.info("Federating #{id} to #{inbox}")
-    host = URI.parse(inbox).host
+    %{host: host, path: path} = URI.parse(inbox)
 
     digest = "SHA-256=" <> (:crypto.hash(:sha256, json) |> Base.encode64())
 
@@ -54,6 +56,7 @@ defmodule Pleroma.Web.ActivityPub.Publisher do
 
     signature =
       Pleroma.Signature.sign(actor, %{
+        "(request-target)": "post #{path}",
         host: host,
         "content-length": byte_size(json),
         digest: digest,
@@ -117,8 +120,6 @@ defmodule Pleroma.Web.ActivityPub.Publisher do
     |> Enum.map(& &1.ap_id)
   end
 
-  @as_public "https://www.w3.org/ns/activitystreams#Public"
-
   defp maybe_use_sharedinbox(%User{info: %{source_data: data}}),
     do: (is_map(data["endpoints"]) && Map.get(data["endpoints"], "sharedInbox")) || data["inbox"]
 
@@ -145,7 +146,7 @@ defmodule Pleroma.Web.ActivityPub.Publisher do
       type == "Delete" ->
         maybe_use_sharedinbox(user)
 
-      @as_public in to || @as_public in cc ->
+      Pleroma.Constants.as_public() in to || Pleroma.Constants.as_public() in cc ->
         maybe_use_sharedinbox(user)
 
       length(to) + length(cc) > 1 ->

lib/pleroma/web/activity_pub/relay.ex

@@ -14,6 +14,7 @@ defmodule Pleroma.Web.ActivityPub.Relay do
     |> User.get_or_create_service_actor_by_ap_id()
   end
 
+  @spec follow(String.t()) :: {:ok, Activity.t()} | {:error, any()}
   def follow(target_instance) do
     with %User{} = local_user <- get_actor(),
          {:ok, %User{} = target_user} <- User.get_or_fetch_by_ap_id(target_instance),
@@ -21,12 +22,17 @@ defmodule Pleroma.Web.ActivityPub.Relay do
       Logger.info("relay: followed instance: #{target_instance}; id=#{activity.data["id"]}")
       {:ok, activity}
     else
+      {:error, _} = error ->
+        Logger.error("error: #{inspect(error)}")
+        error
+
       e ->
         Logger.error("error: #{inspect(e)}")
         {:error, e}
     end
   end
 
+  @spec unfollow(String.t()) :: {:ok, Activity.t()} | {:error, any()}
   def unfollow(target_instance) do
     with %User{} = local_user <- get_actor(),
          {:ok, %User{} = target_user} <- User.get_or_fetch_by_ap_id(target_instance),
@@ -34,20 +40,27 @@ defmodule Pleroma.Web.ActivityPub.Relay do
       Logger.info("relay: unfollowed instance: #{target_instance}: id=#{activity.data["id"]}")
       {:ok, activity}
     else
+      {:error, _} = error ->
+        Logger.error("error: #{inspect(error)}")
+        error
+
       e ->
         Logger.error("error: #{inspect(e)}")
         {:error, e}
     end
   end
 
+  @spec publish(any()) :: {:ok, Activity.t(), Object.t()} | {:error, any()}
   def publish(%Activity{data: %{"type" => "Create"}} = activity) do
     with %User{} = user <- get_actor(),
          %Object{} = object <- Object.normalize(activity) do
       ActivityPub.announce(user, object, nil, true, false)
     else
-      e -> Logger.error("error: #{inspect(e)}")
+      e ->
+        Logger.error("error: #{inspect(e)}")
+        {:error, inspect(e)}
     end
   end
 
-  def publish(_), do: nil
+  def publish(_), do: {:error, "Not implemented"}
 end

lib/pleroma/web/activity_pub/transmogrifier.ex

@@ -19,12 +19,14 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   import Ecto.Query
 
   require Logger
+  require Pleroma.Constants
 
   @doc """
   Modifies an incoming AP object (mastodon format) to our internal format.
   """
   def fix_object(object, options \\ []) do
     object
+    |> strip_internal_fields
     |> fix_actor
     |> fix_url
     |> fix_attachments
@@ -33,7 +35,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     |> fix_emoji
     |> fix_tag
     |> fix_content_map
-    |> fix_likes
     |> fix_addressing
     |> fix_summary
     |> fix_type(options)
@@ -102,8 +103,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
 
     follower_collection = User.get_cached_by_ap_id(Containment.get_actor(object)).follower_address
 
-    explicit_mentions =
+    explicit_mentions = explicit_mentions ++ [Pleroma.Constants.as_public(), follower_collection]
-      explicit_mentions ++ ["https://www.w3.org/ns/activitystreams#Public", follower_collection]
 
     fix_explicit_addressing(object, explicit_mentions, follower_collection)
   end
@@ -115,11 +115,11 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
 
     if followers_collection not in recipients do
       cond do
-        "https://www.w3.org/ns/activitystreams#Public" in cc ->
+        Pleroma.Constants.as_public() in cc ->
           to = to ++ [followers_collection]
           Map.put(object, "to", to)
 
-        "https://www.w3.org/ns/activitystreams#Public" in to ->
+        Pleroma.Constants.as_public() in to ->
           cc = cc ++ [followers_collection]
           Map.put(object, "cc", cc)
 
@@ -151,20 +151,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     |> Map.put("actor", Containment.get_actor(%{"actor" => actor}))
   end
 
-  # Check for standardisation
-  # This is what Peertube does
-  # curl -H 'Accept: application/activity+json' $likes | jq .totalItems
-  # Prismo returns only an integer (count) as "likes"
-  def fix_likes(%{"likes" => likes} = object) when not is_map(likes) do
-    object
-    |> Map.put("likes", [])
-    |> Map.put("like_count", 0)
-  end
-
-  def fix_likes(object) do
-    object
-  end
-
   def fix_in_reply_to(object, options \\ [])
 
   def fix_in_reply_to(%{"inReplyTo" => in_reply_to} = object, options)
@@ -347,13 +333,15 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
 
   def fix_type(object, options \\ [])
 
-  def fix_type(%{"inReplyTo" => reply_id} = object, options) when is_binary(reply_id) do
+  def fix_type(%{"inReplyTo" => reply_id, "name" => _} = object, options)
+      when is_binary(reply_id) do
     reply =
-      if Federator.allowed_incoming_reply_depth?(options[:depth]) do
+      with true <- Federator.allowed_incoming_reply_depth?(options[:depth]),
-        Object.normalize(reply_id, true)
+           {:ok, object} <- get_obj_helper(reply_id, options) do
+        object
       end
 
-    if reply && (reply.data["type"] == "Question" and object["name"]) do
+    if reply && reply.data["type"] == "Question" do
       Map.put(object, "type", "Answer")
     else
       object
@@ -480,8 +468,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
          {:ok, %User{} = follower} <- User.get_or_fetch_by_ap_id(follower),
          {:ok, activity} <- ActivityPub.follow(follower, followed, id, false) do
       with deny_follow_blocked <- Pleroma.Config.get([:user, :deny_follow_blocked]),
-           {_, false} <-
+           {_, false} <- {:user_blocked, User.blocks?(followed, follower) && deny_follow_blocked},
-             {:user_blocked, User.blocks?(followed, follower) && deny_follow_blocked},
            {_, false} <- {:user_locked, User.locked?(followed)},
            {_, {:ok, follower}} <- {:follow, User.follow(follower, followed)},
            {_, {:ok, _}} <-
@@ -609,16 +596,22 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     with %User{ap_id: ^actor_id} = actor <- User.get_cached_by_ap_id(object["id"]) do
       {:ok, new_user_data} = ActivityPub.user_data_from_user_object(object)
 
-      banner = new_user_data[:info]["banner"]
+      banner = new_user_data[:info][:banner]
-      locked = new_user_data[:info]["locked"] || false
+      locked = new_user_data[:info][:locked] || false
+      attachment = get_in(new_user_data, [:info, :source_data, "attachment"]) || []
+
+      fields =
+        attachment
+        |> Enum.filter(fn %{"type" => t} -> t == "PropertyValue" end)
+        |> Enum.map(fn fields -> Map.take(fields, ["name", "value"]) end)
 
       update_data =
         new_user_data
         |> Map.take([:name, :bio, :avatar])
-        |> Map.put(:info, %{"banner" => banner, "locked" => locked})
+        |> Map.put(:info, %{banner: banner, locked: locked, fields: fields})
 
       actor
-      |> User.upgrade_changeset(update_data)
+      |> User.upgrade_changeset(update_data, true)
       |> User.update_and_set_cache()
 
       ActivityPub.update(%{
@@ -656,20 +649,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
       nil ->
         case User.get_cached_by_ap_id(object_id) do
           %User{ap_id: ^actor} = user ->
-            {:ok, followers} = User.get_followers(user)
+            User.delete(user)
-
-            Enum.each(followers, fn follower ->
-              User.unfollow(follower, user)
-            end)
-
-            {:ok, friends} = User.get_friends(user)
-
-            Enum.each(friends, fn followed ->
-              User.unfollow(user, followed)
-            end)
-
-            User.invalidate_cache(user)
-            Repo.delete(user)
 
           nil ->
             :error
@@ -727,8 +707,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
         } = _data,
         _options
       ) do
-    with true <- Pleroma.Config.get([:activitypub, :accept_blocks]),
+    with %User{local: true} = blocked <- User.get_cached_by_ap_id(blocked),
-         %User{local: true} = blocked <- User.get_cached_by_ap_id(blocked),
          {:ok, %User{} = blocker} <- User.get_or_fetch_by_ap_id(blocker),
          {:ok, activity} <- ActivityPub.unblock(blocker, blocked, id, false) do
       User.unblock(blocker, blocked)
@@ -742,8 +721,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
         %{"type" => "Block", "object" => blocked, "actor" => blocker, "id" => id} = _data,
         _options
       ) do
-    with true <- Pleroma.Config.get([:activitypub, :accept_blocks]),
+    with %User{local: true} = blocked = User.get_cached_by_ap_id(blocked),
-         %User{local: true} = blocked = User.get_cached_by_ap_id(blocked),
          {:ok, %User{} = blocker} = User.get_or_fetch_by_ap_id(blocker),
          {:ok, activity} <- ActivityPub.block(blocker, blocked, id, false) do
       User.unfollow(blocker, blocked)
@@ -798,7 +776,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     |> add_mention_tags
     |> add_emoji_tags
     |> add_attributed_to
-    |> add_likes
     |> prepare_attachments
     |> set_conversation
     |> set_reply_to_uri
@@ -985,22 +962,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     |> Map.put("attributedTo", attributed_to)
   end
 
-  def add_likes(%{"id" => id, "like_count" => likes} = object) do
-    likes = %{
-      "id" => "#{id}/likes",
-      "first" => "#{id}/likes?page=1",
-      "type" => "OrderedCollection",
-      "totalItems" => likes
-    }
-
-    object
-    |> Map.put("likes", likes)
-  end
-
-  def add_likes(object) do
-    object
-  end
-
   def prepare_attachments(object) do
     attachments =
       (object["attachment"] || [])
@@ -1016,6 +977,7 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
   defp strip_internal_fields(object) do
     object
     |> Map.drop([
+      "likes",
       "like_count",
       "announcements",
       "announcement_count",
@@ -1090,10 +1052,6 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
         PleromaJobQueue.enqueue(:transmogrifier, __MODULE__, [:user_upgrade, user])
       end
 
-      if Pleroma.Config.get([:instance, :external_user_synchronization]) do
-        update_following_followers_counters(user)
-      end
-
       {:ok, user}
     else
       %User{} = user -> {:ok, user}
@@ -1126,27 +1084,4 @@ defmodule Pleroma.Web.ActivityPub.Transmogrifier do
     data
     |> maybe_fix_user_url
   end
-
-  def update_following_followers_counters(user) do
-    info = %{}
-
-    following = fetch_counter(user.following_address)
-    info = if following, do: Map.put(info, :following_count, following), else: info
-
-    followers = fetch_counter(user.follower_address)
-    info = if followers, do: Map.put(info, :follower_count, followers), else: info
-
-    User.set_info_cache(user, info)
-  end
-
-  defp fetch_counter(url) do
-    with {:ok, %{body: body, status: code}} when code in 200..299 <-
-           Pleroma.HTTP.get(
-             url,
-             [{:Accept, "application/activity+json"}]
-           ),
-         {:ok, data} <- Jason.decode(body) do
-      data["totalItems"]
-    end
-  end
 end

lib/pleroma/web/activity_pub/utils.ex

@@ -18,6 +18,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
   import Ecto.Query
 
   require Logger
+  require Pleroma.Constants
 
   @supported_object_types ["Article", "Note", "Video", "Page", "Question", "Answer"]
   @supported_report_states ~w(open closed resolved)
@@ -250,20 +251,6 @@ defmodule Pleroma.Web.ActivityPub.Utils do
 
   def insert_full_object(map), do: {:ok, map, nil}
 
-  def update_object_in_activities(%{data: %{"id" => id}} = object) do
-    # TODO
-    # Update activities that already had this. Could be done in a seperate process.
-    # Alternatively, just don't do this and fetch the current object each time. Most
-    # could probably be taken from cache.
-    relevant_activities = Activity.get_all_create_by_object_ap_id(id)
-
-    Enum.map(relevant_activities, fn activity ->
-      new_activity_data = activity.data |> Map.put("object", object.data)
-      changeset = Changeset.change(activity, data: new_activity_data)
-      Repo.update(changeset)
-    end)
-  end
-
   #### Like-related helpers
 
   @doc """
@@ -346,8 +333,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
            |> Map.put("#{property}_count", length(element))
            |> Map.put("#{property}s", element),
          changeset <- Changeset.change(object, data: new_data),
-         {:ok, object} <- Object.update_and_set_cache(changeset),
+         {:ok, object} <- Object.update_and_set_cache(changeset) do
-         _ <- update_object_in_activities(object) do
       {:ok, object}
     end
   end
@@ -388,6 +374,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
         [state, actor, object]
       )
 
+      User.set_follow_state_cache(actor, object, state)
       activity = Activity.get_by_id(activity.id)
       {:ok, activity}
     rescue
@@ -396,12 +383,16 @@ defmodule Pleroma.Web.ActivityPub.Utils do
     end
   end
 
-  def update_follow_state(%Activity{} = activity, state) do
+  def update_follow_state(
+        %Activity{data: %{"actor" => actor, "object" => object}} = activity,
+        state
+      ) do
     with new_data <-
            activity.data
            |> Map.put("state", state),
          changeset <- Changeset.change(activity, data: new_data),
-         {:ok, activity} <- Repo.update(changeset) do
+         {:ok, activity} <- Repo.update(changeset),
+         _ <- User.set_follow_state_cache(actor, object, state) do
       {:ok, activity}
     end
   end
@@ -418,7 +409,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
       "type" => "Follow",
       "actor" => follower_id,
       "to" => [followed_id],
-      "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
+      "cc" => [Pleroma.Constants.as_public()],
       "object" => followed_id,
       "state" => "pending"
     }
@@ -510,7 +501,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
       "actor" => ap_id,
       "object" => id,
       "to" => [user.follower_address, object.data["actor"]],
-      "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
+      "cc" => [Pleroma.Constants.as_public()],
       "context" => object.data["context"]
     }
 
@@ -530,7 +521,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
       "actor" => ap_id,
       "object" => activity.data,
       "to" => [user.follower_address, activity.data["actor"]],
-      "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
+      "cc" => [Pleroma.Constants.as_public()],
       "context" => context
     }
 
@@ -547,7 +538,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
       "actor" => ap_id,
       "object" => activity.data,
       "to" => [user.follower_address, activity.data["actor"]],
-      "cc" => ["https://www.w3.org/ns/activitystreams#Public"],
+      "cc" => [Pleroma.Constants.as_public()],
       "context" => context
     }
 
@@ -556,7 +547,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
 
   def add_announce_to_object(
         %Activity{
-          data: %{"actor" => actor, "cc" => ["https://www.w3.org/ns/activitystreams#Public"]}
+          data: %{"actor" => actor, "cc" => [Pleroma.Constants.as_public()]}
         },
         object
       ) do
@@ -765,7 +756,7 @@ defmodule Pleroma.Web.ActivityPub.Utils do
        ) do
     cc = Map.get(data, "cc", [])
     follower_address = User.get_cached_by_ap_id(data["actor"]).follower_address
-    public = "https://www.w3.org/ns/activitystreams#Public"
+    public = Pleroma.Constants.as_public()
 
     case visibility do
       "public" ->

lib/pleroma/web/activity_pub/views/object_view.ex

@@ -66,8 +66,10 @@ defmodule Pleroma.Web.ActivityPub.ObjectView do
       "orderedItems" => items
     }
 
-    if offset < total do
+    if offset + length(items) < total do
       Map.put(map, "next", "#{iri}?page=#{page + 1}")
+    else
+      map
     end
   end
 end

lib/pleroma/web/activity_pub/views/user_view.ex

@@ -65,7 +65,7 @@ defmodule Pleroma.Web.ActivityPub.UserView do
     do: render("service.json", %{user: user})
 
   def render("user.json", %{user: %User{nickname: "internal." <> _} = user}),
-    do: render("service.json", %{user: user})
+    do: render("service.json", %{user: user}) |> Map.put("preferredUsername", user.nickname)
 
   def render("user.json", %{user: user}) do
     {:ok, user} = User.ensure_keys_present(user)
@@ -80,6 +80,17 @@ defmodule Pleroma.Web.ActivityPub.UserView do
       |> Transmogrifier.add_emoji_tags()
       |> Map.get("tag", [])
 
+    fields =
+      user.info
+      |> User.Info.fields()
+      |> Enum.map(fn %{"name" => name, "value" => value} ->
+        %{
+          "name" => Pleroma.HTML.strip_tags(name),
+          "value" => Pleroma.HTML.filter_tags(value, Pleroma.HTML.Scrubber.LinksOnly)
+        }
+      end)
+      |> Enum.map(&Map.put(&1, "type", "PropertyValue"))
+
     %{
       "id" => user.ap_id,
       "type" => "Person",
@@ -98,6 +109,7 @@ defmodule Pleroma.Web.ActivityPub.UserView do
         "publicKeyPem" => public_key
       },
       "endpoints" => endpoints,
+      "attachment" => fields,
       "tag" => (user.info.source_data["tag"] || []) ++ user_tags
     }
     |> Map.merge(maybe_make_image(&User.avatar_url/2, "icon", user))

lib/pleroma/web/activity_pub/visibility.ex

@@ -8,14 +8,14 @@ defmodule Pleroma.Web.ActivityPub.Visibility do
   alias Pleroma.Repo
   alias Pleroma.User
 
-  @public "https://www.w3.org/ns/activitystreams#Public"
+  require Pleroma.Constants
 
   @spec is_public?(Object.t() | Activity.t() | map()) :: boolean()
   def is_public?(%Object{data: %{"type" => "Tombstone"}}), do: false
   def is_public?(%Object{data: data}), do: is_public?(data)
   def is_public?(%Activity{data: data}), do: is_public?(data)
   def is_public?(%{"directMessage" => true}), do: false
-  def is_public?(data), do: @public in (data["to"] ++ (data["cc"] || []))
+  def is_public?(data), do: Pleroma.Constants.as_public() in (data["to"] ++ (data["cc"] || []))
 
   def is_private?(activity) do
     with false <- is_public?(activity),
@@ -73,10 +73,10 @@ defmodule Pleroma.Web.ActivityPub.Visibility do
     cc = object.data["cc"] || []
 
     cond do
-      @public in to ->
+      Pleroma.Constants.as_public() in to ->
         "public"
 
-      @public in cc ->
+      Pleroma.Constants.as_public() in cc ->
         "unlisted"
 
       # this should use the sql for the object's activity

lib/pleroma/web/admin_api/admin_api_controller.ex

@@ -379,6 +379,16 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
     end
   end
 
+  def migrate_to_db(conn, _params) do
+    Mix.Tasks.Pleroma.Config.run(["migrate_to_db"])
+    json(conn, %{})
+  end
+
+  def migrate_from_db(conn, _params) do
+    Mix.Tasks.Pleroma.Config.run(["migrate_from_db", Pleroma.Config.get(:env), "true"])
+    json(conn, %{})
+  end
+
   def config_show(conn, _params) do
     configs = Pleroma.Repo.all(Config)
 
@@ -392,9 +402,9 @@ defmodule Pleroma.Web.AdminAPI.AdminAPIController do
       if Pleroma.Config.get([:instance, :dynamic_configuration]) do
         updated =
           Enum.map(configs, fn
-            %{"group" => group, "key" => key, "delete" => "true"} ->
+            %{"group" => group, "key" => key, "delete" => "true"} = params ->
-              {:ok, _} = Config.delete(%{group: group, key: key})
+              {:ok, config} = Config.delete(%{group: group, key: key, subkeys: params["subkeys"]})
-              nil
+              config
 
             %{"group" => group, "key" => key, "value" => value} ->
               {:ok, config} = Config.update_or_create(%{group: group, key: key, value: value})

lib/pleroma/web/admin_api/config.ex

@@ -55,8 +55,19 @@ defmodule Pleroma.Web.AdminAPI.Config do
 
   @spec delete(map()) :: {:ok, Config.t()} | {:error, Changeset.t()}
   def delete(params) do
-    with %Config{} = config <- Config.get_by_params(params) do
+    with %Config{} = config <- Config.get_by_params(Map.delete(params, :subkeys)) do
+      if params[:subkeys] do
+        updated_value =
+          Keyword.drop(
+            :erlang.binary_to_term(config.value),
+            Enum.map(params[:subkeys], &do_transform_string(&1))
+          )
+
+        Config.update(config, %{value: updated_value})
+      else
         Repo.delete(config)
+        {:ok, nil}
+      end
     else
       nil ->
         err =

lib/pleroma/web/auth/authenticator.ex

@@ -21,8 +21,7 @@ defmodule Pleroma.Web.Auth.Authenticator do
   def create_from_registration(plug, registration),
     do: implementation().create_from_registration(plug, registration)
 
-  @callback get_registration(Plug.Conn.t()) ::
+  @callback get_registration(Plug.Conn.t()) :: {:ok, Registration.t()} | {:error, any()}
-              {:ok, Registration.t()} | {:error, any()}
   def get_registration(plug), do: implementation().get_registration(plug)
 
   @callback handle_error(Plug.Conn.t(), any()) :: any()

lib/pleroma/web/chat_channel.ex

@@ -33,9 +33,11 @@ defmodule Pleroma.Web.ChatChannel do
 end
 
 defmodule Pleroma.Web.ChatChannel.ChatChannelState do
+  use Agent
+
   @max_messages 20
 
-  def start_link do
+  def start_link(_) do
     Agent.start_link(fn -> %{max_id: 1, messages: []} end, name: __MODULE__)
   end
 

lib/pleroma/web/common_api/common_api.ex

@@ -5,6 +5,7 @@
 defmodule Pleroma.Web.CommonAPI do
   alias Pleroma.Activity
   alias Pleroma.ActivityExpiration
+  alias Pleroma.Conversation.Participation
   alias Pleroma.Formatter
   alias Pleroma.Object
   alias Pleroma.ThreadMute
@@ -172,21 +173,25 @@ defmodule Pleroma.Web.CommonAPI do
     end)
   end
 
-  def get_visibility(%{"visibility" => visibility}, in_reply_to)
+  def get_visibility(_, _, %Participation{}) do
+    {"direct", "direct"}
+  end
+
+  def get_visibility(%{"visibility" => visibility}, in_reply_to, _)
       when visibility in ~w{public unlisted private direct},
       do: {visibility, get_replied_to_visibility(in_reply_to)}
 
-  def get_visibility(%{"visibility" => "list:" <> list_id}, in_reply_to) do
+  def get_visibility(%{"visibility" => "list:" <> list_id}, in_reply_to, _) do
     visibility = {:list, String.to_integer(list_id)}
     {visibility, get_replied_to_visibility(in_reply_to)}
   end
 
-  def get_visibility(_, in_reply_to) when not is_nil(in_reply_to) do
+  def get_visibility(_, in_reply_to, _) when not is_nil(in_reply_to) do
     visibility = get_replied_to_visibility(in_reply_to)
     {visibility, visibility}
   end
 
-  def get_visibility(_, in_reply_to), do: {"public", get_replied_to_visibility(in_reply_to)}
+  def get_visibility(_, in_reply_to, _), do: {"public", get_replied_to_visibility(in_reply_to)}
 
   def get_replied_to_visibility(nil), do: nil
 
@@ -212,7 +217,9 @@ defmodule Pleroma.Web.CommonAPI do
     with status <- String.trim(status),
          attachments <- attachments_from_ids(data),
          in_reply_to <- get_replied_to_activity(data["in_reply_to_status_id"]),
-         {visibility, in_reply_to_visibility} <- get_visibility(data, in_reply_to),
+         in_reply_to_conversation <- Participation.get(data["in_reply_to_conversation_id"]),
+         {visibility, in_reply_to_visibility} <-
+           get_visibility(data, in_reply_to, in_reply_to_conversation),
          {_, false} <-
            {:private_to_public, in_reply_to_visibility == "direct" && visibility != "direct"},
          {content_html, mentions, tags} <-
@@ -225,8 +232,9 @@ defmodule Pleroma.Web.CommonAPI do
          mentioned_users <- for({_, mentioned_user} <- mentions, do: mentioned_user.ap_id),
          addressed_users <- get_addressed_users(mentioned_users, data["to"]),
          {poll, poll_emoji} <- make_poll_data(data),
-         {to, cc} <- get_to_and_cc(user, addressed_users, in_reply_to, visibility),
+         {to, cc} <-
-         context <- make_context(in_reply_to),
+           get_to_and_cc(user, addressed_users, in_reply_to, visibility, in_reply_to_conversation),
+         context <- make_context(in_reply_to, in_reply_to_conversation),
          cw <- data["spoiler_text"] || "",
          sensitive <- data["sensitive"] || Enum.member?(tags, {"#nsfw", "nsfw"}),
          {:ok, expires_at} <- check_expiry_date(data["expires_at"]),
@@ -321,8 +329,7 @@ defmodule Pleroma.Web.CommonAPI do
            }
          } = activity <- get_by_id_or_ap_id(id_or_ap_id),
          true <- Visibility.is_public?(activity),
-         %{valid?: true} = info_changeset <-
+         %{valid?: true} = info_changeset <- User.Info.add_pinnned_activity(user.info, activity),
-           User.Info.add_pinnned_activity(user.info, activity),
          changeset <-
            Ecto.Changeset.change(user) |> Ecto.Changeset.put_embed(:info, info_changeset),
          {:ok, _user} <- User.update_and_set_cache(changeset) do

lib/pleroma/web/common_api/utils.ex

@@ -8,6 +8,7 @@ defmodule Pleroma.Web.CommonAPI.Utils do
   alias Calendar.Strftime
   alias Pleroma.Activity
   alias Pleroma.Config
+  alias Pleroma.Conversation.Participation
   alias Pleroma.Formatter
   alias Pleroma.Object
   alias Pleroma.Plugs.AuthenticationPlug
@@ -19,11 +20,17 @@ defmodule Pleroma.Web.CommonAPI.Utils do
   alias Pleroma.Web.MediaProxy
 
   require Logger
+  require Pleroma.Constants
 
   # This is a hack for twidere.
   def get_by_id_or_ap_id(id) do
     activity =
-      Activity.get_by_id_with_object(id) || Activity.get_create_by_object_ap_id_with_object(id)
+      with true <- Pleroma.FlakeId.is_flake_id?(id),
+           %Activity{} = activity <- Activity.get_by_id_with_object(id) do
+        activity
+      else
+        _ -> Activity.get_create_by_object_ap_id_with_object(id)
+      end
 
     activity &&
       if activity.data["type"] == "Create" do
@@ -41,32 +48,61 @@ defmodule Pleroma.Web.CommonAPI.Utils do
 
   def get_replied_to_activity(_), do: nil
 
-  def attachments_from_ids(data) do
+  def attachments_from_ids(%{"media_ids" => ids, "descriptions" => desc} = _) do
-    if Map.has_key?(data, "descriptions") do
+    attachments_from_ids_descs(ids, desc)
-      attachments_from_ids_descs(data["media_ids"], data["descriptions"])
-    else
-      attachments_from_ids_no_descs(data["media_ids"])
-    end
   end
 
-  def attachments_from_ids_no_descs(ids) do
+  def attachments_from_ids(%{"media_ids" => ids} = _) do
-    Enum.map(ids || [], fn media_id ->
+    attachments_from_ids_no_descs(ids)
-      Repo.get(Object, media_id).data
-    end)
   end
 
+  def attachments_from_ids(_), do: []
+
+  def attachments_from_ids_no_descs([]), do: []
+
+  def attachments_from_ids_no_descs(ids) do
+    Enum.map(ids, fn media_id ->
+      case Repo.get(Object, media_id) do
+        %Object{data: data} = _ -> data
+        _ -> nil
+      end
+    end)
+    |> Enum.filter(& &1)
+  end
+
+  def attachments_from_ids_descs([], _), do: []
+
   def attachments_from_ids_descs(ids, descs_str) do
     {_, descs} = Jason.decode(descs_str)
 
-    Enum.map(ids || [], fn media_id ->
+    Enum.map(ids, fn media_id ->
-      Map.put(Repo.get(Object, media_id).data, "name", descs[media_id])
+      case Repo.get(Object, media_id) do
+        %Object{data: data} = _ ->
+          Map.put(data, "name", descs[media_id])
+
+        _ ->
+          nil
+      end
     end)
+    |> Enum.filter(& &1)
   end
 
-  @spec get_to_and_cc(User.t(), list(String.t()), Activity.t() | nil, String.t()) ::
+  @spec get_to_and_cc(
+          User.t(),
+          list(String.t()),
+          Activity.t() | nil,
+          String.t(),
+          Participation.t() | nil
+        ) ::
           {list(String.t()), list(String.t())}
-  def get_to_and_cc(user, mentioned_users, inReplyTo, "public") do
+
-    to = ["https://www.w3.org/ns/activitystreams#Public" | mentioned_users]
+  def get_to_and_cc(_, _, _, _, %Participation{} = participation) do
+    participation = Repo.preload(participation, :recipients)
+    {Enum.map(participation.recipients, & &1.ap_id), []}
+  end
+
+  def get_to_and_cc(user, mentioned_users, inReplyTo, "public", _) do
+    to = [Pleroma.Constants.as_public() | mentioned_users]
     cc = [user.follower_address]
 
     if inReplyTo do
@@ -76,9 +112,9 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     end
   end
 
-  def get_to_and_cc(user, mentioned_users, inReplyTo, "unlisted") do
+  def get_to_and_cc(user, mentioned_users, inReplyTo, "unlisted", _) do
     to = [user.follower_address | mentioned_users]
-    cc = ["https://www.w3.org/ns/activitystreams#Public"]
+    cc = [Pleroma.Constants.as_public()]
 
     if inReplyTo do
       {Enum.uniq([inReplyTo.data["actor"] | to]), cc}
@@ -87,12 +123,12 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     end
   end
 
-  def get_to_and_cc(user, mentioned_users, inReplyTo, "private") do
+  def get_to_and_cc(user, mentioned_users, inReplyTo, "private", _) do
-    {to, cc} = get_to_and_cc(user, mentioned_users, inReplyTo, "direct")
+    {to, cc} = get_to_and_cc(user, mentioned_users, inReplyTo, "direct", nil)
     {[user.follower_address | to], cc}
   end
 
-  def get_to_and_cc(_user, mentioned_users, inReplyTo, "direct") do
+  def get_to_and_cc(_user, mentioned_users, inReplyTo, "direct", _) do
     if inReplyTo do
       {Enum.uniq([inReplyTo.data["actor"] | mentioned_users]), []}
     else
@@ -100,7 +136,7 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     end
   end
 
-  def get_to_and_cc(_user, mentions, _inReplyTo, {:list, _}), do: {mentions, []}
+  def get_to_and_cc(_user, mentions, _inReplyTo, {:list, _}, _), do: {mentions, []}
 
   def get_addressed_users(_, to) when is_list(to) do
     User.get_ap_ids_by_nicknames(to)
@@ -230,8 +266,12 @@ defmodule Pleroma.Web.CommonAPI.Utils do
 
   defp maybe_add_nsfw_tag(data, _), do: data
 
-  def make_context(%Activity{data: %{"context" => context}}), do: context
+  def make_context(_, %Participation{} = participation) do
-  def make_context(_), do: Utils.generate_context_id()
+    Repo.preload(participation, :conversation).conversation.ap_id
+  end
+
+  def make_context(%Activity{data: %{"context" => context}}, _), do: context
+  def make_context(_, _), do: Utils.generate_context_id()
 
   def maybe_add_attachments(parsed, _attachments, true = _no_links), do: parsed
 
@@ -241,20 +281,18 @@ defmodule Pleroma.Web.CommonAPI.Utils do
   end
 
   def add_attachments(text, attachments) do
-    attachment_text =
+    attachment_text = Enum.map(attachments, &build_attachment_link/1)
-      Enum.map(attachments, fn
+    Enum.join([text | attachment_text], "<br>")
-        %{"url" => [%{"href" => href} | _]} = attachment ->
+  end
+
+  defp build_attachment_link(%{"url" => [%{"href" => href} | _]} = attachment) do
     name = attachment["name"] || URI.decode(Path.basename(href))
     href = MediaProxy.url(href)
     "<a href=\"#{href}\" class='attachment'>#{shortname(name)}</a>"
-
-        _ ->
-          ""
-      end)
-
-    Enum.join([text | attachment_text], "<br>")
   end
 
+  defp build_attachment_link(_), do: ""
+
   def format_input(text, format, options \\ [])
 
   @doc """
@@ -314,7 +352,7 @@ defmodule Pleroma.Web.CommonAPI.Utils do
         sensitive \\ false,
         merge \\ %{}
       ) do
-    object = %{
+    %{
       "type" => "Note",
       "to" => to,
       "cc" => cc,
@@ -324,18 +362,20 @@ defmodule Pleroma.Web.CommonAPI.Utils do
       "context" => context,
       "attachment" => attachments,
       "actor" => actor,
-      "tag" => tags |> Enum.map(fn {_, tag} -> tag end) |> Enum.uniq()
+      "tag" => Keyword.values(tags) |> Enum.uniq()
     }
+    |> add_in_reply_to(in_reply_to)
+    |> Map.merge(merge)
+  end
 
-    object =
+  defp add_in_reply_to(object, nil), do: object
-      with false <- is_nil(in_reply_to),
+
-           %Object{} = in_reply_to_object <- Object.normalize(in_reply_to) do
+  defp add_in_reply_to(object, in_reply_to) do
+    with %Object{} = in_reply_to_object <- Object.normalize(in_reply_to) do
       Map.put(object, "inReplyTo", in_reply_to_object.data["id"])
     else
       _ -> object
     end
-
-    Map.merge(object, merge)
   end
 
   def format_naive_asctime(date) do
@@ -367,17 +407,16 @@ defmodule Pleroma.Web.CommonAPI.Utils do
     |> String.replace(~r/(\.\d+)?$/, ".000Z", global: false)
   end
 
-  def to_masto_date(date) do
+  def to_masto_date(date) when is_binary(date) do
-    try do
+    with {:ok, date} <- NaiveDateTime.from_iso8601(date) do
-      date
+      to_masto_date(date)
-      |> NaiveDateTime.from_iso8601!()
+    else
-      |> NaiveDateTime.to_iso8601()
+      _ -> ""
-      |> String.replace(~r/(\.\d+)?$/, ".000Z", global: false)
-    rescue
-      _e -> ""
     end
   end
 
+  def to_masto_date(_), do: ""
+
   defp shortname(name) do
     if String.length(name) < 30 do
       name
@@ -422,7 +461,7 @@ defmodule Pleroma.Web.CommonAPI.Utils do
 
     object_data =
       cond do
-        !is_nil(object) ->
+        not is_nil(object) ->
           object.data
 
         is_map(data["object"]) ->
@@ -466,9 +505,9 @@ defmodule Pleroma.Web.CommonAPI.Utils do
 
   def maybe_extract_mentions(%{"tag" => tag}) do
     tag
-    |> Enum.filter(fn x -> is_map(x) end)
+    |> Enum.filter(fn x -> is_map(x) && x["type"] == "Mention" end)
-    |> Enum.filter(fn x -> x["type"] == "Mention" end)
     |> Enum.map(fn x -> x["href"] end)
+    |> Enum.uniq()
   end
 
   def maybe_extract_mentions(_), do: []

lib/pleroma/web/controller_helper.ex

@@ -33,4 +33,80 @@ defmodule Pleroma.Web.ControllerHelper do
   end
 
   defp param_to_integer(_, default), do: default
+
+  def add_link_headers(
+        conn,
+        method,
+        activities,
+        param \\ nil,
+        params \\ %{},
+        func3 \\ nil,
+        func4 \\ nil
+      ) do
+    params =
+      conn.params
+      |> Map.drop(["since_id", "max_id", "min_id"])
+      |> Map.merge(params)
+
+    last = List.last(activities)
+
+    func3 = func3 || (&mastodon_api_url/3)
+    func4 = func4 || (&mastodon_api_url/4)
+
+    if last do
+      max_id = last.id
+
+      limit =
+        params
+        |> Map.get("limit", "20")
+        |> String.to_integer()
+
+      min_id =
+        if length(activities) <= limit do
+          activities
+          |> List.first()
+          |> Map.get(:id)
+        else
+          activities
+          |> Enum.at(limit * -1)
+          |> Map.get(:id)
+        end
+
+      {next_url, prev_url} =
+        if param do
+          {
+            func4.(
+              Pleroma.Web.Endpoint,
+              method,
+              param,
+              Map.merge(params, %{max_id: max_id})
+            ),
+            func4.(
+              Pleroma.Web.Endpoint,
+              method,
+              param,
+              Map.merge(params, %{min_id: min_id})
+            )
+          }
+        else
+          {
+            func3.(
+              Pleroma.Web.Endpoint,
+              method,
+              Map.merge(params, %{max_id: max_id})
+            ),
+            func3.(
+              Pleroma.Web.Endpoint,
+              method,
+              Map.merge(params, %{min_id: min_id})
+            )
+          }
+        end
+
+      conn
+      |> put_resp_header("link", "<#{next_url}>; rel=\"next\", <#{prev_url}>; rel=\"prev\"")
+    else
+      conn
+    end
+  end
 end

lib/pleroma/web/fallback_redirect_controller.ex

@@ -0,0 +1,77 @@
+# Pleroma: A lightweight social networking server
+# Copyright © 2017-2019 Pleroma Authors <https://pleroma.social/>
+# SPDX-License-Identifier: AGPL-3.0-only
+
+defmodule Fallback.RedirectController do
+  use Pleroma.Web, :controller
+  require Logger
+  alias Pleroma.User
+  alias Pleroma.Web.Metadata
+
+  def api_not_implemented(conn, _params) do
+    conn
+    |> put_status(404)
+    |> json(%{error: "Not implemented"})
+  end
+
+  def redirector(conn, _params, code \\ 200)
+
+  # redirect to admin section
+  # /pleroma/admin -> /pleroma/admin/
+  #
+  def redirector(conn, %{"path" => ["pleroma", "admin"]} = _, _code) do
+    redirect(conn, to: "/pleroma/admin/")
+  end
+
+  def redirector(conn, _params, code) do
+    conn
+    |> put_resp_content_type("text/html")
+    |> send_file(code, index_file_path())
+  end
+
+  def redirector_with_meta(conn, %{"maybe_nickname_or_id" => maybe_nickname_or_id} = params) do
+    with %User{} = user <- User.get_cached_by_nickname_or_id(maybe_nickname_or_id) do
+      redirector_with_meta(conn, %{user: user})
+    else
+      nil ->
+        redirector(conn, params)
+    end
+  end
+
+  def redirector_with_meta(conn, params) do
+    {:ok, index_content} = File.read(index_file_path())
+
+    tags =
+      try do
+        Metadata.build_tags(params)
+      rescue
+        e ->
+          Logger.error(
+            "Metadata rendering for #{conn.request_path} failed.\n" <>
+              Exception.format(:error, e, __STACKTRACE__)
+          )
+
+          ""
+      end
+
+    response = String.replace(index_content, "<!--server-generated-meta-->", tags)
+
+    conn
+    |> put_resp_content_type("text/html")
+    |> send_resp(200, response)
+  end
+
+  def index_file_path do
+    Pleroma.Plugs.InstanceStatic.file_path("index.html")
+  end
+
+  def registration_page(conn, params) do
+    redirector(conn, params)
+  end
+
+  def empty(conn, _params) do
+    conn
+    |> put_status(204)
+    |> text("")
+  end
+end

lib/pleroma/web/federator/retry_queue.ex

@@ -13,7 +13,7 @@ defmodule Pleroma.Web.Federator.RetryQueue do
     {:ok, %{args | queue_table: queue_table, running_jobs: :sets.new()}}
   end
 
-  def start_link do
+  def start_link(_) do
     enabled =
       if Pleroma.Config.get(:env) == :test,
         do: true,

lib/pleroma/web/mailer/subscription_controller.ex

@@ -0,0 +1,20 @@
+defmodule Pleroma.Web.Mailer.SubscriptionController do
+  use Pleroma.Web, :controller
+
+  alias Pleroma.JWT
+  alias Pleroma.Repo
+  alias Pleroma.User
+
+  def unsubscribe(conn, %{"token" => encoded_token}) do
+    with {:ok, token} <- Base.decode64(encoded_token),
+         {:ok, claims} <- JWT.verify_and_validate(token),
+         %{"act" => %{"unsubscribe" => type}, "sub" => uid} <- claims,
+         %User{} = user <- Repo.get(User, uid),
+         {:ok, _user} <- User.switch_email_notifications(user, type, false) do
+      render(conn, "unsubscribe_success.html", email: user.email)
+    else
+      _err ->
+        render(conn, "unsubscribe_failure.html")
+    end
+  end
+end

lib/pleroma/web/mastodon_api/mastodon_api.ex

@@ -13,10 +13,8 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPI do
   alias Pleroma.User
   alias Pleroma.Web.CommonAPI
 
+  @spec follow(User.t(), User.t(), map) :: {:ok, User.t()} | {:error, String.t()}
   def follow(follower, followed, params \\ %{}) do
-    options = cast_params(params)
-    reblogs = options[:reblogs]
-
     result =
       if not User.following?(follower, followed) do
         CommonAPI.follow(follower, followed)
@@ -24,19 +22,25 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPI do
         {:ok, follower, followed, nil}
       end
 
-    with {:ok, follower, followed, _} <- result do
+    with {:ok, follower, _followed, _} <- result do
-      reblogs
+      options = cast_params(params)
-      |> case do
+
-        false -> CommonAPI.hide_reblogs(follower, followed)
+      case reblogs_visibility(options[:reblogs], result) do
-        _ -> CommonAPI.show_reblogs(follower, followed)
-      end
-      |> case do
         {:ok, follower} -> {:ok, follower}
         _ -> {:ok, follower}
       end
     end
   end
 
+  defp reblogs_visibility(false, {:ok, follower, followed, _}) do
+    CommonAPI.hide_reblogs(follower, followed)
+  end
+
+  defp reblogs_visibility(_, {:ok, follower, followed, _}) do
+    CommonAPI.show_reblogs(follower, followed)
+  end
+
+  @spec get_followers(User.t(), map()) :: list(User.t())
   def get_followers(user, params \\ %{}) do
     user
     |> User.get_followers_query()

lib/pleroma/web/mastodon_api/mastodon_api_controller.ex

@@ -4,6 +4,10 @@
 
 defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   use Pleroma.Web, :controller
+
+  import Pleroma.Web.ControllerHelper,
+    only: [json_response: 3, add_link_headers: 5, add_link_headers: 4, add_link_headers: 3]
+
   alias Ecto.Changeset
   alias Pleroma.Activity
   alias Pleroma.Bookmark
@@ -46,6 +50,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   import Ecto.Query
 
   require Logger
+  require Pleroma.Constants
 
   @rate_limited_relations_actions ~w(follow unfollow)a
 
@@ -74,6 +79,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   plug(RateLimiter, :app_account_creation when action == :account_register)
   plug(RateLimiter, :search when action in [:search, :search2, :account_search])
   plug(RateLimiter, :password_reset when action == :password_reset)
+  plug(RateLimiter, :account_confirmation_resend when action == :account_confirmation_resend)
 
   @local_mastodon_name "Mastodon-Local"
 
@@ -132,7 +138,9 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     emojis_text = (user_params["display_name"] || "") <> (user_params["note"] || "")
 
     user_info_emojis =
-      ((user.info.emoji || []) ++ Formatter.get_emoji_map(emojis_text))
+      user.info
+      |> Map.get(:emoji, [])
+      |> Enum.concat(Formatter.get_emoji_map(emojis_text))
       |> Enum.dedup()
 
     info_params =
@@ -151,6 +159,12 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
         end)
       end)
       |> add_if_present(params, "default_scope", :default_scope)
+      |> add_if_present(params, "fields", :fields, fn fields ->
+        fields = Enum.map(fields, fn f -> Map.update!(f, "value", &AutoLinker.link(&1)) end)
+
+        {:ok, fields}
+      end)
+      |> add_if_present(params, "fields", :raw_fields)
       |> add_if_present(params, "pleroma_settings_store", :pleroma_settings_store, fn value ->
         {:ok, Map.merge(user.info.pleroma_settings_store, value)}
       end)
@@ -337,71 +351,6 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     json(conn, mastodon_emoji)
   end
 
-  defp add_link_headers(conn, method, activities, param \\ nil, params \\ %{}) do
-    params =
-      conn.params
-      |> Map.drop(["since_id", "max_id", "min_id"])
-      |> Map.merge(params)
-
-    last = List.last(activities)
-
-    if last do
-      max_id = last.id
-
-      limit =
-        params
-        |> Map.get("limit", "20")
-        |> String.to_integer()
-
-      min_id =
-        if length(activities) <= limit do
-          activities
-          |> List.first()
-          |> Map.get(:id)
-        else
-          activities
-          |> Enum.at(limit * -1)
-          |> Map.get(:id)
-        end
-
-      {next_url, prev_url} =
-        if param do
-          {
-            mastodon_api_url(
-              Pleroma.Web.Endpoint,
-              method,
-              param,
-              Map.merge(params, %{max_id: max_id})
-            ),
-            mastodon_api_url(
-              Pleroma.Web.Endpoint,
-              method,
-              param,
-              Map.merge(params, %{min_id: min_id})
-            )
-          }
-        else
-          {
-            mastodon_api_url(
-              Pleroma.Web.Endpoint,
-              method,
-              Map.merge(params, %{max_id: max_id})
-            ),
-            mastodon_api_url(
-              Pleroma.Web.Endpoint,
-              method,
-              Map.merge(params, %{min_id: min_id})
-            )
-          }
-        end
-
-      conn
-      |> put_resp_header("link", "<#{next_url}>; rel=\"next\", <#{prev_url}>; rel=\"prev\"")
-    else
-      conn
-    end
-  end
-
   def home_timeline(%{assigns: %{user: user}} = conn, params) do
     params =
       params
@@ -430,6 +379,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
       |> Map.put("local_only", local_only)
       |> Map.put("blocking_user", user)
       |> Map.put("muting_user", user)
+      |> Map.put("user", user)
       |> ActivityPub.fetch_public_activities()
       |> Enum.reverse()
 
@@ -491,12 +441,9 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
          activities <-
            ActivityPub.fetch_activities_for_context(activity.data["context"], %{
              "blocking_user" => user,
-             "user" => user
+             "user" => user,
+             "exclude_id" => activity.id
            }),
-         activities <-
-           activities |> Enum.filter(fn %{id: aid} -> to_string(aid) != to_string(id) end),
-         activities <-
-           activities |> Enum.filter(fn %{data: %{"type" => type}} -> type == "Create" end),
          grouped_activities <- Enum.group_by(activities, fn %{id: id} -> id < activity.id end) do
       result = %{
         ancestors:
@@ -531,8 +478,8 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
       |> put_view(StatusView)
       |> try_render("poll.json", %{object: object, for: user})
     else
-      nil -> render_error(conn, :not_found, "Record not found")
+      error when is_nil(error) or error == false ->
-      false -> render_error(conn, :not_found, "Record not found")
+        render_error(conn, :not_found, "Record not found")
     end
   end
 
@@ -880,8 +827,8 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   end
 
   def favourited_by(%{assigns: %{user: user}} = conn, %{"id" => id}) do
-    with %Activity{data: %{"object" => object}} <- Activity.get_by_id(id),
+    with %Activity{} = activity <- Activity.get_by_id_with_object(id),
-         %Object{data: %{"likes" => likes}} <- Object.normalize(object) do
+         %Object{data: %{"likes" => likes}} <- Object.normalize(activity) do
       q = from(u in User, where: u.ap_id in ^likes)
 
       users =
@@ -897,8 +844,8 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
   end
 
   def reblogged_by(%{assigns: %{user: user}} = conn, %{"id" => id}) do
-    with %Activity{data: %{"object" => object}} <- Activity.get_by_id(id),
+    with %Activity{} = activity <- Activity.get_by_id_with_object(id),
-         %Object{data: %{"announcements" => announces}} <- Object.normalize(object) do
+         %Object{data: %{"announcements" => announces}} <- Object.normalize(activity) do
       q = from(u in User, where: u.ap_id in ^announces)
 
       users =
@@ -939,6 +886,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
       |> Map.put("local_only", local_only)
       |> Map.put("blocking_user", user)
       |> Map.put("muting_user", user)
+      |> Map.put("user", user)
       |> Map.put("tag", tags)
       |> Map.put("tag_all", tag_all)
       |> Map.put("tag_reject", tag_reject)
@@ -1220,10 +1168,9 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
 
       recipients =
         if for_user do
-          ["https://www.w3.org/ns/activitystreams#Public"] ++
+          [Pleroma.Constants.as_public()] ++ [for_user.ap_id | for_user.following]
-            [for_user.ap_id | for_user.following]
         else
-          ["https://www.w3.org/ns/activitystreams#Public"]
+          [Pleroma.Constants.as_public()]
         end
 
       activities =
@@ -1346,6 +1293,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
         params
         |> Map.put("type", "Create")
         |> Map.put("blocking_user", user)
+        |> Map.put("user", user)
         |> Map.put("muting_user", user)
 
       # we must filter the following list for the user to avoid leaking statuses the user
@@ -1686,45 +1634,35 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
         |> String.replace("{{user}}", user)
 
       with {:ok, %{status: 200, body: body}} <-
-             HTTP.get(
+             HTTP.get(url, [], adapter: [recv_timeout: timeout, pool: :default]),
-               url,
-               [],
-               adapter: [
-                 recv_timeout: timeout,
-                 pool: :default
-               ]
-             ),
            {:ok, data} <- Jason.decode(body) do
         data =
           data
           |> Enum.slice(0, limit)
           |> Enum.map(fn x ->
-            Map.put(
+            x
-              x,
+            |> Map.put("id", fetch_suggestion_id(x))
-              "id",
+            |> Map.put("avatar", MediaProxy.url(x["avatar"]))
-              case User.get_or_fetch(x["acct"]) do
+            |> Map.put("avatar_static", MediaProxy.url(x["avatar_static"]))
-                {:ok, %User{id: id}} -> id
-                _ -> 0
-              end
-            )
-          end)
-          |> Enum.map(fn x ->
-            Map.put(x, "avatar", MediaProxy.url(x["avatar"]))
-          end)
-          |> Enum.map(fn x ->
-            Map.put(x, "avatar_static", MediaProxy.url(x["avatar_static"]))
           end)
 
-        conn
+        json(conn, data)
-        |> json(data)
       else
-        e -> Logger.error("Could not retrieve suggestions at fetch #{url}, #{inspect(e)}")
+        e ->
+          Logger.error("Could not retrieve suggestions at fetch #{url}, #{inspect(e)}")
       end
     else
       json(conn, [])
     end
   end
 
+  defp fetch_suggestion_id(attrs) do
+    case User.get_or_fetch(attrs["acct"]) do
+      {:ok, %User{id: id}} -> id
+      _ -> 0
+    end
+  end
+
   def status_card(%{assigns: %{user: user}} = conn, %{"id" => status_id}) do
     with %Activity{} = activity <- Activity.get_by_id(status_id),
          true <- Visibility.visible_for_user?(activity, user) do
@@ -1803,7 +1741,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
 
     conversations =
       Enum.map(participations, fn participation ->
-        ConversationView.render("participation.json", %{participation: participation, user: user})
+        ConversationView.render("participation.json", %{participation: participation, for: user})
       end)
 
     conn
@@ -1816,7 +1754,7 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
            Repo.get_by(Participation, id: participation_id, user_id: user.id),
          {:ok, participation} <- Participation.mark_as_read(participation) do
       participation_view =
-        ConversationView.render("participation.json", %{participation: participation, user: user})
+        ConversationView.render("participation.json", %{participation: participation, for: user})
 
       conn
       |> json(participation_view)
@@ -1839,6 +1777,16 @@ defmodule Pleroma.Web.MastodonAPI.MastodonAPIController do
     end
   end
 
+  def account_confirmation_resend(conn, params) do
+    nickname_or_email = params["email"] || params["nickname"]
+
+    with %User{} = user <- User.get_by_nickname_or_email(nickname_or_email),
+         {:ok, _} <- User.try_send_confirmation_email(user) do
+      conn
+      |> json_response(:no_content, "")
+    end
+  end
+
   def try_render(conn, target, params)
       when is_binary(target) do
     case render(conn, target, params) do

lib/pleroma/web/mastodon_api/views/account_view.ex

@@ -28,7 +28,7 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
       id: to_string(user.id),
       acct: user.nickname,
       username: username_from_nickname(user.nickname),
-      url: user.ap_id
+      url: User.profile_url(user)
     }
   end
 
@@ -37,11 +37,11 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
   end
 
   def render("relationship.json", %{user: %User{} = user, target: %User{} = target}) do
-    follow_activity = Pleroma.Web.ActivityPub.Utils.fetch_latest_follow(user, target)
+    follow_state = User.get_cached_follow_state(user, target)
 
     requested =
-      if follow_activity && !User.following?(target, user) do
+      if follow_state && !User.following?(user, target) do
-        follow_activity.data["state"] == "pending"
+        follow_state == "pending"
       else
         false
       end
@@ -50,13 +50,13 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
       id: to_string(target.id),
       following: User.following?(user, target),
       followed_by: User.following?(target, user),
-      blocking: User.blocks?(user, target),
+      blocking: User.blocks_ap_id?(user, target),
-      blocked_by: User.blocks?(target, user),
+      blocked_by: User.blocks_ap_id?(target, user),
       muting: User.mutes?(user, target),
       muting_notifications: User.muted_notifications?(user, target),
       subscribing: User.subscribed_to?(user, target),
       requested: requested,
-      domain_blocking: false,
+      domain_blocking: User.blocks_domain?(user, target),
       showing_reblogs: User.showing_reblogs?(user, target),
       endorsed: false
     }
@@ -72,6 +72,13 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
     image = User.avatar_url(user) |> MediaProxy.url()
     header = User.banner_url(user) |> MediaProxy.url()
     user_info = User.get_cached_user_info(user)
+
+    following_count =
+      ((!user.info.hide_follows or opts[:for] == user) && user_info.following_count) || 0
+
+    followers_count =
+      ((!user.info.hide_followers or opts[:for] == user) && user_info.follower_count) || 0
+
     bot = (user.info.source_data["type"] || "Person") in ["Application", "Service"]
 
     emojis =
@@ -87,12 +94,18 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
       end)
 
     fields =
-      (user.info.source_data["attachment"] || [])
+      user.info
-      |> Enum.filter(fn %{"type" => t} -> t == "PropertyValue" end)
+      |> User.Info.fields()
-      |> Enum.map(fn fields -> Map.take(fields, ["name", "value"]) end)
+      |> Enum.map(fn %{"name" => name, "value" => value} ->
+        %{
+          "name" => Pleroma.HTML.strip_tags(name),
+          "value" => Pleroma.HTML.filter_tags(value, Pleroma.HTML.Scrubber.LinksOnly)
+        }
+      end)
+
+    raw_fields = Map.get(user.info, :raw_fields, [])
 
     bio = HTML.filter_tags(user.bio, User.html_filter_policy(opts[:for]))
-
     relationship = render("relationship.json", %{user: opts[:for], target: user})
 
     %{
@@ -102,11 +115,11 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
       display_name: display_name,
       locked: user_info.locked,
       created_at: Utils.to_masto_date(user.inserted_at),
-      followers_count: user_info.follower_count,
+      followers_count: followers_count,
-      following_count: user_info.following_count,
+      following_count: following_count,
       statuses_count: user_info.note_count,
       note: bio || "",
-      url: user.ap_id,
+      url: User.profile_url(user),
       avatar: image,
       avatar_static: image,
       header: header,
@@ -117,6 +130,7 @@ defmodule Pleroma.Web.MastodonAPI.AccountView do
       source: %{
         note: HTML.strip_tags((user.bio || "") |> String.replace("<br>", "\n")),
         sensitive: false,
+        fields: raw_fields,
         pleroma: %{}
       },
 

lib/pleroma/web/mastodon_api/views/conversation_view.ex

@@ -11,8 +11,8 @@ defmodule Pleroma.Web.MastodonAPI.ConversationView do
   alias Pleroma.Web.MastodonAPI.AccountView
   alias Pleroma.Web.MastodonAPI.StatusView
 
-  def render("participation.json", %{participation: participation, user: user}) do
+  def render("participation.json", %{participation: participation, for: user}) do
-    participation = Repo.preload(participation, conversation: :users)
+    participation = Repo.preload(participation, conversation: [], recipients: [])
 
     last_activity_id =
       with nil <- participation.last_activity_id do
@@ -28,7 +28,7 @@ defmodule Pleroma.Web.MastodonAPI.ConversationView do
 
     # Conversations return all users except the current user.
     users =
-      participation.conversation.users
+      participation.recipients
       |> Enum.reject(&(&1.id == user.id))
 
     accounts =

lib/pleroma/web/mastodon_api/views/status_view.ex

@@ -5,8 +5,12 @@
 defmodule Pleroma.Web.MastodonAPI.StatusView do
   use Pleroma.Web, :view
 
+  require Pleroma.Constants
+
   alias Pleroma.Activity
   alias Pleroma.ActivityExpiration
+  alias Pleroma.Conversation
+  alias Pleroma.Conversation.Participation
   alias Pleroma.HTML
   alias Pleroma.Object
   alias Pleroma.Repo
@@ -25,19 +29,19 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
   defp get_replied_to_activities(activities) do
     activities
     |> Enum.map(fn
-      %{data: %{"type" => "Create", "object" => object}} ->
+      %{data: %{"type" => "Create"}} = activity ->
-        object = Object.normalize(object)
+        object = Object.normalize(activity)
-        object.data["inReplyTo"] != "" && object.data["inReplyTo"]
+        object && object.data["inReplyTo"] != "" && object.data["inReplyTo"]
 
       _ ->
         nil
     end)
     |> Enum.filter(& &1)
-    |> Activity.create_by_object_ap_id()
+    |> Activity.create_by_object_ap_id_with_object()
     |> Repo.all()
     |> Enum.reduce(%{}, fn activity, acc ->
       object = Object.normalize(activity)
-      Map.put(acc, object.data["id"], activity)
+      if object, do: Map.put(acc, object.data["id"], activity), else: acc
     end)
   end
 
@@ -69,12 +73,14 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
 
   def render("index.json", opts) do
     replied_to_activities = get_replied_to_activities(opts.activities)
+    parallel = unless is_nil(opts[:parallel]), do: opts[:parallel], else: true
 
     opts.activities
     |> safe_render_many(
       StatusView,
       "status.json",
-      Map.put(opts, :replied_to_activities, replied_to_activities)
+      Map.put(opts, :replied_to_activities, replied_to_activities),
+      parallel
     )
   end
 
@@ -89,6 +95,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
     reblogged_activity =
       Activity.create_by_object_ap_id(activity_object.data["id"])
       |> Activity.with_preloaded_bookmark(opts[:for])
+      |> Activity.with_set_thread_muted_field(opts[:for])
       |> Repo.one()
 
     reblogged = render("status.json", Map.put(opts, :activity, reblogged_activity))
@@ -143,6 +150,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
     object = Object.normalize(activity)
 
     user = get_user(activity.data["actor"])
+    user_follower_address = user.follower_address
 
     like_count = object.data["like_count"] || 0
     announcement_count = object.data["announcement_count"] || 0
@@ -158,7 +166,11 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
     mentions =
       (object.data["to"] ++ tag_mentions)
       |> Enum.uniq()
-      |> Enum.map(fn ap_id -> User.get_cached_by_ap_id(ap_id) end)
+      |> Enum.map(fn
+        Pleroma.Constants.as_public() -> nil
+        ^user_follower_address -> nil
+        ap_id -> User.get_cached_by_ap_id(ap_id)
+      end)
       |> Enum.filter(& &1)
       |> Enum.map(fn user -> AccountView.render("mention.json", %{user: user}) end)
 
@@ -178,7 +190,7 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
     thread_muted? =
       case activity.thread_muted? do
         thread_muted? when is_boolean(thread_muted?) -> thread_muted?
-        nil -> CommonAPI.thread_muted?(user, activity)
+        nil -> (opts[:for] && CommonAPI.thread_muted?(opts[:for], activity)) || false
       end
 
     attachment_data = object.data["attachment"] || []
@@ -232,7 +244,20 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
       if user.local do
         Pleroma.Web.Router.Helpers.o_status_url(Pleroma.Web.Endpoint, :notice, activity)
       else
-        object.data["external_url"] || object.data["id"]
+        object.data["url"] || object.data["external_url"] || object.data["id"]
+      end
+
+    direct_conversation_id =
+      with {_, true} <- {:include_id, opts[:with_direct_conversation_id]},
+           {_, %User{} = for_user} <- {:for_user, opts[:for]},
+           %{data: %{"context" => context}} when is_binary(context) <- activity,
+           %Conversation{} = conversation <- Conversation.get_for_ap_id(context),
+           %Participation{id: participation_id} <-
+             Participation.for_user_and_conversation(for_user, conversation) do
+        participation_id
+      else
+        _e ->
+          nil
       end
 
     %{
@@ -273,7 +298,8 @@ defmodule Pleroma.Web.MastodonAPI.StatusView do
         in_reply_to_account_acct: reply_to_user && reply_to_user.nickname,
         content: %{"text/plain" => content_plaintext},
         spoiler_text: %{"text/plain" => summary_plaintext},
-        expires_at: expires_at
+        expires_at: expires_at,
+        direct_conversation_id: direct_conversation_id
       }
     }
   end

lib/pleroma/web/media_proxy/media_proxy.ex

@@ -4,6 +4,7 @@
 
 defmodule Pleroma.Web.MediaProxy do
   alias Pleroma.Config
+  alias Pleroma.Upload
   alias Pleroma.Web
 
   @base64_opts [padding: false]
@@ -26,7 +27,18 @@ defmodule Pleroma.Web.MediaProxy do
   defp whitelisted?(url) do
     %{host: domain} = URI.parse(url)
 
-    Enum.any?(Config.get([:media_proxy, :whitelist]), fn pattern ->
+    mediaproxy_whitelist = Config.get([:media_proxy, :whitelist])
+
+    upload_base_url_domain =
+      if !is_nil(Config.get([Upload, :base_url])) do
+        [URI.parse(Config.get([Upload, :base_url])).host]
+      else
+        []
+      end
+
+    whitelist = mediaproxy_whitelist ++ upload_base_url_domain
+
+    Enum.any?(whitelist, fn pattern ->
       String.equivalent?(domain, pattern)
     end)
   end

lib/pleroma/web/nodeinfo/nodeinfo_controller.ex

@@ -34,64 +34,18 @@ defmodule Pleroma.Web.Nodeinfo.NodeinfoController do
   def raw_nodeinfo do
     stats = Stats.get_stats()
 
-    exclusions = Config.get([:instance, :mrf_transparency_exclusions])
-
-    mrf_simple =
-      Config.get(:mrf_simple)
-      |> Enum.map(fn {k, v} -> {k, Enum.reject(v, fn v -> v in exclusions end)} end)
-      |> Enum.into(%{})
-
-    # This horror is needed to convert regex sigils to strings
-    mrf_keyword =
-      Config.get(:mrf_keyword, [])
-      |> Enum.map(fn {key, value} ->
-        {key,
-         Enum.map(value, fn
-           {pattern, replacement} ->
-             %{
-               "pattern" =>
-                 if not is_binary(pattern) do
-                   inspect(pattern)
-                 else
-                   pattern
-                 end,
-               "replacement" => replacement
-             }
-
-           pattern ->
-             if not is_binary(pattern) do
-               inspect(pattern)
-             else
-               pattern
-             end
-         end)}
-      end)
-      |> Enum.into(%{})
-
-    mrf_policies =
-      MRF.get_policies()
-      |> Enum.map(fn policy -> to_string(policy) |> String.split(".") |> List.last() end)
-
     quarantined = Config.get([:instance, :quarantined_instances], [])
 
     staff_accounts =
       User.all_superusers()
       |> Enum.map(fn u -> u.ap_id end)
 
-    mrf_user_allowlist =
-      Config.get([:mrf_user_allowlist], [])
-      |> Enum.into(%{}, fn {k, v} -> {k, length(v)} end)
-
     federation_response =
       if Config.get([:instance, :mrf_transparency]) do
-        %{
+        {:ok, data} = MRF.describe()
-          mrf_policies: mrf_policies,
+
-          mrf_simple: mrf_simple,
+        data
-          mrf_keyword: mrf_keyword,
+        |> Map.merge(%{quarantined_instances: quarantined})
-          mrf_user_allowlist: mrf_user_allowlist,
-          quarantined_instances: quarantined,
-          exclusions: length(exclusions) > 0
-        }
       else
         %{}
       end
@@ -165,6 +119,7 @@ defmodule Pleroma.Web.Nodeinfo.NodeinfoController do
         },
         accountActivationRequired: Config.get([:instance, :account_activation_required], false),
         invitesEnabled: Config.get([:instance, :invites_enabled], false),
+        mailerEnabled: Config.get([Pleroma.Emails.Mailer, :enabled], false),
         features: features,
         restrictedNicknames: Config.get([Pleroma.User, :restricted_nicknames]),
         skipThreadContainment: Config.get([:instance, :skip_thread_containment], false)

lib/pleroma/web/oauth/oauth_controller.ex

@@ -365,8 +365,7 @@ defmodule Pleroma.Web.OAuth.OAuthController do
   def register(%Plug.Conn{} = conn, %{"authorization" => _, "op" => "connect"} = params) do
     with registration_id when not is_nil(registration_id) <- get_session_registration_id(conn),
          %Registration{} = registration <- Repo.get(Registration, registration_id),
-         {_, {:ok, auth}} <-
+         {_, {:ok, auth}} <- {:create_authorization, do_create_authorization(conn, params)},
-           {:create_authorization, do_create_authorization(conn, params)},
          %User{} = user <- Repo.preload(auth, :user).user,
          {:ok, _updated_registration} <- Registration.bind_to_user(registration, user) do
       conn

lib/pleroma/web/oauth/token.ex

@@ -44,8 +44,7 @@ defmodule Pleroma.Web.OAuth.Token do
     |> Repo.find_resource()
   end
 
-  @spec exchange_token(App.t(), Authorization.t()) ::
+  @spec exchange_token(App.t(), Authorization.t()) :: {:ok, Token.t()} | {:error, Changeset.t()}
-          {:ok, Token.t()} | {:error, Changeset.t()}
   def exchange_token(app, auth) do
     with {:ok, auth} <- Authorization.use_token(auth),
          true <- auth.app_id == app.id do

lib/pleroma/web/oauth/token/clean_worker.ex

@@ -6,36 +6,30 @@ defmodule Pleroma.Web.OAuth.Token.CleanWorker do
   @moduledoc """
   The module represents functions to clean an expired oauth tokens.
   """
+  use GenServer
+
+  @ten_seconds 10_000
+  @one_day 86_400_000
 
-  # 10 seconds
-  @start_interval 10_000
   @interval Pleroma.Config.get(
-              # 24 hours
               [:oauth2, :clean_expired_tokens_interval],
-              86_400_000
+              @one_day
             )
-  @queue :background
 
   alias Pleroma.Web.OAuth.Token
 
-  def start_link, do: GenServer.start_link(__MODULE__, nil)
+  def start_link(_), do: GenServer.start_link(__MODULE__, %{})
 
   def init(_) do
-    if Pleroma.Config.get([:oauth2, :clean_expired_tokens], false) do
+    Process.send_after(self(), :perform, @ten_seconds)
-      Process.send_after(self(), :perform, @start_interval)
     {:ok, nil}
-    else
-      :ignore
-    end
   end
 
   @doc false
   def handle_info(:perform, state) do
+    Token.delete_expired_tokens()
+
     Process.send_after(self(), :perform, @interval)
-    PleromaJobQueue.enqueue(@queue, __MODULE__, [:clean])
     {:noreply, state}
   end
-
-  # Job Worker Callbacks
-  def perform(:clean), do: Token.delete_expired_tokens()
 end

lib/pleroma/web/ostatus/activity_representer.ex

@@ -9,6 +9,7 @@ defmodule Pleroma.Web.OStatus.ActivityRepresenter do
   alias Pleroma.Web.OStatus.UserRepresenter
 
   require Logger
+  require Pleroma.Constants
 
   defp get_href(id) do
     with %Object{data: %{"external_url" => external_url}} <- Object.get_cached_by_ap_id(id) do
@@ -34,7 +35,7 @@ defmodule Pleroma.Web.OStatus.ActivityRepresenter do
     Enum.map(to, fn id ->
       cond do
         # Special handling for the AP/Ostatus public collections
-        "https://www.w3.org/ns/activitystreams#Public" == id ->
+        Pleroma.Constants.as_public() == id ->
           {:link,
            [
              rel: "mentioned",
@@ -182,6 +183,7 @@ defmodule Pleroma.Web.OStatus.ActivityRepresenter do
     author = if with_author, do: [{:author, UserRepresenter.to_simple_form(user)}], else: []
 
     retweeted_activity = Activity.get_create_by_object_ap_id(activity.data["object"])
+    retweeted_object = Object.normalize(retweeted_activity)
     retweeted_user = User.get_cached_by_ap_id(retweeted_activity.data["actor"])
 
     retweeted_xml = to_simple_form(retweeted_activity, retweeted_user, true)
@@ -196,7 +198,7 @@ defmodule Pleroma.Web.OStatus.ActivityRepresenter do
       {:"activity:verb", ['http://activitystrea.ms/schema/1.0/share']},
       {:id, h.(activity.data["id"])},
       {:title, ['#{user.nickname} repeated a notice']},
-      {:content, [type: 'html'], ['RT #{retweeted_activity.data["object"]["content"]}']},
+      {:content, [type: 'html'], ['RT #{retweeted_object.data["content"]}']},
       {:published, h.(inserted_at)},
       {:updated, h.(updated_at)},
       {:"ostatus:conversation", [ref: h.(activity.data["context"])],