Our test environment cheats by constructing a conn with a custom oauth_access/2 function. This assigns a :token to the conn but due to the way it is constructed it has the :user preloaded. When the OAuth Plug fetches a token it does not preload the user, so the check for user.disclose_client was always nil and assumed to be false.
Preloading the :user ensures the test environment matches reality.
It is angry we are making a fake %Plug.Conn{} to pass through Signature.validate_signature/1. We can work around it by making the code support a map, but then we lose the benefit of being able to use put_req_header/3
This logic only exists in the Plug, so attempting to validate the signature by calling the library function HTTPSignature.validate_conn/2 directly will never work because we do not attempt to construct the (request-target) and @request-target headers with both the commonly misinterpreted and correct implementation of this field. Therefore all attempts to validate a signature from an Oban Job will fail.
When signatures fail on incoming activities we put the job into Oban to be processed later instead of doing the user fetching and validation inline which is expensive and increases latency on the incoming POST request. Unfortunately we did not retain the :method, :request_path, and :query_string parameters from the conn so the signature validation and Oban Job would always fail.
This was most obvious when Mastodon sends Deletes for users your server has never seen before.
This is for a normal HTTP error response or timeout while receiving the data. A hard error from a process crash, DNS lookup failure, etc should produce a different response than {:ok, %Tesla.Env{}} and the request/job will be retryable.
This should fix WithClauseError resulting in Oban jobs for processing
incoming deletes being retried without getting cancelled when those
deletes are MRF rejected.
Fixes module name being not fully qualified
warning: AdminAPI.FallbackController.call/2 is undefined (module AdminAPI.FallbackController is not available or is yet to be defined)
│
5 │ defmodule Pleroma.Web.AdminAPI.RuleController do
│ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
│
└─ lib/pleroma/web/admin_api/controllers/rule_controller.ex:5: Pleroma.Web.AdminAPI.RuleController.action/2
warning: AdminAPI.FallbackController.init/1 is undefined (module AdminAPI.FallbackController is not available or is yet to be defined)
│
5 │ defmodule Pleroma.Web.AdminAPI.RuleController do
│ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
│
└─ lib/pleroma/web/admin_api/controllers/rule_controller.ex:5: Pleroma.Web.AdminAPI.RuleController.action/2
Key the cache on the image being used and the commit sha. This should allow the cache to be reused by the same runner across multiple jobs/stages where appropriate.
Also update other places where we use the term "send" instead of "stream". This should make it clearer that we are streaming these over websockets / web push and not sending an activity.
:idna.encode/1 expects a charlist even though it will accept a binary string. That functionality is undocumented / not part of its typespec, so we should turn it into a charlist first. Also switch to using match?/2
lib/pleroma/user.ex:2056:call
The function call will not succeed.
:idna.encode(_host :: binary())
will never return since the success typing is:
(string()) :: string()
and the contract is
(string()) :: string()
Wrong @spec name for remove_from_block/2
lib/pleroma/user.ex:2721:overlapping_contract
Overloaded contract for Pleroma.User.add_to_block/2 has
overlapping domains; such contracts are currently unsupported and
are simply ignored.
WebPushEncryption.send_web_push/4 was written to raise on erroroneus input, so we must guard against that.
lib/pleroma/web/push/impl.ex:65:no_return Function push_message/4 has no local return.
lib/pleroma/web/media_proxy/media_proxy_controller.ex:154:pattern_match
The pattern can never match the type.
Pattern:
{:ok, _thumbnail_binary}
Type:
{:error, boolean() | {:ffmpeg, :command_not_found}}
The callback already defines the @spec and these do not match it.
lib/pleroma/upload/filter/exiftool/strip_location.ex:12:callback_spec_type_mismatch
The @spec return type does not match the expected return type
for filter/1 callback in Pleroma.Upload.Filter behaviour.
Actual:
@spec filter(...) :: {:ok, _}
Expected:
@spec filter(...) :: {:error, _} | {:ok, :filtered | :noop} | {:ok, :filtered, struct()}
lib/pleroma/notification.ex:492:invalid_contract
The @spec for the function does not match the success typing of the function.
Function:
Pleroma.Notification.get_notified_from_activity/2
Success typing:
@spec get_notified_from_activity(_, _) :: [any()]
Sharing this pool with regular Media is problematic as Rich Media will connect to many different
domains and thrash the pool, but regular Media will have predictable connections to the webservers
hosting media for the fediverse servers you peer with.
From realpath(1) in POSIX 202x Draft 4.1:
> If file does not name a symbolic link, readlink shall write a diagnostic
> message to standard error and exit with non-zero status.
Which also doesn't includes `-f`, in preference of `realpath`.
Websites are increasingly getting more bloated with tricks like inlining content (e.g., CNN.com) which puts pages at or above 5MB. This value may still be too low.
Rich Media parsing was previously handled on-demand with a 2 second HTTP request timeout and retained only in Cachex. Every time a Pleroma instance is restarted it will have to request and parse the data for each status with a URL detected. When fetching a batch of statuses they were processed in parallel to attempt to keep the maximum latency at 2 seconds, but often resulted in a timeline appearing to hang during loading due to a URL that could not be successfully reached. URLs which had images links that expire (Amazon AWS) were parsed and inserted with a TTL to ensure the image link would not break.
Rich Media data is now cached in the database and fetched asynchronously. Cachex is used as a read-through cache. When the data becomes available we stream an update to the clients. If the result is returned quickly the experience is almost seamless. Activities were already processed for their Rich Media data during ingestion to warm the cache, so users should not normally encounter the asynchronous loading of the Rich Media data.
Implementation notes:
- The async worker is a Task with a globally unique process name to prevent duplicate processing of the same URL
- The Task will attempt to fetch the data 3 times with increasing sleep time between attempts
- The HTTP request obeys the default HTTP request timeout value instead of 2 seconds
- URLs that cannot be successfully parsed due to an unexpected error receives a negative cache entry for 15 minutes
- URLs that fail with an expected error will receive a negative cache with no TTL
- Activities that have no detected URLs insert a nil value in the Cachex :scrubber_cache so we do not repeat parsing the object content with Floki every time the activity is rendered
- Expiring image URLs are handled with an Oban job
- There is no automatic cleanup of the Rich Media data in the database, but it is safe to delete at any time
- The post draft/preview feature makes the URL processing synchronous so the rendered post preview will have an accurate rendering
Overall performance of timelines and creating new posts which contain URLs is greatly improved.
This test should not have been passing. The search result's activity id should not be the same id as the local post.
capture_log was not being used. Removed.
These were only used in dev and served no specific purpose. The equivalent settings for Bandit are under a key called :http1_options and the default values are set to 10_000.
The value here gets passesd to :crypto.pbkdf2_hmac and it expects one of these atoms: :sha | :sha224 | :sha256 | :sha384 | :sha512 so it will always exist
It is not allowed to use the Sec-WebSocket-Protocol header for arbitrary values. This was possible due to the raw websocket handling we were doing with Cowboy, but Phoenix.Socket.Transport does not allow this as the value of this header is compared against a static list of subprotocols.
https://hexdocs.pm/phoenix/Phoenix.Endpoint.html#socket/3-websocket-configuration
Additionally I cannot find anywhere that we depended on this behavior. Setting the Sec-WebSocket-Protocol header does not appear to be a part of PleromaFE.
This was recently changed to solve a Dialyzer error, but the replacement logic was faulty as "retry" would only be compared to :error and not have its truthiness evaluated.
The original logic was also faulty as it returned {:error, :pool_full} even retry was true. It never retried when the pool was full.
Also consolidate Tesla mocks into the HttpRequestMock module.
Tests were not exercising the real codepaths. The Rich Media Preview only works with https, but most of these tests were only mocking http.
The Rich Media Previews were not regenerated when a post was updated due to a cache invalidation issue. They are now cached by the activity id so they can be evicted with the other activity cache objects in the :scrubber_cache.
lib/pleroma/config/deprecation_warnings.ex: The atom :error on line 292 is expected to have type :ok | nil but it has type :error
lib/pleroma/config/deprecation_warnings.ex: The function call move_namespace_and_warn(
[
{Pleroma.ActivityExpiration, Pleroma.Workers.PurgeExpiredActivity,
"
* `config :pleroma, Pleroma.ActivityExpiration` is now `config :pleroma, Pleroma.Workers.PurgeExpiredActivity`"}
],
warning_preface
) on line 350 is expected to have type :ok | nil but it has type :ok | nil | :error
lib/pleroma/config/deprecation_warnings.ex: The function call move_namespace_and_warn(
[
{Pleroma.Plugs.RemoteIp, Pleroma.Web.Plugs.RemoteIp, "
* `config :pleroma, Pleroma.Plugs.RemoteIp` is now `config :pleroma, Pleroma.Web.Plugs.RemoteIp`"}
],
warning_preface
) on line 366 is expected to have type :ok | nil but it has type :ok | nil | :error
lib/pleroma/config/deprecation_warnings.ex: The atom :error on line 390 is expected to have type :ok | nil but it has type :error
lib/pleroma/config/deprecation_warnings.ex: The atom :error on line 413 is expected to have type :ok | nil but it has type :error
lib/pleroma/emoji/pack.ex: The tuple {:cwd, tmp_dir} on line 103 is expected to have type :cooked
| :keep_old_files
| :memory
| :verbose
| {:cwd, list(char())}
| {:file_filter, (record(:zip_file) -> boolean())}
| {:file_list, list(:file.name())} but it has type {:cwd, binary()}
lib/mix/tasks/pleroma/emoji.ex: The tuple {:cwd, pack_path} on line 114 is expected to have type :cooked
| :keep_old_files
| :memory
| :verbose
| {:cwd, list(char())}
| {:file_filter, (record(:zip_file) -> boolean())}
| {:file_list, list(:file.name())} but it has type {:cwd, binary()}
lib/pleroma/mfa.ex: The map %{error: msg} on line 80 is expected to have type {:ok, list(binary())} | {:error, String.t()} but it has type %{required(:error) => any()}
lib/pleroma/config/deprecation_warnings.ex: The atom :error on line 278 is expected to have type :ok | nil but it has type :error
lib/pleroma/config/deprecation_warnings.ex: The atom :error on line 292 is expected to have type :ok | nil but it has type :error
lib/pleroma/config/deprecation_warnings.ex: The atom :error on line 390 is expected to have type :ok | nil but it has type :error
lib/pleroma/config/deprecation_warnings.ex: The atom :error on line 413 is expected to have type :ok | nil but it has type :error
validate_scopes/2 can never receive a map as it is only called in one place with a guard requiring a list
lib/pleroma/web/o_auth/o_auth_controller.ex:615:guard_fail
The guard test:
is_map(_params :: maybe_improper_list())
can never succeed.
lib/pleroma/web/pleroma_api/controllers/mascot_controller.ex:37:pattern_match
The pattern can never match the type.
Pattern:
{:content_type, _}
Type:
{:error, _}
________________________________________________________________________________
lib/pleroma/web/pleroma_api/controllers/mascot_controller.ex:40:pattern_match
The pattern can never match the type.
Pattern:
{:upload, {:error, _}}
Type:
{:error, _}
lib/pleroma/web/admin_api/controllers/user_controller.ex:333:no_return
Function index/2 has no local return.
________________________________________________________________________________
lib/pleroma/web/admin_api/controllers/user_controller.ex:357:unused_fun
Function maybe_parse_filters/1 will never be called.
________________________________________________________________________________
lib/pleroma/web/admin_api/controllers/user_controller.ex:366:no_return
Function page_params/1 has no local return.
________________________________________________________________________________
lib/pleroma/web/admin_api/controllers/user_controller.ex:368:call
The function call will not succeed.
Pleroma.Web.ControllerHelper.fetch_integer_param(_params :: any(), :page, 1)
breaks the contract
(map(), String.t(), integer() | nil) :: integer() | nil
lib/pleroma/web/media_proxy/media_proxy_controller.ex:55:no_return
Function handle_preview/2 has no local return.
________________________________________________________________________________
lib/pleroma/web/media_proxy/media_proxy_controller.ex:59:call
The function call will not succeed.
Pleroma.HTTP.request(<<72, 69, 65, 68>>, _media_proxy_url :: any(), [], [], [{:pool, :media}])
will never return since the success typing is:
(
:delete | :get | :head | :options | :patch | :post | :put | :trace,
binary(),
any(),
[{binary(), binary()}],
Keyword.t()
) :: any()
and the contract is
(
method(),
Pleroma.HTTP.Request.url(),
String.t(),
Pleroma.HTTP.Request.headers(),
:elixir.keyword()
) :: {:ok, Tesla.Env.t()} | {:error, any()}
________________________________________________________________________________
lib/pleroma/web/media_proxy/media_proxy_controller.ex:95:unused_fun
Function handle_preview/3 will never be called.
________________________________________________________________________________
lib/pleroma/web/media_proxy/media_proxy_controller.ex:111:unused_fun
Function handle_png_preview/2 will never be called.
________________________________________________________________________________
lib/pleroma/web/media_proxy/media_proxy_controller.ex:134:unused_fun
Function handle_jpeg_preview/2 will never be called.
________________________________________________________________________________
lib/pleroma/web/media_proxy/media_proxy_controller.ex:152:unused_fun
Function handle_video_preview/2 will never be called.
________________________________________________________________________________
lib/pleroma/web/media_proxy/media_proxy_controller.ex:164:unused_fun
Function drop_static_param_and_redirect/1 will never be called.
________________________________________________________________________________
lib/pleroma/web/media_proxy/media_proxy_controller.ex:173:unused_fun
Function fallback_on_preview_error/2 will never be called.
________________________________________________________________________________
lib/pleroma/web/media_proxy/media_proxy_controller.ex:177:unused_fun
Function put_preview_response_headers/1 will never be called.
________________________________________________________________________________
lib/pleroma/web/media_proxy/media_proxy_controller.ex:177:unused_fun
Function put_preview_response_headers/2 will never be called.
________________________________________________________________________________
lib/pleroma/web/media_proxy/media_proxy_controller.ex:187:unused_fun
Function thumbnail_max_dimensions/0 will never be called.
________________________________________________________________________________
lib/pleroma/web/media_proxy/media_proxy_controller.ex:196:unused_fun
Function min_content_length_for_preview/0 will never be called.
________________________________________________________________________________
lib/pleroma/web/media_proxy/media_proxy_controller.ex:200:unused_fun
Function media_preview_proxy_config/0 will never be called.
lib/pleroma/web/mastodon_api/controllers/directory_controller.ex:6:unused_fun
Function skip_auth/2 will never be called.
________________________________________________________________________________
lib/pleroma/web/mastodon_api/controllers/directory_controller.ex:6:unused_fun
Function skip_plug/2 will never be called.
________________________________________________________________________________
lib/pleroma/web/mastodon_api/controllers/directory_controller.ex:18:guard_fail
The guard clause:
when _action :: atom() == <<105, 110, 100, 101, 120>>
can never succeed.
lib/pleroma/web/pleroma_api/controllers/chat_controller.ex:91:pattern_match
The pattern can never match the type.
Pattern:
{:reject, _message}
Type:
nil
________________________________________________________________________________
lib/pleroma/web/pleroma_api/controllers/chat_controller.ex:96:pattern_match
The pattern can never match the type.
Pattern:
{:error, _message}
Type:
nil
lib/pleroma/web/pleroma_api/controllers/mascot_controller.ex:33:pattern_match
The pattern can never match the type.
Pattern:
{:content_type, _}
Type:
{:error, _}
lib/pleroma/web/activity_pub/side_effects.ex:622:callback_type_mismatch
Type mismatch for @callback handle_after_transaction/1 in Pleroma.Web.ActivityPub.SideEffects.Handling behaviour.
Expected type:
map()
Actual type:
Keyword.t()
lib/pleroma/web/activity_pub/side_effects.ex:622:callback_arg_type_mismatch
The inferred type for the 1st argument is not a
supertype of the expected type for the handle_after_transaction/1 callback
in the Pleroma.Web.ActivityPub.SideEffects.Handling behaviour.
Success type:
Keyword.t()
Behaviour callback type:
map()
lib/pleroma/web/activity_pub/side_effects.ex:328:pattern_match
The pattern can never match the type.
Pattern:
{:actor, _}
Type:
{:error, boolean()}
lib/pleroma/web/activity_pub/side_effects.ex:328:pattern_match
The pattern can never match the type.
Pattern:
{:actor, _}
Type:
nil
lib/pleroma/web/activity_pub/builder.ex:115:pattern_match
The pattern can never match the type.
Pattern:
_emojo = %{:file => _path}
Type:
nil | binary()
lib/pleroma/reverse_proxy.ex:225:pattern_match
The pattern can never match the type.
Pattern:
:done
Type:
{:ok, :no_duration_limit, :no_duration_limit}
lib/pleroma/reverse_proxy.ex:226:pattern_match
The pattern can never match the type.
Pattern:
{:error, _error}
Type:
{:ok, :no_duration_limit, :no_duration_limit}
lib/pleroma/reverse_proxy.ex:391:pattern_match
The pattern can never match the type.
Pattern:
__duration = nil, _max
Type:
integer(), _
lib/pleroma/helpers/qt_fast_start.ex:129:improper_list_constr
List construction (cons) will produce an improper list, because its second argument is <<_::32>>.
lib/pleroma/helpers/qt_fast_start.ex:129:improper_list_constr
List construction (cons) will produce an improper list, because its second argument is <<_::64>>.
Incorrect spec. Both search backends return :ok so that is what should be the spec.
lib/pleroma/search/database_search.ex:56:callback_type_mismatch
Type mismatch for @callback remove_from_index/1 in Pleroma.Search.SearchBackend behaviour.
Expected type:
{:error, _} | {:ok, _}
Actual type:
:ok
lib/pleroma/signature.ex:30:pattern_match
The pattern can never match the type.
Pattern:
%{<<97, 112, 95, 105, 100>> => _ap_id}
Type:
{:error, _} | {:ok, map()}
I have opted to set this to :upper as this retains the same behavior but clears up the error.
lib/pleroma/upload.ex:178:call
The function call will not succeed.
Base.encode16(binary(), [{:lower, true}])
breaks the contract
(binary(), [{:case, encode_case()}]) :: binary()
lib/pleroma/web/twitter_api/controllers/util_controller.ex:300:pattern_match
The pattern can never match the type.
Pattern:
{:error, :no_such_alias}
Type:
{:not_found, nil}
lib/pleroma/web/twitter_api/controllers/util_controller.ex:304:pattern_match
The pattern can never match the type.
Pattern:
{:error, _error}
Type:
{:not_found, nil}
This check was recently improved in Credo and it does make sense for readability.
The offending functions in Pleroma have been renamed and a couple missing the ? suffix have been fixed as well.
lib/pleroma/web/mastodon_api/controllers/status_controller.ex:333:pattern_match
The pattern can never match the type.
Pattern:
{:ok, _activity}
Type:
{:error, _}
lib/pleroma/migrators/context_objects_deletion_migrator.ex:13:exact_eq
The test :error | float() == 0 can never evaluate to 'true'.
lib/pleroma/migrators/hashtags_table_migrator.ex:13:exact_eq
The test :error | float() == 0 can never evaluate to 'true'.
This is for streaming media to ffmpeg thumbnailer. The existing implementation relies on undocumented behavior.
Erlang open_port/2 does not officially support passing a string of a file path for opening. The specs clearly state you are to provide one of the following for open_port/2:
{spawn, Command :: string() | binary()} |
{spawn_driver, Command :: string() | binary()} |
{spawn_executable, FileName :: file:name_all()} |
{fd, In :: integer() >= 0, Out :: integer() >= 0}
Our method technically works but is strongly discouraged as it can block the scheduler and dialyzer throws errors as it recognizes we're breaking the contract and some of the functions we wrote may never return.
This is indirectly covered by the Erlang FAQ section "9.12 Why can't I open devices (e.g. a serial port) like normal files?"
https://www.erlang.org/faq/problems#idm1127
This type is not exported and usable. FlakeId intends to return the type as :uuid, so we replace it in the typespecs with Ecto.UUID.t() which assuages the dialyzer errors
e.g.,
lib/pleroma/bookmark.ex:25:unknown_type
Unknown type: FlakeId.Ecto.CompatType.t/0.
lib/pleroma/application_requirements.ex:19:unknown_type
Unknown type: Pleroma.ApplicationRequirements.VerifyError.t/0.
lib/pleroma/application_requirements.ex:199:pattern_match_cov
The pattern
variable_result
can never match, because previous clauses completely cover the type
:ok.
lib/mix/tasks/pleroma/instance.ex:356:pattern_match_cov
The pattern
:variable_
can never match, because previous clauses completely cover the type
%{
:anonymize => boolean(),
:dedupe => boolean(),
:read_description => boolean(),
:strip_location => boolean()
}.
This is not necessary for the tests to pass and breaks other tests as this change doesn't get cleanly reverted causing the hostname to stay set this way and leak into other test causing failures with "sub.example.com" not matching "localhost"
We were overzealous with matching on a raw error from the object fetch that should have never been relied on like this. If we can't fetch successfully we should assume that the collection is private.
Building a more expressive and universal error struct to match on may be something to consider.
These tests relied on the removed Fetcher.fetch_object_from_id!/2 function injecting the error tuple into a log message with the exact words "Object containment failed."
We will keep this behavior by generating a similar log message, but perhaps this should do a better job of matching on the error tuple returned by Transmogrifier.handle_incoming/1
Old way was wrong for multiple reasons. If we do this as a config value it fixes :slave.start/3 being picked up as a compile warning on OTP26.
Also if we want to do any real clustering we'll need something like this to support OTP25 and older.
Rework inbound federation to accept requests optimistically. The HTTP Signatures Plug will not attempt to fetch the actor or key and will fail early.
If the signature cannot be validated we pass the required data into the Oban job with a reduced priority and increase the timeout to 20 seconds. The Oban job will handle the actor and key fetching before attempting to validate the activity again. This job will be retried 5 times by default.
Another welcome side effect is that actors who change their keys can federate to Pleroma instances immediately instead of needing to wait the default value of 86400s / 24 hours before the key will be fetched again.
Recommending use of the separate HTTP server for exposing the metrics
and securing it externally on your firewall or reverse proxy. It will
listen on port 4021 by default.
- Index unlisted posts
- Move version check outside of the streaming and only do it once
- Use a PUT request instead of checking manually if there is need to insert
- Add error handling, sort of
While here fix the naming convention of the old module attribute restricting caching and add a new one for the default cache value
All frontends should be shipped with versioned assets. There be dragons if you don't.
`<code>` can be anything, but we recommend using a more or less unique identifier to avoid collisions, such as the branch name.
`<type>` can be `add`, `remove`, `fix`, `security` or `skip`. `skip` is only used if there is no user-visible change in the MR (for example, only editing comments in the code). Otherwise, choose a type that corresponds to your change.
`<type>` can be `add`, `change`, `remove`, `fix`, `security` or `skip`. `skip` is only used if there is no user-visible change in the MR (for example, only editing comments in the code). Otherwise, choose a type that corresponds to your change.
In the file, write the changelog entry. For example, if an MR adds group functionality, we can create a file named `group.add` and write `Add group functionality` in it.
@ -4,10 +4,158 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## 2.6.3
## 2.7.0
### Security
- HTTP Security: By default, don't allow unsafe-eval. The setting needs to be changed to allow Flash emulation.
- Fix webfinger spoofing.
- Use proper workers for fetching pins instead of an ad-hoc task, fixing a potential fetch loop
### Changed
- Update to Phoenix 1.7
- Elixir Logger configuration is now longer permitted through AdminFE and ConfigDB
- Refactor the user backups code and improve test coverage
- Invalid activities delivered to the inbox will be rejected with a 400 Bad Request
- Support Bandit as an alternative to Cowboy for the HTTP server.
- Update Bandit to 1.5.2
- Replace eblurhash with rinpatch_blurhash. This also removes a dependency on ImageMagick.
- Elixir 1.13 is the minimum required version.
- Document maximum supported version of Erlang & Elixir
- Update and extend NetBSD installation docs
- Make `/api/v1/pleroma/federation_status` publicly available
- Increase outgoing federation parallelism
- Change Hackney connection pool timeouts to align with the values Gun uses
- Transmogrifier: handle non-validate errors on incoming Delete activities
- Remote object fetch failures will prevent the object fetch job from retrying if the object request returns 401, 403, 404, 410, or exceeds the maximum thread depth.
- - Change AccountView `last_status_at` from a datetime to a date (as done in Mastodon 3.1.0)
- Improve error logging when LDAP authentication fails.
- Publisher jobs will not retry if the error received is a 400
- PollWorker jobs will not retry if the activity no longer exists.
- Improved detecting unrecoverable errors for incoming federation jobs
- Changed some jobs to return :cancel on unrecoverable errors that should not be retried
- Discard Remote Fetcher jobs which errored due to an MRF rejection.
- Oban queues have refactored to simplify the queue design
- Ensure all Oban jobs have timeouts defined
- Optimistic Inbox reduces the processing overhead of incoming activities without instantly verifiable signatures.
- HTTP connection pool adjustments
- Disable jit by default for PostgreSQL
- Update the documentation for configuring Prometheus metrics.
- Change the prometheus library to PromEx.
- Publisher jobs now store the the activity id instead of inserting duplicate JSON data in the Oban queue for each delivery.
- Activity publishing failures will prevent the job from retrying if the publishing request returns a 403 or 410
- Publisher errors will now emit logs indicating the inbox that was not available for delivery.
- Reduce the reachability timestamp update to a single upsert query
- A 422 error is returned when attempting to reply to a deleted status
- Rich Media backfilling is now an Oban job
- Refactored Rich Media to cache the content in the database. Fetching operations that could block status rendering have been eliminated.
- Set default values on validators for transient objects (attachment, poll options)
- User profile refreshes are now asynchronous
- Change mediaproxy previews to use vips to generate thumbnails instead of ImageMagick
- Render nice web push notifications for polls
- Refactor the Mastodon /api/v1/streaming websocket handler to use Phoenix.Socket.Transport
### Added
- Uploader: Add support for uploading attachments using IPFS
- Add NSFW-detecting MRF
- Add DNSRBL MRF
- Add options to the mix prune_objects task
- Add Anti-mention Spam MRF backported from Rebased
- Support /authorize-interaction route used by Mastodon
- Add an option to reject certain domains when authorized fetch is enabled.
- Include following/followers in backups
- Allow to group bookmarks in folders
- Include image description in status media cards
- Implement `/api/v1/accounts/familiar_followers`
- Add support for configuring favicon, embed favicon and PWA manifest in server-generated meta
- Implement FEP-2c59, add "webfinger" to user actor
- Framegrabs with ffmpeg will execute with a 5 second timeout and cache the URLs of failures with a TTL of 15 minutes to prevent excessive retries.
- Added a Mix task "pleroma.config fix_mrf_policies" which will remove erroneous MRF policies from ConfigDB.
- Add ForceMention MRF
- [docs] add frontends management documentation
- Implement group actors
- Add contact account to InstanceView
- Add instance rules
- Implement /api/v2/instance route
- Verify profile link ownership with rel="me"
- Logger metadata is now attached to some logs to help with troubleshooting and analysis
- Add new parameters to /api/v2/instance: configuration[accounts][max_pinned_statuses] and configuration[statuses][characters_reserved_per_url]
- Add meilisearch, make search engines pluggable
- Add missing indexes on foreign key relationships
- Startup detection for configured MRF modules that are missing or incorrectly defined
- Permit passing --chunk and --step values to the Pleroma.Search.Indexer Mix task
- Deleting, Unfavoriting, Unrepeating, or Unreacting will cancel undelivered publishing jobs for the original activity.
- Oban jobs can now be viewed in the Live Dashboard
- Add media proxy to opengraph rich media cards
- Support for Erlang OTP 26
- Prioritize mentioned recipients (i.e., those that are not just followers) when federating.
- PromEx documentation
- Expose nonAnonymous field from Smithereen polls
- Add Qdrant/OpenAI embedding search
- Adds the capability to add a URL to a scrobble (optional field)
- scrubbers/default: Add more formatting elements from HTML4 / GoToSocial (acronym, bdo, big, cite, dfn, ins, kbd, q, samp, s, tt, var, wbr)
- Monitoring of search backend health to control the processing of jobs in the search indexing Oban queue
- Display reposted replies with exclude_replies: true
- Add "status" notification type
- Support honk-style attachment summaries as alt-text.
### Fixed
- Fix Emoji object IDs not always being valid
- Remove checking ImageMagick's commands for Pleroma.Upload.Filter.AnalyzeMetadata
- Ensure that StripLocation actually removes everything resembling GPS data from PNGs
- Fix authentication check on account rendering when bio is defined
- ap userview: add outbox field.
- Fix #strip_report_status_data
- Fix federation with Convergence AP Bridge
- ChatMessage: Tolerate attachment field set to an empty array
- Config: Check the permissions of the linked file instead of the symlink
- MediaProxy was setting the content-length header which is not permitted by RFC9112§6.2 when we are chunking the reply as it conflicts with the existence of the transfer-encoding header.
- Restore Cowboy's ability to stream MediaProxy responses without Chunked encoding.
- Fix the processing of email digest jobs.
- Client application data was always missing from the status
- Elixir 1.15 compatibility
- When downloading remote emojis packs, account for pagination
- Make remote emoji packs API use specifically the V1 URL. Akkoma does not understand it without V1, and it works either way with normal pleroma, so no reason to not do this
- Following HTTP Redirects when the HTTP Adapter is Finch
- Video framegrabs were not working correctly after the change to use Exile to execute ffmpeg
- Deactivated groups would still try to repeat a post.
- Fix logic error in Gun connection pooling which prevented retries even when the worker was launched with retry = true
- Connection pool errors when publishing an activity is a soft-error that will be retried shortly.
- Gun Connection Pool was not retrying to acquire a connection if the pool was full and stale connections were reclaimed
- TwitterAPI: Return proper error when healthcheck is disabled
- Handle cases when users.inbox is nil.
- Fix LDAP support
- Use correct domain for fqn and InstanceView
- The query for marking notifications as read has been simplified
- Mastodon API /api/v1/directory: Fix listing directory contents when not authenticated
- Ensure MediaProxy HTTP requests obey all the defined connection settings
- Fix a memory leak caused by Websocket connections that would not enter a state where a full garbage collection run could be triggered.
- Fix OpenGraph and Twitter metadata providers when parsing objects with no content or summary fields.
- MRF: Log sensible error for subdomains_regex
- MRF.StealEmojiPolicy: Properly add fallback extension to filenames missing one
- Federated timeline removal of hashtags via MRF HashtagPolicy
- Support objects with a null contentMap (firefish)
- Fix notifications query which was not using the index properly
- Notifications: improve performance by filtering on users table instead of activities table
- Prevent Rich Media backfill jobs from retrying in cases where it is likely they will fail again.
- Oban Jobs for refreshing users were not respecting the uniqueness setting
- Fix Optimistic Inbox for failed signatures
- MediaProxy Preview failures prevented when encountering certain video files
- pleroma_ctl: Use realpath(1) instead of readlink(1)
- ReceiverWorker: Make sure non-{:ok, _} is returned as {:error, …}
- Harden Rich Media parsing against very slow or malicious URLs
- Rich Media Preview cache eviction when the activity is updated.
- Parsing of RichMedia TTLs for Amazon URLs when query parameters are nil
- End of poll notifications were not streamed over websockets or web push
- Fix eblurhash and elixir-captcha not using system cflags
- Video thumbnails were not being generated due to a negative cache lookup logic error
- Fix web push notifications not successfully delivering
- Web Push notifications are no longer generated for muted/blocked threads and users.
- Fix validate_webfinger when running a different domain for Webfinger
### Removed
- Mastodon API: Remove deprecated GET /api/v1/statuses/:id/card endpoint https://github.com/mastodon/mastodon/pull/11213
- Removed support for multiple federator modules as we only support ActivityPub
## 2.6.2
@ -72,7 +220,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## 2.5.4
## Security
- Fix XML External Entity (XXE) loading vulnerability allowing to fetch arbitary files from the server's filesystem
- Fix XML External Entity (XXE) loading vulnerability allowing to fetch arbitrary files from the server's filesystem
## 2.5.3
@ -88,7 +236,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
## 2.5.4
## Security
- Fix XML External Entity (XXE) loading vulnerability allowing to fetch arbitary files from the server's filesystem
- Fix XML External Entity (XXE) loading vulnerability allowing to fetch arbitrary files from the server's filesystem
## 2.5.3
@ -128,7 +276,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
- Fix `block_from_stranger` setting
- Fix rel="me"
- Docker images will now run properly
- Fix inproper content being cached in report content
- Fix improper content being cached in report content
- Notification filter on object content will not operate on the ones that inherently have no content
- ZWNJ and double dots in links are parsed properly for Plain-text posts
- OTP releases will work on systems with a newer libcrypt
@ -794,7 +942,7 @@ switched to a new configuration mechanism, however it was not officially removed
- Rate limiter crashes when there is no explicitly specified ip in the config
- 500 errors when no `Accept` header is present if Static-FE is enabled
- Instance panel not being updated immediately due to wrong `Cache-Control` headers
- Statuses posted with BBCode/Markdown having unncessary newlines in Pleroma-FE
- Statuses posted with BBCode/Markdown having unnecessary newlines in Pleroma-FE
- OTP: Fix some settings not being migrated to in-database config properly
- No `Cache-Control` headers on attachment/media proxy requests
- Reverse Proxy limiting `max_body_length` was incorrectly defined and only checked `Content-Length` headers which may not be sufficient in some circumstances
### Added
- Expiring/ephemeral activites. All activities can have expires_at value set, which controls when they should be deleted automatically.
- Expiring/ephemeral activities. All activities can have expires_at value set, which controls when they should be deleted automatically.
- Mastodon API: in post_status, the expires_in parameter lets you set the number of seconds until an activity expires. It must be at least one hour.
- Mastodon API: all status JSON responses contain a `pleroma.expires_at` item which states when an activity will expire. The value is only shown to the user who created the activity. To everyone else it's empty.
- Configuration: `ActivityExpiration.enabled` controls whether expired activites will get deleted at the appropriate time. Enabled by default.
- Configuration: `ActivityExpiration.enabled` controls whether expired activities will get deleted at the appropriate time. Enabled by default.
- Conversations: Add Pleroma-specific conversation endpoints and status posting extensions. Run the `bump_all_conversations` task again to create the necessary data.
- MRF: Support for priming the mediaproxy cache (`Pleroma.Web.ActivityPub.MRF.MediaProxyWarmingPolicy`)
- MRF: Support for excluding specific domains from Transparency.
"The instance thumbnail can be any image that represents your instance and is used by some apps or services when they display information about your instance.",
"Maximum number of requests waiting for other requests to finish. After this number is reached, the pool will start returning errrors when a new request is made",
"Maximum number of requests waiting for other requests to finish. After this number is reached, the pool will start returning errors when a new request is made",
description:"Limit user to export not more often than once per N days",
suggestions:[7]
},
%{
key::process_wait_time,
type::integer,
label:"Process Wait Time",
description:
"The amount of time to wait for backup to report progress, in milliseconds. If no progress is received from the backup job for that much time, terminate it and deem it failed.",
suggestions:[30_000]
},
%{
key::process_chunk_size,
type::integer,
label:"Process Chunk Size",
description:"The number of activities to fetch in the backup job for each chunk.",
suggestions:[100]
},
%{
key::timeout,
type::integer,
label:"Timeout",
description:"The amount of time to wait for backup to complete in seconds.",
- `<path>` - where to save migrated config. E.g. `--path=/tmp`. If file saved into non standart folder, you must manually copy file into directory where Pleroma can read it. For OTP install path will be `PLEROMA_CONFIG_PATH` or `/etc/pleroma`. For installation from source - `config` directory in the pleroma folder.
- `<path>` - where to save migrated config. E.g. `--path=/tmp`. If file saved into non-standard folder, you must manually copy file into directory where Pleroma can read it. For OTP install path will be `PLEROMA_CONFIG_PATH` or `/etc/pleroma`. For installation from source - `config` directory in the pleroma folder.
- `<env>` - environment, for which is migrated config. By default is `prod`.
- To delete transferred settings from database optional flag `-d` can be used
@ -154,4 +154,19 @@ This forcibly removes all saved values in the database.
```sh
mix pleroma.config [--force] reset
```
## Remove invalid MRF modules from the database
This forcibly removes any enabled MRF that does not exist and will fix the ability of the instance to start.
- `--vacuum` - run `VACUUM FULL` after the embedded objects are replaced with their references
## Prune old remote posts from the database
This will prune remote posts older than 90 days (configurable with [`config :pleroma, :instance, remote_post_retention_days`](../../configuration/cheatsheet.md#instance)) from the database, they will be refetched from source when accessed.
This will prune remote posts older than 90 days (configurable with [`config :pleroma, :instance, remote_post_retention_days`](../../configuration/cheatsheet.md#instance)) from the database. Pruned posts may be refetched in some cases.
!!! note
The disk space will only be reclaimed after a proper vacuum. By default Postgresql does this for you on a regular basis, but if your instance has been running for a long time and there are many rows deleted, it may be advantageous to use `VACUUM FULL` (e.g. by using the `--vacuum` option).
!!! danger
The disk space will only be reclaimed after `VACUUM FULL`. You may run out of disk space during the execution of the task or vacuuming if you don't have about 1/3rds of the database size free.
You may run out of disk space during the execution of the task or vacuuming if you don't have about 1/3rds of the database size free. Vacuum causes a substantial increase in I/O traffic, and may lead to a degraded experience while it is running.
=== "OTP"
@ -45,7 +47,11 @@ This will prune remote posts older than 90 days (configurable with [`config :ple
```
### Options
- `--vacuum` - run `VACUUM FULL` after the objects are pruned
- `--keep-threads` - Don't prune posts when they are part of a thread where at least one post has seen local interaction (e.g. one of the posts is a local post, or is favourited by a local user, or has been repeated by a local user...). It also won't delete posts when at least one of the posts in that thread is kept (e.g. because one of the posts has seen recent activity).
- `--keep-non-public` - Keep non-public posts like DM's and followers-only, even if they are remote.
- `--prune-orphaned-activities` - Also prune orphaned activities afterwards. Activities are things like Like, Create, Announce, Flag (aka reports). They can significantly help reduce the database size. Note: this can take a very long time.
- `--vacuum` - Run `VACUUM FULL` after the objects are pruned. This should not be used on a regular basis, but is useful if your instance has been running for a long time before pruning.
## Create a conversation for all existing DMs
@ -93,6 +99,9 @@ Can be safely re-run
## Vacuum the database
!!! note
By default Postgresql has an autovacuum deamon running. While the tasks described here can help in some cases, they shouldn't be needed on a regular basis. See [the Postgresql docs on vacuuming](https://www.postgresql.org/docs/current/sql-vacuum.html) for more information on this.
### Analyze
Running an `analyze` vacuum job can improve performance by updating statistics used by the query planner. **It is safe to cancel this.**
1. Optionally you can remove the users of your instance. This will trigger delete requests for their accounts and posts. Note that this is 'best effort' and doesn't mean that all traces of your instance will be gone from the fediverse.
* You can do this from the admin-FE where you can select all local users and delete the accounts using the *Moderate multiple users* dropdown.
* You can also list local users and delete them individualy using the CLI tasks for [Managing users](./CLI_tasks/user.md).
* You can also list local users and delete them individually using the CLI tasks for [Managing users](./CLI_tasks/user.md).
2. Stop the Pleroma service `systemctl stop pleroma`
3. Disable pleroma from systemd `systemctl disable pleroma`
4. Remove the files and folders you created during installation (see installation guide). This includes the pleroma, nginx and systemd files and folders.
@ -41,6 +41,7 @@ To add configuration to your config file, you can copy it from the base config.
* `allow_relay`: Permits remote instances to subscribe to all public posts of your instance. This may increase the visibility of your instance.
* `public`: Makes the client API in authenticated mode-only except for user-profiles. Useful for disabling the Local Timeline and The Whole Known Network. Note that there is a dependent setting restricting or allowing unauthenticated access to specific resources, see `restrict_unauthenticated` for more details.
* `quarantined_instances`: ActivityPub instances where private (DMs, followers-only) activities will not be send.
* `rejected_instances`: ActivityPub instances to reject requests from if authorized_fetch_mode is enabled.
* `allowed_post_formats`: MIME-type list of formats allowed to be posted (transformed into HTML).
* `extended_nickname_format`: Set to `true` to use extended local nicknames format (allows underscores/dashes). This will break federation with
older software for theses nicknames.
@ -154,14 +155,15 @@ To add configuration to your config file, you can copy it from the base config.
* `Pleroma.Web.ActivityPub.MRF.MentionPolicy`: Drops posts mentioning configurable users. (See [`:mrf_mention`](#mrf_mention)).
* `Pleroma.Web.ActivityPub.MRF.VocabularyPolicy`: Restricts activities to a configured set of vocabulary. (See [`:mrf_vocabulary`](#mrf_vocabulary)).
* `Pleroma.Web.ActivityPub.MRF.ObjectAgePolicy`: Rejects or delists posts based on their age when received. (See [`:mrf_object_age`](#mrf_object_age)).
* `Pleroma.Web.ActivityPub.MRF.ActivityExpirationPolicy`: Sets a default expiration on all posts made by users of the local instance. Requires `Pleroma.Workers.PurgeExpiredActivity` to be enabled for processing the scheduled delections.
* `Pleroma.Web.ActivityPub.MRF.ActivityExpirationPolicy`: Sets a default expiration on all posts made by users of the local instance. Requires `Pleroma.Workers.PurgeExpiredActivity` to be enabled for processing the scheduled deletions.
* `Pleroma.Web.ActivityPub.MRF.ForceBotUnlistedPolicy`: Makes all bot posts to disappear from public timelines.
* `Pleroma.Web.ActivityPub.MRF.FollowBotPolicy`: Automatically follows newly discovered users from the specified bot account. Local accounts, locked accounts, and users with "#nobot" in their bio are respected and excluded from being followed.
* `Pleroma.Web.ActivityPub.MRF.AntiFollowbotPolicy`: Drops follow requests from followbots. Users can still allow bots to follow them by first following the bot.
* `Pleroma.Web.ActivityPub.MRF.KeywordPolicy`: Rejects or removes from the federated timeline or replaces keywords. (See [`:mrf_keyword`](#mrf_keyword)).
* `Pleroma.Web.ActivityPub.MRF.ForceMentionsInContent`: Forces every mentioned user to be reflected in the post content.
* `Pleroma.Web.ActivityPub.MRF.InlineQuotePolicy`: Forces quote post URLs to be reflected in the message content inline.
* `Pleroma.Web.ActivityPub.MRF.QuoteToLinkTagPolicy`: Force a Link tag for posts quoting another post. (may break outgoing federation of quote posts with older Pleroma versions)
* `Pleroma.Web.ActivityPub.MRF.QuoteToLinkTagPolicy`: Force a Link tag for posts quoting another post. (may break outgoing federation of quote posts with older Pleroma versions).
* `Pleroma.Web.ActivityPub.MRF.ForceMention`: Forces posts to include a mention of the author of parent post or the author of quoted post.
* `transparency`: Make the content of your Message Rewrite Facility settings public (via nodeinfo).
* `transparency_exclusions`: Exclude specific instance names from MRF transparency. The use of the exclusions feature will be disclosed in nodeinfo as a boolean value.
@ -272,6 +274,10 @@ Notes:
#### :mrf_inline_quote
* `template`: The template to append to the post. `{url}` will be replaced with the actual link to the quoted post. Default: `<bdi>RT:</bdi> {url}`
#### :mrf_force_mention
* `mention_parent`: Whether to append mention of parent post author
* `mention_quoted`: Whether to append mention of parent quoted author
### :activitypub
* `unfollow_blocked`: Whether blocks result in people getting unfollowed
* `outgoing_blocks`: Whether to federate blocks to other instances
@ -279,6 +285,7 @@ Notes:
* `deny_follow_blocked`: Whether to disallow following an account that has blocked the user in question
* `sign_object_fetches`: Sign object fetches with HTTP signatures
* `authorized_fetch_mode`: Require HTTP signatures for AP fetches
* `authorized_fetch_mode_exceptions`: List of IPs (CIDR format accepted) to exempt from HTTP Signatures requirement (for example to allow debugging, you shouldn't otherwise need this)
* `ignore_hosts`: list of hosts which will be ignored by the metadata parser. For example `["accounts.google.com", "xss.website"]`, defaults to `[]`.
* `ignore_tld`: list TLDs (top-level domains) which will ignore for parse metadata. default is ["local", "localdomain", "lan"].
* `parsers`: list of Rich Media parsers.
* `failure_backoff`: Amount of milliseconds after request failure, during which the request will not be retried.
* `timeout`: Amount of milliseconds after which the HTTP request is forcibly terminated.
## HTTP server
@ -467,6 +474,7 @@ This will make Pleroma listen on `127.0.0.1` port `8080` and generate urls start
* ``ct_max_age``: The maximum age for the `Expect-CT` header if sent.
* ``referrer_policy``: The referrer policy to use, either `"same-origin"` or `"no-referrer"`.
* ``report_uri``: Adds the specified url to `report-uri` and `report-to` group in CSP header.
* `allow_unsafe_eval`: Adds `wasm-unsafe-eval` to the CSP header. Needed for some non-essential frontend features like Flash emulation.
### Pleroma.Web.Plugs.RemoteIp
@ -506,7 +514,7 @@ config :pleroma, :rate_limit,
Means that:
1. In 60 seconds, 15 authentication attempts can be performed from the same IP address.
2. In 1 second, 10 search requests can be performed from the same IP adress by unauthenticated users, while authenticated users can perform 30 search requests per second.
2. In 1 second, 10 search requests can be performed from the same IP address by unauthenticated users, while authenticated users can perform 30 search requests per second.
Supported rate limiters:
@ -656,6 +664,19 @@ config :ex_aws, :s3,
host: "s3.eu-central-1.amazonaws.com"
```
#### Pleroma.Uploaders.IPFS
* `post_gateway_url`: URL with port of POST Gateway (unauthenticated)
* `get_gateway_url`: URL of public GET Gateway
Example:
```elixir
config :pleroma, Pleroma.Uploaders.IPFS,
post_gateway_url: "http://localhost:5001",
get_gateway_url: "http://{CID}.ipfs.mydomain.com"
```
### Upload filters
#### Pleroma.Upload.Filter.AnonymizeFilename
@ -832,7 +853,7 @@ config :logger,
backends: [{ExSyslogger, :ex_syslogger}]
config :logger, :ex_syslogger,
level: :warn
level: :warning
```
Another example, keeping console output and adding the pid to syslog output:
Boolean, enables/disables in-database configuration. Read [Transfering the config to/from the database](../administration/CLI_tasks/config.md) for more information.
Boolean, enables/disables in-database configuration. Read [Transferring the config to/from the database](../administration/CLI_tasks/config.md) for more information.
## :database_config_whitelist
@ -1142,7 +1163,7 @@ Control favicons for instances.
!!! note
Requires enabled email
* `:purge_after_days` an integer, remove backup achives after N days.
* `:purge_after_days` an integer, remove backup achieves after N days.
* `:limit_days` an integer, limit user to export not more often than once per N days.
* `:dir` a string with a path to backup temporary directory or `nil` to let Pleroma choose temporary directory in the following order:
1. the directory named by the TMPDIR environment variable
@ -1150,6 +1171,7 @@ Control favicons for instances.
3. the directory named by the TMP environment variable
4. C:\TMP on Windows or /tmp on Unix-like operating systems
5. as a last resort, the current working directory
The files should be PNG (APNG is okay with `.png` for `image/png` Content-type) and under 50kb for compatibility with mastodon.
Default file extentions and locations for emojis are set in `config.exs`. To use different locations or file-extentions, add the `shortcode_globs` to your secrets file (`prod.secret.exs` or `dev.secret.exs`) and edit it. Note that not all fediverse-software will show emojis with other file extentions:
Default file extensions and locations for emojis are set in `config.exs`. To use different locations or file-extensions, add the `shortcode_globs` to your secrets file (`prod.secret.exs` or `dev.secret.exs`) and edit it. Note that not all fediverse-software will show emojis with other file extensions:
This guide is going to focus on the Pleroma federation aspect. The actual installation is neatly explained in the official documentation, and more likely to remain up-to-date.
It might be added to this guide if there will be a need for that.
Pleroma is built upon the Erlang/OTP VM known as BEAM. The BEAM VM is highly optimized for latency, but this has drawbacks in environments without dedicated hardware. One of the tricks used by the BEAM VM is [busy waiting](https://en.wikipedia.org/wiki/Busy_waiting). This allows the application to pretend to be busy working so the OS kernel does not pause the application process and switch to another process waiting for the CPU to execute its workload. It does this by spinning for a period of time which inflates the apparent CPU usage of the application so it is immediately ready to execute another task. This can be observed with utilities like **top(1)** which will show consistently high CPU usage for the process. Switching between procesess is a rather expensive operation and also clears CPU caches further affecting latency and performance. The goal of busy waiting is to avoid this penalty.
Pleroma is built upon the Erlang/OTP VM known as BEAM. The BEAM VM is highly optimized for latency, but this has drawbacks in environments without dedicated hardware. One of the tricks used by the BEAM VM is [busy waiting](https://en.wikipedia.org/wiki/Busy_waiting). This allows the application to pretend to be busy working so the OS kernel does not pause the application process and switch to another process waiting for the CPU to execute its workload. It does this by spinning for a period of time which inflates the apparent CPU usage of the application so it is immediately ready to execute another task. This can be observed with utilities like **top(1)** which will show consistently high CPU usage for the process. Switching between processes is a rather expensive operation and also clears CPU caches further affecting latency and performance. The goal of busy waiting is to avoid this penalty.
This strategy is very successful in making a performant and responsive application, but is not desirable on Virtual Machines or hardware with few CPU cores. Pleroma instances are often deployed on the same server as the required PostgreSQL database which can lead to situations where the Pleroma application is holding the CPU in a busy-wait loop and as a result the database cannot process requests in a timely manner. The fewer CPUs available, the more this problem is exacerbated. The latency is further amplified by the OS being installed on a Virtual Machine as the Hypervisor uses CPU time-slicing to pause the entire OS and switch between other tasks.
While it has no external dependencies, it has problems with performance and relevancy.
## QdrantSearch
This uses the vector search engine [Qdrant](https://qdrant.tech) to search the posts in a vector space. This needs a way to generate embeddings and uses the [OpenAI API](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings). This is implemented by several project besides OpenAI itself, including the python-based fastembed-server found in `supplemental/search/fastembed-api`.
The default settings will support a setup where both the fastembed server and Qdrant run on the same system as pleroma. To use it, set the search provider and run the fastembed server, see the README in `supplemental/search/fastembed-api`:
Then, start the Qdrant server, see [here](https://qdrant.tech/documentation/quick-start/) for instructions.
You will also need to create the Qdrant index once by running `mix pleroma.search.indexer create_index`. Running `mix pleroma.search.indexer index` will retroactively index the last 100_000 activities.
### Indexing and model options
To see the available configuration options, check out the QdrantSearch section in `config/config.exs`.
The default indexing option work for the default model (`snowflake-arctic-embed-xs`). To optimize for a low memory footprint, adjust the index configuration as described in the [Qdrant docs](https://qdrant.tech/documentation/guides/optimize/). See also [this blog post](https://qdrant.tech/articles/memory-consumption/) that goes into detail.
Different embedding models will need different vector size settings. You can see a list of the models supported by the fastembed server [here](https://qdrant.github.io/fastembed/examples/Supported_Models), including their vector dimensions. These vector dimensions need to be set in the `qdrant_index_configuration`.
E.g, If you want to use `sentence-transformers/all-MiniLM-L6-v2` as a model, you will not need to adjust things, because it and `snowflake-arctic-embed-xs` are both 384 dimensional models. If you want to use `snowflake/snowflake-arctic-embed-l`, you will need to adjust the `size` parameter in the `qdrant_index_configuration` to 1024, as it has a dimension of 1024.
When using a different model, you will need do drop the index and recreate it (`mix pleroma.search.indexer drop_index` and `mix pleroma.search.indexer create_index`), as the different embeddings are not compatible with each other.
## Meilisearch
Note that it's quite a bit more memory hungry than PostgreSQL (around 4-5G for ~1.2 million
posts while idle and up to 7G while indexing initially). The disk usage for this additional index is also
around 4 gigabytes. Like [RUM](./cheatsheet.md#rum-indexing-for-full-text-search) indexes, it offers considerably
higher performance and ordering by timestamp in a reasonable amount of time.
Additionally, the search results seem to be more accurate.
Due to high memory usage, it may be best to set it up on a different machine, if running pleroma on a low-resource
computer, and use private key authentication to secure the remote search instance.
To use [meilisearch](https://www.meilisearch.com/), set the search module to `Pleroma.Search.Meilisearch`:
# Differences in Mastodon API responses from vanilla Mastodon
A Pleroma instance can be identified by "<Mastodonversion> (compatible; Pleroma <version>)" present in `version` field in response from `/api/v1/instance`
A Pleroma instance can be identified by "<Mastodonversion> (compatible; Pleroma <version>)" present in `version` field in response from `/api/v1/instance` and `/api/v2/instance`
## Flake IDs
@ -39,6 +39,9 @@ Has these additional fields under the `pleroma` object:
- `emoji_reactions`: A list with emoji / reaction maps. The format is `{name: "☕", count: 1, me: true}`. Contains no information about the reacting users, for that use the `/statuses/:id/reactions` endpoint.
- `parent_visible`: If the parent of this post is visible to the user or not.
- `pinned_at`: a datetime (iso8601) when status was pinned, `null` otherwise.
- `quotes_count`: the count of status quotes.
- `non_anonymous`: true if the source post specifies the poll results are not anonymous. Currently only implemented by Smithereen.
- `bookmark_folder`: the ID of the folder bookmark is stored within (if any).
The `GET /api/v1/statuses/:id/source` endpoint additionally has the following attributes:
@ -64,6 +67,12 @@ Some apps operate under the assumption that no more than 4 attachments can be re
Pleroma does not process remote images and therefore cannot include fields such as `meta` and `blurhash`. It does not support focal points or aspect ratios. The frontend is expected to handle it.
## Bookmarks
The `GET /api/v1/bookmarks` endpoint accepts optional parameter `folder_id` for bookmark folder ID.
The `POST /api/v1/statuses/:id/bookmark` endpoint accepts optional parameter `folder_id` for bookmark folder ID.
## Accounts
The `id` parameter can also be the `nickname` of the user. This only works in these endpoints, not the deeper nested ones for following etc.
@ -304,19 +313,27 @@ Has these additional parameters (which are the same as in Pleroma-API):
`GET /api/v1/instance` has additional fields
- `max_toot_chars`: The maximum characters per post
- `max_media_attachments`: Maximum number of post media attachments
- `chat_limit`: The maximum characters per chat message
- `description_limit`: The maximum characters per image description
- `poll_limits`: The limits of polls
- `shout_limit`: The maximum characters per Shoutbox message
- `upload_limit`: The maximum upload file size
- `avatar_upload_limit`: The same for avatars
- `background_upload_limit`: The same for backgrounds
- `banner_upload_limit`: The same for banners
- `background_image`: A background image that frontends can use
- `pleroma.metadata.account_activation_required`: Whether users are required to confirm their emails before signing in
- `pleroma.metadata.birthday_required`: Whether users are required to provide their birth day when signing in
- `pleroma.metadata.birthday_min_age`: The minimum user age (in days)
- `pleroma.metadata.features`: A list of supported features
- `pleroma.metadata.federation`: The federation restrictions of this instance
- `pleroma.metadata.fields_limits`: A list of values detailing the length and count limitation for various instance-configurable fields.
- `pleroma.metadata.post_formats`: A list of the allowed post format types
- `vapid_public_key`: The public key needed for push messages
- `pleroma.stats.mau`: Monthly active user count
- `pleroma.vapid_public_key`: The public key needed for push messages
In, `GET /api/v2/instance` Pleroma-specific fields are all moved into `pleroma` object. `max_toot_chars`, `poll_limits` and `upload_limit` are replaced with their MastoAPI counterparts.
Pleroma Conversations have the same general structure that Mastodon Conversations have. The behavior differs in the following ways when using these endpoints:
@ -382,7 +452,7 @@ Pleroma Conversations have the same general structure that Mastodon Conversation
Conversations have the additional field `recipients` under the `pleroma` key. This holds a list of all the accounts that will receive a message in this conversation.
The status posting endpoint takes an additional parameter, `in_reply_to_conversation_id`, which, when set, will set the visiblity to direct and address only the people who are the recipients of that Conversation.
The status posting endpoint takes an additional parameter, `in_reply_to_conversation_id`, which, when set, will set the visibility to direct and address only the people who are the recipients of that Conversation.
⚠ Conversation IDs can be found in direct messages with the `pleroma.direct_conversation_id` key, do not confuse it with `pleroma.conversation_id`.
* `enabled` (Pleroma extension) enables the endpoint
* `ip_whitelist` (Pleroma extension) could be used to restrict access only to specified IPs
* `auth` sets the authentication (`false` for no auth; configurable to HTTP Basic Auth, see [prometheus-plugs](https://github.com/deadtrickster/prometheus-plugs#exporting) documentation)
* `format` sets the output format (`:text` or `:protobuf`)
* `path` sets the path to app metrics page
## `/api/pleroma/app_metrics`
### Exports Prometheus application metrics
* Method: `GET`
* Authentication: not required by default (see configuration options above)
* Params: none
* Response: text
## Grafana
### Config example
The following is a config example to use with [Grafana](https://grafana.com)
PromEx supports the ability to automatically publish dashboards to your Grafana server as well as register Annotations. If you do not wish to configure this capability you must generate the dashboard JSON files and import them directly. You can find the mix commands in the upstream [documentation](https://hexdocs.pm/prom_ex/Mix.Tasks.PromEx.Dashboard.Export.html). You can find the list of modules enabled in Pleroma for which you should generate dashboards for by examining the contents of the `lib/pleroma/prom_ex.ex` module.
## prometheus.yml
The following is a bare minimum config example to use with [Prometheus](https://prometheus.io) or Prometheus-compatible software like [VictoriaMetrics](https://victoriametrics.com).
@ -15,7 +15,7 @@ Pleroma requires some adjustments from the defaults for running the instance loc
2. Change the dev.secret.exs
* Change the scheme in `config :pleroma, Pleroma.Web.Endpoint` to http (see examples below)
* If you want to change other settings, you can do that too
3. You can now start the server `mix phx.server`. Once it's build and started, you can access the instance on `http://<host>:<port>` (e.g.http://localhost:4000 ) and should be able to do everything locally you normaly can.
3. You can now start the server `mix phx.server`. Once it's build and started, you can access the instance on `http://<host>:<port>` (e.g.http://localhost:4000 ) and should be able to do everything locally you normally can.
Example config to change the scheme to http. Change the port if you want to run on another port.
```elixir
@ -38,7 +38,7 @@ config :logger, :console,
## Testing
1. Create a `test.secret.exs` file with the content as shown below
1. Create a `config/test.secret.exs` file with the content as shown below
2. Create the database user and test database.
1. You can use the `config/setup_db.psql` as a template. Copy the file if you want and change the database name, user and password to the values for the test-database (e.g. 'pleroma_local_test' for database and user). Then run this file like you did during installation.
2. The tests will try to create the Database, so we'll have to allow our test-database user to create databases, `sudo -Hu postgres psql -c "ALTER USER pleroma_local_test WITH CREATEDB;"`
@ -59,7 +59,7 @@ Gentoo quite pointedly does not come with a cron daemon installed, and as such i
If you would not like to install the optional packages, remove them from this line.
If you're running this from a low-powered virtual machine, it should work though it will take some time. There were no issues on a VPS with a single core and 1GB of RAM; if you are using an even more limited device and run into issues, you can try creating a swapfile or use a more powerful machine running Gentoo to [cross build](https://wiki.gentoo.org/wiki/Cross_build_environment). If you have a wait ahead of you, now would be a good time to take a break, strech a bit, refresh your beverage of choice and/or get a snack, and reply to Arch users' posts with "I use Gentoo btw" as we do.
If you're running this from a low-powered virtual machine, it should work though it will take some time. There were no issues on a VPS with a single core and 1GB of RAM; if you are using an even more limited device and run into issues, you can try creating a swapfile or use a more powerful machine running Gentoo to [cross build](https://wiki.gentoo.org/wiki/Cross_build_environment). If you have a wait ahead of you, now would be a good time to take a break, stretch a bit, refresh your beverage of choice and/or get a snack, and reply to Arch users' posts with "I use Gentoo btw" as we do.
### Install PostgreSQL
@ -104,7 +104,7 @@ Not only does this make it much easier to deploy changes you make, as you can co
* Add a new system user for the Pleroma service and set up default directories:
Remove `,wheel` if you do not want this user to be able to use `sudo`, however note that being able to `sudo` as the `pleroma` user will make finishing the insallation and common maintenence tasks somewhat easier:
Remove `,wheel` if you do not want this user to be able to use `sudo`, however note that being able to `sudo` as the `pleroma` user will make finishing the installation and common maintenance tasks somewhat easier:
@ -49,7 +49,7 @@ Gentoo quite pointedly does not come with a cron daemon installed, and as such i
If you would not like to install the optional packages, remove them from this line.
If you're running this from a low-powered virtual machine, it should work though it will take some time. There were no issues on a VPS with a single core and 1GB of RAM; if you are using an even more limited device and run into issues, you can try creating a swapfile or use a more powerful machine running Gentoo to [cross build](https://wiki.gentoo.org/wiki/Cross_build_environment). If you have a wait ahead of you, now would be a good time to take a break, strech a bit, refresh your beverage of choice and/or get a snack, and reply to Arch users' posts with "I use Gentoo btw" as we do.
If you're running this from a low-powered virtual machine, it should work though it will take some time. There were no issues on a VPS with a single core and 1GB of RAM; if you are using an even more limited device and run into issues, you can try creating a swapfile or use a more powerful machine running Gentoo to [cross build](https://wiki.gentoo.org/wiki/Cross_build_environment). If you have a wait ahead of you, now would be a good time to take a break, stretch a bit, refresh your beverage of choice and/or get a snack, and reply to Arch users' posts with "I use Gentoo btw" as we do.
Currently there are two options available for NetBSD: manual installation (from source) or using experimental package from [pkgsrc-wip](https://github.com/NetBSD/pkgsrc-wip/tree/master/pleroma).
WIP package can be installed via pkgsrc and can be crosscompiled for easier binary distribution. Source installation most probably will be restricted to a single machine.
## pkgsrc installation
WIP package creates Mix.Release (similar to how Docker images are built) but doesn't bundle Erlang runtime, listing it as a dependency instead. This allows for easier and more modular installations, especially on weaker machines. Currently this method also does not support all features of `pleroma_ctl` command (like changing installation type or managing frontends) as NetBSD is not yet a supported binary flavour of Pleroma's CI.
In any case, you can install it the same way as any other `pkgsrc-wip` package:
Use `bmake package` to create a binary package. This can come especially handy if you're targeting embedded or low-power systems and are crosscompiling on a more powerful machine.
> Note: Elixir has [endianness bug](https://github.com/elixir-lang/elixir/issues/2785) which requires it to be compiled on a machine with the same endianness. In other words, package crosscompiled on amd64 (little endian) won't work on powerpc or sparc machines (big endian). While _in theory™_ nothing catastrophic should happen, one can see that for example regexes won't work properly. Some distributions just strip this warning away, so it doesn't bother the users... anyway, you've been warned.
## Source installation
pkgin should have been installed by the NetBSD installer if you selected
the right options. If it isn't installed, install it using pkg_add.
the right options. If it isn't installed, install it using `pkg_add`.
Note that `postgresql11-contrib` is needed for the Postgres extensions
Pleroma uses.
> Note: you can use modern versions of PostgreSQL. In this case, just use `postgresql16-contrib` and so on.
The `mksh` shell is needed to run the Elixir `mix` script.
### Install media / graphics packages (optional, see [`docs/installation/optional/media_graphics_packages.md`](../installation/optional/media_graphics_packages.md))
Configure Pleroma. Note that you need a domain name at this point:
Get deps and compile:
```
$ cd /home/pleroma/pleroma
$ export MIX_ENV=prod
$ mix deps.get
$ MIX_ENV=prod mix pleroma.instance gen # You will be asked a few questions here.
$ mix compile
```
Since Postgres is configured, we can now initialize the database. There should
now be a file in `config/setup_db.psql` that makes this easier. Edit it, and
*change the password* to a password of your choice. Make sure it is secure, since
## Install media / graphics packages (optional, see [`docs/installation/optional/media_graphics_packages.md`](../installation/optional/media_graphics_packages.md))
From now on, you may encounter `$PREFIX` variable in the paths. This variable indicates your current local pkgsrc prefix. Usually it's `/usr/pkg` unless you configured it otherwise. Translating to pkgsrc's lingo, it's called `LOCALBASE`, which essentially means the same this. You may want to set it up for your local shell session (this uses `mksh` which should already be installed as one of the required dependencies):
```
$ export PREFIX=$(pkg_info -Q LOCALBASE mksh)
$ echo $PREFIX
/usr/pkg
```
## Setting up your instance
Now, you need to configure your instance. During this initial configuration, you will be asked some questions about your server. You will need a domain name at this point; it doesn't have to be deployed, but changing it later will be very cumbersome.
If you've installed via pkgsrc, `pleroma_ctl` should already be in your `PATH`; if you've installed from source, it's located at `/home/pleroma/pleroma/release/bin/pleroma_ctl`.
```
$ su -l pleroma
$ pleroma_ctl instance gen --output $PREFIX/etc/pleroma/config.exs --output-psql /tmp/setup_db.psql
```
During installation, you will be asked about static and upload directories. Don't forget to create them and update permissions:
Postgres allows connections from all users without a password by default. To
fix this, edit `/usr/pkg/pgsql/data/pg_hba.conf`. Change every `trust` to
fix this, edit `$PREFIX/pgsql/data/pg_hba.conf`. Change every `trust` to
`password`.
Once this is done, restart Postgres with `# /etc/rc.d/pgsql restart`.
Run the database migrations.
### pkgsrc installation
```
pleroma_ctl migrate
```
### Source installation
You will need to do this whenever you update with `git pull`:
```
$ cd /home/pleroma/pleroma
$ MIX_ENV=prod mix ecto.migrate
```
## Configuring nginx
Install the example configuration file
`/home/pleroma/pleroma/installation/pleroma.nginx` to
`/usr/pkg/etc/nginx.conf`.
(`$PREFIX/share/examples/pleroma/pleroma.nginx` or `/home/pleroma/pleroma/installation/pleroma.nginx`) to
`$PREFIX/etc/nginx.conf`.
Note that it will need to be wrapped in a `http {}` block. You should add
settings for the nginx daemon outside of the http block, for example:
@ -176,27 +237,45 @@ Let's add auto-renewal to `/etc/daily.local`
--stateless
```
## Creating a startup script for Pleroma
## Autostart
Copy the startup script to the correct location and make sure it's executable:
For properly functioning instance, you will need pleroma (backend service), nginx (reverse proxy) and postgresql (database) services running. There's no requirement for them to reside on the same machine, but you have to provide autostart for each of them.
To check that it started properly and didn't fail right after starting, you can run `ps aux | grep postgres`, there should be multiple lines of output.
#### httpd
httpd will have three fuctions:
httpd will have three functions:
* redirect requests trying to reach the instance over http to the https URL
* serve a robots.txt file
@ -225,7 +225,7 @@ pass in quick on $if inet6 proto icmp6 to ($if) icmp6-type { echoreq unreach par
pass in quick on $if proto tcp to ($if) port { http https } # relayd/httpd
pass in quick on $if proto tcp from $authorized_ssh_clients to ($if) port ssh
```
Replace *<network interface\>* by your server's network interface name (which you can get with ifconfig). Consider replacing the content of the authorized\_ssh\_clients macro by, for exemple, your home IP address, to avoid SSH connection attempts from bots.
Replace *<network interface\>* by your server's network interface name (which you can get with ifconfig). Consider replacing the content of the authorized\_ssh\_clients macro by, for example, your home IP address, to avoid SSH connection attempts from bots.
Check pf's configuration by running `pfctl -nf /etc/pf.conf`, load it with `pfctl -f /etc/pf.conf` and enable pf at boot with `rcctl enable pf`.
@ -238,7 +238,7 @@ At this point if you open your (sub)domain in a browser you should see a 502 err
systemctl enable pleroma
```
If everything worked, you should see Pleroma-FE when visiting your domain. If that didn't happen, try reviewing the installation steps, starting Pleroma in the foreground and seeing if there are any errrors.
If everything worked, you should see Pleroma-FE when visiting your domain. If that didn't happen, try reviewing the installation steps, starting Pleroma in the foreground and seeing if there are any errors.
Questions about the installation or didn’t it work as it should be, ask in [#pleroma:libera.chat](https://matrix.to/#/#pleroma:libera.chat) via Matrix or **#pleroma** on **libera.chat** via IRC, you can also [file an issue on our Gitlab](https://git.pleroma.social/pleroma/pleroma-support/issues/new).
@ -292,7 +292,7 @@ defmodule Mix.Tasks.Pleroma.Instance do
ifdb_configurable?do
shell_info(
" Please transfer your config to the database after running database migrations. Refer to \"Transfering the config to/from the database\" section of the docs for more information."
" Please transfer your config to the database after running database migrations. Refer to \"Transferring the config to/from the database\" section of the docs for more information."
)
end
else
@ -352,6 +352,4 @@ defmodule Mix.Tasks.Pleroma.Instance do
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
