Anon/pleroma
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
d21aa1a77c
pleroma/changelog.d
History
Mark Felder ede414094f RichMedia refactor 
Rich Media parsing was previously handled on-demand with a 2 second HTTP request timeout and retained only in Cachex. Every time a Pleroma instance is restarted it will have to request and parse the data for each status with a URL detected. When fetching a batch of statuses they were processed in parallel to attempt to keep the maximum latency at 2 seconds, but often resulted in a timeline appearing to hang during loading due to a URL that could not be successfully reached. URLs which had images links that expire (Amazon AWS) were parsed and inserted with a TTL to ensure the image link would not break.

Rich Media data is now cached in the database and fetched asynchronously. Cachex is used as a read-through cache. When the data becomes available we stream an update to the clients. If the result is returned quickly the experience is almost seamless. Activities were already processed for their Rich Media data during ingestion to warm the cache, so users should not normally encounter the asynchronous loading of the Rich Media data.

Implementation notes:

- The async worker is a Task with a globally unique process name to prevent duplicate processing of the same URL
- The Task will attempt to fetch the data 3 times with increasing sleep time between attempts
- The HTTP request obeys the default HTTP request timeout value instead of 2 seconds
- URLs that cannot be successfully parsed due to an unexpected error receives a negative cache entry for 15 minutes
- URLs that fail with an expected error will receive a negative cache with no TTL
- Activities that have no detected URLs insert a nil value in the Cachex :scrubber_cache so we do not repeat parsing the object content with Floki every time the activity is rendered
- Expiring image URLs are handled with an Oban job
- There is no automatic cleanup of the Rich Media data in the database, but it is safe to delete at any time
- The post draft/preview feature makes the URL processing synchronous so the rendered post preview will have an accurate rendering

Overall performance of timelines and creating new posts which contain URLs is greatly improved.
2024-05-07 19:54:56 -04:00
..
2.6.0-mergeback.skip Bump package version for mergeback 2023-10-31 20:49:31 -04:00
2.6.1-mergeback.skip Skip changelog for 2.6.1 mergeback 2023-12-15 19:45:23 -05:00
3900.change Update to Phoenix 1.7 2023-06-03 14:06:28 -04:00
3987.fix add changelog 2023-12-07 20:33:23 +09:00
account-rendering-auth-check.fix Fix authentication check on account rendering when bio is defined 2024-01-04 16:57:48 +03:00
add-outbox.fix ap userview: add outbox field. 2023-12-15 09:55:06 +04:00
anonymous-exception-else.fix Fix #strip_report_status_data 2023-11-27 17:56:43 +04:00
api-docs.skip Update API docs for my changes 2024-01-17 17:13:21 +01:00
atom-leak.skip Changelog 2024-02-14 18:04:39 -05:00
authorize-interaction.add Support /authorize-interaction route used by Mastodon 2023-11-02 13:47:23 +01:00
backups-follows.add Include following/followers in backups 2024-03-10 23:53:12 +01:00
bad_inbox_request.change Return a 400 from a bad delivery attempt to the inbox 2023-12-19 13:56:17 -05:00
bandit.change Revert "Add support for a "deps" changelog type and document deps changes since 2.6.1 release" 2024-02-16 12:53:18 -05:00
bare_uri_test.skip ObjectValidators.BareUriTest: Replace calls of SafeText to BareUri 2023-11-08 02:17:50 +01:00
benchee.skip Ensure benchee doesn't run unless we are executing benchmarks 2023-11-08 12:44:57 -05:00
blurhash.change Docs: Remove rust references 2023-11-28 15:21:34 +04:00
bookmark-folders.add Allow to group bookmarks in folders 2024-03-01 11:04:01 +01:00
bookmark-folders.skip Fix BookmarkFolderView, add test 2024-03-20 13:24:43 +01:00
bugfix-ccworks.fix Test incoming federation from Convergence AP Bridge 2024-01-26 17:05:18 +01:00
build-release-with-local-libvips.skip Changelog 2023-12-11 12:48:53 +04:00
chat-attachment-empty-array.fix ChatMessage: Tolerate attachment field set to an empty array 2023-12-29 05:32:47 +01:00
config-stat-symlink.fix Config: Check the permissions of the linked file instead of the symlink↵ 2024-02-14 18:19:50 +01:00
content-length.fix MediaProxy RFC compliance 2024-02-14 13:25:52 -05:00
deprecations2.skip More deprecation fixes 2023-12-20 16:47:58 -05:00
deprecations.skip Chase deprecations/warnings for Elixir 1.15 2023-12-20 20:17:25 +00:00
deps-bump-2024-01-25.skip Bump dependencies 2024-01-26 15:50:45 -05:00
description-meilisearch-type.skip Fix type in description 2024-04-27 20:29:23 +02:00
dialyzer2.skip Changelog 2024-01-26 21:05:43 -05:00
dialyzer3.skip Changelog 2024-01-31 10:13:02 -05:00
dialyzer4.skip Changelog 2024-02-02 12:14:21 -05:00
dialyzer.skip Dialyzer fixes 2024-01-20 19:29:29 -05:00
digest_emails.fix Fix digest email processing, consolidate Oban queues 2023-11-07 15:14:36 -05:00
doc-fix.skip Add changelog 2023-11-29 19:42:40 +04:00
docs-max-elixir-erlang.change Bump version to 2.6.1 2023-12-14 20:14:41 -05:00
emoji-download-paginate.fix Fetch count before downloading the pack and use that as page size 2024-01-07 17:07:50 +03:00
emoji-use-v1.fix Make remote emoji packs API use specifically the V1 URL 2024-01-07 15:30:52 +03:00
exile-bsds.skip Exile: switch to fork with BSD compile fix 2024-01-22 13:44:20 -05:00
exile-freebsd.skip Exile: change to upstream pre-release commit that fixes build on FreeBSD 2024-02-23 15:36:37 -05:00
exile-macos.skip Exile: fix for MacOS dev environments 2024-01-22 18:37:13 -05:00
exile.skip Replace custom fifo implementation with Exile 2024-01-22 10:13:17 -05:00
favicon.add Add support for configuring a favicon and embed PWA manifest in server-generated-meta 2023-11-14 11:05:23 +01:00
federation_status-access.change router: Make /federation_status publicly available 2023-11-14 10:48:30 +01:00
federator-modules.remove Changelog 2023-12-28 14:55:26 -05:00
federator.skip Changelog 2023-12-28 23:32:21 -05:00
fep-2c59.add FEP-2c59, add "webfinger" to user actor 2024-04-11 17:50:11 +02:00
ffmpeg-limiter.add MediaHelper: cache failed URLs for 15 minutes to prevent excessive retries 2024-03-19 12:14:03 -04:00
finch_redirects.fix Fix following redirects with Finch 2023-12-29 00:50:50 -05:00
fix-bookmark-folder-tests.skip Add changelog 2024-03-19 15:03:16 +04:00
fix-dockerfile.skip Dockerfile: Use platform libvips. 2023-12-12 11:36:34 +04:00
fix-duplicate-inbox-deliveries.fix Fix duplicate inbox deliveries 2024-01-11 20:48:35 +03:00
fix-otp-comparison.skip Fix invalid string comparison for OTP versions and replace with config 2023-12-20 23:13:33 +00:00
fix-tests.skip StatusViewTest: Fix tests. 2023-12-13 11:19:08 +04:00
force-mention-mrf.add Add ForceMention mrf 2024-03-01 18:16:09 +01:00
framegrabs.fix Fix ffmpeg framegrabs with Exile 2024-03-07 17:38:21 -05:00
frontend-management.add Bump version to 2.6.1 2023-12-14 20:14:41 -05:00
generate-unset-user-keys-migration.skip Fix GenerateUnsetUserKeys migration 2023-11-13 14:18:22 +01:00
group-actor.add Add changelog for group actors 2023-12-27 12:28:11 -05:00
gun_pool2.fix Prevent publisher jobs from erroring if the connection pool is full 2024-02-22 14:14:40 -05:00
gun_pool3.skip Rename variable to make the worker retry logic easier to read 2024-02-23 11:12:10 -05:00
gun_pool.fix Fix Gun connection supervisor logic error 2024-02-09 10:48:40 -05:00
gun-logs.skip Change some Gun connection pool logs to debug level 2024-02-14 13:28:32 -05:00
handle_object_fetch_failures.change Handle 401s as I have observed it in the wild 2023-12-28 23:09:33 -05:00
healthcheck-disabled-error.fix Bump version to 2.6.1 2023-12-14 20:14:41 -05:00
instance-contact-account.add Add contact account to InstanceView 2023-10-26 23:32:08 +02:00
instance-defdelegates.skip These functions in Pleroma.Instances should be defdelegates to Pleroma.Instances.Instance 2023-12-27 12:44:16 -05:00
instance-v2.add Implement api/v2/instance route 2023-08-11 00:08:05 +02:00
instance-v2.skip Add some missing fields to instanceV2 2024-02-27 13:25:26 +01:00
issue-3241.fix add changelog.d 2024-03-02 18:24:39 +09:00
last_status_at.change MastoAPI AccountView: Change last_status_at to be a date 2023-11-16 15:52:01 +01:00
link-verification.add Verify link ownership with rel="me" 2023-11-02 13:30:46 +01:00
loading-order-test-fix.skip Add changelog. 2023-11-30 09:55:04 +04:00
local-webfinger.fix update changelog 2023-11-02 13:31:59 +01:00
mastodon_directory.fix Pleroma.Web.MastodonAPI.DirectoryController: dialyzer errors 2024-01-28 17:41:07 -05:00
meilisearch.add Add changelog. 2023-11-12 15:00:51 +04:00
memleak.fix Force more frequent full_sweep GC runs on the Websocket processes 2024-02-17 09:53:52 -05:00
mergeback-2.6.2.skip Mergeback of security release 2.6.2 2024-02-20 09:34:11 +01:00
migration-fix.skip Permit the index creation to run concurrently 2023-12-08 15:48:00 -05:00
mrf_hashtags.fix MRF.HashtagPolicy: fix dialyzer error 2024-01-22 18:37:19 -05:00
mrf-regex-error.fix MRF: Log sensible error for subdomains_regex 2024-01-02 15:37:41 +01:00
mrf-steal-emoji-extname.fix MRF.StealEmojiPolicy: Properly add fallback extension to filenames missing one 2024-01-07 19:37:35 +01:00
nil-content-map.fix Support objects with a nil contentMap (firefish) 2024-01-11 07:16:45 +01:00
no-async-with-clear-config.skip Tests: Don't run tests that use clear_config asynchronously. 2023-12-10 16:27:23 +04:00
notifications-index.fix Changelog for notifications fix pulled in from Rebased 2024-02-12 14:14:38 -05:00
notifications.fix Notifications: filter on users rather than activities 2024-03-17 20:58:59 +01:00
oauth-nickname.skip Use User.full_nickname/1 in oauth html template 2024-02-09 16:46:45 +01:00
opengraph-rich-media-proxy.add Add media proxy to opengraph rich media cards 2023-12-12 18:32:02 +03:00
optimistic-inbox.change Optimistic Inbox 2023-12-08 18:13:43 -05:00
otp26.add Support for Erlang OTP 26 2023-12-29 23:22:31 -05:00
postgres-jit.change Update minimum Postgres version to 11.0; disable JIT 2024-03-18 15:36:26 -04:00
prioritize-direct-recipients.add Actually write changelog 2023-12-16 21:41:28 +04:00
promex.change Revert "Add support for a "deps" changelog type and document deps changes since 2.6.1 release" 2024-02-16 12:53:18 -05:00
public-polls.add Add entry to @context, tests 2023-10-29 00:02:17 +02:00
publisher_discard.change Activity publishing failures will prevent the job from retrying if the publishing request returns a 403 or 410 2023-12-29 00:25:33 -05:00
publisher_log.change Make the Publisher log error less noisy 2023-12-29 00:25:11 -05:00
qtfaststart.fix Change QTFastStart to recover gracefully if it encounters an error during bitstring matching 2023-12-29 14:12:44 -05:00
quotes-count.skip Count and display post quotes 2023-11-12 13:38:08 +00:00
reachability.change Change set_reachable/1 to an upsert 2023-12-06 15:50:00 -05:00
receiverworker-error-handling.fix ReceiverWorker: Make sure non-{:ok, _} is returned as {:error, …} 2024-04-17 07:43:47 +02:00
remote-fetcher-error.skip RemoteFetcherWorker: Make sure {:error, _} is returned on failure 2024-02-23 01:59:30 +01:00
rich_media_refactor.change RichMedia refactor 2024-05-07 19:54:56 -04:00
rich_media_tests.skip RichMedia.Helpers: move the validate_page_url/1 function to the Parser module 2024-02-06 18:34:02 -05:00
rich_media.fix Fix Rich Media Previews for updated activities 2024-02-04 23:47:04 -05:00
scrobble-url.add Add optional URL value for scrobbles 2023-11-23 04:51:51 -07:00
scrubbers-html4-GtS.add scrubbers/default: Add more formatting elements from HTML4 / GoToSocial 2023-11-16 18:27:32 +01:00
system-cflags.fix Bump version to 2.6.1 2023-12-14 20:14:41 -05:00
tesla.deps Tesla changelog 2024-02-15 10:24:32 -05:00
test-improvements.skip Add changelog 2024-03-19 10:38:29 +04:00
testsecrets.skip Clarify location of test.secret.exs file 2023-12-20 14:12:06 -05:00
transient-validators-defaults.change Update transient-validators-defaults.change 2024-03-19 12:03:43 +00:00
typo.skip Fix some more typos 2023-12-28 00:17:04 +01:00
vips.change Update changelogs 2023-11-17 11:21:46 -05:00
web_push.fix Fix Web Push notification delivery 2023-12-19 10:56:55 -05:00
websocket-refactor.change Websocket refactor changelog 2024-02-14 15:27:12 -05:00