Merge pull request #487 from kmycode/kb-draft-10.1

Release: 10.1

.bundler-audit.yml

@@ -0,0 +1,6 @@
+---
+ignore:
+  # devise-two-factor advisory about brute-forcing TOTP
+  # We have rate-limits on authentication endpoints in place (including second
+  # factor verification) since Mastodon v3.2.0
+  - CVE-2024-0227

.ruby-version

@@ -1 +1 @@
-3.2.2
+3.2.3

CHANGELOG.md

@@ -2,6 +2,65 @@
 
 All notable changes to this project will be documented in this file.
 
+## [4.2.4] - 2024-01-24
+
+### Fixed
+
+- Fix error when processing remote files with unusually long names ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28823))
+- Fix processing of compacted single-item JSON-LD collections ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28816))
+- Retry 401 errors on replies fetching ([ShadowJonathan](https://github.com/mastodon/mastodon/pull/28788))
+- Fix `RecordNotUnique` errors in LinkCrawlWorker ([tribela](https://github.com/mastodon/mastodon/pull/28748))
+- Fix Mastodon not correctly processing HTTP Signatures with query strings ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28443), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/28476))
+- Fix potential redirection loop of streaming endpoint ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28665))
+- Fix streaming API redirection ignoring the port of `streaming_api_base_url` ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28558))
+- Fix error when processing link preview with an array as `inLanguage` ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28252))
+- Fix unsupported time zone or locale preventing sign-up ([Gargron](https://github.com/mastodon/mastodon/pull/28035))
+- Fix "Hide these posts from home" list setting not refreshing when switching lists ([brianholley](https://github.com/mastodon/mastodon/pull/27763))
+- Fix missing background behind dismissable banner in web UI ([Gargron](https://github.com/mastodon/mastodon/pull/27479))
+- Fix line wrapping of language selection button with long locale codes ([gunchleoc](https://github.com/mastodon/mastodon/pull/27100), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/27127))
+- Fix `Undo Announce` activity not being sent to non-follower authors ([MitarashiDango](https://github.com/mastodon/mastodon/pull/18482))
+- Fix N+1s because of association preloaders not actually getting called ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28339))
+- Fix empty column explainer getting cropped under certain conditions ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28337))
+- Fix `LinkCrawlWorker` error when encountering empty OEmbed response ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28268))
+- Fix call to inefficient `delete_matched` cache method in domain blocks ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28367))
+
+### Security
+
+- Add rate-limit of TOTP authentication attempts at controller level ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28801))
+
+## [4.2.3] - 2023-12-05
+
+### Fixed
+
+- Fix dependency on `json-canonicalization` version that has been made unavailable since last release
+
+## [4.2.2] - 2023-12-04
+
+### Changed
+
+- Change dismissed banners to be stored server-side ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27055))
+- Change GIF max matrix size error to explicitly mention GIF files ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27927))
+- Change `Follow` activities delivery to bypass availability check ([ShadowJonathan](https://github.com/mastodon/mastodon/pull/27586))
+- Change single-column navigation notice to be displayed outside of the logo container ([renchap](https://github.com/mastodon/mastodon/pull/27462), [renchap](https://github.com/mastodon/mastodon/pull/27476))
+- Change Content-Security-Policy to be tighter on media paths ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/26889))
+- Change post language code to include country code when relevant ([gunchleoc](https://github.com/mastodon/mastodon/pull/27099), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/27207))
+
+### Fixed
+
+- Fix upper border radius of onboarding columns ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27890))
+- Fix incoming status creation date not being restricted to standard ISO8601 ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27655), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/28081))
+- Fix some posts from threads received out-of-order sometimes not being inserted into timelines ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27653))
+- Fix posts from force-sensitized accounts being able to trend ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27620))
+- Fix error when trying to delete already-deleted file with OpenStack Swift ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27569))
+- Fix batch attachment deletion when using OpenStack Swift ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27554))
+- Fix processing LDSigned activities from actors with unknown public keys ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27474))
+- Fix error and incorrect URLs in `/api/v1/accounts/:id/featured_tags` for remote accounts ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27459))
+- Fix report processing notice not mentioning the report number when performing a custom action ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27442))
+- Fix handling of `inLanguage` attribute in preview card processing ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27423))
+- Fix own posts being removed from home timeline when unfollowing a used hashtag ([kmycode](https://github.com/mastodon/mastodon/pull/27391))
+- Fix some link anchors being recognized as hashtags ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27271), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/27584))
+- Fix format-dependent redirects being cached regardless of requested format ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27634))
+
 ## [4.2.1] - 2023-10-10
 
 ### Added

Dockerfile

@@ -1,11 +1,7 @@
 # syntax=docker/dockerfile:1.4
 
-# Please see https://docs.docker.com/engine/reference/builder for information about
-# the extended buildx capabilities used in this file.
-# Make sure multiarch TARGETPLATFORM is available for interpolation
-# See: https://docs.docker.com/build/building/multi-platform/
-ARG TARGETPLATFORM=${TARGETPLATFORM}
-ARG BUILDPLATFORM=${BUILDPLATFORM}
+FROM ghcr.io/moritzheiber/ruby-jemalloc:3.2.3-slim as ruby
+FROM node:${NODE_VERSION} as build
 
 # Ruby image to use for base image, change with [--build-arg RUBY_VERSION="3.2.2"]
 ARG RUBY_VERSION="3.2.2"

Gemfile.lock

@@ -481,7 +481,8 @@ GEM
       timeout
     net-smtp (0.4.0)
       net-protocol
-    nio4r (2.5.9)
+    net-ssh (7.1.0)
+    nio4r (2.7.0)
     nokogiri (1.16.0)
       mini_portile2 (~> 2.8.2)
       racc (~> 1.4)
@@ -544,7 +545,7 @@ GEM
     psych (5.1.2)
       stringio
     public_suffix (5.0.4)
-    puma (6.4.1)
+    puma (6.4.2)
       nio4r (~> 2.0)
     pundit (2.3.1)
       activesupport (>= 3.0.0)

app/controllers/api/v1/streaming_controller.rb

@@ -2,7 +2,7 @@
 
 class Api::V1::StreamingController < Api::BaseController
   def index
-    if Rails.configuration.x.streaming_api_base_url == request.host
+    if same_host?
       not_found
     else
       redirect_to streaming_api_url, status: 301, allow_other_host: true
@@ -11,6 +11,11 @@ def index
 
   private
 
+  def same_host?
+    base_url = Addressable::URI.parse(Rails.configuration.x.streaming_api_base_url)
+    request.host == base_url.host && request.port == (base_url.port || 80)
+  end
+
   def streaming_api_url
     Addressable::URI.parse(request.url).tap do |uri|
       base_url = Addressable::URI.parse(Rails.configuration.x.streaming_api_base_url)

app/controllers/auth/sessions_controller.rb

@@ -1,6 +1,10 @@
 # frozen_string_literal: true
 
 class Auth::SessionsController < Devise::SessionsController
+  include Redisable
+
+  MAX_2FA_ATTEMPTS_PER_HOUR = 10
+
   layout 'auth'
 
   skip_before_action :check_self_destruct!
@@ -130,9 +134,23 @@ def clear_attempt_from_session
     session.delete(:attempt_user_updated_at)
   end
 
+  def clear_2fa_attempt_from_user(user)
+    redis.del(second_factor_attempts_key(user))
+  end
+
+  def check_second_factor_rate_limits(user)
+    attempts, = redis.multi do |multi|
+      multi.incr(second_factor_attempts_key(user))
+      multi.expire(second_factor_attempts_key(user), 1.hour)
+    end
+
+    attempts >= MAX_2FA_ATTEMPTS_PER_HOUR
+  end
+
   def on_authentication_success(user, security_measure)
     @on_authentication_success_called = true
 
+    clear_2fa_attempt_from_user(user)
     clear_attempt_from_session
 
     user.update_sign_in!(new_sign_in: true)
@@ -164,4 +182,8 @@ def on_authentication_failure(user, security_measure, failure_reason)
       user_agent: request.user_agent
     )
   end
+
+  def second_factor_attempts_key(user)
+    "2fa_auth_attempts:#{user.id}:#{Time.now.utc.hour}"
+  end
 end

app/controllers/concerns/auth/two_factor_authentication_concern.rb

@@ -66,6 +66,11 @@ def authenticate_with_two_factor_via_webauthn(user)
   end
 
   def authenticate_with_two_factor_via_otp(user)
+    if check_second_factor_rate_limits(user)
+      flash.now[:alert] = I18n.t('users.rate_limited')
+      return prompt_for_two_factor(user)
+    end
+
     if valid_otp_attempt?(user)
       on_authentication_success(user, :otp)
     else

app/helpers/jsonld_helper.rb

@@ -163,7 +163,7 @@ def safe_for_forwarding?(original, compacted)
     end
   end
 
-  def fetch_resource(uri, id, on_behalf_of = nil)
+  def fetch_resource(uri, id, on_behalf_of = nil, request_options: {})
     unless id
       json = fetch_resource_without_id_validation(uri, on_behalf_of)
 
@@ -172,14 +172,14 @@ def fetch_resource(uri, id, on_behalf_of = nil)
       uri = json['id']
     end
 
-    json = fetch_resource_without_id_validation(uri, on_behalf_of)
+    json = fetch_resource_without_id_validation(uri, on_behalf_of, request_options: request_options)
     json.present? && json['id'] == uri ? json : nil
   end
 
-  def fetch_resource_without_id_validation(uri, on_behalf_of = nil, raise_on_temporary_error = false)
+  def fetch_resource_without_id_validation(uri, on_behalf_of = nil, raise_on_temporary_error = false, request_options: {})
     on_behalf_of ||= Account.representative
 
-    build_request(uri, on_behalf_of).perform do |response|
+    build_request(uri, on_behalf_of, options: request_options).perform do |response|
       raise Mastodon::UnexpectedResponseError, response unless response_successful?(response) || response_error_unsalvageable?(response) || !raise_on_temporary_error
 
       body_to_json(response.body_with_limit) if response.code == 200
@@ -212,8 +212,8 @@ def response_error_unsalvageable?(response)
     response.code == 501 || ((400...500).cover?(response.code) && ![401, 408, 429].include?(response.code))
   end
 
-  def build_request(uri, on_behalf_of = nil)
-    Request.new(:get, uri).tap do |request|
+  def build_request(uri, on_behalf_of = nil, options: {})
+    Request.new(:get, uri, **options).tap do |request|
       request.on_behalf_of(on_behalf_of) if on_behalf_of
       request.add_headers('Accept' => 'application/activity+json, application/ld+json')
     end

app/javascript/mastodon/features/compose/components/privacy_dropdown.jsx

@@ -169,6 +169,7 @@ class PrivacyDropdown extends PureComponent {
     value: PropTypes.string.isRequired,
     onChange: PropTypes.func.isRequired,
     noDirect: PropTypes.bool,
+    noLimited: PropTypes.bool,
     replyToLimited: PropTypes.bool,
     container: PropTypes.func,
     disabled: PropTypes.bool,
@@ -260,6 +261,10 @@ class PrivacyDropdown extends PureComponent {
       );
     }
 
+    if (this.props.noLimited) {
+      this.options = this.options.filter((opt) => !['mutual', 'circle'].includes(opt.value));
+    }
+
     this.selectableOptions = [...this.options];
 
     if (!enableLoginPrivacy) {

app/javascript/mastodon/features/ui/components/boost_modal.jsx

@@ -110,6 +110,7 @@ class BoostModal extends ImmutablePureComponent {
           {status.get('visibility') !== 'private' && !status.get('reblogged') && (
             <PrivacyDropdown
               noDirect
+              noLimited
               value={privacy}
               container={this._findContainer}
               onChange={this.props.onChangeBoostPrivacy}

app/lib/account_statuses_filter.rb

@@ -33,7 +33,8 @@ def results
     available_visibilities -= [:unlisted] if (domain_block&.detect_invalid_subscription || misskey_software?) && @account.user&.setting_reject_unlisted_subscription
     available_visibilities -= [:login] if current_account.nil?
 
-    scope.merge!(scope.where(spoiler_text: ['', nil])) if domain_block&.reject_send_sensitive
+    scope.merge!(scope.where(sensitive: false)) if domain_block&.reject_send_sensitive
+
     scope.merge!(scope.where(searchability: available_searchabilities))
     scope.merge!(scope.where(visibility: available_visibilities))
 
@@ -153,9 +154,9 @@ def truthy_param?(key)
   end
 
   def domain_block
-    return nil if @account.nil? || @account.local?
+    return nil if @current_account.nil? || @current_account.local?
 
-    @domain_block = DomainBlock.find_by(domain: @account.domain)
+    @domain_block = DomainBlock.find_by(domain: @current_account.domain)
   end
 
   def misskey_software?

app/lib/status_reach_finder.rb

@@ -38,47 +38,39 @@ def all_inboxes
   private
 
   def reached_account_inboxes
-    # When the status is a reblog, there are no interactions with it
-    # directly, we assume all interactions are with the original one
-
-    if @status.reblog?
-      []
-    elsif @status.limited_visibility?
-      Account.where(id: mentioned_account_ids).where.not(domain: banned_domains).inboxes
-    else
-      Account.where(id: reached_account_ids).where.not(domain: banned_domains + friend_domains).inboxes
-    end
+    Account.where(id: reached_account_ids).where.not(domain: banned_domains).inboxes
   end
 
   def reached_account_inboxes_for_misskey
-    if @status.reblog? || @status.limited_visibility?
-      []
-    else
-      Account.where(id: reached_account_ids, domain: banned_domains_for_misskey - friend_domains).inboxes
-    end
+    Account.where(id: reached_account_ids, domain: banned_domains_for_misskey - friend_domains).inboxes
   end
 
   def reached_account_inboxes_for_friend
-    if @status.reblog? || @status.limited_visibility?
-      []
-    else
-      Account.where(id: reached_account_ids, domain: friend_domains).inboxes
-    end
+    Account.where(id: reached_account_ids, domain: friend_domains).inboxes
   end
 
   def reached_account_ids
-    [
-      replied_to_account_id,
-      reblog_of_account_id,
-      mentioned_account_ids,
-      reblogs_account_ids,
-      favourites_account_ids,
-      replies_account_ids,
-      quoted_account_id,
-    ].tap do |arr|
-      arr.flatten!
-      arr.compact!
-      arr.uniq!
+    # When the status is a reblog, there are no interactions with it
+    # directly, we assume all interactions are with the original one
+
+    if @status.reblog?
+      [reblog_of_account_id]
+    elsif @status.limited_visibility?
+      [mentioned_account_ids]
+    else
+      [
+        replied_to_account_id,
+        reblog_of_account_id,
+        mentioned_account_ids,
+        reblogs_account_ids,
+        favourites_account_ids,
+        replies_account_ids,
+        quoted_account_id,
+      ].tap do |arr|
+        arr.flatten!
+        arr.compact!
+        arr.uniq!
+      end
     end
   end
 

app/models/concerns/account/other_settings.rb

@@ -15,13 +15,6 @@ def translatable_private?
       false
     end
 
-    def link_preview?
-      return user.setting_link_preview if local? && user.present?
-      return settings['link_preview'] if settings.present? && settings.key?('link_preview')
-
-      true
-    end
-
     def allow_quote?
       return user.setting_allow_quote if local? && user.present?
       return settings['allow_quote'] if settings.present? && settings.key?('allow_quote')
@@ -95,7 +88,6 @@ def public_settings
         'hide_following_count' => hide_following_count?,
         'hide_followers_count' => hide_followers_count?,
         'translatable_private' => translatable_private?,
-        'link_preview' => link_preview?,
         'allow_quote' => allow_quote?,
         'emoji_reaction_policy' => Setting.enable_emoji_reaction ? emoji_reaction_policy.to_s : 'block',
       }

app/models/concerns/user/has_settings.rb

@@ -127,6 +127,10 @@ def setting_allow_quote
     settings['allow_quote']
   end
 
+  def setting_reject_send_limited_to_suspects
+    settings['reject_send_limited_to_suspects']
+  end
+
   def setting_noindex
     settings['noindex']
   end
@@ -135,10 +139,6 @@ def setting_translatable_private
     settings['translatable_private']
   end
 
-  def setting_link_preview
-    settings['link_preview']
-  end
-
   def setting_dtl_force_visibility
     settings['dtl_force_visibility']&.to_sym || :unchange
   end