Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(sender): graceful restart #1567

Merged
merged 9 commits into from
Dec 2, 2024

Conversation

colinlyguo
Copy link
Member

@colinlyguo colinlyguo commented Nov 28, 2024

Purpose or design rationale of this PR

This PR adds:

  • db status rollback.
  • blob tx fees precheck.

Note that:

  • use permanent delete (Unscoped) instead of soft delete to prevent database bloat, as repeated SendTransaction failures could write a large number of transactions to the database.
  • the db role should have "delete" permission after this PR.

PR title

Your PR title must follow conventional commits (as we are doing squash merge for each PR), so it must start with one of the following types:

  • fix: A bug fix

Deployment tag versioning

Has tag in common/version.go been updated or have you added bump-version label to this PR?

  • Yes

Breaking change label

Does this PR have the breaking-change label?

  • No, this PR is not a breaking change

Summary by CodeRabbit

Summary by CodeRabbit

  • New Features

    • Updated version number to v4.4.79.
    • Introduced a method to delete pending transactions by transaction hash.
  • Bug Fixes

    • Enhanced error handling and transaction management, including a rollback mechanism for failed transactions.
  • Improvements

    • Added validation for gas fees in transaction creation to ensure they meet minimum thresholds.
    • Improved clarity and consistency in method naming for transaction status updates in tests.

@colinlyguo colinlyguo added the bump-version Bump the version tag for deployment label Nov 28, 2024
Copy link

coderabbitai bot commented Nov 28, 2024

Walkthrough

The pull request introduces changes across several files, focusing on a version increment and enhancements to transaction handling. The version number in common/version/version.go is updated from "v4.4.78" to "v4.4.79". In rollup/internal/controller/sender/sender.go, modifications improve error handling and transaction management, including a rollback mechanism and stricter validation for transaction fees. Additionally, new methods for deleting transactions and updating statuses are added in rollup/internal/orm/pending_transaction.go.

Changes

File Path Change Summary
common/version/version.go Updated tag variable from "v4.4.78" to "v4.4.79".
rollup/internal/controller/sender/sender.go Enhanced Sender struct methods with improved error handling, transaction management, and validation.
rollup/internal/orm/pending_transaction.go Added DeleteTransactionByTxHash method to delete transactions by their hash; renamed UpdatePendingTransactionStatusByTxHash to UpdateTransactionStatusByTxHash.
rollup/internal/orm/orm_test.go Updated tests to reflect method renaming and added assertions for improved clarity and consistency.

Possibly related PRs

Suggested reviewers

  • georgehao
  • 0xmountaintop
  • Thegaram
  • jonastheis

🐇 In the meadow, a change so bright,
Version updated, a joyful sight!
Transactions handled with care and grace,
Errors caught in a swift embrace.
A hop, a skip, through code we flow,
With every change, our progress will grow! 🌼

Warning

There were issues while running some tools. Please review the errors and either fix the tool’s configuration or disable the tool if it’s a critical failure.

🔧 golangci-lint (1.62.2)

level=warning msg="[runner] Can't run linter goanalysis_metalinter: buildir: failed to load package : could not load export data: no export data for "scroll-tech/rollup/mock_bridge""
level=error msg="Running error: can't run linter goanalysis_metalinter\nbuildir: failed to load package : could not load export data: no export data for "scroll-tech/rollup/mock_bridge""


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Outside diff range and nitpick comments (1)
rollup/internal/controller/sender/sender.go (1)

625-641: Enhance error handling when rollback fails

When both SendTransaction and the rollback transaction fail, the code logs an error but does not handle the failure further. Consider implementing additional error handling to maintain system consistency, such as retrying the rollback or alerting system operators.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 438a9fb and 8371843.

📒 Files selected for processing (3)
  • common/types/db.go (2 hunks)
  • common/version/version.go (1 hunks)
  • rollup/internal/controller/sender/sender.go (3 hunks)
✅ Files skipped from review due to trivial changes (1)
  • common/version/version.go
🔇 Additional comments (4)
common/types/db.go (3)

313-314: LGTM: New transaction status enhances error handling granularity

The addition of TxStatusSentFailed helps distinguish between transactions that fail during sending versus those that fail after confirmation, which is crucial for implementing proper rollback mechanisms.


327-328: LGTM: String representation properly implemented

The String() method is correctly updated to handle the new status, maintaining consistency with the existing pattern.


Line range hint 313-328: Verify handling of the new status across the codebase

Since TxStatus is part of the public API, we should verify:

  1. All switch statements handling TxStatus are updated
  2. Database migrations if TxStatus is stored as an integer
  3. Error handling logic in the rollup relayer properly sets this status
✅ Verification successful

All switch statements and database handling properly account for the new TxStatusSentFailed status

The verification results show:

  1. The new TxStatusSentFailed is properly used in the sender implementation (rollup/internal/controller/sender/sender.go) when transaction sending fails
  2. Database schema migrations exist and handle transaction status (bridge-history-api/internal/orm/migrate/migrations/)
  3. All switch statements handling TxStatus in the codebase are for different status types (btypes.TxStatusType) and not related to this enum
  4. The status is properly defined in the enum and its String() method
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify complete handling of TxStatusSentFailed across the codebase

# Find all switch statements handling TxStatus
echo "Checking switch statements handling TxStatus..."
rg -A 10 "switch.*TxStatus" 

# Find database schema references to TxStatus
echo "Checking database schema references..."
fd -e sql . | xargs rg -l "tx_status|transaction_status"

# Find all direct references to TxStatus constants
echo "Checking usage of TxStatus constants..."
rg "TxStatus(Unknown|Pending|Replaced|Confirmed|ConfirmedFailed|SentFailed)"

Length of output: 10391

rollup/internal/controller/sender/sender.go (1)

466-474: Confirm gas fee doubling logic is accurate

The added code correctly ensures that the adjusted gas fees (gasTipCap, gasFeeCap, blobGasFeeCap) are at least double the original values. This validation aligns with network requirements for fee escalation.

rollup/internal/controller/sender/sender.go Outdated Show resolved Hide resolved
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Outside diff range and nitpick comments (1)
common/types/db.go (1)

313-314: Clarify the distinction between TxStatusSentFailed and TxStatusConfirmedFailed

The new status TxStatusSentFailed is well-documented, but its relationship with TxStatusConfirmedFailed could be clearer. Consider enhancing the comment to explain:

  • When exactly this status is set (e.g., network issues, gas estimation failures)
  • How it differs from TxStatusConfirmedFailed (which occurs after the transaction is mined)
  • Whether transactions in this status can be retried
 // TxStatusSentFailed indicates that the transaction has failed to be sent.
+// This status is set when the transaction fails during the sending process (e.g., 
+// network issues, gas estimation failures) and differs from TxStatusConfirmedFailed 
+// which occurs when a sent transaction fails during execution.
 TxStatusSentFailed
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 438a9fb and 8371843.

📒 Files selected for processing (3)
  • common/types/db.go (2 hunks)
  • common/version/version.go (1 hunks)
  • rollup/internal/controller/sender/sender.go (3 hunks)
✅ Files skipped from review due to trivial changes (1)
  • common/version/version.go
🔇 Additional comments (2)
common/types/db.go (1)

327-328: LGTM! String() method properly updated

The String() method has been correctly updated to handle the new status, maintaining consistency with the existing implementation.

rollup/internal/controller/sender/sender.go (1)

466-473: Validation for gas fees doubling is correctly implemented

The added code in lines 466-473 properly ensures that the new gas fees (gasTipCap, gasFeeCap, blobGasFeeCap) are at least double the original values. This validation aligns with the gas fee escalation policies and enhances transaction reliability.

rollup/internal/controller/sender/sender.go Outdated Show resolved Hide resolved
rollup/internal/controller/sender/sender.go Outdated Show resolved Hide resolved
@colinlyguo colinlyguo changed the title fix(rollup-relayer): graceful restart fix(sender): graceful restart Nov 28, 2024
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Outside diff range and nitpick comments (2)
rollup/internal/orm/pending_transaction.go (1)

153-167: Consider documenting soft delete behavior

Since this method uses GORM's soft delete feature (via the DeletedAt field), it's important to document that deleted transactions are not physically removed from the database. This behavior is crucial for the "graceful restart" feature as it allows for potential recovery scenarios.

Consider adding a comment like:

-// DeletePendingTransactionByTxHash deletes a pending transaction record from the database by transaction hash.
+// DeletePendingTransactionByTxHash soft deletes a pending transaction record from the database by transaction hash.
+// The record will be marked as deleted but retained in the database for potential recovery scenarios.
 func (o *PendingTransaction) DeletePendingTransactionByTxHash(ctx context.Context, hash common.Hash, dbTX ...*gorm.DB) error {
rollup/internal/controller/sender/sender.go (1)

626-642: Consider improving error handling after rollback

While the rollback mechanism is well-implemented, the current implementation returns immediately after a successful rollback, which might be too aggressive. Consider continuing the loop to process other pending transactions even if one transaction fails.

Apply this diff to improve the error handling:

 if rollbackErr := s.db.Transaction(func(tx *gorm.DB) error {
     // Restore original transaction status back to pending
     if updateErr := s.pendingTransactionOrm.UpdatePendingTransactionStatusByTxHash(s.ctx, originalTx.Hash(), types.TxStatusPending, tx); updateErr != nil {
         return fmt.Errorf("failed to rollback status of original transaction, err: %w", updateErr)
     }
     // Delete the new transaction that was inserted
     if updateErr := s.pendingTransactionOrm.DeletePendingTransactionByTxHash(s.ctx, newSignedTx.Hash(), tx); updateErr != nil {
         return fmt.Errorf("failed to delete new transaction, err: %w", updateErr)
     }
     return nil
 }); rollbackErr != nil {
     // Both SendTransaction and rollback failed
     log.Error("failed to rollback database after SendTransaction failed", "tx hash", newSignedTx.Hash().String(), "from", s.transactionSigner.GetAddr().String(), "nonce", newSignedTx.Nonce(), "sendTxErr", err, "rollbackErr", rollbackErr)
-    return
+    continue
 }

 log.Error("failed to send replacing tx", "tx hash", newSignedTx.Hash().String(), "from", s.transactionSigner.GetAddr().String(), "nonce", newSignedTx.Nonce(), "err", err)
-return
+continue
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 8371843 and 2eb7266.

📒 Files selected for processing (2)
  • rollup/internal/controller/sender/sender.go (3 hunks)
  • rollup/internal/orm/pending_transaction.go (1 hunks)
🔇 Additional comments (3)
rollup/internal/orm/pending_transaction.go (1)

153-167: Verify transaction status before deletion

Given that this PR involves "precheck for blob transaction fees", it would be prudent to add validation ensuring we only delete transactions in appropriate states.

Let's verify the transaction status handling in the codebase:

Consider adding status validation:

 func (o *PendingTransaction) DeletePendingTransactionByTxHash(ctx context.Context, hash common.Hash, dbTX ...*gorm.DB) error {
     db := o.db
     if len(dbTX) > 0 && dbTX[0] != nil {
         db = dbTX[0]
     }
     db = db.WithContext(ctx)
     db = db.Model(&PendingTransaction{})

+    // Verify transaction is in a deletable state
+    var tx PendingTransaction
+    if err := db.Where("hash = ?", hash.String()).First(&tx).Error; err != nil {
+        return fmt.Errorf("failed to find pending transaction: %w", err)
+    }
+    if tx.Status == types.TxStatusConfirmed {
+        return fmt.Errorf("cannot delete confirmed transaction: %s", hash.String())
+    }

     result := db.Where("hash = ?", hash.String()).Delete(&PendingTransaction{})
     if result.Error != nil {
         return fmt.Errorf("failed to delete pending transaction, err: %w", result.Error)
     }
     return nil
 }
rollup/internal/controller/sender/sender.go (2)

235-240: LGTM: Proper cleanup of failed transactions

The implementation correctly handles the cleanup of failed transactions by removing them from the pending transaction table, with appropriate error handling and logging.


467-474: LGTM: Robust validation of gas fee increments

The implementation correctly enforces the minimum gas fee increment requirement for transaction replacement, ensuring compliance with EIP-1559 and EIP-4844 specifications. The error logging is comprehensive, including all relevant fee values for debugging.

rollup/internal/orm/pending_transaction.go Outdated Show resolved Hide resolved
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (1)
rollup/internal/orm/pending_transaction.go (1)

153-154: Enhance method documentation

Consider adding more details to the documentation about return values and the soft delete behavior:

-// DeletePendingTransactionByTxHash deletes a pending transaction record from the database by transaction hash.
+// DeletePendingTransactionByTxHash soft deletes a pending transaction record from the database by transaction hash.
+// Returns nil if the transaction was successfully deleted, an error if the transaction wasn't found,
+// or a wrapped error if the database operation failed.
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 2eb7266 and 0866f99.

📒 Files selected for processing (1)
  • rollup/internal/orm/pending_transaction.go (1 hunks)
🔇 Additional comments (1)
rollup/internal/orm/pending_transaction.go (1)

162-169: LGTM! Robust error handling implementation

The implementation includes proper validation for both database errors and "not found" cases, making it easy to distinguish between different error scenarios. The soft delete operation (due to DeletedAt field) ensures data can be recovered if needed.

@codecov-commenter
Copy link

codecov-commenter commented Nov 28, 2024

Codecov Report

Attention: Patch coverage is 35.29412% with 33 lines in your changes missing coverage. Please review.

Project coverage is 52.06%. Comparing base (438a9fb) to head (b527cd5).

Files with missing lines Patch % Lines
rollup/internal/controller/sender/sender.go 10.34% 23 Missing and 3 partials ⚠️
rollup/internal/orm/pending_transaction.go 68.18% 5 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1567      +/-   ##
===========================================
- Coverage    52.11%   52.06%   -0.06%     
===========================================
  Files          157      157              
  Lines        12433    12479      +46     
===========================================
+ Hits          6480     6497      +17     
- Misses        5395     5421      +26     
- Partials       558      561       +3     
Flag Coverage Δ
rollup 57.12% <35.29%> (-0.24%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Thegaram
Thegaram previously approved these changes Nov 28, 2024
rollup/internal/controller/sender/sender.go Outdated Show resolved Hide resolved
rollup/internal/controller/sender/sender.go Show resolved Hide resolved
rollup/internal/controller/sender/sender.go Show resolved Hide resolved
jonastheis
jonastheis previously approved these changes Dec 2, 2024
@colinlyguo colinlyguo dismissed stale reviews from jonastheis and Thegaram via 538e7d1 December 2, 2024 06:29
georgehao
georgehao previously approved these changes Dec 2, 2024
jonastheis
jonastheis previously approved these changes Dec 2, 2024
@colinlyguo colinlyguo dismissed stale reviews from jonastheis and georgehao via 528c89d December 2, 2024 10:32
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (1)
rollup/internal/orm/orm_test.go (1)

598-609: Consider enhancing test assertions for deletion verification.

While the test cases cover the main scenarios, consider adding more specific assertions:

 // Test DeleteTransactionByTxHash
 err = pendingTransactionOrm.DeleteTransactionByTxHash(context.Background(), tx0.Hash())
 assert.NoError(t, err)

 // Verify the transaction is deleted
 status, err = pendingTransactionOrm.GetTxStatusByTxHash(context.Background(), tx0.Hash())
-assert.NoError(t, err)
-assert.Equal(t, types.TxStatusUnknown, status) // Should return unknown status for deleted transaction
+assert.ErrorContains(t, err, "record not found")

 // Try to delete non-existent transaction
 err = pendingTransactionOrm.DeleteTransactionByTxHash(context.Background(), common.HexToHash("0x123"))
-assert.Error(t, err) // Should return error for non-existent transaction
+assert.ErrorContains(t, err, "no transaction found with hash")
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between feb3294 and b527cd5.

📒 Files selected for processing (2)
  • rollup/internal/orm/orm_test.go (3 hunks)
  • rollup/internal/orm/pending_transaction.go (3 hunks)
🔇 Additional comments (3)
rollup/internal/orm/pending_transaction.go (2)

154-177: LGTM! Well-implemented transaction deletion.

The implementation is robust with proper error handling, logging, and validation. The use of permanent deletion is appropriate for preventing database bloat from repeated transaction failures.


Line range hint 179-191: LGTM! Method rename improves clarity.

The rename from UpdatePendingTransactionStatusByTxHash to UpdateTransactionStatusByTxHash better reflects the method's functionality, as it handles all transaction states, not just pending ones.

rollup/internal/orm/orm_test.go (1)

Line range hint 563-580: LGTM! Comprehensive test coverage for transaction status updates.

The test cases thoroughly verify the transaction status update functionality and proper retrieval of transaction metadata.

@georgehao georgehao merged commit fa2401c into develop Dec 2, 2024
6 checks passed
@georgehao georgehao deleted the fix-rollup-relayer-resubmitting-blob-txn branch December 2, 2024 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bump-version Bump the version tag for deployment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants