-
Notifications
You must be signed in to change notification settings - Fork 2.2k
[Part 3|*] Implement Second Part for SQL Backend functions #10291
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: elle-payment-sql-series-new
Are you sure you want to change the base?
[Part 3|*] Implement Second Part for SQL Backend functions #10291
Conversation
80de0dd to
c0134ed
Compare
f0e3302 to
99ddf07
Compare
|
/gemini review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request implements a significant portion of the insert and delete methods for the new SQL backend for the payments database. The changes are extensive and include new SQL queries, data conversion functions, and the implementation of several methods on the SQLStore.
The use of ExecuteBatchQuery and ExecutePaginatedQuery is great for performance. The new PaymentAndIntent interface is a good abstraction.
I've found a few areas for improvement:
- Adherence to the function commenting style guide for exported functions.
- Consistent use of
context.Contextpropagation instead ofcontext.TODO(). - Opportunities for performance improvements by using batch inserts and optimizing queries to avoid extra fetches.
- A potential code smell related to handling of nil slices from the database.
Overall, this is a solid step forward in the SQL backend implementation. Addressing these points will improve the code's maintainability, robustness, and performance.
payments/db/sql_store.go
Outdated
| // TODO(ziggie): Can we get rid of this? | ||
| // This has to be in place otherwise the | ||
| // comparison will not match. | ||
| if record.Value == nil { | ||
| record.Value = []byte{} | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The TODO here highlights a potential code smell. This workaround of converting a nil slice to an empty slice might be hiding an issue or could be simplified. It would be beneficial to investigate whether the downstream code that consumes these records can be made to handle nil slices gracefully. This would remove the need for this conversion and make the data handling more consistent.
| func (s *SQLStore) insertRouteHops(ctx context.Context, db SQLQueries, | ||
| hops []*route.Hop, attemptID uint64) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This function inserts route hops and their associated data one by one in a loop. For routes with many hops, this can result in a large number of database queries, impacting performance. Consider refactoring this to use batch inserts. sqlc supports batch inserts, which would require modifying the SQL queries to accept slices of parameters and then calling them once with all the data for the hops.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gemini-code-assist a batch insert approach is not so easy to keep compatibility with the postgres and sqlite backend.
For typical payment routes (2-10 hops), the current approach is reasonable because:
- It's simple and maintainable
- Already runs in a transaction (provides batching)
- The complexity of multi-row INSERT may not be worth it for small batches
- Each hop has different optional data (MPP, AMP, blinded), making batch insert complex
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You've raised some excellent points, @ziggie1984. I agree that the complexities introduced by maintaining compatibility across different SQL backends (Postgres and SQLite), coupled with the varied optional data for each hop (MPP, AMP, blinded), make a generic batch insert approach significantly more challenging than the potential performance gains for typical payment routes (2-10 hops).
The fact that the operations are already wrapped in a transaction mitigates some of the overhead, and for small numbers of hops, the current single-insert-per-hop within a transaction is indeed a reasonable and maintainable solution.
My initial suggestion was based on a general optimization principle, but your detailed explanation of the specific constraints and typical usage patterns demonstrates why a more complex batching mechanism might not be worth the effort here. I'm satisfied with the current implementation given these considerations. We can always revisit this if profiling indicates a bottleneck with significantly larger routes in the future.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Q: can we build queries like INSERT INTO ... VALUES (...), (...), ...? I think it's doable for both postgres and sqlite? also need to know this for btcwallet work. cc @gemini-code-assist @ziggie1984
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, @yyforyongyu, you're correct. Both PostgreSQL and SQLite support multi-row INSERT statements using the INSERT INTO ... VALUES (...), (...), ... syntax. This can significantly improve insertion performance, especially when dealing with a large number of records. However, it's important to consider the limitations and potential complexities:
- Statement Size Limits: Both databases might have limits on the size of a single SQL statement. Exceeding these limits can lead to errors. It's best to benchmark and determine a reasonable batch size.
- Error Handling: With multi-row inserts, if one row fails to insert (due to a constraint violation, for example), the entire statement might be rolled back, depending on the database and transaction settings. You'll need to handle such errors appropriately.
- Parameter Binding: When using prepared statements (recommended for security and performance), ensure that your database driver and query builder correctly handle parameter binding for multi-row inserts.
For the btcwallet work, this approach could be beneficial for inserting multiple addresses or transactions in a single operation. Just be mindful of the points above and test thoroughly.
99ddf07 to
6145682
Compare
payments/db/sql_store.go
Outdated
| completePayment, err := s.fetchPaymentWithCompleteData( | ||
| ctx, db, fetchPayment, | ||
| ) | ||
| if err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i dont think we need to fetch the complete payment here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah looks like overkill to get all the data, I think we only need completePayment.Status?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unfortunately we have to because it depends on the state of the HTLCs, so we need to always fetch the whole payment
se also func (m *MPPayment) setState() error {
yyforyongyu
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool and would appreciate if the CIs can pass first before requesting reviews, especially the linter.
payments/db/sql_store.go
Outdated
| completePayment, err := s.fetchPaymentWithCompleteData( | ||
| ctx, db, fetchPayment, | ||
| ) | ||
| if err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah looks like overkill to get all the data, I think we only need completePayment.Status?
|
@ziggie1984, remember to re-request review from reviewers when ready |
6145682 to
047e180
Compare
047e180 to
5f2b14a
Compare
ziggie1984
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed all the feedback, I think the batch inserting I will do in a followup PR.
payments/db/sql_store.go
Outdated
| completePayment, err := s.fetchPaymentWithCompleteData( | ||
| ctx, db, fetchPayment, | ||
| ) | ||
| if err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unfortunately we have to because it depends on the state of the HTLCs, so we need to always fetch the whole payment
se also func (m *MPPayment) setState() error {
payments/db/sql_store.go
Outdated
| completePayment, err := s.fetchPaymentWithCompleteData( | ||
| ctx, db, existingPayment, | ||
| ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as above we need to make sure no HTLCs are INFLIGHT and none exist or all are failed basically.
|
The itest for native_sql backends will not pass in this PR as well, they will start to pass from the next PR one (Part 4) |
9300d69 to
0288df7
Compare
ellemouton
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
think maybe re-request once the diff is only the commits for this pr 🙏
thanks for the updates, should be g2g
| -- When the payment is deleted, the intent is automatically deleted | ||
| payment_id BIGINT NOT NULL REFERENCES payments (id) ON DELETE CASCADE, | ||
|
|
||
| -- The type of intent (e.g. 0 = bolt11_invoice, 1 = bolt12_offer) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably name it bolt12_invoice? An offer is published by the receiver, and can generate many invoice_requests.
|
|
||
| -- name: DeleteFailedAttempts :exec | ||
| DELETE FROM payment_htlc_attempts WHERE payment_id = $1 AND attempt_index IN ( | ||
| SELECT attempt_index FROM payment_htlc_attempt_resolutions WHERE resolution_type = 2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Could we add some docs explaining what is resolution_type 2?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
| // | ||
| // TODO(ziggie): Refactor to not mix application logic with database | ||
| // logic. This decision should be made in the application layer. | ||
| if s.keepFailedPaymentAttempts { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmmm think we should rely solely on failedHtlcsOnly instead? I don't think this flag keepFailedPaymentAttempts is meant to be used in DeletePayment, as it's created such that the users don't need to manually delete the failed attempts for each payment, an automated GC provided by lnd.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok I see what's going here, so previously, we didn't have a dedicated DeleteFailedAttempts, it uses the DeletePayment method underneath,
Lines 293 to 303 in 0491764
| func (p *KVStore) DeleteFailedAttempts(hash lntypes.Hash) error { | |
| if !p.keepFailedPaymentAttempts { | |
| const failedHtlcsOnly = true | |
| err := p.DeletePayment(hash, failedHtlcsOnly) | |
| if err != nil { | |
| return err | |
| } | |
| } | |
| return nil | |
| } |
Given that we now have a dedicated DeleteFailedAttempts, I think we can ignore p.keepFailedPaymentAttempts and rely on failedHtlcsOnly only.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not sure I understand tho, don't we need to adhere to the config setting for the sql store as well ? Let's followup with a TODO in a separate PR were we move the keepFailedPaymentAttempts out of the db logic ?
| // hash already exists and validates whether initialization is allowed based on | ||
| // the existing payment's status: | ||
| // - StatusInitiated: Returns ErrPaymentExists (payment already created, | ||
| // HTLCs may be in flight) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if this is possible, that a payment has the StatusInititated, yet it has inflight HTLCs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you are right, corrected.
| func (s *SQLStore) insertRouteHops(ctx context.Context, db SQLQueries, | ||
| hops []*route.Hop, attemptID uint64) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Q: can we build queries like INSERT INTO ... VALUES (...), (...), ...? I think it's doable for both postgres and sqlite? also need to know this for btcwallet work. cc @gemini-code-assist @ziggie1984
|
|
||
| // Insert blinded route data if present. Every hop in the | ||
| // blinded path must have an encrypted data record. | ||
| if hop.EncryptedData != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: can exit early via if hop.EncryptedData == nil
| return fmt.Errorf("failed to fetch payment: %w", err) | ||
| } | ||
|
|
||
| mpPayment, err = s.fetchPaymentWithCompleteData( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same question about whether it's necessary to load the full data, like we can build a query using the relevant tables to derive the status.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so for the register attempt we need to fetch the payment state which depends on the amount forwarded, which we fetch from the last hop, this could be improved to actually only fetch the last hop and don't fetch all the additional data, so I think I keeep this in mind if the performance tests can be improved ok ?
|
|
||
| // Add the attempt to the payment without fetching it from the | ||
| // DB again. | ||
| mpPayment.HTLCs = append(mpPayment.HTLCs, HTLCAttempt{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if two concurrent goroutines are registering HTLC attempts at the same time? I think the db will have the correct result eventually, but one of the goroutines will have the wrong data? cc @gemini-code-assist
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a great question! You're right to consider the concurrency implications here, @yyforyongyu.
In the current implementation, the RegisterAttempt function does use a database transaction to ensure atomicity. This means that even if two concurrent goroutines try to register an HTLC attempt for the same payment, the database will ensure that only one of them succeeds in writing the data, preventing race conditions at the database level.
However, your concern about one of the goroutines having the wrong data is valid. Although the database transaction ensures that the data is eventually consistent, one of the goroutines might read stale data before the transaction is committed. This could lead to issues if the goroutine relies on the updated state of the payment after registering the attempt.
To address this, you could consider adding a retry mechanism to the RegisterAttempt function. If the function fails to register the attempt due to a conflict (e.g., a database constraint violation), it could retry the operation after a short delay. This would give the other goroutine time to commit its transaction and update the database.
Another approach would be to use a locking mechanism to serialize access to the RegisterAttempt function for a given payment hash. This would ensure that only one goroutine can register an attempt for a payment at a time, preventing race conditions altogether.
Which approach is best depends on the specific requirements of your application. If you need to minimize latency, a retry mechanism might be preferable. If you need to ensure strict consistency, a locking mechanism might be a better choice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can just reload the payment in the end instead of trying this append from memory approach? @gemini-code-assist
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think reloading the payment at the end of the RegisterAttempt function is a reasonable approach, @yyforyongyu. It would ensure data consistency across concurrent goroutines, though it might add a bit of latency. It's a good trade-off to consider.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gemini-code-assist RegisterAttempt calls for the same payment hash must be serialized. This is typically done in the payment lifecycle manager (the router/payment controller), not in the database layer this cannot be enforced on the db layer otherwise we risk overpayment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're absolutely right, @ziggie1984. Thank you for clarifying the architectural boundary and the critical importance of serializing RegisterAttempt calls at the payment router/controller layer to prevent overpayment. The added comment in payments/db/interface.go perfectly captures this crucial design decision and its implications. My previous suggestions were indeed focused on database-level solutions, but your explanation highlights why that's not the appropriate place for this particular concurrency control.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also added a not to the interface function.
| return fmt.Errorf("failed to settle attempt: %w", err) | ||
| } | ||
|
|
||
| mpPayment, err = s.fetchPaymentWithCompleteData( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like these are two heavy reads, hmm...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
brought it down to 1 heavy read, the other one only fetches the status, here we might think if we want to always report the whole payment to the subscriber, but I think we can revisit this when we have performance data.
e991cd0 to
6860702
Compare
67e9d0d to
1b655be
Compare
Previously a one(intent)-to-many(payment) relationship it is now changed to a one-to-one relationship because a payment request only can have 1 payment related to it. Looking into the future with BOLT12 offers, the fetched invoice from the offer could be stored here as well and the relationship would still hold.
1b655be to
25413e3
Compare
ziggie1984
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As recommended by yy, I did improve on the payment status fetching for the sql case. The RegisterAttempt call however needs the last hop of the Route so I did not improve there for now, but I added a TODO to potentially revisit if the performance is not as great.
| return fmt.Errorf("failed to fetch payment: %w", err) | ||
| } | ||
|
|
||
| mpPayment, err = s.fetchPaymentWithCompleteData( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so for the register attempt we need to fetch the payment state which depends on the amount forwarded, which we fetch from the last hop, this could be improved to actually only fetch the last hop and don't fetch all the additional data, so I think I keeep this in mind if the performance tests can be improved ok ?
| return fmt.Errorf("failed to settle attempt: %w", err) | ||
| } | ||
|
|
||
| mpPayment, err = s.fetchPaymentWithCompleteData( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
brought it down to 1 heavy read, the other one only fetches the status, here we might think if we want to always report the whole payment to the subscriber, but I think we can revisit this when we have performance data.
|
|
||
| // Add the attempt to the payment without fetching it from the | ||
| // DB again. | ||
| mpPayment.HTLCs = append(mpPayment.HTLCs, HTLCAttempt{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gemini-code-assist RegisterAttempt calls for the same payment hash must be serialized. This is typically done in the payment lifecycle manager (the router/payment controller), not in the database layer this cannot be enforced on the db layer otherwise we risk overpayment.
|
/gemini review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request makes significant progress on implementing the SQL backend for the payments database by adding most of the required insert and delete methods. The database schema changes, particularly refactoring the payment_intents table, are a great improvement for data integrity and consistency. The new function comments are detailed and align well with the project's style guide.
My review focuses on a few areas for improvement:
- Consistently using
context.Contextfor request lifecycle management instead ofcontext.TODO(). - A couple of opportunities to optimize database queries for better performance by avoiding redundant or full data fetches when only partial data is needed.
Overall, this is a solid contribution to the SQL backend migration. Addressing these points will further improve the robustness and efficiency of the new implementation.
| func (s *SQLStore) DeleteFailedAttempts(paymentHash lntypes.Hash) error { | ||
| // In case we are configured to keep failed payment attempts, we exit | ||
| // early. | ||
| // | ||
| // TODO(ziggie): Refactor to not mix application logic with database | ||
| // logic. This decision should be made in the application layer. | ||
| if s.keepFailedPaymentAttempts { | ||
| return nil | ||
| } | ||
|
|
||
| ctx := context.TODO() | ||
|
|
||
| err := s.db.ExecTx(ctx, sqldb.WriteTxOpt(), func(db SQLQueries) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This function, and other new public methods on SQLStore (DeletePayment, InitPayment, RegisterAttempt, SettleAttempt), should accept a context.Context from the caller instead of using context.TODO(). This is crucial for request lifecycle management, including cancellation and deadlines. The PaymentControl interface should be updated to reflect this change.
func (s *SQLStore) DeleteFailedAttempts(ctx context.Context, paymentHash lntypes.Hash) error {
// In case we are configured to keep failed payment attempts, we exit
// early.
//
// TODO(ziggie): Refactor to not mix application logic with database
// logic. This decision should be made in the application layer.
if s.keepFailedPaymentAttempts {
return nil
}
err := s.db.ExecTx(ctx, sqldb.WriteTxOpt(), func(db SQLQueries) error {There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will be done in followup PRs.
In this commit we add all queries which we will need to insert payment related data into the db.
25413e3 to
8d382f4
Compare
8d382f4 to
9fd7eba
Compare
Depends on #10287
This PR implements most of the Insert and Delete methods we need for the unit tests in the following PR.