-
Notifications
You must be signed in to change notification settings - Fork 3.9k
Add builders for classes based on UnifiedJedis #4263
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
* Add toxyproxy * Add test endpoints * Add example tests
…se (#4176) * enforce formating for mcf * enforce formating for mcf * Trigger integration test jobs feature branches
…t breaker before retries (#4178) * fix: Change evaluation order of circuit breaker and retry mechanisms This commit modifies the order in which circuit breaker and retry mechanisms are evaluated during failover scenarios. Previously, the circuit breaker was applied after all retries were exhausted, which could delay failover in unstable network conditions. Now, the circuit breaker counts each connection error separately, even during retry attempts. * format * format * use java-8 compatible formater version * fix AutomaticFailoverTest * formating
…4175) * Implement failover retry for in-flight commands This change adds support for retrying in-flight commands when a Redis connection fails and the client automatically fails over to another cluster. The feature is configurable through the FailoverOptions builder. Key changes: - Added retryFailedInflightCommands option to FailoverOptions - Implemented retry logic in CircuitBreakerCommandExecutor - Added integration tests to verify both retry and no-retry behavior - Created utility methods for test setup and configuration This enhancement improves resilience for long-running commands like blocking operations, allowing them to transparently continue on the failover cluster without client-side errors. * Update src/main/java/redis/clients/jedis/mcf/CircuitBreakerCommandExecutor.java Co-authored-by: Copilot <[email protected]> # Conflicts: # src/main/java/redis/clients/jedis/mcf/CircuitBreakerCommandExecutor.java * format format & clean-up * fix FailoverIntegrationTest.testInflightCommandsAreNotRetriedAfterFailover blpop timeout should be less than async command timeout to prevent completing with java.util.concurrent.TimeoutException instead of actuall command failure * Address comments from review - rename retryFailedInflightCommands->retryOnFailover - remove check for CB in OPEN state * remove unused method --------- Co-authored-by: Copilot <[email protected]>
…es (#4183) * fix formatting for anticipated failover changes * additional formatting to cover given folders
#4189) * - weighted cluster seleciton - Healtstatus manager with initial listener and registration logic - pluggable health checker strategy introduced, these are draft NoOpStrategy, EchoStrategy, LagAwareStrategy, - fix failing tests impacted from weighted clusters * - add builder for ClusterConfig - add echo ot CommandObjects and UnifiedJEdis - improve StrategySupplier by accepting jedisclientconfig - adapt EchoStrategy to StrategySupplier. Now it handles the creation of connection by accepting endpoint and JedisClientConfig - make healthchecks disabled by default - drop noOpStrategy - add unit&integration tests for health check * - fix naming * clean up and mark override methods * fix link in javadoc * fix formatting * - fix double registered listeners in healtstatusmgr - clear redundant catch - replace failover options and drop failoveroptions class - remove forced_unhealthy from healthstatus - fix failback check - add disabled flag to cluster - update/fix related tests * Update src/main/java/redis/clients/jedis/mcf/EchoStrategy.java Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]>
* - weighted cluster seleciton - Healtstatus manager with initial listener and registration logic - pluggable health checker strategy introduced, these are draft NoOpStrategy, EchoStrategy, LagAwareStrategy, - fix failing tests impacted from weighted clusters * - add builder for ClusterConfig - add echo ot CommandObjects and UnifiedJEdis - improve StrategySupplier by accepting jedisclientconfig - adapt EchoStrategy to StrategySupplier. Now it handles the creation of connection by accepting endpoint and JedisClientConfig - make healthchecks disabled by default - drop noOpStrategy - add unit&integration tests for health check * - fix naming * clean up and mark override methods * fix link in javadoc * fix formatting * - fix double registered listeners in healtstatusmgr - clear redundant catch - replace failover options and drop failoveroptions class - remove forced_unhealthy from healthstatus - fix failback check - add disabled flag to cluster - update/fix related tests * Update src/main/java/redis/clients/jedis/mcf/EchoStrategy.java Co-authored-by: Copilot <[email protected]> * - add remove endpoints * - replace cluster disabled with failbackCandidate - replace failback enabled with failbacksupported in client - fix formatting - set defaults * - remove failback candidate - fix failing tests * - fix remove logic - fix failing tests --------- Co-authored-by: Copilot <[email protected]>
* - weighted cluster seleciton - Healtstatus manager with initial listener and registration logic - pluggable health checker strategy introduced, these are draft NoOpStrategy, EchoStrategy, LagAwareStrategy, - fix failing tests impacted from weighted clusters * - add builder for ClusterConfig - add echo ot CommandObjects and UnifiedJEdis - improve StrategySupplier by accepting jedisclientconfig - adapt EchoStrategy to StrategySupplier. Now it handles the creation of connection by accepting endpoint and JedisClientConfig - make healthchecks disabled by default - drop noOpStrategy - add unit&integration tests for health check * - fix naming * clean up and mark override methods * fix link in javadoc * fix formatting * - fix double registered listeners in healtstatusmgr - clear redundant catch - replace failover options and drop failoveroptions class - remove forced_unhealthy from healthstatus - fix failback check - add disabled flag to cluster - update/fix related tests * Update src/main/java/redis/clients/jedis/mcf/EchoStrategy.java Co-authored-by: Copilot <[email protected]> * - add remove endpoints * - replace cluster disabled with failbackCandidate - replace failback enabled with failbacksupported in client - fix formatting - set defaults * - remove failback candidate - fix failing tests * - fix remove logic - fix failing tests * - periodic failback checks - introduce graceperiod - fix issue when CB is forced_open and gracePeriod is completed * - format * - fix timing issue --------- Co-authored-by: Copilot <[email protected]>
…ient init (#4207) * - weighted cluster seleciton - Healtstatus manager with initial listener and registration logic - pluggable health checker strategy introduced, these are draft NoOpStrategy, EchoStrategy, LagAwareStrategy, - fix failing tests impacted from weighted clusters * - add builder for ClusterConfig - add echo ot CommandObjects and UnifiedJEdis - improve StrategySupplier by accepting jedisclientconfig - adapt EchoStrategy to StrategySupplier. Now it handles the creation of connection by accepting endpoint and JedisClientConfig - make healthchecks disabled by default - drop noOpStrategy - add unit&integration tests for health check * - fix naming * clean up and mark override methods * fix link in javadoc * fix formatting * - fix double registered listeners in healtstatusmgr - clear redundant catch - replace failover options and drop failoveroptions class - remove forced_unhealthy from healthstatus - fix failback check - add disabled flag to cluster - update/fix related tests * Update src/main/java/redis/clients/jedis/mcf/EchoStrategy.java Co-authored-by: Copilot <[email protected]> * - add remove endpoints * - replace cluster disabled with failbackCandidate - replace failback enabled with failbacksupported in client - fix formatting - set defaults * - remove failback candidate - fix failing tests * - fix remove logic - fix failing tests * - periodic failback checks - introduce graceperiod - fix issue when CB is forced_open and gracePeriod is completed * - introduce StatusTracker with purpose of waiting initial healthcheck results during consturction of provider - add HealthStatus.UNKNOWN as default for Cluster - handle status changes in order of events during initialization - add tests for status tracker and orderingof events - fix impacted unit&integ tests * - introduce forceActiveCluster by duration - fix formatting * - fix failing tests by waiting on clusters to get healthy * - fix failing scenario test - downgrade logback version for slf4j compatibility - increase timeouts for faultInjector * - adressing reviews and feedback * - fix formatting * - fix formatting * - get rid of the queue and event ordering for healthstatus change in MultiClusterPooledConnectionProvider - add test for init and post init events - fix failing tests * - replace use of reflection with helper methods - fix failing tests due to method name change * - replace 'sleep' with 'await', feedback from @a-TODO-rov --------- Co-authored-by: Copilot <[email protected]>
…e approach with builders (#4226) * - weighted cluster seleciton - Healtstatus manager with initial listener and registration logic - pluggable health checker strategy introduced, these are draft NoOpStrategy, EchoStrategy, LagAwareStrategy, - fix failing tests impacted from weighted clusters * - add builder for ClusterConfig - add echo ot CommandObjects and UnifiedJEdis - improve StrategySupplier by accepting jedisclientconfig - adapt EchoStrategy to StrategySupplier. Now it handles the creation of connection by accepting endpoint and JedisClientConfig - make healthchecks disabled by default - drop noOpStrategy - add unit&integration tests for health check * - fix naming * clean up and mark override methods * fix link in javadoc * fix formatting * - fix double registered listeners in healtstatusmgr - clear redundant catch - replace failover options and drop failoveroptions class - remove forced_unhealthy from healthstatus - fix failback check - add disabled flag to cluster - update/fix related tests * Update src/main/java/redis/clients/jedis/mcf/EchoStrategy.java Co-authored-by: Copilot <[email protected]> * - add remove endpoints * - replace cluster disabled with failbackCandidate - replace failback enabled with failbacksupported in client - fix formatting - set defaults * - remove failback candidate - fix failing tests * - fix remove logic - fix failing tests * - periodic failback checks - introduce graceperiod - fix issue when CB is forced_open and gracePeriod is completed * - introduce StatusTracker with purpose of waiting initial healthcheck results during consturction of provider - add HealthStatus.UNKNOWN as default for Cluster - handle status changes in order of events during initialization - add tests for status tracker and orderingof events - fix impacted unit&integ tests * - introduce forceActiveCluster by duration - fix formatting * - fix failing tests by waiting on clusters to get healthy * - fix failing scenario test - downgrade logback version for slf4j compatibility - increase timeouts for faultInjector * - adressing reviews and feedback * - fix formatting * - fix formatting * - get rid of the queue and event ordering for healthstatus change in MultiClusterPooledConnectionProvider - add test for init and post init events - fix failing tests * - replace use of reflection with helper methods - fix failing tests due to method name change * - introduce clusterSwitchEvent and drop clusterFailover post processor - fix broken echostrategy due to connection issue - make healtthCheckStrategy closable and close on - adding fastfailover mode to config and provider - add local failover tests for total failover duration * - introduce fastfailover using objectMaker injection into connectionFactory * - polish * - cleanup * - improve healtcheck thread visibility * - introduce TrackingConnectionPool with FailFastConnectionFactory - added builders to connection and connectionFactory - introduce initializtionTracker to track list of connections during their construction. * - return broken source as usual - do not throw exception is failover already happening * - unblock waiting threads * - failover by closing the pool * - formatting * - check waiters and active/idle connections to force disconnect * - add builder to trackingconnectionpool * - fix failing tests due to mocked ctor for trackingConnectionPool * - replace initTracker with split initializtion of conn * - refactor on builders and ctors * - undo format * - clena up * - add exceptions to logs * - add max wait duration and minConsecutiveSuccessCount to healthCheckStrategy * - replace 'sleep' with 'await', feedback from @a-TODO-rov * - fix status tracker tests --------- Co-authored-by: Copilot <[email protected]>
…on error (#4230) * Add tests: EchoStrategy does recover on connection error * log error's on exception
…4236) * - replace SimpleEntry with HealthCheckResult * - format
…4248) * Add option for rate limiter to MultiThreadedFakeApp * retrigger checks
* - lag-aware api * - introduce healtCheckMetaConfig - add config objects for both EchoStrategy and LagAwareStrategy - make StrategySupplier interface generic - fix impacted tests * - fix issues with docs and log * Fix EchoStrategyIntegrationTest after merge * Implement resolve bdbId based on configured HostAndPort * format & merge RedisRestAPI unit tests * introduce builder for LagAwareStrategy * configurable extended-check - provide option to enable/disable lag check - default to lag-aware check with AVAILABILITY_LAG_TOLERANCE_DEFAULT of 100ms * - fix java doc issue * - formatting * -fix doc issue * - formatting * revert introduce healtCheckMetaConfig - introduces type unsafe warnings, revert till they are resovled * Add list constructor * Add option for rate limiter to MultiThreadedFakeApp * Use /v1/local/bdbs/<int:uid>/endpoint/availability - use /v1/local/bdbs/<int:uid>/endpoint/availability instead of availability address vs /v1/bdbs/<int:uid>/availability * Add builder factory method for base HealthCheckStrategy.Config * format * format * revert to /v1/bdbs/<int:uid>/availability reading again the spec /local is inteded to be used when configuring LoadBalancers * revert to /v1/bdbs/<int:uid>/availability reading again the spec /local is inteded to be used when configuring LoadBalancers * format * fix unit test after revert of local * format * Address review comments - remove misleading default value comment - renamed Config.endpoint to Config.restEndpoint - renamed dataPathAvailability -> databaseAvailability --------- Co-authored-by: ggivo <[email protected]>
… `HealthCheck` itself (#4244) * - set sleep duration 1 ms for forceDisconnect * - refactor cluster to replace HealthStatus with HealthCheck itself * - move Endpoint to parent package * - format * - fix typo * - adjust tests with new provider behaviour with HealthCheck * Address review comments - rename getTimeout() -> getMaxWaitFor() - remove unnecessary noHealthChecks * resolve merge conflicts * format * format --------- Co-authored-by: Ivo Gaydazhiev <[email protected]>
* Improve AA failover test * Fix endpoint name * Count failedCommandsAfterFailover * do not enforce formating on ActiveActiveFailoverTest --------- Co-authored-by: Ivo Gaydazhiev <[email protected]>
…al state changes #4255 (#4256) * fix: only notify health status listeners on actual state changes #4255 - Replace overlapping health checks with sequential execution - use wasUpdated flag to prevent false notifications from timestamp conflicts. Eliminates race conditions causing spurious failovers. Closes: #4255 * format * Update src/main/java/redis/clients/jedis/mcf/HealthCheckImpl.java Co-authored-by: atakavci <[email protected]> * revert to fixed rate health checks * fix ActiveActiveLocalFailoverTest - reduced endpoint pause to 10s to speed up test - ensure test endpoint is eavailable before startign the test (awaits up to endpoint pause time) --------- Co-authored-by: atakavci <[email protected]>
* Initial draft * Cover health checks * Add javadoc's for MultiClusterClientConfig * Fix docs for health checks * Fix javadoc's * Update wordlist.txt --------- Co-authored-by: Ivo Gaydazhiev <[email protected]>
- ensure hbase-formatter.xml match current one in master
…ailover # Conflicts: # .github/wordlist.txt # pom.xml # src/test/java/redis/clients/jedis/misc/AutomaticFailoverTest.java
- setClusterFailoverPostProcessor renamed - format AutomaticFailoverTest
Skip the reflection tests if the "with-param-names" flag is not enabled during the test run
Address review comments - replace magic numbers in tests with constats
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approach with the builders looks good.
I have put some comments, and have concerns around tests validating the constructor coverage.
*/ | ||
public T cache(Cache cache) { | ||
this.cache = cache; | ||
this.redisProtocol = RedisProtocol.RESP3; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure about silently overriding the protocol.
I will suggest validating it during build if cache is configured and configured protocol does not support it..
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
*/ | ||
public T cacheConfig(CacheConfig cacheConfig) { | ||
this.cacheConfig = cacheConfig; | ||
this.redisProtocol = RedisProtocol.RESP3; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as above, don't think we should silently set the protocol;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
* @throws IllegalArgumentException if any common configuration is invalid | ||
*/ | ||
protected void validateCommonConfiguration() { | ||
if (poolConfig == null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about using custom connection provider that does not require poolConfig?
I think pool config should be optional
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
* @param cache the client-side cache (may be null) | ||
* @return the configured Redis client instance | ||
*/ | ||
protected abstract C createClient(CommandExecutor commandExecutor, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding new components (metrics, tracing, etc.) would require breaking changes. No need to pass explicitly commandExecutor
,... They are already available to extending classes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch, thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
} | ||
|
||
@Test | ||
void cacheEnablesRESP3() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
related to previous comment. Don't think we should silently set the protocol when enabling client side cache
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
} | ||
|
||
@Test | ||
void standaloneValidateHostPortRequired() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Relates to previous comment. We should not enforce poolConfig as required. For example when using custom ConnectionProvide we might not use appache commons pool
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This test is just off; it should test the host and port instead. I'm going to fix it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
|
||
@Test | ||
void sentinelValidateMasterAndSentinels() { | ||
assertThrows(IllegalArgumentException.class, () -> new JedisSentineled.Builder() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the reason for Illegal argument exception? Should we have sensible defaults and build() pass with them?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Primary name and sentinels config are required and cannot have a default value. See https://github.com/redis/jedis/pull/4263/files/183b00d9046022a45ebe162c1342663c4f53d1b2#diff-af5d7cdfaeacc02ac468d5007e9860e001b9f90f80b80288ce3dccd22ae390b7R93-R104 for more details
verify(exec).executeCommand(cap.capture()); | ||
List<String> args = argsToStrings(cap.getValue()); | ||
// SET prefix:k v | ||
org.hamcrest.MatcherAssert.assertThat(args.get(0), containsString("SET")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
static imports
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
/** | ||
* Reflection-based coverage test for JedisCluster constructors against builder API. The test | ||
* verifies each constructor parameter can be provided via: - ClusterClientBuilder methods, or - | ||
* DefaultJedisClientConfig.Builder methods, or - Custom ConnectionProvider (manual coverage) | ||
*/ | ||
@EnabledIfSystemProperty(named = "with-param-names", matches = "true") | ||
public class JedisClusterConstructorReflectionTest { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have some concerns about the current reflection-based testing approach in JedisPooledConstructorReflectionTest
.
The tests rely on validating hardcoded internal state, which can become outdated quickly and may fail to catch real functional differences.
Some issues with this approach:
- It can lead to false positives.
- It only validates that parameters can be provided, not that they work correctly.
Instead of reflection-based coverage validation, have you considered testing functional equivalence between constructor-created and builder-created clients?
This way, the tests also serve a documentation purpose while validating actual functionality parity.
For example:
@Test
void constructorBuilderEquivalence() {
// Ensure constructor and builder produce functionally equivalent clients
// Constructor approach
JedisPooled constructorClient = new JedisPooled("localhost", 6379, "user", "pass");
// Builder approach
JedisPooled builderClient = JedisPooled.builder()
.hostAndPort("localhost", 6379)
.clientConfig(DefaultJedisClientConfig.builder()
.user("user")
.password("pass")
.build())
.build();
// Validate functional equivalence
assertClientsEquivalent(constructorClient, builderClient);
}
Where assertClientsEquivalent(...)
could validate the equivalence of initialized client components such as:
ConnectionProvider provider
Pool<Connection> pool
CommandExecutor executor
CommandObjects commandObjects
Cache cache
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The approach I used serves as a safeguard to prevent the addition of new constructors and provides an efficient way to check the builder's coverage. Once we reach Jedis v9, we should drop it altogether with all deprecated constructors.
Another problem with functional tests is that they only work for simple cases like the one you have mentioned. Also, creating reasonable assertClientsEquivalent
is quite tricky. I gave up on that approach after seeing that it became a reflection monster to expose all the internal states of UnifiedJedis and co. At best, with a functional approach, we end up testing that both client instances can connect to the same redis-server instance, which is not sufficient and can be achieved by just instantiating UnifiedJedis and co with the new builder in plenty of integration tests that we already have.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see your point.
Yes, I was thinking that we need a mix of functional tests and unit tests. We can still use the reflection, but based on comparing the actual objects built. e.g, ConnectionProvider built via the constructor is equivalent to ConnectionProvider built with the corresponding builder.
IMO, the current approach is also hard to verify via review.
It might take some time till v9 is out. Existing tests will give some confidence, but I still think it makes sense to open a follow-up issue and add additional tests focused on object build using a concrete constructor vs the corresponding one using builder.
- resolve cannot find symbol Error: symbol: class MultiClusterPooledConnectionProvider
Motivation
This change creates a foundation for providing a new client class(es) for AA failover, allowing developers to easily instantiate JedisPooled, JedisSentineled, JedisCluster, etc.
Review
Summary
This commit introduces a formal Builder pattern for UnifiedJedis-based clients (JedisPooled, JedisCluster, JedisSentineled) and adds a shared builder base (AbstractClientBuilder) plus specialized builders for standalone, cluster, and sentinel deployments. It also exposes a couple of configuration hooks, aligns RediSearch default dialect handling, and adds reflection-based tests to ensure builder coverage over the existing constructor surfaces. Build tooling is adjusted to preserve parameter names for those tests.
High-level Impact
Detailed Changes
1) Builders
Key capabilities (AbstractClientBuilder):
Specialized builders:
2) Jedis* classes integrate builders
3) UnifiedJedis visibility
4) CommandObjects tuning
5) RediSearch protocol constant usage
6) Build tooling to support reflection tests
7) Tests
API/Compatibility Assessment