Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Server-assisted Client-side Caching #3757

Merged
merged 71 commits into from
Sep 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
da9c463
Initial support for client-side caching (#3658)
sazzad16 Dec 28, 2023
0ca0dd1
Merge branch 'master' into 5.2.0
sazzad16 Dec 28, 2023
5fa2c80
Merge branch 'master' into 5.2.0
sazzad16 Jan 1, 2024
89617c9
Support for client-side caching - phase 2 (#3673)
sazzad16 Jan 8, 2024
6d4930f
Merge branch 'master' into 5.2.0
sazzad16 Jan 10, 2024
fca975f
Fix transaction failure tests using mock (#3683)
sazzad16 Jan 11, 2024
d87fc6e
Merge branch 'master' into 5.2.0
sazzad16 Jan 17, 2024
3ab6bdc
Support client-side caching from UnifiedJedis (#3691)
sazzad16 Jan 17, 2024
5f1d8c6
Client-side caching by hashing command arguments (#3700)
sazzad16 Feb 15, 2024
4cada22
Cover Redis commands for client side caching (#3702)
sazzad16 Feb 15, 2024
2480b02
Support Client-side caching through URI/URL (#3703)
sazzad16 Feb 15, 2024
26606b9
Test GuavaCSC and CaffeineCSC (#3742)
sazzad16 Feb 28, 2024
333dcd7
Support white-list and black-list commands and keys (#3755)
sazzad16 Mar 6, 2024
e66f498
Introduce interface(s) for hashing CommandObject (#3743)
sazzad16 Mar 6, 2024
c02e5be
Merge branch 'master' into 5.2.0
sazzad16 Mar 6, 2024
1651b26
Client-side cache related naming changes (#3758)
sazzad16 Mar 10, 2024
b897094
Reformat clientSideCache variable names (#3761)
sazzad16 Mar 10, 2024
dc35d45
Merge branch 'master' into 5.2.0
sazzad16 Mar 10, 2024
a2f5d16
Format tabs in pom.xml
sazzad16 Mar 21, 2024
39fc618
Merge branch 'master' into 5.2.0
sazzad16 Mar 21, 2024
6a488b6
Merge branch 'master' into 5.2.0
sazzad16 Mar 27, 2024
a4737e0
Use Experimental annotation
sazzad16 Mar 27, 2024
b7881ac
Merge branch 'master' into 5.2.0
sazzad16 Apr 3, 2024
767fc01
Fix client side cache tests (#3799)
sazzad16 Apr 4, 2024
3bd45a4
Remove openhft hashing from source dependency (#3800)
sazzad16 Apr 5, 2024
bb99c16
Merge branch 'master' into 5.2.0
sazzad16 Apr 29, 2024
82c0226
Test different functionalities of client side cache (#3828)
sazzad16 Apr 29, 2024
6a1dfc8
Test JedisURIHelper#getClientSideCache(URI) (#3835)
sazzad16 May 6, 2024
27e1553
Merge branch 'master' into 5.2.0
sazzad16 May 7, 2024
e45e4a7
Merge branch 'master' into 5.2.0
sazzad16 Jun 6, 2024
103575d
Merge fix: after introducing EndpointConfig in #3836
sazzad16 Jun 6, 2024
a347d7c
Tweak maximumSize test in CaffeineClientSideCacheTest
sazzad16 Jun 6, 2024
11ce88e
Little more tweak maximumSize test in CaffeineClientSideCacheTest
sazzad16 Jun 6, 2024
6b9d338
Fix incompatibilities with the latest RedisStack (#3855)
uglide Jun 6, 2024
a0b0c59
Revert "Fix incompatibilities with the latest RedisStack (#3855)"
sazzad16 Jun 7, 2024
8310408
Merge branch 'master' into 5.2.0
sazzad16 Jun 7, 2024
dbc6e4f
Merge branch 'master' into 5.2.0
sazzad16 Jun 10, 2024
92c09f3
[TEMPORARY] [TEST] Use redis-stack-server:7.4.0-rc1 image for testing
sazzad16 Jun 13, 2024
33d4771
Support RediSearch DIALECT 5 (#3831)
sazzad16 Jun 13, 2024
7c898b8
Support FLOAT16 and BFLOAT16 VecSim storage types (#3849)
sazzad16 Jun 13, 2024
ef79d54
Test: INTERSECTS and DIJOINT conditions support in GeoSearch (#3862)
sazzad16 Jun 13, 2024
ac18fd0
Support IGNORE and other optional arguments for timeseries commands (…
sazzad16 Jun 14, 2024
5cba18c
Polish #3860: Separate params for TS.INCRBY and TS.DECRBY (#3863)
sazzad16 Jun 14, 2024
f136c6f
Support indexing of MISSING and EMPTY values (#3866)
sazzad16 Jun 15, 2024
3534996
Little tweak maximumSize test in CaffeineClientSideCacheTest
sazzad16 Jun 30, 2024
819447c
Inject ClientSideCacheable via set method (#3882)
sazzad16 Jul 4, 2024
a89b2a9
Use CommandObject(s) as cache-key (#3875)
sazzad16 Jul 4, 2024
fb75443
Merge branch 'master' into 5.2.0
sazzad16 Jul 15, 2024
3d9c09e
#3886 merge fix
sazzad16 Jul 15, 2024
1c12fdd
Revert "[TEMPORARY] [TEST] Use redis-stack-server:7.4.0-rc1 image for…
sazzad16 Jul 15, 2024
fed9aaf
More tweak maximumSize test in CaffeineClientSideCacheTest
sazzad16 Jul 15, 2024
94a2523
Remove client side cache support through uri/url (#3892)
sazzad16 Jul 15, 2024
25ca055
Bump com.google.guava:guava from 33.0.0-jre to 33.2.1-jre (#3893)
sazzad16 Jul 15, 2024
148d4bb
Merge branch 'master' into 5.2.0
sazzad16 Jul 17, 2024
8b83218
Prepare client side caching - design 2 (#3889)
sazzad16 Jul 25, 2024
229399f
Refactor Client-Side Caching implementation (#3900)
atakavci Aug 13, 2024
f78fc08
Merge branch 'master' into 5.2.0
sazzad16 Aug 13, 2024
9bddabd
Jedis test plan coverage for CSC (#3918)
sazzad16 Aug 15, 2024
6707327
Use ExecutorService.shutdownNow() in tests (#3922)
sazzad16 Aug 15, 2024
6ffdcba
Merge branch 'master' into 5.2.0
sazzad16 Aug 21, 2024
eb7e9fe
[minor change] Avoid creating same CacheKey twice
sazzad16 Aug 25, 2024
747e391
Merge branch 'master' into 5.2.0
sazzad16 Aug 25, 2024
e96e2e3
Support caching null values (#3939)
atakavci Aug 28, 2024
88cf9bc
Adding CacheConfig (#3919)
atakavci Aug 28, 2024
1d213bb
Polish "Adding CacheConfig"
sazzad16 Aug 28, 2024
ce210ea
Adding Cache class to CacheConfig (#3942)
atakavci Aug 29, 2024
d369cf6
Merge branch 'master' into 5.2.0
sazzad16 Aug 29, 2024
b94fdf2
Server version check for CSC activation (#3954)
atakavci Sep 18, 2024
da236cf
Merge branch 'master' into 5.2.0
sazzad16 Sep 23, 2024
fcde2ff
Merge branch 'master' into 5.2.0
uglide Sep 27, 2024
ec6ca7b
Merge branch 'master' into 5.2.0
uglide Sep 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@
<version>2.11.0</version>
</dependency>

<!-- Optional dependencies -->

<!-- UNIX socket connection support -->
<dependency>
<groupId>com.kohlschutter.junixsocket</groupId>
Expand All @@ -90,6 +92,7 @@
<version>1.20.0</version>
<scope>test</scope>
</dependency>

<!-- test -->
<dependency>
<groupId>junit</groupId>
Expand Down
27 changes: 27 additions & 0 deletions src/main/java/redis/clients/jedis/CommandArguments.java
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,12 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;

import redis.clients.jedis.annots.Experimental;
import redis.clients.jedis.annots.Internal;
import redis.clients.jedis.args.Rawable;
import redis.clients.jedis.args.RawableFactory;
import redis.clients.jedis.commands.ProtocolCommand;
Expand All @@ -17,6 +20,8 @@ public class CommandArguments implements Iterable<Rawable> {
private CommandKeyArgumentPreProcessor keyPreProc = null;
private final ArrayList<Rawable> args;

private List<Object> keys;

private boolean blocking;

private CommandArguments() {
Expand All @@ -26,6 +31,8 @@ private CommandArguments() {
public CommandArguments(ProtocolCommand command) {
args = new ArrayList<>();
args.add(command);

keys = Collections.emptyList();
}

public ProtocolCommand getCommand() {
Expand Down Expand Up @@ -127,9 +134,24 @@ public CommandArguments key(Object key) {
throw new IllegalArgumentException("\"" + key.toString() + "\" is not a valid argument.");
}

addKeyInKeys(key);

return this;
}

private void addKeyInKeys(Object key) {
if (keys.isEmpty()) {
keys = Collections.singletonList(key);
} else if (keys.size() == 1) {
List oldKeys = keys;
keys = new ArrayList();
keys.addAll(oldKeys);
keys.add(key);
} else {
keys.add(key);
}
}

public final CommandArguments keys(Object... keys) {
Arrays.stream(keys).forEach(this::key);
return this;
Expand Down Expand Up @@ -178,6 +200,11 @@ public Iterator<Rawable> iterator() {
return args.iterator();
}

@Internal
public List<Object> getKeys() {
return keys;
}

public boolean isBlocking() {
return blocking;
}
Expand Down
38 changes: 38 additions & 0 deletions src/main/java/redis/clients/jedis/CommandObject.java
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
package redis.clients.jedis;

import java.util.Iterator;
import redis.clients.jedis.args.Rawable;

public class CommandObject<T> {

private final CommandArguments arguments;
Expand All @@ -17,4 +20,39 @@ public CommandArguments getArguments() {
public Builder<T> getBuilder() {
return builder;
}

@Override
public int hashCode() {
int hashCode = 1;
for (Rawable e : arguments) {
hashCode = 31 * hashCode + e.hashCode();
}
hashCode = 31 * hashCode + builder.hashCode();
return hashCode;
}

@Override
public boolean equals(Object o) {
if (o == this) {
return true;
}
if (!(o instanceof CommandObject)) {
return false;
}

Iterator<Rawable> e1 = arguments.iterator();
Iterator<Rawable> e2 = ((CommandObject) o).arguments.iterator();
while (e1.hasNext() && e2.hasNext()) {
Rawable o1 = e1.next();
Rawable o2 = e2.next();
if (!(o1 == null ? o2 == null : o1.equals(o2))) {
return false;
}
}
if (e1.hasNext() || e2.hasNext()) {
return false;
}

return builder == ((CommandObject) o).builder;
}
}
126 changes: 80 additions & 46 deletions src/main/java/redis/clients/jedis/Connection.java
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.function.Supplier;

import redis.clients.jedis.Protocol.Command;
Expand All @@ -31,7 +32,7 @@
public class Connection implements Closeable {

private ConnectionPool memberOf;
private RedisProtocol protocol;
protected RedisProtocol protocol;
private final JedisSocketFactory socketFactory;
private Socket socket;
private RedisOutputStream outputStream;
Expand All @@ -41,6 +42,8 @@ public class Connection implements Closeable {
private boolean broken = false;
private boolean strValActive;
private String strVal;
protected String server;
protected String version;

public Connection() {
this(Protocol.DEFAULT_HOST, Protocol.DEFAULT_PORT);
Expand All @@ -55,9 +58,7 @@ public Connection(final HostAndPort hostAndPort) {
}

public Connection(final HostAndPort hostAndPort, final JedisClientConfig clientConfig) {
this(new DefaultJedisSocketFactory(hostAndPort, clientConfig));
this.infiniteSoTimeout = clientConfig.getBlockingSocketTimeoutMillis();
initializeFromClientConfig(clientConfig);
this(new DefaultJedisSocketFactory(hostAndPort, clientConfig), clientConfig);
}

public Connection(final JedisSocketFactory socketFactory) {
Expand Down Expand Up @@ -373,16 +374,40 @@ protected void flush() {
}
}

@Experimental
protected Object protocolRead(RedisInputStream is) {
return Protocol.read(is);
}

@Experimental
protected void protocolReadPushes(RedisInputStream is) {
}

protected Object readProtocolWithCheckingBroken() {
if (broken) {
throw new JedisConnectionException("Attempting to read from a broken connection.");
}

try {
return Protocol.read(inputStream);
// Object read = Protocol.read(inputStream);
// System.out.println(redis.clients.jedis.util.SafeEncoder.encodeObject(read));
// return read;
return protocolRead(inputStream);
} catch (JedisConnectionException exc) {
broken = true;
throw exc;
}
}

protected void readPushesWithCheckingBroken() {
if (broken) {
throw new JedisConnectionException("Attempting to read from a broken connection.");
}

try {
if (inputStream.available() > 0) {
protocolReadPushes(inputStream);
}
} catch (IOException e) {
broken = true;
throw new JedisConnectionException("Failed to check buffer on connection.", e);
} catch (JedisConnectionException exc) {
setBroken();
throw exc;
Expand All @@ -404,6 +429,7 @@ public List<Object> getMany(final int count) {

/**
* Check if the client name libname, libver, characters are legal
*
* @param info the name
* @return Returns true if legal, false throws exception
* @throws JedisException if characters illegal
Expand All @@ -419,7 +445,7 @@ private static boolean validateClientInfo(String info) {
return true;
}

private void initializeFromClientConfig(final JedisClientConfig config) {
protected void initializeFromClientConfig(final JedisClientConfig config) {
try {
connect();

Expand All @@ -430,12 +456,12 @@ private void initializeFromClientConfig(final JedisClientConfig config) {
final RedisCredentialsProvider redisCredentialsProvider = (RedisCredentialsProvider) credentialsProvider;
try {
redisCredentialsProvider.prepare();
helloOrAuth(protocol, redisCredentialsProvider.get());
helloAndAuth(protocol, redisCredentialsProvider.get());
} finally {
redisCredentialsProvider.cleanUp();
}
} else {
helloOrAuth(protocol, credentialsProvider != null ? credentialsProvider.get()
helloAndAuth(protocol, credentialsProvider != null ? credentialsProvider.get()
: new DefaultRedisCredentials(config.getUser(), config.getPassword()));
}

Expand All @@ -447,7 +473,9 @@ private void initializeFromClientConfig(final JedisClientConfig config) {
}

ClientSetInfoConfig setInfoConfig = config.getClientSetInfoConfig();
if (setInfoConfig == null) setInfoConfig = ClientSetInfoConfig.DEFAULT;
if (setInfoConfig == null) {
setInfoConfig = ClientSetInfoConfig.DEFAULT;
}

if (!setInfoConfig.isDisabled()) {
String libName = JedisMetaInfo.getArtifactId();
Expand Down Expand Up @@ -492,50 +520,56 @@ private void initializeFromClientConfig(final JedisClientConfig config) {
}
}

private void helloOrAuth(final RedisProtocol protocol, final RedisCredentials credentials) {

if (credentials == null || credentials.getPassword() == null) {
if (protocol != null) {
sendCommand(Command.HELLO, encode(protocol.version()));
getOne();
private void helloAndAuth(final RedisProtocol protocol, final RedisCredentials credentials) {
Map<String, Object> helloResult = null;
if (protocol != null && credentials != null && credentials.getUser() != null) {
byte[] rawPass = encodeToBytes(credentials.getPassword());
try {
helloResult = hello(encode(protocol.version()), Keyword.AUTH.getRaw(), encode(credentials.getUser()), rawPass);
} finally {
Arrays.fill(rawPass, (byte) 0); // clear sensitive data
}
return;
} else {
auth(credentials);
helloResult = protocol == null ? null : hello(encode(protocol.version()));
}
if (helloResult != null) {
server = (String) helloResult.get("server");
version = (String) helloResult.get("version");
}

// Source: https://stackoverflow.com/a/9670279/4021802
ByteBuffer passBuf = Protocol.CHARSET.encode(CharBuffer.wrap(credentials.getPassword()));
byte[] rawPass = Arrays.copyOfRange(passBuf.array(), passBuf.position(), passBuf.limit());
Arrays.fill(passBuf.array(), (byte) 0); // clear sensitive data
// clearing 'char[] credentials.getPassword()' should be
// handled in RedisCredentialsProvider.cleanUp()
}

private void auth(RedisCredentials credentials) {
if (credentials == null || credentials.getPassword() == null) {
return;
}
byte[] rawPass = encodeToBytes(credentials.getPassword());
try {
/// actual HELLO or AUTH -->
if (protocol != null) {
if (credentials.getUser() != null) {
sendCommand(Command.HELLO, encode(protocol.version()),
Keyword.AUTH.getRaw(), encode(credentials.getUser()), rawPass);
getOne(); // Map
} else {
sendCommand(Command.AUTH, rawPass);
getStatusCodeReply(); // OK
sendCommand(Command.HELLO, encode(protocol.version()));
getOne(); // Map
}
} else { // protocol == null
if (credentials.getUser() != null) {
sendCommand(Command.AUTH, encode(credentials.getUser()), rawPass);
} else {
sendCommand(Command.AUTH, rawPass);
}
getStatusCodeReply(); // OK
if (credentials.getUser() == null) {
sendCommand(Command.AUTH, rawPass);
} else {
sendCommand(Command.AUTH, encode(credentials.getUser()), rawPass);
}
/// <-- actual HELLO or AUTH
} finally {

Arrays.fill(rawPass, (byte) 0); // clear sensitive data
}
getStatusCodeReply();
}

// clearing 'char[] credentials.getPassword()' should be
// handled in RedisCredentialsProvider.cleanUp()
protected Map<String, Object> hello(byte[]... args) {
sendCommand(Command.HELLO, args);
return BuilderFactory.ENCODED_OBJECT_MAP.build(getOne());
}

protected byte[] encodeToBytes(char[] chars) {
// Source: https://stackoverflow.com/a/9670279/4021802
ByteBuffer passBuf = Protocol.CHARSET.encode(CharBuffer.wrap(chars));
byte[] rawPass = Arrays.copyOfRange(passBuf.array(), passBuf.position(), passBuf.limit());
Arrays.fill(passBuf.array(), (byte) 0); // clear sensitive data
return rawPass;
}

public String select(final int index) {
Expand Down
Loading
Loading