How to achieve high throughput using Jedis library

[dsinghal-nice]

@sazzad16
I have a requirement where I have to use Redis cache for storing event data received from various streams, and fetch these data by scanning the keys and returning the data in JSON format. Data will be stored for a day only, but will be in millions. Request per second will be 3000 for fetching the data, I am using redis server with redis json module only.

I have created a multi-threaded environment to first scan all the keys and then multiple threads will fire JSON.MGET command to fetch the data. But even doing this I am not able to get the latency below <100ms, the scan command is taking more than 1000 ms to fetch all the keys. I am currently doing this simulation on 7000 JSON objects only, but in real-time it will be in millions.

Below is how my configuration looks like

@Configuration
public class RedisConfig {
    private static final int CORE_POOL_SIZE = 15;
    private static final int MAX_POOL_SIZE = 25;

    @Value("${spring.data.redis.host}")
    private String redisHost;

    @Value("${spring.data.redis.port}")
    private int redisPort;

    @Bean
    public JedisConnectionFactory jedisConnectionFactory() {
        RedisStandaloneConfiguration standaloneConfiguration = new RedisStandaloneConfiguration(redisHost, redisPort);

        JedisPoolConfig poolConfig = new JedisPoolConfig();
        poolConfig.setMaxTotal(500);
        poolConfig.setMaxIdle(300);
        poolConfig.setMinEvictableIdleTimeMillis(300);
        
        poolConfig.setBlockWhenExhausted(true);
        poolConfig.setMaxWaitMillis(100);

        JedisClientConfiguration jedisClientConfiguration = JedisClientConfiguration.builder()
                .usePooling()
                .poolConfig(poolConfig)
                .build();

        return new JedisConnectionFactory(standaloneConfiguration, jedisClientConfiguration);
    }

    @Bean
    public JedisPooled getUnifiedJedis(JedisConnectionFactory jedisConnectionFactory) {
        HostAndPort hostAndPort = new HostAndPort(jedisConnectionFactory.getHostName(), jedisConnectionFactory.getPort());
        return new JedisPooled(hostAndPort);
    }

    @Bean(name = "redisThreadPoolExecutor")
    public ThreadPoolTaskExecutor redisThreadPoolExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(CORE_POOL_SIZE);
        executor.setMaxPoolSize(MAX_POOL_SIZE);
        executor.setThreadNamePrefix("fetch_data_from_redis");
        return executor;
    }
}