java - Any workaround for distributing(sharding) key of collections like Lists, Sets etc -

we using redis 2.8.17 jobqueues.

we using rpush , brpoplpush making reliable queue.

as per our current design multiple app-servers push(rpush) jobs single job queue. since brpoplpush operation atomic in redis, jobs later poped(brpoplpush) , processed of server's consumers.

since app servers capable of scaling out, bit concerned redis might become bottleneck in future.

i learnt following documentation on redis partitioning: "it not possible shard dataset single huge key big sorted set"

i wonder whether pre-sharding queues app servers option scale out.

is there cluster can in above design?

the main thing need consider whether you'll need shard @ all. entire stackexchange network (not stackoverflow -- all network) runs off of 2 redis servers (one of i'm pretty sure exists redundancy), uses aggressively. take @ http://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-overflow/

redis absurdly fast (and remarkably space-efficient), 1 caveat: deleting entire list/set/sorted set/hash o(n), n number of elements contains. long don't (operations rpop, brpop, brpoplpush, etc. don't count -- constant time), should golden.

tldr: unless you're planning go bigger stackoverflow, won't need shard, simple job queue.