Redis [redis.io] is an in-memory data structure storage database. Redis supports data structures such as hashes, lists, sets, sorted sets with range queries. Redis also provides a PUB/SUB messaging system.
I’ve been using Redis to store and process the data required for the BeefPool.net Ethereum mining pool running on a Ubuntu 14.04.4 LTS on a virtual server. As mining is a calculation competition between miners, every millisecond counts. Using Redis to record data sent to and from the miners adds minimal delays to the communications between the pool and the miners.
The mining pool back-end has been developed using the Go [golang.org] programming language and uses the godis [github.com/simonz05/godis] Go Redis client module.
I’m in the process of writing the mining pool front-end to perform the accounting calculations and make the automated payments. This is written in Java and uses the lettuce [github.com/mp911de/lettuce] Java Redis client module.
Notes on the installation and usage of Redis on the Linux platform follow.
In summary, Redis is easy to install and use. It is fast enough for my requirements, does not use much CPU or disk resources and can be accessed from various programming languages as well as the command. You will have to spend some time to design your data models for easy storage and access of the data. Although I have not used this functionality, Redis databases can be clustered.
Installation Of Redis On Ubuntu Linux
It’s simple to install Redis from the Ubuntu repository:
1 |
beefee@BeefPool:~$ sudo apt-get install redis-server |
Here’s a benchmark of Redis to check it is running:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
beefee@BeefPool:~$ redis-benchmark -q -n 1000 -c 10 -P 5 PING_INLINE: 200000.00 requests per second PING_BULK: 249999.98 requests per second SET: 333333.34 requests per second GET: 249999.98 requests per second INCR: 249999.98 requests per second LPUSH: 249999.98 requests per second LPOP: 249999.98 requests per second SADD: 333333.34 requests per second SPOP: 249999.98 requests per second LPUSH (needed to benchmark LRANGE): 249999.98 requests per second LRANGE_100 (first 100 elements): 249999.98 requests per second LRANGE_300 (first 300 elements): 249999.98 requests per second LRANGE_500 (first 450 elements): 249999.98 requests per second LRANGE_600 (first 600 elements): 249999.98 requests per second MSET (10 keys): 166666.67 requests per second |
Remember to set a password in /etc/redis/redis.conf to access Redis:
1 |
requirepass onebeefypassword |
And restart Redis server for password to take effect:
1 |
beefee@BeefPool:~$ sudo service redis-server restart |
The Redis data can be accessed from the command line:
1 2 3 |
beefee@BeefPool:~$ redis-cli 127.0.0.1:6379> AUTH onebeefypassword OK |
You can also specify the password from the command line:
1 2 |
beefee@BeefPool:~$ redis-cli -a onebeefypassword 127.0.0.1:6379> |
The Redis installation uses port 6379 on localhost by default:
1 2 3 4 5 6 |
beefee@BeefPool:~$ netstat -an | grep 6379 tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:47068 127.0.0.1:6379 ESTABLISHED tcp 0 0 127.0.0.1:6379 127.0.0.1:47070 ESTABLISHED tcp 0 0 127.0.0.1:6379 127.0.0.1:47068 ESTABLISHED tcp 0 0 127.0.0.1:47070 127.0.0.1:6379 ESTABLISHED |
Using Redis In The Go Mining Pool Back-End
Import the godis module into your source code:
1 2 3 4 5 |
import ( ... "github.com/simonz05/godis/redis" ... ) |
Declare a client variable:
1 2 |
var redisPassword = "onebeefypassword" var redisClient *redis.Client |
Open the Redis connection:
1 2 3 4 5 6 |
var err error redisClient = redis.New("", 0, redisPassword) if err != nil { logError.Println("Unable to open redis connection:", err) return } |
Here’s how I save a miner’s reported hashrate into a Redis sorted set. The sort key is a 64-bit floating point number and I am using a number of millseconds from 01/01/1970 as the sort key:
1 2 3 4 5 6 |
now := time.Now() var s = workerId + ":" + strconv.FormatFloat(minerDifficulty, 'f', -1, 64) + ":" + strconv.FormatUint(hashrate, 10) if res, err := redisClient.Zadd("reportedHashrate:"+miner+":"+worker, float64(now.Unix()*1000), s); err != nil { logError.Println("redis.Zadd reportedHashrate:"+miner+":"+worker, s, err, res) return } |
Here’s how I save a miner’s successful mining of a block into a Redis sorted set:
1 2 3 |
if res, err := redisClient.Zadd("mined:"+miner+":"+worker, float64(now.Unix()*1000), s); err != nil { logError.Println("redis.Zadd mined:"+miner+":"+worker, float64(now.Unix()*1000), s, err, res) } |
And here’s how I save a miner’s preferences into a Redis hash:
1 2 3 4 5 |
s = strconv.FormatFloat(payout, 'f', -1, 64) + ":" + email + ":" + strconv.Itoa(emailLevel) if err := redisClient.Set("minerPref:"+miner, s); err != nil { logError.Println("redis.Set minerPref:"+miner, s, err) return } |
View The Saved Redis Data
Let’s view the data saved by the mining pool back-end. First we will check the keys created:
1 2 3 4 5 6 7 8 9 10 11 |
beefee@BeefPool:~$ redis-cli -a onebeefypassword 127.0.0.1:6379> KEYS * 1) "reportedHashrate:917a3086e47ff0d9efc40221f454aa510c507191:Kumquat" 2) "reportedHashrate:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8:Masterator" 3) "reportedHashrate:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8:Rasterbator" 4) "shares:917a3086e47ff0d9efc40221f454aa510c507191:Kumquat" 5) "minerPref:917a3086e47ff0d9efc40221f454aa510c507191" 6) "shares:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8:Masterator" 7) "shares:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8:Rasterbator" 8) "minerPref:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8" 9) "mined:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8:Masterator" |
View the miner’s preference stored in a Redis hash:
1 2 |
127.0.0.1:6379> GET minerPref:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8 "0.7:my.email@mydomainname.com.au:1" |
Let’s look at the last three reported hashrate by a miner. I have saved this information in a Redis sorted set using using a sort key (called “score” in Redis) of the number of milliseconds since 01/01/1970. Here the ZREVRANGE command to view the latest 3 reported hashrate records:
1 2 3 4 5 6 7 8 9 |
127.0.0.1:6379> ZREVRANGE reportedHashrate:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8:Rasterbator 0 3 WITHSCORES 1) "0x85cc6559409b94fcaf0c7c66da554a2c931d2f84188a6f1527335fa9693af12b:58:60512952" 2) "1459091750000" 3) "0x85cc6559409b94fcaf0c7c66da554a2c931d2f84188a6f1527335fa9693af12b:58:67855271" 4) "1459091749000" 5) "0x85cc6559409b94fcaf0c7c66da554a2c931d2f84188a6f1527335fa9693af12b:58:59433457" 6) "1459091748000" 7) "0x85cc6559409b94fcaf0c7c66da554a2c931d2f84188a6f1527335fa9693af12b:58:65604337" 8) "1459091747000" |
Let’s retrieve a miner’s reported hashrate between two time ranges:
1 2 3 4 5 |
127.0.0.1:6379> ZREVANGEBYSCORE reportedHashrate:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8:Rasterbator 1459091750000 1459091747000 WITHSCORES 1) "0x85cc6559409b94fcaf0c7c66da554a2c931d2f84188a6f1527335fa9693af12b:58:60512952" 2) "1459091750000" 3) "0x85cc6559409b94fcaf0c7c66da554a2c931d2f84188a6f1527335fa9693af12b:58:65604337" 4) "1459091747000" |
Let’s look at the data recording the blocks mined by a miner:
1 2 3 4 5 6 7 |
127.0.0.1:6379> ZREVRANGE mined:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8:Masterator 0 -1 WITHSCORES 1) "5700000000:0x0ec775775084922e3ea0a88bd20fb238e24105278d873d274c97c7945268feb5:0x77bbb4b9dc43d3a0f9bf9cb261018d99817f70b7f815c8066ca21f6fad25c11a:9f0a1660c7a4e1dc" 2) "1459079638000" 3) "5700000000:0x3f5a9789bb8530a1a5aabb5cddba815ba71d33c8091f81d5cb1851425699d6af:0xfb728fc90ca599dd9655ee3fc83a878f75a44f67ee9b44fba8f85668e8f7c6ef:609c8a9b08c9cb06" 4) "1458940562000" 5) "5700000000:0x425fd257eee29a756be24deef0b87b6b195ac2469605a5c7a851c789dd38a82e:0xa8e60f3b81044de1a012a33a00e99e57a05b2e0dca8f8d0c3c7c4f011a1b90d6:374f655c95bb11ca" 6) "1458636306000" |
This corresponds to the last three blocks mined by this BeefPool.net mining pool. From etherchain.org/account/0xbEeF281B81d383336AcA8B2B067a526227638087#blocks:
Accessing The Saved Redis Data from Java
Add the following Maven dependency:
1 2 3 4 5 |
<dependency> <groupId>biz.paluch.redis</groupId> <artifactId>lettuce</artifactId> <version>4.1.1.Final</version> </dependency> |
Declare the RedisClient variable:
1 |
private final RedisClient redisClient; |
Instantiate the Redis client variables and connect to Redis:
1 2 3 4 |
this.redisClient = RedisClient.create("redis://onebeefypassword@localhost:6379"); ... StatefulRedisConnection<String, String> connection = redisClient.connect(); RedisCommands<String, String> redisCommands = connection.sync(); |
Retrieve the miner’s preferences from the Redis hash:
1 2 3 4 5 |
List<String> keys = redisCommands.keys("minerPref:*"); for (String key : keys) { String value = redisCommands.get(key); // Process your key:value data here } |
Retrieve the miner’s hashrate for the last 20 minutes:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
long activeTime = System.currentTimeMillis(); long previousTime = currentTime - 60 * 20 * 1000; List<String> keys = redisCommands.keys("reportedHashrate:*"); for (String key : keys) { List<ScoredValue<String>> hashrateRecords = redisCommands.zrevrangebyscoreWithScores(key, currentTime, previousTime); if (hashrateRecords.size() > 0) { for (ScoredValue<String> hashrateRecord : hashrateRecords) { // Process values String[] fields = StringUtils.split(hashrateRecord.value, ":"); // And the scores if (hashrateRecord.score > activeTime) { ... } } } } |
Further Redis Features
Saving Data To Disk
By default Redis saves it’s in-memory data to disk after a specified interval if more that a specified number of keys are modified/written. To manually save the Redis data held in memory into a disk file, issue the command:
1 2 |
127.0.0.1:6379> save OK |
The data is saved into the the following file by default:
1 2 |
beefee@BeefPool:~$ ls -al /var/lib/redis/dump.rdb -rw-rw---- 1 redis redis 4061415 Mar 28 02:31 /var/lib/redis/dump.rdb |
Separate Databases
You can store your data in separate databases within your Redis instance. The number of databases by default is 16 – this can be configured with the ‘database’ key within /etc/redis/redis.conf .
The following snippet demonstrates how to switch databases:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
beefee@BeefPool:~$ redis-cli -a onebeefypassword 127.0.0.1:6379> select 1 OK 127.0.0.1:6379[1]> keys * (empty list or set) 127.0.0.1:6379[1]> select 0 OK 127.0.0.1:6379> keys sha* 1) "shares:917a3086e47ff0d9efc40221f454aa510c507191:Kumquat" 2) "shares:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8:Masterator" 3) "shares:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8:Rasterbator" 127.0.0.1:6379> select 1 OK 127.0.0.1:6379[1]> SET "testkey" "testvalue" OK 127.0.0.1:6379[1]> keys * 1) "testkey" 127.0.0.1:6379[1]> select 0 OK 127.0.0.1:6379> keys sha* 1) "shares:917a3086e47ff0d9efc40221f454aa510c507191:Kumquat" 2) "shares:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8:Masterator" 3) "shares:f8b8de9241af1a69eca7b404ee988a6cdfdf98b8:Rasterbator" |
Publish/Subscribe
Start Redis in two separate sessions. Here is session #1 where I have subscribed to the topic “test”.
1 2 3 4 5 6 7 8 9 |
beefee@BeefPool:~$ redis-cli -a onebeefypassword 127.0.0.1:6379> SUBSCRIBE test Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "test" 3) (integer) 1 1) "message" 2) "test" 3) "Hello" |
1 2 3 4 |
beefee@BeefPool:~$ redis-cli -a onebeefypassword 127.0.0.1:6379> PUBLISH test "Hello" (integer) 1 127.0.0.1:6379> |
1 2 3 4 5 6 7 8 9 10 |
beefee@BeefPool:~$ redis-cli -a onebeefypassword 127.0.0.1:6379> PSUBSCRIBE Topic.A* Reading messages... (press Ctrl-C to quit) 1) "psubscribe" 2) "Topic.A*" 3) (integer) 1 1) "pmessage" 2) "Topic.A*" 3) "Topic.All" 4) "Hello" |
1 2 3 4 5 6 7 8 9 |
beefee@BeefPool:~$ redis-cli -a onebeefypassword 127.0.0.1:6379> PUBLISH test "Hello" (integer) 0 127.0.0.1:6379> PUBLISH A "Hello" (integer) 0 127.0.0.1:6379> PUBLISH Topic.All "Hello" (integer) 1 127.0.0.1:6379> PUBLISH Topic.BAll "Hello" (integer) 0 |
Transactions
Redis commands can be grouped together in transactions so that either all commands will successfully be run or all commands will fail:
1 2 3 4 5 6 7 8 9 10 |
beefee@BeefPool:~$ redis-cli -a onebeefypassword 127.0.0.1:6379[1]> MULTI OK 127.0.0.1:6379[1]> SET key1 value1 QUEUED 127.0.0.1:6379[1]> SET key2 value2 QUEUED 127.0.0.1:6379[1]> EXEC 1) OK 2) OK |
Keys With Automatic Expiry
I’ll delete all existing keys, then create a key with a 5 second expiry:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
beefee@BeefPool:~$ redis-cli -a onebeefypassword 127.0.0.1:6379[1]> KEYS * 1) "key2" 2) "key1" 3) "test" 127.0.0.1:6379[1]> DEL test key1 key2 (integer) 3 127.0.0.1:6379[1]> KEYS * (empty list or set) 127.0.0.1:6379[1]> SET keya valuea OK 127.0.0.1:6379[1]> EXPIRE keya 5 (integer) 1 127.0.0.1:6379[1]> KEYS * 1) "keya" 127.0.0.1:6379[1]> KEYS * (empty list or set) 127.0.0.1:6379[1]> |