Advanced Usage

This section explains less common functionality of REEM.

Custom Datatypes

REEM is designed to be customizable. Out of the box, it supports transferring native python types and numpy arrays. You can, however, define how any type of data is stored in Redis using a Ship object.

Inside the module, reem.ships is the abstract class SpecialDatatypeShip. If you define your own ship, you must subclass SpecialDatatypeShip and fill in the methods. The class’s documentation is below

class reem.ships.SpecialDatatypeShip
check_fit(value)

Determine if this ship will handle value

This method returns true if value is data that this ship is supposed to handle. If this ship handled all numpy arrays, it would check if value’s type is a numpy array.

Parameters

value – object to check

Returns: True if ship will handle value

write(key, value, client)

Write value to Redis at the specified key using client

Given a Redis client, execute any number of needed commands to store the value in Redis. You are required to use the key given for REEM to find it. If you must store multiple pieces of information, use a Redis Hash which acts like a one level dictionary.

Parameters
  • key (str) – The Redis key name this ship must store data under

  • value – The value to write into Redis

  • client – A ReJSON Redis Client pipeline

Returns: None

read(key, client)

Retrieve necessary information from Redis

Given a Redis client, execute ONE command to retrieve all the information you need to rebuild the data that was stored in write from Redis. This method should execute the command that allows you to retrieve all data stored under key

Parameters
  • key (str) – a keyname that contains data stored by write

  • client

    A ReJSON Redis Client pipeline

Returns: None

interpret_read(responses)

Translate Redis data into a local object

Redis will reply to you with something according to what read command you executed in read. This method takes whatever Redis replied with and turns it into an object identical to what was initially passed to write as value.

Parameters

responses – Redis’s reply data based on read method

Returns: An object identical to what was initially written to Redis.

get_label()

Return a unique string identifier

This method should return a string that uniquely identifies this ship. REEM will use it to determine what ship to use to decode data that is already stored in Redis.

Returns

the string identifier

Return type

str

To use a ship, include it as an argument when creating a RedisInterface object.

interface = RedisInterface(host="localhost", ships=[CustomShip()])

Numpy Arrays

Numpy Arrays are stored in Redis through ships. If you want to keep the default ship for numpy arrays when including your custom ships, you must include the default numpy ship in your list of ships.

interface = RedisInterface(host="localhost", ships=[reem.ships.NumpyShip(), CustomShip()])

See the implementation of the Numpy Ship below

class NumpyShip(SpecialDatatypeShip):
    def check_fit(self, value):
        return type(value) in [np.array, np.ndarray]

    def write(self, key, value, client):
        client.hset(key, "arr", bytes(memoryview(value.data)))
        client.hset(key, "dtype", str(value.dtype))
        client.hset(key, "shape", str(value.shape))
        client.hset(key, "strides", str(value.strides))

    def get_label(self):
        return "default_numpy_handler"

    def read(self, key,  client):
        client.hgetall(key)

    def interpret_read(self, responses):
        hash = responses[0]
        dtype = eval("np.{}".format(hash[b'dtype'].decode('utf-8')))
        shape = hash[b'shape'].decode("utf-8")[1:-1]
        shape = tuple([int(s) for s in shape.split(",") if len(s) > 0])
        arr = np.frombuffer(hash[b'arr'], dtype)
        arr = np.reshape(arr, shape)
        return arr