Advanced Usage

This section explains less common functionality of REEM.

Custom Datatypes

REEM is designed to be customizable. Out of the box, it supports transferring native python types and numpy arrays. You can, however, define how any type of data is stored in Redis using a Marshaller object.

Inside the module reem.marshalling is the abstract class SpecialDatatypeMarshaller. If you define your own marshaller, you must subclass SpecialDatatypeMarshaller and fill in the methods. The class’s documentation is below

class reem.marshalling.SpecialDatatypeMarshaller

Abstract base class for marshallers that convert Python objects to/from ReJSON commands.

abstract check_fit(value)

Determine if this marshaller will handle value

This method returns true if value is data that this marshaller is supposed to handle. If this marshaller handled all numpy arrays, it would check if value’s type is a numpy array.

Parameters

value – object to check

Returns: True if marshaller will handle value

abstract write(key, value, client) None

Write value to Redis at the specified key using client

Given a Redis client, execute any number of needed commands to store the value in Redis. You are required to use the key given for REEM to find it. If you must store multiple pieces of information, use a Redis Hash which acts like a one level dictionary.

Parameters
  • key (str) – The Redis key name this marshaller must store data under

  • value – The value to write into Redis

  • client – A Redis Client

Returns: None

abstract read(key, client)

Retrieve necessary information from Redis

Given a Redis client, execute ONE command to retrieve all the information you need to rebuild the data that was stored in write from Redis. This method should execute the command that allows you to retrieve all data stored under key.

Parameters
  • key (str) – a keyname that contains data stored by write

  • client

    A Redis Client

Returns: None

abstract delete(key, client)

Delete information from redis at the specified key using client.

Given a Redis client, execute any number of commands to retrieve all the information you need to delete all of the data stored under key

Parameters
  • key (str) – a keyname that contains data stored by write

  • client

    A Redis Client

Returns: None

abstract interpret_read(responses)

Translate Redis data into a local object

Redis will reply to you with something according to what read command you executed in read. This method takes whatever Redis replied with and turns it into an object identical to what was initially passed to write as value.

Parameters

responses – Redis’s reply data based on read method

Returns: An object identical to what was initially written to Redis.

abstract get_label()

Return a unique string identifier

This method should return a string that uniquely identifies this marshaller. REEM will use it to determine what marshaller to use to decode data that is already stored in Redis.

Returns

the string identifier

Return type

str

To use a marshaller, include it as an argument when creating a RedisInterface object.

from reem import RedisInterface
interface = RedisInterface(host="localhost", marshallers=[CustomMarshaller()])

This interface object can be passed to KeyValueStore, PublishSpace, or the XSubscriber classes instead of an IP address.

Numpy Arrays

Numpy Arrays are stored in Redis through marshalling. If you want to keep the default marshaller for numpy arrays when including your custom marshallers, you must include the default numpy marshaller in the initializer.

interface = RedisInterface(host="localhost", marshallers=[reem.marshalling.NumpyMarshaller(), CustomMarshaller()])

See the implementation of the Numpy marshaller below

class NumpyMarshaller(SpecialDatatypeMarshaller):
    """A marshaller for Numpy arrays."""
    def check_fit(self, value):
        return type(value) in [np.array, np.ndarray]

    def get_label(self):
        return "default_numpy_handler"

    def write(self, key, value, client):
        client.hset(key, "arr", memoryview(value.data).tobytes())
        client.hset(key, "dtype", str(value.dtype))
        client.hset(key, "shape", str(value.shape))
        client.hset(key, "strides", str(value.strides))

    def delete(self, key, client):
        client.delete(key)

    def read(self, key,  client):
        client.hgetall(key)

    def interpret_read(self, responses):
        hash = responses[0]
        dtype = eval("np.{}".format(hash[b'dtype'].decode('utf-8')))
        shape = hash[b'shape'].decode("utf-8")[1:-1]
        shape = tuple([int(s) for s in shape.split(",") if len(s) > 0])
        arr = np.frombuffer(hash[b'arr'], dtype)
        arr = np.reshape(arr, shape)
        return arr