Advanced Usage
This section explains less common functionality of REEM.
Custom Datatypes
REEM is designed to be customizable. Out of the box, it supports transferring native python types and numpy arrays.
You can, however, define how any type of data is stored in Redis using a Marshaller
object.
Inside the module reem.marshalling
is the abstract class SpecialDatatypeMarshaller
. If you define your own marshaller, you must
subclass SpecialDatatypeMarshaller
and fill in the methods. The class’s documentation is below
- class reem.marshalling.SpecialDatatypeMarshaller
- abstract check_fit(value)
Determine if this marshaller will handle
value
This method returns true if
value
is data that this marshaller is supposed to handle. If this marshaller handled all numpy arrays, it would check ifvalue
’s type is a numpy array.- Parameters
value – object to check
Returns: True if marshaller will handle
value
- abstract write(key, value, client)
Write
value
to Redis at the specifiedkey
usingclient
Given a Redis client, execute any number of needed commands to store the
value
in Redis. You are required to use the key given for REEM to find it. If you must store multiple pieces of information, use a Redis Hash which acts like a one level dictionary.- Parameters
key (str) – The Redis key name this marshaller must store data under
value – The value to write into Redis
client – A ReJSON Redis Client pipeline
Returns: None
- abstract read(key, client)
Retrieve necessary information from Redis
Given a Redis client, execute ONE command to retrieve all the information you need to rebuild the data that was stored in
write
from Redis. This method should execute the command that allows you to retrieve all data stored under key.- Parameters
key (str) – a keyname that contains data stored by
write
client –
A ReJSON Redis Client pipeline
Returns: None
- abstract delete(key, client)
Delete information from redis at the specified
key
usingclient
.Given a Redis client, execute any number of commands to retrieve all the information you need to delete all of the data stored under
key
- Parameters
key (str) – a keyname that contains data stored by
write
client –
A ReJSON Redis Client pipeline
Returns: None
- abstract interpret_read(responses)
Translate Redis data into a local object
Redis will reply to you with something according to what read command you executed in
read
. This method takes whatever Redis replied with and turns it into an object identical to what was initially passed towrite
as value.- Parameters
responses – Redis’s reply data based on
read
method
Returns: An object identical to what was initially written to Redis.
- abstract get_label()
Return a unique string identifier
This method should return a string that uniquely identifies this marshaller. REEM will use it to determine what marshaller to use to decode data that is already stored in Redis.
- Returns
the string identifier
- Return type
str
To use a marshaller, include it as an argument when creating a RedisInterface
object.
from reem import RedisInterface
interface = RedisInterface(host="localhost", ships=[CustomMarshaller()])
This interface object can be passed to KeyValueStore
, PublishSpace
, or the XSubscriber
classes instead of an IP address.
Numpy Arrays
Numpy Arrays are stored in Redis through marshalling. If you want to keep the default marshaller for numpy arrays when including your custom marshallers, you must include the default numpy marshaller in the initializer.
interface = RedisInterface(host="localhost", ships=[reem.marshalling.NumpyMarshaller(), CustomMarshaller()])
See the implementation of the Numpy marshaller below
class NumpyMarshaller(SpecialDatatypeMarshaller):
def check_fit(self, value):
return type(value) in [np.array, np.ndarray]
def get_label(self):
return "default_numpy_handler"
def write(self, key, value, client):
client.hset(key, "arr", memoryview(value.data).tobytes())
client.hset(key, "dtype", str(value.dtype))
client.hset(key, "shape", str(value.shape))
client.hset(key, "strides", str(value.strides))
def delete(self, key, client):
client.delete(key)
def read(self, key, client):
client.hgetall(key)
def interpret_read(self, responses):
hash = responses[0]
dtype = eval("np.{}".format(hash[b'dtype'].decode('utf-8')))
shape = hash[b'shape'].decode("utf-8")[1:-1]
shape = tuple([int(s) for s in shape.split(",") if len(s) > 0])
arr = np.frombuffer(hash[b'arr'], dtype)
arr = np.reshape(arr, shape)
return arr