API Documentation

reem

class reem.RedisInterface(host: str = 'localhost', marshallers=[<reem.marshalling.NumpyMarshaller object>], *args, **kwargs)

Low level class for interfacing with Redis. Notably, stores the set of marshallers used for accessing Numpy and other user data.

Variable args and keyword arguments are passed to a redis.Redis() constructor.

hostname

the hostname of the Redis server (default localhost)

Type

str

marshallers

the list of marshallers used for decoding Python objects like Numpy arrays.

Type

List[Marshaller]

client

the ReJSON client

Type

redis.Redis

client_no_decode

the ReJSON client used for communicating with Redis without decoding JSON results.

Type

redis.Redis

client_copy()

class reem.KeyValueStore(interface: Union[str, reem.connection.RedisInterface, reem.connection.KeyValueStore] = 'localhost', *args, **kwargs)

Dictionary Like object used for get/set paradigm

The KeyValueStore object is one that users will frequently use. It keeps Reader and Writer objects for each key that the user has read or written to. It does not actually handle the getting and setting of data but produces KeyAccessor objects that assist with path construction and call the reader and writer’s write and read methods.

When constructing this class, users can pass through *args and **kwargs that will be forwarded to the RedisInterface and eventually to the Redis client. This allows users to set things like socket timeouts. See https://redis.readthedocs.io/en/latest/connections.html for a full list of options.

interface

Defines the connection to Redis this reader will use. If a str, then a RedisInterface will be created and connected to automatically.

Type

str, RedisInterface, or KeyValueStore

get(path: str) → Any

Reads a path in the form ‘key.element1.element2’ from the redis server. If the path doesn’t exist, an error is raised.

Marginally faster than nested accessors.

set(path: str, value: Any) → None

Sets a value in a path in the form ‘key.element1.element2’ from the redis server. If any elements along the path, except for the last key, do not exist, an error is raised.

value must be a JSON-serializable object.

Marginally faster than nested accessors.

track_schema_changes(track: bool, keys=None) → None

Performance optimization for skipping schema update checks

Stop checking for schema updates when setting data. Use ONLY if your data’s schema is static and you are really trying to eke out every bit of optimization.

Parameters

• track (bool) – True/False indicating if the keys’ schema should be tracked

• keys (List[str]) – List of keys to track. If None, all present and future keys are tracked according to track

Returns: None

class reem.TolerantKeyValueStore(interface: Union[str, reem.connection.RedisInterface, reem.connection.KeyValueStore] = 'localhost', refreshRate: float = 0, *args, **kwargs)

TolerantKeyValueStore acts almost like a normal KeyValueStore, except for the following advantages when using the var(), set(), and get() methods:

• More tolerant of uninitialized keys and will fill in empty dicts as needed.

• Can accept paths in the form of string paths ['key1','key2'] or period- separated keys 'key1.key2'

Note

Does not accept path strings that include integer array indices, yet. To access arrays, you will need to pass in arrays, e.g. ['key1','key2',3]

There are also some performance enhancements when using var():

• If you are accessing a sub-key many times in success to obtain its sub-objects, storing a var() once and accessing its sub-keys is much faster because it will cache the object and avoid making many calls to the server.

• If you are accessing a sub-key of a large object, use var(path_to_item) rather than var(key1)[key2][...]. The latter will retrieve the whole object referenced by key1.

• You may play around with the cache refresh time with the second argument to var() or with the attribute defaultVarRefreshRate. If you have a slow item that only updates occasionally, e.g., by user input, set a larger refresh, e.g. var(path,2) will make at most one query to the server per 2 seconds.

var(path: Union[str, list], refreshRate: Optional[float] = None) → reem.convenience.TolerantKVSVar

Given a path (period-separated str or list of str), returns a TolerantKVSVar object that accesses the key using get()/set().

All parent keys in path will be created as empty dictionaries if they don’t exist.

If refreshRate!=0, values will be cached, only refreshing after refreshRate seconds elapse. This minimizes server traffic.

get(path: Union[str, list]) → Any

Given a path (period-separated str or list of str), returns a JSON object or numpy array at path by recursively descending into self.

set(path: Union[str, list], value: Any)

Given a path (period-separated str or list of str), sets a value in the state server. The item must be json-encodable or a numpy array.

If you are accessing a path with multiple subkeys that don’t exist, the subkeys will be added.

class reem.PublishSpace(interface: Union[str, reem.connection.RedisInterface, reem.connection.KeyValueStore] = 'localhost', *args, **kwargs)

Convenience class for publishing

This class keeps track of PublishWriter objects for each key the user has published on.

class reem.CallbackSubscriber(channel: str, interface: Union[str, reem.connection.RedisInterface], callback_function: Callable, kwargs)

A Subscriber that calls a user-defined callback every time a publisher sends a message. You can also access the most up-to-date value using value() or [key].

The callback function is of the form f(channel,message,**kwargs) and is called every time it receives a message published on channel_name*

call_user_function(channel, message)

Wrapper callback function (wrapping user function) for this class to work with a RawSubscriber object Fits required interface for a ChannelSubscriber callback function :param channel: channel published to :param message: message that was published :return: None :rtype: None

class reem.SilentSubscriber(channel: str, interface: Union[str, reem.connection.RedisInterface])

A Subscriber that reads updates into a local dict, and users access the most up-to-date values using value() or [key].

update_local_copy(channel, message)

Update the local copy of the data stored under this channel name in redis.

Parameters

• channel – the name of the channel that was published.

• message – message published on that channel

Returns: None

listen() → None

Makes this subscriber start listening

Returns: None

value()

Get data stored at root

Where as the reader can do server["foo"].read() with if server is a KeyValueStore, accessing the root value of a subscriber is not as easy. This method retrieves all data stored underneath a top level key.

Returns: all data stored underneath a top level Redis key.

reem.connection

class reem.connection.RedisInterface(host: str = 'localhost', marshallers=[<reem.marshalling.NumpyMarshaller object>], *args, **kwargs)

Low level class for interfacing with Redis. Notably, stores the set of marshallers used for accessing Numpy and other user data.

Variable args and keyword arguments are passed to a redis.Redis() constructor.

hostname

the hostname of the Redis server (default localhost)

Type

str

marshallers

the list of marshallers used for decoding Python objects like Numpy arrays.

Type

List[Marshaller]

client

the ReJSON client

Type

redis.Redis

client_no_decode

the ReJSON client used for communicating with Redis without decoding JSON results.

Type

redis.Redis

client_copy()

class reem.connection.KeyValueStore(interface: Union[str, reem.connection.RedisInterface, reem.connection.KeyValueStore] = 'localhost', *args, **kwargs)

Dictionary Like object used for get/set paradigm

The KeyValueStore object is one that users will frequently use. It keeps Reader and Writer objects for each key that the user has read or written to. It does not actually handle the getting and setting of data but produces KeyAccessor objects that assist with path construction and call the reader and writer’s write and read methods.

When constructing this class, users can pass through *args and **kwargs that will be forwarded to the RedisInterface and eventually to the Redis client. This allows users to set things like socket timeouts. See https://redis.readthedocs.io/en/latest/connections.html for a full list of options.

interface

Defines the connection to Redis this reader will use. If a str, then a RedisInterface will be created and connected to automatically.

Type

str, RedisInterface, or KeyValueStore

get(path: str) → Any

Reads a path in the form ‘key.element1.element2’ from the redis server. If the path doesn’t exist, an error is raised.

Marginally faster than nested accessors.

set(path: str, value: Any) → None

Sets a value in a path in the form ‘key.element1.element2’ from the redis server. If any elements along the path, except for the last key, do not exist, an error is raised.

value must be a JSON-serializable object.

Marginally faster than nested accessors.

track_schema_changes(track: bool, keys=None) → None

Performance optimization for skipping schema update checks

Stop checking for schema updates when setting data. Use ONLY if your data’s schema is static and you are really trying to eke out every bit of optimization.

Parameters

• track (bool) – True/False indicating if the keys’ schema should be tracked

• keys (List[str]) – List of keys to track. If None, all present and future keys are tracked according to track

Returns: None

class reem.connection.PublishSpace(interface: Union[str, reem.connection.RedisInterface, reem.connection.KeyValueStore] = 'localhost', *args, **kwargs)

Convenience class for publishing

This class keeps track of PublishWriter objects for each key the user has published on.

class reem.connection.RawSubscriber(channel_name, interface, callback_function, kwargs)

A Subscriber that calls a callback function of the form f(channel,message,**kwargs) every time it receives a message published on channel_name*.

listen()

class reem.connection.SilentSubscriber(channel: str, interface: Union[str, reem.connection.RedisInterface])

A Subscriber that reads updates into a local dict, and users access the most up-to-date values using value() or [key].

update_local_copy(channel, message)

Update the local copy of the data stored under this channel name in redis.

Parameters

• channel – the name of the channel that was published.

• message – message published on that channel

Returns: None

listen() → None

Makes this subscriber start listening

Returns: None

value()

Get data stored at root

Where as the reader can do server["foo"].read() with if server is a KeyValueStore, accessing the root value of a subscriber is not as easy. This method retrieves all data stored underneath a top level key.

Returns: all data stored underneath a top level Redis key.

class reem.connection.CallbackSubscriber(channel: str, interface: Union[str, reem.connection.RedisInterface], callback_function: Callable, kwargs)

A Subscriber that calls a user-defined callback every time a publisher sends a message. You can also access the most up-to-date value using value() or [key].

The callback function is of the form f(channel,message,**kwargs) and is called every time it receives a message published on channel_name*

call_user_function(channel, message)

Wrapper callback function (wrapping user function) for this class to work with a RawSubscriber object Fits required interface for a ChannelSubscriber callback function :param channel: channel published to :param message: message that was published :return: None :rtype: None

class reem.connection.Writer(top_key_name: str, interface: reem.connection.RedisInterface)

Responsible for setting data inside Redis

The Writer class is an internal class that is used for all data sent to Redis (not including pub/sub messages). Each key that will have nested data below requires a new instantiation of Writer

top_key_name

The name of the Redis key under which JSON data will be stored. To Redis, this will become a ReJSON key name. It is also used to generate the Redis key name that marshallers use to store non JSON data.

Type

str

interface

Defines the connection to Redis this writer will use

Type

RedisInterface

send_to_redis(set_path, set_value)

Execute equivalent of JSON.SET self.top_key_name <set_path> <set_value>

From the user’s perspective, it executes JSON.SET self.top_key_name <set_path> <set_value> except that set_value can be json-incompatible. In such a case, it determines what non-serializable types are inside set_value, stores the serializable data as a JSON, and stores the non-serializable data using self.interface’s marshallers.

Parameters

• set_path (str) – path underneath JSON key to set

• set_value – value to set

Returns

None

delete_from_redis(del_path: str)

Execute equivalent of JSON.DEL self.top_key_name <set_path>

Parameters

del_path (str) – path underneath JSON key to delete

Returns

None

class reem.connection.Reader(top_key_name, interface)

Responsible for getting data from Redis

The Reader class is an internal class that is used for all read from Redis (not including pub/sub messages). Each key that will have nested data below requires a new instantiation of Reader

top_key_name

The name of the Redis key under which JSON data is stored

Type

str

interface

Defines the connection to Redis this reader will use

Type

RedisInterface

read_from_redis(read_path)

Read specified path from Redis

This is the only public method of the Reader class. It will retrieve the data stored at a specified path from Redis. At a high level, it reads data stored with ReJSON and inserts non-JSON compatible data at appropriate paths using the metadata associated with this key.

Parameters

read_path (str) – path the user wants to read

Returns: data stored at value in Redis

update_metadata()

Update the local copy of metadata if a relevant path has been updated.

The metadata listener is a redis client subscribed to key-space notifications. If a relevant path is updated, this Reader’s pull_metadata flag will be turned on. If pull_metadata is True, then the reader will fetch metadata from the Redis server.

Returns: None

queue_reads(read_path)

Queue reads in a pipeline

Queue all redis queries necessary to read data at path into the appropriate redis pipeline. First, queue decoded pipeline with the ReJSON query Next, queue all the special path reads with the non-decoded pipeline and marshallers

Parameters

read_path – path user wants to read

Returns: None

build_dictionary(read_path)

Execute pipelines and consolidate data into a dictionary

Parameters

read_path – path user wants to read

Returns: The data stored at read_path in Redis

pull_special_path(read_path)

Directly pull a non-JSON path

If the user specified path is not in JSON, this will retrieve the data directly without going through ReJSON.

Parameters

read_path – path user wants to read

Returns:

class reem.connection.PublishWriter(top_key_name, interface)

Identical to Writer, but publishes a message when it writes a value.

send_to_redis(set_path, set_value)

PublishWriter equivalent of Writer send_to_redis

This is an equivalent function to Writer’s send_to_redis method but also publishes a message indicating what channel has been updated.

Parameters

• set_path (str) – path underneath JSON key to set

• set_value – value to set

Returns: None

delete_from_redis(del_path)

Execute equivalent of JSON.DEL self.top_key_name <set_path>

Parameters

del_path (str) – path underneath JSON key to delete

Returns

None

reem.accessors

class reem.accessors.MetadataListener(client)

add_listener(key_name, reader)

flush()

class reem.accessors.KeyAccessor(parent, writer, reader, initial_path=[])

Main class for accessing sub-keys of a KeyValueStore.

get(key: str, default_value: Optional[Any] = None) → Any

Similar to dict’s get() method, returns a default value if the key doesn’t exist.

Essentially equivalent to ``` try:

value = self[key].read()

except:

value = default_value

```

read() → Any

Actually read the value referred to by this accessor.

write(value: Any) → None

Writes value to the path referred to by this accessor.

type() → str

Returns the type of the object

append(rhs: Any) → None

Appends a value to an array

Type checking is not performed, so the user should know what they’re doing.

class reem.accessors.WriteOnlyKeyAccessor(*args, **kwargs)

read()

Actually read the value referred to by this accessor.

append(rhs)

Appends a value to an array

Type checking is not performed, so the user should know what they’re doing.

class reem.accessors.ActiveSubscriberKeyAccessor(*args, **kwargs)

write()

Writes value to the path referred to by this accessor.

append(rhs)

Appends a value to an array

Type checking is not performed, so the user should know what they’re doing.

read() → Any

Actually read the value referred to by this accessor.

class reem.accessors.ChannelListener(client, channel_name, callback_function, kwargs)

run()

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

reem.convenience

class reem.convenience.TolerantKeyValueStore(interface: Union[str, reem.connection.RedisInterface, reem.connection.KeyValueStore] = 'localhost', refreshRate: float = 0, *args, **kwargs)

TolerantKeyValueStore acts almost like a normal KeyValueStore, except for the following advantages when using the var(), set(), and get() methods:

• More tolerant of uninitialized keys and will fill in empty dicts as needed.

• Can accept paths in the form of string paths ['key1','key2'] or period- separated keys 'key1.key2'

Note

Does not accept path strings that include integer array indices, yet. To access arrays, you will need to pass in arrays, e.g. ['key1','key2',3]

There are also some performance enhancements when using var():

• If you are accessing a sub-key many times in success to obtain its sub-objects, storing a var() once and accessing its sub-keys is much faster because it will cache the object and avoid making many calls to the server.

• If you are accessing a sub-key of a large object, use var(path_to_item) rather than var(key1)[key2][...]. The latter will retrieve the whole object referenced by key1.

• You may play around with the cache refresh time with the second argument to var() or with the attribute defaultVarRefreshRate. If you have a slow item that only updates occasionally, e.g., by user input, set a larger refresh, e.g. var(path,2) will make at most one query to the server per 2 seconds.

var(path: Union[str, list], refreshRate: Optional[float] = None) → reem.convenience.TolerantKVSVar

Given a path (period-separated str or list of str), returns a TolerantKVSVar object that accesses the key using get()/set().

All parent keys in path will be created as empty dictionaries if they don’t exist.

If refreshRate!=0, values will be cached, only refreshing after refreshRate seconds elapse. This minimizes server traffic.

get(path: Union[str, list]) → Any

Given a path (period-separated str or list of str), returns a JSON object or numpy array at path by recursively descending into self.

set(path: Union[str, list], value: Any)

Given a path (period-separated str or list of str), sets a value in the state server. The item must be json-encodable or a numpy array.

If you are accessing a path with multiple subkeys that don’t exist, the subkeys will be added.

class reem.convenience.TolerantKVSVar(server: reem.convenience.TolerantKeyValueStore, path: Union[str, list], refreshRate: float = 0)

An accessor for a TolerantKeyValueStore. Can be more convenient than accessing the server via server[key].read() / write() because it will create parent keys from a period-separated path, and can be more efficient on reads due to caching and pre-creation of path strings.

Supports operators [], del [], type(), len(), +=, -=, *=, /=, and append().

get()

Reads the value at the given key from the state server

set(value)

Sets the value at the given key into the state server.

type()

append(rhs)

class reem.convenience.TolerantKVSVarSubkey(root: reem.convenience.TolerantKVSVar, parentnode, key: str)

Accessor for sub-keys of TolerantKVSVar’s. Respects cache, and refreshes the cache of the original object referenced by var().

Performs copy-on-write semantics.

Supports operators [], del [], type(), len(), +=, -=, *=, /=, and append().

get()

set(value)

type()

append(rhs)

reem.marshalling

class reem.marshalling.SpecialDatatypeMarshaller

Abstract base class for marshallers that convert Python objects to/from ReJSON commands.

abstract check_fit(value)

Determine if this marshaller will handle value

This method returns true if value is data that this marshaller is supposed to handle. If this marshaller handled all numpy arrays, it would check if value’s type is a numpy array.

Parameters

value – object to check

Returns: True if marshaller will handle value

abstract write(key, value, client) → None

Write value to Redis at the specified key using client

Given a Redis client, execute any number of needed commands to store the value in Redis. You are required to use the key given for REEM to find it. If you must store multiple pieces of information, use a Redis Hash which acts like a one level dictionary.

Parameters

• key (str) – The Redis key name this marshaller must store data under

• value – The value to write into Redis

• client – A Redis Client

Returns: None

abstract read(key, client)

Retrieve necessary information from Redis

Given a Redis client, execute ONE command to retrieve all the information you need to rebuild the data that was stored in write from Redis. This method should execute the command that allows you to retrieve all data stored under key.

Parameters

• key (str) – a keyname that contains data stored by write

• client –

  A Redis Client

Returns: None

abstract delete(key, client)

Delete information from redis at the specified key using client.

Given a Redis client, execute any number of commands to retrieve all the information you need to delete all of the data stored under key

Parameters

• key (str) – a keyname that contains data stored by write

• client –

  A Redis Client

Returns: None

abstract interpret_read(responses)

Translate Redis data into a local object

Redis will reply to you with something according to what read command you executed in read. This method takes whatever Redis replied with and turns it into an object identical to what was initially passed to write as value.

Parameters

responses – Redis’s reply data based on read method

Returns: An object identical to what was initially written to Redis.

abstract get_label()

Return a unique string identifier

This method should return a string that uniquely identifies this marshaller. REEM will use it to determine what marshaller to use to decode data that is already stored in Redis.

Returns

the string identifier

Return type

str

class reem.marshalling.NumpyMarshaller

A marshaller for Numpy arrays.

check_fit(value)

Determine if this marshaller will handle value

This method returns true if value is data that this marshaller is supposed to handle. If this marshaller handled all numpy arrays, it would check if value’s type is a numpy array.

Parameters

value – object to check

Returns: True if marshaller will handle value

get_label()

Return a unique string identifier

This method should return a string that uniquely identifies this marshaller. REEM will use it to determine what marshaller to use to decode data that is already stored in Redis.

Returns

the string identifier

Return type

str

write(key, value, client)

Write value to Redis at the specified key using client

Given a Redis client, execute any number of needed commands to store the value in Redis. You are required to use the key given for REEM to find it. If you must store multiple pieces of information, use a Redis Hash which acts like a one level dictionary.

Parameters

• key (str) – The Redis key name this marshaller must store data under

• value – The value to write into Redis

• client –

  A Redis Client

Returns: None

delete(key, client)

Delete information from redis at the specified key using client.

Given a Redis client, execute any number of commands to retrieve all the information you need to delete all of the data stored under key

Parameters

• key (str) – a keyname that contains data stored by write

• client –

  A Redis Client

Returns: None

read(key, client)

Retrieve necessary information from Redis

Given a Redis client, execute ONE command to retrieve all the information you need to rebuild the data that was stored in write from Redis. This method should execute the command that allows you to retrieve all data stored under key.

Parameters

• key (str) – a keyname that contains data stored by write

• client –

  A Redis Client

Returns: None

interpret_read(responses)

Translate Redis data into a local object

Redis will reply to you with something according to what read command you executed in read. This method takes whatever Redis replied with and turns it into an object identical to what was initially passed to write as value.

Parameters

responses – Redis’s reply data based on read method

Returns: An object identical to what was initially written to Redis.