Data Snack is a minimalistic framework for storing and accessing structured data.
It uses an Entity
objects to define a schema for your data. Snack
provides an interface
for automatically serializing and storing entities in a cache database of you choice.
General interface that allows you to use different backends: redis, memcached.
Entity
objects are stored in a compress form to reduce memory usage.Snack
is usingEntity
fields to define a unique key to represent an object stored in the db.Snack
is supporting batch saving and reading data to achieve high performance.
Entity
- a class defines a schema of single object stored in dbkey fields
- a list of fields (defined as a list ofstr
values) that will be used to create a key for a givenEntity
object.key values
- a list of values forkey fields
from givenEntity
key
- astr
value created for a given Entity- created in a format:
<Entity type name>-<key value 1>_<key value 2>...<key value N>
- created in a format:
Data Snack can be easily installed using pypi repository.
pip install data_snack
This examples shows a basic usage of defining an entity and using Snack
to save and load it from the cache.
More examples can be found in the Examples section.
The first thing you need to do is to define an Entity
.
Entities are used to define a common structure of the objects stores in your database.
We are recommending adding data validation to your entities.
The easiest way is using pydantic
for type validation of all entity fields.
from pydantic.dataclasses import dataclass
from typing import Text
from data_snack.entities import Entity
@dataclass
class Person(Entity):
index: Text
name: Text
Connect to you a cache database of your choice.
In this example we are using Redis
, but you could also use Memcached
if you want.
import redis
redis_connection = redis.Redis(host='127.0.0.1', port=6379, password='')
In this step we create a Snack
instance and connect it to our Redis
database.
Notice, that Redis
client is wrapped in our RedisConnection
class to ensure shared interface.
And at least we can register all entities that will be used in our project.
For each entity we specify a list of fields that will be used to define keys when saving our data.
from data_snack import Snack
from data_snack.connections.redis import RedisConnection
snack = Snack(connection=RedisConnection(redis_connection)) # create instance
snack.register_entity(Person, key_fields=['index']) # register your entity
You are ready to save and load data using Snack
.
snack.set(Person("1", "John"))
# 'Person-1'
entity = snack.get(Person, ["1"])
# Person(index='1', name='John')
snack.set_many([Person("1", "John"), Person("2", "Anna")])
# ['Person-1', 'Person-2']
entities = snack.get_many(Person, [["1"], ["2"]])
# [Person(index='1', name='John'), Person(index='2', name='Anna')]
After you're done with your data you can delete it using Snack
.
snack.delete(Person, ["1"])
# Person(index='1', name='John')
snack.delete_many(Person, [["1"], ["2"]])
# [Person(index='1', name='John'), Person(index='2', name='Anna')]
WIP. Documentation will be hosted on github pages.
Setup documentation directory
mkdir docs
cd docs
Create documentation scaffold. Make sure to select an option with separated directories for source
and build
.
sphinx-quickstart
Update extensions
in docs/source/conf.py
.
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.napoleon']
Before you start make sure to import project src
directory at the very top of docs/source/conf.py
file.
import sys
import os
sys.path.insert(0, os.path.abspath(os.path.join('..', '..', 'src')))
Since documentation uses additional modules (other than base data-snack
), we need to install additional requirements:
pip install -r docs/requirements.txt
Update the scaffold and generate the html docs.
sphinx-apidoc -o ./source ../src/data_snack
make html
Plugin was created by the Data Science team from Webinterpret.