nosqlapi is a library for building standard NOSQL python libraries.
Full documentation: Read the docs
This library is defined to encourage similarity between Python modules used to access NOSQL databases. In this way, I hope for consistency that leads to more easily understood modules, code that generally gets more portability across databases and a broader scope of database connectivity from Python.
This document describes the Python NOSQL database API specification.
Access to the database is made available through connection objects. The module must provide the following constructor for these:
Connection(parameters...)
Constructor for creating a connection to the database.
This object has a connect method that returns a Session object. It takes a number of parameters which are database dependent.
api_level
String constant stating the supported DB API level.
Currently, only the strings "1.0".
CONNECTION
Connection object global variable.
SESSION
Session object global variable.
Error
Exception that is the base class of all other error exceptions. You can use this to catch all errors with one single except statement.
UnknownError
Exception raised when an unspecified error occurred.
It must be a subclass of Error.
ConnectError
Exception raised for errors that are related to the database connection.
It must be a subclass of Error.
CloseError
Exception raised for errors that are related to the database close connection.
It must be a subclass of Error.
DatabaseError
Exception raised for errors that are related to the database, generally.
It must be a subclass of Error.
DatabaseCreationError
Exception raised for errors that are related to the creation of a database.
It must be a subclass of DatabaseError.
DatabaseDeletionError
Exception raised for errors that are related to the deletion of a database.
It must be a subclass of DatabaseError.
SessionError
Exception raised for errors that are related to the session, generally.
It must be a subclass of Error.
SessionInsertingError
Exception raised for errors that are related to the inserting data on a database session.
It must be a subclass of SessionError.
SessionUpdatingError
Exception raised for errors that are related to the updating data on a database session.
It must be a subclass of SessionError.
SessionDeletingError
Exception raised for errors that are related to the deletion data on a database session.
It must be a subclass of SessionError.
SessionClosingError
Exception raised for errors that are related to the closing database session.
It must be a subclass of SessionError.
SessionFindingError
Exception raised for errors that are related to the finding data on a database session.
It must be a subclass of SessionError.
SessionACLError
Exception raised for errors that are related to the grant or revoke permission on a database.
It must be a subclass of SessionError.
SelectorError
Exception raised for errors that are related to the selectors in general.
It must be a subclass of Error.
SelectorAttributeError
Exception raised for errors that are related to the selectors attribute.
It must be a subclass of SelectorError.
This is the exception inheritance layout:
Exception
|__Error
|__UnknownError
|__ConnectError
|__CloseError
|__DatabaseError
| |__DatabaseCreationError
| |__DatabaseDeletionError
|__SessionError
| |__SessionInsertingError
| |__SessionUpdatingError
| |__SessionDeletingError
| |__SessionClosingError
| |__SessionFindingError
| |__SessionACLError
|__SelectorError
|__SelectorAttributeError
Connection objects should respond to the following methods.
.connected
This read-only attribute contains a boolean value.
.close(parameters...)
Closing the connection now.
.connect(parameters...)
Connecting to database with the arguments when object has been instantiated and create a Session object to database.
.create_database(parameters...)
Creating a single database with position and keyword arguments.
.has_database(parameters...)
Checking if exists a single database with position and keyword arguments.
.delete_database(parameters...)
Deleting of a single database with position and keyword arguments.
.databases(parameters...)
List all databases.
.show_database(parameters...)
Show an information of a specific database
Session objects should respond to the following methods.
ATTENTION: Session object it will come instantiated if the
connectionvalue contains a compliant API Connection object.databasevalue is optional.
.connection
This read-only attribute contains the connection object or other object to serve connection (like sockets).
.description
This read-only attribute contains the session parameters (can be string, list or dictionary).
.item_count
This read-only attribute contains the number of object returned of an operations.
.database
This read-only attribute contains the name of database in current session.
.acl
This read-only attribute contains the Access Control List in the current session.
.indexes
This read-only attribute contains the name of indexes of the current database.
.get(parameters...)
Getting one or more data on specific database with position and keyword arguments.
.insert(parameters...)
Inserting one data on specific database with position and keyword arguments.
.insert_many(parameters...)
Inserting one or more data on specific database with position and keyword arguments.
.update(parameters...)
Updating one existing data on specific database with position and keyword arguments.
.update_many(parameters...)
Updating one or more existing data on specific database with position and keyword arguments.
.delete(parameters...)
Deleting one existing data on specific database with position and keyword arguments.
.close(parameters...)
Closing the session and connection now.
.find(parameters...)
Finding data on specific database with string selector or Selector object with position and keyword arguments.
.grant(parameters...)
Granting ACL on specific database with position and keyword arguments.
.revoke(parameters...)
Revoking ACL on specific database with position and keyword arguments.
.new_user(parameters...)
Creating new normal or admin user.
.set_user(parameters...)
Modifying exists user or reset password.
.delete_user(parameters...)
Deleting exists user.
.add_index(parameters...)
Adding a new index to database.
.delete_index(parameters...)
Deleting exists index to database.
.call(batch, parameters...)
Calling a batch object to execute one or more statement.
Selector objects should respond to the following methods.
.selector
This read/write attribute represents the selector key/value than you want search.
.fields
This read/write attribute represents the fields key that returned from find operations.
.partition
This read/write attribute represents the name of partition/collection in a database.
.condition
This read/write attribute represents other condition to apply a selectors.
.order
This read/write attribute represents order returned from find operations.
.limit
This read/write attribute represents limit number of objects returned from find operations.
.build(parameters...)
Building a selector string in the dialect of a NOSQL language based on various property of the Selector object.
Response objects should respond to the following attributes.
Responseobjects is a species of an either-data type, because contains both success and error values
.data
This read-only attribute represents the effective data than returned (Any object).
.code
This read-only attribute represents a number code of error or success in an operation.
.header
This read-only attribute represents a string information (header) of an operation.
.error
This read-only attribute that throw an Exception if it has been set.
.dict
This read-only attribute represents a dictionary transformation of Response object.
.throw()
Raise exception stored in error property.
Batch objects should respond to the following methods.
.session
This read/write attribute represents a Session object.
.batch
This read/write attribute represents a batch operation.
.execute(parameters...)
Executing a batch operation with position and keyword arguments.
Each database type may contain unique method names for the database type itself. For example, in the abstract class
nosqlapi.docdb.DocumentConnectionthere is thecopy_databasemethod which is not present in the other types. These characteristics distinguish the substantial differences between the four databases.
The package nosqlapi is a collection of interface and utility class and functions for build your own NOSQL python package.
To test this package.
$ git clone https://github.com/MatteoGuadrini/nosqlapi.git
$ cd nosqlapi
$ python -m unittest discover testsInstead, to install package.
$ pip install nosqlapi #from pypi
$ git clone https://github.com/MatteoGuadrini/nosqlapi.git #from official repo
$ cd nosqlapi
$ python setup.py installNoSql databases are of four types:
- Key/Value database
- Column database
- Document database
- Graph database
For each type of database, nosqlapi offers standard interfaces, in order to unify as much as possible the names of methods and functions.
For an example, just look at the relevant package test files.
Look as each test module has same class and each class has same method name. For instance, look at MyDBSession class as inherit for each nosqlapi type of abstract class:
$ grep "class MyDBSession" tests/*
tests/test_columndb.py:class MyDBSession(nosqlapi.columndb.ColumnSession):
tests/test_docdb.py:class MyDBSession(nosqlapi.docdb.DocSession):
tests/test_graphdb.py:class MyDBSession(nosqlapi.graphdb.GraphSession):
tests/test_kvdb.py:class MyDBSession(nosqlapi.kvdb.KVSession):A key–value database, or key–value store, is a data storage paradigm designed for storing, retrieving, and managing associative arrays, and a data structure more commonly known today as a dictionary or hash table. Dictionaries contain a collection of objects, or records, which in turn have many fields within them, each containing data. These records are stored and retrieved using a key that uniquely identifies the record, and is used to find the data within the database.
import nosqlapi.kvdb
# Redis like database
class RedisConnection(nosqlapi.kvdb.KVConnection):...
class RedisSession(nosqlapi.kvdb.KVSession):...
class RedisResponse(nosqlapi.kvdb.KVResponse):...
class RedisBatch(nosqlapi.kvdb.KVBatch):...
class RedisSelector(nosqlapi.kvdb.KVSelector):...
# Use Redis library
conn = RedisConnection(host='server.local', username='admin', password='pass', database='db')
sess = conn.connect() # return RedisSession object
# Create a new database
conn.create_database('new_db')
# CRUD operation
C = sess.insert('key', 'value') # Create
R = sess.get('key') # Read
U = sess.update('key', 'new_value') # Update
D = sess.delete('key') # Delete
print(R) # {'key': 'value'}
print(type(R)) # <class 'RedisResponse'>
print(isinstance(R, nosqlapi.Response)) # True
# Extended CRUD operations
sess.insert_many({'key1': 'value1', 'Key2': 'value2'})
sess.update_many({'key1': 'new_value1', 'Key2': 'new_value2'})
# Complex select operation
sel = RedisSelector(selector='key:value id:1 ttl:300', limit=2)
sess.find(sel)
# Batch operations
op = 'SET hello "Hello"\nSET mykey "new"\nGET mykey\nSET anotherkey "will expire in a minute" EX 60'
batch = RedisBatch(batch=op, session=sess)
resp = batch.execute()
print(resp) # ('OK', 'OK', {'mykey': 'new'}, 'OK')
print(type(resp)) # <class 'RedisResponse'>A column-oriented DBMS or columnar DBMS is a database management system (DBMS) that stores data tables by column rather than by row. Practical use of a column store versus a row store differs little in the relational DBMS world. Both columnar and row databases can use traditional database query languages like SQL to load data and perform queries. Both row and columnar databases can become the backbone in a system to serve data for common extract, transform, load (ETL) and data visualization tools. However, by storing data in columns rather than rows, the database can more precisely access the data it needs to answer a query rather than scanning and discarding unwanted data in rows.
import nosqlapi.columndb
# Cassandra like database
class CassandraConnection(nosqlapi.columndb.ColumnConnection):...
class CassandraSession(nosqlapi.columndb.ColumnSession):...
class CassandraResponse(nosqlapi.columndb.ColumnResponse):...
class CassandraBatch(nosqlapi.columndb.ColumnBatch):...
class CassandraSelector(nosqlapi.columndb.ColumnSelector):...
# Use Cassandra library
conn = CassandraConnection(host='server.local', username='admin', password='pass', database='db')
sess = conn.connect() # return CassandraSession object
# Create a new database
conn.create_database('new_db')
# CRUD operation
C = sess.insert(table='hitchhikers', columns=('id', 'name', 'age'), values=(1, 'Arthur', 42)) # Create
R = sess.get(table='hitchhikers', columns=('id', 'name', 'age')) # Read
U = sess.update(table='hitchhikers', columns=('id', 'name', 'age'), values=(1, 'Arthur', 43)) # Update
D = sess.delete(table='hitchhikers', conditions=["name='Arthur'", 'age=43']) # Delete
print(R) # [(1, 'Arthur', 42)]
print(type(R)) # <class 'CassandraResponse'>
print(isinstance(R, nosqlapi.Response)) # True
# Extended CRUD operations
sess.insert_many(table='hitchhikers', columns=('id', 'name', 'age'), values=[(1, 'Arthur', 42), (2, 'Arthur', 43)])
sess.update_many(table='hitchhikers', columns=('id', 'name', 'age'), values=[(1, 'Arthur', 44), (2, 'Arthur', 45)])
# Complex select operation
sel = CassandraSelector()
# Table
sel.selector = 'hitchhikers'
# Columns
sel.fields = ['id', 'name', 'age']
sess.find(sel)
# Batch operations
op = "BEGIN BATCH\nINSERT INTO hitchhikers (id, name, age)\n VALUES (1, 'Arthur', 42)\nIF NOT EXISTS;\nAPPLY BATCH;"
batch = CassandraBatch(batch=op, session=sess)
resp = batch.execute()
print(resp) # [(1, 'Arthur', 42)]
print(type(resp)) # <class 'CassandraResponse'>A document-oriented database, or document store, is a computer program and data storage system designed for storing, retrieving and managing document-oriented information, also known as semi-structured data.
Document-oriented databases are one of the main categories of NoSQL databases, and the popularity of the term "document-oriented database" has grown with the use of the term NoSQL itself. Graph databases are similar, but add another layer, the relationship, which allows them to link documents for rapid traversal.
import nosqlapi.docdb
# MongoDB like database
class MongoConnection(nosqlapi.docdb.DocConnection):...
class MongoSession(nosqlapi.docdb.DocSession):...
class MongoResponse(nosqlapi.docdb.DocResponse):...
class MongoBatch(nosqlapi.docdb.DocBatch):...
class MongoSelector(nosqlapi.docdb.DocSelector):...
# Use MongoDB library
conn = MongoConnection(host='server.local', username='admin', password='pass')
sess = conn.connect() # return MongoSession object
# Create a new database
conn.create_database('new_db')
# CRUD operation
C = sess.insert(path='db/doc1', doc={"_id": "5099803df3f4948bd2f98391", "name": "Arthur", "age": 42}) # Create
R = sess.get(path='db/doc1') # Read
U = sess.update(path='db/doc1', doc={"_id": "5099803df3f4948bd2f98391", "name": "Arthur", "age": 43}, rev=2) # Update
D = sess.delete(path='db/doc1', rev=2) # Delete
print(R) # {"_id": "5099803df3f4948bd2f98391", "rev"= 2, "name": "Arthur", "age": 42}
print(type(R)) # <class 'MongoResponse'>
print(isinstance(R, nosqlapi.Response)) # True
# Extended CRUD operations
sess.insert_many(database='db', docs=[{"_id": "5099803df3f4948bd2f98391", "name": "Arthur", "age": 42},
{"_id": "5099803df3f4948bd2f98392", "name": "Arthur", "age": 43}])
sess.update_many(database='db', docs=[{"_id": "5099803df3f4948bd2f98391", "name": "Arthur", "age": 42, "rev": 2},
{"_id": "5099803df3f4948bd2f98392", "name": "Arthur", "age": 43, "rev": 2}])
# Complex select operation
sel = MongoSelector(selector={"name": "Arthur"}, fields=['_id', 'name', 'age'], limit=2)
sess.find(sel)
# Batch operations
op = [{"_id": "5099803df3f4948bd2f98391", "name": "Arthur", "age": 42}, {"_id": "5099803df3f4948bd2f98392", "name": "Arthur", "age": 43}]
batch = MongoBatch(batch=op, session=sess)
resp = batch.execute(crud='insert')
print(resp) # [{"_id": "5099803df3f4948bd2f98391", "rev"= 2, "name": "Arthur", "age": 42}, {"_id": "5099803df3f4948bd2f98392", "rev"= 2, "name": "Arthur", "age": 43}]
print(type(resp)) # <class 'MongoResponse'>Graph databases are a type of NoSQL database, created to address the limitations of relational databases. While the graph model explicitly lays out the dependencies between nodes of data, the relational model and other NoSQL database models link the data by implicit connections. In other words, relationships are a first-class citizen in a graph database and can be labelled, directed, and given properties. This is compared to relational approaches where these relationships are implied and must be reified at run-time. Graph databases are similar to 1970s network model databases in that both represent general graphs, but network-model databases operate at a lower level of abstraction and lack easy traversal over a chain of edges.
import nosqlapi.graphdb
# Neo4j like database
class Neo4jConnection(nosqlapi.graphdb.GraphConnection):...
class Neo4jSession(nosqlapi.graphdb.GraphSession):...
class Neo4jResponse(nosqlapi.graphdb.GraphResponse):...
class Neo4jBatch(nosqlapi.graphdb.GraphBatch):...
class Neo4jSelector(nosqlapi.graphdb.GraphSelector):...
# Use Neo4j library
conn = Neo4jConnection(host='server.local', username='admin', password='pass', database='db')
sess = conn.connect() # return Neo4jSession object
# Create a new database
conn.create_database('new_db')
# CRUD operation
C = sess.insert(node='n:Person', properties={'name': 'Arthur', 'age': 42}) # Create
R = sess.get(node='n:Person') # Read
U = sess.update(node='n:Person', properties={'name': 'Arthur', 'age': 42}) # Update
D = sess.delete(node='n:Person') # Delete
print(R) # {'n.name': 'Arthur', 'n.age': 42}
print(type(R)) # <class 'Neo4jResponse'>
print(isinstance(R, nosqlapi.Response)) # True
# Extended CRUD operations
sess.insert_many(nodes=['matteo:Person', 'arthur:Person'], properties=[{'name': 'Matteo', 'age': 35},
{'name': 'Arthur', 'age': 42}])
sess.update_many(nodes=['matteo:Person', 'arthur:Person'], properties=[{'name': 'Matteo', 'age': 35},
{'name': 'Arthur', 'age': 42}])
sess.link(node='arthur:Person{name: "Arthur"}', to_link='book:hitchhikers', relationship=':ACT_IN')
sess.detach(node='arthur:Person{name: "Arthur"}')
# Complex select operation
sel = Neo4jSelector(selector='people:Person', fields=['name', 'age'], condition='people.age>=35', order='age', limit=2)
sess.find(sel)
# Batch operations
op = "MATCH (p:Person {name: 'Arhur'})-[rel:ACT_IN]-(:Book {name: 'hitchhikers'})\nSET rel.startYear = date({year: 2018})\nRETURN p"
batch = Neo4jBatch(batch=op, session=sess)
resp = batch.execute()
print(resp) # {'p.name': 'Arhur', 'p.age': 42}
print(type(resp)) # <class 'Neo4jResponse'>For each type of NOSQL database there is an ODM (Object-relational mapping) module that contains classes and functions relating to the mapping of objects and/or operations concerning the specific database CRUD operation.
In the nosqlapi.common.odm module there are also classes that represent the data types of databases.
>>> import nosqlapi.common.odm
>>> [t for t in dir(nosqlapi.common.odm) if not t.startswith('__')]
['Array', 'Ascii', 'Blob', 'Boolean', 'Counter', 'Date', 'Dc', 'Decimal', 'Double', 'Duration', 'Float', 'Inet', 'Int',
'List', 'Map', 'Null', 'SmallInt', 'Text', 'Time', 'Timestamp', 'Uuid', 'Varchar']The package also comes with useful classes and functions to help migrate a library to these APIs. Besides, these there are also some utilities for end users.
import nosqlapi
from pymongo import Connection
# This decorator allows you to map existing method names to API compliant methods.
@nosqlapi.api(database_names='databases', drop_database='delete_database', close_cursor='close')
class ApiConnection(Connection):
pass
conn = ApiConnection('localhost', 27017, 'test_database')
hasattr(conn, 'databases') # True
conn.databases() # (test_database, 'db1', 'db2')import nosqlapi
from neo4j import Neo4jConnection
from mymongo import MongoConnection
# Create manager for session API
man = nosqlapi.Manager(Neo4jConnection(host='server.local', username='admin', password='pass', database='db'))
print(type(man)) # <class 'Manager'>
print(man) # database=db, description=('db', 'online')
# CRUD operation
C = man.insert(node='n:Person', properties={'name': 'Arthur', 'age': 42}) # Create
R = man.get(node='n:Person') # Read
U = man.update(node='n:Person', properties={'name': 'Arthur', 'age': 42}) # Update
D = man.delete(node='n:Person') # Delete
# Change session
man.change(MongoConnection(host='server.local', username='admin', password='pass', database='db'))
C = man.insert(path='db/doc1', doc={"_id": "5099803df3f4948bd2f98391", "name": "Arthur", "age": 42}) # Create
R = man.get(path='db/doc1') # Read
U = man.update(path='db/doc1', doc={"_id": "5099803df3f4948bd2f98391", "name": "Arthur", "age": 43}, rev=2) # Update
D = man.delete(path='db/doc1', rev=2) # Deletenosqlapi is an open source project. Any contribute, It's welcome.
A great thanks.
For donations, press this
For me
For Telethon
The Telethon Foundation is a non-profit organization recognized by the Ministry of University and Scientific and Technological Research. They were born in 1990 to respond to the appeal of patients suffering from rare diseases. Come today, we are organized to dare to listen to them and answers, every day of the year.
This package is Treeware. If you use it in production, then we ask that you buy the world a tree to thank us for our work. By contributing to the Treeware forest you’ll be creating employment for local families and restoring wildlife habitats.
Thanks to Mark Lutz for writing the Learning Python and Programming Python books that make up my python foundation.
Thanks to Kenneth Reitz and Tanya Schlusser for writing the The Hitchhiker’s Guide to Python books.
Thanks to Dane Hillard for writing the Practices of the Python Pro books.
Special thanks go to my wife, who understood the hours of absence for this development. Thanks to my children, for the daily inspiration they give me and to make me realize, that life must be simple.
Thanks Python!