Datasette API

[jeremy-wayland]

Overview

This update enhances the codebase to directly interact with the online version of our dataset, enabling dynamic data retrieval and analysis. Key features include:
• Data Access: Use SQL queries to extract specific HSAs from the Datasette instance and load them locally.
• Network Generation: Create network structures from raw interaction data, facilitating advanced analysis.
• Feature Computation: Compute curvature-filtrations and standard networkx features on the fly.
• Visualization: Explore networks visually, highlighting their structural features.

The Apparent class will serve as the primary interface for users, orchestrating data retrieval, feature computation, analysis, and visualization. Specialized functionality related to network operations will be implemented through dedicated subclasses within the networks submodule.

Structure

├── README.md
├── apparent 
│   ├── __init__.py
│   ├── apparent.py # Main Object to query from Datasette and drive downstream computations!
│   └── networks
│       ├── __init__.py
│       ├── build.py # Methods for building networks. Support default and allow for customisation.
│       ├── cluster.py # Methods for clustering networks.
│       ├── compare.py # Methods for comparing networks.
│       ├── describe.py # Methods for describing networks/computing network features.
│       ├── embed.py # Methods for embedding networks.
├── notebooks
│   └── tutorial.ipynb # Tutorial Notebook on how to use `Apparent`. Maybe can add an advanced network analysis as well.
├── poetry.lock
├── pyproject.toml
├── queries
│   └── sample.sql #Hold some sample queries that will also be available in the Datasette

Tasks

• Restructure Functionality into usable objects
• Introduce an Apparent object that will handle data fetching and drive communcation with networks functionality.

Once Apparent grabs a subset of HSAs, we want to support the following functionality:

• Compute new features of these networks with describe.py (new curvatures, node/edge features etc.)
• Support Embedding methods for networks (inspo is embedding pairwise curvature-filtration distances) in embed.py
• Visualize "space" of networks (2D representation of original embedding)
• Run clustering algorithms on networks to subset further based on structure and visualize :)

[emsimons]

Klingt ganz toll!

File restructuring & naming intuitively makes sense - I think it will be immediately apparent to the user where to find what they are looking for. :)