Skip to content

Commit

Permalink
Make minor grammatical and style updates to user guide
Browse files Browse the repository at this point in the history
  • Loading branch information
JeremyMcCormick committed Jul 22, 2024
1 parent cc76deb commit 9d90971
Show file tree
Hide file tree
Showing 8 changed files with 141 additions and 106 deletions.
54 changes: 28 additions & 26 deletions docs/user-guide/databases.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ SQL Databases
Felis can create the corresponding database objects from a schema using the command line tool or the Python API.
This includes the schema itself and all of its tables, columns, indexes, and constraints.
The DDL to perform these actions can either be executed automatically or written to a file for later use.
An existing database can be used or a new one can be created, depending on the options provided.
An existing database can be used or a new one created, depending on the options provided.
When creating a new database rather than updating an existing one, the schema will be instantiated using
``CREATE DATABASE`` in MySQL and ``CREATE SCHEMA`` in PostgreSQL.
The user must have the necessary permissions to create databases in the target database server for this to
Expand Down Expand Up @@ -43,7 +43,7 @@ A specific database dialect such as MySQL can be selected in dry run mode using
felis create --engine-url mysql:// schema.yaml
This will generate the SQL in MySQL format and print it to the console but will not actually create or update
the database.
a database.

The dry run mode may also be explicitly enabled using the ``--dry-run`` option:

Expand All @@ -69,23 +69,23 @@ In order to create a persistent database, the ``--engine-url`` must be set to a
The URL format follows `SQLAlchemy engine conventions <https://docs.sqlalchemy.org/en/20/core/engines.html>`_:
``dialect+driver://username:password@host:port/database``.

Each of the parameters is explained below:
The database URL has the following parameters:

- ``dialect``: The name of the database backend, such as ``sqlite``, ``mysql``, or ``postgresql``. The default is ``sqlite``.
- ``driver``: The name of the DBAPI to use, such as ``mysqlconnector``, or ``psycopg2``. This is optional and the default driver for the dialect will be used if not specified.
- ``username``: The username to use when connecting to the database.
- ``password``: The password to use when connecting to the database.
- ``host``: The host to connect to. Typically, this should be set to ``localhost`` if the database is running on the same machine.
- ``port``: The port to connect to. This will use the default port for the dialect if not specified.
- ``database``: The name of the database to create. For MySQL, this should be left blank. For PostgreSQL, this should be the name of the database in which the schema will be created.
- ``host``: The host for connection. Typically, this should be set to ``localhost`` if the database is running on the same machine.
- ``port``: The port for connection. This will use the default port for the dialect if not specified.
- ``database``: The name of the database to create. For MySQL, this should typically be left blank, and Felis will use the name of the schema for the database. For a PostgreSQL connection, this should be the name of the database in which the schema will be created.

The database URL may also be set using the ``FELIS_ENGINE_URL`` environment variable, in which case the
``--engine-url`` option can be omitted.

MySQL
^^^^^

To create a MySQL database from a schema file, the command would look like this:
To create a MySQL database from a schema file, the command would look similar to the following:

.. code-block:: bash
Expand All @@ -104,8 +104,8 @@ dialect:
felis create --engine-url postgresql+psycopg2://username:password@localhost/database schema.yaml
Felis can be used to create the schema, but it cannot create the database itself, which must be included as
part of the URL.
This must be done beforehand using the ``CREATE DATABASE`` command in the PostgreSQL client.
part of the URL; or the command will fail.
The database must be created beforehand using the ``CREATE DATABASE`` command in the PostgreSQL client.

SQLite
^^^^^^
Expand All @@ -116,49 +116,53 @@ To persist a SQLite database, first create an empty database on disk as follows:
sqlite3 /tmp/my.db "VACUUM;"
The database can then be created from a schema file:
Installation of SQLite is not covered; please refer to the `SQLite documentation <https://www.sqlite.org>`_ for more information.

The database objects can then be instantiated from a schema file:

.. code-block:: bash
felis create --engine-url sqlite:////tmp/my.db schema.yaml
After it has been created, uou may open the database file with a SQLite client to inspect the schema as
follows:
After it has been created, you may open the database file with a SQLite client to inspect the schema:

.. code-block:: bash
sqlite3 /tmp/my.db
To show the instantiated tables from the SQLite client, use the following command:
To show the tables which were instantiated, use the following command from within the SQLite client:

..
.tables

SQLite will ignore the name of the schema, as it does not support named schemas or databases internally.
SQLite will ignore the name of the schema, as it does not support named schemas or databases.

Creating a New Database
-----------------------

Felis can also be used to create the database itself, rather than use an existing one, by using the
``--create-if-not-exists`` option:
``--create-if-not-exists`` option.

Here is an example of creating a new MySQL database:

.. code-block:: bash
felis create --engine-url mysql+mysqlconnector://username:password@localhost --create-if-not-exists schema.yaml
Felis can also drop an existing database first and recreate it:
Felis can also drop an existing database first and then recreate it:

.. code-block:: bash
felis create --engine-url mysql+mysqlconnector://username:password@localhost --drop-if-exists schema.yaml
The commands to create or drop databases will require that the database user has the necessary permissions.
The commands to create or drop databases will require that the database user has the necessary permissions on
the server.

Using a Different Schema Name
-----------------------------

Typically, the name of the schema in the database will be the same as the ``name`` field in the YAML file, but
The name of the schema in the database will by default be the same as the ``name`` field in the YAML file, but
this can be overridden using the ``--schema-name`` option:

.. code-block:: bash
Expand All @@ -174,25 +178,23 @@ The Python API can also be used to create a database from a schema.
First, the schema object should be loaded from a YAML file, following the instructions in
:ref:`validating-with-python-api`.

Once the schema object has been created after being successfuly validated, the builder can be used to create
the SQLAlchemy metadata object:
Once the schema object has been successfuly created, the builder can be used to create the SQLAlchemy metadata
object:

.. code-block:: python
from felis.metadata import MetaDataBuilder
metadata = MetaDataBuilder(schema).build()
The metadata object can be used to create the database using standard SQLAlchemy commands.
For example, to create an in-memory SQLite database:
For example, the following command will create an in-memory SQLite database in Python:

.. code-block:: python
from sqlalchemy import create_engine
engine = create_engine("sqlite:///:memory:")
metadata.create_all(engine)
The above command will just create an in-memory SQLite database and will not persist the database to disk.

To create a MySQL database, the engine URL should be changed to something like this:

.. code-block:: python
Expand All @@ -203,7 +205,7 @@ To create a MySQL database, the engine URL should be changed to something like t
The database will then be created on the MySQL server at ``localhost``.

Felis also provides the `DatabaseContext class <../dev/internals/felis.db.utils.DatabaseContext.html>`_ which
can be used to create a database from a schema.
supports creation of the database or schema itself:

.. code-block:: python
Expand All @@ -213,5 +215,5 @@ can be used to create a database from a schema.
ctx.create_all()
An advantage of using this class is that it can automatically handle the creation of the database if it does
not already exist with the ``create_if_not_exists`` method or drop and recreate the database with the
``drop_and_create`` method.
not already exist with the ``create_if_not_exists`` method; an existing database may also be dropped and
recreated using the ``drop_and_create`` method.
8 changes: 4 additions & 4 deletions docs/user-guide/datatypes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
Data Types
##########

Felis has an internal type system that is used to define the data types of columns in a schema.
These are mapped to the appropriate SQL data types when the schema is created in a database and the
corresponding `VOTable <https://ivoa.net/documents/VOTable/>`__ data type when writing information to
Felis provides an internal type system that is used to define the data types of columns in a schema.
These data types are mapped to the appropriate SQL types when instantiating a database and the corresponding
`VOTable <https://ivoa.net/documents/VOTable/>`__ data types when writing information to
`TAP_SCHEMA <https://www.ivoa.net/documents/TAP/20190927/REC-TAP-1.1.html#tth_sEc4>`_.

The following values are supported for the
The following values are supported by the
`datatype <../dev/internals/felis.datamodel.Column.html#felis.datamodel.Column.datatype>`_ field of a column
object:

Expand Down
32 changes: 20 additions & 12 deletions docs/user-guide/examples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,10 @@
Examples
########

The `SDM Schemas github repository <https://github.com/lsst/sdm_schemas>`_ contains Felis schema files used by the Rubin Observatory, which will be used as examples to illustrate the features of Felis.
The following is an excerpt from the `DP0.2 DC2 schema <https://github.com/lsst/sdm_schemas/blob/main/yml/dp02_dc2.yaml>`_:
The `SDM Schemas github repository <https://github.com/lsst/sdm_schemas>`_ contains Felis schema files used by
the Rubin Observatory, which will be used as examples to illustrate the features of Felis.
The following is an excerpt from the
`DP0.2 DC2 schema <https://github.com/lsst/sdm_schemas/blob/main/yml/dp02_dc2.yaml>`_:

.. code-block:: yaml
:linenos:
Expand Down Expand Up @@ -33,22 +35,27 @@ The following is an excerpt from the `DP0.2 DC2 schema <https://github.com/lsst/
ivoa:ucd: pos.eq.ra;meta.main
Lines 2-6 define the schema name, id, and description.
Name and id are always required.
Name and id are required for all objects in Felis schemas.
The description is optional but highly recommended for documentation purposes.

Next is a list of table definitions, starting with the Object table on line 18.
Each table definition includes a name, id, description, and a list of columns, and may also include TAP-specific metadata.

Each table defines one or more columns, starting with the objectId column on line 23.
A column must have a name, id, and datatype, and an optional (but highly recommended to include) description.
The `Column <../dev/internals/felis.datamodel.Column.html>`_ class provides a full list of available fields, including TAP and VOTable-specific metadata.
A table is comprised of one or more columns which must must have a name, id, and datatype, and an optional
(but highly recommended to include) description.
The `Column <../dev/internals/felis.datamodel.Column.html>`_ class provides a full list of available fields,
including TAP and VOTable-specific metadata.

The second exerted column is coord_ra, which is a measurement field including units of measurement.

Both fields shown here have an `IVOA UCD <https://www.ivoa.net/documents/cover/UCD-20050812.html>`_ field, which is a "vocabulary for describing astrononomical data quantities," describing the semantics of the fields.
Both fields shown here have an `IVOA UCD <https://www.ivoa.net/documents/cover/UCD-20050812.html>`_ field,
which is a "vocabulary for describing astrononomical data quantities," describing the semantics of the fields.
The first column in the Object table is ``objectId``, which is a long integer field defining a unique identifier
for records in the table.
The ``meta.id`` word in the column's UCD flags the field semantically as an identifier.
The second exerted column is ``coord_ra``, which is a measurement field including units of measurement.

Felis also supports table constraints, such as foreign keys.
The DP0.2 DC2 schema includes a foreign key constraint on the ccdVisitId field of the Source table, defined as follows:
The `DP0.2 DC2 schema <https://github.com/lsst/sdm_schemas/blob/main/yml/dp02_dc2.yaml>`_ includes a foreign
key constraint on the ``ccdVisitId`` field of the ``Source`` table, defined as follows:

.. code-block:: yaml
:linenos:
Expand All @@ -63,6 +70,7 @@ The DP0.2 DC2 schema includes a foreign key constraint on the ccdVisitId field o
referencedColumns:
- "#CcdVisit.ccdVisitId"
The ccdVisitId field in the Source table is linked to the ccdVisitId field in the CcdVisit table.
The ``ccdVisitId`` field in the ``Source`` table is linked to the ``ccdVisitId`` field in the ``CcdVisit``
table.

Felis supports many additional features. Refer to the `model documentation <model>`_ for a complete list.
Felis schemas support many additional features. Refer to the `model documentation <model>`_ for a complete list.
13 changes: 7 additions & 6 deletions docs/user-guide/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,31 +3,32 @@ Installation
############

Felis is deployed to PyPI on a weekly release schedule.

To install the latest release, use the following command:

.. code-block:: bash
pip install lsst-felis
To install a specific release, use a command similar to the following:
A specific release can be installed using a command similar to the following:

.. code-block:: bash
pip install 'lsst-felis==27.2024.2700'
This is just an example. If you want to install a different version, replace the version number from above
with the one you want to install.
This is just an example.
If you want to install a different version, replace the version number from above.

To install Felis from the Github main branch, use the following command:
To install Felis directly from the Github ``main`` branch, use the following command:

.. code-block:: bash
pip install 'lsst-felis @ git+http://github.com/lsst/felis.git@main'
To depend on Felis in your project, add the following to your ``requirements.txt`` file:
Felis may also be added as a dependency of your Python project by putting the following in ``requirements.txt``:

.. code-block:: text
lsst-felis
Felis requires Python 3.11 or later and is tested on Linux, macOS, and Windows WSL.
Felis requires Python 3.11 or later and is regularly tested on Linux, macOS, and Windows WSL.
22 changes: 11 additions & 11 deletions docs/user-guide/intro.rst
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
Introduction
------------

Data catalogs are a fundamental part of modern astronomical research.
They provide a means to describe the data that is available, and to search for data that matches certain
criteria.
Tabular data models are a common way to represent such catalogs.
A canonical format for describing these types of models is SQL Data Definition Language (DDL).
Data catalogs are a fundamental part of modern astronomical research, providing a means to describe available
data and search for data matching certain criteria.
Tabular data models are a common way to represent such catalogs, and a canonical format for describing them is
SQL Data Definition Language (DDL).
However, DDL does not provide a way to describe the semantics of the data, such as the meaning of each column,
the units of measurement, or the relationships between tables.
Felis provides a way to describe these semantics in a user-friendly YAML format.

Within astronomy, the `International Virtual Observatory Alliance <https://ivoa.net/>`__ (IVOA) has developed
a standard for describing tabular data models called
`Table Access Protocol <https://www.ivoa.net/documents/TAP/>`__ (TAP).
Metadata for a specific TAP service is typically provided in a schema called TAP_SCHEMA that describes the
tables and columns available in the service.
Metadata for a specific TAP service is typically provided in a schema called TAP_SCHEMA describing tables and
columns available in the service.
Felis provides a tool for translating its schema representation into TAP_SCHEMA, making a catalog's metadata
available through a standard TAP service.
Compatible TAP services can use this data to populate their ``/tables`` output as well.
Compatible TAP services can use this data to populate their ``/tables`` output.

Felis also provides a mechanism for instantiating an empty catalog from its schema into relational database
objects such as tables and columns.
Supported databases include SQLite, MySQL/MariaDB, and PostgreSQL.
Felis also provides a mechanism for instantiating a catalog from its schema representation into an (empty)
relational database including the table, constraint and column definitions.
This can be done using the command line interface or the Python API.
Supported databases include SQLite, MySQL/MariaDB, and PostgreSQL.

Loading

0 comments on commit 9d90971

Please sign in to comment.