Merge pull request #167 from mgckind/1.4.6

1.4.6

AUTHORS.md

@@ -0,0 +1,10 @@
+##  Contributors to `easyaccess`
+This is an alphabetical list of contributors, if you feel you have contributed to the project and we have neglected adding you to the list please accept our apologizes and [let us know](mailto:[email protected]) to correct it.
+
+- Matias Carrasco Kind
+- Alex Drlica-Wagner
+- Landon Gelman
+- Audrey Koziol
+- Donald Petravick
+- Eli Rykoff
+- Ignacio Sevilla

CHANGES.md

@@ -1,11 +1,22 @@
 # Changes
 
+## v1.4.6
+#### 2019-JAN-14
+- No minor changes to the code, just the general documentation
+- Adding Contribution guidelines and Coda of Conduct
+- Adding Authors
+- Adding FAQ
+- Updating Readme and Installation instructions
+- Reformatting code
+- Update paper and references
+
+
 ## v1.4.5
 #### 2018-OCT-04
 - Restructuring code structure (PR #151)
 - Add paper reference
 - Minor fixes in unitests
-- Update documentation 
+- Update documentation
 - Fix conda issues
 
 ## v1.4.4

CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at [email protected]. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

CONTRIBUTING.md

@@ -0,0 +1,40 @@
+# Contributing to `easyaccess`
+
+
+When contributing to this repository, please first discuss the change you wish to make via [issues](https://github.com/mgckind/easyaccess/issues),
+[email](mailto:[email protected]), or any other method with the owners of this repository before making a change.
+
+Please note we have a [code of conduct](CODE_OF_CONDUCT.md) for this project, please follow it in all your interactions with the project.
+
+## Reporting Bugs
+
+When creating a bug or an issue please include as many details as possible. Create your issues [here](https://github.com/mgckind/easyaccess/issues) and follow this steps:
+
+- Check [previous closed issues](https://github.com/mgckind/easyaccess/issues?q=is%3Aissue+is%3Aclosed) , [current open issues](https://github.com/mgckind/easyaccess/issues?q=is%3Aopen+is%3Aissue) or our FAQ for a list of common questions or previously solved problems
+- Use a clear and descriptive title for the issue to identify the problem.
+- Provide all information about OS
+- Provide information about the installation process (source, pip, conda, Docker, etc)
+- software version including Python
+- Detailed information on how to reproduce the problem
+- Provide screenshots when useful
+
+## Other contributions
+
+All kind of contributions are welcome, from fixing bugs, resolving issues or suggesting features. Look at the [current issues](https://github.com/mgckind/easyaccess/issues) to see how you can contribute and provide help.
+
+## Pull Request Process
+
+1. Ensure any install or build dependencies are removed before the end of the layer when doing a
+   build.
+2. Update the README.md with details of changes to the interface, this includes new environment
+   variables, exposed ports, useful file locations and container parameters.
+3. Increase the version numbers in any examples files and the README.md to the new version that this
+   Pull Request would represent. The versioning scheme we use is [SemVer](http://semver.org/).
+4. You may merge the Pull Request in once you have the sign-off of two other developers, or if you
+   do not have permission to do that, you may request the second reviewer to merge it for you.
+
+## Authors
+
+Please see our [list of contributors](AUTHORS.md), if you feel you have contributed to the project and we have neglected adding you to the list please accept our apologizes and [let us know](mailto:[email protected]) to correct it.
+
+

FAQ.md

@@ -0,0 +1,22 @@
+## Frequently Asked Questions
+
+Here we collect a list of FAQ related to installation, usage, etc. Please refre to the [issues page](https://github.com/mgckind/easyaccess/issues) for more information.
+
+- The installation with `pip` went smoothly but it was not obvious where "easyaccess" was installed.
+    - When installed using pip, one can use the following: `pip show -f easyaccess` to check the installation folders
+
+- When trying to connect I keep getting this error `ORA-21561: OID generation failed`. Any idea how to solve it?
+    - Most of the time this problem can be solved by adding the name of your computer in the `/etc/hosts` file, next to the line that says 127.0.0.1 localhost. Just add the name of your computer (type `hostname`) to that line, so it looks like `127.0.0.1 localhost my-computer`
+- Where is the configuration file and authentication file?
+    - Usually the configuration file by default is at `$HOME/.easyaccess/config.ini`  and the authentication file is at `$HOME/.desservices.ini` or can be set at the env variable `$DES_SERVICES`
+- I am a DES Collaborator, where can I reset my credentials?
+  - For collaborators only: Please use [this](https://deslogin.wufoo.com/forms/help-me-with-my-desdm-account/) form
+- The client hangs after getting an Oracle error and I need to close to the window.
+  - This is a long-standing [issue](https://github.com/mgckind/easyaccess/issues/130) but we haven't been able to fix it or to reproduce it in all the systems. Please report it there and add details about OS and versions. We'd recommend a clean conda installation which sometimes fixes the issue.
+- How can I install the Oracle client by myself.
+  - Please follow the instructions posted [here](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html)
+- Are there other requirements besides python modules and Oracle clients?
+  - Usually not, for new OS or Virtual Machines you'd need to install `libaio` and `libbz2`
+- How can I contribute to the project?
+  - Please take a look st our [Code of Conduct](CODE_OF_CONDUCT.md) and or [contribution guide](CONTRIBUTING.md)
+

README.md

@@ -1,15 +1,16 @@
-# easyaccess <a href="https://github.com/mgckind/easyaccess/releases/tag/1.4.5"> <img src="https://img.shields.io/badge/release-v1.4.5-blue.svg" alt="latest release" /></a> <a href="https://github.com/mgckind/easyaccess/blob/master/LICENSE.txt"><img src="https://img.shields.io/badge/license-NCSA%20License-blue.svg" alt="License" /> </a> <a href="https://pypi.python.org/pypi/easyaccess/1.4.5"><img src="https://img.shields.io/badge/pypi-v1.4.5-orange.svg" alt="pypi version"/></a><a href="https://anaconda.org/mgckind/easyaccess"> <img src="https://img.shields.io/badge/Anaconda Cloud-v1.4.5-blue.svg" /> </a>
-![help_screen](data/help.gif)
+# easyaccess <a href="https://github.com/mgckind/easyaccess/releases/tag/1.4.6"> <img src="https://img.shields.io/badge/release-v1.4.6-blue.svg" alt="latest release" /></a> <a href="https://github.com/mgckind/easyaccess/blob/master/LICENSE.txt"><img src="https://img.shields.io/badge/license-NCSA%20License-blue.svg" alt="License" /> </a> <a href="https://pypi.python.org/pypi/easyaccess/1.4.6"><img src="https://img.shields.io/badge/pypi-v1.4.6-orange.svg" alt="pypi version"/></a> ![](https://img.shields.io/conda/v/mgckind/easyaccess.svg) ![](https://img.shields.io/conda/pn/mgckind/easyaccess.svg) ![](https://img.shields.io/badge/python-2.7%7C3.6-blue.svg)
 
-Enhanced command line SQL interpreter client for astronomical databases.
+Enhanced command line SQL interpreter client for astronomical surveys.
+![help_screen](data/help.gif)
 
-Python Command Line Interpreter to access Oracle DES DB
-using cx_Oracle
+## Description
+`easyaccess` is an enhanced command line interpreter and Python package created to facilitate access to astronomical catalogs stored in SQL Databases. It provides a custom interface with custom commands and was specifically designed to access data from the Dark Energy Survey Oracle database, including autocompletion of tables, columns, users and commands, simple ways to upload and download tables using csv, fits and HDF5 formats, iterators, search and description of tables among others. It can easily be extended to another surveys or SQL databases. The package was completely written in Python and support customized addition of commands and functionalities.
 
 For a short tutorial check [here](http://matias-ck.com/easyaccess)
 
-**Current version = 1.4.5**
+**Current version = 1.4.6**
 
+#### DES DR1 users
 For DES public data release, you can start `easyaccess` with:
 
     easyaccess -s desdr
@@ -18,24 +19,89 @@ To create an account click [here](https://des.ncsa.illinois.edu/easyweb/signup/)
 
 ## Requirements
 
-- Oracle Client > 11g.2 (External library, no python)
-  Check [here](https://opensource.ncsa.illinois.edu/confluence/display/DESDM/Instructions+for+installing+Oracle+client+and+easyaccess+without+EUPS) for instructions on how to install these libraries
-- [cx_Oracle](https://bitbucket.org/anthony_tuininga/cx_oracle)
-
-  Note that cx_Oracle needs libaio on some Linux systems (e.g., #98)
-
-  Note that cx_Oracle needs libbz2 on some Linux systems
+- [Oracle Client](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html) > 11g.2 (External library, no python)
+  Check [here](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html) for instructions on how to install these libraries
+- [cx_Oracle](https://cx-oracle.readthedocs.io/en/latest/index.html)
+  - Note that cx_Oracle needs libaio on some Linux systems
+  - Note that cx_Oracle needs libbz2 on some Linux systems
 - [fitsio](https://github.com/esheldon/fitsio) >= 0.9.6
 - [pandas](http://pandas.pydata.org/) >= 0.14
+- [numpy](https://docs.scipy.org/doc/numpy-1.15.1/reference/index.html)
 - [termcolor](https://pypi.python.org/pypi/termcolor)
 - [PyTables](http://pytables.github.io/) (optional, for hdf5 output)
 - [future](http://python-future.org/) (for python 2/3 compatibility)
 - [requests](http://docs.python-requests.org/en/master/)
 - [gnureadline](https://github.com/ludwigschwardt/python-gnureadline) (optional, for better console behavior in OS X)
-- importlib (This is only needed if running python 2.6)
 
+## Installation
+
+Installing `easyaccess` can be a little bit tricky given the external libraries required, in particular the Oracle libraries which are free to use. `easyaccess` is based heavily on the Oracle python client `cx_Oracle`, you can follow the installation instructions from [here](https://cx-oracle.readthedocs.io/en/latest/installation.html#quick-start-cx-oracle-installation). For `cx_Oracle` to work, you will need the Oracle Instant Client packages which can be obtained from [here](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html).
+
+Make sure you have these libraries installed before proceeding to the installation of easyaccess, you can try by opening a Python interpreter and type:
+
+    import cx_Oracle
+
+If you have issues, please check the [Troubleshooting page](https://cx-oracle.readthedocs.io/en/latest/installation.html#troubleshooting) or our [FAQ page](FAQ.md).
+
+#### Source Installation
+
+You can clone this repository and install `easyaccess` with:
+
+    python setup.py install
+
+#### Pip installation
+`easyaccess` can also be installed using `pip` but it'd require the installation of the oracle instant client first as described above
+
+    pip install easyaccess==1.4.6
+
+or directly from github:
+
+    pip install git+https://github.com/mgckind/easyaccess.git
+
+#### Conda installation
+For Collaborators, now easyaccess can be installed using [conda](http://conda.pydata.org/docs/install/quick.html) out of the box!
 
-## Some *nice* features
+    conda install easyaccess==1.4.6 -c mgckind -c anaconda
+
+#### Docker
+For collaborators, We have a Docker image with easyaccess pre-installed which you can obtained from:
+
+    docker pull mgckind/easyaccess
+
+## FAQ
+We have a running list of [FAQ](FAQ.md) which we will constantly update, please check [here](FAQ.md).
+
+#### Contributing
+Please take a look st our [Code of Conduct](CODE_OF_CONDUCT.md) and or [contribution guide](CONTRIBUTING.md).
+
+
+## Citation
+If you use `easyaccess` in your work we would encourage to use this reference [https://arxiv.org/abs/1810.02721](https://arxiv.org/abs/1810.02721) or copy/paste this BibTeX:
+```
+@ARTICLE{2018arXiv181002721C,
+       author = {{Carrasco Kind}, M. and {Drlica-Wagner}, A. and {Koziol}, A.~M.~G. and
+        {Petravick}, D.},
+        title = "{easyaccess: Enhanced SQL command line interpreter for astronomical surveys}",
+      journal = {arXiv e-prints},
+     keywords = {Astrophysics - Instrumentation and Methods for Astrophysics},
+         year = 2018,
+        month = Oct,
+          eid = {arXiv:1810.02721},
+        pages = {arXiv:1810.02721},
+archivePrefix = {arXiv},
+       eprint = {1810.02721},
+ primaryClass = {astro-ph.IM},
+       adsurl = {https://ui.adsabs.harvard.edu/\#abs/2018arXiv181002721C},
+      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
+}
+```
+
+
+## Usage
+
+For a short tutorial and documentation see [here](http://matias-ck.com/easyaccess), note that not all the features are available for the public use, i.e., DR1 users.
+
+#### Some *great* features
 - Nice output format (using pandas)
 - Very flexible configuration
 - Smart tab autocompletion for commands, table names, column names, and file paths
@@ -48,23 +114,14 @@ To create an account click [here](https://des.ncsa.illinois.edu/easyweb/signup/)
 - Show the execution plan of a query if needed
 - Python functions can be run in a inline query
 
-## Conda installation
-Now easyaccess can be installed using [conda](http://conda.pydata.org/docs/install/quick.html) out of the box!
-
-    conda install easyaccess==1.4.5 -c mgckind
-
-## Pip installation
-easyaccess can also be installed using `pip` but it'd require the installation of the oracle instant client first
 
-    pip install easyaccess==1.4.5
-
-## Interactive interpreter
+#### Interactive interpreter
 
 Assuming that ```easyaccess``` is in your path, you can enter the interactive interpreter by calling ```easyaccess``` without any command line arguments:
 
         easyaccess
 
-### Running SQL commands
+#### Running SQL commands
 Once inside the interpreter run SQL queries by adding a ";" at the end::
 
         DESDB ~> select ... from ... where ... ;
@@ -75,15 +132,15 @@ To save the results into a table add ">" after the end of the query (after ";")
 
 The file types supported so far are: .csv, .tab, .fits, and .h5. Any other extension is ignored.
 
-### Load tables
+#### Load tables
 To load a table it needs to be in a csv format with columns names in the first row
 the name of the table is taken from filename or with optional argument --tablename
 
         DESDB ~> load_table <filename> --tablename <mytable> --chunksize <number of rows to read/upload> --memsize <memory in MB to read at a time>
 
 The --chunsize and --memsize are optional arguments to facilitate uploading big files.
 
-### Load SQL queries
+#### Load SQL queries
 To load SQL queries just run:
 
         DESDB ~> loadsql <filename.sql>
@@ -93,7 +150,7 @@ or
 
 The query format is the same as the interpreter, SQL statement must end with ";" and to write output files the query must be followed by " > <output file>"
 
-### Configuration
+#### Configuration
 
 The configuration file is located at ```$HOME/.easyaccess/config.ini``` but everything can be configured from inside easyaccess type:
 
@@ -111,8 +168,14 @@ and to see any particular option (e.g., timeout):
 
         DESDB ~> config timeout show
 
-## Command line usage
+#### Command line usage
 
 Much of the functionality provided through the interpreter is also available directly from the command line. To see a list of command-line options, use the ```--help``` option
 
         easyaccess --help
+
+## Architecture
+
+We have included a simplified UML diagram describing the architecture and dependencies of `easyaccess` which shows only the different methods for a given class and the name of the file hosting a given class. The main class, `easy_or()`, inherits all methods from all different subclasses, making this model flexible and extendable to other surveys or databases. These methods are then converted to command line commands and functions that can be called inside `easyaccess`. Given that there are some DES specific functions, we have moved DES methods into a separate class `DesActions()`.
+
+![`easyaccess` architecture diagram](paper/classes_simple.png)