diff --git a/AUTHORS.md b/AUTHORS.md new file mode 100644 index 0000000..7b29971 --- /dev/null +++ b/AUTHORS.md @@ -0,0 +1,10 @@ +## Contributors to `easyaccess` +This is an alphabetical list of contributors, if you feel you have contributed to the project and we have neglected adding you to the list please accept our apologizes and [let us know](mailto:mgckind@gmail.com) to correct it. + +- Matias Carrasco Kind +- Alex Drlica-Wagner +- Landon Gelman +- Audrey Koziol +- Donald Petravick +- Eli Rykoff +- Ignacio Sevilla diff --git a/CHANGES.md b/CHANGES.md index 356f292..0dd8e63 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,11 +1,22 @@ # Changes +## v1.4.6 +#### 2019-JAN-14 +- No minor changes to the code, just the general documentation +- Adding Contribution guidelines and Coda of Conduct +- Adding Authors +- Adding FAQ +- Updating Readme and Installation instructions +- Reformatting code +- Update paper and references + + ## v1.4.5 #### 2018-OCT-04 - Restructuring code structure (PR #151) - Add paper reference - Minor fixes in unitests -- Update documentation +- Update documentation - Fix conda issues ## v1.4.4 diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..77595ae --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,76 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to making participation in our project and +our community a harassment-free experience for everyone, regardless of age, body +size, disability, ethnicity, sex characteristics, gender identity and expression, +level of experience, education, socio-economic status, nationality, personal +appearance, race, religion, or sexual identity and orientation. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment +include: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +* The use of sexualized language or imagery and unwelcome sexual attention or + advances +* Trolling, insulting/derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or electronic + address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable +behavior and are expected to take appropriate and fair corrective action in +response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other contributions +that are not aligned to this Code of Conduct, or to ban temporarily or +permanently any contributor for other behaviors that they deem inappropriate, +threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies both within project spaces and in public spaces +when an individual is representing the project or its community. Examples of +representing a project or community include using an official project e-mail +address, posting via an official social media account, or acting as an appointed +representative at an online or offline event. Representation of a project may be +further defined and clarified by project maintainers. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported by contacting the project team at mgckind@gmail.com. All +complaints will be reviewed and investigated and will result in a response that +is deemed necessary and appropriate to the circumstances. The project team is +obligated to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good +faith may face temporary or permanent repercussions as determined by other +members of the project's leadership. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, +available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html + +[homepage]: https://www.contributor-covenant.org + +For answers to common questions about this code of conduct, see +https://www.contributor-covenant.org/faq diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..bea1d90 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,40 @@ +# Contributing to `easyaccess` + + +When contributing to this repository, please first discuss the change you wish to make via [issues](https://github.com/mgckind/easyaccess/issues), +[email](mailto:mgckind@gmail.com), or any other method with the owners of this repository before making a change. + +Please note we have a [code of conduct](CODE_OF_CONDUCT.md) for this project, please follow it in all your interactions with the project. + +## Reporting Bugs + +When creating a bug or an issue please include as many details as possible. Create your issues [here](https://github.com/mgckind/easyaccess/issues) and follow this steps: + +- Check [previous closed issues](https://github.com/mgckind/easyaccess/issues?q=is%3Aissue+is%3Aclosed) , [current open issues](https://github.com/mgckind/easyaccess/issues?q=is%3Aopen+is%3Aissue) or our FAQ for a list of common questions or previously solved problems +- Use a clear and descriptive title for the issue to identify the problem. +- Provide all information about OS +- Provide information about the installation process (source, pip, conda, Docker, etc) +- software version including Python +- Detailed information on how to reproduce the problem +- Provide screenshots when useful + +## Other contributions + +All kind of contributions are welcome, from fixing bugs, resolving issues or suggesting features. Look at the [current issues](https://github.com/mgckind/easyaccess/issues) to see how you can contribute and provide help. + +## Pull Request Process + +1. Ensure any install or build dependencies are removed before the end of the layer when doing a + build. +2. Update the README.md with details of changes to the interface, this includes new environment + variables, exposed ports, useful file locations and container parameters. +3. Increase the version numbers in any examples files and the README.md to the new version that this + Pull Request would represent. The versioning scheme we use is [SemVer](http://semver.org/). +4. You may merge the Pull Request in once you have the sign-off of two other developers, or if you + do not have permission to do that, you may request the second reviewer to merge it for you. + +## Authors + +Please see our [list of contributors](AUTHORS.md), if you feel you have contributed to the project and we have neglected adding you to the list please accept our apologizes and [let us know](mailto:mgckind@gmail.com) to correct it. + + diff --git a/FAQ.md b/FAQ.md new file mode 100644 index 0000000..c048f0e --- /dev/null +++ b/FAQ.md @@ -0,0 +1,22 @@ +## Frequently Asked Questions + +Here we collect a list of FAQ related to installation, usage, etc. Please refre to the [issues page](https://github.com/mgckind/easyaccess/issues) for more information. + +- The installation with `pip` went smoothly but it was not obvious where "easyaccess" was installed. + - When installed using pip, one can use the following: `pip show -f easyaccess` to check the installation folders + +- When trying to connect I keep getting this error `ORA-21561: OID generation failed`. Any idea how to solve it? + - Most of the time this problem can be solved by adding the name of your computer in the `/etc/hosts` file, next to the line that says 127.0.0.1 localhost. Just add the name of your computer (type `hostname`) to that line, so it looks like `127.0.0.1 localhost my-computer` +- Where is the configuration file and authentication file? + - Usually the configuration file by default is at `$HOME/.easyaccess/config.ini` and the authentication file is at `$HOME/.desservices.ini` or can be set at the env variable `$DES_SERVICES` +- I am a DES Collaborator, where can I reset my credentials? + - For collaborators only: Please use [this](https://deslogin.wufoo.com/forms/help-me-with-my-desdm-account/) form +- The client hangs after getting an Oracle error and I need to close to the window. + - This is a long-standing [issue](https://github.com/mgckind/easyaccess/issues/130) but we haven't been able to fix it or to reproduce it in all the systems. Please report it there and add details about OS and versions. We'd recommend a clean conda installation which sometimes fixes the issue. +- How can I install the Oracle client by myself. + - Please follow the instructions posted [here](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html) +- Are there other requirements besides python modules and Oracle clients? + - Usually not, for new OS or Virtual Machines you'd need to install `libaio` and `libbz2` +- How can I contribute to the project? + - Please take a look st our [Code of Conduct](CODE_OF_CONDUCT.md) and or [contribution guide](CONTRIBUTING.md) + diff --git a/README.md b/README.md index f0b088a..c63d2b2 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,16 @@ -# easyaccess latest release License pypi version -![help_screen](data/help.gif) +# easyaccess latest release License pypi version ![](https://img.shields.io/conda/v/mgckind/easyaccess.svg) ![](https://img.shields.io/conda/pn/mgckind/easyaccess.svg) ![](https://img.shields.io/badge/python-2.7%7C3.6-blue.svg) -Enhanced command line SQL interpreter client for astronomical databases. +Enhanced command line SQL interpreter client for astronomical surveys. +![help_screen](data/help.gif) -Python Command Line Interpreter to access Oracle DES DB -using cx_Oracle +## Description +`easyaccess` is an enhanced command line interpreter and Python package created to facilitate access to astronomical catalogs stored in SQL Databases. It provides a custom interface with custom commands and was specifically designed to access data from the Dark Energy Survey Oracle database, including autocompletion of tables, columns, users and commands, simple ways to upload and download tables using csv, fits and HDF5 formats, iterators, search and description of tables among others. It can easily be extended to another surveys or SQL databases. The package was completely written in Python and support customized addition of commands and functionalities. For a short tutorial check [here](http://matias-ck.com/easyaccess) -**Current version = 1.4.5** +**Current version = 1.4.6** +#### DES DR1 users For DES public data release, you can start `easyaccess` with: easyaccess -s desdr @@ -18,24 +19,89 @@ To create an account click [here](https://des.ncsa.illinois.edu/easyweb/signup/) ## Requirements -- Oracle Client > 11g.2 (External library, no python) - Check [here](https://opensource.ncsa.illinois.edu/confluence/display/DESDM/Instructions+for+installing+Oracle+client+and+easyaccess+without+EUPS) for instructions on how to install these libraries -- [cx_Oracle](https://bitbucket.org/anthony_tuininga/cx_oracle) - - Note that cx_Oracle needs libaio on some Linux systems (e.g., #98) - - Note that cx_Oracle needs libbz2 on some Linux systems +- [Oracle Client](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html) > 11g.2 (External library, no python) + Check [here](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html) for instructions on how to install these libraries +- [cx_Oracle](https://cx-oracle.readthedocs.io/en/latest/index.html) + - Note that cx_Oracle needs libaio on some Linux systems + - Note that cx_Oracle needs libbz2 on some Linux systems - [fitsio](https://github.com/esheldon/fitsio) >= 0.9.6 - [pandas](http://pandas.pydata.org/) >= 0.14 +- [numpy](https://docs.scipy.org/doc/numpy-1.15.1/reference/index.html) - [termcolor](https://pypi.python.org/pypi/termcolor) - [PyTables](http://pytables.github.io/) (optional, for hdf5 output) - [future](http://python-future.org/) (for python 2/3 compatibility) - [requests](http://docs.python-requests.org/en/master/) - [gnureadline](https://github.com/ludwigschwardt/python-gnureadline) (optional, for better console behavior in OS X) -- importlib (This is only needed if running python 2.6) +## Installation + +Installing `easyaccess` can be a little bit tricky given the external libraries required, in particular the Oracle libraries which are free to use. `easyaccess` is based heavily on the Oracle python client `cx_Oracle`, you can follow the installation instructions from [here](https://cx-oracle.readthedocs.io/en/latest/installation.html#quick-start-cx-oracle-installation). For `cx_Oracle` to work, you will need the Oracle Instant Client packages which can be obtained from [here](https://www.oracle.com/technetwork/database/database-technologies/instant-client/overview/index.html). + +Make sure you have these libraries installed before proceeding to the installation of easyaccess, you can try by opening a Python interpreter and type: + + import cx_Oracle + +If you have issues, please check the [Troubleshooting page](https://cx-oracle.readthedocs.io/en/latest/installation.html#troubleshooting) or our [FAQ page](FAQ.md). + +#### Source Installation + +You can clone this repository and install `easyaccess` with: + + python setup.py install + +#### Pip installation +`easyaccess` can also be installed using `pip` but it'd require the installation of the oracle instant client first as described above + + pip install easyaccess==1.4.6 + +or directly from github: + + pip install git+https://github.com/mgckind/easyaccess.git + +#### Conda installation +For Collaborators, now easyaccess can be installed using [conda](http://conda.pydata.org/docs/install/quick.html) out of the box! -## Some *nice* features + conda install easyaccess==1.4.6 -c mgckind -c anaconda + +#### Docker +For collaborators, We have a Docker image with easyaccess pre-installed which you can obtained from: + + docker pull mgckind/easyaccess + +## FAQ +We have a running list of [FAQ](FAQ.md) which we will constantly update, please check [here](FAQ.md). + +#### Contributing +Please take a look st our [Code of Conduct](CODE_OF_CONDUCT.md) and or [contribution guide](CONTRIBUTING.md). + + +## Citation +If you use `easyaccess` in your work we would encourage to use this reference [https://arxiv.org/abs/1810.02721](https://arxiv.org/abs/1810.02721) or copy/paste this BibTeX: +``` +@ARTICLE{2018arXiv181002721C, + author = {{Carrasco Kind}, M. and {Drlica-Wagner}, A. and {Koziol}, A.~M.~G. and + {Petravick}, D.}, + title = "{easyaccess: Enhanced SQL command line interpreter for astronomical surveys}", + journal = {arXiv e-prints}, + keywords = {Astrophysics - Instrumentation and Methods for Astrophysics}, + year = 2018, + month = Oct, + eid = {arXiv:1810.02721}, + pages = {arXiv:1810.02721}, +archivePrefix = {arXiv}, + eprint = {1810.02721}, + primaryClass = {astro-ph.IM}, + adsurl = {https://ui.adsabs.harvard.edu/\#abs/2018arXiv181002721C}, + adsnote = {Provided by the SAO/NASA Astrophysics Data System} +} +``` + + +## Usage + +For a short tutorial and documentation see [here](http://matias-ck.com/easyaccess), note that not all the features are available for the public use, i.e., DR1 users. + +#### Some *great* features - Nice output format (using pandas) - Very flexible configuration - Smart tab autocompletion for commands, table names, column names, and file paths @@ -48,23 +114,14 @@ To create an account click [here](https://des.ncsa.illinois.edu/easyweb/signup/) - Show the execution plan of a query if needed - Python functions can be run in a inline query -## Conda installation -Now easyaccess can be installed using [conda](http://conda.pydata.org/docs/install/quick.html) out of the box! - - conda install easyaccess==1.4.5 -c mgckind - -## Pip installation -easyaccess can also be installed using `pip` but it'd require the installation of the oracle instant client first - pip install easyaccess==1.4.5 - -## Interactive interpreter +#### Interactive interpreter Assuming that ```easyaccess``` is in your path, you can enter the interactive interpreter by calling ```easyaccess``` without any command line arguments: easyaccess -### Running SQL commands +#### Running SQL commands Once inside the interpreter run SQL queries by adding a ";" at the end:: DESDB ~> select ... from ... where ... ; @@ -75,7 +132,7 @@ To save the results into a table add ">" after the end of the query (after ";") The file types supported so far are: .csv, .tab, .fits, and .h5. Any other extension is ignored. -### Load tables +#### Load tables To load a table it needs to be in a csv format with columns names in the first row the name of the table is taken from filename or with optional argument --tablename @@ -83,7 +140,7 @@ the name of the table is taken from filename or with optional argument --tablena The --chunsize and --memsize are optional arguments to facilitate uploading big files. -### Load SQL queries +#### Load SQL queries To load SQL queries just run: DESDB ~> loadsql @@ -93,7 +150,7 @@ or The query format is the same as the interpreter, SQL statement must end with ";" and to write output files the query must be followed by " > " -### Configuration +#### Configuration The configuration file is located at ```$HOME/.easyaccess/config.ini``` but everything can be configured from inside easyaccess type: @@ -111,8 +168,14 @@ and to see any particular option (e.g., timeout): DESDB ~> config timeout show -## Command line usage +#### Command line usage Much of the functionality provided through the interpreter is also available directly from the command line. To see a list of command-line options, use the ```--help``` option easyaccess --help + +## Architecture + +We have included a simplified UML diagram describing the architecture and dependencies of `easyaccess` which shows only the different methods for a given class and the name of the file hosting a given class. The main class, `easy_or()`, inherits all methods from all different subclasses, making this model flexible and extendable to other surveys or databases. These methods are then converted to command line commands and functions that can be called inside `easyaccess`. Given that there are some DES specific functions, we have moved DES methods into a separate class `DesActions()`. + +![`easyaccess` architecture diagram](paper/classes_simple.png) diff --git a/config/conda/meta.yaml b/config/conda/meta.yaml index 2537806..f7ea65c 100644 --- a/config/conda/meta.yaml +++ b/config/conda/meta.yaml @@ -1,9 +1,9 @@ package: name: easyaccess - version: "1.4.5" + version: "1.4.6" source: - git_rev: 1.4.5 + git_rev: 1.4.6 git_url: https://github.com/mgckind/easyaccess.git build: @@ -20,6 +20,7 @@ requirements: - python - future >=0.15.0 - requests + - pandas - termcolor >=1.1.0 - fitsio >=0.9.11 - oracle-instantclient ==11.2.0.4.0 @@ -31,6 +32,7 @@ requirements: - python - future >=0.15.0 - requests + - pandas - termcolor >=1.1.0 - fitsio >=0.9.11 - oracle-instantclient ==11.2.0.4.0 @@ -41,7 +43,7 @@ requirements: test: # Python imports imports: - #- easyaccess + - easyaccess - cx_Oracle - fitsio diff --git a/easyaccess/eaparser.py b/easyaccess/eaparser.py index 25ac211..40378d8 100644 --- a/easyaccess/eaparser.py +++ b/easyaccess/eaparser.py @@ -8,9 +8,9 @@ class MyParser(argparse.ArgumentParser): def error(self, message): - print('\n*****************') - sys.stderr.write('error: %s \n' % message) - print('*****************\n') + print("\n*****************") + sys.stderr.write("error: %s \n" % message) + print("*****************\n") self.print_help() sys.exit(2) @@ -18,48 +18,94 @@ def error(self, message): def get_args(config_file): conf = config_mod.get_config(config_file) parser = MyParser( - description='Easy access to the DES database. There is a configuration file ' - 'located in %s for more customizable options' % config_file) - parser.add_argument("-v", "--version", action="store_true", - help="print version number and exit") - parser.add_argument("-c", "--command", dest='command', - help="Executes command and exit") - parser.add_argument("-l", "--loadsql", dest='loadsql', - help="Loads a sql command, execute it and exit") - parser.add_argument("-lt", "--load_table", dest='loadtable', - help="Loads data from a csv, tab, or fits formatted file \ + description="Easy access to the DES database. There is a configuration file " + "located in %s for more customizable options" % config_file + ) + parser.add_argument( + "-v", "--version", action="store_true", help="print version number and exit" + ) + parser.add_argument( + "-c", "--command", dest="command", help="Executes command and exit" + ) + parser.add_argument( + "-l", + "--loadsql", + dest="loadsql", + help="Loads a sql command, execute it and exit", + ) + parser.add_argument( + "-lt", + "--load_table", + dest="loadtable", + help="Loads data from a csv, tab, or fits formatted file \ into a DB table using the filename as the table name or a custom \ - name with --tablename MYTABLE. Not available in desdr.") - parser.add_argument("-at", "--append_table", dest='appendtable', - help="Appends data from a csv, tab, or fits formatted file \ + name with --tablename MYTABLE. Not available in desdr.", + ) + parser.add_argument( + "-at", + "--append_table", + dest="appendtable", + help="Appends data from a csv, tab, or fits formatted file \ into a DB table using the filename as the table name or a custom \ - name with --tablename MYABLE. Not available in desdr.") - parser.add_argument("--tablename", dest='tablename', - help="Custom table name to be used with --load_table\ - or --append_table. Not available in desdr.") - parser.add_argument("--chunksize", dest='chunksize', type=int, default=None, - help="Number of rows to be inserted at a time. Useful for large files " - "that do not fit in memory. Use with --load_table or --append_table") - parser.add_argument("--memsize", dest='memsize', type=int, default=None, - help=" Size of chunk to be read at a time in Mb. Use with --load_table or " - "--append_table") - parser.add_argument("-s", "--db", dest='db', - choices=['dessci', 'desoper', 'destest', 'desdr'], - help="Override database name [dessci,desoper,destest,desdr]") - parser.add_argument("-q", "--quiet", action="store_true", dest='quiet', - help="Silence initialization, no loading bar") - parser.add_argument("-u", "--user", dest='user') - parser.add_argument("-p", "--password", dest='password') - parser.add_argument("-nr", "--no_refresh", dest='norefresh', action="store_true", - help="Do not refresh metadata at starting up to speed initialization. " - "Metadata can always be refreshed from inside using the " - "refresh_metadata command") - parser.add_argument("--config", help="--config show, will print content of " - "config file\n" - "--config reset will reset config to default " - "values\n" - "--config set param1=val1 param2=val2 will " - "modify parameters for the session only", nargs='+') + name with --tablename MYABLE. Not available in desdr.", + ) + parser.add_argument( + "--tablename", + dest="tablename", + help="Custom table name to be used with --load_table\ + or --append_table. Not available in desdr.", + ) + parser.add_argument( + "--chunksize", + dest="chunksize", + type=int, + default=None, + help="Number of rows to be inserted at a time. Useful for large files " + "that do not fit in memory. Use with --load_table or --append_table", + ) + parser.add_argument( + "--memsize", + dest="memsize", + type=int, + default=None, + help=" Size of chunk to be read at a time in Mb. Use with --load_table or " + "--append_table", + ) + parser.add_argument( + "-s", + "--db", + dest="db", + choices=["dessci", "desoper", "destest", "desdr"], + help="Override database name [dessci,desoper,destest,desdr]", + ) + parser.add_argument( + "-q", + "--quiet", + action="store_true", + dest="quiet", + help="Silence initialization, no loading bar", + ) + parser.add_argument("-u", "--user", dest="user") + parser.add_argument("-p", "--password", dest="password") + parser.add_argument( + "-nr", + "--no_refresh", + dest="norefresh", + action="store_true", + help="Do not refresh metadata at starting up to speed initialization. " + "Metadata can always be refreshed from inside using the " + "refresh_metadata command", + ) + parser.add_argument( + "--config", + help="--config show, will print content of " + "config file\n" + "--config reset will reset config to default " + "values\n" + "--config set param1=val1 param2=val2 will " + "modify parameters for the session only", + nargs="+", + ) args = parser.parse_args() if args.version: @@ -67,42 +113,44 @@ def get_args(config_file): sys.exit() if args.config: - if args.config[0] == 'show': - print('\n Showing content of the config file (%s) :\n' % config_file) - file_temp = open(config_file, 'r') + if args.config[0] == "show": + print("\n Showing content of the config file (%s) :\n" % config_file) + file_temp = open(config_file, "r") for line in file_temp.readlines(): print(line.strip()) file_temp.close() sys.exit() - elif args.config[0] == 'reset': - print('\n ** Reset config file (%s) to its default!! **:\n' % config_file) - check = input(' Proceed? (y/[n]) : ') - if check.lower() == 'y': + elif args.config[0] == "reset": + print("\n ** Reset config file (%s) to its default!! **:\n" % config_file) + check = input(" Proceed? (y/[n]) : ") + if check.lower() == "y": os.remove(config_file) conf = config_mod.get_config(config_file) sys.exit() - elif args.config[0] == 'set': + elif args.config[0] == "set": if len(args.config) == 1: parser.print_help() sys.exit() - entries = ','.join(args.config[1:]) - entries = entries.replace(',,', ',') - entries = entries.split(',') + entries = ",".join(args.config[1:]) + entries = entries.replace(",,", ",") + entries = entries.split(",") for e in entries: - if e == '': + if e == "": continue updated = False try: - key, value = e.split('=') - for section in (conf.sections()): + key, value = e.split("=") + for section in conf.sections(): if conf.has_option(section, key): conf.set(section, key, str(value)) updated = True if not updated: raise except: - print("Check the key exists or that you included the '=' for the " - "parameter\nFor more info use --help.") + print( + "Check the key exists or that you included the '=' for the " + "parameter\nFor more info use --help." + ) sys.exit() else: parser.print_help() diff --git a/easyaccess/version.py b/easyaccess/version.py index f13762b..92e3f3f 100644 --- a/easyaccess/version.py +++ b/easyaccess/version.py @@ -3,29 +3,31 @@ import logging import warnings from datetime import datetime + warnings.filterwarnings("ignore") def last_pip_version(): import requests + logging.getLogger("requests").setLevel(logging.WARNING) """ Return last available version of easyaccess from pypi """ - url = "https://pypi.python.org/pypi/%s/json" % ('easyaccess',) + url = "https://pypi.python.org/pypi/%s/json" % ("easyaccess",) data = requests.get(url, verify=False).json() uploads = [] - for k in data['releases'].keys(): + for k in data["releases"].keys(): try: - up_time = data['releases'][k][0]['upload_time'] - uploads.append([k, datetime.strptime(up_time, '%Y-%m-%dT%H:%M:%S')]) + up_time = data["releases"][k][0]["upload_time"] + uploads.append([k, datetime.strptime(up_time, "%Y-%m-%dT%H:%M:%S")]) except: pass return sorted(uploads, key=lambda x: x[1])[-1][0] -version_tag = (1, 4, 5) -__version__ = '.'.join(map(str, version_tag[:3])) +version_tag = (1, 4, 6) +__version__ = ".".join(map(str, version_tag[:3])) if len(version_tag) > 3: - __version__ = '%s-%s' % (__version__, version_tag[3]) + __version__ = "%s-%s" % (__version__, version_tag[3]) diff --git a/paper/paper.md b/paper/paper.md index b7f4011..08bc021 100644 --- a/paper/paper.md +++ b/paper/paper.md @@ -31,12 +31,12 @@ bibliography: paper.bib # Summary -`easyaccess` is an enhanced command line interpreter and Python package created to facilitate access to astronomical catalogs stored in SQL Databases. It provides a custom interface with custom commands and was specifically designed to access data from the [Dark Energy Survey](https://www.darkenergysurvey.org/) Oracle database, although it can easily be extended to another survey or SQL database. The package was completely written in [Python](https://www.python.org/) and support customized addition of commands and functionalities. +`easyaccess` is an enhanced command line interpreter and Python package created to facilitate access to astronomical catalogs stored in SQL Databases. It provides a custom interface with custom commands and was specifically designed to access data from the [Dark Energy Survey](https://www.darkenergysurvey.org/) Oracle database, although it can easily be extended to another survey or SQL database. The package was completely written in [Python](https://www.python.org/) and supports customized addition of commands and functionalities. Visit [https://github.com/mgckind/easyaccess](https://github.com/mgckind/easyaccess) to view installation instructions, tutorials, and the Python source code for `easyaccess`. # The Dark Energy Survey -The Dark Energy Survey (DES) [@DES2005; @DES2016] is an international, collaborative effort of over 500 scientists from 26 institutions in seven countries. The primary goals of DES are reveal the nature of the mysterious dark energy and dark matter by mapping hundreds of millions of galaxies, detecting thousands of supernovae, and finding patterns in the large-scale structure of the Universe. Survey operations began on on August 31, 2013 and will conclude in early 2019. For about 500 nights, DES has been taking thousands of deep images of southern sky, which are transferred and processed at the National Center for Supercomputing Applications ([NCSA](http://www.ncsa.illinois.edu/)). The images are processed to produce catalogs of astronomical sources with hundreds of millions of entries (billions in the case of individual detections), describing the sources found within the images and other relevant metadata. +The Dark Energy Survey (DES) [@DES2005; @DES2016] is an international, collaborative effort of over 500 scientists from 26 institutions in seven countries. The primary goals of DES are to reveal the nature of the mysterious dark energy and dark matter by mapping hundreds of millions of galaxies, detecting thousands of supernovae, and finding patterns in the large-scale structure of the Universe. Survey operations began on on August 31, 2013 and will conclude in early 2019. For about 500 nights, DES has been taking thousands of deep images of southern sky, which are transferred and processed at the National Center for Supercomputing Applications ([NCSA](http://www.ncsa.illinois.edu/)). The images are processed to produce catalogs of astronomical sources with hundreds of millions of entries (billions in the case of individual detections), describing the sources found within the images and other relevant metadata. A significant subset of the DES data was recently [made public](https://des.ncsa.illinois.edu/releases/dr1) [@DR1] and can be accessed through several mechanisms including `easyaccess` and [web interfaces](https://des.ncsa.illinois.edu/easyweb/) that run `easyaccess` as a backend. This public release includes information for almost 400 million astrophysical sources and complementary tables to allow scientific analysis. ## DES users @@ -47,7 +47,7 @@ The first release of `easyaccess` was on February 17th, 2015 and since then, ove # `easyaccess` -`easyaccess` is a command line interpreter that is heavily based on `termcolor` [@termcolor] and the [`cmd`](https://docs.python.org/3/library/cmd.html) Python core module. It interfaces with `cx_Oracle` [@cxoracle] to communicate with Oracle, `requests` [@requests] for external URL requests, and other external open source libraries, including NumPy [@NumPy], `pandas` [@pandas], `fitsio` [@fitsio] and `h5py` [@h5py] to handle and transform array data. +`easyaccess` is a command line interpreter that is heavily based on `termcolor` [@termcolor] and the [`cmd`](https://docs.python.org/3/library/cmd.html) Python core module. It interfaces with `cx_Oracle` [@cxoracle] to communicate with Oracle, `requests` [@requests] for external URL requests, and other external open source libraries, including NumPy [@NumPy], `pandas` [@pandas], `fitsio` [@fitsio] and `h5py` [@h5py] to handle and transform array data. Figure 2 shows an example of the welcome screen as seen by a DES user. ![Welcome screenshot](easyaccess_welcome.png) @@ -62,35 +62,14 @@ One can also load SQL queries from a file into the database, or run SQL queries Users can also use `easyaccess` to submit and request cutouts around specific positions or objects which are generated from the images. This allows better integration with other data services for a richer scientific workflow. -## Architecture - -We have included a simplified UML diagram describing the architecture and dependencies of `easyaccess` (Figure 3). Figure 3 shows only the different methods for a given class and the name of the file hosting a given class. The main class, `easy_or()`, inherits all methods from all different subclasses, making this model flexible and extendable to other surveys or databases. These methods are then converted to command line commands and functions that can be called inside `easyaccess`. Given that there are some DES specific functions, we have moved DES methods into a separate class `DesActions()`. - -![`easyaccess` architecture diagram](classes_simple.png) ## Installation -To download easyaccess you can clone the source code from GitHub at [https://github.com/mgckind/easyaccess](https://github.com/mgckind/easyaccess) or follow any of the standard installation channels described below. - -- From [source](https://github.com/mgckind/easyaccess) - - `python setup.py install` - -- [conda](https://conda.io/docs/) - - `conda install easyaccess -c mgckind` - -- [Docker](https://hub.docker.com/r/mgckind/easyaccess/) - - `docker pull mgckind/easyaccess` - -- [pip](https://pypi.org/project/easyaccess/1.4.4/) - - `pip install easyaccess` +API documentation and installation instructions are available on the [online documentation](http://matias-ck.com/easyaccess/) or and the [Readme](https://github.com/mgckind/easyaccess) file on the repository # Acknowledgments -The DES Data Management System is supported by the National Science Foundation under Grant NSF AST 07-15036 and NSF AST 08-13543. +The DES Data Management System is supported by the National Science Foundation under Grant NSF AST 07-15036 and NSF AST 08-13543. # References diff --git a/paper/paper.pdf b/paper/paper.pdf index 2cb02e3..7649c16 100644 Binary files a/paper/paper.pdf and b/paper/paper.pdf differ