update readme firstpass

Cluster.md

@@ -1,3 +1,7 @@
+**The information on this page is and the functionality
+it pertains to is based on the v1 release, and has 
+not been updated.**
+
 ## Running a Local DPN Cluster
 
 You can run a local DPN REST cluster using the run_cluster script in the script

InternodeTesting.md

@@ -1,3 +1,7 @@
+**The information on this page is and the functionality
+it pertains to is based on the v1 release, and has 
+not been updated.**
+
 ## Internode Testing
 
 Each DPN node runs a demo or staging server, and from time to time, as

README.md

@@ -7,191 +7,124 @@
 A Rails implementation of the DPN RESTful communication layer. The current
 maintainer is [Bryan Hockey](https://github.com/malakai97).
 
-
-# Development and Test
+# Dependencies
 
 The project is built and tested with mri. For specific ruby version support,
 consult `.travis.yml` or the Travis CI build page.  Bundler is required.
 All other dependencies are described in the gemfile.
 
+# Development and Test
+
 ```
 git clone [email protected]:dpn-admin/dpn-server.git
 bundle install --path .bundle
 bundle exec rake config
 bundle exec rake db:setup
+bundle exec rspec
 ```
 
-Then, if you wish to run the tests:
-
-``` bundle exec rspec ```
-
-Or if you want to poke around the admin interface:
-
-```
-bundle exec rake db:admin_user # this will display the login credentials.
-bundle exec rails server
-```
-
-Then visit http://localhost:3000/admin.
-
-
-# Running a Local DPN Cluster
-It is possible to run a local DPN cluster.  This can be quite useful
-for debugging and development.  More information can be found in
-[Cluster.md](Cluster.md).
-
 # Production
 
-## Dependencies
-
-The project is built and tested with mri. For specific ruby version support,
-consult `.travis.yml` or the Travis CI build page.  Bundler is required.
-All other dependencies are described in the gemfile.
-
-## Getting the Files
-
-```
-git clone [email protected]:dpn-admin/dpn-server.git
-bundle install --path .bundle
-```
-
 ## Configuration
 
-Configuration is set in config/database.yml and
-config/secrets.yml.  These two files are excluded from git.  You
-may also wish to include additional gems specific to your
+### Configuration Files
+Configuration is set in the yaml files in the `config` folder
+that are generated by `bundle exec rake config`.  These files
+are excluded from version control.  Example files (denoted by
+the `.example` suffix) are the best source of information for
+configuring these files.
+
+### Gems Specific to My Environment
+You may wish to include additional gems specific to your
 deployment, e.g. "mysql2".  This can be done by creating a file
 Gemfile.local with your gems.  See Gemfile.local.example for
 more information.
 
-A generator exists to build these basic files, but it only
-includes development and test configurations.  You may wish
-to use it to get started, but the example files are the best
-source of information.  The generator is ```bundle exec rake config```
-
-### Changes from Previous Versions
-Previously, we used [dotenv](https://github.com/bkeepers/dotenv) for
-configuration.  This usage is now deprecated, in favor of editing
-the above yaml files, which are excluded from git.  You may wish to
-continue using the old functionality at this time.  To facilitate this,
-a set of generators are provided for you.  Simply run
-```bundle exec rake config:deprecated```, which will generate the
-above described configuration files configured to read the same environment
-variables as before.
-
-### Assets
+### Firewall
+Each other node will need incoming https to your instance.
 
-You will need to precompile the assets. This will create a ```public/assets/``` directory.
+### Node Configuration
 
-```
-bundle exec rake assets:precompile
-```
+Authentication is handled via authentication tokens, which must be generated on a
+per-node basis.
 
-## Database
+Authentication of incoming requests is allowed via Node.private_auth_token.
+You should mint these tokens for each node.  You will not need them again
+after storing them. A one-way hash function is applied to the token, so if the
+authenticated node loses the token a new one will need to be minted.
 
-You should create a database for your environment.  To date, the project
-has been tested with MySQL and PostgreSQL.
+Likewise, the corresponding token used to make requests against other nodes is stored as
+Node.auth_credential.  These are subject to two-way encryption.  These must be obtained
+from each other node.
 
-Define your database in config/database.yml.  See the configuration section
-for more information.
+Note that for the local node, the auth_credential and private_auth_token should be the
+same (before encryption).
 
-You should not have to create or populate the tables.  Rails will do this
-for you, assuming you have populated the configuration variables.
+Nodes must be configured once via either the rails console
+or the admin interface.  E.g.:
 
+```ruby
+Node.create!(
+  name: "Hathitrust", # Note this need *not* be the canonical name
+  namespace: "hathi",
+  api_root: "https://dpn.hathitrust.org/",
+  private_auth_token: "unencrypted_token",
+  auth_credential: "unencrypted_credential")
 ```
-bundle exec rake db:setup
-```
-
-## Storage
-
-Separate staging and repository directories are required
-(and specified in config/secrets.yml). Pruning of old files
-in the staging directory is considered environment-specific and
-therefore out of scope for this project.  (We use find. You may
-want something more robust if your bandwidth isn't "free.")
-
 
-## Administration
+### Installation
 
-In order to use the administration interface, you must create an admin user.
-You will need to do this in the rails console in order for the user's password
-to be correctly encrypted.
+Once the files are in place and configured appropriately 
+on your application server:
 
 ```
-RAILS_ENV=production bundle exec rails console
-```
-
-```ruby
-User.create!(
-  email: "[email protected]",
-  admin: true,
-  password: "unencrypted_passoword")
+bundle install --path .bundle
+bundle exec rake db:migrate
+bundle exec rake assets:precompile
 ```
 
-The admin interface can be reached at /admin/.  The admin interface is not
-necessarily easier to use than the rails console for all tasks.
-
-## Firewall
+## Replication and Synchronization
 
-Each other node will need incoming https and ssh (for rsync) access to your instance.
+dpn-server now provides built-in clients to handle replication and
+synchronization from other nodes.  If you wish to use this feature, 
+you must additionally perform the following configuration steps.
 
-## RSync
+To disable this feature, simply specify `:disbled` in `config/dpn.yml`.
+Consult `config/dpn.yml.example` for more information.
 
-To allow rsync over ssh, you will need to install each other node's public keys
-in the rsync user's authorized keys.  Since this grants ssh access for a real
-user, we recommend using a chroot environment and restricting the shell using
-[rssh](http://www.pizzashack.org/rssh/).
+### Firewall 
+Each other node will need ssh (for rsync) access to your instance.
 
+### RSync
 In order to get bags from other nodes, you'll need to mint an ssh keypair.  The
 private key should be installed locally, and the public key issued to each other
 node.
 
-## Jobs
+### Storage
+Separate staging and repository directories are required
+(and specified in config/secrets.yml). Pruning of old files
+in the staging directory is considered environment-specific and
+therefore out of scope for this project.
 
-Various jobs power some of the facilities of the application.  They use
-[ActiveJob](http://edgeguides.rubyonrails.org/active_job_basics.html), which allows
-you to substitute the job runner of your choice.  The default job runner is
-[Delayed Job](https://github.com/collectiveidea/delayed_job), which makes use of
-the "delayed_jobs" database table that was generated for you.
+### Redis
 
-## Node Configuration
+You must install redis.  On Debian-based machines, sufficient packages are
+available in apt: `apt install redis-server redis-tools`
 
-Nodes must be configured once via either the rails console
-or the admin interface.  E.g.:
+### Resque-Pool
+All jobs utilize a pool of workers.  To start the pool, spawn the process via:
 
-```ruby
-Node.create!(
-  namespace: "hathi",
-  api_root: "https://dpn.hathitrust.org/",
-  private_auth_token: "unencrypted_token",
-  auth_credential: "unencrypted_credential")
+```
+bundle exec rake resque:pool
 ```
 
-Do to the secret nature of some of these credentials, no script has yet been
-created to automate this task.  See below for more information.
-
-### Identification
-
-At minimum, you will need to know each node's namespace string and api root.  The
-rest will be obtained when the project queries the node.
-
-### Authentication
-
-Authentication is handled via authentication tokens, which must be generated on a
-per-node basis.
-
-Authentication of incoming requests is allowed via Node.private_auth_token.
-You should mint these tokens for each node.  You will not need them again
-after storing them. A one-way hash function is applied to the token, so if the
-authenticated node loses the token a new one will need to be minted.
-
-Likewise, the corresponding token used to make requests against other nodes is stored as
-Node.auth_credential.  These are subject to two-way encryption.  These must be obtained
-from each other node.
-
-Note that for the local node, the auth_credential and private_auth_token should be the
-same (before encryption).
+### Resque-Scheduler
+Synchronization jobs are driven by resque-scheduler.  This process can
+be started via:
 
+```
+bundle exec rake resque:scheduler
+```
 
 # Contributing
 

config/resque-pool.yml.example

@@ -4,6 +4,9 @@ development:
 test:
   sync: 1
   repl: 1
+
+# This is only required if you wish to use the replication
+# and synchronization features of dpn-server.
 production:
   sync: 4
   repl: 2

config/resque.yml.example

@@ -4,4 +4,10 @@ development:
 
 test:
   host: localhost
+  port: 6379
+
+# This is only required if you wish to use the replication
+# and synchronization features of dpn-server.
+production:
+  host: redis.mydomain.edu
   port: 6379