Welcome to the Distant Reading Early Modernity (DREaM) Project!
2025 Updates
- Access to CMU Libraries Server of DREaM
- Public Deployment of DREaM with HTTPS
- Software Archaeology of DREaM and Voyant Tools
- UI Changes to DREaM and Voyant Tools
- Creation of Matomo Website Analytics
Helpful Links
Deployed Websites
-
CMU Clone of Voyant Tools: https://dream.library.cmu.edu/
2025 Box for Voyant Server and DREaM
2025 Technical Documentation
2025 CMU Libraries Sustainability Plan Proposal
How DREaM Works
Parameters
When choosing parameters in the corpus creation tool what DREaM is doing is looking through all the metafile data for all the documents and finding documents that match.
Downloading as a Zip
When zipping it then makes a temporary corpus and downloads all the files that match your parameters as XML files in a folder. This tmp corpus will be in the data folder however it is not set up to be opened in Voyant analytics. The zip file that was downloaded would need to be reuploaded onto Voyant in order for the analytics tools to find all the proper information.
Exporting to Voyant
One of the major recommendations for future teams is to fix the “Export Corpus to Voyant” button as this is one of the main functionalities of DREaM. As of now, a user is able to navigate to Voyant Tools from the DREaM navigation bar and upload their corpus zip file manually. The “Export Corpus to Voyant” button should work to streamline this process and have the temporary corpus automatically uploaded to Voyant Tools and ready to analyze.
Reference this video for the ideal workflow: https://youtu.be/aZkz7qn6hzo
Public Deployment: CMU Libraries Server
How to login into server:
-
%
ssh [email protected]- will prompt you to enter password - ask Jonathan Kiritharan from CMU Libraries
-
%
cd DREaM-Voyant-App/
Getting Code Off of DREaM SSH Server:
-
In your own terminal window enter the following command:
rsync -avz [email protected]:/usr/local/share/dreamer/DREaM-Voyant-App /path/to/your/code/- Replace
/path/to/your/code/with the file path to your 67-373 folder on your local device
- Replace
-
Enter DREaM server password
-
A copy of the code should be made in your local 67-373 folder (will take some time)
Opening DREaM Locally:
-
In your own terminal window, navigate to your local 67-373 folder where you copied the code from the server
-
Navigate into the DREaM folder
cd DREaM-Voyant-App -
Change local Java version to Java 8
export JAVA_HOME=`/usr/libexec/java_home -v 1.8` -
Run Voyant Jar File
java -jar VoyantServer.jar -
Voyant should open locally in your web browser http://127.0.0.1:8888/
-
To navigate to DREaM enter the following URL http://127.0.0.1:8888/dream
Sending Updated Code/Code Changes Back to DREaM SSH Server:
-
In your own terminal window enter the following command:
rsync -avz --update --dry-run /path/to/your/code/ [email protected]:/usr/local/share/dreamer-
Replace
/path/to/your/code/with the file path to your 67-373 folder on your local device -
--dry-run- shows what files will be sent to the server (make sure this aligns with what you changes you want to send)
-
-
Once you confirm, the files being sent are correct enter the following command:
rsync -avz --update /path/to/your/code/ [email protected]:/usr/local/share/dreamer- Replace
/path/to/your/code/with the file path to your 67-373 folder on your local device
- Replace
-
Enter DREaM server password
-
Code changes should be sent to SSH server *Can ensure this is correct by cd-ing into files on server or for front-end changes doing
cmd-shift-ron https://dream.library.cmu.edu/dream/ or https://dream.library.cmu.edu
Handling Website Outages
If DREaM goes down/not live on website link:
-
Login into server
-
cd /usr/local/share/dreamer/DREaM-Voyant-App -
Run command
nohup java -Dorg.voyanttools.server.host=0.0.0.0 -Dorg.voyanttools.server.port=8080 -jar VoyantServer.jar headless=true &
Server Reboot
Cache Memory Issue/ngninx error when opening website and is not resolved with the steps above
-
Reboot server
sudo reboot -
Kill current Voyant processes
pkill -f VoyantServer.jar -
Confirm they are gone
ps aux | grep VoyantServer.jar -
Disable firewalld
sudo service firewalld stop -
cd /usr/local/share/dreamer/DREaM-Voyant-App -
Run the following command to start Voyant
nohup java -Dorg.voyanttools.server.host=0.0.0.0 -Dorg.voyanttools.server.port=8080 -jar VoyantServer.jar headless=true & -
Open website link and ensure DREaM is back up
Important Files & Folders
-
_app/dream folder - all files & images related to DREaM
-
_app/dream/index.jsp - DREaM home page, almost all JavaScript for the DREaM corpus filtering features, and includes Matomo code
-
_app/dream/info.jsp - DREaM information page, created using just HTML and CSS, also includes Matomo code
-
_app/resources/voyant/current/voyant.min.js - Where you would edit the JavaScript to add/change features that cannot be done in _app/dream/index.jsp. For example, if changes want to be done to the Download ZIP Archive window (panel that appears when you click Download ZIP Archive button), changes would be done in this file. This can be done by editing a relevant Voyant Widget in the file.
- Note: _app/resources/voyant/current/voyant.js and files in _app/app/widget have the same or very similar code, however changes for DREaM can only be seen if they are made in voyant.min.js
-
_app/resources/voyant/current/voyant-locale-en.js - Used to redefine string variables in DREaM. For example, if the text for a tool tip or field label needs to be changed, it would be done in this file. The sections in the file correspond to where the variable is used.
- For example:
- In voyant-locale-en.js: Ext.apply(Voyant.widget.DownloadFileFormat.i18n, { "TXTTip":"This will produce a plain text version of each document." .....});
- TXTTip would be referenced under Ext.define("Voyant.widget.DownloadFileFormat" in voyant.min.js
- For example:
-
_app/resources/ext/6.2.0/theme-crisp/resources/theme-crisp-all_1.css - Where style changes are made to DREaM
-
_app/resources/css/styles.css - Where the appearance of CMU Clone of Voyant Tools can be changed
-
nohup.out - Where the output of the terminal can be seen (This is where the “no segments” error related to the Export Corpus to Voyant button/Server 500 error can be seen)
Note: Most of the files are written using ExtJS, a JavaScript framework