Carlyle Letters Online hosts a digital archive containing thousands of letters of great historical import, dating back to the Victorian Era. The influential pair of thinkers whose letter correspondence make up our archive - Thomas and Jane Welsh Carlyle - are timeless figures needing a group of individuals to modernize their writings. For more non-technical information about this project, consult the about-project page on the site itself.
While most of these dependencies are installed automatically by executing the scripts in bin/, see below for a list of the major dependencies required to build and deploy CLO3. This list is not exhaustive, and one should consult those scripts, as well as the respective package.json files for a complete list.
You will need root access to the machine to build CLO3. Our team currently uses Ubuntu 20.04 for its remote servers.
Attempting the install on your own local machine, especially if you're on Mac, might be difficult. If you're struggling, ask the sysadmins to set up a clean virtual machine for you.
- Git
- python3-pip
- Nodeenv
- Node (^12.18.3)
- MongoDB
- ElasticSearch (^7.8)
- Angular (^9.0.4)
- jquery (^3.4.1)
Install Git using your system's package manager. See [https://git-scm.com/book/en/v2/Getting-Started-Installing-Git](their installation directions). On Debian/Ubuntu based systems, the command is:
sudo apt-get update
sudo apt-get install git python3-pip certbot python3-certbot-nginx
This may take awhile, especially on the line "processing triggers for man-db". Be patient.
Clone Repository
Clone the repository to your command line. On the home page of the repository, locate the green button above the list of folders and files that says "Code," and click on it to pull up the HTTPS of this repository. Click the button to the right of the url. This copies it, you'll need to paste it into the cloning command.
git clone <url>
The bin/ directory at the root of the project contains all the build scripts. These scripts are named sequentially. Most are small, and do not do more than a few different things. Be sure that you are in the /bin directory before executing the first script. Executing the first script outside of /bin directory may prevent correct configuration of the environment variable. The scripts for building the site are listed below.
When going through the scripts, it's important to know what is going on in them, rather than just going through the following steps described in this section. To that end, it's reccomended that you read the scripts prior to execution. Here is a link to the folder which contains our scripts. Open that page in a new tab and follow along that way as we progress in this section or open each script individually in a new window as we come to it - the links to that particular script will be in each section.
Before executing each script, you can expect a brief description on its functionality. Let's begin!
A1
Let's check out the first script. Right click the link to the first script and choose the option "open in a different window," or just click on the script "A1-configure-nodeenv.sh," from the bin/ folder.
The first script serves two important functions. First, it configure the CLO_ROOT environment variable. Second, it creates the Node environment.
Configuring a variable initially is only tentative. The change in our bash configuration file will only propogate after exiting and re-logging into the server (quitting and restarting SSH) or by running the "source" command on your .bashrc file in the terminal (source ~/.bashrc).
./A1-configure_nodeenv.sh
Before running the next script, ensure the environment variable is set.
echo $CLO_ROOT
Expected Output After Echo Statement
$ echo $CLO_ROOT
/home/kennethj/clo3
If it is empty, the environment variable is not set. Check that the necessary "export" line has been appended to your ~/.bashrc file.
Click here if You're Having Trouble
Let's issue a command to see if the line was appended...
`tail -r ~/.bashrc` `q`
Example Output
$ tail -r ~/.bashrc
export CLO_ROOT="/home/mitchelllambert/clo3"
alias python=...
PS1=...
This command prints out the last few lines of our bashrc file in reverse order. So, if you do not see the export statement at the top of this command output, the export line was not appended.
Un-collapse this section to move on from this if the line was appended or execute the following steps to set the variable manually.
Find your current path: pwd
Create Variable & Put it in bashrc file: export CLO_ROOT=your current path here .
Save Changes Permanently to Bashrc: source ~/.bashrc
Now, run the echo steps again to verify it's set.
Now that we have the first part taken care of, we can move onto activating the Node environment. Be sure to have this environment activated at any time you're working with CLO3.
source ../env/bin/activate or source $CLO_ROOT/env/bin/activate
If this doesn't work, you may need to execute the script again (in the /bin directory), and then try the above source command again.
Ensure that you're in the /bin folder first!
$ pwd
/home/mitchelllambert/clo3/bin
You in the /bin folder? If so, go ahead and execute the script then activate the environment. If not, navigate to it before moving on.
./A1-configure_nodeenv.sh
source ../env/bin/activate
What we've just installed and activated is Node.js, this is a JavaScript runtime environment that lets us execute our JavaScript code outside a web browser. Whenever you are working with CLO3, be sure to have the nodeenv activated.
A2
Let's check out the the second script in a new window.
The second script decends into 'clo-angular' and 'clo-api' to install the required node packages. Rember that last script installed Node for us. Node comes with a default (node) package manager. Which means Node's default package manager is a dependency of Node itself.
As you can see if you're following along in the script, our package manager installs dependencies after descending into the two directories using the aptly named...
npm install...
... command. Now, to introduce you to our dependency structure, let's check where our package manager is obtaining the dependency information.
They can be found in the package.json files of Angular's & the API's root directory. Check them out by either opening these files in a text editor or by executing a command to print in your terminal the particular segment of these two files from which our package manager is getting the required information.
So where is NPM locating Dependency Information?
Locate and Open the Files NPM is Using
cd ../clo-angular && vim package.json
Backtrack to the project's root direct then move into angular's root directory, and open up the file (we use vim here but you can use whichever text editor you're most comfortable with.
Take note of the dependencies & devDependencies section, these are angular's dependencies and all the key-value pairs listed here is where our package manager is reading from in the second script.
:q!
Exit the vim editor.
Now you can repeat this process for checking out the API's dependencies, subsiting the API's root folder in for step 1.
View File Contents NPM is Using from Command Line
sed -n '/"dependencies'/,$p' ../clo-angular/package.json
Expected Output
$ sed -n '/"dependencies"/,$p' ../clo-angular/package.json
"dependencies": {
"@angular/common": "^9.0.4",
"@angular/compiler": "^9.0.4",
"@angular/compiler-cli": "^9.0.4",
"@angular/core": "^9.0.4",
"@angular/forms": "^9.0.4",
"@angular/localize": "^9.0.4",
"@angular/platform-browser": "^9.0.4",
"@angular/platform-browser-dynamic": "^9.0.4",
"@angular/router": "^9.0.4",
"@fortawesome/angular-fontawesome": "^0.6.0",
"@fortawesome/fontawesome-svg-core": "^1.2.27",
"@fortawesome/free-solid-svg-icons": "^5.12.1",
"@ng-bootstrap/ng-bootstrap": "^6.0.0",
"bootstrap": "^4.4.1",
"font-awesome": "^4.7.0",
"jquery": "^3.4.1",
"mark.js": "^8.11.1",
"nan": "^2.14.0",
"popper.js": "^1.16.1",
"rxjs": "^6.5.4",
"tslib": "^1.11.1",
"zone.js": "~0.10.2"
},
"devDependencies": {
"@angular-devkit/build-angular": "~0.900.4",
"@angular/cli": "~9.0.4",
"@fortawesome/fontawesome-free": "^5.12.1",
"jasmine-core": "~3.5.0",
"jasmine-spec-reporter": "~4.2.1",
"karma": "^4.4.1",
"karma-chrome-launcher": "~3.1.0",
"karma-coverage-istanbul-reporter": "^2.1.1",
"karma-jasmine": "~3.1.1",
"karma-jasmine-html-reporter": "^1.5.2",
"protractor": "^5.4.3",
"ts-node": "~8.6.2",
"tslint": "~6.0.0",
"typescript": "^3.7.5",
"viewerjs": "^1.5.0"
}
}
Similarly, we can run the same command, substituting the api's package.json in as argument file to check the api's dependencies:
sed -e '/"dependencies": {/,$p' clo-api/package.json
Now that we've checked out where our package manager is getting it's information, let's execute the script.
./A2-install_dependencies.sh
Note that the script will prompt the user for returning feedback about Angular to Google. Answer as you please.
A3
Let's pull up the third script.
This script creates and populates the .env file in the api.
./A3-api_env_file.sh
Let's ensure its correctly configured by checking the .env file contents.
cat $CLO_ROOT/clo-api/.env.
Expected Output
$ cat $CLO_ROOT/clo-api/.env
DB_HOST=mongodb://127.0.0.1.27017/clo
ES_HOST=http://127.0.0.1:9200
B1
This script and the next one together deploy clo-api. This Mongo installation script is found here. This script install MongoDB and restores the contents of the database../B1-install_mongo.sh
Let's check to see if Mongo is running.
ps -aux | grep -e "mongo" | grep -v "grep".
B2
I am here.
For Mac users, rather than executing the build script you'll need to download with Homebrew or from archive . Assuming you don't execute the build script, you'll also need to remember to start the elasticsearch process as a service before proceeding, using brew or elastic search's executable file (elasticsearch-$VERSION_NUMBER/bin/elasticsearch).
This script installs elasticsearch to the the API's bin directory ($CLO_ROOT/clo-api/bin directory).
ElasticSearch is a database used with MongoDB because of its phenomenal ability to search through files.
./B2-install_elastic.sh
Let's check to make sure elastic search is running.
ps -aux | grep -e "elastic" | grep -v "grep".
Now we are going to run a script that takes all of the entries in the Mongo database and put the relevant information into the ElasticSearch database so that the user can search through the volumes.
cd ../clo-api
node elasticSync.js
nohup ./bin/www &
This last command runs the equivalent of 'npm start' using the nohup ("no hangup") command. The "&" sends the process to the background immediately. This allows the process to continue running after the shell has been detached.
Due to a inconsistency in the output of nohup, you may need to press Enter/Return to get another command prompt in your terminal. This is normal.
At this point, your API is running. Ensure it is configured correctly by checking the contents of the logging file with.
cat nohup.out
Expected Output
$ cat nohup.out
Connected to MongoDB at URL: mongodb://127.0.0.1:27017/clo
Connected to Elasticsearch at URL: http://127.0.0.1:9200
Important Info About Nohup For SysAdmins
Maintaining the process in background after terminal session is ended will work only if the terminal session was ended properly. To do this, before exiting, you must press Control D in the terminal.
C1
Open me in a new window.
cd $CLO_ROOT/bin
./C1-build_site_PROD.sh
This script builds the Angular front-end. It also creates a symlink from the build artifact ('dist') to the /srv/ directory as per the LFSH. This is the directory that NGINX points to.
This script also copies the HTTP version of the Nginx config to the '/etc/nginx/sites-available' directory.
Note that this will take awhile to run. Developers often report the longest wait at 92% compiling. Be patient.
Configure NGINX
cd /etc/nginx/sites-enabled
sudo unlink default
sudo cp /<path>/<to>/clo3/docs/nginx-configs/clo.dev.HTTP.conf /etc/nginx/sites-available
Run ls to check that this has been done correctly. You should see clo.dev.HTTP.conf .
If you are working on a virtual machine provided by the sysadmin, you will need to change to localhost:
sudo vim clo.dev.HTTP.conf
Type i to go into insert mode, and then change server_name to localhost.
esc and :wq to save and exit.
At this point, run nginx -s reload and confirm that the config is valid. You can check at this point if the front end is available at http://clo.dev.cdhsc.org.
Be sure to check the server_name directive. It defaults to a non-existent clo.dev2.cdhsc.org.
At this point, speak to the DevOps team to setup DNS if you have not already.
Enable SSL/HTTPS Support
`sudo certbot --nginx`
Answer all the questions as given, and be sure to choose the redirect option for the final question. Please note that changes to the DNS can take up to 5 minutes to propogate. Also, make sure your browser is not using a cached version of the site if you are not seeing the changes you expected. It is common to use the "private mode" of a browser for troubleshooting of this type.
sudo nginx -s reload
Production Nginx/SSL
The production CLO VM has a special requirement - it must support two domains: https://clo.cdhsc.org, and https://carlyleletters.dukeupress.edu. In order for this to work, the Nginx config file in /etc/nginx/sites-enabled must have them both listed in the 'server' direcive. Once they are both in the config file, use certbot twice to get SSL support for both domains.
Expectations
Clear and concise commit messages are a must. Let's all shoot for unambiguous language.
Here are some examples of insufficient messages...
Making some changes.
Fixed a bug.
Let's fix those up a bit...
Made a change that fixes rendering errors associated with database field.
Fixed a bug in Authentication Fields.
Please make a serious attempt to be both thorough and brief. Describe what changes were made, describe which bug was fixed, etc.
A good rule of thumb is that messages should have a specific object. Also, for new members especially, it could help to format your messages in a way that completes this sentence:
If applied, this commit will...
Changelog
It is vital that we understand what's been changed and when, not only for developmental purposes, but for our project managers as well so they can obtain this neccessary information promptly when needed to report them on grant requests. All major changes should be documented in the changelog.
- Jerrod Mathis
- Caleb Kitzmann
- Prithvi Tippabhatla
- Joshua Nelson
- Kenny Johnson
- Ian McDowell
- Tyron Schultz
- Mitchell Lambert
- Stella Masucci
-
[ATTENTION] It is absolutely important at all times to understand your dependency structure. It is important to document if a command is run with
sudo. It is important to understand whether Node is using a globally installed package, a package it knows to install w/npm install, or if the dependency is not being tracked by node at all. -
If working on a feature branch other than master, you can checkout that branch via
git checkout <branch-name>. Use the '-b' flag to create a NEW branch.