README
¶
enm - DLTS Enhanced Network Monographs
CLI application for generating the Enhanced Networked Monographs (ENM) static website and Solr index.
Overview
enm is a CLI application for performing various backend ENM functions:
- Create static pages
- About
- Home
- Create browse topics lists: e.g. featured topics
- Create topic pages: e.g. culture -- popular
- Load
enm-pagesSolr index - Automatically create cached data files which can be used in place of the Postgres database in subsequent jobs: nyudlts/enm-cache
The enm program has not been productionized. Enough development was done to
create an initial stable and correct demo site. ENM data is currently frozen and
there are no active plans to add or change data at this time.
Getting Started
Prerequisities
Database
The Postgres database from TCT should already be set up on devdb1.dlib.nyu.edu with the correct user credentials. See https://jira.nyu.edu/jira/browse/NYUP-437 for details.
Installation and setup
Installation using go get:
go get github.com/nyulibraries/dlts-enm
cd dlts-enm/
git remote rm origin
git remote add origin git@github.com:NYULibraries/dlts-enm.git
git fetch --all
dep ensure
go build
mv dlts-enm enm
Installation using git clone:
mkdir -p $GOPATH/src/github.com/NYULibraries/
cd $GOPATH/src/github.com/NYULibraries/
git clone git@github.com:NYULibraries/dlts-enm.git
cd dlts-enm/
dep ensure
go build
mv dlts-enm enm
Set environment variables
Set environment variables for database access:
export ENM_POSTGRES_DATABASE=enm
export ENM_POSTGRES_DATABASE_HOSTNAME=127.0.0.1
export ENM_POSTGRES_DATABASE_USERNAME=enm_readonly
export ENM_POSTGRES_DATABASE_PASSWORD=[password for devdb1.dlib.nyu.edu:enm database for user enm_readonly]
Note that we use 127.0.0.1 even though the database is remote because we'll need to access the remote Postgres server through an SSH tunnel through the bastion host.
These environment variables must be set before running any enm command, even
the ones that do not technically need database access. Failure to do so will
cause a panic.
Set location of the cache using ENM_CACHE:
export ENM_CACHE=$HOME/enm-cache/
This environment variable is optional. If it is set and the path that is pointed
to does not already exist, enm will create it, along with any needed intermediate
directories.
If ENM_CACHE is not set, the location of the cache defaults to /tmp/enm-cache/.
Set up SSH tunnel to devdb1
Set up SSH tunneling to port 5432 on Postgres host devdb1.dlib.nyu.edu through bastion host by running this command in a separate terminal:
ssh -N -L 5432:devdb1.dlib.nyu.edu:5432 [USERNAME]@b.dlib.nyu.edu
This will allow remote database access via local port 5432.
A less verbose command can be run if the following is set up in .ssh/config:
Host devdb1
Hostname devdb1.dlib.nyu.edu
ProxyCommand ssh bastion -W %h:%p
User [USERNAME]
...
Host bastion
Hostname b.dlib.nyu.edu
User [USERNAME]
In a separate terminal, run this command:
ssh -N -L 5432:devdb1:5432 bastion
Usage
Deploy site
There is a deploy script that can generate the full website and copy it to the dev,
stage, and prod web servers. The easiest way to use it is to run the wrapper script
that will prompt for information and then run the deploy script with the proper
options and credentials. The rsync commands are run using expect
to automatically enter the user credentials for the bastion and web server hosts.
bin/deploy-site_expect.sh devbin/deploy-site_expect.sh stagebin/deploy-site_expect.sh prod
The deploy script runs all the sitegen commands detailed below with destination
set to directories in dist/.
See examples for a full demonstration of how to use this wrapper script.
Note that the deploy script does static page generation and copying to server only, it does not perform Solr indexing.
Get general help
./enm help
Get help on a command
./enm help sitegen
Get help on a subcommmand
./enm help sitegen browsetopicslists
Create browse topics lists using Postgres database
./enm sitegen browsetopicslists --destination=[DESTINATION]
Create browse topics lists using cache files
./enm sitegen browsetopicslists --destination=[DESTINATION] --source=cache
Create page About, Home, etc.
./enm sitegen sitepages --destination=[DESTINATION]
Create all topic pages using Postgres database
This automatically creates cache files in /tmp/enm-cache/sitegen-topicpages/ that can be used as the data source for subsequent topic pages generation runs:
./enm sitegen topicpages --source=database --destination=[DESTINATION]
Create all topic pages using cache files in /tmp/enm-cache/sitegen-topicpages/
./enm sitegen topicpages --source=cache --destination=[DESTINATION]
Create topic pages for 2 topics only using Postgres database
./enm sitegen topicpages --source=database --destination=[DESTINATION] [TOPIC ID 1] [TOPIC ID 2]
Create topic pages for 2 topics only using cache files
./enm sitegen topicpages --source=cache --destination=[DESTINATION] [TOPIC ID 1] [TOPIC ID 2]
Load enm-pages Solr index using Postgres database
./enm solr load --server=[SOLR SERVER] --port 8983
Load enm-pages Solr index using cache files
./enm solr load --server=[SOLR SERVER] --source=cache --port 8983
Examples
Create the production website from latest cache files cloned to $HOME/enm-cache/ and deploy to the dev web server
In the example below, it is assumed that the dlts-enm repo is located at
$GOPATH/src/github.com/nyulibraries/dlts-enm/, and the https://github.com/nyudlts/enm-cache
repo has already been cloned to $HOME.
$ export ENM_CACHE=$HOME/enm-cache/
$ bin/deploy-site_expect.sh dev
Do complete regeneration of the site before copying to server? [y/n] y
Use the cache for regenerating the site? [y/n] y
Username for b.dlib.nyu.edu and devweb1.dlib.nyu.edu: someuser
Password for b.dlib.nyu.edu and devweb1.dlib.nyu.edu:
spawn /Users/someuser/Documents/programming/go/gopath/src/github.com/nyulibraries/dlts-enm/bin/deploy-site.sh -c -g -u arjanik dev
Generating site pages...
Generating browse topics lists...
Generating topic pages...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
[...SNIPPED...]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
someuser@b.dlib.nyu.edu's password:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
[...SNIPPED...]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
someuser@devweb1.dlib.nyu.edu's password:
building file list ... done
about.html
sent 127 bytes received 66 bytes 77.20 bytes/sec
total size is 2.46K speedup is 12.76
rsync #1 completed successfully.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
[...SNIPPED...]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
someuser@b.dlib.nyu.edu's password:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
[...SNIPPED...]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
someuser@devweb1.dlib.nyu.edu's password:
building file list ... done
index.html
sent 127 bytes received 66 bytes 128.67 bytes/sec
total size is 2.45K speedup is 12.69
rsync #2 completed successfully.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
[...SNIPPED...]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
someuser@b.dlib.nyu.edu's password:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
[...SNIPPED...]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
someuser@devweb1.dlib.nyu.edu's password:
building file list ... done
./
0-9.html
a.html
b.html
c.html
d.html
e.html
enm-picks.html
f.html
g.html
h.html
i.html
j.html
k.html
l.html
m.html
n.html
non-alphanumeric.html
o.html
p.html
q.html
r.html
s.html
t.html
u.html
v.html
w.html
x.html
y.html
z.html
sent 1.76K bytes received 69.40K bytes 28.46K bytes/sec
total size is 8.47M speedup is 119.01
rsync #3 completed successfully.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
[...SNIPPED...]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
someuser@b.dlib.nyu.edu's password:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
[...SNIPPED...]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
someuser@devweb1.dlib.nyu.edu's password:
building file list ... done
sent 2.50K bytes received 20 bytes 1.68K bytes/sec
total size is 27.13M speedup is 10761.88
rsync #4 completed successfully.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
[...SNIPPED...]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
someuser@b.dlib.nyu.edu's password:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: UNAUTHORIZED PERSONS ........ DO NOT PROCEED
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
[...SNIPPED...]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
someuser@devweb1.dlib.nyu.edu's password:
building file list ... done
./
00/
00/00/
00/00/00/
00/00/00/00/
00/00/00/00/0000000002.html
00/00/00/00/0000000003.html
00/00/00/00/0000000004.html
00/00/00/00/0000000005.html
[...SNIPPED...]
sent 2.74M bytes received 3.08M bytes 125.19K bytes/sec
total size is 222.68M speedup is 38.25
rsync #5 completed successfully.
ENM site deployment completed.
Load prod Solr index from Postgres database
./enm solr load --server=discovery1.dlib.nyu.edu --port 8983
Load prod Solr index from cache files at $ENM_CACHE if set, or default cache location /tmp/enm-cache/
./enm solr load --server=discovery1.dlib.nyu.edu --source=cache --port 8983
Running the tests
Make sure to set up access to the Postgres database before running the tests. See Set environment variables.
go test ./...
Generation of code files in db/postgres/models
The Go code in db/postgres/models was generated automatically by xo
using custom xo templates. If changes are made to the Postgres database,
the models can be updated by running go generate at the root of the project.
Configuration
Configuration is done through command/subcommand options and environment variables, which must be set before running any commands, including those that do not require database access.
Future improvements
- Real error handling/recovery/messaging/logging instead of the liberal use of
paniccalls - (maybe) Embedding of
sitegentemplates into theenmbinary using something like https://github.com/jteeuwen/go-bindata (for motivation see comment insitegen/sitegen.go) - Write more tests, and stub out Postgres in the tests in the
solrandsitegenpackages (and in all future tests).
ENM project Github repos
- dlts-enm
- dlts-enm-search-application
- dlts-enm-tct-backend
- dlts-enm-tct-developer
- dlts-enm-tct-frontend
- dlts-enm-verifier
- dlts-enm-web
Built With
License
This project is licensed under the Apache License Version 2.0 - see the LICENSE file for details.
Documentation
¶
There is no documentation for this package.
Source Files
Directories