Installing programs for your personal Stanford account
As you may have noticed while using corn.stanford.edu, you have limited access to the filesystem. That is, you don't have the permissions to wipe out another user's home directory. And you also don't have the ability to install programs that run system-wide; imagine if someone replaced the system cat program with a similarly-named program that, say, printed cat emoji instead of concatenated files.
But luckily, you can install a program to run from inside your home directory, which is good enouhg for our purposes. You just have to do a few steps to configure your "environment" to use these programs from a private directory of yours.
General installation steps
The instructions here go through the steps of installing pup, a command-line HTML parser. The steps would be the same for installing and accessing any program, not just pup:
1. Create the ~/bin_compciv directory
If you visit /bin, i.e. the bin directory that's right at the "root" of the system and list the files, you'll notice a lot of your favorite programs, such as cat and rm. Your shell (e.g. bash) has been configured to look inside /bin when you run a command such as cat file.txt. Without this configuration, the shell would have no idea where to find the cat program on the computer.
So the first thing we need to do is create a place for the shell to look for. We want to do this from our home directory. For the purposes of this class, we'll use bin_compciv as the directory name:
mkdir ~/bin_compciv
2. Download the pup program
The pup program can be found as a zipped file at this address (as of January 2015):
https://github.com/EricChiang/pup/releases/download/v0.3.7/pup_linux_amd64.zip
Go ahead and cd into ~/bin_compciv and download and unzip the zip file:
cd ~/bin_compciv
curl -Lo pup.zip \
https://github.com/EricChiang/pup/releases/download/v0.3.7/pup_linux_amd64.zip
unzip pup.zip
(the -L option for curl is needed to follow where that URL actually redirects)
You'll see this output from the unzip command:
Archive: pup_linux.zip
extracting: pup
Running ls will show that a file named pup is now in your directory. You can go ahead and delete the pup.zip file, too.
3. Running pup
In one sense, the pup program is "installed" for our usage. You can run it like so:
curl -s http://example.com | ~/bin_compciv/pup 'h1'
Which should output the relevant HTML as text:
<h1>
Example Domain
</h1>
But try running pup without the reference to the directory that it currently lives in (i.e. ~/bin_compciv). The shell should return with an error like:
pup: command not found
(23) Failed writing body
What gives? As I mentioned earlier, your shell environment needs to be configured to know that ~/bin_compciv is a path (i.e. directory) in which the shell should search to find executable programs.
4a. Understanding the PATH environment variable
(Note: This subsection just contains a little explanation of why we're doing what we're doing. There's nothing to actually do.)
The PATH variable contains the list of paths in which the shell will look for executable programs.
You can see what it currently contains simply by __echo__ing it:
echo $PATH
The response will look something (but not exactly) like:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
It's hard to tell, but the colon symbol separates the different paths. Pipe that long string of output into tr and turn the colons into newlines:
echo $PATH | tr ':' '\n'
The output will look something like:
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
Notice that /bin is part of the PATH variable (at the very bottom). Each time you run a command that invokes the name of a program, whether it is cat or echo or pup, bash searches for that name starting from the directory at the top of the list to the very bottom.
Run the following command:
which cat
Unless you've altered your workarea to have another cat program (for whatever reason), the response should be:
/bin/cat
That response means that when you want to execute the cat program, the shell actually executes /bin/cat. Now, if you were to place a new program named cat anywhere higher on the path, for example /usr/local/bin, running which cat would return:
/usr/local/bin/cat
4b. Extend the PATH environment variable
So what does PATH mean for us and for pup? The most direct and obvious answer: move pup into one of the existing PATH directories:
mv ~/bin_compciv/pup /bin/pup
But on corn.stanford.edu, you (and me and every other average Stanfordian) are just a peon without the privileges to dump things into /bin, so you'll receive this error:
mv: cannot create regular file '/bin/pup': Permission denied
So let's extend the path with this line, which you can add to ~/.bashrc using the nano text editor (put this line somewhere near the bottom):
export PATH="$HOME/bin_compciv:$PATH"
If you can't be bothered to open up nano, then just run this one-liner below to append to ~/.bashrc
echo 'export PATH="$HOME/bin_compciv:$PATH"' >> ~/.bashrc
If you're curious as to what PATH is set to, run this, and you'll see that ~/bin_compciv is now at the beginning of the PATH variable:
echo "$HOME/bin_compciv:$PATH"
5. Running programs from ~/bin_compciv
For the changes to ~/.bashrc to take effect, you can either logout and log back in. Or run source ~/.bashrc. If you then echo $PATH, you should see ~/bin_compciv.
To make sure it works, the following should execute without error:
curl -s http://example.com | pup 'h1'
Congrats, you now have a place – ~/bin_compciv – in which you can download self-contained programs into and use from your corn.stanford.edu account.
If you have a personal laptop that runs on Linux or OS X, you probably don't need to jump through these hoops. Why not? Because you are presumably the owner of your laptop and have administrative privileges – you're free to install software wherever you want on your own laptop.
Other fun software
csvfix
wget https://bitbucket.org/neilb/csvfix/get/version-1.6.zip
unzip version-1.6.zip && rm version-1.6.zip
cd neilb-csvfix-e804a794d175
make lin
cp ./csvfix/bin/csvfix ~/bin_compciv/
jq
# download directly into ~/bin_compciv
curl http://stedolan.github.io/jq/download/linux64/jq -o ~/bin_compciv/jq
# give it executable permissions
chmod a+x ~/bin_compciv/jq
gifify
A command-line tool for turning video files into GIFs. Warning: this tool runs a bit slow on corn.stanford.edu, so try not to gifify more than a few seconds of video.
Installation
gifify has several dependencies that need to be installed beforehand:
ffmpeg - corn.stanford.edu does not have this video library. It does, however, have avconv, so we can hack around the issue by symlinking (i.e. when gifify tries calling ffmpeg, it will call avconv instead):
ln -s /usr/bin/avconv ~/bin_compciv/ffmpeg
giflossy - A fork of the gifsicle library used to work with GIFs
git clone https://github.com/pornel/giflossy
cd giflossy
autoreconf -i
./configure
make
cd src
/usr/bin/install -c gifsicle gifview gifdiff ~/bin_compciv/
node - corn.stanford.edu has nodejs installed, but refers to it as "nodejs", whereas gifify is expecting "node". So we do another symlink:
ln -s /usr/bin/nodejs ~/bin_compciv/node
Finally, we install gifify
cd ~/bin_compciv
npm install gifify
ln -s node_modules/gifify/bin/gifify ./gifify
Basic usage of gifify
# first download a movie file
curl -o puppies.mp4 http://stash.compciv.org/multimedia/spinning-dogs.mp4
# then run gifify and save the gif to your web directory
gifify puppies.mp4 -o ~/WWW/puppies.gif
# see it at: http://www.stanford.edu/~your_sunet_id/puppies.gif
gifify and pipes
From a standard movie file:
curl http://stash.compciv.org/multimedia/spinning-dogs.mp4 | \
gifify -o ~/WWW/piped-puppies.gif --text "WHO LOVES PIPES?!"
Piping in from youtube-dl:
youtube-dl hmJ1gp2iHV0 -o - | \
gifify -o ~/WWW/youtube-puppies.gif --text "PUPPIES PIPED IN YOUR TUBES"
bpm-tools
Combined with the sox package that's installed on FarmShare, the bpm-tools utility lets you detect the number of beats per minute in a given sound clip.
curl http://www.pogo.org.uk/~mark/bpm-tools/releases/bpm-tools-0.3.tar.gz \
| tar xz
cd bpm-tools-0.3/
make && make install PREFIX=~/bin_compciv
cd ~/bin_compciv
mv ./bin/bpm* .
The sox program (as configured by default) can't read MP3s, so we use the avconv library to convert to a .wav file, then run sox test.wav and pipe it into bpm:
avconv -i test.mp3 -f wav test.wav
sox test.wav -t raw -r 44100 -e float -c 1 - | bpm
youtube-dl
A command-line tool for accessing video files from popular video websites
Installation
cd ~/bin_compciv
wget https://yt-dl.org/downloads/2015.01.11/youtube-dl
chmod a+x youtube-dl
Usage
youtube-dl hmJ1gp2iHV0 --restrict-filenames