As you may have noticed while using
corn.stanford.edu, you have limited access to the filesystem. That is, you don't have the permissions to wipe out another user's home directory. And you also don't have the ability to install programs that run system-wide; imagine if someone replaced the system
cat program with a similarly-named program that, say, printed cat emoji instead of concatenated files.
But luckily, you can install a program to run from inside your home directory, which is good enouhg for our purposes. You just have to do a few steps to configure your "environment" to use these programs from a private directory of yours.
The instructions here go through the steps of installing pup, a command-line HTML parser. The steps would be the same for installing and accessing any program, not just pup:
If you visit
/bin, i.e. the
bin directory that's right at the "root" of the system and list the files, you'll notice a lot of your favorite programs, such as cat and rm. Your shell (e.g. bash) has been configured to look inside
/bin when you run a command such as
cat file.txt. Without this configuration, the shell would have no idea where to find the
cat program on the computer.
So the first thing we need to do is create a place for the shell to look for. We want to do this from our home directory. For the purposes of this class, we'll use
bin_compciv as the directory name:
The pup program can be found as a zipped file at this address (as of January 2015):
Go ahead and
~/bin_compciv and download and unzip the zip file:
cd ~/bin_compciv curl -Lo pup.zip \ https://github.com/EricChiang/pup/releases/download/v0.3.7/pup_linux_amd64.zip unzip pup.zip
-L option for
curl is needed to follow where that URL actually redirects)
You'll see this output from the
Archive: pup_linux.zip extracting: pup
ls will show that a file named
pup is now in your directory. You can go ahead and delete the
pup.zip file, too.
In one sense, the pup program is "installed" for our usage. You can run it like so:
curl -s http://example.com | ~/bin_compciv/pup 'h1'
Which should output the relevant HTML as text:
<h1> Example Domain </h1>
But try running pup without the reference to the directory that it currently lives in (i.e.
~/bin_compciv). The shell should return with an error like:
pup: command not found (23) Failed writing body
What gives? As I mentioned earlier, your shell environment needs to be configured to know that
~/bin_compciv is a path (i.e. directory) in which the shell should search to find executable programs.
(Note: This subsection just contains a little explanation of why we're doing what we're doing. There's nothing to actually do.)
PATH variable contains the list of paths in which the shell will look for executable programs.
You can see what it currently contains simply by __echo__ing it:
The response will look something (but not exactly) like:
It's hard to tell, but the colon symbol separates the different paths. Pipe that long string of output into
tr and turn the colons into newlines:
echo $PATH | tr ':' '\n'
The output will look something like:
/usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /bin
/bin is part of the
PATH variable (at the very bottom). Each time you run a command that invokes the name of a program, whether it is
pup, bash searches for that name starting from the directory at the top of the list to the very bottom.
Run the following command:
Unless you've altered your workarea to have another
cat program (for whatever reason), the response should be:
That response means that when you want to execute the
cat program, the shell actually executes
/bin/cat. Now, if you were to place a new program named
cat anywhere higher on the path, for example
which cat would return:
So what does
PATH mean for us and for pup? The most direct and obvious answer: move
pup into one of the existing
mv ~/bin_compciv/pup /bin/pup
corn.stanford.edu, you (and me and every other average Stanfordian) are just a peon without the privileges to dump things into
/bin, so you'll receive this error:
mv: cannot create regular file '/bin/pup': Permission denied
So let's extend the path with this line, which you can add to
~/.bashrc using the nano text editor (put this line somewhere near the bottom):
If you can't be bothered to open up nano, then just run this one-liner below to append to
echo 'export PATH="$HOME/bin_compciv:$PATH"' >> ~/.bashrc
If you're curious as to what
PATH is set to, run this, and you'll see that
~/bin_compciv is now at the beginning of the
For the changes to
~/.bashrc to take effect, you can either logout and log back in. Or run
source ~/.bashrc. If you then
echo $PATH, you should see
To make sure it works, the following should execute without error:
curl -s http://example.com | pup 'h1'
Congrats, you now have a place –
~/bin_compciv – in which you can download self-contained programs into and use from your
If you have a personal laptop that runs on Linux or OS X, you probably don't need to jump through these hoops. Why not? Because you are presumably the owner of your laptop and have administrative privileges – you're free to install software wherever you want on your own laptop.
wget https://bitbucket.org/neilb/csvfix/get/version-1.6.zip unzip version-1.6.zip && rm version-1.6.zip cd neilb-csvfix-e804a794d175 make lin cp ./csvfix/bin/csvfix ~/bin_compciv/
# download directly into ~/bin_compciv curl http://stedolan.github.io/jq/download/linux64/jq -o ~/bin_compciv/jq # give it executable permissions chmod a+x ~/bin_compciv/jq
A command-line tool for turning video files into GIFs. Warning: this tool runs a bit slow on corn.stanford.edu, so try not to gifify more than a few seconds of video.
gifify has several dependencies that need to be installed beforehand:
ffmpeg - corn.stanford.edu does not have this video library. It does, however, have avconv, so we can hack around the issue by symlinking (i.e. when gifify tries calling ffmpeg, it will call avconv instead):
ln -s /usr/bin/avconv ~/bin_compciv/ffmpeg
giflossy - A fork of the gifsicle library used to work with GIFs
git clone https://github.com/pornel/giflossy cd giflossy autoreconf -i ./configure make cd src /usr/bin/install -c gifsicle gifview gifdiff ~/bin_compciv/
node - corn.stanford.edu has nodejs installed, but refers to it as "nodejs", whereas gifify is expecting "node". So we do another symlink:
ln -s /usr/bin/nodejs ~/bin_compciv/node
Finally, we install gifify
cd ~/bin_compciv npm install gifify ln -s node_modules/gifify/bin/gifify ./gifify
# first download a movie file curl -o puppies.mp4 http://stash.compciv.org/multimedia/spinning-dogs.mp4 # then run gifify and save the gif to your web directory gifify puppies.mp4 -o ~/WWW/puppies.gif # see it at: http://www.stanford.edu/~your_sunet_id/puppies.gif
From a standard movie file:
curl http://stash.compciv.org/multimedia/spinning-dogs.mp4 | \ gifify -o ~/WWW/piped-puppies.gif --text "WHO LOVES PIPES?!"
Piping in from youtube-dl:
youtube-dl hmJ1gp2iHV0 -o - | \ gifify -o ~/WWW/youtube-puppies.gif --text "PUPPIES PIPED IN YOUR TUBES"
Combined with the sox package that's installed on FarmShare, the bpm-tools utility lets you detect the number of beats per minute in a given sound clip.
curl http://www.pogo.org.uk/~mark/bpm-tools/releases/bpm-tools-0.3.tar.gz \ | tar xz cd bpm-tools-0.3/ make && make install PREFIX=~/bin_compciv cd ~/bin_compciv mv ./bin/bpm* .
The sox program (as configured by default) can't read MP3s, so we use the
avconv library to convert to a
.wav file, then run
sox test.wav and pipe it into bpm:
avconv -i test.mp3 -f wav test.wav sox test.wav -t raw -r 44100 -e float -c 1 - | bpm
cd ~/bin_compciv wget https://yt-dl.org/downloads/2015.01.11/youtube-dl chmod a+x youtube-dl
youtube-dl hmJ1gp2iHV0 --restrict-filenames