Programming

Set Up an Ubuntu VM for Standard Ebooks Tools on a Mac

Update: This Guide is Deprecated Because the Tools Now Support the Mac Natively


(Deprecated -- just use the tools directly on your Mac now!)

This guide will walk you through the exact steps to get all dependencies set up for Standard Ebooks (SE) book production work on a Mac.

This guide assumes your Mac is around 8 years old or newer, so that it is 64-bit. If you are running anything newer than Snow Leopard then that is probably the case. (This was tested on MacOS 10.12 Sierra.)

Convenient, cross-platform development

With Vagrant's shared folder (synced folder) feature, the files are hosted in a normal folder on your Mac, and mounted to /vagrant/ inside the VM. You can use all of your Mac tools (iBooks for viewing .epub files, your favorite text editor(s), visual git tools such as Xcode's FileMerge or Atlasian SourceTree, etc.) to work on the files (and manipulate the Git repo) and still run the SE command line tools (build-images, build, typogrify, etc.) from inside the Ubuntu VM. It is kind of magical.

First, install VirtualBox and Vagrant

Vagrant Ubuntu VMs

For this guide, we will be using Xenial Xerus, or Ubuntu 16.04 LTS.

For more info:

For more Vagrant Ubuntu VMs:

According to Stack Overflow the VM downloaded for us by Vagrant will be hidden in your user folder here:

~/.vagrant.d/boxes

Open Terminal and decide where to put your working directory and Vagrantfile on your Mac

For this guide we will assume you are putting it in ~/Documents/setools/

In a Terminal window (do not type the $ belowl; it is just an indicator of what commands are typed into the shell.

$ cd ~/Documents

At this point you may see a prompt like this:

my-mac:Documents my-username$

but for simplicity I will just write $

Now make the folder and cd into it

$ mkdir setools && cd setools

Have Vagrant install the VM for us

$ vagrant init ubuntu/xenial64

This created a Vagrantfile for us.

IMPORTANT: increase the VM's RAM to 1024 MiB so GCC does not crash!

Before we proceed, we edit the Vagrantfile to get the VM to have enough RAM for purposes. Find the section like the following, and uncomment a few lines so it looks like this:

config.vm.provider "virtualbox" do |vb|
  # Display the VirtualBox GUI when booting the machine
  #vb.gui = true
  # Customize the amount of memory on the VM:
  vb.memory = "1024"
end

Ready to download and boot

Now we wait for it to download and install (bandwidth usage of about 450 MiB? + another 1,500 MiB after dependencies install -- but on-disk is smaller when we are done, around 290 MiB) and boot the VM (still on the Mac side of the Terminal, in the ~/setools/ folder):

$ vagrant up --provider virtualbox

This Mac folder (~/Documents/setools) and the Vagrant VM share a folder which stays there on the Mac side when the VM is turned off. It is mounted in the Ubuntu VM's filesystem at /vagrant/

Note that the Vagrant VM / Virtualbox .box file lives in ~/.vagrant.d/boxes and not in this mostly empty folder, with just the Vagrantfile

Time passes...

Let Vagrant do its download business.

Ready to log in finally

Inside Terminal, from our (Mac) working folder (where the Vagrantfile lives), we can log in with:

$ vagrant ssh

Ubuntu prints this friendly welcome screen each time we login:

Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-81-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

ubuntu@ubuntu-xenial:~$

The user is vagrant and vagrant ssh will log us in without having to type a password.

Congratulations, you are over the hump.

Inside the Ubuntu VM

Commands shown below are assumed to be entered into the VM's command line, once you are logged in (after vagrant ssh. You may see something like:

ubuntu@ubuntu-xenial:~$

But again I will just write $ for the prompt, where you type the stuff besides the $

Verify RAM settings

$ top

Top is like Activity Monitor on the Mac. The fourth line should say something like

KiB Mem :  1015968 total,   640844 free,    51932 used,   323192 buff/cache

As long as we have >= 1,000,000 KiB (>= 1 GiB) we should be fine.

(press q to quit)

Get all the dependencies, in the right order.

First we make sure our package sources are up to date

$ sudo apt update

And wait a little bit.

Now we get Debian package dependencies, using APT

$ sudo apt install python3-pip xsltproc libxml2-utils xmlstarlet libxml-xpath-perl recode html-xml-utils librsvg2-bin libimage-exiftool-perl zip epubcheck calibre default-jre

It will print a bunch of package names, then at the bottom:

After this operation, 1,044 MB of additional disk space will be used.

Press y to coninue and then wait for a while as the packages are downloaded and installed for you.

More dependencies

To help with the Python image library stuff (pacakge Pillow) for Ubuntu 16 (see https://pillow.readthedocs.io/en/latest/installation.html#building-on-linux.

$ sudo apt install libtiff5-dev libjpeg8-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python-tk

(May have to press y to continue)

In order to get Python lxml package to install, and gitdb, which needs realpath (and to use build-images) we need a few more packages (and their dependencies):

$ sudo apt install libxml2-dev libxslt-dev realpath inkscape

Panoply of Python Packages

Now we install the SETools' Python dependencies, which should have their dependencies satisfied by the above package installs.

$ sudo pip3 install pyhyphen roman titlecase beautifulsoup4 smartypants pillow gitpython cssselect regex lxml gitdb termcolor terminaltables

Install the fonts for the book covers / title pages

mkdir -p ~/.fonts/
wget -O ~/.fonts/LeagueSpartan-Bold.otf "https://github.com/theleagueof/league-spartan/blob/master/LeagueSpartan-Bold.otf?raw=true"
wget -O ~/.fonts/OFLGoudyStM.otf "https://github.com/theleagueof/sorts-mill-goudy/blob/master/OFLGoudyStM.otf?raw=true"
wget -O ~/.fonts/OFLGoudyStM-Italic.otf "https://github.com/theleagueof/sorts-mill-goudy/blob/master/OFLGoudyStM-Italic.otf?raw=true"
sudo fc-cache -fv

Continuing with Step 1 of the SE Producing an Ebook guide

Install hyphenation dictionaries for the pyhyphen library:

$ sudo python3 -c "exec(\"from hyphen import dictools\\ndictools.install('en_GB')\\ndictools.install('en_US')\")"

And of course: install Git for the Ubuntu command line

$ sudo apt install git

Now for the 'easy' part of step 1 of the guide!

$ cd ~/
$ git clone https://github.com/standardebooks/tools.git

We make a working folder that is shared between the Mac and the Ubuntu side:

$ cd /vagrant/
$ mkdir books
$ cd books

Test an existing book

Let's test that the tools work on an existing repo:

$ cd /vagrant/books/
$ git clone https://github.com/standardebooks/henry-david-thoreau_walden.git
$ cd henry-david-thoreau_walden/
$ ~/tools/build-images .

You can open the vectorized SVG files on your Mac, in the Finder.

Navigate to ~/Documents/setools/books/henry-david-thoreau_walden/images and press the space bar to use MacOS's built-in Quick Look preview ability to see the SVG file, complete with backtground art and vector text.

Cool tip: Install the same fonts (below) on your Mac

Then you can use Finder's space bar quick look feature to preview .SVG source files (not the ones generated by the build-images script) and you can see how the book cover will look, before you even run the build-images script.

Test the epub build process

Continuing in the Thoreau / Walden folder:

$ ~/tools/build --output-dir="dist" --kindle --check .

The End

On your Mac, use the Finder to navigate to the Documents/setools/books folder. Drill down to the books/whoever-whatever/dist folder and double-click the .epub file to open it in iBooks.

Congratulations: you have built an .epub file (~1 MiB) with the Standard Ebooks tools (290 MiB VM on disk).

Now you can make your own skeleton(s) for your own book(s):

$ cd /vagrant/books/
$ ~/tools/create-draft --author="John Doe" --title="Anonymous Cowards"

Have fun with steps 2–21!

Next steps:

You are done with step 1 of the Producing an Ebook Step by Step guide.

(Don't worry: it is a lot more fun and more interesting from here on out. And hopefully this tutorial has been explicit enough that you will not have had to do any distracting, undocumented Side Quests!)

Small tip

If you are proofreading an ebook and making changes to it, then syncing and pulling in the file from Dropbox or iCloud Drive, and if your iPad iBooks does not show the updated version, try this:

  1. Delete the old version from iBooks on your iPad.
  2. Before you sync it in your Dropbox, or iCloud Drive, rename it with an increasing version number: h-d-t-walden-42.epub, then after another change, h-d-t-walden-43.epub.
  3. Now grab the newest file, h-d-t-walden-43.epub from Dropbox (for example) and Export > Open in ... > Send to iBooks
  4. When you tap to open, it should have the up-to-date draft you are working on!

(Obviously you don't have to release a weirdly-numbered .epub, this is just a hack to get around iPad/iBooks inability to use the newer file, each time you make changes. iBooks on the Mac seems well behaved if we just delete and then double click the .epub in the Finder to re-import.)


Optional, Useful Tidbits

Vagrant Tips and Tricks

If you change your Vagrantfile (use less RAM -- untested) and want to restart the VM:

(inside of the Ubuntu VM, opposite of vagrant ssh)

$ exit

In Mac Terminal, in your ~/Documents/setools/ folder

$ vagrant reload

Once it finishes booting:

$ vagrant ssh

To shutdown:

In the Terminal, in the (Mac side) folder where your Vagrant file lives (e.g. ~/Documents/setools)

$ vagrant suspend

To boot it from suspended state:

In the Terminal, in the (Mac side) folder where your Vagrant file lives (e.g. ~/Documents/setools)

$ vagrant up

And to log back in:

$ vagrant ssh

While you are editing Ebooks, you may want to know...

How to type real, typographic, “smart,” curved quotes on your Mac

Option + [ “ (double left quote)
Shift + Option + [ ” (double right quote)
Option + ] ‘ (single left quote)
Shift + Option + ] ’ (single right quote, and real typographic curved apostrophe, too)

Optionally: to install emacs (command-line no-X version) if you do not like the nano text editor:

$ sudo apt install emacs24-nox

but you can just use whatever you have on your Mac, in the Terminal (or any App / Finder, etc.) on the ~/Documents/setools side

Using command-line git to see word-by-word or character-by-character diffs:

Works on the Mac or Linux side:

$ git diff --word-diff

Assuming you have Xcode installed, run from the Mac side, in Terminal:

$ git difftool -t opendiff -y

or

$ git difftool -t opendiff

Go into Preferences and make sure Wrap Text is checked. Also, you may need to resize the window to have the wrap text feature work properly.


(Deprecated -- just use the tools directly on your Mac now!)