Writing in private repository while publishing in public

Sometimes you want to keep ongoing writing in a private repository and make only selected chapters and/or files publicly visible. In such cases one can set up the book project structure in a private repository, but use a public repository instead of the doc/pub directory for publishing selected compiled documents. This is easy: just change the dest= line, where the publishing directory is defined, in all make*.sh scripts in doc/src/chapters. The files will then be copied to this alternative destination.

Often, you want to publish the software associated with the book project, stored in doc/src/chapter/nickname/src-nickname, as a part of the public repository. Such files can also easily be copied, say to src/nickname in the public repository. However, software files often change names and locations in subdirectories, and then you need to be very careful with updating the Git commands in the public repository every time you do git add or git rm locally in the private repository. This problem occurs with text files too, but maybe less often, so the recipe given below applies to all kind of files you want to mirror from a private to a public repository.

We have made a script rsync_git.py that can copy files from one repository to another and log files that are removed or deleted and then take the appropriate Git actions. Running

Terminal> rsync_git.py src-mychap $HOME/repos/pub/mybook/src/mychap

will copy all files from src-mychap to $HOME/repos/pub/mybook/src/mychap, find which files that are new in src-mychap and must be added to the destination directory, and which files that are removed in src-mychap and should be removed in the destination directory as well. An rsync command is run to the physical copy and removal of files, followed by git add and git rm commands. In this way, you can automatically keep the public repository as a mirror of parts of your private repository! [1]

[1]

This functionality should be a part of Git, but no Git expert I have talked to has ever seen use for merging a flexibly defined subset of a repository with another repository. (The current functionality of Git is not capable of working with, e.g., branches that merge with only parts of another branch.)

The rsync_git.py script is listed below for reference. Note that a file $HOME/.rsyncexclude can be made to filter out certain files that you never want to copy (this is always a good idea!).

#!/usr/bin/env python
"""
Sync two directory trees with rsync and perform corresponding
git operations (add or rm).
Skip files listed in $HOME/.rsyncexclude.

Usage:   rsync_git.py from-dir to-dir
Example: rsync_git.py src-mychap $HOME/repos/pub/mybook/src/mychap

The from-dir is the source and the to-dir is the destination
(e.g. a public directory where resources are exposed).
The script must be run from a dir within the repo of to-dir.
"""

# Typical rsync output:
"""
sending incremental file list
deleting decay7.py
decay_TULL.py

sent 675 bytes  received 34 bytes  1418.00 bytes/sec
total size is 94788  speedup is 133.69
"""

# Example on $HOME/.rsyncexclude file
"""
.#*
#*
*.rsync~
*.a
*.o
*.so
*~
.*~
*.log
*.dvi
*.aux
*.old
tmp_*
.tmp*
*.tar
*.tar.gz
*.tgz
*.pyc
"""

import commands, os, sys

from_ = sys.argv[1]
to_ = sys.argv[2]
cmd = 'rsync -rtDvz -u -e ssh -b ' + \
      '--exclude-from=$HOME/.rsyncexclude ' + \
      '--suffix=.rsync~ --delete --force %s/ %s' % (from_, to_)
print cmd
failure, output = commands.getstatusoutput(cmd)
print output

delete = []
add = []
for line in output.splitlines():
    relevant_line = True
    for text in 'sending incremental file list', \
        'sent ', 'total size is':
        if line.startswith(text):
            relevant_line = False
    if relevant_line and line != '':
        if line.startswith('deleting'):
            delete.append(line.split()[1])
        else:
            add.append(line.strip())

print delete
print add

for filename in delete:
    option = '-rf' if os.path.isdir('%s/%s' % (to_, filename)) else '-f'
    cmd = 'git rm %s %s/%s' % (option, to_, filename)
    print cmd
    os.system(cmd)
for filename in add:
    cmd = 'git add %s/%s' % (to_, filename)
    print cmd
    os.system(cmd)