.. !split

Writing in private repository while publishing in public
========================================================

.. index:: private repos

.. index:: mirroring repos

Sometimes you want to keep ongoing writing in a *private* repository
and make only *selected* chapters and/or files publicly visible.
In such cases
one can set up the book project structure in a private repository, but
use a public repository instead of the ``doc/pub`` directory for publishing
selected compiled documents. This is easy: just change the ``dest=``
line, where the publishing directory is defined, in all ``make*.sh``
scripts in ``doc/src/chapters``. The files will then be copied to
this alternative destination.

Often, you want to publish the software associated with the book
project, stored in ``doc/src/chapter/nickname/src-nickname``, as a part
of the public repository. Such files can also easily be copied, say to
``src/nickname`` in the public repository. However, software files often
change names and locations in subdirectories, and then you need to be
very careful with updating the Git commands in the public repository
every time you do ``git add`` or ``git rm`` locally in the private
repository.  This problem occurs with text files too, but maybe less
often, so the recipe given below applies to all kind of files you want
to mirror from a private to a public repository.

We have made a script `rsync_git.py <https://github.com/hplgit/setup4book-doconce/tree/master/doc/src/chapters/rules/src-rules/rsync_git.py>`__
that can copy files from one repository to another and log files that
are removed or deleted and then take the appropriate Git actions.
Running

.. code-block:: text

        Terminal> rsync_git.py src-mychap $HOME/repos/pub/mybook/src/mychap

will copy all files from ``src-mychap`` to ``$HOME/repos/pub/mybook/src/mychap``,
find which files that are new in ``src-mychap`` and
must be added to the destination directory, and which files
that are removed in ``src-mychap`` and should be removed in the destination
directory as well.
An ``rsync`` command is run to the physical copy and removal of
files, followed by ``git add`` and ``git rm``
commands. In this way, you can automatically keep the public repository
as a mirror *of parts* of your private repository! [#rsync-git]_

.. [#rsync-git] This functionality should be a part of Git, but no Git expert I have talked to has ever seen use for merging a flexibly defined subset of a repository with another repository. (The current functionality of Git is not capable of working with, e.g., branches that merge with only parts of another branch.)

The ``rsync_git.py`` script is listed below for reference. Note that
a file ``$HOME/.rsyncexclude`` can be made to filter out certain files
that you never want to copy (this is always a good idea!).

.. code-block:: python

        #!/usr/bin/env python
        """
        Sync two directory trees with rsync and perform corresponding
        git operations (add or rm).
        Skip files listed in $HOME/.rsyncexclude.
        
        Usage:   rsync_git.py from-dir to-dir
        Example: rsync_git.py src-mychap $HOME/repos/pub/mybook/src/mychap
        
        The from-dir is the source and the to-dir is the destination
        (e.g. a public directory where resources are exposed).
        The script must be run from a dir within the repo of to-dir.
        """
        
        # Typical rsync output:
        """
        sending incremental file list
        deleting decay7.py
        decay_TULL.py
        
        sent 675 bytes  received 34 bytes  1418.00 bytes/sec
        total size is 94788  speedup is 133.69
        """
        
        # Example on $HOME/.rsyncexclude file
        """
        .#*
        #*
        *.rsync~
        *.a
        *.o
        *.so
        *~
        .*~
        *.log
        *.dvi
        *.aux
        *.old
        tmp_*
        .tmp*
        *.tar
        *.tar.gz
        *.tgz
        *.pyc
        """
        
        import commands, os, sys
        
        from_ = sys.argv[1]
        to_ = sys.argv[2]
        cmd = 'rsync -rtDvz -u -e ssh -b ' + \
              '--exclude-from=$HOME/.rsyncexclude ' + \
              '--suffix=.rsync~ --delete --force %s/ %s' % (from_, to_)
        print cmd
        failure, output = commands.getstatusoutput(cmd)
        print output
        
        delete = []
        add = []
        for line in output.splitlines():
            relevant_line = True
            for text in 'sending incremental file list', \
                'sent ', 'total size is':
                if line.startswith(text):
                    relevant_line = False
            if relevant_line and line != '':
                if line.startswith('deleting'):
                    delete.append(line.split()[1])
                else:
                    add.append(line.strip())
        
        print delete
        print add
        
        for filename in delete:
            option = '-rf' if os.path.isdir('%s/%s' % (to_, filename)) else '-f'
            cmd = 'git rm %s %s/%s' % (option, to_, filename)
            print cmd
            os.system(cmd)
        for filename in add:
            cmd = 'git add %s/%s' % (to_, filename)
            print cmd
            os.system(cmd)