| Print dvi | usdvi |
| Download tgz | bintgz |

WSync user manual

Christophe Deleuze

Table of Contents

1  Introduction

WSync is a web uploading tool, i.e. a kind of ``web mirroring'' program, but it operates by ``pushing'' (uploading) files rather than downloading them. If you maintain a web site, you probably generate HTML pages on a local repository on your computer, and then upload them to the web server. WSync can help you a lot.

You give WSync a list of files and it keeps track of what versions of what files have been uploaded, and can thus find out which files have changed on your local repository, and upload them automatically. It does that by maintaining locally a database of files ``signatures'' (made of a timestamp and a digest). Actual uploading is made through a ``backend'', performing network operations. Currently, only the File Transfer Protocol is available as backend (an FTP library is included in WSync). In the future, I plan to add new backends supportting other protocols, especially because FTP is known to be insecure (see [1] for details).

When you run WSync it computes the signature for all files in the local repository, and check if they match those stored in its database. It does that by first comparing timestamps. If they match, then the file hasn't changed since last upload. If they don't, it computes the digest and compares it to the database digest. If they differ, then the file has changed and needs to be uploaded, otherwise, it means that the file has been regenerated, but with no changes, and there is thus no need to upload it. This can often happen when you generate the HTML (or whatever) files from some higher level description language (such as LaTeX or XML, and in particular if you generate several HTML files from one source file).

WSync also detects when you add or remove files in the list, and uploads or remote-deletes them automatically. Thanks to the files digests it maintains, it can even detect pairs of add/remove that actually are a renaming of a file. So it can do the right thing, i.e. rename the file on the remote server, rather than deleting it under the old name and uploading it under the new name.

It also creates directories as needed (but never delete directories after deleting files).

2  Usage

Two data files are necessary for WSync operation. A list file containing the list of files that are to be synched with the remote server, and a command file giving configuration informations.

2.1  Command file

Here is a template command file, that is pretty self-explanatory. Currently, the only available backend is ``FTP'', and its parameters are host, login, and password. The log file is automatically created if it doesn't exist. 
# template command file for wsync
#
# Syntax
# one directive per non blank non comment line
# '\' as last character (no trailing whitespaces) enables wrapping
# 
# key : value
# key : par1 = val1, par2= val2 ,par3  = val3


# backend to perform upload
backend: FTP

# connection parameters for the backend

params: host = my.host, login = mylogin, \
 password = mypass

# path to the local repository
repository: path/to/files/

# name of file listing the synched files
list_file: filename

# name of db file (where files status is stored)
db: .wsync.db

# name of log file (defaults to 'wsync.log')
#log_file: filename

2.2  List file

The list file is basically a simple list, with one file per line. Here is a sample. You can use the special file name '*' for ``all the files in this directory'', as well as including the content of another list file by putting the file name on a line by itself, prefixed by the `<' character. The parsing of this file is very simple and stupid, so be sure to conform exactly to this! 
#-- some files in top dir

index.html
academ.html
traduc.html
docs.html

#-- all files from a subdir

these/*

# include another list file
<list.latextab

#-- files in various dirs

M/cfip97.ps.gz
M/dnac99.ps.gz

R/measured_performance.pdf
R/QoS/BZ96a.ps.gz

2.3  Operations and file states

You run WSync by giving it the name of the command file as an argument. It reads the list file, its database, and displays what the situation is.

It may warn you about a number of lost files. These are files that are in some directory of the local repository (i.e. a directory with files listed in the list file) but not in the list file themselves. Maybe this is because you generated new files and forgot to add them to the list file. You can give WSync the -l option to get the list of these lost files displayed.

It then lists the files that need to be uploaded (for beeing new or changed, if any), with their sizes and showing the total amount of data that needs to be uploaded. It also list files to be deleted, renamed, and directories to be created.

By default, it displays this information but doesn't call any backend to perform any action on the remote server. It will if you give it the -d (for doit) option. Of course, if any such operation fails, the corresponding database entry is not updated, so the operation will be tried again next time.

Some files may also be listed as ``touched''. These are files whose timestamp has changed, but whose digest not. They won't be uploaded and are listed just for information. Next time you run WSync they won't be in the ``touched'' list again (except if their timestamp has changed again). This is because the signature in the database is updated with the new timestamp, so that if you don't touch them again, WSync will known they haven't changed without needing to compute the digest.

Finally, files may be ``unknown'', meaning that they're present both in the list file and in the database, but can't be found in the local repository. You may thus delete some files in the repository (e.g. to save disk space) without causing any trouble (but be sure they've been uploaded before to do that). This is meaningfull since if you generate HTML files from some other language, you shouln't have to keep local copies of the HTML pages themselves, once they have been uploaded. Next time you generate these files, WSync will realize they're back and treat them normally (i.e. upload them if they have changed.)

A variant of this is ``added but not present'', showing files that are new in the list file but can't be found in the local repository. You probably need to fix something in this case.

It may happen that some files have changed, but you don't want all of them to be uploaded, for example if you know you're going to change them again very soon. In this case, you should delete (or move out) the files you don't want WSync to upload from the local repository. WSync will thus finds them to be ``unknown'', and will not attempt to upload them. Do not delete them from the list file, or WSync will think you want them to be removed from the remote server! Another way to do that is to use the --prevent option, discussed later.

2.4  Log file

All operations performed (or attempted) on the remote server are logged in the log file that looks like that: 
06/10/03 07:36:40  ( FTP my.host.com, mylogin
06/10/03 07:36:42  + prgm.html
06/10/03 07:36:43  + emacs.html
06/10/03 07:36:49  * Private
06/10/03 07:36:50  + Private/fp.tgz
06/10/03 07:36:50  - msx.html
06/10/03 07:36:51  > latxe.html --> latex.html
06/10/03 07:36:51  )
As you can see, each line shows a timestamp, an operation symbol (`(' for opening the connection, `)' for closing it, `+' for upload, `-' for delete, `*' for creating a directory, `>' for renaming) followed by parameters of the operation. If the operation has failed, the string `**FAILED**' ends the line.

2.5  Manual operations

WSync tries to act automatically as most as possible. But you can also explicitally direct it to do certain things. For all the following operations, there are two ways so specify a list of files:

2.5.1  Forcing/Avoiding uploads

You may want WSync to upload a given set of files, even if it thinks they don't need to. Such files are called ``forced''. You specify them using either --force file-list or --force-from file.

The reverse operation is to avoid uploading of files WSync thinks need to be uploaded. Such files are called ``avoided''. You specify them using either --avoid file-list or --avoid-from file.

2.5.2  Faking synchronization

You can direct WSync to perform a ``fake sync'': this is like a sync, but the remote server is not contacted, WSync updates its DB juste like if the operations on the remote server had been performed successfully. This can be useful if the remote server state has been modified independently of WSync. For example, if you have deleted files ``by hand'' on the remote server and then remove them from the list file, WSync will try (and fail) to delete theses files each time you sync. To perform a fake sync, use the -f command line option instead of the -d one.

Fake operations are logged with an initial ``!'' character in the log file.

2.6  FTP options

--no-safeput
by default, files are uploaded under a temporary name, which is their name plus the .tmp suffix. Only when the file has been successfully uploaded is the file renamed to its real name. This is so that if the upload is interrupted, the previous version of the file will still be safe on the remote server. This command line option allows to disable this feature: files are then uploaded directly under their name.

--no-passive
disable FTP passive mode, that is set by default. Passive mode allows FTP to work even if you're behind a NAT, and is a bit more secure (though FTP is never secure anyway) . You should probably never need to disable it.

2.7  Reference

2.7.1  Files states


File state Meaning
Unknown file with no local copy available, so freshness is unknown
Touched file whose timestamp has changed, but not digest's
Deleted file removed from the list file
Added but  
not present file new in the list file, with no local copy
Added file new in the list file
Changed file whose signature has changed since last upload
Renamed file whose name has changed
Forced file that shouldn't be uploaded, but so required by the user
Avoided file that should be updated, but required not by the user



2.7.2  Command line options


Option Meaning
-d doit, sync (upload/delete/rename) files on the remote server
-f fake doit
-l show the list of lost files
-v show version information
--no-safeput disable FTP safeput feature
--no-passive disable passive FTP
--force file-list force upload of listed files
--force-from file force upload of files listed in file
--avoid file-list avoid upload of listed files
--avoid-from file avoid upload of files listed in file
--version ...
-help, --help guess it


3  Installation

The core of WSync is written in Objective Caml, and linked with an FTP library written in C, derived from the traditional Linux FTP client netkit-ftp. The OCaml Unix module is used, so it won't run on non-Unix platforms.

Sources and compiled files are available from the top right ``Download'' links on this page. All are distributed under the General Public Licence (GPL).

3.1  From sources

Installation from sources requires the Objective Caml compiler version 3.06 or above, a C compiler, and the FTPlib source package. Compilation must be performed from ftplib source directory, as shown above:
  1. tar xvzf ftplib.tgz
  2. tar xvzf wsync.tgz
  3. cd ftplib
  4. cp ../wsync/wsync.ml .
  5. for the native code version: make libftp.a ftp.cmxa ; make -f ../wsync/Makefile wsync.opt
    for the bytecode version: make libftp.a ftp.cma ; make -f ../wsync/Makefile wsync.b
If something goes wrong, make sure the caml bindings are enabled (the NO_CAML macro should not be defined, it is commented out in ftplib makefile by default), and do a make clean for ftplib before compiling.

3.2  Binary

A binary package for i386 Linux is also available for download. This is a native code binary, and so does not need any OCaml tool installed to run.

4  Feedback

Send any bug report, comment, suggestion or anything to elviok AT free DOT fr. Make sure to mention that you're talking about WSync and give version information if applicable.

References

[1]
D. J. Bernstein. FTP: File transfer protocol. web site. [ url ]