［每周讨论专题］－－第六期－－WEB服务器的数据远程同步

［每周讨论专题］－－第六期－－WEB服务器的数据远程同步
每周讨论专题【第六期】.......................................................点这里查看其他讨论专题

　　WEB服务器的数据远程同步


感谢 wingger 点题
本期主持：wingger

两个WEB远 程同步可以吗

或是如何负载匀横

I think only web can't do so.maybe you can use scp or sftp/ftp do so.

如果是数据同步，而且使用了数据库的话，可以考虑采用数据库的同步复制功能；
当然也可以对web系统自己预留接口，定期自动查询同步。最笨了，而且性能不好控制

rsync这个软件可以实现mirror.

使用rsync


下面是rsync网站上的一些例子

每隔七天将数据往中心服务器做增量备份


#!/bin/sh

# This script does personal backups to a rsync backup server. You will end up
# with a 7 day rotating incremental backup. The incrementals will go
# into subdirectories named after the day of the week, and the current
# full backup goes into a directory called "current"
# tridge@linuxcare.com

# directory to backup
BDIR=/home/$USER

# excludes file - this contains a wildcard pattern per line of files to exclude
EXCLUDES=$HOME/cron/excludes

# the name of the backup machine
BSERVER=owl

# your password on the backup server
export RSYNC_PASSWORD=XXXXXX


########################################################################

BACKUPDIR=`date +%A`
OPTS="--force --ignore-errors --delete-excluded --exclude-from=$EXCLUDES
      --delete --backup --backup-dir=/$BACKUPDIR -a"

export PATH=$PATH:/bin:/usr/bin:/usr/local/bin

# the following line clears the last weeks incremental directory
[ -d $HOME/emptydir ] || mkdir $HOME/emptydir
rsync --delete -a $HOME/emptydir/ $BSERVER:USER/$BACKUPDIR/
rmdir $HOME/emptydir

# now the actual transfer
rsync $OPTS $BDIR $BSERVER:USER/current


备份至一个空闲的硬盘

I do local backups on several of my machines using rsync. I have an
extra disk installed that can hold all the contents of the main
disk. I then have a nightly cron job that backs up the main disk to
the backup. This is the script I use on one of those machines.

    #!/bin/sh

    export PATH=/usr/local/bin:/usr/bin:/bin

    LIST="rootfs usr data data2"

    for d in $LIST; do
        mount /backup/$d
        rsync -ax --exclude fstab --delete /$d/ /backup/$d/
        umount /backup/$d
    done

    DAY=`date "+%A"`
   
    rsync -a --delete /usr/local/apache /data2/backups/$DAY
    rsync -a --delete /data/solid /data2/backups/$DAY

   

The first part does the backup on the spare disk. The second part
backs up the critical parts to daily directories.  I also backup the
critical parts using a rsync over ssh to a remote machine.



对vger的cvs树进行镜像

The vger.rutgers.edu cvs tree is mirrored onto cvs.samba.org via
anonymous rsync using the following script.

    #!/bin/bash

    cd /var/www/cvs/vger/
    PATH=/usr/local/bin:/usr/freeware/bin:/usr/bin:/bin

    RUN=`lps x | grep rsync | grep -v grep | wc -l`
    if [ "$RUN" -gt 0 ]; then
            echo already running
            exit 1
    fi

    rsync -az vger.rutgers.edu::cvs/CVSROOT/ChangeLog $HOME/ChangeLog

    sum1=`sum $HOME/ChangeLog`
    sum2=`sum /var/www/cvs/vger/CVSROOT/ChangeLog`

    if [ "$sum1" = "$sum2" ]; then
            echo nothing to do
            exit 0
    fi

    rsync -az --delete --force vger.rutgers.edu::cvs/ /var/www/cvs/vger/
    exit 0

Note in particular the initial rsync of the ChangeLog to determine if
anything has changed. This could be omitted but it would mean that the
rsyncd on vger would have to build a complete listing of the cvs area
at each run. As most of the time nothing will have changed I wanted to
save the time on vger by only doing a full rsync if the ChangeLog has
changed. This helped quite a lot because vger is low on memory and
generally quite heavily loaded, so doing a listing on such a large
tree every hour would have been excessive.


在家自动备份

我用rsync to backup my wifes home directory across a modem link each
night. The cron job looks like this

    #!/bin/sh
    cd ~susan
    {
    echo
    date
    dest=~/backup/`date +%A`
    mkdir $dest.new
    find . -xdev -type f \( -mtime 0 -or -mtime 1 \) -exec cp -aPv "{}"
    $dest.new \;
    cnt=`find $dest.new -type f | wc -l`
    if [ $cnt -gt 0 ]; then
      rm -rf $dest
      mv $dest.new $dest
    fi
    rm -rf $dest.new
    rsync -Cavze ssh . samba:backup
    } >;>; ~/backup/backup.log 2>;&1


note that most of this script isn't anything to do with rsync, it just
creates a daily backup of Susans work in a ~susan/backup/ directory so
she can retrieve any version from the last week. The last line does
the rsync of her directory across the modem link to the host
samba. Note that I am using the -C option which allows me to add
entries to .cvsignore for stuff that doesn't need to be backed up.





Fancy footwork with remote file lists

One little known feature of rsync is the fact that when run over a
remote shell (such as rsh or ssh) you can give any shell command as
the remote file list. The shell command is expanded by your remote
shell before rsync is called. For example, see if you can work out
what this does:

        rsync -avR remote:'`find /home -name "*.[ch]"`' /tmp/

note that that is backquotes enclosed by quotes (some browsers don't
show that correctly).

Rsync for Windows（我没测试成功，有兴趣的可以试试，我有空会把这篇翻译成中文）

导言
我写这份文档可以帮助那些想通过使用Rsync备份 windowsrs 到LINUX servers的人或 Windows 工作站/服务器. 这是经过测试过的。


--------------------------------------------------------------------------------

1: Cygwin环境安装

你需要下载和安装Cygwin系统 .
安装时记住要选择 Rsync from the +Net package list, 和a suitable Editor (因为这个系统不会默认安装). 要用到一些编辑器  Pico (在 +Mail之下的part of Pine living ) 和 Nano (living under +Editors).

下面这段都是讲如何在win下设置环境变量的
在环境变量中增加C:\Cygwin\bin.

否则 会报错：apps called from outside Cygwin will fail.

On Windows 2000/XP, open the Control Panel and double click on the System applet. Click on the Advanced tab, then click the Environment Variables button. Double click on the PATH statement in the 'System Variable' screen (lower of the two), add the path on the end, and click OK. Click OK to close the Environment Variables screen, then click OK to close the System Properties dialogue box. The path will be dynamically reloaded (no need to reboot).

注意：
If the end of the path looks something like this: C:\Somepath don't forget to add ; before you add the C:\Cygwin\bin; e.g. C:\Somepath;C:\Cygwin\bin;


--------------------------------------------------------------------------------

2: 服务端分LINUX和WIN服务端两种

Linux Rsync 服务安装
1. 确认 Rsync 已经安装在你的LINUX上 [rpm -q rsync (Red Hat)] .

2. Choose the path for your backup area: This can be on a per-user basis (backup a Users data to their /home/user area) or on a system level basis (a single machine backing up to one directory.)

Real world example:
All rsync data on the Gaztronics Server sits on /dev/hda2 as /data/rsync_dump

3. Create the /etc/rsyncd.conf and /etc/rsyncd.secrets files. [Check out the Manual pages for rsyncd.conf, or see the documentation on the Rsync website for more info.]

Here is an example of an rsyncd.conf file where the backup area drops into the user 'Fred's' home drive:

[computername]

    path = /home/fred/backup
    comment = Fred's Offsite storage area (requires authentication)
    uid = fred
    gid = users
    read only = false
    auth users = fred
    secrets file = /etc/rsyncd.secrets  

The permissions for this file should be: -rw-r--r-- (644) and root root.

The corresponding rsyncd.secrets file contains the following entry:

fred:BackUpPassword

The permissions for this file should be: -rw------- (600) and root root.

4. Start Rsync in daemon mode.

Linux Tip: Linux Distros usually run rsync from xinetd. You might need to run /usr/sbin/setup (Red Hat) and select 'rsync' in the 'System Services'; or you can edit the 'rsync' file in /etc/xinetd.d and set disable = no (don't forget to restart xinetd!).


Windows Rsync 服务安装

Setting up Rsync as a Server under Windows is a little more tricky, due to the differences in security and paths, and the inability to use the 'authenticate user' mode of Rsync. For this reason, I would not recommend setting up an Rsync Server on Windows for use over the Internet. Keep the installation within a secure Local Area Network.

Method update:  Windows 2003 Server has thrown a spanner in the works. Microsoft have set the paranoid level to maximum which has resulted in Cygwin based services failing to start. This method has been updated to take this into account.

This method comes without warranty, but it should work for: Windows NT 4.0 Server; Windows NT 4.0 Workstation; Windows 2000 Server; Windows 2000 Workstation; Windows XP Workstation.

Method 1. Without installing Cygwin in the full: You might be interested in this Rsync Server project. The Windows package installs the bare essentials to create an Rsync Server.
This method has yet to be tested with 2003 Server!

Method 2. Install Cygwin in full and use its directory structure for storage. (This is recommended if you are new to Linux as the directory permissions are stored *nix style and you will need Cygwin to change them - you cannot change them from Windows!)

Both methods require a valid rsyncd.conf, such as the one below.


--------------------------------------------------------------------------------

Method 2. Step 1: Install Cygwin as in Section 1.

Method 2. Step 2: Choose an area to backup the files to. (In this example I have used /var/rsync_dump in the Cygwin tree.)

Method 2. Step 3: Create the /etc/rsyncd.conf file, as in the example below:

use chroot = false
strict modes = false

[modulename]

    path = /cygdrive/c/cygwin/var/rsync_dump
    comment = Rsync storage area
    read only = false  

Note: The path = /cygdrive/c/cygwin/var/rsync_dump looks a little odd. This is Cygwin convention for defining Windows paths from within a *nix emulator.

Method 2. Step 4: If you are setting up on Windows 2003 Server (otherwise skip to the next step):

(i) Open the Windows File Explorer and go to the C: drive.

(ii) Right click on the 'Cygwin' directory and select 'Properties'.

(iii) Click on the 'Security' tab. The user 'Administrator' should be the first in the list and it will not have any permissions set for this folder.
       (If the user 'Administrator' is not listed, you will need to add it.)

(iv) Tick the 'Allow - Full Control' box in the "ermissions for Administrator" window.

(v) Click the Advanced button and tick the box for "Replace permission entries on all child objects with entries shown here that apply to child objects".

(vi) Click the Apply button to set the permissions.

(vii) Click the OK button to close the Advanced settings dialogue box.

(viii) Click the OK button to close the Cygwin properties dialogue box.

Method 2. Step 5 Install Rsync as a Service from a 'Command Prompt' window with the following command line:

cygrunsrv.exe -I "Rsync" -p /cygdrive/c/cygwin/bin/rsync.exe -a "--config=/cygdrive/c/cygwin/etc/rsyncd.conf --daemon --no-detach"
-f "Rsync" -u Administrator -w password

Note: This is all one line!

The section -u Administrator -w password installs the service to run as the user 'Administrator' (password is the Administrator's account password) and is required by Windows 2003 Server, else the service will fail to start correctly. The extra settings have not been needed for Windows NT 4.0 or Windows 2000, but may be a good idea.

Windows Tip: Once you have the service running as Administrator, you can, for good security practice, create an account with Admin priveleges which Rsync can use. Remember to change the permissions on the C:\Cygwin directory and for the Service.

Method 2. Final Step From the same command prompt, start the Service with   net start rsync.

If all has gone well, you should be ready to accept incoming client connections.


--------------------------------------------------------------------------------

3: 客户端

In order to backup your Windows machine effectively, I would recommend the use of a batch file, as in the following:

@ cls
@ echo off

rem Rsync job control file

C:\Cygwin\bin\rsync -vrtz --password-file=c:\cygwin\secret --delete /cygdrive/d/Data fred@company.com::computername

An explanation:

C:\Cygwin\bin\rsync    -    is the full path to 'rysync.exe'.

-vrtz    -    See the rsync documentation for details.

--password-file=c:\cygwin\secret   -    Path to 'secret' file. (Note: Remember this is for backup to a Linux based Rsync Server; a Windows based Rsync Server cannot authenticate!)

--delete   -    delete remote files that are deleted locally.

/cygdrive/d/Data    -    in this example means D:\Data.

fred@company.com::computername    -    is the user ID, hostname (can be IP address if over Local Network), and the module connection name (in this example 'computername').

Windows Tip: We assume you wish to run the backup automatically! In that case, you will have to create the 'secret' file that Rsync uses for its authenticating password. Here is how to do that:

1. Login to Cygwin.

2. Create a file called 'secret' in the root of the Cygwin application (i.e. cd /) with the Rsync server password (in this example: 'BackUpPassword') and give it 600 permissions (chmod 600 secret). This is the file '--password-file=c:\cygwin\secret' as referred to above.

3. Exit Cygwin.

Windows Tip: You may call this batch file from the Startup Group, the Scheduler (and AT scheduler), or from the Logon / Logoff features of the Group Policy (Win2k/XP) by running gpedit.msc.


--------------------------------------------------------------------------------

Further reading: check out the documentation on the Rsync website.

Happy backing up!!

版主去哪了，

其实远程数据同步的方案是比较多的。
如果数据量不大用rsync就完全可以实现了，可以参看
http://www.fanqiang.com/a1/b1/20011019/0800011478.html
另外如果数据量大的话可以采用rsync＋sgi－fam
sgi－fam在redhat AS3 里面就有，不用编译内核
具体做法可以参看
http://oldsite.linuxaid.com.cn/training/showtri.jsp?i=221
如果是专业的网站就可以考虑用商业的数据同步软件来做
比如snapmirror等等，甚至可以用netapp等专业的存储设备加以实现。

如果用root根用户，用rsync之前最好小心点，此命令像rm命令一样“杀伤力”具大。