11 Practical Examples of Rsync Command in Linux and Unix

1
6490

Rsync ( Remote Sync) is an open source command utility that provides fast incremental file transfer from one host to another. You can copy and synchronize  your data with sync remotely and locally across directories, across disks and networks, perform data backups and mirroring between two Linux and Unix machines.

It is one of the lightweight application, because the file transfers are incremental. Rsync copies only the differences of files that have actually changed, compressed through ssh.You don’t need to be root user to run rsync command.

Some advantages and features of Rsync command

  • Rsync allows encryption of data using ssh protocol during transfer.
  • Supports copying links, devices, owners, groups and permissions.
  • No special privileges are required to install and execute rsync
  • It’s faster than scp (Secure Copy) because rsync uses remote-update protocol which allows to transfer just the differences between two sets of files. First time, it copies the whole content of a file or a directory from source to destination but from next time, it copies only the changed blocks and bytes to the destination.
  • Rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols.

Before you install Rsync on your server, make sure to run below commands as root user.

yum install rsync (On Red Hat based systems)
apt-get install rsync (On Debian based systems)

Basic syntax of rsync command
# rsync options source destination

Some common options used with rsync commands

  1. -v : verbose
  2. -r : copies data recursively (but don’t preserve timestamps and permission while transferring data
  3. -a : archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps
  4. -z : compress file data
  5. -h : human-readable, output numbers in a human-readable format.

1Copy/Sync Files and Directory Locally

The following command will sync a single file on a local machine from one location to another location.

[root@instance-1 log]# rsync -zvh log.tar /root/emre/newdirectory
log.tar
sent 43.61K bytes received 31 bytes 87.28K bytes/sec
total size is 532.48K speedup is 12.20

"newdirectory" automatically created.

2Copy/Sync a Directory on Local Computer

The following command sync two directories in a local computer, use the following rsync -avzh command.Below command; /var/log contains some log files files and you want that directory to be copied inside /home/emre/rsyncfold folder. “rsyncfold” folder is created automatically.

[root@instance-1 log]# rsync -avzh /var/log /root/emre/rsyncfolder/
sending incremental file list
created directory /root/emre/rsyncfolder
log/
log/boot.log
log/btmp
log/cron
log/dmesg
log/firewalld
log/grubby
log/grubby_prune_debug
log/lastlog
log/log.tar
log/maillog
log/messages
log/secure
log/spooler
log/tallylog
log/wtmp
log/tuned/tuned.log
sent 87.52K bytes received 378 bytes 175.79K bytes/sec
total size is 1.05M speedup is 11.91

3Copy or Sync Synchronize Files From Local to Remote Server

Copy a Directory from Local Server to a Remote Server

This command will sync a directory from a local machine to a remote machine. For example: There is a folder in your local computer “/var/log” which contains some log packages and you want that local directory’s content send to a remote server, you can use following command.

[root@ip-172-31-37-79 .ssh]# rsync -avz /var/log/ root@172.31.42.177:/root/rsyncfolder/
sending incremental file list
./
lastlog
maillog
messages
secure
audit/audit.log
chrony/
qemu-ga/
rhsm/
tuned/
tuned/tuned.log

sent 277,838 bytes received 602 bytes 185,626.67 bytes/sec
total size is 2,831,543 speedup is 10.17
Copy/Sync a Remote Directory to a Local Machine

This command will help you sync a remote directory to a local directory. Here in this example, a directory /root/rsyncfolder which is on a remote server is being copied in your local computer in /root/emre

root@ip-172-31-37-79 .ssh]# rsync -avzh root@172.31.42.177:/root/rsyncfolder/ /root/emre/
receiving incremental file list
created directory /root/emre
./
boot.log
btmp
choose_repo.log
cloud-init.log
cron
dmesg
grubby_prune_debug
lastlog
sent 610 bytes received 277.81K bytes 185.62K bytes/sec
total size is 2.83M speedup is 10.17

4Rsync Over SSH

With rsync, we can use SSH (Secure Shell) for data transfer, using SSH protocol while transferring our data you can be ensured that your data is being transferred in a secured connection with encryption so that nobody can read your data while it is being transferred over the wire on the internet.

Copy a File from a Remote Server to a Local Server with SSH

-e” option is ssh.You can use other protocol what do you want.

[root@ip-172-31-37-79 ec2-user]# rsync -avzhe ssh root@172.31.42.177:/var/log /root/emrenewlog
receiving incremental file list
created directory /root/emrenewlog
log/
log/boot.log
log/btmp
log/choose_repo.log
log/cloud-init.log
log/cron
log/dmesg
log/dmesg.old
sent 649 bytes received 325.02K bytes 72.37K bytes/sec
total size is 3.23M speedup is 9.92
Copy a File from a Local Server to a Remote Server with SSH
[root@ip-172-31-37-79 ec2-user]# rsync -avzhe ssh /var/log/ root@172.31.42.177:/root/newfolder/
sending incremental file list
created directory /root/newfolder
./
boot.log
btmp
choose_repo.log
cloud-init.log
cron
dmesg
dmesg.old
grubby_prune_debug
sent 320.64K bytes received 678 bytes 71.40K bytes/sec
total size is 3.16M speedup is 9.82

5Show Progress While Transferring Data with rsync

When you use rsync for copy or backup, maybe you  want to know the progress of the backup or copy. For example;how many files are copies, at what rate it is copying the file, etc.

rsync –progress option displays detailed progress of rsync execution as shown below.

[root@ip-172-31-37-79 ~]# rsync -avzhe ssh --progress /var/log root@172.31.42.177:/root/emre/
sending incremental file list
created directory /root/emre
log/
log/boot.log
 0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=35/37)
log/btmp
 1.15K 100% 0.00kB/s 0:00:00 (xfr#2, to-chk=34/37)
log/choose_repo.log
 1.14K 100% 1.09MB/s 0:00:00 (xfr#3, to-chk=33/37)
log/cloud-init.log
 121.75K 100% 58.05MB/s 0:00:00 (xfr#4, to-chk=32/37)
log/cron
 2.15K 100% 1.02MB/s 0:00:00 (xfr#5, to-chk=31/37)
log/dmesg
 30.74K 100% 9.77MB/s 0:00:00 (xfr#6, to-chk=30/37)
log/grubby_prune_debug
 193 100% 62.83kB/s 0:00:00 (xfr#7, to-chk=29/37)
log/lastlog
 292.29K 100% 46.46MB/s 0:00:00 (xfr#8, to-chk=28/37)
log/maillog
 210 100% 34.18kB/s 0:00:00 (xfr#9, to-chk=27/37)
log/messages
 84.66K 100% 10.09MB/s 0:00:00 (xfr#10, to-chk=26/37)
log/secure
 5.05K 100% 616.94kB/s 0:00:00 (xfr#11, to-chk=25/37)
log/spooler
 0 100% 0.00kB/s 0:00:00 (xfr#12, to-chk=24/37)
log/tallylog
 0 100% 0.00kB/s 0:00:00 (xfr#13, to-chk=23/37)
log/wtmp
 3.07K 100% 375.00kB/s 0:00:00 (xfr#14, to-chk=22/37)
log/yum.log
 0 100% 0.00kB/s 0:00:00 (xfr#15, to-chk=21/37)
sent 282.46K bytes received 655 bytes 566.23K bytes/sec
total size is 2.90M speedup is 10.24

6Synchronize Only One File

To copy only one file, specify the file name to rsync command.

[root@ip-172-31-37-79 ec2-user]# rsync -v /root/.ssh/id_rsa.pub root@172.31.42.177:/root/
id_rsa.pub

sent 516 bytes received 35 bytes 157.43 bytes/sec
total size is 429 speedup is 0.78

7Use of –include and –exclude Options

These two options allows us to include and exclude files by specifying parameters with these option helps us to specify those files or directories which you want to include in your sync and exclude files and folders with you don’t want to be transferred.

Here in this example, rsync command will include those files and directory only which starts with ‘s’ and exclude all other files and directory.

[root@ip-172-31-37-79 ec2-user]# rsync -avze ssh --include 's*' --exclude '*' root@172.31.42.177:/var/log/ /root/
receiving incremental file list
./
secure
spooler

sent 80 bytes received 1,995 bytes 592.86 bytes/sec
total size is 14,185 speedup is 6.84

8Use of –delete Option

If a file or directory not exist at the source, but already exists at the destination, you might want to delete that existing file/directory at the target while syncing .

We can use ‘–delete‘ option to delete files that are not there in source directory.

Source and target are in sync. Now creating new file sysaix.txt at the target.

[root@ip-172-31-37-79 emre]# touch sysaix.txt
[root@ip-172-31-37-79 emre]# rsync -avz --delete root@172.31.42.177:/root/emre/ .
receiving incremental file list
deleting sysaix.txt
./sent 27 bytes received 43 bytes 20.00 bytes/sec
total size is 0 speedup is 0.00

9View the Changes Between Source and Destination

This option is useful to view the difference in the files or directories between source and destination.

At the source:

[root@ip-172-31-37-79 emre]# ls -ltr
toplam 0
-rw-r--r--. 1 root root 0 Nis 22 20:02 unix.txt
-rw-r--r--. 1 root root 0 Nis 22 20:02 linux.txt
-rw-r--r--. 1 root root 0 Nis 22 20:02 bkm.txt

At the destination:

[root@ip-172-31-42-177 emre]# ls -ltr
toplam 0
-rwxrwxrwx. 1 root root 0 Nis 22 20:03 unix.txt
-rw-r--r--. 1 root root 0 Nis 22 20:03 linux.txt
-rw-r--r--. 1 bin bin 0 Nis 22 20:03 bkm.txt

Now let us see how rsync displays this difference. -i option displays the item changes.

[root@ip-172-31-37-79 emre]# rsync -avzi root@172.31.42.177:/root/emre/ /root/emre/
receiving incremental file list
.d..t...... ./
>f..t.og... bkm.txt
>f..t...... linux.txt
>f..tp..... unix.txt

sent 84 bytes received 241 bytes 92.86 bytes/sec
total size is 0 speedup is 0.00

In our example, the letters in front of the Basenames (and Dirnames) says the following:

> specifies that a file is being transferred to the local host.
f represents that it is a file.
s represents size changes are there.
t represents timestamp changes are there.
o owner changed
g group changed.

10Setting Max Size of Files to be Transferred

You can specify the Max file size to be transferred or sync. You can do it with “–max-size” option. For example, Max file size is 100k, so this command will transfer only those files which are equal or smaller than 100k.

[root@ip-172-31-37-79 emre]# rsync -avz --max-size='100K' root@172.31.42.177:/var/log/ /root/
receiving incremental file list
./
boot.log
btmp
choose_repo.log

sent 515 bytes received 48,092 bytes 13,887.71 bytes/sec
total size is 3,443,343 speedup is 70.84

11Set Bandwidth Limit and Transfer File

You can set the bandwidth limit while transferring data from one machine to another machine with the the help of ‘–bwlimit‘ option. This options helps us to limit I/O bandwidth.

[root@ip-172-31-37-79 emre]# rsync --bwlimit=100 -avzhe ssh /var/log/ root@172.31.42.177:/root/emre/
sending incremental file list
./
anaconda/syslog
audit/
audit/audit.log
chrony/
qemu-ga/
rhsm/
rhsm/rhsm.log
tuned/
tuned/tuned.log
sent 321.11K bytes received 644 bytes 49.50K bytes/sec
total size is 3.16M speedup is 9.82

1 COMMENT

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.