Rsync ( Remote Sync) is an open source command utility that provides fast incremental file transfer from one host to another. You can copy and synchronize your data with sync remotely and locally across directories, across disks and networks, perform data backups and mirroring between two Linux and Unix machines.
It is one of the lightweight application, because the file transfers are incremental. Rsync copies only the differences of files that have actually changed, compressed through ssh.You don’t need to be root user to run rsync command.
Some advantages and features of Rsync command
- Rsync allows encryption of data using ssh protocol during transfer.
- Supports copying links, devices, owners, groups and permissions.
- No special privileges are required to install and execute rsync
- It’s faster than scp (Secure Copy) because rsync uses remote-update protocol which allows to transfer just the differences between two sets of files. First time, it copies the whole content of a file or a directory from source to destination but from next time, it copies only the changed blocks and bytes to the destination.
- Rsync uses compression and decompression of data block by block at the sending and receiving end respectively. So the bandwidth used by rsync will be always less compared to other file transfer protocols.
Before you install Rsync on your server, make sure to run below commands as root user.
yum install rsync (On Red Hat based systems) apt-get install rsync (On Debian based systems)
Basic syntax of rsync command
# rsync options source destination
Some common options used with rsync commands
- -v : verbose
- -r : copies data recursively (but don’t preserve timestamps and permission while transferring data
- -a : archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps
- -z : compress file data
- -h : human-readable, output numbers in a human-readable format.
1Copy/Sync Files and Directory Locally
The following command will sync a single file on a local machine from one location to another location.
[root@instance-1 log]# rsync -zvh log.tar /root/emre/newdirectory log.tar sent 43.61K bytes received 31 bytes 87.28K bytes/sec total size is 532.48K speedup is 12.20 "newdirectory" automatically created.
2Copy/Sync a Directory on Local Computer
The following command sync two directories in a local computer, use the following rsync -avzh command.Below command; /var/log contains some log files files and you want that directory to be copied inside /home/emre/rsyncfold folder. “rsyncfold” folder is created automatically.
[root@instance-1 log]# rsync -avzh /var/log /root/emre/rsyncfolder/ sending incremental file list created directory /root/emre/rsyncfolder log/ log/boot.log log/btmp log/cron log/dmesg log/firewalld log/grubby log/grubby_prune_debug log/lastlog log/log.tar log/maillog log/messages log/secure log/spooler log/tallylog log/wtmp log/tuned/tuned.log sent 87.52K bytes received 378 bytes 175.79K bytes/sec total size is 1.05M speedup is 11.91
3Copy or Sync Synchronize Files From Local to Remote Server
Copy a Directory from Local Server to a Remote Server
This command will sync a directory from a local machine to a remote machine. For example: There is a folder in your local computer “/var/log” which contains some log packages and you want that local directory’s content send to a remote server, you can use following command.
[root@ip-172-31-37-79 .ssh]# rsync -avz /var/log/ email@example.com:/root/rsyncfolder/ sending incremental file list ./ lastlog maillog messages secure audit/audit.log chrony/ qemu-ga/ rhsm/ tuned/ tuned/tuned.log sent 277,838 bytes received 602 bytes 185,626.67 bytes/sec total size is 2,831,543 speedup is 10.17
Copy/Sync a Remote Directory to a Local Machine
This command will help you sync a remote directory to a local directory. Here in this example, a directory /root/rsyncfolder which is on a remote server is being copied in your local computer in /root/emre
root@ip-172-31-37-79 .ssh]# rsync -avzh firstname.lastname@example.org:/root/rsyncfolder/ /root/emre/ receiving incremental file list created directory /root/emre ./ boot.log btmp choose_repo.log cloud-init.log cron dmesg grubby_prune_debug lastlog sent 610 bytes received 277.81K bytes 185.62K bytes/sec total size is 2.83M speedup is 10.17
4Rsync Over SSH
With rsync, we can use SSH (Secure Shell) for data transfer, using SSH protocol while transferring our data you can be ensured that your data is being transferred in a secured connection with encryption so that nobody can read your data while it is being transferred over the wire on the internet.
Copy a File from a Remote Server to a Local Server with SSH
“-e” option is ssh.You can use other protocol what do you want.
[root@ip-172-31-37-79 ec2-user]# rsync -avzhe ssh email@example.com:/var/log /root/emrenewlog receiving incremental file list created directory /root/emrenewlog log/ log/boot.log log/btmp log/choose_repo.log log/cloud-init.log log/cron log/dmesg log/dmesg.old sent 649 bytes received 325.02K bytes 72.37K bytes/sec total size is 3.23M speedup is 9.92
Copy a File from a Local Server to a Remote Server with SSH
[root@ip-172-31-37-79 ec2-user]# rsync -avzhe ssh /var/log/ firstname.lastname@example.org:/root/newfolder/ sending incremental file list created directory /root/newfolder ./ boot.log btmp choose_repo.log cloud-init.log cron dmesg dmesg.old grubby_prune_debug sent 320.64K bytes received 678 bytes 71.40K bytes/sec total size is 3.16M speedup is 9.82
5Show Progress While Transferring Data with rsync
When you use rsync for copy or backup, maybe you want to know the progress of the backup or copy. For example;how many files are copies, at what rate it is copying the file, etc.
rsync –progress option displays detailed progress of rsync execution as shown below.
[root@ip-172-31-37-79 ~]# rsync -avzhe ssh --progress /var/log email@example.com:/root/emre/ sending incremental file list created directory /root/emre log/ log/boot.log 0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=35/37) log/btmp 1.15K 100% 0.00kB/s 0:00:00 (xfr#2, to-chk=34/37) log/choose_repo.log 1.14K 100% 1.09MB/s 0:00:00 (xfr#3, to-chk=33/37) log/cloud-init.log 121.75K 100% 58.05MB/s 0:00:00 (xfr#4, to-chk=32/37) log/cron 2.15K 100% 1.02MB/s 0:00:00 (xfr#5, to-chk=31/37) log/dmesg 30.74K 100% 9.77MB/s 0:00:00 (xfr#6, to-chk=30/37) log/grubby_prune_debug 193 100% 62.83kB/s 0:00:00 (xfr#7, to-chk=29/37) log/lastlog 292.29K 100% 46.46MB/s 0:00:00 (xfr#8, to-chk=28/37) log/maillog 210 100% 34.18kB/s 0:00:00 (xfr#9, to-chk=27/37) log/messages 84.66K 100% 10.09MB/s 0:00:00 (xfr#10, to-chk=26/37) log/secure 5.05K 100% 616.94kB/s 0:00:00 (xfr#11, to-chk=25/37) log/spooler 0 100% 0.00kB/s 0:00:00 (xfr#12, to-chk=24/37) log/tallylog 0 100% 0.00kB/s 0:00:00 (xfr#13, to-chk=23/37) log/wtmp 3.07K 100% 375.00kB/s 0:00:00 (xfr#14, to-chk=22/37) log/yum.log 0 100% 0.00kB/s 0:00:00 (xfr#15, to-chk=21/37) sent 282.46K bytes received 655 bytes 566.23K bytes/sec total size is 2.90M speedup is 10.24
6Synchronize Only One File
To copy only one file, specify the file name to rsync command.
[root@ip-172-31-37-79 ec2-user]# rsync -v /root/.ssh/id_rsa.pub firstname.lastname@example.org:/root/ id_rsa.pub sent 516 bytes received 35 bytes 157.43 bytes/sec total size is 429 speedup is 0.78
7Use of –include and –exclude Options
These two options allows us to include and exclude files by specifying parameters with these option helps us to specify those files or directories which you want to include in your sync and exclude files and folders with you don’t want to be transferred.
Here in this example, rsync command will include those files and directory only which starts with ‘s’ and exclude all other files and directory.
[root@ip-172-31-37-79 ec2-user]# rsync -avze ssh --include 's*' --exclude '*' email@example.com:/var/log/ /root/ receiving incremental file list ./ secure spooler sent 80 bytes received 1,995 bytes 592.86 bytes/sec total size is 14,185 speedup is 6.84
8Use of –delete Option
If a file or directory not exist at the source, but already exists at the destination, you might want to delete that existing file/directory at the target while syncing .
We can use ‘–delete‘ option to delete files that are not there in source directory.
Source and target are in sync. Now creating new file sysaix.txt at the target.
[root@ip-172-31-37-79 emre]# touch sysaix.txt [root@ip-172-31-37-79 emre]# rsync -avz --delete firstname.lastname@example.org:/root/emre/ . receiving incremental file list deleting sysaix.txt ./sent 27 bytes received 43 bytes 20.00 bytes/sec total size is 0 speedup is 0.00
9View the Changes Between Source and Destination
This option is useful to view the difference in the files or directories between source and destination.
At the source:
[root@ip-172-31-37-79 emre]# ls -ltr toplam 0 -rw-r--r--. 1 root root 0 Nis 22 20:02 unix.txt -rw-r--r--. 1 root root 0 Nis 22 20:02 linux.txt -rw-r--r--. 1 root root 0 Nis 22 20:02 bkm.txt
At the destination:
[root@ip-172-31-42-177 emre]# ls -ltr toplam 0 -rwxrwxrwx. 1 root root 0 Nis 22 20:03 unix.txt -rw-r--r--. 1 root root 0 Nis 22 20:03 linux.txt -rw-r--r--. 1 bin bin 0 Nis 22 20:03 bkm.txt Now let us see how rsync displays this difference. -i option displays the item changes.
[root@ip-172-31-37-79 emre]# rsync -avzi email@example.com:/root/emre/ /root/emre/ receiving incremental file list .d..t...... ./ >f..t.og... bkm.txt >f..t...... linux.txt >f..tp..... unix.txt sent 84 bytes received 241 bytes 92.86 bytes/sec total size is 0 speedup is 0.00
In our example, the letters in front of the Basenames (and Dirnames) says the following:
> specifies that a file is being transferred to the local host. f represents that it is a file. s represents size changes are there. t represents timestamp changes are there. o owner changed g group changed.
10Setting Max Size of Files to be Transferred
You can specify the Max file size to be transferred or sync. You can do it with “–max-size” option. For example, Max file size is 100k, so this command will transfer only those files which are equal or smaller than 100k.
[root@ip-172-31-37-79 emre]# rsync -avz --max-size='100K' firstname.lastname@example.org:/var/log/ /root/ receiving incremental file list ./ boot.log btmp choose_repo.log sent 515 bytes received 48,092 bytes 13,887.71 bytes/sec total size is 3,443,343 speedup is 70.84
11Set Bandwidth Limit and Transfer File
You can set the bandwidth limit while transferring data from one machine to another machine with the the help of ‘–bwlimit‘ option. This options helps us to limit I/O bandwidth.
[root@ip-172-31-37-79 emre]# rsync --bwlimit=100 -avzhe ssh /var/log/ email@example.com:/root/emre/ sending incremental file list ./ anaconda/syslog audit/ audit/audit.log chrony/ qemu-ga/ rhsm/ rhsm/rhsm.log tuned/ tuned/tuned.log sent 321.11K bytes received 644 bytes 49.50K bytes/sec total size is 3.16M speedup is 9.82