Linux equivalent to xcopy /m

[Herman Viaene]

Still learning a lot.....

On the "other OS" I often used the "xcopy /m" command to keep a backup of a
directory somewhere on another disk on the same machine.

I wonder what the Linux equivalent of that could be. I found rsync, but none
of its possibilities is a good match - as far as I understood all of it. In
particular I do not get what the command does on subsequent runs (i.e.
where most of the files/directories already exist at the destination).

Any suggestions?

[Wayne]

Post by Herman Viaene
Still learning a lot.....
On the "other OS" I often used the "xcopy /m" command to keep a backup of
a directory somewhere on another disk on the same machine.
I wonder what the Linux equivalent of that could be. I found rsync, but
none of its possibilities is a good match - as far as I understood all of
it. In particular I do not get what the command does on subsequent runs
(i.e. where most of the files/directories already exist at the
destination).
Any suggestions?

http://www.yolinux.com/TUTORIALS/unix_for_dos_users.html

Wayne

[Whiskers]

Post by Wayne

Post by Herman Viaene
Still learning a lot.....
On the "other OS" I often used the "xcopy /m" command to keep a backup of
a directory somewhere on another disk on the same machine.
I wonder what the Linux equivalent of that could be. I found rsync, but
none of its possibilities is a good match - as far as I understood all of
it. In particular I do not get what the command does on subsequent runs
(i.e. where most of the files/directories already exist at the
destination).
Any suggestions?

http://www.yolinux.com/TUTORIALS/unix_for_dos_users.html
Wayne

Good link :))

[Lordy]

Post by Whiskers

Post by Wayne

Post by Herman Viaene
Still learning a lot.....
On the "other OS" I often used the "xcopy /m" command to keep a backup
I found rsync, but...
Any suggestions?

http://www.yolinux.com/TUTORIALS/unix_for_dos_users.html
Wayne

Good link :))

Unfortunately, in this case, it doesnt answer the question and may even
lead one away from a better solution.

Lordy

[Wayne]

Post by Whiskers

Post by Wayne
http://www.yolinux.com/TUTORIALS/unix_for_dos_users.html
Wayne

Good link :))

The magic of Google :-)

Wayne

[Herman Viaene]

Post by Wayne

Post by Herman Viaene
Still learning a lot.....
On the "other OS" I often used the "xcopy /m" command to keep a backup of
a directory somewhere on another disk on the same machine.
I wonder what the Linux equivalent of that could be. I found rsync, but
none of its possibilities is a good match - as far as I understood all of
it. In particular I do not get what the command does on subsequent runs
(i.e. where most of the files/directories already exist at the
destination).
Any suggestions?

http://www.yolinux.com/TUTORIALS/unix_for_dos_users.html
Wayne

I'll answer on your and Donalds and Lordy's post in this one.
I bookmarked the link above, I think it can come in handy in future, tx.

Rsync just copies "differences" as Lordy states, but that is just my
question, what are differences, how does it define that?

As far Donald's remark, it is a valuable thing to know that rsync removes
files in the backup. But I am not sure that this is an advantage. Sure it
takes away the necessity of checking and removing "rubbish", but I feel not
quite confident. I've see, to often in the past people deleting files and
only realizing a few days later this was a mistake. If rsync was run in the
mean time (via a cron entry for instance), the files are definitively lost.
I'll have to check whether rsync's options can avoid this.

Tx anyway.

[Lordy]

Post by Herman Viaene
Rsync just copies "differences" as Lordy states, but that is just my
question, what are differences, how does it define that?

Difference is at the byte level (more or less). Differences are the bits
that are not the same :) I'm not sure I understand your concern here?

Rsync is defined to be network efficient and thus if only a few bytes
have changed in a 200M file, it will just copy a few blocks over to patch
the target file.

http://samba.anu.edu.au/rsync/tech_report/node2.html

You dont really need to get down to that level. 'rysnc' MIRRORS a
source at a destination, quite efficiently.

Post by Herman Viaene
As far Donald's remark, it is a valuable thing to know that rsync removes
files in the backup. But I am not sure that this is an advantage.

man rsync
[big snip]
-b, --backup make backups (see --suffix & --backup-dir)
--backup-dir=DIR make backups into hierarchy based in DIR
--suffix=SUFFIX backup suffix (default ~ w/o --backup-dir)
[snip]
--delete delete files that don't exist on sender

Lots more info in the man page but --delete is optional.

Lordy

[Herman Viaene]

Post by Lordy

Post by Herman Viaene
Rsync just copies "differences" as Lordy states, but that is just my
question, what are differences, how does it define that?

Difference is at the byte level (more or less). Differences are the bits
that are not the same :) I'm not sure I understand your concern here?

OK, xcopy works on a file attribute, the archive bit. It copies only the
files that have this bit set, and resets it. The archive bit gets set (by
the OS) as soon as the file is saved again, no matter if it is really
changed or just overwritten with the same contents.
So, rsync really goes into comparing every single bit of the files from
source to destination. Well.....I'm not sure, updating by just sending a
few blocks over sounds efficient, but before that the whole file will have
travelled anyway to make that comparison, so where is the gain?

And tx for the info on the delete option.

Herman

[Lordy]

Post by Herman Viaene
into comparing every single bit of the files from
source to destination. Well.....I'm not sure, updating by just sending a
few blocks over sounds efficient, but before that the whole file will have
travelled anyway to make that comparison, so where is the gain?

No, From http://samba.anu.edu.au/rsync/tech_report/node1.html

"The problem is that the normal methods for creating a set of differences
between two files rely on being able to read both files. Thus they require
that both files are available beforehand at one end of the link. If they
are not both available on the same machine, these algorithms cannot be
used (once you had copied the file over, you wouldn't need the
differences). This is the problem that rsync addresses."

Read rsync manpage and if you really want to know how it works
http://samba.anu.edu.au/rsync/tech_report/node1.html.

Rsync is what you want. All info is in the man pages.

Lordy

[Lordy]

Post by Herman Viaene
into comparing every single bit of the files from
source to destination. Well.....

No. You asked how it works. The link I posted
http://samba.anu.edu.au/rsync/tech_report/node2.html showed that it doesnt
compare every byte. You didnt read it? Then why keep asking qs and not
reading answers or looking at rysnc man page?

Post by Herman Viaene
I'm not sure, updating by just sending a
few blocks over sounds efficient, but before that the whole file will
have travelled anyway to make that comparison, so where is the gain?

No, From http://samba.anu.edu.au/rsync/tech_report/node1.html

"The problem is that the normal methods for creating a set of differences
between two files rely on being able to read both files. Thus they require
that both files are available beforehand at one end of the link. If they
are not both available on the same machine, these algorithms cannot be
used (once you had copied the file over, you wouldn't need the
differences). This is the problem that rsync addresses."

Read rsync manpage and if you really want to know how it works
http://samba.anu.edu.au/rsync/tech_report/node1.html.

Rsync is what you want. All info is in the man pages.

Lordy

[Herman Viaene]

Post by Lordy

Post by Herman Viaene
into comparing every single bit of the files from
source to destination. Well.....

No. You asked how it works. The link I posted
http://samba.anu.edu.au/rsync/tech_report/node2.html showed that it doesnt
compare every byte. You didnt read it? Then why keep asking qs and not
reading answers or looking at rysnc man page?

No, I didn't read it before. I saw "samba" in the link and thought it might
be specific to this protocol. Assuming too much. Error corrected now, this
indeed explains what I wanted to know, tx.

Herman

[Donald Tees]

Post by Herman Viaene

Post by Wayne

Post by Herman Viaene
Still learning a lot.....
On the "other OS" I often used the "xcopy /m" command to keep a backup of
a directory somewhere on another disk on the same machine.
I wonder what the Linux equivalent of that could be. I found rsync, but
none of its possibilities is a good match - as far as I understood all of
it. In particular I do not get what the command does on subsequent runs
(i.e. where most of the files/directories already exist at the
destination).
Any suggestions?

http://www.yolinux.com/TUTORIALS/unix_for_dos_users.html
Wayne

I'll answer on your and Donalds and Lordy's post in this one.
I bookmarked the link above, I think it can come in handy in future, tx.
Rsync just copies "differences" as Lordy states, but that is just my
question, what are differences, how does it define that?
As far Donald's remark, it is a valuable thing to know that rsync removes
files in the backup. But I am not sure that this is an advantage. Sure it
takes away the necessity of checking and removing "rubbish", but I feel not
quite confident. I've see, to often in the past people deleting files and
only realizing a few days later this was a mistake. If rsync was run in the
mean time (via a cron entry for instance), the files are definitively lost.
I'll have to check whether rsync's options can avoid this.
Tx anyway.

Yes, it could be dangerous, but do note it is an *option* to do the
deletes. The default is not to. Used correctly rsynch is safer that
xcopy as it actually checksums the files to make the copy/no copy
decision. Xcopy simply checks the archive bit, which has a couple of
problems. For example, you can defeat it by manually reseting that bit,
or, if you use the xcopy reset, you cannot do the xcopy to more than one
place. I always used the /d xcopy option for that reason.

I agree with your point that often one removes files, and only realizes
later that it was a mistake, and over-writing backups ... well, it is
quite dismaying to discover a large chunk of code you have not worked on
for six months is *gone*, from *everywhere*.

I tend to do cleanups about every three months, but only after I have
done a permament hard copy backup for my archives. I do that to DVD now,
but use to do it to tape. I use rsynch for daily "oh damnit" problems,
and updating things like libraries on other machines.

What I would like to find is a program that divides 40-50 gigs of files,
all under 4 gigs, into sets that will fit on single DVDs.

Donald

[David W. Hodgins]

Post by Donald Tees
What I would like to find is a program that divides 40-50 gigs of files,
all under 4 gigs, into sets that will fit on single DVDs.

dar & kdar.
My scripts were generated with kdar, and then manually entered to created
cd backups of my entire system (I run them in run level 1). To create dvd
images, just change the size. The manual editing consisted of finetuning
which directories to exclude, and in the full backup, generating the date
for the file name. The second one could be modified to find that latest
full backup, and use a generated date for the incremental file name, I
just haven't gotten around to that yet<g>.

darfull.sh
#!/bin/bash
time dar -v -c /mnt/hdb1/full`date +'%G%m%d'` -R / -w -s 737148928 -D -y -m 150 -P "mnt" -P "dev" -P "sys" -P "proc" -P "var/ftp"

darinc.sh
#!/bin/bash
#Note: Manually alter dates before running.
time dar -v -c /mnt/hdb1/inc20051015 -R / -A /mnt/hdb1/full20051001 -w -s 737148928 -D -y -m 150 -P "dev" -P "mnt" -P "proc" -P "sys" -P "var/ftp"

Regards, Dave Hodgins

[Donald Tees]

On Wed, 19 Oct 2005 09:00:19 -0400, Donald Tees

Post by Donald Tees
What I would like to find is a program that divides 40-50 gigs of files,
all under 4 gigs, into sets that will fit on single DVDs.

dar & kdar.
My scripts were generated with kdar, and then manually entered to created
cd backups of my entire system (I run them in run level 1). To create dvd
images, just change the size. The manual editing consisted of finetuning
which directories to exclude, and in the full backup, generating the date
for the file name. The second one could be modified to find that latest
full backup, and use a generated date for the incremental file name, I
just haven't gotten around to that yet<g>.
darfull.sh
#!/bin/bash
time dar -v -c /mnt/hdb1/full`date +'%G%m%d'` -R / -w -s 737148928 -D -y
-m 150 -P "mnt" -P "dev" -P "sys" -P "proc" -P "var/ftp"
darinc.sh
#!/bin/bash
#Note: Manually alter dates before running.
time dar -v -c /mnt/hdb1/inc20051015 -R / -A /mnt/hdb1/full20051001 -w
-s 737148928 -D -y -m 150 -P "dev" -P "mnt" -P "proc" -P "sys" -P "var/ftp"
Regards, Dave Hodgins

Thank you. I have copied that to a file, and expect it will give me
several evenings of entertainment<g>.

Donald

[Lordy]

On Tue, 18 Oct 2005 15:31:44 +020The rsync remote-update protocol allows rsync to transfer just the dif-
ferences between two sets of files across the network connection0,

Post by Herman Viaene
I wonder what the Linux equivalent of that could be. I found rsync, but none
of its possibilities is a good match - as far as I understood all of it.

Most of us guessing xcopy /m copies files thant have not
changed since the last xcopy /m?

rsync is good. Especially for large files, as it only copies differences
at the byte level.

Post by Herman Viaene
particular I do not get what the command does on subsequent runs (i.e.
where most of the files/directories already exist at the destination).

"The rsync remote-update protocol allows rsync to transfer just the dif-
ferences between two sets of files across the network connection"

Lordy

[Donald Tees]

Post by Lordy
On Tue, 18 Oct 2005 15:31:44 +020The rsync remote-update protocol allows
rsync to transfer just the dif-
ferences between two sets of files across the network connection0,

Post by Herman Viaene
I wonder what the Linux equivalent of that could be. I found rsync, but
none of its possibilities is a good match - as far as I understood all of
it.

Most of us guessing xcopy /m copies files thant have not
changed since the last xcopy /m?
rsync is good. Especially for large files, as it only copies differences
at the byte level.

Post by Herman Viaene
particular I do not get what the command does on subsequent runs (i.e.
where most of the files/directories already exist at the destination).

"The rsync remote-update protocol allows rsync to transfer just the dif-
ferences between two sets of files across the network connection"
Lordy

The other advantage of rsync is that it can remove files that are no longer
on the originating drive. Xcopy simply copies, rather than synchronizes.
Any file cleanup that you do in the library being backed up either has to
be done again in the copy library, or the copy library has to be deleted
and a fresh copy made. It tends to accumulate garbage as a result, and
if/when you need to do a restore, you end up with lots of post restore
cleanup to do.

I used Xcopy for years, though I tended to use /s/e/d/v(subdirectories,
empty directories, dates changed, and verify) rather than /M(archive bit
set). rsync is much cleaner.

Donald