How do you back up your work?

Discussion in 'Photography' started by Dallas, Mar 7, 2012.

  1. I did invest in ECC memory for my file server. I figure I look at
    things after editing them and writing them to disk, so ECC errors on my
    editing system will get caught immediately, but I want higher
    reliability out of my file server.

    It's not just 1/8 more for extra bits, it also adds some time to the
    basic memory cycle.
    Ctein once tracked down an intermittent data problem with one of his
    disks to a cable problem. Going back and recomputing, basically the
    bad cable was raising the undetected error rate from 1 in 10^14 all the
    way to 1 in 10^13. And he noticed this, and was able to track it down.

    This leaves me feeling 1 in 65536 is not very good, just off the top of
    my head .
     
    David Dyer-Bennet, Mar 14, 2012
    #41
    1. Advertisements

  2. Dallas

    Pete A Guest

    You know me well enough by now, Wolfgang :)

    With that in mind, how about the recent reliability of cloud storage.
    Its name conjures up the phrase "pie in the sky". I can pay to backup
    my data somewhere in the ether and a "ruling", or even just a company
    whim, can make my data inaccessible - iDisk being a soon to be
    fulfilled example.

    And what about the plethora of hardware that lies in its response to
    fsync() or equivalent API call: the list of IDE and USB drives that ACK
    a low-level flush command after they've written the data to their cache
    instead of after it is successfully committed to permanent storage is
    despicable.

    And what about the fact that certain mass storage devices have been
    shown to be not fully compatible with motherboard chipsets. I know this
    one very well: it took both me and the computer supplier months of hard
    work to figure out the source of data errors. Hey, MD5 may not be very
    strong evidence, but it's strong enough to wallop hardware vendors into
    admitting the truth when used enough times.

    Of course I'm confused over the terms "archival" and "backup" - the
    computer itself hasn't the faintest idea of what the software is
    requesting it to do and the computer certainly doesn't give a rat's
    arse if it's doing what the unsuspecting user is expecting it to do.

    The humble fly is far more intelligence than your computer. So, next
    time a fly lands on your face after it has be eating something gross,
    use it as a lesson to learn how much intelligent creatures in the
    hardware and software industry really care about your health,
    well-being, data integrity, archival, and backup storage.
     
    Pete A, Mar 14, 2012
    #42
    1. Advertisements

  3. Dallas

    Savageduck Guest

    Sorry to hear that.

    How is the cat doing?
     
    Savageduck, Mar 14, 2012
    #43
  4. Dallas

    Pete A Guest



    Having partially composed myself from stitches of laughter: get some
    software and a suitable drive to show you the massive number of C1 and
    C2 errors from pressed CDs and laser recorded CDs. I have several
    pressed music CDs where the C1 errors have caused so many C2 errors
    that good CD players eventually revert to "STOP" mode and most players
    either get stuck ("broken record") or keep skipping across the disc.
    These CDs do what CDs always do - get worse as they get older.

    Next test: record CDs at different recording speeds and notice how the
    C1 errors massively increase towards the rated recording speed. You
    will also notice that some CDs increase the C1 errors when the
    recording speed is reduced below their optimal recording speed.

    The cross-interleaved Reed-Solomon code (CIRC) encoding and decoding is
    a masterpiece of engineering design. One of the Philips test CDs has a
    sequence of "drilled holes" of various sizes across the disc. Sadly,
    very few players pass the full set of tests. I say "sadly" because for
    a reason unknown to me, it is legal to sell CD players and recorders
    that fall a long way short of the original standards/specifications
    even though the devices are stated by their manufacturer to conform to
    a <Colour> Book Standard.

    The CIRC is to combat surface and substrate regular pattern defects
    (usually visibly identifiable) such as scratches, fingerprints, and
    hairs. These cause bursts of errors. CIRC increases the "Hamming
    distance" between adjacent errors caused by typical media defects
    thereby reducing the amount of forward error correction required to
    provide reasonable data recovery. CIRC does nothing to help correct
    underlying problems with the substrate, which degrades with time.

    It is well-known that Linux has various tools to read CDs and DVDs with
    the lowest possible error-rate, but these algorithms are not used by
    regular CD/DVD applications. Why? Because it can take many hours (even
    days) to recover reasonably accurate data from CDs and DVDs. Yes, I
    know data CDs and DVDs have a so-called data protection layer on top of
    the raw disc data just as TCP has a so-called guarantee of accurate
    data transmission. We all know that TCP delivers corrupted files far
    more often than it should, which according to Wolfgang this should be
    almost impossible.

    The Shannon-Hartley theorem of data transmission cannot be overcome by
    using any form of clever digital technology. The probability of having
    unrecoverable errors is always greater than zero and always increases
    towards maximum media read speed. If you think that by ticking a check
    box in a typically application's options you are guaranteed accurate
    data recording and retrieval then you do not understand the
    implications of having inaccurate data on your computer system. Either
    that or you employ enough lawyers to work around data corruptions as
    they occur.

    I shall not bother to address each of your points, Wolfgang, because
    you have completely failed to understand why the existing CRC
    mechanisms used for data integrity have proved themselves to archaic to
    the point of being next to useless for any other purposes than cost
    reduction and marketing hype.


     
    Pete A, Mar 15, 2012
    #44
  5. Dallas

    tony cooper Guest

    The newspapers haven't covered that. No pet obit listings.
     
    tony cooper, Mar 15, 2012
    #45
  6. Dallas

    Burt Johnson Guest

    And I've seen LOTS of cases where the disk died "the day before I was
    going to fully back it up!"...

    That is why an automated, daily incremental backup is so important. I
    have a local one made hourly (TimeMachine) and a remote one done daily
    (CrashPlan).

    No muss, no fuss, no forgetting, no extra work. Once it is set up, it
    can be forgotten until needed. It is then also easy to recover just a
    few files that were deleted 3 months ago when needed (happened to my
    wife recently).
     
    Burt Johnson, Mar 16, 2012
    #46
  7. [major snip, since Pete A does not answer lots of things
    where he's clearly wrong]
    "massive number"? Compared to what, the number of errors
    that can be corrected?

    You might also note that if the CD plays, the output of the
    CD data is beween zero and not noticeable.
    Old CD, right?
    Yep, the early CDs literally rust internally and thus have a
    rather short lifetime.

    How good a given recordable CD and a burner work together (measured
    by C1 and C2 error rates) varies *a lot* from burner to burner
    and from lot to lot.

    Actually, it seems legal to sell music CDs that
    - very creatively don't follow the Red Book Standard (and
    thus play on any given player just by pure chance)
    - install rootkit software (looking at you, Sony!)

    Just as, say, heart medicine products are to combat various heart
    illnesses and conditions .... but do nothing to help correct
    the underlying problems with the tissue, which degrades with
    time and will stop functioning within usually 100 years from
    production.

    Error correction helps to work with the inevitably existing
    error rates, even when they increase over time. Proof: The
    CD wouldn't even be playable without error correction.
    The various tools you mention often work around the limits of,
    say the Red Book Standard (no proper sector adressing possible,
    one long spiral (think 33,3 rpm record of olden days) instead of
    tracks and sectors (think floppy disk, hard disk)). Worse, it's
    not needed for most cases, it's only needed for damaged/'special'
    disks, where the reader's ECC doesn't cancel the errors good
    enough for the purpose used.

    And TCP doesn't guarantee 100% error free transmission. Never was
    designed for it. It only used 16 bit checksums (one for the TCP
    header and data, one for the IP header). It's a virtual
    circuit, not a virtual error free connection.

    So do you have any statistics how often a TCP transmission
    delivers wrong data, and have you compared that to the to be
    expected error rates? No?

    BTW, I haven't noticed "TCP" delivering corrupted files.
    That may, of course, be because applications layering on TCP
    may use their own error checking data.

    Actually, sending any kind of data below the channel capacity
    can be done at arbitrary low error rates. You want an error
    rate of 1 error in the lifetime of the universe? Sure. Can
    be done.
    I think you are wrong here. >0, sure, but arbitrary low (if
    the engineers make that effort) and I don't think it's read
    speed related. Or would you say the probability of unrecoverable
    errors per bit read is larger on the outer rim of a HDD than on
    the inner rim?

    Additionally a well designed medium can have a maximum read
    speed so that the signals at that speed are read best and are
    worse at any slower speeds. (magnet patterns passing under a
    reading head elict a stronger signal when they pass faster.
    In fact, at speed 0 they don't elict a signal. So increasing
    the speed *improves* the signal.)

    If you think that I'm a GUI constrained checkbox clicker you
    are sadly mistaken.

    Do you have looked at the undetectable error rate of your hard
    drives (according to the manufacturer) recently? Of course
    you have or will have inaccurate data on your computer
    system. What will you do about it?
    Lawyers restore undetectabble errors? That's a new one.

    You are the one to use *communication channel bandwidth* theorems
    on *storage media* where the channel isn't the bottleneck of
    reliability or speed.

    OK, what's your payable, scaleable suggestion for the
    "existing CRC mechanisms" in, say, hard disks (where they
    don't even exist in the form you claim)?


    [major snip, since Pete A does not answer lots of things
    where he's clearly wrong]

    -Wolfgang
     
    Wolfgang Weisselberg, Mar 18, 2012
    #47
  8. Dallas

    tcroyer Guest

    --
    Tom Royer
    If you're not free to fail, you're not free. -- Gene Burns
    From the Carbonite web site:

    Bank-level privacy protection
    To prevent unauthorized access to your files, we encrypt them with 128-bit
    Blowfish encryption before they even leave your computer. Then they're
    transmitted to our data centers using Secure Socket Layer (SSL) technology,
    the same security technology used in online banking and e-commerce
    transactions. Your files remain encrypted on our servers. We even pay a
    professional security firm to test our intrusion defenses to make sure no
    one else can access your files.
     
    tcroyer, Mar 18, 2012
    #48
  9. Dallas

    Alan Browne Guest

    On 2012-03-09 10:38 , tcroyer wrote:
    <snip>

    Don't top post.
     
    Alan Browne, Mar 18, 2012
    #49
  10. Dallas

    Dallas Guest

    burned down / was burglarized, I go to the CrashPlan backup.

    But does the CrashPlan make a bootable clone of the hard disk?

    If the answer is 'no' then recovering from a burned down house would
    look like this:

    1) Buy new computer with OS installed.
    2) Purchase all my software again because install CDs were lost in the
    fire.. i.e. MS Office, Photoshop, Outlook, etc.
    3) Install all the Applications, connect to your ISP ect.
    4) Recover your data from CrashPlan and restore it to your new computer
    in correct locations.

    If you have an off site clone of a hard drive, you'd just install it in
    a new box. Now, I have no illusions about trying to boot that hard
    drive in a new computer... in theory you'd boot into safe mode and
    begin the process of installing new drivers. But, completing that,
    you'd have all your programs and data ready to go.
     
    Dallas, Mar 21, 2012
    #50
  11. Dallas

    Burt Johnson Guest

    And you would lose all the data from the last time you made the clone.
    You certainly won't be making a clone daily, or taking your backup
    offsite daily.

    Go ahead and make your clone if you really want, but make sure you also
    have an automated daily offsite backup, so the work you are doing today
    is safe, not just what you did 6 months ago.

    The 'burned down house' scenario is the extreme end, and has a low
    probability of being needed. Not zero, and we have had major regional
    fires burn within a few doors of our houses on two occasions (two
    different houses for that matter).

    if that really does happen, rebuilding the computer would be one of the
    minor issue to address, and I would rather know that I lost NO data at
    all, rather than maybe saving a few hours of recovery, but not having
    anything I worked on for the past xx months.
     
    Burt Johnson, Mar 21, 2012
    #51
  12. Dallas

    Mark F Guest

    Beforehand:
    1. Use your favorite stand-alone utility to make a single file image
    of your system disk to a file on a drive.
    2. Use your favorite archiving program (WINRAR?) to split the
    single file into multiple files and put them on a new recovery
    drive. (Backup and restore of a 40GB file over the web with
    throttled backups isn't fun.)
    3. Repeat 1 and 2 so you wind up with a single disk can be used
    to make clones for all of your system disks.
    4. stick ISO's of all of your software on the recovery drive as
    you get the software. (You won't use these for disaster recovery
    but it will help when you move to a new operating system
    or get a new computer.)
    5. Connect the drive with the split images to one system for backup
    to the cloud. (Clone this drive or the one with the unsplit
    image backups and keep copies at your friends or in your safe
    deposit box.)
    NOTE: Encrypt or use self encrypting disks for the backups as you
    desire.
    Only need one for all of your computers, just to un WINRAR the
    split-image files.
    All ISO's of all of these were kept in my step 4, but you don't
    use them for recovery.
    Correct: my additional steps show a practical way of getting
    files of a practical size to backup to the cloud so you can recover
    from your bank also being under water.
    In addition, everyone probably has many programs that were only
    available for installation from the web with no way to save the
    installation files. Many of these are no longer available, have to
    be installed in the same order to get things to work the same way,
    etc. Basically, no hope of installing any system from scratch and
    having it work the same way. (Some people install from scratch
    regularly and learn to deal with the most common things that mess up
    or can't be installed on an operating system that can be installed
    on currently available computers.)
     
    Mark F, Mar 21, 2012
    #52
  13. Buy just the OS; you'll install the rest of the applications from your
    archived installation files after you've restored your data. (Most of
    the applications I got on CD have upgraded over the net a few times by
    now, so I've got files in store. And I create files documenting serial
    numbers that live in the same directory.)
    All my data files live on my fileserver. The backups of that are
    directly readable drives (for any Solaris server; they're ZFS
    filesystems). I can get to them without building a new server, simply
    by booting any x86 computer from a Solaris LiveCD.
     
    David Dyer-Bennet, Mar 21, 2012
    #53
  14. [/QUOTE]
    Deoending on the company that you trust your data with, the
    major problem will not be that data is lost, but that the
    company or product folds. But any other backup system has the
    same problem, it's not 100%. HDDs die, tapes break, CDs, DVDs,
    BlueRays deteriorate, break, become unreadable, ...

    So don't use one cloud based system as the sole storage for
    important data.

    True. Try not to buy them.
    Well, one of the two isn't compatible enough to the standards.
    And MD5 is very strong evidence: a different MD5 does mean
    different input data.

    Weaker it may be in cryptographic terms. But that means stuff like
    replacing the original with a fake that has the identical MD5 ...

    Your body also has not the faintest idea what your brain is
    trying to tell it, and doesn't give a rat's ass either.
    That completely depends on what software you're using.
    There's a lot of rather intelligent programs that do tons of
    things a fly couldn't even comprehend. There are learning
    programs (e.g. learning to classify between spam and
    non-spam), and programs that learn what you like to see on
    TV and act upon it ...
    Non-sequitur: Completely unconnected to each other.

    -Wolfgang
     
    Wolfgang Weisselberg, Mar 25, 2012
    #54
  15. Not necessary.
    You just lost *all* your hardware, including the bootable clone
    copy (if you have one). Without a computer with OS you won't be
    able to access computer data. After all, you need to access your
    CrashPlan account in some way.

    Aren't they bound to your hardware? In which case they won't
    work with your new computer anyway ... whether your house burned
    down or you just upgraded your computer.

    Other software, you just copy the files and settings to the right
    place (and that's where the Windows registry sucks 10s of miles
    sized asteroids through thin straws).

    Connecting to your ISP is step 1.3, downloading your data is
    1.6 and then you do have all your applications.
    you have lost your access to it. The key to the bank vault
    is lost.

    Wrong.
    1) Buy a new computer without an OS and without a harddrive.
    (have fun finding one)
    2) fetch the clone drive.
    3) Install the clone drive into the computer. Replace the
    mainboard, since the hard drive doesn't connect to the mainboard
    any more.
    4) Try to boot the computer.
    5) Hope you have all the drivers for everything with your computer.
    Install them.
    6) Connect to the internet. No problem, but ... your whole system
    is months or even years out of date! Including the AV software.
    It takes longer just to update a new Windows than the usual
    timespan between attacks on the holes it patches on a given IP,
    so ...
    7) ... try to update all your software faster than an attack
    gets you.
    8) Try to make sure your computer has not been attacked yet.
    9) buy all the stuff you bought again since you made the last
    clone and install it.
    10) Cry for all the rest you didn't backup since then ...
    The good thing about theory and praxis is that in theory they
    both behave the same. In praxis ... well, that's different.

    -Wolfgang
     
    Wolfgang Weisselberg, Mar 25, 2012
    #55
  16. No, with the possible exception of Windows itself (depends on what kind
    of license you have; the default that comes with big-name hardware is
    rather limited, but I've rarely had that license). But everything else
    you can move to the new computer (I've moved Photoshop, Bibble Pro,
    Photo Mechanic, Thumbs Plus, various plugins, Picture Window Pro, and
    oodles of other stuff across a number of computers).
    Funny, because they invented the registry to make problems from people
    moving stuff around smaller.
    You've probably got your ID (it lives in my wallet in my pocket, so if I
    survive my ID probably does too), you can walk into the bank and get
    them to being a locksmith to let you into your vault.

    I think you get two copies of the key, anyway. One can live at home,
    one in your car, or something. Or in a drawer at work.
    Trivially easy, I've bought that way frequently.
    No, install the proper drivers for the new mainboard. It may well do
    this itself (Windows generally does).
    I've never had problems with this when configuring new computers or
    doing motherboard replacements. The key is, your computer shouldn't
    ever be *directly exposed* on the internet, where other things can
    connect to it (unless it's a hardened server system). But mostly at
    home you're behind NAT, in which case outside systems can't reach yours.

    So then, just avoid browsing anywhere except Windows Update until you're
    updated.

    Or if you're badly paranoid, download them on another computer onto a
    DVD.
    Which is why automatic backups are good, yes. And why automatic
    off-site is especially good.

    Still, I brought 16GB of data back from a photo trip to the zoo
    yesterday, and that takes a while to backup via my outbound Internet
    bandwidth.
    I've had consistently good results with moterhboard upgrades, though,
    which is what this is (the hard part).
     
    David Dyer-Bennet, Mar 26, 2012
    #56
  17. Never have *just one* backup copy, is the rule. This rule is
    medium-agnostic.

    And remember that if you lose your backup -- you haven't lost your data
    yet.
    Can be a nasty problem, especially because you may not notice until much
    later. I dunno, can fraud suits be brought against them? It seems to
    me they're not actually implementing the standards they claim to be
    implementing.
     
    David Dyer-Bennet, Mar 26, 2012
    #57
  18. Dallas

    Pete A Guest

    Fraud? This is my interpretation of EULAs: I cannot install the
    OS/hardwar driver/software/service unless I actively agree to what in
    essence says "You agree that you are installing a bug-ridden consumer
    toy that will occasionally fail and destroy your data. This product is
    for entertainment/amusement purposes only therefore the vendor cannot
    be held responsible for any data loss and/or leakage of personal
    information to third parties." Have I missed something?
     
    Pete A, Mar 26, 2012
    #58
  19. You've missed that such contract of adhesion are specifically forbidden
    in many states.
     
    David Dyer-Bennet, Mar 26, 2012
    #59
  20. Dallas

    Pete A Guest

    Obviously not in the states where it is of utmost importance:

    <http://www.bbc.co.uk/news/technology-17486847>

    I don't see any mention of suing the software/harware vendors.
     
    Pete A, Mar 26, 2012
    #60
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.