Best lossless compression for DV-AVI files?

Discussion in 'Amateur Video Production' started by JT, Nov 21, 2004.

  1. JT

    JT Guest

    Are there lossless (or nearly so, perfection isn't required)
    compressors that can scrunch DV-AVI files 2:1 or more? I'd like to
    archive files that may later be converted back to DV for editing with
    the likes of Premiere, so the process should be fairly undemanding of
    CPU time.
     
    JT, Nov 21, 2004
    #1
    1. Advertisements

  2. JT

    david.mccall Guest

    Sorry, but you aren't allowed to use lossless and compression
    in the same sentence. DV is already pretty compressed. Even
    recompression to the same codec is considered something
    to avoid, if you can. If you cut that in half you will likely see
    some deterioration, but if you then edit that in Premiere it
    will suffer another recompression (at minimum everything that
    changes (as in text overlays, color correction dissolves, etc.).

    If you then put that to DVD, you will get another generate another
    layer of heavy recompression. Compared to a VHS dub, it
    will still look pretty good, but it will be quite a bit different from
    the quality of your original DV footage.

    When people talk lossless they are talking about schemes
    that keep everything that changes within a frame. You save
    a lot on a completely black screen with very little graphics
    or text on it, but a frame with a lot of detail won't compress
    very well without loosing information.

    David
     
    david.mccall, Nov 21, 2004
    #2
    1. Advertisements

  3. JT

    JT Guest

    Lemme see, are you the guy with the perfection complex, or were you
    trying to amuse? I make that mistake, too.

    I assume you must know that there are dozens of lossless compression
    scheme for all kinds of data, and were making a joke. Pkzip is a
    simple one and for the range of avi files I sampled one day it ranged
    from 10% to 50%, but unfortunately the 10% end was much too prevalant
    - not enough to be worth the bother to me.

    The best compression schemes are tailored to and designed with as much
    knowledge as possible of the material being compressed and can almost
    handily beat a general approach. RLL is a good place to begin, after
    finding the best direction and/or frame size. I generally go for
    deltas after that.

    I've not worked with video data but I guess it's obvious that the
    first two choices to investigate would be field/frame temporal, and
    areal. I was/am hoping that someone suggests a piece of software that
    does that rather than explain that it can't be done. It certainly can.
     
    JT, Nov 21, 2004
    #3
  4. "JT" wrote ...
    Of course youall realize that DV is already compressed 5:1 by that
    methodology. It already creates artifacts visible to all but the most
    casual TV viewer.
     
    Richard Crowley, Nov 21, 2004
    #4
  5. JT

    Johan Stäck Guest

    david.mccall wrote:

    There *are* indeed situations were losless compression can be
    accomplished. (e.g. RLE coding of cartoons)
    However DV is not among them.
    You can try and zip a DV file and probably get a few % compression. This
    is because there is some repetitive (non-video) data in the dv frames
    that is suitable for compression.
    The compressed video will not compress further without loss of picture
    quality.

    However, if you decompress the DV frames and recompress them with a good
    mpeg codec you will (admittedly, this is subjective) be able to get a
    high rate of compression with a rather small loss in picture quality.

    /Johan
     
    Johan Stäck, Nov 21, 2004
    #5
  6. Not exactly correct, for example '.zip' is compressed, AND lossless.
    True
    You could for example zip a DV format file, but probably it will not make it a
    lot shorter.
    There are algos around that may still have some effect...
    For the original poster he could encode to mpeg2 or DivX perhaps.
    'slightly' lossy is vague, what you can do depends on the type of material,
    say how much motion and detail, a factor 2 with DivX is no problem I think.
    JP
     
    Jan Panteltje, Nov 21, 2004
    #6
  7. JT

    david.mccall Guest

    I wouldn't call it a perfectionist complex, but I am a professional
    that feels a need to provide his clients with appropriate levels of
    quality when I do work for them. However, not everybody needs
    the very best quality possible and corners can sometimes be cut
    if the needs are modest. For instance home movies, items that
    will only be viewed on the web at small sizes, documentation that
    needs to be kept small to fit on a CD, etc.
    Perhaps someone with better knowlege of compression will jump in,
    but there is a term for that kind of compression, and in fact, I think that
    type of compression is the is first step in most compression scheme.
    After everything that is redundent has been removed, then the compressor
    starts looking for stuff to throw ayay that you might not notice, then it
    starts
    in on the stuff you might be able to notice if you look close, but still
    might
    not care. DV is already at this level, so PK-Zip isn't going to make much
    of an impact on DV footage. The only way you can further compress DV
    is to throw away more important information that was removed on earlier
    passes. Al you can do is expiriment to see how much loss is acceptable
    to your aplication.
    But, you see, that has already been done with the first DV compression
    before going to tape. Chances are RLL algorythms won't do much for
    DV compressed material. You could gain a little more by doing some
    delta compression, but that makes editing harder. I'm not even sure
    many editing programs can deal with delta compressed video. The
    problems is that you can't just jump in the middle of delta compressed
    video and play or make a cut. The editing system has to backtrack to
    the previous I frame and then reconstruct the video for that section
    before you can scrub backwards or nake a cut. HDV compression
    has to do this which is why you can just edit it in every software. I don't
    know how smooth it is to edit HDV at this point, but it will get better
    as processor speed goes up.
    If only it were true. I wish you luck, but I think you will find that your
    video
    looks somewhat different once you throw away enough detail to achieve
    2 to 1 on top of 5 to 1 compression. This level of quality may be
    acceptable to your application. As I said before, it will still beat a VHS
    dub,
    and god knows a lot of people found that acceptable for years. Even if you
    do that and output MPEG-2 (much heavier compression than DV) it still
    might be good enough for your needs. However many applications would
    find that much recompression unacceptable.

    I did intend a touch of levity in that first statement about not being
    allowed
    to use lossless and compression in the same sentence, but beyond that,
    I'm being serious.

    Your best bet would be to dump the DV into an editor and cut out the
    useless parts, then dump the remainder back to DV tape or cut it into
    4 gig, or smaller, chunks and then store the data on DVDs as raw DV.

    David
     
    david.mccall, Nov 21, 2004
    #7
  8. JT

    david.mccall Guest

    It may be acceptable for his needs, but it may not be much fun
    to edit with, even if his editor can handle MPEG-2 or DivX,
    and the recompression to store the result will add another hit.
    It may still be acceptable to some, but to think you can do any
    decent level of further compression, without loss, with a DV
    source is a pipe dream.

    David
     
    david.mccall, Nov 21, 2004
    #8
  9. JT

    Jim Gunn Guest

    This question keeps popping up again and again, but the answer is
    clear . For long term archiving and backups, mini-DV tapes are the
    only sensible way to go. Assuming you acquire footage on mini-DV,
    your backups will be the same format. Just copy your original camera
    masters from deck to deck or camera to camera (or even pc to deck
    using Scenalyzer Live like I often do) The tapes cost only about
    $4.50 apiece in bulk and they can hold over an hour of DV-AVI video
    (13Gb+). Why try to compress to a file and lose quality and keep your
    important and presumably irreplaceable footage on risky magnetic
    temporary storage like a hard disk drive?
     
    Jim Gunn, Nov 21, 2004
    #9
  10. My experience with tape is that it is much more unreliable then say CDR
    or DVD-R.
    Especially in moist and tropical oriented climates all sorts of stuff
    starts growing on tapes... and floppies etc...
    So, why not binary split the DV and make a couple of DVD+R?
    I have hundreds full of video...
    And, for Dave, regarding his remarks.. about editing. Of cause it is correct
    that you can cut DV material easy, it is just a collection of jpgs right?
    But for a mpeg2 or DivX encoded movie you can also cut on one frame.
    There will hardly be any quality loss if you work with the right bitrates and
    encoding parameters.
    So it all depends on what you have to store it on.
    But in no circumstance would I use tapes... I have never seen an old tape
    without some sort of problem.... dropouts.
    Go optical if you MUST keep the original.
    JP
     
    Jan Panteltje, Nov 21, 2004
    #10
  11. JT

    david.mccall Guest

    You make some very good points. I have seen tape fail a lot over the years.
    Unfortunately, recordable DVD media hasn't been around long enough to
    know what they look like in 30 years under various conditions.

    But, you are right. We know that tapes can fail. I've had
    very little luck with tape backups on a computer, and I'm
    not sure why DV tapes would fare better.

    David
     
    david.mccall, Nov 22, 2004
    #11
  12. "david.mccall" wrote ...
    And yet likely way over 90% of the planet's computer data is
    backed up on tape! There are whole mines filled with backup
    tapes from every industrial company you've ever heard of.
     
    Richard Crowley, Nov 22, 2004
    #12
  13. JT

    david.mccall Guest

    Ain't it the truth.

    I used to handle my records wit reasonable care,
    but they all went "clickity popity"

    I had a friend that just piled them up on the table,
    occasionally knocking them on to the floor, then
    drop an ash tray on the mess then play the record
    and it sounded better than mine, even though his
    system wasn't as expensive as mine. Go figure.
    Perhaps if I had kept my records in a mine?

    David
     
    david.mccall, Nov 22, 2004
    #13
  14. JT

    stankley Guest

    No offense intended, but I do find comments about 'will I be able to
    get data from this media in 30 years' a little silly. Since the data
    is in digital form, one can move it about from medium to medium to
    one's heart's content. I would anticipate that even in 10 years,
    we'll look back at DVD's and laugh at their puny capacity. In 30
    years, we'll probably have holographic media that can hold collosal
    amounts of data.

    If I were you, I'd split your DV AVI files across DVDs and store them
    carefully, perhaps multiple copies in multiple locations if the data
    is that precious. Then, when the next generation of media comes
    along, copy the files to this format. And let your children carry on
    with this tradition, and then your childrens' children, and your
    childrens' childrens' children...
     
    stankley, Nov 22, 2004
    #14
  15. JT

    david.mccall Guest

    Only if you have a drive that will take the disk, the integrity of the
    recorded
    material is still near perfect, and you still have access to the original
    codec.

    Sure, bits are bits, but if you were to be given an 8" floppy and had
    to get a file off of it, you might be in trouble. Likewise, in the analog
    universe, a 1" type A analog tape would be difficult, and even the
    various 1/2" reel to reel formats are getting difficult to access. Some
    laser disc (videodisc) are quite noisy due to the "laser rot". I have
    heard that some DVDs and CDs are starting to show signs of "laser rot"
    so that could be an issue, especially for digital information. Analog media
    can be pretty damaged and still work even though the image might be of
    poor quality. With digital you get some error correction, but once the
    damage
    gets beyond what the error correction can deal with you get nothing but
    garbage.

    David
     
    david.mccall, Nov 22, 2004
    #15

  16. Of course the key is to switch it over before the old format becomes a
    footnote in the history books. I hope someday we have star trek
    isolinear chips but as you know even those are vulnerable to power
    surges in the computer cord caused by Romulin fire :)
     
    Chris Phillipo, Nov 22, 2004
    #16
  17. JT

    nappy-iou Guest

    It's those ashes that polished the scratches out of the grooves. Learned
    that back in the 60's
     
    nappy-iou, Nov 23, 2004
    #17
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.