CoolPix 950, FAT16 problem with large card

Discussion in 'Digital Cameras' started by Dave Platt, Jan 14, 2008.

  1. Dave Platt

    Dave Platt Guest

    I just encountered a problem with a Nikon CoolPix 950 which caused
    inconvenience and might have resulted in a loss of images from the CF
    card. As a similar problem might exist with other camera models,
    other people might want to be aware of the issue.

    I just bought the 950 used. It came with the original 8MB Compact
    Flash card... much too small to be useful. I stopped by an
    electronics store and bought the smallest-capacity CF card they had in
    stock... 1 GB.

    Popped it in the camera, let the camera format it, and started taking
    photos. All seemed to be well. After a few days, I pulled the card,
    popped it into a CF/PCMCIA adapter, and inserted the adapter into my
    laptop's card slot.

    Voila... nothing happened. The icon didn't appear. No files available.

    I dug around a bit, and found that the Linux kernel's support module
    for the FAT/VFAT filesystem objected to the filesystem on the card.
    The partition table was valid, there was a single FAT16 partition on
    the card, but the filesystem on that partition was unmountable. The
    kernel reported that the filesystem had almost a quarter-of-a-million
    clusters, which is more than the limit allowed for a FAT16 filesystem.

    I tried formatting the partition using the linux "mkfs.vfat" program.
    The filesystem formatted OK, the filesystem-test program "fsck.vfat"
    said that it was OK... but when I put the card back into the camera,
    the camera insisted that the card was unformatted. When I allowed the
    camera to reformat the card again, the same problem resulted.

    So... Linux wouldn't read the camera's filesystem, and the camera
    wouldn't read the Linux-constructed filesystem.

    This problem didn't exist for the 8MB card which came with the 950,
    nor with a 128 MB card which came with a CoolPix 800 I'd bought
    earlier.

    Further testing showed why... the Linux utility was (silently)
    deciding to build a FAT32 filesystem on the 1-gig card due to the
    large number of sectors, and the CoolPix 950 can't read FAT32.

    So, I used "mkfs.vfat -f 16" on the Linux system, forcing the system
    to format the card with a FAT16 filesystem. The Linux utility did so,
    simply increasing the size of the clusters so that the number of
    clusters didn't exceed 2^16.

    Problem solved, I believe. The CoolPix 950 happily accepted the card
    with this FAT16 partition without requiring a reformat, built the dcim
    directory hierarchy, and has allowed me to take a bunch of pictures.
    Linux can mount the filesystem and read the files, and the Linux
    filesystem-check program declares that the filesystem is clean (no
    internal inconsistencies).

    I suspect that the original formatting of the card, by the CoolPix
    950, was a case of data corruption waiting to happen. The camera
    appears to have built an "illegal" filesystem... one which had cluster
    numbers that couldn't be stored in 16 bits. I'd guess that the
    camera's firmware wasn't written to handle such large cards, and
    doesn't fail gracefully when faced by them.

    If I'd stored enough images on the card, there's a fair chance that
    some of the images (old or new) would become corrupt or unreadable...
    or the card would have "run out of space" when it was only about 25%
    full.

    Fortunately, the camera firmware seems able to handle FAT16
    filesystems with larger cluster sizes. I'll have to try filling it up
    with big TIFF images and make sure that everything still works OK when
    the card has lots of data on it.
     
    Dave Platt, Jan 14, 2008
    #1
    1. Advertisements

  2. Dave Platt

    Dave Platt Guest

    I believe you're incorrect, John.

    FAT16 has an ultimate limit of 2 GB, which is set by the maximum
    number of clusters, multiplied by the maximum cluster size. As the
    entry you cited pointed out,

    The limit on partition size was now dictated by the 8-bit signed count
    of sectors-per-cluster, which had a maximum power-of-two value of 64.
    With the usual hard disk sector size of 512 bytes, this gives 32 KB
    clusters, thereby fixing the "definitive" limit for the FAT16
    partition size at 2 gibibytes.

    2 gibibytes is, indeed, 65536 sectors times 32768 bytes per sector...
    and according to the Wikipedia article, the limit in FAT16 is actually
    only 65517 clusters.

    However, you can only reach the 2-gib limit if you use 32k-byte
    sectors. If you use fewer sectors per cluster, the 65517-cluster
    limit still applies and the maximum size of the filesystem is reduced.

    The problem with the Coolpix 950 seems to be that its filesystem
    formatter does *not* use 32k-byte clusters. It formats the filesystem
    using fewer sectors per cluster... enough fewer that the required
    number of clusters exceeds 65517. It shouldn'ta done that, I think.
    The cluster size used by the camera allows support of a 128MB flash,
    and I think it would allow a 256MB flash, but it won't go beyond that.

    While the "Final FAT16" format does increase the number of sectors
    allowed on the disk to a 32-bit number, it didn't increase the cluster
    index fields beyond 16 bits. That was left for the later FAT32
    filesystem formats.
     
    Dave Platt, Jan 14, 2008
    #2
    1. Advertisements

  3. Dave Platt

    Don Wiss Guest

    I've been taking digital pictures for more than eight years. I've never
    formatted a CF card. They come formatted.

    Don <www.donwiss.com> (e-mail link at home page bottom).
     
    Don Wiss, Jan 14, 2008
    #3
  4. Dave Platt

    ray Guest

    Rather strange. I've used quite a variety of CF and SD cards from 8mb to
    2gb on several Linux systems and never had an issue. What distro are you
    running?
     
    ray, Jan 14, 2008
    #4
  5. Dave Platt

    ray Guest

    As I understand it, FAT16 has a maximum theoretical limit of 4gb though
    virtually all implementations have a practical limit of 2gb.

    Note: 2**32 = 4gb. Most implementations seem to want to avoid the full use
    of unsigned integers.
     
    ray, Jan 14, 2008
    #5
  6. Dave Platt

    Dave Platt Guest

    Debian (testing/unstable), with a 2.6.23 kernel.

    My guess at this point is that the CoolPix 950 firmware has a hard-coded
    4k-byte cluster size coded in its FAT16 formatting logic. This was
    probably seen as adequate to handle all of the CF cards then on the
    market, back when the camera was released, and the camera was simply
    not tested with cards larger than around 128 MB.

    Formatting the card as FAT16, with a suitably-large cluster size (16k
    or 32k bytes) seems to entirely resolve the problem.
     
    Dave Platt, Jan 14, 2008
    #6
  7. All sorts of "hard coding" logic (sic) can bite you in the @$$ as
    technology moves forward while your piece of crap^h^h^h^h equipment
    (camera) stands still. :)

    I have an old Kodak DC-280 which is only capable of displaying the
    (approx.) number of images available on the CF card as "nnn" (i.e., a
    max count of 999.) With the camera set at its max ("High")resolution a
    1 GB card will play nice in the camera, with a image count that will
    only display as' "---" in the nnn-count filed. With an empty 2GB card
    installed, the camera claims only 12 images are left. And, indeed, it
    would only take 12 pictures and then displayed "Card Memory Full".

    I believe the counting problem is the typical binary (or decimal) field
    overflow (without detection) and the rest of the camera's logic
    continues on with illogical/erroneous values.

    In a slightly-not-as-old Kodak DX3600, the 1 GB card claims 1580 images
    are available. With the 2 GB card, it claims 3158 images are available.
    From date-of-purchase to this day I am sure I have not taken 3158
    pictures with this camera. Maybe not even 1580...

    Oh, those were the days! 2 MB Pixel technology.

    Jonesy
     
    Allodoxaphobia, Jan 15, 2008
    #7
  8. Dave Platt

    Don Wiss Guest

    More than enough for eBay. And the 950, with its excellent macro
    capability, makes a good eBay camera.

    Don <www.donwiss.com> (e-mail link at home page bottom).
     
    Don Wiss, Jan 15, 2008
    #8
  9. Dave Platt

    Dave Platt Guest

    *sigh*. Right you are. I shouldn't type in long messages when I'm
    tired... fingers run ahead of brain and eyes don't see the typos.
    That explanation, although perhaps convenient, fails to account for
    another important fact I've just confirmed:

    ->> Windows 98 also says that the filesystem format is bad! <<-

    I reformatted the CF card in the camera, powered down, moved the card
    to an adapter in an old Windows 98 laptop I keep around (I knew there
    was a reason to keep it!) and ran the Win98 Scandisk utility on the
    filesystem.

    Scandisk reported no less than 6 severe errors. Everything on the
    card (two subdirectories and one file) were reported as being stored
    in an illegal location. Scandisk's final diagnosis was along the
    lines of "Seriously ill".

    I then used Win98 to reformat the filesystem. This worked
    fine... Windows used a cluster size of 32 sectors (16kb), just as the
    Linux utilities had done. The Linux and Win98 formattings are nearly
    identical... Windows set aside some hidden sectors, and didn't use the
    last odd sector in the partition, but there are no other significant
    differences.

    The camera accepted the Windows-formatted card without complaint. So
    does my Linux laptop. So, it seems that the 950's formatter is the
    odd-man-out: neither Windows 98 nor Linux consider its formatting of
    this card to be correct, while Windows and Linux produce compatible
    formattings that the 950 will also accept.

    In order to get a Linux-independent description of how the filesystem
    should look, I went back to the Wikipedia page you cited, and followed
    the link to the ECMA EC-107 standard, which documents the FAT12 and
    FAT16 filesystem interchange formats.

    Experimental procedure: I zeroed out the first megabyte of the
    CF card, erasing the MBR if present, the partition table, and the boot
    block and FAT and root directory of the filesystem on the card. I
    then inserted the card into the Coolpix 950 camera and powered it on.
    The camera quite properly reported that the card was not formatted,
    and offered to do the formatting. I accepted. A few seconds later,
    when formatting was complete and the camera was ready to record
    photos, I powered it off, removed the card, and put the card back into
    my laptop's slot.

    Once again, Linux recognized a valid partition table, but objected to
    the FAT16 filesystem, saying that it had too many clusters.

    I bulk-copied the first megabyte of partition 1 and saved it for
    further analysis.

    Here's a hex dump of the boot block and the first sector of the first
    copy of the FAT:

    0000000 e9 00 00 20 20 20 20 20 20 20 20 00 02 08 01 00
    0000016 02 00 02 00 00 f8 c8 03 3f 00 10 00 3f 00 00 00
    0000032 f1 38 1e 00 00 00 00 00 00 00 00 00 00 00 00 00
    0000048 00 00 00 00 00 00 46 41 54 31 36 20 20 20 00 00
    0000064 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    *
    0000496 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa
    0000512 f8 ff ff ff ff ff ff ff ff ff 00 00 00 00 00 00
    0000528 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
    *
    0001024

    Offsets from the beginning are in decimal, counting from 0 (the EC-107
    spec starts counting from 1, and I'll use that convention below).
    Multibyte integers are of course in little-endian storage order.

    My interpretation of this, based on the EC-107 spec, is as follows:

    - The creating system identifier, in byte positions 4 through 11, is
    all blanks.

    - The SS (sector size) in byte positions 12 and 13 is 0x0200, or 512.

    - The SC (sectors per cluster) count in byte position 14 is 8. Since
    the cluster size equals SS*SC, the cluster size is therefore 4096
    bytes.

    - The RSC (reserved sector count) in byte position 15 and 16 is 1.

    - The FN (number of FATs) in byte position 17 is 2.

    - The RDE (count of entries in the root directory) in byte positions
    18 and 19 is 0x0200, or 512.

    - The TS (total sector count) in byte positions 20 and 21 is zero.
    This is to be expected for an extended FDC descriptor which
    describes a filesystem having more than 63353 sectors.

    - The SF (sectors per FAT) in byte positions 23 and 24 is 0x03C8,
    or 968.

    - The SPT (sectors per track) in byte positions 25 and 26 is 0x003F,
    or 63.

    - The NOS (number of sides/heads) in byte positions 27 and 28 is
    0x0010, or 16.

    - The TS (total sector count) value in byte positions 33 through 36 is
    0x1E38F1, or 1980657.

    - The volume ID number in byte positions 40 through 43, and the volume
    label in byte positions 44 through 54, are all zeros.

    - The file system type in byte positions 55 through 62 is "FAT16 ".

    Looking at the partition table itself with the Linux "fdisk" program, I
    see:

    Disk /dev/sdb: 1014 MB, 1014644736 bytes
    16 heads, 63 sectors/track, 1966 cylinders
    Units = cylinders of 1008 * 512 = 516096 bytes
    Disk identifier: 0x00000000

    Device Boot Start End Blocks Id System
    /dev/sdb1 * 1 1965 990328+ 6 FAT16

    The block count is in 1k blocks, and the "+" indicates that there's an
    additional sector at the end. Twice 990328, plus 1, equals 1980657,
    indicating that the space described by the disk's partition table, and
    the space described by the filesystem boot block, are identical.

    The boot block / FDC says that the filesystem has 1980657 sectors, and
    that there are 8 sectors per cluster. This would require 247582
    clusters (minus a few to account for the size of the system area).

    Since a FAT16 FAT uses two bytes per cluster, each sector in the FAT
    will hold 256 cluster entries. Dividing 247582 clusters by 256 clusters
    per FAT sector, and rounding up, we find that we'd need 968 sectors of
    FAT... and that's just what the SF value in the boot block says that
    the filesystem is using.

    If I run the Linux "fsck.vfat" program on the partition, here's how it
    interprets what it sees:

    dosfsck 2.11, 12 Mar 2005, FAT32, LFN
    Checking we can access the last sector of the filesystem
    Boot sector contents:
    System ID " "
    Media byte 0xf8 (hard disk)
    512 bytes per logical sector
    4096 bytes per cluster
    1 reserved sector
    First FAT starts at byte 512 (sector 1)
    2 FATs, 16 bit entries
    495616 bytes per FAT (= 968 sectors)
    Root directory starts at byte 991744 (sector 1937)
    512 root directory entries
    Data area starts at byte 1008128 (sector 1969)
    247336 data clusters (1013088256 bytes)
    63 sectors/track, 16 heads
    63 hidden sectors
    1980657 sectors total

    The data cluster count shown by the checker accounts for the size of
    the system area, while my crude calculation above does not, so the
    smaller number of clusters is to be expected.

    So, Mr. Navas, I'd like to ask you two questions:

    (1) Based on the actual data shown above, do you agree that the
    CoolPix 950 did create a FAT16 filesystem which contains more
    than 247,000 clusters of 8 sectors (4kb) each?

    (2) Is a FAT16 filesystem capable of being legal and self-consistent
    if its maximum number of clusters exceeds 0xFFF7?

    As far as I can tell, the way that the Linux kernel and utilities are
    interpreting the FAT16 boot block are quite consistent with ECMA
    EC-107, and also to match the Win98 interpretation. The camera really
    did create a filesystem which is not capable of correct operation -
    disaster will occur once the camera starts trying to use clusters
    numbered above 0xFFFF. Neither the directory entries (Table 5) nor
    the cluster chains in the FAT can store such cluster numbers without
    truncation. Lost data and "duplicate allocation" errors are sure to
    result.

    As an experiment, I instructed the Linux mkfs.vfat program to format
    the partition in FAT16 mode with 8 sectors per cluster (-F 16 -S 8).
    The program flatly refused to do so, telling me that this would
    require more clusters than the FAT can hold and that I should
    specify a larger number of sectors per cluster.

    The sectors-per-cluster value of 8 used by the Coolpix 950 would work
    correctly with CF cards of up to 128 MB, and it should work correctly
    with a 256 MB card. With a 512MB card or above, the cluster-number
    indices will exceed 16 bits in width, will be truncated, and Bad
    Things will happen.

    If you disagree with my conclusions about the (in)correctness of the
    filesystem created by the CoolPix 950, I'd really appreciate a
    *specific* pointer to how you disagree with my interpretation of the
    data in the filesystem boot block, or a pointer to technical documents
    which show how a FAT16 filesystem can actually hold more than 2^16
    clusters.
     
    Dave Platt, Jan 16, 2008
    #9
  10. Dave Platt

    Dave Platt Guest

    It seems you were correct about the problem. Good job.[/QUOTE]

    Thank you.

    I'll probably go out and buy myself a few new-old-stock 128MB CF
    cards, just so I know that I have a backup card or two that the camera
    can format correctly.
     
    Dave Platt, Jan 16, 2008
    #10
  11. Why not simply format the cards in the computer?

    jue
     
    Jürgen Exner, Jan 16, 2008
    #11
  12. Dave Platt

    Dave Platt Guest

    Oh, that's what I intend to do for my existing 1-gig card.

    However, I have a minor (at least) case of "absent-minded professor"
    personality, and I might suffer a brainfart and use the camera's
    "format card" feature as a fast way of doing a "delete all". That
    would leave me with a card filesystem that I couldn't trust, until I
    got around to (somehow) pulling the photos off of the mis-formatted
    card and reformatting it again in the computer. I'd be seriously
    annoyed with myself.

    Since I want to have a spare card or two anyhow, it won't kill me to
    have them be of a somewhat smaller capacity... a 120MB card will hold
    a couple of hundred photos at Normal resolution on that camera, and
    that's probably more than I'm ever likely to need to shoot between
    downloads.
     
    Dave Platt, Jan 16, 2008
    #12
  13. Dave Platt

    Dave Platt Guest

    Because the camera might not be able to handle it properly in all
    situations, particularly when well over the maximum size it was designed
    and tested for. Would be a shame to lose lots of pictures.[/QUOTE]

    That's the next test I really need to do. Set up the camera with a
    well-formatted 1-gig card and a proper AC adapter, aim it at a clock,
    tape down the shutter release button :)-) and let it take pictures as
    fast as it will until it fills the card. Then, see if they're all
    readable and intact. Delete a few, record a couple more, and see if
    the camera handles the situation correctly.

    Most operating systems would handle this situation properly. Others
    would not. I recall that some years ago, the VxWorks o/s code for the
    FAT file system would read the entire FAT and all of the directory
    hierarchy into RAM when the filesystem was mounted, and keep them
    there for as long as the filesystem was in use. Tolerable, perhaps,
    if you're dealing with a single slow floppy disk, but it was sheer
    unnecessary hell if you had a couple of gigabytes of small files on a
    hard drive. It took megabytes of RAM for the in-memory index, and
    several minutes to mount the drive. Darn near torpedoed the embedded
    system we were trying to ship... we had to invent an alternative
    data-storage methodology in a terrible hurry.
     
    Dave Platt, Jan 17, 2008
    #13
  14. That's the next test I really need to do. Set up the camera with a
    well-formatted 1-gig card and a proper AC adapter, aim it at a clock,
    tape down the shutter release button :)-) and let it take pictures as
    fast as it will until it fills the card. Then, see if they're all
    readable and intact. Delete a few, record a couple more, and see if
    the camera handles the situation correctly.

    Most operating systems would handle this situation properly. Others
    would not. I recall that some years ago, the VxWorks o/s code for the
    FAT file system would read the entire FAT and all of the directory
    hierarchy into RAM when the filesystem was mounted, and keep them
    there for as long as the filesystem was in use. Tolerable, perhaps,
    if you're dealing with a single slow floppy disk, but it was sheer
    unnecessary hell if you had a couple of gigabytes of small files on a
    hard drive. It took megabytes of RAM for the in-memory index, and
    several minutes to mount the drive. Darn near torpedoed the embedded
    system we were trying to ship... we had to invent an alternative
    data-storage methodology in a terrible hurry.[/QUOTE]

    That was also a failure mode of the Mars rovers, Spirit and Opportunity.
    The solution was to delete a whole bunch of files that were only needed
    during flight.

    --Gene
     
    Gene S. Berkowitz, Jan 20, 2008
    #14
  15. Dave Platt

    Dave Platt Guest

    That was also a failure mode of the Mars rovers, Spirit and Opportunity.
    The solution was to delete a whole bunch of files that were only needed
    during flight.[/QUOTE]

    Oh, yeah... I remember breaking out in disbelieving laughter when I
    read about the rovers' problems (and the cause and solution) in the
    paper. I never expected the problem I ran into, to ever show up again
    after so many years (and in such a high-profile situation)! It was
    certainly a *very* good thing that the project team for the rovers had
    implemented a safe-boot mode which bypassed the flash disk and let
    them get the system back on the air.

    You sure do have to be careful when the nearest bootable-floppy drive
    is going to be millions of miles away :)
     
    Dave Platt, Jan 20, 2008
    #15
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.