The Creekside Digital Blog

Frontispiece from a recent project scanned at 400dpi (click for detail)

It’s been awhile since our last update to the blog.  We’ve been heads-down with a lot of important digitization projects lately.  About a month ago, we performed onsite scanning on a large job for a legal services company here in the Northeast.  Currently, we have two large jobs in the shop.  One is a 461 roll conversion of 16mm microfilm for a large utility customer in the Western US.  This job includes Creekside Digital’s ultra high-quality film scanning, OCR, and PDF compression.  At the same time, we’re running a 134 roll 35mm microfilmed newspaper project for a library system (also out west), which includes scanning, OCR, and creation of FullText indices to allow the library’s staff and patrons to keyword search all 134 rolls of film simultaneously.  In between, of course, we’ve been running a variety of small jobs for folks all over the US and Canada: single rolls for individuals; orders which are a handful of rolls for engineering firms, hospitals, and churches; and plenty of fiche jobs for parts suppliers, school districts, and litigation service companies.

Very soon, microfiche orders will be accepted through our Small Order Form as well as rollfilm.  That means that if you have a small number of sheets / jackets of microfiche (up to 1,000), you’ll be able to simply fill out the form in your browser, entering the number of sheets you have and the type of processing you need (PDF or JPG, OCR, etc.).  The form will automatically calculate the exact price for you (including shipping!), and you’ll print it out and mail it to us with you fiche.  We’ll process your order and send your fiche back along with a DVD of your digitized fiche images.  We expect to have microfiche digitization available through the Small Order Form within the next few weeks.

Have a great 4th of July!

– Jim and the folks at Creekside Digital

.

Nobody has more microfilm than the US Government — and no one has created more public domain microfilm publications than the National Archives and Records Administration, or NARA.  In fact, Creekside Digital first ventured into the high-quality microfilm digitization industry as part of a software project built around NARA microfilm, and it continues to be a large part of our business today.

Period document from 35mm NARA microfilm (18x) preserving original paper texture and creases (click for detail: 300dpi JPEG @ 60% quality, 1.51 MB)

Currently, we’ve got another project in the shop that is all National Archives microfilm — in this case, period documents from the 1860s.  About half is 16mm, and the other half is 35mm.  Typical of NARA rollfilm, the original documents on the 16mm rolls are the same size has those on the 35mm rolls, but the reduction ratio is higher — meaning the frames are smaller, and more documents can fit on a single roll.  This also means that there’s less image information in each frame with which to work, which means it’s harder to get a nice, tight focus and set the scanner’s critical settings.  You really have to take your time.  Each roll of film is different, and there’s no “magic button” which automatically configures the best scanner settings whenever a new roll is loaded.  That’s up to the operator, and that’s where experience with the equipment and having “an eye” for what great images look like makes all the difference.

One common issue with scanning period docs on NARA microfilm is the tremendous variation in image density on the same roll.  Often, a roll will have a sequence of very dark documents, followed by extremely light papers.  This is makes it very difficult to optimally set lamp and gamma: too low, and you end up with a bunch of dark and mostly illegible pages (virtually impossible to frame detect) followed by a document scanned at “normal” brightness.  Too hot, and yes, the darker frames become legible — but the previously “normal” documents become overexposed and details like the paper grain start to burn out.  It takes a lot of extra time to get everything set “just right” in order to output great images across an entire 100′ roll of film.

16mm NARA roll in the Auditor showing tremendous variation in film density

Creekside Digital’s scanning platform offers an unprecedented level of bandwidth to allow it to better capture the full range of image density found in a given roll of film.  Grayscale images no longer look like a poor Xerox copy or over-dithered; shades of gray are as true to the original document as the source film allows.  And although perfect automatic frame detection remains a pipe dream with film of highly-variable density, we have software tools which allow a person to visually audit and correct the frame boundaries proposed by the machine.

The screenshot shown here illustrates the high variation in film density we’re encountering on this project.  Notice the automatic frame detection has done a good job at establishing initial frame boundaries (the yellow boxes).  Any part of any yellow box can be grabbed by the operator with the mouse and dragged into the correct position.  Frames can also be removed entirely, and dark frames which were cropped or skipped entirely by the machine can be added by a person prior to output.  Try that with first-generation technology that outputs the images “on-the-fly” as the film passes through the scanner!

.

What’s going on at Creekside Digital?  Well, it’s unseasonably cold here in Maryland today.  It was 12 degrees this morning and it’s up to a balmy 21 now.  It’s windy too, and the wind chill has given rise to a Severe Weather alert for our area.

Recently, we’ve been working on a pair of 35mm archival conversions for libraries in Southern California.  This means that the rollfilm is scanned at much higher resolution than normal — in this case, 400dpi — and that the output scans are 8-bit grayscale lossless TIFF images.  It’s a more expensive, time-consuming process than “ordinary” 200 / 300dpi scanning with lossy compression, but the result is as close to a bit-for-bit copy of your source rollfilm as you can get given the current state of scanner technology.

Archival 400dpi lossless image at 100% zoom (click for detail)

A few things to keep in mind:

• 8-bit grayscale TIFFs at 400dpi are huge.  A single 11″ x 17″ document might be 22 MB in size.  Architectural drawings or a single 2-up newspaper frame might be up to 100MB or more.  So delivery via external hard drive is pretty much a requirement for anything over a few rolls of film.
• Creekside Digital delivers its 8-bit grayscale TIFFs using LZW compression.  This is a “lossless” compression scheme which uses rather complex mathematical algorithms to shrink down the file size while preserving bit-for-bit perfection of the original scan (i.e., an LZW-compressed file is visually identical to its uncompressed counterpart).  The file size savings are marginal compared to “lossy” compression schemes, such as the popular JPEG format, but there’s no degradation of the image’s quality, either.


"Reader" 150dpi lossy image at 350% zoom (click for detail)

• Files this large are clumsy to work with, even on a fast, new computer.  They consume vast amounts of disk space and can take awhile to copy or print.  Generally, they’re too large to email.  Because of this, we usually create downscaled “reader copies” in PDF format.  These are lower-resolution files which library staff and patrons work with day-to-day; when they want to examine the original document with ALL of its high-resolution detail, they can pull the corresponding TIFF image.  Additionally, it’s possible to deliver multipage reader copies on DVD (preferred by librarians) for ease of use on non-networked PCs, with each multipage file corresponding to an entire source roll of microfilm.  And because the reader copies are so much smaller, you might fit fifteen rolls of film on a single DVD.
• PDF reader copies can be OCR’d, of course, allowing keyword search of typed / printed documents.

To see the difference between archival- and reader-quality images, click each of the two thumbnails above.  The top image is a section of a 400dpi archival scan viewed at 100% zoom (no magnification) — you can see that the high resolution of the scan captures a tremendous amount of detail, and that there is no noise or other “artifacts” associated with compression (as lossless compression was used).  The bottom image is the same frame, but as the image was a 150dpi reader, it was necessary to zoom to 350% magnification in order to view the same section of the original document at the same size — yes, archival scans are that large.  Note the compression artifacts and loss of image detail on the reader copy.  Again, readers are great for day-to-day use, but are no substitute for the detail captured with archival scans.

When you want the best-possible microfilm scan quality combined with the portability of the PDF format, contact Creekside Digital about archival scanning of your microfilm.

Happy Holidays!

.

We get a lot of questions from folks who have documents on microform but are not sure exactly what they have.  Hopefully this post will clarify things a bit.

Creekside Digital can scan all of the following major types of microform:

• Rollfilm.  As its name implies, rollfilm is a roll or spool of film containing a sequential series of frames.  It comes it two widths: 16mm and 35mm.  Typically, one finds rollfilm in lengths of 100′ or less.
  

  16mm rollfilm

  16mm rollfilm is the “typical” film width used for office-sized 8-1/2″ x 11″ / A4 documents.  At 24x reduction ratio, it’s possible to fit about 2,500 documents on a 100′ roll of microfilm.  One often finds check stubs, time cards, bills and invoices, and similar documents on this type of rollfilm.  The frames can be quite small, especially with duplex microfilm (which captures both sides of each original document), so care and quality equipment are required to get sharp images which are true to the original.  Sometimes spools of 16mm rollfilm will be enclosed in a square, open plastic frame to allow them to work with a specific microfilm reader / printer — but they will still have the same type of plastic spool and hub shown here.

  

  35mm rollfilm

  35mm rollfilm is used where larger frames are required due to bigger source documents.  It is very popular for newspapers, architectural drawings, and engineering diagrams.  Even lighting and quality optics with a wide depth-of-field are required to ensure there is no darkening or bluriness around the edges of the frames.  35mm rollfilm is also the film of choice for archiving older historic documents at a lower reduction ratio so the larger frames can be used to capture the original document at a much higher resolution.

  

  3M cartridge

  3M cartridges are essentially 16mm rollfilm in a square, hard plastic case or “cartridge.” Rather than a single square hole in the middle of a plastic spool, 3M cartridges have a metal hub with four smaller, round holes.  They often require a special adapter to allow them to be loaded onto a scanner, which not all vendors have (yes — Creekside Digital can work with 3M cartridges).  Other than their cartridge format, 3M cartridges are identical to normal 16mm film in terms of what types of documents they’re typically used to store.

  

  Jacket microfiche

• Microfiche.  Microfiche comes in several different types:
  Jacket microfiche are strips of 16mm or 35mm film stored in a clear Mylar jacket or envelope.  Jackets are typically 4″ x 6″ and can hold up to six strips of film, such that a matrix of images is created.  A header strip across the top of the envelope will usually have the fiche’s title and indicate its contents.
  

  COM microfiche

  COM microfiche is typically used for technical manuals, parts lists, operation handbooks, etc.  These are 4″ x 6″ sheets which have small individual frames arranged into a permanent grid pattern.  Each sheet will have a header which has a title and describes its contents.  COM fiche usually uses a higher reduction ratio (up to 50x or more) and a single sheet can hold up to 300 frames.

  

  Aperture card

• Aperture Cards are punch cards (Hollerith cards) which have a single 35mm microfilm frame mounted in them. Aperture cards are typically used in engineering applications. The actual punch card portion of an aperture card may also contain index information about its image.

The bottom line: Creekside Digital can convert all of the above types of microform into digital images that you can view, email, and print from any PC. Still have questions about what type of film you have? Contact us!

.

Optical Character Recognition, or OCR for short, is a type of software designed to extract text from images (for example, digitized images of your rollfilm) and output it to a file such as a PDF or text file. Creekside Digital often runs OCR on digitized rollfilm and creates searchable PDF files.  But will it work with YOUR film, and is it worth the extra cost?

While OCR works very well with typewritten and printed text, the technology is currently very limited in its ability to recognize hardwritten (cursive) script, particularly on older documents of dubious quality.  The one exception is with engineering drawings and architectural diagrams; we’ve found that quite often, the consistent “block” handwriting on such documents OCRs very well.  Other commonly OCR’d documents include newspapers and parts catalogs on microfiche.  Most commonly, Creekside will deliver PDF files with an invisible layer of text underneath the document image, which may be copied and pasted, searched, indexed, etc., just like any other office document.  We can also provide other formats such as text files and spreadsheets for custom applications — just ask us.  We scan all film to be OCR’d in grayscale at 300dpi if possible (as recommended by the publisher of the OCR software).

How about printed documents in other languages?  Sure.  Our server-based OCR engine recognizes a total of 184 languages, so newspapers and other printed documents in non-English languages are fair game.

How accurate is the OCR process?  That depends on the quality of your source microfilm.  If the document images are very clean and in a more modern font or typeface that’s very easy for the OCR engine to recognize, it’s not uncommon to see accuracy close to 100%.  The accuracy may drop depending on several factors such as film sharpness, reduction ratio and resolution of the scans, (the smaller the frames, the less image information there is to recognize), how “clean” the documents are (folds, lines, and shadows across letters can confuse the OCR engine), etc.  Having said that, Creekside Digital has successfully OCR’d microfilm of newspapers which are more than 150 years old, with excellent results.

So while 11th Century manuscripts and handwritten meeting minutes may not OCR very well, if you need to quickly find numbers, names, and other data within printed docs, optical character recognition is the way to go.  It’s as easy as checking a box on our Small Order Form.  (And if you need handwritten documents converted to electronic, searchable text, ask us about our data entry and manual indexing services).

.

Creekside Digital does things differently than most of the other microfilm digitization vendors out there: all of the rollfilm projects we quote are bid using flat per-roll pricing.  There are a few reasons we do this, which we’ll get into shortly, but the bottom line is that what’s good for our customers is good for our company.

Let’s start with a dirty little secret that the microfilm digitization industry DOESN’T want you to know: 50% of the actual work involved with scanning a roll of microfilm goes into (or at least, should go into) getting the optimal settings for that particular roll.  This means that we adjust a number of variables including lamp brightness, gamma correction, focus / image sharpness, etc., EVERY TIME a new roll of microfilm is loaded into the scanner.  Why?  Because every roll is a little bit different.  When your film was created thirty-five years ago, perhaps the camera operator was having a bad day, and after lunch he got a little sleepy and careless and overexposed a few rolls a little before (if) he found his mistake.  If we cut corners and use the same settings for every roll or sheet in your project, odds are that some images will be overexposed, some will be too dark, and others will be skipped or poorly cropped due to misfiring frame detection (more on that later).  We don’t do that, but a lot of other vendors out there do.

So prior to capture, we reset all of the critical settings for that particular roll of film.  We double-check the focus, the reduction ratio, and the lamp brightness and gamma settings.  We look at histograms to ensure that we’re maximzing the CCD camera’s available bandwidth.  The dirty little secret is that, with modern production scanners, it doesn’t matter whether or not you have 20 frames or 2,000 frames — once we have everything set up, we hit “go” and the scanner processes your roll of film in its entirity, quickly and automatically.  Due to the fact that a real person visually QAs everything after that initial capture process is complete (the other 50% of the work — or again, it should be), we’re able to capture many rolls quickly and correctly, GUARANTEE that you’re receiving everything that’s on your original source film, and charge you one competitive per-roll rate which covers all of the above processing.  Once we give you a flat, per-roll quote, it’s easy to budget for your project, as you know exactly how much it we’ll cost (and we’ll put it in writing).

Most of the other vendors out there base their pricing on a per-image model.  We can’t understand how anyone wouild be able to accurately budget their project unless they have an exceptionally good (and accurate) idea of how many images they had — which is rarely the case.  What if your sample rolls from the vendor (you are getting complete sample rolls from the vendors you’re evaluating, aren’t you?) scan at an average of 2,400 images / roll but most of your rolls end up actually having 3,200 images / roll?  To make matters worse, we hear of other fees associated with the per-image model such as “roll handling charges,” “per-DVD costs,” (if you don’t know how many images you have, how could you possibly know how many DVDs your project will consume?), and the dreaded “attended scanning rate,” which you must usually pay to guarantee that no images are skipped or improperly cropped.  An extra charge to ensure quality?  Come on.  And what about duplex film?  They’re not going to charge you for the front AND back of every document, are they?   It’s insane, and there’s no reason for it if the vendor uses modern production microfilm scanning equipment and does things RIGHT.

All of Creekside’s rollfilm orders — from single rolls submitted using our Small Order Form through large corporate and institutional projects consisting of thousands of rolls — are billed using flat-rate per-roll pricing.  Period.  No tricks or hidden fees.  At Creekside Digital, our focus is on giving you the highest quality scans available today at reasonable prices, not nickle-and-diming you to death for everything we can think of.   If your current microfilm scanning vendor doesn’t do this, you may want to ask them why.

.

This is Creekside Digital’s new blog.  If you’re reading this, you’ve probably stumbled upon it while looking for information on getting microfilm or microfiche converted into digital files.  You’ve come to the right place — that’s the only thing that we do here at Creekside, and we do it very well.

This blog will hopefully fill in some of the “human” element missing from so many of the digitization vendors’ websites out there right now, including our own.  We’ve converted a lot of microfilm to digital, and we have developed some significant insights into the process.  We’ll be sharing a lot of that knowledge right here, so stay tuned!

– Jim Studnicki, President of Creekside Digital

.