Making pdf Documents

© Brooke Clarke 2006 - 2007

This came about because I wanted to make pdf versions of military and test instrument Technical Manuals that range in size from dozens to hundreds of pages.  So here are some of the things I've learned in the past few years.  There are three main steps to making a great manual.  Scanning, Post Processing and Acrobat processing.  The following is based on the idea that a CD-ROM or DVD will be used as the distribution medium, not an on line document.  When a document is to be on line the file size needs to be minimized both to reduce the storage requirement and to shorten the download time at the expense of quality.

The first, but for most also essentially the last, step is to scan the document. 

There are two kinds of scanners, manual and auto feed. I use a manual flat bed scanner and it has the advantage that when individual pages are scanned they are aligned and not rotated.  An auto feeder saves on labor, but creates a number of problems.  Since there needs to be some clearance between the edges of the paper and the feed slot, say it's 0.1 inches, then the paper can rotate by some small amount (0.1"/8.5" = 0.6 degrees).  0.6 degrees is a very noticeable amount of rotation and typically many pages have some rotation.  The feed rollers sometimes grab two sheets either skipping a sheet, making some combination of two sheets, or distorting a single sheet by smearing the letters.

Bleed through

When copying double sided sheets the back side image bleeds through and shows up in the scan of the front side.  The main reason for bleed through is that the scanner lid has a white lining.  This is about as bad as a mirror and reflects light back through the paper.  In my opinion the lid should be painted flat black, or what I do is tape a sheet of flat black paper to the lid.  Now light is only reflected by the front of the page knocking the bleed through down a lot.

Histogram

The four images below show what bleed through looks like on a histogram.  The upper left "Exposure Adjustment" window shows the classical bleed through peak on the right.  In the image below it you can see the word "INDEX" in the bleed through. 

The upper right "Exposure Adjustment" window shows the highlight cursor has been moved from 233 to 169.  169 was choosen because it's at the left toe of the bleed through curve.  The image below shows the output.

The only change made was to the highlight cursor.  Both images are the raw .bmp files directly from the HP 6200 scanner.

Normal	Bleed Through Cut

The next step in eliminating bleed through is to set the white threshold using the histogram.  On the HP 6200C flat bed scanner, when doing gray scale or color scans, you can adjust the black threshold, white threshold and the gamma (plus stuff with the 3 color channels).  The threshold controls are directly below the histogram and move cursors on the histogram.  So by placing the white (right hand side) cursor just to the left of the toe of the hump that's the bleed through you eliminate it completely.  Note this is a trade off since you are also cutting out some of the highlight detail in the image.

Note that if the page is Black and White (no gray) then by scanning it in gray scale and setting the scanner controls you can completely eliminate bleed though.  But it there is a photograph or other gray scale on the page eliminating bleed through and the quality of the image are a tradeoff and the histogram gets to be very important.

When scanning a bound book insert a sheet of black paper behind the page being scanned.

I tried the HP 8400 flat bed scanner and although they "show" a histogram, there's no way it can be used as described above since the controls were somewhere else and there were no cursors.  I turned it back and stuck with the 6200.

See Post Processing Bleed Through below for how to fix bleed through in an existing image file.

File Format

I expect that what most people do is scan in jpg or pdf format.  This is a mistake if you're going to do any post processing since these are lossy formats and degrade each time there's a new generation.  A non lossy format like Bit Map (.bmp) is a better choice to maintain high quality.  Bit map also includes the physical size of the image which is not the case with Tagged Image Format (TIF).

File Naming

When working with hundreds of pages there will be mistakes and you will need to rescan a page or two.  So it's a very good idea if your file naming scheme somehow will allow you match to the actual book.  Most of the TMs that I scan use a chapter-page system where the first page in chapter 4 is called 4-1. 

Also the file name should be such that the computer file manager will alphabetize them into the same order that they appear in the book.  Otherwise you will need to do a lot of manual work to get the pages in order.

The answer for me has been a file name like nnn-mmm.bmp.  Where nnn is the chapter number starting with 000 for stuff prior to chapter 1 and after the last chapter keep using the next number, so if appendix A comes after chapter 9 then it's 010-mmm.bmp.  Where mmm is the page number.  Note nnn and mmm are always say 3 digits so the front cover is 001-001, not 1-1.  This is needed to keep the file sort order correct.

A schematic might be 004-037L.bmp for the left side and 004-037R.bmp for the right side.  If there are more than two scans you can use A, B, C etc.  This way when making multiple scans of what is really one page number you don't use up page numbers that are needed for other actual pages.  For a huge book you may need to provide for a 4 digit page number like nnn-mmmm.

When scanning you don't need to manually enter all of the file name.  When the save file button is pressed the default file name is the last name stored and you can just place the cursor in front of the digit that needs to be changed and type: delete, the new digit, enter.

Dots Per Inch (DPI)

This has a lot to do with the source material.  If the source is line art or text made prior to laser printers then 300 DPI is very good.  But if there are schematic diagrams with very small print size (like a C size drawing that has been photo reduced by 4x) then 600 DPI is needed.  Photos are discussed separately.

Black & White, Gray scale or Color

If the source document has color then the scan should be in color.  When scanning very old books where the pages have yellowed sometimes using color will make the post processing easier.  For everything else I use gray scale.  Black and White has an advantage when you are trying for the smallest file size, but for me it's too much of a quality reduction.  Note that even though you are scanning a black and white document you need gray scale so that when half a pixel sees black and the other half sees white it can make a gray.  If B&W was used in this situation there would be a 1 pixel error either into the black or into the white.

Frame Size

For most documents you can set the frame size to just a little smaller than the page image then there will not be black borders.  When working with schematics it's good to expand the frame size to capture as much of the schematic as possible so that when stitching you will have more choice of where to place the seam.  But remember to put it back for text pages.

Scanning Blank Pages

Books are laid out so that new chapters always start on a right side (odd numbered) page.  This is a good thing to do for a physical book since it allows thumbing for new chapters, but has no advantage in an electronic only book.  If making a pdf where it's planned to print all of it then scanning blank pages will maintain the odd page on the right concept.

This is where a number of things get fixed and the advantages of an electronic manual start to show up.   This is done using a photo editing package like Photoshop.  These packages process image files and although they can do some text that's not their main use.

Bleed Through

Recently I received images of pages scanned by someone else that had noticeable bleed through on many pages.  But in Photoshop you can Image\Adjust\Levels and on the histogram move the right cursor to the left so that it's on the toe of the bleed through curve.  Also moving the left cursor to  the right makes the blacks blacker and the whole page look better.  This is much better than trying to use the "magic wand" to get rid of the bleed though.  This works so well because the bleed through is in the form of light gray images not the black images that are desired.

Portrait or Landscape

In a hard bound book all the pages must be in the same orientation, but that's not the case for a pdf document.  So if there's a diagram or table that's better viewed in landscape mode the page should be rotated into landscape format.  Note that if the document is later printed Adobe will automatically rotate it.

Angle Correction Rotation

If an auto feeder was used in any of the prior generations then there will be pages with rotations typically less than 2 degrees that will need to be rotated to within about 0.3 degrees of true.  0.7 degrees is very noticeable and anything over 1 degree is really noticeable.  If it's a schematic that's going to be stitched all the pages need to be the same rotation.  This means that you can stitch a couple of pates where they are both 0.6 degrees, but not if one is 0.0 and the other is 0.6.

Clean

The idea is erasing things that are not wanted.  Binder and staple holes are an example.  Older copy processes have the tendency to leave small black specks much like finely ground pepper.  Older books where the pages have yellowed have a grainy background.  If you have set the frame size too big or the page got rotated there may be black borders that need to be erased.  Some copy machines leave streaks, like there was a scratch on the drum.  The fold lines on a schematic are another thing that can be erased.  A properly made scan of a clean page may not need any cleaning.  An antique book may need an hour of cleaning for each page.

Sometimes rather than erasing to white you need to use a copy and paste method, like for eliminating the binder holes in a color cover sheet.

The image at the left comes from the 1928 K&E catalog.  Many hours of cleaning were required to get it looking like this.  The photo is a reduced resolution image, at full size it's even more impressive.  Note that like all the illustrations in the catalog this is hand drawn using K&E drafting supplies not a photo.

Stitch

Fold out pages need to be stitched together.  This makes it very easy to look at a schematic on the computer screen.  When a 2, 3 or 4 page fold out schematic is broken up into seperate pages it's almost impossible to work with it on a computer.  What's most commonly done is to get out the tape and scissors and make a hard copy.

Most schematics that were drawn prior to laser printers were done either by hand or a plotting machine, but in either case there was a pen or  pencil used that could not draw anything finer than about 0.3 mm.  This is a much wider line than a laser printer can draw.  So if you have a schematic that's on a B (2 x letter size) or C (4 x letter size) sheet you can stitch it together.  Do NOT shrink the page size, leave it as is since the  Adobe printing default will shrink the page to fit the printer.  This way the end user has the Adobe option to use tiling (cut and tape) to get a full size print.  There was a time when photo reductions were used typically to move a drawing to the next smaller sheet size.  For these you need to go up to 600 DPI when scanning to maintain the annotations.

When stitching you can place the stitch anywhere in the area where both images overlap.  Rather than just take all of the second image it's good to look for a place where there's a minimal amount of text that will cross the stitch.  It's common that there's a small scale and rotation difference between the two images that are being joined so even if you pick a good stitch line you may still need to determine where on the line the best match will occur and as you go farther away the match gets poorer.

Vector Graphics

Line art, like schematics, can be stored as either an image or vector file.  Photoshop, Paint, etc. are image processing programs.  Autocad and the old HP ME (Mechanical Engineering) are vector processing programs.  A vector based pdf "D" size drawing fits into a few hundred kilo bytes, but if the same file with the same resolution is converted to an image format it will be a few hundred Mega bytes, i.e. about 1,000 times larger.  This was made real to me when making the web page for the HP E1938 OCXO where the complete drawing package was very small as a vector pdf but huge when done as an image file.

I haven't found a free image to vector converter application.  If you know of one please let me know.

Adding Color

Often in schematics a box is drawn around part of the circuit to define some function.  Since these lines look very similar to the trace lines they add confusion.  But they can be manually erased and replaced by either a colored line or a gray line.  Greatly adding to the understanding of the circuit.

The trace lines can also be made much more understandable by doing things like making all the ground lines wider and solid black.  The Vcc lines can be made red and the signal lines some other color.  This goes even further in making the schematic easier to understand at a glance.

Some colors, like yellow, look good on the computer screen, but do not show up when a page is printed.  So when choosing colors be sure they have at least 15% of red, blue and green components, so there's some gray to print.  Better is to make a trial print to test each new color.

In Acrobat 7 there is a "make pdf from multiple files" option and a browse function.  So if you have the files named as described above it's just a few clicks and you will have a single document that combines all the pages.  I put this step in the intro to the Acrobat processing sections because it's just he beginning, not the end of what's needed.

Navigation

An electronic document is different from a physical document and how you find what you want is different.  You can not "thumb" an electronic document like you can a book.  But a book does not have the instant access that you get with an electronic document.  When pages are numbered using the chapter-page method there's no way to correlate that with the pdf document page number.

Bookmarks

What's wrong with most Bookmarks

Good Bookmarks

Links

OCR

Page Numbers

An area where a electronic document is very different from a printed one is the case of photographs.  A pdf document allows the user to change the size of the displayed image.  I find this very useful since my reading vision is not as good as it once was, but it's fantastic to be able to zoom in on a high resolution color photo to the point that you are seeing macroscopic detail.

Taking a high resolution color photo takes some skill and is the subject of many books and college level classes.  When using a digital camera if at all possible use the raw file format (the one that makes the largest file size).  Note that a color scanner can make a 30 Mega byte file at only 300 DPI and HUGE files at higher DPI values like 600 or 1200.  These provide macroscopic views or even microscopic views when enlarged.  You can see way more in one of these images than you can with a magnifying glass.

I frequently see things in my photos that I did not see with my eyes.

Making a high resolution color photo into a pdf does not result in much file size reduction and may even make it larger, I feel it's the right thing to do.