Calibration or Ground-Truthing Is Critical

Harrison Eiteljorg, II

(See email contacts page for the author's email address.)

"The maximum heart rate (HR _max) is the highest heart rate an individual can safely achieve through exercise stress, and depends on age." (Wikipedia, http://en.wikipedia.org/wiki/Heart_rate, accessed 3/9/11) The maximum rate is often casually defined as 220 minus the age of the individual. Thus, a heart rate during high-stress exercise of 198 would imply a 22-years-old person. I had such a heart rate on a recent workout in my local gym, and, for readers who do not know me, I am well over 22, not to mention 32, 42, 52, and 62. (A more complex and statistically-supported formula would suggest that my maximum heart rate should be 160. Other more modern calculations would suggest a similar number.) By all odds, I should have been about to have a heart attack. When I asked the person in charge of the equipment about the displayed heart rate on an earlier occasion — when it was "only" 188 — I was told that the equipment had been calibrated on installation (then only a few weeks prior) but that wearing a transmitter around my chest would provide more accurate results. Neither the attendant nor I worried about the high pulse rate because we both knew that the machine was simply not accurate enough to cause concern.

What has this to do with archaeology? Recently I have heard and read papers about the application of new technologies to archaeological research. In four cases the speakers/authors have uncritically accepted results from bleeding-edge equipment with no attempt to validate the information. (To be fair, at least two of those authors/speakers may have attempted validation and not reported it. The other two, when questioned, seemed to see no need.) It is all too easy to use new techniques that have arisen in other fields and apply them to our work without worrying about whether or not the techniques in question are appropriate for our use. Easy as it is, though, it seems extremely risky.

When a 3D scanner can provide data coordinates of points to the nearest .01 mm., according to the manufacturer, we cannot verify the results of a given scan. Similarly, when an aerial image shows a scale within it, we cannot easily measure objects on the ground to check the relationship between observed dimensions and dimensions calculated from the aerial image and its scale. Nor can we obtain for comparison the actual volume of every pot that is subjected to a modern CAD process to get its volume. In each of those cases, however, there are things that can be done to provide some confidence that the information supplied is accurate and precise within the range of our needs.

Let us take the examples just presented, in reverse order, to see what might be done to add confidence.

1. Pottery volume. If a specific computer procedure is to be used to calculate the volume of a transport amphora or some other pot, there is always the problem of obtaining a complete pot for experimental purposes. I would suggest, however, that it should not be difficult to obtain pots of various sizes — modern ones — and to subject them both to measurement for the computer procedure and to the simple exercise of filling them (with small plastic beads or, if the vessel is able to tolerate it, a liquid) and measuring the resulting volume by pouring the contents into measuring containers. This is not the proverbial rocket science, but it would provide some measure of confidence concerning the computer-based method — or render the method suspect. (In fact, Cemal Pulak, Vice President of the Institute for Nautical Archaeology and holder of the Frederick R. Mayer Fellowship in Nautical Archaeology at Texas A & M University, has carefully and self-consciously carried out such experiments with ancient transport amphorae, without needing modern examples. Professor Pulak found there to be significant differences between measured volumes and computer-calculated volumes. He believes that the differences may reflect irregularities in the pottery shapes or inaccurate drawings.¹)

2. Aerial images. Here again, checking accuracy may be impossible for any given photograph. If the image is of an area that cannot be reached — the obvious reason for relying on an aerial photo in the first place — how could one check it? The answer, of course, is that one could not check it. One can, however, check for accuracy in a more general sense. To do so one might simply use aerial photographs from the same purveyor — but of other places where checks can be made or where, for instance, excavation has been carried out such that measurements are already available for standing walls visible in the aerial photographs — and to make the checks in those places. Here again, the check would enable a scholar to have some confidence in the accuracy and precision of the photographs for the purposes at hand, providing accuracy and precision limits that would be very helpful.

3. 3D scans. A scanner whose manufacturer asserts accuracy to the nearest .01 mm. cannot readily be checked by any machinery we archaeologists have access to. On the other hand, it is highly unlikely that we will actually want accuracy of to that level of precision. What we will want to be sure of is probably much simpler — accuracy to the nearest mm. For such a measurement an easily-purchased caliper will do nicely. One need only scan a few simple objects and compare some dimensions from the resulting 3D models to the measurements of the same dimension with a caliper. Once again, confidence can be added to the results so that the user can rest assured that the necessary standards of accuracy and precision have been met.²)

It may seem odd to write about this issue, but it is so easy to accept claims about accuracy and precision from manufacturers that we are prone to doing so uncritically. Having done that, research proceeds and methodological limits are not simply pushed into the background; they are treated as non-existent. In the end, assertions are made that rest entirely on a house of cards that begins with a manufacturer's claims of accuracy and precision. Few of us would uncritically accept a claim of superiority for one manufacturer's product over another's. We would be naturally skeptical about such a claim. It seems to me that we should be equally skeptical about a manufacturer's claims of accuracy and precision, especially if it is possible to test them — and it usually is. In the cases cited here, I would expect there to be some problems serious enough to impact the use of the technologies in question for archaeological work. In these and other cases, my concern is the fact that serious scholarship may depend upon the data gathered with such technologies; the data, in turn, may depend upon devices tested only by their manufacturers/designers. Without some unbiased testing of the accuracy and precision of the basic data-gathering processes, we risk using data that may not be as accurate and precise as we believe — not accurate and precise enough for the analyses based upon them.

-- Harrison Eiteljorg, II

Notes:

1. Having heard Professor Pulak refer to these problems in a lecture, the author corresponded with him by email and received the information included here. Return to text.

2. A modern caliper can measure to the nearest .01 mm, but the difficulty of locating and measuring identical places on an object and a CAD model of it suggests that two identical measurements at such precision would, if found, be the result of good fortune more than good measurement. Return to text.