Refraction Error Correction in Sextant Measurements

Quote of the Day

We must teach our children to dream with their eyes open.

— Harry Edwards


Introduction

Figure 1: Illustration of Light Refraction Through the Atmosphere.

Figure 1: Illustration of Light Refraction Through the Atmosphere (source).

Refraction is probably the most difficult to understand of all the altitude observation corrections. It is also the most difficult to estimate accurately because it depends so strongly on atmospheric conditions, particularly the rate of temperature variation with altitude (see lapse rate). I will derive in this post a commonly used expression for the refraction correction required for a celestial object with an altitude greater than or equal to 15°. The accuracy of this expression degrades significantly for objects below 15°.

Formulas exist for refraction correction on objects with altitudes lower than 15°, but they are complex and are subject to substantial uncertainties. Since my goal here is provide some insight into how refraction is modeled, I will keep things simple and assume that we are working with objects relatively high in the sky.

Background

Definitions

refraction
Refraction is the change in direction of propagation of a wave due to a change in its transmission medium (source).
index of refraction
The index of refraction is a parameter that relates the speed of light in the material to the speed of light in a vacuum, i.e. v=\frac{c}{n}, where c is the speed of light in a vacuum, n is the index of refraction, and v is the speed light in the material with index n. I discuss how the index of refraction varies with air's density in Appendix A.
angle of incidence (θ1 in Figure 2)
The angle of incidence is the angle between a ray incident on a surface and the line perpendicular to the surface at the point of incidence, called the normal (source).
angle of refraction (θ2 in Figure 2)
The angle formed by a refracted ray or wave and a line perpendicular to the refracting surface at the point of refraction (source).

Index of Refraction

Model

Figure 2 shows how we usually model refraction at the interface between two materials through which the light passes.

Figure M: Refraction Model (Wikipedia).

Figure 2: Refraction Model (Wikipedia).

Refraction effects are zero for celestial objects at our zenith and a maximum for objects on the horizon. Refraction causes celestial objects to have an apparent altitude that is higher than their true altitude. Thus, refraction corrections always involve subtracting a correction term from a measured altitude.

Snell's Law

Snell's law relates the angle of incidence to the angle of refraction, which is usually expressed as shown in Equation 1.

Eq. 1 \displaystyle \frac{{\sin {{\theta }_{1}}}}{{\sin {{\theta }_{2}}}}=\frac{{{{v}_{1}}}}{{{{v}_{2}}}}=\frac{{{{n}_{2}}}}{{{{n}_{1}}}}

where

  • θ1 is the light's angle of incidence.
  • θ2 is the light's angle of refraction.
  • v1 is the speed of light in material 1.
  • v2 is the speed of light in material 2.
  • n1 is the index of refraction of in material 1.
  • n2 is the index of refraction of in material 2.

Atmospheric Assumption

For celestial bodies with altitudes greater than 15°, we can model refraction with sufficient accuracy for navigational purposes by assuming that the atmosphere is series of flat layers with different densities. This assumption is not true for celestial objects with altitudes lower than 15° because the light from these objects will pass through atmospheric layers that are significantly curved.

To keep this discussion simple, I will only model refraction for objects with altitudes greater than 15°.

Analysis

Key Result

Eq. 2 \displaystyle R =\left( {{{n}_{{Air}}}-1} \right)\cdot \cot \left( \theta \right)=0.974\text{ arcmin}\cdot \text{cot}\left( \theta \right)=58\text{ arcsec}\cdot\text{cot}\left( \theta \right)

where

  • nAir is the index of refraction at the Earth's surface with a pressure of 760 mmHg and a temperature of 10 °C. It is usually assumed to have a mean value of 1.0002835. The reference temperature and pressure varies with the person doing the modeling. I often see 1010 millibars and 0 °C used as well.
  • θ is the altitude of the celestial object.
  • R is the angular measure of the amount of refraction.

Derivation

For a very detailed version of this discussion, see the excellent work in  "Textbook on Spherical Astronomy" by Smart, which is my favorite text on the subject of astronomical measurements. Fortunately, this section of the text on refraction is available in its Google Books preview on page 58 (at least it was when I looked).

We begin deriving Equation 2 by assuming the atmosphere is composed of layers of air with different refractive indexes as shown in Figure 3. We can relate the angle of incidence in space (z) to the angle of incidence on the ground (ζ) using the argument shown in Figure 4. The idea here is is to look at the angles of incidence and refraction for each layer. As is shown in the Figure 4, all the intermediate indexes of refraction can be eliminated, leaving only the index of refraction of space (n = 1) and at the observer. The angles of incidence and refraction will be converted to altitudes in Figure 5.

Figure 4: Key Relationship Between True and Observed Altitude.

Figure 4: Key Relationship Between True and Observed Altitude.

Given the relationship (yellow highlight) shown in Figure 4, we can now derive a commonly seen approximation for the refraction correction as a function of altitude (Figure 5).

Figure 5: Derivation of Equation 2.

Figure 5: Derivation of Equation 2.

Figure 5 shows how to derive Equation 2 and presents it in the two forms that it is usually seen – one for arcminutes and the other for arcseconds. As an example of the use of these expressions, I am including a link to a surveying text that uses the arcsecond form.

Conclusion

Equation 2 is a commonly seen expression for the effect of refraction on celestial objects with relatively high altitudes (>15°).  We can correct a sextant reading for refraction by subtracting the value generated by Equation 2 from the altitude reading we obtained from our sextant.

My next post will discuss index error, which is an error that is much simpler to describe than refraction.

Appendix A: Atmospheric Refraction Index

We need to have a model of the air's index of refraction in order to create an effective model refraction on sextant measurements. I will model the air's index of refraction using Equation 3.

Eq. 3 \displaystyle {{n}_{{Air}}}=1+2.9\cdot {{10}^{{-4}}}\cdot \frac{P}{{{{P}_{0}}}}\cdot \frac{{{{T}_{0}}}}{T}

where

  • P0 at the reference pressure, which is assumed to be 760 mmHg.
  • T0 is the reference temperature of T0 = 0°C. I see different references temperatures used.
  • 2.9E-4 is the index of refraction air at a pressure of 760 mmHg and temperature of 0 °C.

I show my derivation for Equation 3 in Figure 6.

Figure M: Air Index Linear Model Variation with Pressure and Temperature.

Figure 6: Air Index Linear Model Variation with Pressure and Temperature.

I can graph Equation 3 and compare it other models for air's index of refraction as shown in Figure 7. The agreement is good.

Figure M: Comparison of the Linear Model to Model Found on Web.

Figure 7: Comparison of the Linear Model to Model Found on Web.

Web index reference
 
This entry was posted in Astronomy, Naval History, Navigation. Bookmark the permalink.

One Response to Refraction Error Correction in Sextant Measurements

  1. Pingback: The Earth is flat? It's a circular flat disk surrounded by a wall of ice? - Page 4 - Christian Chat Rooms & Forums

Leave a Reply

Your email address will not be published. Required fields are marked *