EDM atmospheric and prism corrections

Corrects a measured distance for the air it traveled through (in ppm, entered or worked out from temperature, pressure, and humidity) and for the prism constant, each listed.

Experimental: not yet fully verified. How results are checked

Computing your own values needs JavaScript and WebAssembly. The worked example below was computed when this page was built, and the documentation under "How we got this" applies.

999.982m

The corrected distance is 999.982 m: 12 ppm for the air and -0.03 m for the prism.

Atmospheric correction
Atmospheric correction
Prism correction

Provenance

Computed by: survey.reduction.edm-correction 1.0.0, core 0.1.0
Model: N_G = 287.6155 + 4.88660 ÷ λ² + 0.06800 ÷ λ⁴ (λ in µm); N_L = (273.15 ÷ 1013.25) · N_G · p ÷ T − 11.27 · e ÷ T, with p and e in hPa and T in kelvins (IAG 1999 Resolution 3); ppm = N_ref − N_L; corrected = D + ppm · 10⁻⁶ · D + prism constant. Vapor pressure from humidity by the Magnus form
Accuracy: The closed formula is within 0.25 ppm of the full Ciddor and Hill procedure for ordinary survey conditions; use the instrument's own N_ref and wavelength. Instruments that apply ppm internally need no second correction
Notes: 1 shown with the answer
Cites: Landgate (Western Australian Land Information Authority) with ICSM, The Survey Instrumentation Calibration Technical User Manual

Something look off?

Your values

Showing an example. Change anything.

Measured distance As the instrument displayed it, like 1000.000 m · plain numbers mean m Prism constantoptional Added to the distance, like -30 mm · plain numbers mean mm Atmospheric correctionoptional In ppm if you have it, like 12; or give temperature and pressure Air temperatureoptional Like 28 °C · plain numbers mean °C Air pressureoptional Station pressure, like 980 hPa · plain numbers mean hPa

More options 3

Relative humidityoptional Percent, like 60; dry air if left out Carrier wavelengthoptional Micrometers, from the instrument manual, like 0.658 Reference refractivityoptional The instrument's N_ref in ppm, from its manual, like 286.34

Run many at once from a CSV

Loading…

How we got thisFormula, worked example, sources, and proof

Model: N_G = 287.6155 + 4.88660 ÷ λ² + 0.06800 ÷ λ⁴ (λ in µm); N_L = (273.15 ÷ 1013.25) · N_G · p ÷ T − 11.27 · e ÷ T, with p and e in hPa and T in kelvins (IAG 1999 Resolution 3); ppm = N_ref − N_L; corrected = D + ppm · 10⁻⁶ · D + prism constant. Vapor pressure from humidity by the Magnus form

Show your work

Atmospheric correction
ppm × 10⁻⁶ × measured distance
12 × 10⁻⁶ × 1,000 = 0.012 m
Corrected distance
$measured + atmospheric + prism$
1,000 + 0.012 + -0.03 = 999.982 m

The same steps an agent gets from the MCP server with explain: true.

Accuracy: The closed formula is within 0.25 ppm of the full Ciddor and Hill procedure for ordinary survey conditions; use the instrument's own N_ref and wavelength. Instruments that apply ppm internally need no second correction

Worked example: 1,000 m with +12 ppm and a -30 mm prism. Source: add-survey-suite ppm scenario: 1,000.000 + 0.012 − 0.030 = 999.982 m, each correction listed. It is golden test vector v001, and every build checks the tool still gives its answer within its tolerance.

You enter

Measured distance: 1000 m
Atmospheric correction: 12
Prism constant: -30 mm

You get

Corrected distance: 999.982 m
Atmospheric correction: 12
Atmospheric correction: 0.012 m
Prism correction: -0.03 m

Review: Not yet independently reviewed by a licensed surveyor.

Last verified: 2026-09-22, when a maintainer last confirmed this tool's sources at the issuer. See the sources ledger.

Status: version 1.0.0, core 0.1.0. See this tool in the verification report.

Checked against: 9 golden test vectors (download the test vectors, each with its source and tolerance). See how results are checked and every source.

Sources

The Survey Instrumentation Calibration Technical User Manual, Landgate (Western Australian Land Information Authority) with ICSM, Online edition, updated 24 November 2025. Section 4.2.3, equations 4.6 and 4.7 (the IAG 1999 Resolution 3 closed formula for group refractivity) and equation 4.1 (the first velocity correction).

Terms

EDM: electronic distance measurement: Measuring a distance by timing light or infrared bounced off a prism or a surface, as a total station does. The air slows the light a little, so the reading is corrected for temperature and pressure, in parts per million. Source: The Survey Instrumentation Calibration Technical User Manual

Experimental means this tool has not yet met the stable bar: at least 20 golden vectors, differential tests, and an independent worked example.

EDM — electronic distance measurement

Measuring a distance by timing light or infrared bounced off a prism or a surface, as a total station does. The air slows the light a little, so the reading is corrected for temperature and pressure, in parts per million.

Source: The Survey Instrumentation Calibration Technical User Manual