geoprimsField-grade geospatial math

EDM atmospheric and prism corrections

Corrects a measured distance for the air it traveled through, using entered ppm, custom instrument constants, or a named Trimble Access formula, and lists the atmospheric and prism corrections separately.

Experimental: not yet fully verified. How results are checked

999.982m

The corrected distance is 999.982 m: 12 ppm for the air and -0.03 m for the prism.

Atmospheric correction
Atmospheric correction
Prism correction
Provenance
Computed by
survey.reduction.edm-correction 1.1.0, core 0.1.0
Model
Custom IAG 1999: N_G = 287.6155 + 4.88660 ÷ λ² + 0.06800 ÷ λ⁴; N_L = (273.15 ÷ 1013.25) · N_G · p ÷ T − 11.27 · e ÷ T; ppm = N_ref − N_L. Trimble Access presets: ppm = J − N·p ÷ (273.16 + t), using the guide's J and N for the selected instrument family. Corrected = D + ppm · 10⁻⁶ · D + prism constant
Accuracy
The custom IAG 1999 formula is within 0.25 ppm of the full Ciddor and Hill procedure for ordinary survey conditions. The named presets reproduce Trimble Access's published pressure-and-temperature equation and constants. Instruments that apply ppm internally need no second correction
Notes
1 shown with the answer
Cites
Landgate (Western Australian Land Information Authority) with ICSM, The Survey Instrumentation Calibration Technical User Manual; Trimble Inc., Trimble Access Calculations Reference Guide

Something look off?

Your values

Showing an example. Change anything.
More options 4
Run many at once from a CSV

Loading…

How we got thisFormula, worked example, sources, and proof

Model: Custom IAG 1999: N_G = 287.6155 + 4.88660 ÷ λ² + 0.06800 ÷ λ⁴; N_L = (273.15 ÷ 1013.25) · N_G · p ÷ T − 11.27 · e ÷ T; ppm = N_ref − N_L. Trimble Access presets: ppm = J − N·p ÷ (273.16 + t), using the guide's J and N for the selected instrument family. Corrected = D + ppm · 10⁻⁶ · D + prism constant

Show your work

  1. Atmospheric correction

    ppm × 10⁻⁶ × measured distance

    12 × 10⁻⁶ × 1,000 = 0.012 m

  2. Corrected distance

    measured+atmospheric+prism

    1,000 + 0.012 + -0.03 = 999.982 m

The same steps an agent gets from the MCP server with explain: true.

Accuracy: The custom IAG 1999 formula is within 0.25 ppm of the full Ciddor and Hill procedure for ordinary survey conditions. The named presets reproduce Trimble Access's published pressure-and-temperature equation and constants. Instruments that apply ppm internally need no second correction

When to use this: Use this when you need to correct an EDM distance for the air and the prism, such as when the ppm was not set in the instrument. Enter the measured distance, the prism constant, and either the ppm or temperature and pressure. For the latter, choose an exact Trimble Access instrument-family preset or give the instrument's wavelength, reference refractivity, and optional humidity. It returns the corrected distance with each correction listed.

Limitations: If the instrument already applied ppm, do not apply it again. Use a named preset only for that exact instrument family; current Trimble VX/S/M instruments supply their own constants and are not preset here. The Trimble Access formula does not use humidity. Temperature and pressure are taken at one spot, while the air along the line may differ. The tool does not reduce slope to horizontal or apply elevation or grid scale factors; those come next.

Worked example: 1,000 m with +12 ppm and a -30 mm prism. Source: add-survey-suite ppm scenario: 1,000.000 + 0.012 − 0.030 = 999.982 m, each correction listed. It is golden test vector v001, and every build checks the tool still gives its answer within its tolerance.

You enter

Measured distance
1000 m
Atmospheric correction
12
Prism constant
-30 mm

You get

Corrected distance
999.982 m
Atmospheric correction
12
Atmospheric correction
0.012 m
Prism correction
-0.03 m

Review: Not yet independently reviewed by a licensed surveyor.

Last verified: 2026-09-22, when a maintainer last confirmed this tool's sources at the issuer. See the sources ledger.

Status: version 1.1.0, core 0.1.0. See this tool in the verification report.

Changes

Checked against: 25 golden test vectors (download the test vectors, each with its source and tolerance). See how results are checked and every source.

Constants this tool assumes

Values you do not enter, and where each comes from.

Group refractivity constant term (IAG 1999)
287.6155 — The Survey Instrumentation Calibration Technical User Manual
Group refractivity coefficient on 1/wavelength squared
4.88660 um2 — The Survey Instrumentation Calibration Technical User Manual
Group refractivity coefficient on 1/wavelength to the fourth
0.06800 um4 — The Survey Instrumentation Calibration Technical User Manual
Reference pressure for the group refractivity
1013.25 hPa — The Survey Instrumentation Calibration Technical User Manual
Water vapor refractivity coefficient
11.27 K/hPa — The Survey Instrumentation Calibration Technical User Manual
Magnus coefficient a (saturation vapor pressure)
17.625 — Alduchov and Eskridge (1996) Magnus form
Magnus coefficient b
243.04 degC — Alduchov and Eskridge (1996) Magnus form
Magnus coefficient c
6.1094 hPa — Alduchov and Eskridge (1996) Magnus form
Trimble Access 5600 J constant
274.41 — Trimble Access Calculations Reference Guide
Trimble Access 5600 N constant
79.39 K/hPa — Trimble Access Calculations Reference Guide
Trimble Access 3300/3600 J constant
278.77 — Trimble Access Calculations Reference Guide
Trimble Access 3300/3600 N constant
80.653 K/hPa — Trimble Access Calculations Reference Guide
Trimble Access TTS300/500 J constant
270.0 — Trimble Access Calculations Reference Guide
Trimble Access TTS300/500 N constant
79.167 K/hPa — Trimble Access Calculations Reference Guide

Sources

Terms

EDM: electronic distance measurement
Measuring a distance by timing light or infrared bounced off a prism or a surface, as a total station does. The air slows the light a little, so the reading is corrected for temperature and pressure, in parts per million. Source: The Survey Instrumentation Calibration Technical User Manual

Experimental means this tool has not yet met the stable bar: at least 20 golden vectors, differential tests, and an independent worked example.

EDM — electronic distance measurement

Measuring a distance by timing light or infrared bounced off a prism or a surface, as a total station does. The air slows the light a little, so the reading is corrected for temperature and pressure, in parts per million.

Source: The Survey Instrumentation Calibration Technical User Manual