11.4 Data Storage Precision
Data storage of both geometry and attributes can also affect data accuracy whenever entries comprise many significant figures. Storage with floating point single precision, in which a single computer word (string of bits) is used to represent a number, results in precision to seven significant figures. Storage with double precision, in which double the usual number of bits are used to represent a number, results in precision of up to 15 significant figures. If data are transferred from a single-precision system to a double-precision system, digits after the sixth or seventh are unreliable. Single-precision systems may also introduce noticeable errors in rounding off when decimal numbers are repeatedly multiplied, as, for example, when computing nodes.
Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Read now
Unlock full access