Questions tagged [significant-digits]

Significant digits are a way of describing the precision of measurements in a scale-independent way.

Typically if a number is in normal scientific notation, the number of significant digits is the length (number of digits) of the mantissa or significand. The significant figures of a number are those digits that carry meaning contributing to its precision. This includes all digits except: All leading zeros, Trailing zeros when they are merely placeholders to indicate the scale of the number (exact rules are explained at Identifying significant figures), and spurious digits introduced, for example, by calculations carried out to greater precision than that of the original data, or measurements reported to a greater precision than the equipment supports.

Numbers are often rounded to avoid reporting insignificant figures. For instance, if a device measures to the nearest gram and gives a reading of 12.345 kg, it would create false precision to express this measurement as 12.34500 kg. Significant Figures - Wikipedia

178 questions
1
vote
4 answers

Remove digits representing irrelevant precision from a number with JavaScript

I'm looking for a function in Javascript (or JQuery) to remove digits representing irrelevant precision from a number. I'm calculating an estimated probability, and it could be a range of values - and the exact decimal isn't very relevant - just the…
Ben in CA
  • 688
  • 8
  • 22
1
vote
1 answer

Why does my data visual in ggplot2 display 4 significant figures when I specify 3?

I'm trying to create a bar chart the displays the unemployment rate for people who majored in the social sciences for 4 different age groups. In my visual, I want to display the unemployment rate on each bar, but I'm having trouble getting the…
1
vote
1 answer

R: Getting dplyr & summary to show more than 3 digits

This fails. I need 3 digits to the right of the decimal point. require("dplyr") options(digits = 10) group_by(MyData, Type_1, Type_2) %>% summarise( count = n(), mean = mean(Rate, na.rm = TRUE), sd = sd(Rate, na.rm = TRUE) ) # A tibble: 4…
user13248694
1
vote
3 answers

Storing individual bits in memory

So I want to store random bits of length 1 to 8 (a BYTE) in memory. I know that computer aren't efficient enough to store individual bits in memory and that we must store at least a BYTE data on most modern machines. I have been doing some research…
lag
  • 520
  • 2
  • 10
  • 26
1
vote
2 answers

SQL Server significant digits

I have a variable that I want to return a max of 5 to the left of the decimal and always 3 to the right CONVERT(VARCHAR(14),CONVERT(DECIMAL(8,3), DATEDIFF(MILLISECOND,@ProcessTime,GETDATE()))/@OneThousand) Now variables are defined as…
Mike
  • 5,918
  • 9
  • 57
  • 94
1
vote
1 answer

Choose the power of 10 to display in R, for formatting error in scientific notation

I'm writing a program in R in order to estimate the value of a physical constant. At the end of the program, after I get the result I wanted I'd like to put it in a plot in the format "value +- error". In order to make that more readable I want the…
1
vote
0 answers

OpenCV Python -> C++ : decimal digits precision and divergence

I am currently trying to translate a Python image detection algorithm in C++ (in order to use it in a smartphone app) using OpenCV. I have similar results to a certain point, where the two algorithms seem to diverge due to the accumulation of small…
1
vote
2 answers

Lua significant figures

I'm trying to make a function that rounds a number up to a certain number of significant figures given by the user, for example if the user gives me the number 234.235534 with 5 significant numbers, the function should return 234.24
1
vote
1 answer

Printing significant figures regardless of decimal point

The table below (mean ± se) contains true data that will be presented in a publication. As you can see, the data has a very wide range. Using R, I rounded all numbers to 3 significant figures, but I want to include trailing zeros as well as zeros to…
J.Con
  • 4,101
  • 4
  • 36
  • 64
1
vote
2 answers

Round up values to a specific significant figure in R

What would be the simplest way to round up a value to a specific significant figure? Something like the function signif(), but only perform rounding up (not down) For example: Round up 0.001145288 to 1 significant figure would yield 0.002 Any…
Tara Sutjarittham
  • 366
  • 1
  • 6
  • 18
1
vote
1 answer

R : round p values to two significant digits in xtable and ReporteRs export to Latex/Office/LibreOffice

I am using a combination of xtable, rtable and ReporteRs to export summary tables of various objects (aov, glm's, lm's etc) to Latex/Word/Powerpoint. I'm not so fond of the default way, however, in which p values are given with a fixed number of…
Tom Wenseleers
  • 7,535
  • 7
  • 63
  • 103
1
vote
1 answer

writing matrix columns with different precisions(significant digits) in matlab

I have a matrix m=[10 15.675; 13.5 34.987; 20 55.5]; I want to write this matrin on o txt file. I want to use different precisions for each columns. For example 1 for 1st column and 2 for 2nd column. Desired output: 10.0 15.68 13.5 …
1
vote
0 answers

R - RStudio hangs/crashes creating double-precision large numeric vector

I have a numeric vector of 500 values that I'm trying to load into R to do some graphical calculations and displays, and every time I try my client hangs/crashes (RStudio). The system doesn't immediately crash, but hangs indefinitely with a plus…
Alium Britt
  • 1,246
  • 4
  • 13
  • 25
1
vote
1 answer

Floating point error in MATLAB - linearly spaced vectors

OS: Win 7 64bit. Matlab: 2014a, 2015a When I create a vector as follows: a = 0.2:0.2:1 I get: a = 0.2000 0.4000 0.6000 0.8000 1.0000 which is expected. Now when I want to see whether 0.6 exists in my vector I type: a == 0.6 and I…
ilyas
  • 609
  • 9
  • 25
1
vote
0 answers

Data table column conversion from chr to numeric adds decimals?

I'm working with a data table which has a column of results which begin as character values (chr). Because I need to do mathematical calculations with some of these values I copy the original column, but change the values which are non-numerics to…
NWdev
  • 483
  • 1
  • 6
  • 19