precision-CodePudding

Tags > precision

08-24Blockchainfloating-point error fligner.test r function?
08-17BlockchainConvert boost::multiprecision::cpp_dec_float_100 to string with precision
08-17BlockchainCan I naively check if a/b == c/d?
08-07EnterpriseSymmetric band matrix vector multiplication
08-06Back-endPrecision rounding problem with boost multiprecision
08-03NetC# double to float conversion issue
07-13Back-endMATLAB numeric precision when generating a numeric sequence
07-12Software designDouble vs Float vs _Float16 (Running Time)
06-25Back-endPython 3 decimal module devision still has floating point errors
06-24MobileWhy does Clang add extra FMA instructions?
06-18Software engineeringBigDecimal setScale ROUND_HALF_UP don't seem to operate on each number
06-15Back-endhow to enforce the evaluation type of an expression to be double
06-10OSNumpy matrix multiplication issue with 20 elements
06-06OSnumpy.vstack losing precision float16
05-26Enterprisepd.read_csv() keep number of decimals
05-18databaseDo we need epsilon value for lesser or greater comparison for float value?
05-14databaseHow to perform binomial-coefficient and factorial calculation with more precision?
05-08Software engineeringDo FMA instructions of different CPUs have different intermediate accuracy? If yes, then how does a
05-07OSIs it possible to calculate the value after overflow?
05-04Software designWhy is multiplication of numbers with floating point inside a loop isn't as precise as outside
04-30MobileStrange warning when calling sprintf with .* width specifier
04-26BlockchainWhat is the minimum number of significant decimal digits in a floating point literal to represent th
04-20Software designFor loop in which the counter goes up to a value generated by Math.floor()
04-19OSSubtract extremely small number from one in C
04-16EnterpriseIs it ok that with fp:fast 3000.f/1000.f != 3.f?
04-02BlockchainHow can I remove empty space in float from a number I get from web scraping? Error: could not conver
04-01NetLooking for suitable datatype in C to handle two mathematical expressions
03-20Blockchaindouble precision in C , 308 digits or 15 digits?
03-18otherHow to emulate Excel's automatic rounding?
03-18Software designDivision in group by SQL table
03-17NetConverting ctype float to python float properly
03-12MobileHow to perform Row wise sum based on column condition and add Class Wise specific value as Column?
03-03NetString number with limited precision fraction digits in Swift
01-03OSExplanation about Output in C
12-24MobileCheck whether Pandas DataFrame column contains duplicates within floating point precision
12-14OSSafely convert `float` to `double` without loss of precision
12-13Back-endPython: Large float arithmetic for El Gamal decryption
12-13NetWhy do we need std::numeric_limits::max_digits10?
12-12BlockchainFsolve precision issue (python)
12-12OS5456.367 -> 5456.367188. Is this because float precision in c?

64 1 2 Next Last