Maths error by AAB Software

Active

Type:	Bug
ID:	778545
Opened:	2/5/2013 3:16:11 AM
Access Restriction:	Public

Workaround(s)

view

User(s) can reproduce this bug

modular variable stores an integer value of 100000, multiple variable by itself (i.e square it). With compile set to debug mode and hex display turned off look at the value via the immediate window, local window or the displayed result you get 1410065408 and not 10000000000

Details (expand)

Visual Studio 2012

modular variable stores an integer value of 100000, multiple variable by itself (i.e square it). With compile set to debug mode and hex display turned off look at the value via the immediate window, local window or the displayed result you get 1410065408 and not 10000000000

            int test = 100000;
            double val = test * test;
            double val2 = Math.Pow(test, 2);
different results obtained

English

Windows Server 2012

English

1410065408

10000000000

0 attachments

Posted by Microsoft on 4/12/2013 at 3:40 PM

Hi. This is by design. The int operation is overflowing the arithmetic before you store to the double, while Pow is converting internally.

Bill

Posted by Microsoft on 2/5/2013 at 10:30 PM

Thank you for submitting feedback on Visual Studio and .NET Framework. Your issue has been routed to the appropriate VS development team for investigation. We will contact you if we require any additional information.

Posted by Microsoft on 2/5/2013 at 3:50 AM

Thank you for your feedback, we are currently reviewing the issue you have submitted. If this issue is urgent, please contact support directly(http://support.microsoft.com)

Microsoft