这让我疯了。
我花了太多时间在上面。
任何人都可以告诉我为什么在转换数字totext时会得到.562000000000000001而不是.562（参见随附的VEE Pro 7.0程序）？
另外，你有任何想法如何解决这个问题吗？问候，LeeLee Hudson，R＆amp; D科学家**************当然。
你得到的是.562000000000000001，这是IEEE标准要求的精确答案。
由于这是一台数字计算机，它只具有如此高的精度，因此只能近似数字。
也就是说，由于您只有64位可用于IEEE 64位浮点，因此只能表示可以表示的实数的固定子集。
其他一切都是近似的 - 因此是你所看到的东西的起源。许多环境都选择隐藏它。
VEE（无论好坏）决定暴露它。如果你想要更低的精度那么它只是数字fomatting的情况，并且最好的方法之一是使用To String对象，你可以指定任何格式/
你喜欢的精确度.Stan ---------------------------------------------
----------------------------- Stan Bischof安捷伦科技公司707-577-3994 stan_bischof@agilent.com-------
--------------------------------------------------
--------------------您目前订阅了vrf：r***@soco.agilent.com要订阅，请发送一封空白电子邮件至“join-vrf@it.lists。
it.agilent.com“。要取消订阅，请发送一封空白电子邮件至”leave-vrf@it.lists.it.agilent.com“。要发送邮件到此邮件列表，请发送电子邮件至”vrf@agilent.com“。
如果您需要有关邮件列表的帮助，请发送邮件至“owner-vrf@it.lists.it.agilent.com”。
以上来自于谷歌翻译
以下为原文
This is driving me nuts. I've spent way too much time on it. Can anyone tell
me why I get .562000000000000001 instead of .562 when I convert the number to
text (see attached VEE Pro 7.0 program)? Also, do you have any idea how to fix
this problem?
Regards,
Lee
Lee Hudson, R&D Scientist
**************
Sure.
You get .562000000000000001 becuase that's the precise answer that
IEEE standards call for. Since this is a digital computer it
only has so much precision and can therefore only approximate
most numbers. That is, since you only have 64 bits to work with
for IEEE 64-bit floating point there's only a fixed subset of
the real numbers that can be represented. Everything else is
approximate- hence the origins of what you are seeing.
Many environments choose to hide this. VEE ( for better or worse)
decided to expose it.
If you want less precision then it is just a case of number fomatting,
and one of the best ways to do that is to use a To String object
so that you can specifiy whatever format/precision you like.
Stan
--------------------------------------------------------------------------
Stan Bischof Agilent Technologies 707-577-3994
stan_bischof@agilent.com
--------------------------------------------------------------------------
---
You are currently subscribed to vrf as: [email=r***@soco.agilent.com]r***@soco.agilent.com[/email]
To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".
To send messages to this mailing list, email "vrf@agilent.com".
If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".
这让我疯了。
我花了太多时间在上面。
任何人都可以告诉我为什么在转换数字totext时会得到.562000000000000001而不是.562（参见随附的VEE Pro 7.0程序）？
另外，你有任何想法如何解决这个问题吗？问候，LeeLee Hudson，R＆amp; D科学家**************当然。
你得到的是.562000000000000001，这是IEEE标准要求的精确答案。
由于这是一台数字计算机，它只具有如此高的精度，因此只能近似数字。
也就是说，由于您只有64位可用于IEEE 64位浮点，因此只能表示可以表示的实数的固定子集。
其他一切都是近似的 - 因此是你所看到的东西的起源。许多环境都选择隐藏它。
VEE（无论好坏）决定暴露它。如果你想要更低的精度那么它只是数字fomatting的情况，并且最好的方法之一是使用To String对象，你可以指定任何格式/
你喜欢的精确度.Stan ---------------------------------------------
----------------------------- Stan Bischof安捷伦科技公司707-577-3994 stan_bischof@agilent.com-------
--------------------------------------------------
--------------------您目前订阅了vrf：r***@soco.agilent.com要订阅，请发送一封空白电子邮件至“join-vrf@it.lists。
it.agilent.com“。要取消订阅，请发送一封空白电子邮件至”leave-vrf@it.lists.it.agilent.com“。要发送邮件到此邮件列表，请发送电子邮件至”vrf@agilent.com“。
如果您需要有关邮件列表的帮助，请发送邮件至“owner-vrf@it.lists.it.agilent.com”。
以上来自于谷歌翻译
以下为原文
This is driving me nuts. I've spent way too much time on it. Can anyone tell
me why I get .562000000000000001 instead of .562 when I convert the number to
text (see attached VEE Pro 7.0 program)? Also, do you have any idea how to fix
this problem?
Regards,
Lee
Lee Hudson, R&D Scientist
**************
Sure.
You get .562000000000000001 becuase that's the precise answer that
IEEE standards call for. Since this is a digital computer it
only has so much precision and can therefore only approximate
most numbers. That is, since you only have 64 bits to work with
for IEEE 64-bit floating point there's only a fixed subset of
the real numbers that can be represented. Everything else is
approximate- hence the origins of what you are seeing.
Many environments choose to hide this. VEE ( for better or worse)
decided to expose it.
If you want less precision then it is just a case of number fomatting,
and one of the best ways to do that is to use a To String object
so that you can specifiy whatever format/precision you like.
Stan
--------------------------------------------------------------------------
Stan Bischof Agilent Technologies 707-577-3994
stan_bischof@agilent.com
--------------------------------------------------------------------------
---
You are currently subscribed to vrf as: [email=r***@soco.agilent.com]r***@soco.agilent.com[/email]
To subscribe send a blank email to "join-vrf@it.lists.it.agilent.com".
To unsubscribe send a blank email to "leave-vrf@it.lists.it.agilent.com".
To send messages to this mailing list, email "vrf@agilent.com".
If you need help with the mailing list send a message to "owner-vrf@it.lists.it.agilent.com".
举报
