std::ostream fails to write UTF-8 encoded string to console - by mkluwe

Status : 

 


6
0
Sign in
to vote
ID 431244 Comments
Status Active Workarounds
Type Bug Repros 4
Opened 4/7/2009 11:15:30 PM
Access Restriction Public

Description

In my sample program, I expect

    std::cout << "\xc3\xbc";

to output character 'ü' in the console after I've done

    SetConsoleOutputCP( CP_UTF8 );

especially because

    fputs( "\xc3\xbc", stdout );

works as expected (and fputs( "ü", stdout ) as well, choosing to save the source file UTF-8 encoded).

After using std::cout as above, std::cout.fail() is true.

I encountered this problem first trying to read UTF-8 encoded strings from the console, which I tried to discuss in http://groups.google.de/group/microsoft.public.win32.programmer.international/browse_thread/thread/e38aa923a32c20f9.
Sign in to post a comment.
Posted by eraxillan on 5/11/2014 at 12:31 AM
std::cout still can't output UTF-8.
However, WriteConsoleA can. Consider this program - it correctly output UTF-8 under my Windows 8 x64/Visual C++ 2013 (using Lucida Console font in console):
#pragma execution_character_set("utf-8")
#include <windows.h>

int main ()
{
    if (!SetConsoleCP (CP_UTF8))
        std::cout << "Could not set console input code page to UTF-8" << std::endl;
    if (!SetConsoleOutputCP (CP_UTF8))
        std::cout << "Could not set console output code page to UTF-8" << std::endl;

    static char const raw_utf8_text[] = "Blåbærsyltetøy! кошка!";
    DWORD raw_written = 0;
    std::cout << "UTF-8 output using Windows API WriteConsoleA:" << std::endl;
    WriteConsoleA (GetStdHandle (STD_OUTPUT_HANDLE), raw_utf8_text, std::strlen (raw_utf8_text), &raw_written, NULL); // correct output
    std::cout << std::endl;
    std::cout << "UTF-8 output using STL std::cout stream: " << std::endl << raw_utf8_text << std::endl; // FIXME: garbage!

    cin.get (); // for pausing the program
}
Posted by mkluwe on 1/5/2010 at 11:37 AM
Thank you for suggesting the "conversion workauround".

IMHO a much simpler solution would be doing all output using fputs and alike, as I mentioned in my original report from April last year...
Posted by Stephan [MSFT] on 1/4/2010 at 2:37 PM
> The basic question remains: How do I do it?

I suggest converting UTF-8 to UTF-16 and printing that out with _O_U16TEXT and wcout.

Stephan T. Lavavej
Visual C++ Libraries Developer
Posted by mkluwe on 10/2/2009 at 8:04 AM
Dear Visual C++ Team,

thank you for the suggestion to "re-activate this ticket" if I am "still experiencing the issue".

Of course I am as no solution has been made available in the last half-year. I reactivated this item in august.

Regards,
M. Kluwe
Posted by Microsoft on 9/16/2009 at 2:00 PM
Hi mkluwe,

Thank you for taking the time to report this feedback. Please re-activate this ticket if you are still experiencing the issue.

Thank you,

Visual C++ Team
Posted by mkluwe on 8/12/2009 at 9:46 AM
The above said comes as a small surprise. Until now I have assumed that I had to tell the console about what character encoding it had to expect from my application (as it had no way of guessing it). I have thought the extensions of this concept to UTF-8 was quite natural.

Sad enough, nothing in

http://msdn.microsoft.com/en-us/library/dd317756(VS.85).aspx

and

http://msdn.microsoft.com/en-us/library/ms686036(VS.85).aspx

gave me a hint that this is not the case.

I looked into the mentioned blog and altered my example code. Now I have the following:

#include <fcntl.h>
#include <io.h>
#include <iostream>

int main() {
_setmode(_fileno(stdout), _O_U16TEXT);
std::wcout << L"\x00fc";
}

yields the wanted character 'ü'. Unfortunately, I'd like to handle UTF-8 encoded strings, thus I tried

#include <fcntl.h>
#include <io.h>
#include <iostream>

int main() {
_setmode(_fileno(stdout), _O_U8TEXT);
std::cout << "\xc3\xbc";
}

which leaves me with the following message (debug mode):

---------------------------
Microsoft Visual C++ Debug Library
---------------------------
Debug Assertion Failed!

Program: c:\tmp\unicode\Debug\unicode.exe
File: f:\dd\vctools\crt_bld\self_x86\crt\src\fputc.c
Line: 49

Expression: ( (_Stream->_flag & _IOSTRG) || ( fn = _fileno(_Stream), ( (_textmode_safe(fn) == __IOINFO_TM_ANSI) && !_tm_unicode_safe(fn))))

For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts.

(Press Retry to debug the application)

----

The basic question remains: How do I do it?

In addition: I don't think that closing this issue with a reference to a blog which says "content of Michael Kaplan's blog not approved by Microsoft!" is the proper way of customer support.

I will reopen this issue now.
Posted by Stephan [MSFT] on 7/29/2009 at 6:45 PM
Hi,

Thanks for reporting this issue. I've resolved it as By Design, because SetConsoleOutputCP() is not the correct way to do this. Michael Kaplan blogged about the correct way here: http://blogs.msdn.com/michkap/archive/2008/03/18/8306597.aspx

If you have any further questions, feel free to E-mail me at stl@microsoft.com .

Stephan T. Lavavej
Visual C++ Libraries Developer
Posted by mkluwe on 4/23/2009 at 10:25 PM
Screenshot provided, FWIW.

The following information is from Visual Studio's Help/About dialog:

Microsoft Visual Studio 2008
Version 9.0.30729.1 SP
Microsoft .NET Framework
Version 3.5 SP1

Installierte Edition: Professional

Microsoft Visual C++ 2008 91988-270-3422665-60435
Microsoft Visual C++ 2008

Microsoft Visual Web Developer 2008 91988-270-3422665-60435
Microsoft Visual Web Developer 2008

Crystal Reports    AAJ60-H0MSA4K-68000YE
Crystal Reports Basic für Visual Studio 2008


Hotfix für Microsoft Visual Studio 2008 Professional Edition - DEU (KB944899) KB944899
Dieser Hotfix ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieser Hotfix automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/944899.

Hotfix für Microsoft Visual Studio 2008 Professional Edition - DEU (KB945282) KB945282
Dieser Hotfix ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieser Hotfix automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/945282.

Hotfix für Microsoft Visual Studio 2008 Professional Edition - DEU (KB946040) KB946040
Dieser Hotfix ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieser Hotfix automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/946040.

Hotfix für Microsoft Visual Studio 2008 Professional Edition - DEU (KB946308) KB946308
Dieser Hotfix ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieser Hotfix automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/946308.

Hotfix für Microsoft Visual Studio 2008 Professional Edition - DEU (KB946344) KB946344
Dieser Hotfix ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieser Hotfix automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/946344.

Hotfix für Microsoft Visual Studio 2008 Professional Edition - DEU (KB946581) KB946581
Dieser Hotfix ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieser Hotfix automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/946581.

Hotfix für Microsoft Visual Studio 2008 Professional Edition - DEU (KB947171) KB947171
Dieser Hotfix ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieser Hotfix automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/947171.

Hotfix für Microsoft Visual Studio 2008 Professional Edition - DEU (KB947173) KB947173
Dieser Hotfix ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieser Hotfix automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/947173.

Hotfix für Microsoft Visual Studio 2008 Professional Edition - DEU (KB947180) KB947180
Dieser Hotfix ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieser Hotfix automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/947180.

Hotfix für Microsoft Visual Studio 2008 Professional Edition - DEU (KB947540) KB947540
Dieser Hotfix ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieser Hotfix automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/947540.

Hotfix für Microsoft Visual Studio 2008 Professional Edition - DEU (KB947789) KB947789
Dieser Hotfix ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieser Hotfix automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/947789.

Hotfix für Microsoft Visual Studio 2008 Professional Edition - DEU (KB948127) KB948127
Dieser Hotfix ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieser Hotfix automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/948127.

Microsoft Visual Studio 2008 Professional Edition - DEU Service Pack 1 (KB945140) KB945140
Dieses Service Pack ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieses Service Pack automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/945140.

Microsoft Visual Studio 2008 Professional Edition - DEU Service Pack 1 (KB947888) KB947888
Dieses Service Pack ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieses Service Pack automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/947888.

Microsoft Visual Studio 2008 Professional Edition - DEU Service Pack 1 (KB948484) KB948484
Dieses Service Pack ist für Microsoft Visual Studio 2008 Professional Edition - DEU.
Wenn Sie später ein neueres Service Pack installieren, wird dieses Service Pack automatisch deinstalliert.
Weitere Informationen erhalten Sie unter http://support.microsoft.com/kb/948484.
Posted by mkluwe on 4/8/2009 at 11:53 PM
Thanks for looking at the issue. You see two characters written to the console because it does not support the requested encoding in the current font setting.

The see the result I'm seeing here, switch the console font to "Lucida Console" (which is the only option I have here apart from a raster font). This font should support the "Basic Latin" Unicode range as seen in http://www.microsoft.com/typography/fonts/font.aspx?FMID=159.

I'm sorry that I did not include this additional information in my original message, but working here in Germany, _all_ my console windows are switched away from the raster font, obviously, so I did not think of that.
Posted by Microsoft on 4/8/2009 at 6:01 PM
Thanks for reporting the issue.
In order to fix the issue, we must first reproduce the issue in our labs. We are unable to reproduce the issue with the steps you provided. See attached snapshot_kylin.jpg. Could you please provide us with screenshot about this issue? And please provide us with detailed info about your Visual Studio.

Thank you,
Kylin.Ming