Andrey Semashev wrote:
WTF-8 and CESU-8 are not UTF-8 but different encodings. Dealing with them should be the user's explicit choice (e.g. the user should write utf16_to_wtf8 instead of utf16_to_utf8).
The user doesn't write such things in practice. He writes things like string fn = get_file_name(); fopen( fn.c_str() ); and get_file_name and fopen must decide how to encode/decode UTF-8. So get_file_name gets some wchar_t[] sequence from Windows, which happens to be invalid UTF-16. But Windows doesn't care for UTF-16 validity and if you pass this same sequence to it, it will be able to open the file. So your choice is whether you make this work, or make this fail. I choose to make it work. The functions would of course never produce invalid UTF-8 when passed a valid input (and will deterministically produce the least-invalid UTF-8 for a given input) but here again the definition of valid may change with time if, f.ex. more code points are added to Unicode beyond the current limit. You should also keep in mind that Unicode strings can have multiple representations even if using strict UTF-8. So one could argue that using strict UTF-8 provides a false sense of security.