Github repository :https://github.com/github/VisualStudio

What Will I Learn?

• Learn the basic knowledge of mbcs and ANSI.
• Learn how to set local info temporarily.
• Solutions to a problems of garbled when reading ANSI zh-cn files.

Requirements

• The basic knowledge of C++.
• Basic C++ knowledge
• Basic knowledge of mbcs and ANSI.
• Basic knowledge of reading data from file.

Difficulty

• Intermediate

Tutorial Contents

issue raised

Some time ago, I was doing Github development based on Visual Studio 2015 . In this process, I have the need to read some TXT files or XML files from the database. But I found that the data is all garbled when I use the readstring function of CStdioFile to read a TXT file  contain chinese string, which was encoded by ANSI.

My Computer default language is set to English.

The TXT file shown in the following illustration contains some Chinese strings, and the encoded format of the file is ANSI.

Under these conditions, the data read from file will all displayed as garbled if without any special treatment, as shown in the following figure:

Investigate the reason

Reason: I have been investigating this problem for some time before I realized that it was the ReadString function of cstdiofile does not support Chinese files encoded in ANSI format.

1.ANSI and MBCS

MBCS(Multi-Byte Chactacter System)

It is a type of encoding, not a name for a particular encoding.

In order to expand the ASCII encoding to be used to display the native language, different countries and regions have different standards. So there are several different coding standards for GB2312, BIG5, JIS, and so on. These extended encodings, called ANSI encodings, which use 2 bytes to represent a single character, are called the "MBCS (muilti-bytes charecter set, multibyte character set) ". 

In the Simplified Chinese system, ANSI encoding represents GB2312 encoding, in the Japanese operating system, ANSI code for JIS code, so in Chinese windows to turn code into GB2312,GBK only need to save the text as ANSI code. Different ANSI codes are incompatible, and when information is exchanged internationally, text that belongs to both languages cannot be stored in the same section of ANSI-encoded text. A big disadvantage is that the same coded value represents different words in different coding systems. This can easily cause confusion.

2.ReadString of CStdioFile

If you do not convert your localized settings to the current encoding system when use the CStdioFile readstring function, the text you read will be treated as the text of the other encoding system, and the code will appear garbled.

3.Solution: Localization

If I want to continue using Cstudiofile to read files, you should temporarily set the LOACL info to the current encoding system when reading this type of file. The following two functions allow localized settings

char *setlocale(  

   int category,  

   const char *locale   

);  



wchar_t *_wsetlocale(  

   int category,  

   const wchar_t *locale   

);  

Create a CMBCSLocalEnv class

Based on the above questions, I wrote a new LocaleEnv.h file to solve the problem.

1.Enumerate common language types

First, you create an enumeration of language types to set localized information based on commonly used languages. I've only listed a few common types here, and you can add other types to your needs.

2.Record Raw Localized information

Because this class is intended only to temporarily set the locale to the specified, you need to remember the original language and locale information, and then restore the original settings when the class is destructor.

Cleverly using _tsetlocale to get current localized information . The defination of_tsetlocale seein c:\Program Files (x86)\Windows Kits\10\Include\10.0.10240.0\ucrt\tchar.h

This function has two parameters：category and locale

Parameters:

category : Category affected by locale.

locale : Locale specifier.

Return Value

If a valid locale and category are given, returns a pointer to the string associated with the specified locale and category. If the locale or category is not valid, returns a null pointer and the current locale settings of the program are not changed.

Therefore, we can use _tsetlocale (LC_ALL,  NULL ) function to get the current localized information. Use member variable m_OrgMBCSLoc to remember this information.

class CMBCSLocalEnv{

public:

    CMBCSLocalEnv(ECurLangType lID)

    {

      m_OrgMBCSLoc = NULL;

       #ifdef _MSC_VER

      m_OrgMBCSLoc = _tcsdup( _tsetlocale( LC_ALL,  NULL ) );

    }

    ~CMBCSLocalEnv()

    {

    }

private:

    LPTSTR m_OrgMBCSLoc;

}

3.Conditions for Localization settings

Localization settings require three conditions:

a. Language Code

b. Country Code

c. Encoding

Local names can be constructed using the following sections:

Language code _ Country code, such as (Chinese (Simplified)_China.936)

To get the current localized information, you can use the GetLocaleInfo function：

int GetLocaleInfo(

  _In_      LCID   Locale,

  _In_      LCTYPE LCType,

  _Out_opt_ LPTSTR lpLCData,

  _In_      int    cchData

);

First parameter： _In_      LCID   Locale,

Locale [in]

Locale identifier for which to retrieve information. 

Common Use Language ID

Language ID Constants

$(WindowsSdkDir)Include\WinNT.h

You can find some common language IDs in this link：https://blog.csdn.net/TMS_LI/article/details/48000751

Determines the value of the _In_      LCID   Locale based on the value of the lID . The code is as follows:

LCID mbcsID = 0;

if (lID == LA_ZH_CN_NO_LANG)

{

     mbcsID = LANG_CHINESE_SIMPLIFIED;

}

else if (lID == LA_ZH_TW_NO_LANG)

{

     mbcsID = LANG_CHINESE_TRADITIONAL;

}

else if (lID == LA_EN_US_NO_LANG)

{

    mbcsID = LANG_ENGLISH;

}

else if (lID == LA_UNKNOWN)

{

 return;

}

else

  mbcsID = (LCID)lID;

Second parameter： _In_      LCTYPE LCType,

LCType [in]

The locale information to retrieve. we need Language Code and

Country Code and Encoding.

We can get these three messages by GetLocaleInfo function, the code is as follows: 

TCHAR cpname[32];

int nbytes = 0;

GetLocaleInfo (mbcsID, LOCALE_SENGLANGUAGE, cpname, 32);



TCHAR cpchain1[128];

nbytes = GetLocaleInfo (mbcsID, LOCALE_SENGCOUNTRY   , cpchain1, 128);    



TCHAR cpchain[7];

nbytes = GetLocaleInfo (mbcsID, LOCALE_IDEFAULTANSICODEPAGE, cpchain, 7);

4.set local info same as mbcs info temporarily

Use the _tsetlocale function to temporarily set the current localization settings to the desired setting.

TCHAR NewLocal[256];

_stprintf_s( NewLocal, 256,  _T("%s_%s.%s"),cpname,cpchain1,cpchain);

_tsetlocale( LC_ALL,  NewLocal );

5.The processing of destructor

The locale setting should be set again to the original locale when the class is destructed because we just want local information to be set locally and temporarily. The code is shown as following:

~CMBCSLocalEnv()

{

    if (m_OrgMBCSLoc == NULL)

    {

        return;

    }

#ifdef _MSC_VER

    _tsetlocale( LC_ALL,  m_OrgMBCSLoc );

    ASSERT( _tcscmp(m_OrgMBCSLoc, _tsetlocale(LC_ALL,NULL) ) == 0 );

    free(m_OrgMBCSLoc);

#endif

}

6.The complete code for this class

After a few steps, we can now write this class in its entirety:

#ifndef IC_LOCALE_ENV_H

#define IC_LOCALE_ENV_H

#include <locale.h>

#include "mbctype.h"



enum ECurLangType

{

    LA_ZH_CN_NO_LANG,

    LA_ZH_TW_NO_LANG,

    LA_EN_US_NO_LANG,

    LA_UNKNOWN

};



//fix using function ReadString of CStdioFile issue.

//if reading an ANSI text, and the local info is en-us, but mbcs info is zh-cn, the reading result is wrong.

//so we should set local info same as mbcs info temporarily.



class CMBCSLocalEnv

{

public:

    CMBCSLocalEnv(ECurLangType lID)

    {

        m_OrgMBCSLoc = NULL;

#ifdef _MSC_VER

        m_OrgMBCSLoc = _tcsdup( _tsetlocale( LC_ALL,  NULL ) );

        LCID mbcsID = 0;



        if (lID == LA_ZH_CN_NO_LANG)

        {

            mbcsID = LANG_CHINESE_SIMPLIFIED;

        }

        else if (lID == LA_ZH_TW_NO_LANG)

        {

            mbcsID = LANG_CHINESE_TRADITIONAL;

        }

        else if (lID == LA_EN_US_NO_LANG)

        {

            mbcsID = LANG_ENGLISH;

        }

        else if (lID == LA_UNKNOWN)

        {

            return;

        }



        TCHAR cpname[32];



        int nbytes = 0;

        GetLocaleInfo (mbcsID, LOCALE_SENGLANGUAGE, cpname, 32);



        TCHAR cpchain1[128];

        nbytes = GetLocaleInfo (mbcsID, LOCALE_SENGCOUNTRY   , cpchain1, 128);    



        TCHAR cpchain[7];

        nbytes = GetLocaleInfo (mbcsID, LOCALE_IDEFAULTANSICODEPAGE, cpchain, 7);

        TCHAR NewLocal[256];



        _stprintf_s( NewLocal, 256,  _T("%s_%s.%s"),cpname,cpchain1,cpchain);

        _tsetlocale( LC_ALL,  NewLocal );

#endif

    }



    ~CMBCSLocalEnv()

    {

        if (m_OrgMBCSLoc == NULL)

        {

            return;

        }

#ifdef _MSC_VE

        _tsetlocale( LC_ALL,  m_OrgMBCSLoc );

        ASSERT( _tcscmp(m_OrgMBCSLoc, _tsetlocale(LC_ALL,NULL) ) == 0 );

        free(m_OrgMBCSLoc);

#endif

    }

private:

    LPTSTR m_OrgMBCSLoc;

};

#endif

How to use this class 

The way to use this class is simple. Creates a CMBCSLocalEnv temporary variable when reading a TXT file encoded in ANSI format.

At this point, the data read from the file is correct：

Most importantly, this LocaleEnv.h file can be placed in other projects by simply include this header file. It can avoid the redundancy of code. I hope this tutorial will help you.

 Thank you for your attention.
@hushuilan