Nevertheless, even though research on artificial intelligence-based Korean language models, which is the basis of natural language processing, has been actively conducted until recently, the means for diagnosing their social biases are still lacking.
 
Amidst being aware of such problems, the K-StereoSet, which will be presented by Professor Sungroh Yoon's team and has been based on the development set of the "StereoSet" released by MIT to diagnose the social bias of the English language model, will continue to be modified and developed in line with the situation of Korea's reality. The original dataset, consisting of about 4,000 samples, was first translated through the Naver Papago API, and then independently inspected and reviewed by a number of researchers, and was built by post-processing to preserve the original sample form and purpose.  
The field of social bias in the data consists of a total of four fields: gender, religion, occupation, and race, and the sample form for diagnosing bias is classified into two categories.
 
The first is an 'intrasentence' form for testing bias within sentences. (See the example on the left of the image above)
 
When a blank sentence is given, it is diagnosed using which of the three examples provided is given a high score as the content to be filled in the blank. For example, as shown in the example on the left above, it is possible to check whether a person in the profession of 'psychologist' is given a bias of 'self-righteousness' within a sentence.
 
The second is the ‘intersentence’ form which is a bias diagnostic test that may occur between different sentences. (Refer to the example on the right of the image above) When a certain context is given, three options are given as the next sentence, and diagnosis is made using which sentence is chosen to be given the highest score. For example, given the context that a person is 'Hispanic' as in the example on the right above, in the following sentence, it is possible to check whether there is any bias such whether the individual is an 'illegal citizen'.
 
“Amongst the bias diagnosis samples, those within the ‘unrelated’ label words that are completely unrelated to the context are placed in the blanks, and thus it is easy to deviate from the intended meaning of the original text during the process of automatic translation. Furthermore, the example sentences of the inter-sentence bias diagnosis sample were converted while paying attention to special situations, such as cases where contextual sentences were not considered,” said researcher JongYun Song who led the research.
 
"As artificial intelligence-based Korean language models are getting more advanced and commercialized, efforts to secure ethics and eliminate bias are key, and in alignment with Hangul Day, we hope that K-StereoSet will be a small but meaningful first step towards developing artificial intelligence technology that uses better Hangul," said Professor Sungroh Yoon, the research director.