A little girl lost in translation.
Once upon a time, a 10-year-old girl and her family returned to the U.S. after living in South Korea for four years. Born into a family of Korean heritage, the girl was told that education was critical to success in life, so she worked hard in school.
One fateful day in the fifth grade, the girl gazed at her teacher with a puzzled look on her face, wondering, “Why doesn't she believe that I completed my assignment? I finished my rough draft.”
After being accused of lying and being called to a meeting with the principal and her parents, the girl realized that in the U.S., a “rough draft” was actually a cleaned-up version of a paper.
Even though the girl was fluent in English, she interpreted a literal meaning of the term “rough”; the girl understood that this so-called “rough draft” was not the final product in the paper project.
The girl thought the teacher wanted to see the edits and changes she had made to show her progress toward the final product; consequently, she turned in her assignment with bright red markings on her pages. In turn, the teacher thought the student was making excuses for not completing the assignment.
This little girl was me. Although I was born in the U.S., the period in my life in which I was transitioning back into the culture after living in Korea for several years was a challenging and educational experience.
A cross-cultural misunderstanding inspired my career choice.
Now let's fast forward to the present. In the past few years, I’ve been able to apply my personal experiences and expertise in psychometric and assessment training in educational psychology to my work with the Kuder research faculty.
Striving to ensure fairness in education in a practical ways, I've come to understand the importance of collaborative work and using both quantitative and qualitative methods when working to serve people in real-life situations throughout my professional work and studies.
How cultural and linguistic elements are factored in test localization and translation.
Due to the growing phenomena of testing, localization and test translation is becoming more prevalent in cross-cultural comparisons (Van de Vijver & Hambleton, 1996). In the U.S. alone, schools are becoming increasingly more racially and culturally diverse.
Since 2005, almost 50% of U.S. students can be classified as culturally different (Ford, 2010). Also, Lee and Koro-Ljungberg (2007) estimated that by the year 2050, no more than 50% of the U.S. population will be of Anglo ancestry.
Beyond the U.S., countries worldwide are administering assessments for international comparisons. More specifically, as cross-cultural comparisons are becoming more prevalent, there is an increasing need for linguistic and cultural equivalence of original and localized tests.
With the increasing popularity of Kuder assessments in career guidance counseling worldwide, I'm privileged to be a part of a great effort toward obtaining cultural and linguistic equivalency tests.
Consequently, Kuder assessments have been localized from the United States to other countries and regions and translated from English to other languages. By providing career assessments in multiple languages, Kuder assists individuals across all ages to explore career options, plan for education, and prepare for job search. However, test localization can lead to bias and may result in assessments that are not reliable or valid (meaningful) for other culture groups.
Essentially, if not carefully approached, an assessment for one group may become a different type of assessment for another group. Thus, not only is it imperative to use a combination of methods to achieve equivalent assessments, but it's also vital to examine the cultures and consider differences in values that influence the lifestyles and experiences of individuals when adapting an assessment across cultures.
Understanding cultural differences provides a framework within which an assessment can be appropriately constructed for a valid localized assessment.
Specifically, since Kuder assessments were developed in the U.S., there are cultural and linguistic elements embedded in the item content and choice of words that are appropriate for individuals from the U.S.
Kuder uses both quantitative and qualitative approaches to ensure reliable and valid assessments.
Both traditional and modern cultural values and systems are important to consider when adapting Kuder assessments to another language. Currently, the Kuder research team uses both quantitative and qualitative approaches in ensuring reliable and valid assessments for global use.
Quantitatively, differential item functioning (DIF) is a statistical method that allows one to determine whether the items are functioning in the same way for all subgroups of interest. In other words, if there is a DIF item, the item may be biased.
The purpose of using the differential item functioning (DIF) method is to analyze Kuder Assessments for potentially biased items for test fairness in the localized versions.
Although there is no direct connection between DIF and the validity of a test, item analysis using DIF facilitates deeper examination of construct representation (Holland, Wainer, & ETS, 1993). Thus, we also ought to examine item content and have cultural experts involved in the qualitative process as well.
Taking cultural and linguistic background into consideration, cultural and subject area experts both examine the items and study the cultures. When evaluating items, these individuals can look at the grammatical structure, underlying meanings of words or phrases, item format, test layout and directions, and meaningfulness of the construct and score interpretation.
Think-alouds provide the Kuder research faculty with amazing insights.
Cultural sensitivity review is critical to examine item content. Moreover, think-alouds are also helpful to find out if the Kuder assessment items contain any insensitive, offensive, stereotypic or biased language, content or context when used within the social context of a specific culture or group.
The purpose of think-alouds is to understand what students may be thinking as they’re taking the assessments on their own. In the beginning, the researcher asks the student to read the questions out loud, think out loud, and tell the researcher how he/she is responding and what is going through his/her mind. The researcher speaks only at the end for follow-up questions to gain deeper insight on items for which students may have had concerns or difficulties.
Think-aloud exercises allow the researcher to see what it's like for actual test takers from the target group to use it. During the think-aloud session, the researcher asks the students to take the assessment while reading the questions out loud, think out loud, and verbally share how they're responding and what's going through their minds. Essentially, participants are asked to freely speak what they're thinking aloud as they take the assessment.
Think-alouds can also be fun. We learn about other cultures while making personal connections as we're trying to relate to students with various backgrounds.
In a recent think-aloud session with a group from Singapore, a male college student was having some difficulty understanding an item about a “trucking company.” After some dialogue, I was quickly reminded that in Singapore, a “truck” is referred to as a “lorry” due to the British influence.
Immediately, the British TV series about motor vehicles Top Gear (and I'm referring to the original, of course), became a topic of common interest. It was exciting to me because not only is Top Gear one of my favorite shows, but while watching it, I've often thought it would be cool for it to be socially acceptable for me to actually use a word like “lorry.” Now here I was having this experience in the middle of the night with a student from Singapore!
I'm honored to be a part of this exciting and valuable adventure.
With a bicultural and bilingual background, I continue to grow in my appreciation for and sensitivity to others and seeking to better understand their perspectives and experiences. Being committed to the Kuder's mission to encourage students to “dream big,” I love helping individuals of diverse backgrounds receive the guidance and tools they need to be successful – not just in their careers but also in life.
Who knew that the same embarrassed and utterly shy fifth grader would one day have the opportunity to work with at Kuder the research faculty to ensure cultural and linguistic equivalence – specifically, for quality internationally adapted educational tools?
As Kuder continues to serve students from all over the world, I look forward to learning more and living out my love for people and life using effective methods to ensure that students are provided with the high quality and inclusive tools.
Ford, D. (2010). Culturally Responsive Classrooms: Affirming Culturally Different Gifted Students. Gifted Child Today, 33, 50-53.
Lee, I., & Koro-Ljungberg, M. (2007). A phenomenological study of Korean students’ acculturation in middle schools in the USA. Journal of Research in International Education, 6(1), 95-117.
Holland, P. W., Wainer, H., & Educational Testing Service (1993). Differential item functioning. Hillsdale, NJ: Lawrence Erlbaum Associates.
van de Vijver, F. J. R. & Hambleton, R. K. (1996). Translating tests: Some practical guidelines. European Psychologist, 1, 89-99.
- career development theory