Administer > Database administration > Data persistence > IR Expert > Chinese, Japanese, and Korean language analyzer

Chinese, Japanese, and Korean language analyzer

The language analyzer is a text tokenization analysis tool. It uses statistical-based, rule-based, and dictionary-based techniques to quickly determine the correct segmentation of Chinese, Japanese, or Korean text.

In addition to tokenization, the language analyzer offers normalization, part-of-speech information, morphological analysis, and reading information.

Searching for Chinese or Japanese and English words in a Chinese or Japanese environment

For Chinese

  • English stemming and branching are not supported.
  • A mixture of Chinese and English characters is not supported when the characters are separated by a blank space.
  • A mixture of Chinese and English characters is supported when the characters are not separated by a blank space.
  • Multiple Chinese keyword search is not supported.

For Japanese

  • English stemming and branching are not supported.
  • A mixture of Japanese and English characters is not supported when the characters are separated by a blank space.
  • A mixture of Japanese and English characters is supported when the characters are not separated by a blank space.
  • Multiple Japanese keyword search is not supported.

Searching for Chinese or Japanese and English words in an English environment

For Chinese

  • English stemming and branching are supported.
  • A mixture of Chinese and English characters is supported when the words are separated by a blank space.
  • Multiple Chinese keyword search is supported when the words are separated by a blank space.
  • Chinese stop words are not supported.
  • Chinese stemming is supported.

For Japanese

  • English stemming and branching are supported.
  • A mixture of Japanese and English characters is supported when the words are separated by a blank space.
  • Multiple Japanese keyword search is supported when the words are separated by a blank space.
  • Japanese stop words are not supported.
  • Japanese stemming is supported.

 

Related topics

IR Expert
IR Expert tasks
IR Query features
Promoting or discarding a solution candidate
Searching the central Knowledge Base
Standard record lists and IR Expert
Using IR Expert to create a query
What is Knowledge Engineering?
Customizing IR Expert for foreign languages
Creating an IR file
Access IR Expert
Load data files with IR Expert keys
Start IR Asynchronous mode