(This article was written by Jingfang Huang with inputs from Kasun Perera)
Did you know that Ai Palette’s trendspotting and prediction algorithm analyses data from 16 languages? We do this through a method called natural language processing (NLP). But how does our NLP work? This article intends to guide you through the fundamental knowledge about NLP tokenization and introduce you to the tokenization package created by Ai Palette that can be applied to different languages.
What is text tokenisation?
Tokenization is a technique that is used to split any text that the AI system needs to analyse into smaller “tokens”. These tokens can be sentences, phrases or words with which the system can recognize and take meaningful action. As humans, we convey our messages in complete sentences, paragraphs, or, in this case, blogs. We are capable of reading huge blocks of text and understanding what they mean. However, computers cannot yet process such large chunks of information. This is why we need to split it into smaller pieces so they can process and analyse the text. This is a critical step in building an AI system.
How Does Tokenisation Help
Data comes in two forms: numbers, names, addresses and other kinds of data that follow a set pattern are called “structured data”. The ones that don\’t, such as comments on social media, are called “unstructured data”. Tokenisation helps developers of an AI system convert unstructured data into structured data through various processes. Numerous sets of code, called ‘tokenizers’, are available for developers to help them tokenize text in different languages.
Why do different languages have different tokenizers?
When processing data in different languages, we must understand the concept of the smallest language meaningful unit (LMU), otherwise known as a morpheme, in a phrase, sentence or paragraph structure. If morphemes were subdivided any further, they would become meaningless. Tokenization helps identify these small units to split the text meaningfully. Since different written languages have various forms and structures, we need different methods to split them effectively. For example, English uses a space to separate the words, but the Chinese language uses no spaces between words. This is why different languages have different tokenizers.
Ai Palette Offers a Unique Multi-language Tokenizer Package for Developers
As the world’s first AI platform for product concept generation, Ai Palette’s service has covered 21 countries in 16 languages and is still growing. Being able to perform data processing with different languages is a key to our accurate analysis. We have decided to make our tokenization methods standardised. We have made this package available for everyone interested in food tech and NLP!
“aipalettenlp” contains a list of NLP functions that will be used for future tasks in Ai Palette. Many useful modules and functions will be included in the package. For now, it has a module that consists of tokenizers of different languages and another module that has several functions for text preprocessing. A detailed description of each function is included in the PyPI documentation. Click here to access the files.
