টেক্সট প্রসেসিং

মূল ধাপগুলো

1. Cleaning — HTML, URL, emoji, ducplicate ছাঁকা।
2. Normalization — lower-case, unicode NFC।
3. Tokenization — word, subword, character।
4. Stopword / Stemming / Lemmatization — classical NLP।
5. Encoding — BoW, TF-IDF, embedding ids।

Cleaning উদাহরণ

import re, unicodedata

def clean(text):
    text = unicodedata.normalize("NFC", text)
    text = re.sub(r"http\S+", " ", text)         # URL
    text = re.sub(r"<.*?>", " ", text)            # HTML
    text = re.sub(r"[^\w\s\u0980-\u09FF]", " ", text)  # বাংলা+ASCII
    text = re.sub(r"\s+", " ", text).strip().lower()
    return text

Tokenization

Word tokenization

from nltk.tokenize import word_tokenize
print(word_tokenize("আমি ডিপ লার্নিং শিখছি।"))

Subword — BPE / WordPiece / SentencePiece

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-multilingual-cased")
print(tok.tokenize("আমি ট্রান্সফরমার ভালোবাসি"))
# ['আমি', 'ট্রান', '##স', '##ফরম', '##ার', 'ভাল', '##োবাসি']

Stopword, Stemming, Lemmatization

from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer

sw = set(stopwords.words("english"))
ps, wl = PorterStemmer(), WordNetLemmatizer()

words = ["running", "studies", "better"]
print([ps.stem(w) for w in words])           # run, studi, better
print([wl.lemmatize(w, pos="v") for w in words])  # run, study, better

Encoding

Bag of Words

from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer()
X = vec.fit_transform(["i love nlp", "nlp is fun"])

TF-IDF

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(ngram_range=(1,2), min_df=2)
X = tfidf.fit_transform(corpus)

Vocab → IDs (Deep model-এর জন্য)

from collections import Counter
tokens = text.split()
vocab = {w:i+2 for i,(w,_) in enumerate(Counter(tokens).most_common(20000))}
vocab["<pad>"], vocab["<unk>"] = 0, 1
ids = [vocab.get(w, 1) for w in tokens]

বাংলা-নির্দিষ্ট চ্যালেঞ্জ

• Unicode normalization (NFC) আবশ্যক।
• Compound character — sentence-piece tokenizer ভালো।
• সীমিত labeled data → multilingual model (XLM-R, mBERT)।

অনুশীলন

সারসংক্ষেপ