Indian social media is a linguistic mosaic. Users routinely switch between English, Hindi, Bengali, and Gujarati within a single tweet or comment, sometimes within a single sentence. This code-mixing creates a practical problem for downstream NLP tasks: before you can do sentiment analysis, hate speech detection, or machine translation on code-mixed text, you need to know which language each token belongs to. Token-level language identification, the task of labeling every word in a sentence with its source language, is a prerequisite for almost everything else in the code-mixed NLP pipeline. Prior work has handled this task for bilingual mixing, typically one matrix language plus English. IndicTriMix tackles the harder case: trilingual code-mixing involving three languages simultaneously.
Why Two Languages Are Not Enough
India has 22 official languages and roughly 7 percent of the population speaks three languages. On social media, this means a single utterance might weave English, Hindi, and Bengali, or English, Hindi, and Gujarati, in ways that do not follow any fixed pattern. The grammar of the sentence might shift mid-phrase, and the embedding language might appear at any position relative to the matrix language.
Prior work in token-level language identification has been limited to two-language scenarios: one matrix language and one embedded language. The datasets, models, and evaluation protocols all assume this bilingual structure. Existing benchmarks for Indian code-mixed text cover Hindi-English, Bengali-English, and Gujarati-English pairs, but not combinations of three languages in the same sentence. The few works that have explored three-language data have used it only for downstream tasks like offensive language detection or sentiment analysis, without addressing the language identification prerequisite. IndicTriMix fills this gap by building datasets, models, and benchmarks specifically for trilingual code-mixed language identification.
Generating Trilingual Code-Mixed Data
The core difficulty is that trilingual code-mixed corpora do not exist in sufficient quantity. The authors build their datasets from IndicCMix, a corpus of 1.1 million sentences where each unique English sentence is translated into 11 Indic languages. IndicCMix provides both romanized and native-script forms, which eliminates the need for external transliteration and reduces error propagation. The romanized text is the starting point for code-mixed generation.
Two generation approaches produce the training and evaluation data. The rule-based approach takes parallel sentences in three languages and combines phrases from each, enforcing a minimum token count per language to avoid generating sentences where one language dominates with only a handful of words from the other two. The language of each token is unambiguously known because the source sentences are labeled, so the resulting annotations are automatic and noise-free.
The LLM-based approach uses Gemini 2.5 Pro with a detailed prompt that instructs the model to generate naturally flowing trilingual code-mixed sentences. The prompt emphasizes varying the sequence of languages, avoiding rigid ordering, and maintaining grammatical coherence. Two language experts, each a trilingual speaker with postgraduate education, manually annotate the language tag of every token in the generated sentences. This produces roughly 500 sentences per code-mixed setting for both dev and test sets.
The quality of generated sentences is validated using semantic similarity measures. BertScore, Sentence-BERT (MPNET), and LaBSE embeddings are used to compare each code-mixed sentence against its language-specific parallel sentences. All similarity scores exceed 0.85 on average across both generation approaches, indicating that the generated sentences are both fluent and faithful to their source content.
Formulating the Task as Sequence Labeling
The authors formulate token-level language identification as a sequence labeling problem. Given a sentence as a sequence of tokens, the goal is to produce a corresponding sequence of labels where each token is tagged as BEN (Bengali), ENG (English), GUJ (Gujarati), HIN (Hindi), or UNI (punctuation and symbols). This is standard sequence labeling framing, but the challenge lies in the input: romanized text where different languages share the Latin alphabet, making purely character-based approaches insufficient.
The pipeline handles tokenization, subword alignment, and label masking. Since transformer tokenizers split words into subword units, the word-level annotation is assigned only to the first subword token. All subsequent subwords of the same word receive the ignore index of -100, preventing them from contributing to the training loss. This is a common strategy in token classification with subword tokenizers, but the authors note it as a limitation: trailing subword representations do not receive direct supervision from the word-level label.
Two Models, Three Training Configurations
The authors fine-tune two multilingual transformer models: MuRIL (Multilingual Representations for Indian Languages), specifically designed for Indian languages and their code-mixed variations, and XLM-RoBERTa-base, trained on 100+ languages as a general multilingual baseline. Both models are fine-tuned for 10 epochs with a learning rate of 2e-5, batch size of 16, max sequence length of 128 tokens, and weight decay of 0.01, using mixed-precision fp16 optimization.
Three training configurations test different levels of language specificity. Combined data training merges all multilingual data from both language pairs. ENG-HIN-BEN training uses only English-Hindi-Bengali code-mixed text. ENG-HIN-GUJ training uses only English-Hindi-Gujarati code-mixed text. Each configuration is evaluated on its corresponding development and test sets, allowing comparison between a general multilingual model and language-pair-specific specialists.
Results: High Performance, With Hindi as the Hard Case
On rule-based development sets, both models achieve F1 scores above 0.98 across all language classes. The near-perfect scores are expected because rule-based generation produces clean, unambiguous annotations. The more informative results come from the LLM-generated test sets, where the data reflects naturalistic code-mixing with transliteration artifacts and informal language.
On the combined training configuration evaluated on ENG-HIN-BEN test-LLM, MuRIL achieves 0.975 (BEN), 0.993 (ENG), 0.773 (HIN), and 1.0 (UNI). XLM-RoBERTa achieves 0.968 (BEN), 0.99 (ENG), 0.771 (HIN), and 1.0 (UNI). On ENG-HIN-GUJ test-LLM, MuRIL achieves 0.996 (ENG), 0.923 (GUJ), 0.778 (HIN), and 1.0 (UNI). XLM-RoBERTa achieves 0.993 (ENG), 0.927 (GUJ), 0.779 (HIN), and 1.0 (UNI).
The pattern is consistent: English and punctuation are identified with near-perfect accuracy, Gujarati and Bengali score in the 0.92 to 0.98 range, and Hindi is the persistent weak point at 0.76 to 0.78 on LLM-generated test sets. The authors attribute this to transliteration ambiguity. Several Indo-Aryan languages share similar phonetic characteristics when represented using the Latin alphabet, blurring the boundaries between Hindi and its relatives. Hindi also suffers from label imbalance in certain evaluation slices where it has fewer supporting examples, making the F1 score sensitive to precision and recall variations.
MuRIL versus XLM-RoBERTa
MuRIL generally achieves slightly better performance than XLM-RoBERTa, particularly under combined-data training. This advantage is consistent with MuRIL's pre-training emphasis on Indian languages and its design for handling the morphological and phonetic complexities of Indo-Aryan languages in both native scripts and Latin transliteration. However, the difference is small and inconsistent across configurations. XLM-RoBERTa achieves comparable performance in several cases, and in the ENG-HIN-GUJ dev-LLM setting, it actually outperforms MuRIL on Hindi (0.645 versus 0.603). The practical takeaway is that both models are viable, and the choice may depend on deployment constraints rather than raw accuracy.
Language-Pair-Specific versus Combined Training
Models trained on a specific language pair achieve consistently strong performance on the corresponding evaluation set, indicating that specialized training effectively captures language-specific characteristics. But combined multilingual training also produces competitive results across both language pairs. A single model trained on all available data can learn shared multilingual representations while maintaining strong overall performance. This is good news for deployment: one model can serve multiple trilingual settings without retraining, at a modest cost in accuracy compared to language-pair-specific specialists.
What the Limitations Reveal
The authors are straightforward about three limitations. First, label imbalance across evaluation subsets makes F1 scores unstable for underrepresented languages. When a language has very few examples in a particular evaluation slice, small errors swing the score dramatically. Second, transliteration creates genuine ambiguity. Informal abbreviations, slang, and phonetic interference between Indo-Aryan languages written in Latin script make clean token-level boundaries difficult to define, let alone predict. Third, the subword alignment strategy is simple but lossy: only the first subword of each word receives the ground-truth label, and subsequent subwords get no supervision. More sophisticated alignment strategies might improve performance, particularly for languages that tend to produce longer subword sequences.
There is also a scope limitation worth noting. The work addresses language identification as an isolated task, but the real motivation is enabling downstream NLP. The authors do not evaluate whether improved language identification actually improves sentiment analysis, hate speech detection, or other tasks on trilingual code-mixed text. That connection remains to be established.
What This Means for practitioners
If you are building NLP pipelines for Indian social media, the trilingual case is real and under-addressed. The authors release fine-tuned models, datasets, and source code, providing a ready-made benchmark for future work. The rule-based generation approach is reproducible with any three-way parallel corpus, so the methodology extends to other language triples beyond the Hindi-Bengali-Gujarati combinations tested here. The key practical finding is that contextual transformer embeddings handle trilingual code-mixed language identification well, with the main failure mode being phonetic ambiguity between related languages in romanized form. For applications where this matters, incorporating phonetic or lexical features alongside contextual embeddings is a natural next step.