Natural Language Processing on Arshad Siddiqui

Natural Language Processing

Mon, 01 Jan 0001 00:00:00 +0000

Natural Language Processing #

Natural Language Processing (NLP) studies how computers can analyse, understand, represent, and generate human language.

It combines ideas from linguistics, computer science, machine learning, and deep learning to work with text and language-based information.

Natural Language Processing = Linguistics + Computation + Machine Learning

The learning path begins with language understanding and vector representations, progresses through language modelling, tagging, and parsing, and then moves towards transformers, knowledge graphs, Retrieval-Augmented Generation, and modern NLP applications.

N-gram Language Modelling

Mon, 01 Jan 0001 00:00:00 +0000

N-gram Language Modelling #

A language model assigns probabilities to sequences of words. It can compare complete sentences or predict which word is likely to come next.

Key ideas include:

word prediction and sequence probability
the chain rule and Markov assumption
unigram, bigram and trigram models
Maximum Likelihood Estimation
unseen sequences and smoothing
interpolation and backoff
intrinsic and extrinsic evaluation
perplexity

Learning Objectives #

Explain what a language model represents.
Calculate simple unigram and bigram probabilities.
Explain why unseen N-grams create zero probabilities.
Distinguish smoothing, interpolation and backoff.
Interpret perplexity correctly.

Big Picture #

flowchart TD
 A["Training Corpus"] --> B["Count N-grams"]
 B --> C["Estimate Probabilities"]
 C --> D["Handle Unseen Events"]
 D --> E["Score Word Sequences"]
 E --> F["Evaluate Model"]

 style A fill:#E1F5FE
 style B fill:#C8E6C9
 style C fill:#FFF9C4
 style D fill:#EDE7F6
 style E fill:#E1F5FE
 style F fill:#C8E6C9

1. What Is a Language Model? ☆ #

A language model estimates how probable a sequence of words is.

NN and Neural Language Modelling

Mon, 01 Jan 0001 00:00:00 +0000

Neural Networks and Neural Language Modelling #

Neural networks learn useful representations and nonlinear relationships directly from data. In language modelling, they replace discrete N-gram identities with learned word embeddings and use these representations to predict the next word.

Learning Objectives #

Explain the computation performed by a neural unit.
Describe why hidden layers and nonlinear activations are needed.
Explain how feed-forward networks support NLP classification.
Trace the flow through a feed-forward neural language model.
Compare N-gram and neural language models.

Big Picture #

flowchart TD
 A["Context Words"] --> B["One-hot Inputs"]
 B --> C["Embedding Lookup"]
 C --> D["Combined Context"]
 D --> E["Hidden Layer"]
 E --> F["Softmax"]
 F --> G["Next-word Probabilities"]

 style A fill:#E1F5FE
 style B fill:#C8E6C9
 style C fill:#FFF9C4
 style D fill:#EDE7F6
 style E fill:#E1F5FE
 style F fill:#C8E6C9
 style G fill:#FFF9C4

1. Neural Network Units ☆ #

A neural unit receives input values, multiplies them by learned weights, adds a bias, and applies an activation function.

LLM and Prompt Engineering

Mon, 01 Jan 0001 00:00:00 +0000

LLMs and Prompt Engineering #

A Large Language Model extends neural language modelling through much larger datasets, many more parameters, broad pretraining and adaptation to many downstream tasks. Its central operation remains next-token prediction.

Learning Objectives #

Explain how neural language modelling develops into an LLM.
Describe the meaning of large, general-purpose and pretrained.
Explain how a prompt guides generation.
Distinguish zero-shot and few-shot prompting.
Compare prompting with model adaptation.

Big Picture #

flowchart TD
 A["Broad Text Data"] --> B["Large-scale Pretraining"]
 B --> C["General Language Model"]
 C --> D["Prompt or Adaptation"]
 D --> E["Task Output"]

 style A fill:#E1F5FE
 style B fill:#C8E6C9
 style C fill:#FFF9C4
 style D fill:#EDE7F6
 style E fill:#E1F5FE

1. From Neural Language Models to LLMs ☆ #

A neural language model learns a conditional probability for the next token:

Part-of-Speech Tagging and Hidden Markov Models

Mon, 01 Jan 0001 00:00:00 +0000

Part-of-Speech Tagging and Hidden Markov Models #

Part-of-Speech tagging assigns a grammatical category to each word in a sequence. Because many words can play different grammatical roles, a tagger must use surrounding context rather than examine each word independently.

Learning Objectives #

Identify common English word classes and Penn Treebank tags.
Explain why POS tagging is a sequence-labelling problem.
Describe the Markov assumption.
Distinguish a Markov Chain from a Hidden Markov Model.
Explain how an HMM represents POS tagging.

Big Picture #

flowchart TD
 A["Word Sequence"] --> B["Use Context"]
 B --> C["Infer Hidden Tags"]
 C --> D["Tagged Sequence"]

 style A fill:#E1F5FE
 style B fill:#C8E6C9
 style C fill:#FFF9C4
 style D fill:#EDE7F6

1. What Is Part-of-Speech Tagging? ☆ #

A part of speech describes the grammatical role played by a word in a sentence. POS tagging assigns one tag to every word in a sequence.

Statistical, ML and Neural Models of POS Tagging

Mon, 01 Jan 0001 00:00:00 +0000

Statistical, ML and Neural Models of POS Tagging #

‘HMM Inference: Forward and Viterbi Algorithms’ covers the portion:

Forward Algorithm
Viterbi Algorithm
HMM inference for POS tagging

The complete Topic also includes:

Maximum Entropy Markov Models
Bidirectionality
Neural-network models for POS tagging

HMM Inference: Forward and Viterbi Algorithms #

Hidden Markov Models create two closely related inference problems:

Likelihood: How probable is an observed sequence under the model?
Decoding: Which hidden-state sequence most probably generated the observations?

The Forward Algorithm solves the likelihood problem, while the Viterbi Algorithm solves the decoding problem. Both use dynamic programming and a trellis, but they combine paths differently.