Natural Language Processing

Natural Language Processing #

Natural Language Processing (NLP) studies how computers can analyse, understand, represent, and generate human language.

It combines ideas from linguistics, computer science, machine learning, and deep learning to work with text and language-based information.

Natural Language Processing = Linguistics + Computation + Machine Learning

The learning path begins with language understanding and vector representations, progresses through language modelling, tagging, and parsing, and then moves towards transformers, knowledge graphs, Retrieval-Augmented Generation, and modern NLP applications.

Understanding and Generation

NLP - Understanding and Generation #

• The Study of Language.
• Applications of Natural Language Understanding.
• Evaluating Language Understanding Systems.
• Different Levels of Language Analysis.
• Organisation of Natural Language Understanding Systems.

Learning Objectives #

• Explain what Natural Language Processing studies and how it relates to artificial intelligence and linguistics.
• Describe why human language is difficult for computers to process.
• Recognise the main applications and stages of an NLP pipeline.
• Distinguish morphological, lexical, syntactic, semantic, pragmatic, and discourse analysis.
• Explain the relationship between natural language understanding and natural language generation.
• Identify suitable ways to evaluate different NLP systems.

Big Picture #

flowchart TD
    A[Human Language] --> B[Acquire and Prepare Text or Speech]
    B --> C[Analyse Language Structure]
    C --> D[Represent Meaning and Context]
    D --> E[Model or Reason]
    E --> F[Understand, Predict or Generate]
    F --> G[Evaluate and Improve]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9
    style G fill:#FFF9C4

1. What Is Natural Language Processing? ☆ #

Natural language processing (NLP) is the discipline of building machines that can manipulate human language - or data that resembles human language - in the way that it is written, spoken, and organised.

Vector Semantics and Embedding

NLP - Vector Semantics and Embedding #

• Lexical semantics and word meaning.
• Lemmas, senses, and semantic relationships.
• Distributional hypothesis.
• Vector semantics and word embeddings.
• Document and word vectors.
• Dot product and cosine similarity.
• Term Frequency–Inverse Document Frequency (TF-IDF).
• Prediction-based word embeddings and self-supervision.
• Word2Vec using Skip-gram with Negative Sampling and CBOW.
• Embedding matrices, context-window choices, analogies, visualisation, and bias.
• GloVe and global word–word co-occurrence statistics.

Learning Objectives #

• Explain lexical semantics and distinguish a lemma from a word sense.
• Compare synonymy, similarity, relatedness, antonymy, and connotation.
• Explain the distributional hypothesis and its role in modelling meaning.
• Describe how words and documents can be represented as vectors.
• Construct and interpret word–document and word–context matrices.
• Calculate dot product and cosine similarity between vectors.
• Explain why raw word frequency can be misleading.
• Calculate TF, IDF, and TF-IDF weights.
• Explain how Word2Vec learns embeddings from a prediction task.
• Construct positive and negative Skip-gram training pairs.
• Explain how sigmoid, negative sampling, and gradient descent train SGNS.
• Compare Skip-gram with CBOW.
• Explain how context-window size affects the relationships captured.
• Interpret word analogies and two-dimensional embedding visualisations.
• Explain how GloVe combines global counts with learned dense vectors.
• Recognise how social biases can be encoded in word embeddings.

Big Picture #

flowchart TD
    A[Words and Documents] --> B[Observe Their Context]
    B --> C[Represent Them as Vectors]
    C --> D[Compare Vector Directions]
    D --> E[Estimate Semantic Similarity]
    E --> F[Search, Classify, Retrieve or Generate]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9

1. Lexical Semantics ☆ #

Lexical semantics is the linguistic study of word meaning and the relationships between word meanings.

N-gram Language Modelling

N-gram Language Modelling #

A language model assigns probabilities to sequences of words. It can compare complete sentences or predict which word is likely to come next.

Key ideas include:

• word prediction and sequence probability
• the chain rule and Markov assumption
• unigram, bigram and trigram models
• Maximum Likelihood Estimation
• unseen sequences and smoothing
• interpolation and backoff
• intrinsic and extrinsic evaluation
• perplexity

Learning Objectives #

• Explain what a language model represents.
• Calculate simple unigram and bigram probabilities.
• Explain why unseen N-grams create zero probabilities.
• Distinguish smoothing, interpolation and backoff.
• Interpret perplexity correctly.

Big Picture #

flowchart TD
    A["Training Corpus"] --> B["Count N-grams"]
    B --> C["Estimate Probabilities"]
    C --> D["Handle Unseen Events"]
    D --> E["Score Word Sequences"]
    E --> F["Evaluate Model"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9

1. What Is a Language Model? ☆ #

A language model estimates how probable a sequence of words is.

NN and Neural Language Modelling

Neural Networks and Neural Language Modelling #

Neural networks learn useful representations and nonlinear relationships directly from data. In language modelling, they replace discrete N-gram identities with learned word embeddings and use these representations to predict the next word.

Learning Objectives #

• Explain the computation performed by a neural unit.
• Describe why hidden layers and nonlinear activations are needed.
• Explain how feed-forward networks support NLP classification.
• Trace the flow through a feed-forward neural language model.
• Compare N-gram and neural language models.

Big Picture #

flowchart TD
    A["Context Words"] --> B["One-hot Inputs"]
    B --> C["Embedding Lookup"]
    C --> D["Combined Context"]
    D --> E["Hidden Layer"]
    E --> F["Softmax"]
    F --> G["Next-word Probabilities"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9
    style G fill:#FFF9C4

1. Neural Network Units ☆ #

A neural unit receives input values, multiplies them by learned weights, adds a bias, and applies an activation function.

LLM and Prompt Engineering

LLMs and Prompt Engineering #

A Large Language Model extends neural language modelling through much larger datasets, many more parameters, broad pretraining and adaptation to many downstream tasks. Its central operation remains next-token prediction.

Learning Objectives #

• Explain how neural language modelling develops into an LLM.
• Describe the meaning of large, general-purpose and pretrained.
• Explain how a prompt guides generation.
• Distinguish zero-shot and few-shot prompting.
• Compare prompting with model adaptation.

Big Picture #

flowchart TD
    A["Broad Text Data"] --> B["Large-scale Pretraining"]
    B --> C["General Language Model"]
    C --> D["Prompt or Adaptation"]
    D --> E["Task Output"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE

1. From Neural Language Models to LLMs ☆ #

A neural language model learns a conditional probability for the next token:

Part-of-Speech Tagging and Hidden Markov Models

Part-of-Speech Tagging and Hidden Markov Models #

Part-of-Speech tagging assigns a grammatical category to each word in a sequence. Because many words can play different grammatical roles, a tagger must use surrounding context rather than examine each word independently.

Learning Objectives #

• Identify common English word classes and Penn Treebank tags.
• Explain why POS tagging is a sequence-labelling problem.
• Describe the Markov assumption.
• Distinguish a Markov Chain from a Hidden Markov Model.
• Explain how an HMM represents POS tagging.

Big Picture #

flowchart TD
    A["Word Sequence"] --> B["Use Context"]
    B --> C["Infer Hidden Tags"]
    C --> D["Tagged Sequence"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6

1. What Is Part-of-Speech Tagging? ☆ #

A part of speech describes the grammatical role played by a word in a sentence. POS tagging assigns one tag to every word in a sequence.

Statistical, ML and Neural Models of POS Tagging

Statistical, ML and Neural Models of POS Tagging #

‘HMM Inference: Forward and Viterbi Algorithms’ covers the portion:

• Forward Algorithm
• Viterbi Algorithm
• HMM inference for POS tagging

The complete Topic also includes:

• Maximum Entropy Markov Models
• Bidirectionality
• Neural-network models for POS tagging

HMM Inference: Forward and Viterbi Algorithms #

Hidden Markov Models create two closely related inference problems:

• Likelihood: How probable is an observed sequence under the model?
• Decoding: Which hidden-state sequence most probably generated the observations?

The Forward Algorithm solves the likelihood problem, while the Viterbi Algorithm solves the decoding problem. Both use dynamic programming and a trellis, but they combine paths differently.