NLP - Vector Semantics and Embedding #

Lexical semantics and word meaning.
Lemmas, senses, and semantic relationships.
Distributional hypothesis.
Vector semantics and word embeddings.
Document and word vectors.
Dot product and cosine similarity.
Term Frequency–Inverse Document Frequency (TF-IDF).
Prediction-based word embeddings and self-supervision.
Word2Vec using Skip-gram with Negative Sampling and CBOW.
Embedding matrices, context-window choices, analogies, visualisation, and bias.
GloVe and global word–word co-occurrence statistics.

Learning Objectives #

Explain lexical semantics and distinguish a lemma from a word sense.
Compare synonymy, similarity, relatedness, antonymy, and connotation.
Explain the distributional hypothesis and its role in modelling meaning.
Describe how words and documents can be represented as vectors.
Construct and interpret word–document and word–context matrices.
Calculate dot product and cosine similarity between vectors.
Explain why raw word frequency can be misleading.
Calculate TF, IDF, and TF-IDF weights.
Explain how Word2Vec learns embeddings from a prediction task.
Construct positive and negative Skip-gram training pairs.
Explain how sigmoid, negative sampling, and gradient descent train SGNS.
Compare Skip-gram with CBOW.
Explain how context-window size affects the relationships captured.
Interpret word analogies and two-dimensional embedding visualisations.
Explain how GloVe combines global counts with learned dense vectors.
Recognise how social biases can be encoded in word embeddings.

Big Picture #

flowchart TD
    A[Words and Documents] --> B[Observe Their Context]
    B --> C[Represent Them as Vectors]
    C --> D[Compare Vector Directions]
    D --> E[Estimate Semantic Similarity]
    E --> F[Search, Classify, Retrieve or Generate]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9

1. Lexical Semantics ☆ #

Lexical semantics is the linguistic study of word meaning and the relationships between word meanings.

NN and Neural Language Modelling

AI, ML

Natural Language Processing, NLP, Neural Networks, Feed-Forward Networks, Neural Language Models, Word Embeddings

Neural Networks and Neural Language Modelling #

Neural networks learn useful representations and nonlinear relationships directly from data. In language modelling, they replace discrete N-gram identities with learned word embeddings and use these representations to predict the next word.

Learning Objectives #

Explain the computation performed by a neural unit.
Describe why hidden layers and nonlinear activations are needed.
Explain how feed-forward networks support NLP classification.
Trace the flow through a feed-forward neural language model.
Compare N-gram and neural language models.

Big Picture #

flowchart TD
    A["Context Words"] --> B["One-hot Inputs"]
    B --> C["Embedding Lookup"]
    C --> D["Combined Context"]
    D --> E["Hidden Layer"]
    E --> F["Softmax"]
    F --> G["Next-word Probabilities"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9
    style G fill:#FFF9C4

1. Neural Network Units ☆ #

A neural unit receives input values, multiplies them by learned weights, adds a bias, and applies an activation function.