NN and Neural Language Modelling

Neural Networks and Neural Language Modelling #

Neural networks learn useful representations and nonlinear relationships directly from data. In language modelling, they replace discrete N-gram identities with learned word embeddings and use these representations to predict the next word.

Learning Objectives #

• Explain the computation performed by a neural unit.
• Describe why hidden layers and nonlinear activations are needed.
• Explain how feed-forward networks support NLP classification.
• Trace the flow through a feed-forward neural language model.
• Compare N-gram and neural language models.

Big Picture #

flowchart TD
    A["Context Words"] --> B["One-hot Inputs"]
    B --> C["Embedding Lookup"]
    C --> D["Combined Context"]
    D --> E["Hidden Layer"]
    E --> F["Softmax"]
    F --> G["Next-word Probabilities"]

    style A fill:#E1F5FE
    style B fill:#C8E6C9
    style C fill:#FFF9C4
    style D fill:#EDE7F6
    style E fill:#E1F5FE
    style F fill:#C8E6C9
    style G fill:#FFF9C4

1. Neural Network Units ☆ #

A neural unit receives input values, multiplies them by learned weights, adds a bias, and applies an activation function.