Lecture 1 — Beginner tutorial

Function approximation by hand

Lecture 1 Tutorial — Function Approximation (Beginner)

Course: Mathematical Foundations of Machine Learning (Prof. Prathosh A.P.)
Based on: Lec 01 — Overview of Function Approximation

What you will do

1. Create a tiny dataset (like observations in the lecture)
2. Make an initial guess for the unknown function (a model family)
3. Refine that guess using data
4. See overfitting (memorizing data)
5. See that a flexible model can even fit random labels
6. Turn a small “image” into a vector (X-ray idea from class)

How to use this notebook

• Read each markdown cell (the text explanations).
• Run the code cell right below it (Shift + Enter).
• Do not skip cells — later cells need earlier ones.

You do not need to know ML already. If a line of code is new, the text above it explains what it does.

Step 0 — Install / import tools

We need two libraries:

Library	Job
numpy	Work with numbers and arrays (lists of numbers)
matplotlib	Draw plots

If the next cell fails with ModuleNotFoundError, run this once in a cell:

%pip install numpy matplotlib

Then run the import cell again.

import numpy as np
import matplotlib.pyplot as plt

print("Imports OK. numpy version:", np.__version__)

Step 1 — Remember the lecture idea

In class we said:

We have pairs (x, y). We assume there is an unknown function f such that y ≈ f(x).
Goal: estimate f from data.

In this notebook we will pretend the true function is a sine wave:

\[ f(x) = \sin(2\pi x) \]

In real ML we do not know f. Here we keep it secret from the “learner”, but we plot it as a dashed line so you can see the truth.

def true_f(x):
    """The unknown reality. In real problems we do NOT know this."""
    return np.sin(2 * np.pi * x)

# Try it on a few numbers
print("f(0.0) =", true_f(0.0))
print("f(0.25) =", true_f(0.25))  # should be about 1
print("f(0.5) =", true_f(0.5))    # should be about 0

Step 2 — Create the dataset D

Lecture notation:

\[ \mathcal{D} = \{(x_1,y_1),\ldots,(x_n,y_n)\} \]

We will: 1. Pick random x values between 0 and 1 2. Compute y = true_f(x) 3. Add a little noise (real measurements are never perfect)

New words - seed: makes “random” numbers repeatable (same result every run) - noise: small random error added to y

def make_dataset(n=15, noise=0.15, seed=0):
    """Make n noisy observations from true_f."""
    rng = np.random.default_rng(seed)          # random number generator
    x = np.sort(rng.uniform(0, 1, size=n))      # n random x in [0,1], sorted
    y = true_f(x) + rng.normal(0, noise, size=n)  # true value + noise
    return x, y

x_train, y_train = make_dataset()

print("Number of points:", len(x_train))
print("First 5 x values:", x_train[:5])
print("First 5 y values:", y_train[:5])

Plot the data

• Red dots = what we observed (D)
• Black dashed curve = true f (unknown in real life)

Notice: the dots are near the curve, but not exactly on it, because of noise.

x_grid = np.linspace(0, 1, 400)  # many x values for a smooth curve

plt.figure(figsize=(7, 4))
plt.plot(x_grid, true_f(x_grid), "k--", label="true f (unknown in real ML)")
plt.scatter(x_train, y_train, c="red", label="data D", zorder=3)
plt.xlabel("x")
plt.ylabel("y")
plt.title("Observations vs the unknown function")
plt.legend()
plt.show()

Step 3 — Initial guess + refine (the lecture philosophy)

From class:

1. Start with a guess for f (choose a model family)
2. Refine using data D

Our simple guess family: polynomials

Degree	Example shape
1	straight line: \(a + bx\)
2	parabola: \(a + bx + cx^2\)
3	cubic: \(a + bx + cx^2 + dx^3\)
high	can wiggle a lot

np.polyfit = refine the guess (find best coefficients for that degree)
np.polyval = use the fitted polynomial to predict y at new x

def fit_and_predict(x_train, y_train, x_query, degree):
    """
    Guess family: polynomials of this degree.
    Refine: least squares fit to the data.
    """
    coeffs = np.polyfit(x_train, y_train, degree)  # learn coefficients from D
    y_hat = np.polyval(coeffs, x_query)            # predict on new x
    return y_hat

# Quick test: fit a line (degree 1) and predict at x=0.5
y_at_half = fit_and_predict(x_train, y_train, 0.5, degree=1)
print("Degree-1 model prediction at x=0.5:", float(y_at_half))

Step 4 — Compare three guesses (underfit → OK → overfit)

We try three model families on the same data:

Degree	What usually happens
1	Too simple → underfitting (misses the curve)
3	Often a reasonable compromise
12	Too flexible → overfitting (chases noise, wild wiggles)

Overfitting matches what the lecture said: defining the function only to match observed points does not help on new x.

degrees = [1, 3, 12]

fig, axes = plt.subplots(1, 3, figsize=(12, 3.5), sharey=True)

for ax, deg in zip(axes, degrees):
    y_hat = fit_and_predict(x_train, y_train, x_grid, degree=deg)

    ax.plot(x_grid, true_f(x_grid), "k--", lw=1.5, label="true f")
    ax.plot(x_grid, y_hat, lw=2, label=f"degree {deg}")
    ax.scatter(x_train, y_train, c="red", zorder=3, label="data D")
    ax.set_title(f"Guess family: poly degree {deg}")
    ax.set_xlabel("x")
    ax.legend(fontsize=8)

axes[0].set_ylabel("y")
fig.suptitle("Same data D — different initial guesses (model families)")
fig.tight_layout()
plt.show()

What to notice

• Left plot: line cannot bend with the sine wave → bad guess family.
• Middle: closer to truth → useful model (still “wrong”, but useful).
• Right: passes near every red point, but oscillates → memorizing noise.

Lecture proverb: All models are wrong, but some are useful.

Step 5 — Random labels thought experiment

In class: take MNIST digits, shuffle the labels randomly, and you can still fit a powerful model.

Here we do a tiny version:

1. Meaningful labels: 1 if true_f(x) > 0, else 0
2. Random labels: coin-flip 0/1 with no meaning

We use a very flexible polynomial (degree 18).
Point: fitting the training points does not mean the labels make sense.

rng = np.random.default_rng(1)

x = np.linspace(0, 1, 20)
y_meaningful = (true_f(x) > 0).astype(float)          # structured 0/1 labels
y_random = rng.integers(0, 2, size=len(x)).astype(float)  # nonsense labels

print("Meaningful labels:", y_meaningful)
print("Random labels:    ", y_random)

y_hat_meaningful = fit_and_predict(x, y_meaningful, x_grid, degree=18)
y_hat_random = fit_and_predict(x, y_random, x_grid, degree=18)

fig, axes = plt.subplots(1, 2, figsize=(10, 3.5), sharey=True)

axes[0].scatter(x, y_meaningful, c="C0", label="meaningful labels")
axes[0].plot(x_grid, y_hat_meaningful, c="C0")
axes[0].set_title("Fit meaningful labels")
axes[0].set_xlabel("x")
axes[0].legend(fontsize=8)

axes[1].scatter(x, y_random, c="C1", label="random labels")
axes[1].plot(x_grid, y_hat_random, c="C1")
axes[1].set_title("Fit random labels (still possible!)")
axes[1].set_xlabel("x")
axes[1].legend(fontsize=8)

for ax in axes:
    ax.set_ylim(-0.2, 1.2)

fig.suptitle("Flexible models can memorize nonsense")
fig.tight_layout()
plt.show()

This is why the course moves to a probabilistic view later:
we need ideas beyond “draw a curve through the points” (generalization, uncertainty, distributions).

Step 6 — An image is just a vector (X-ray idea)

In lecture: a \(P \times Q\) image → stack columns → one long vector in \(\mathbb{R}^{PQ}\).

We use a tiny fake “image” (4 rows × 5 columns) so you can see every number.

P, Q = 4, 5
img = np.arange(P * Q).reshape(P, Q)  # numbers 0..19 arranged as a grid

print("Image shape (rows, cols):", img.shape)
print("Image (matrix):")
print(img)

# order="F" means Fortran-order = stack COLUMN by column (as in lecture)
vec = img.reshape(-1, order="F")

print("As a vector in R^{}:".format(P * Q))
print(vec)
print("Vector length:", vec.shape[0])

plt.figure(figsize=(3, 3))
plt.imshow(img, cmap="viridis")
plt.title("Tiny fake image")
plt.colorbar()
plt.show()

A real X-ray is the same idea with a much bigger \(P\) and \(Q\)
(for example \(512 \times 512\) → a vector of length \(262\,144\)).

Then ML learns a function:

\[ f : \mathbb{R}^{d} \rightarrow \{0,1\} \]

(image vector → diseased / not diseased).

Step 7 — Mini exercises (try yourself)

Change numbers and re-run. There is no single “correct” answer — watch what happens.

# Exercise A: more noise → harder to see the true curve
x2, y2 = make_dataset(n=15, noise=0.4, seed=0)  # try noise=0.05 vs 0.4

plt.figure(figsize=(7, 4))
plt.plot(x_grid, true_f(x_grid), "k--", label="true f")
plt.scatter(x2, y2, c="red", label="noisier data")
plt.legend()
plt.title("Exercise A: change the noise")
plt.show()

# Exercise B: change the degree yourself
my_degree = 5  # try 0, 2, 8, 14 ...

y_hat = fit_and_predict(x_train, y_train, x_grid, degree=my_degree)

plt.figure(figsize=(7, 4))
plt.plot(x_grid, true_f(x_grid), "k--", label="true f")
plt.plot(x_grid, y_hat, label=f"degree {my_degree}")
plt.scatter(x_train, y_train, c="red", label="data D", zorder=3)
plt.legend()
plt.title("Exercise B: pick your own model family")
plt.show()

Recap (map back to the lecture)

Lecture phrase	What we did
Dataset \(\mathcal{D}\)	red points (x_train, y_train)
Unknown \(f\)	true_f (sine)
Initial guess / model family	polynomial degree
Refine using data	np.polyfit
Overfitting	very high degree chasing noise
Random labels	flexible model fits nonsense
Image as vector	reshape matrix → long list of numbers
Why probability next?	fitting alone is not enough

Next lectures: probability theory — sample space, random variables, distributions.

When you finish, send the Lecture 2 link and we will make the next notes + notebook.