Linear SVM : Example2 - Machine Learning

Dataset

Point	x1	x2	Class
A	1	1	-1
B	2	2	-1
C	4	4	+1
D	5	5	+1

Here:

-1 = Class A
+1 = Class B

Step 1: General Hyperplane Equation

Linear SVM separates classes using a straight line called a hyperplane.

The equation is:

w1x1 + w2x2 + b = 0

Where:

w1, w2 = weights
x1, x2 = input features
b = bias

Step 2: Find Closest Opposite-Class Points

Class -1 points:

A(1,1), B(2,2)

Class +1 points:

C(4,4), D(5,5)

Now calculate distances between opposite-class points.

Distance formula:

Distance = √((x2 - x1)² + (y2 - y1)²)

Distance between B(2,2) and C(4,4)

Distance = √((4 - 2)² + (4 - 2)²)

Distance = √(2² + 2²)

Distance = √(4 + 4)

Distance = √8 = 2.828

Distance between A(1,1) and C(4,4)

Distance = √((4 - 1)² + (4 - 1)²)

Distance = √(3² + 3²)

Distance = √18 = 4.243

Distance between B(2,2) and D(5,5)

Distance = √((5 - 2)² + (5 - 2)²)

Distance = √(3² + 3²)

Distance = √18 = 4.243

Smallest distance is between:

B(2,2) and C(4,4)

So these two points become the main boundary-deciding points.

Step 3: Find the Midpoint

The decision boundary should lie between the closest opposite-class points.

Closest points:

B(2,2) and C(4,4)

Midpoint formula:

Midpoint = ((x1 + x2)/2, (y1 + y2)/2)

Midpoint = ((2 + 4)/2, (2 + 4)/2)

Midpoint = (3,3)

So the decision boundary passes through:

(3,3)

Step 4: Find Weight Vector

The line joining B and C has direction:

C - B = (4 - 2, 4 - 2)

= (2,2)

The SVM hyperplane is perpendicular to the line joining the closest opposite-class points.

So the weight vector is in the same direction as:

(2,2)

We can simplify it as:

w = (1,1)

So:

w1 = 1
w2 = 1

Step 5: Find Bias Value

Hyperplane equation:

w1x1 + w2x2 + b = 0

Substitute:

w1 = 1
w2 = 1

So:

x1 + x2 + b = 0

The boundary passes through midpoint (3,3).

Substitute:

3 + 3 + b = 0

6 + b = 0

b = -6

So the final equation is:

x1 + x2 - 6 = 0

Step 6: Final Decision Boundary

The final hyperplane is:

x1 + x2 - 6 = 0

x1 + x2 = 6

Prediction function:

f(x) = x1 + x2 - 6

Decision rule:

If f(x) > 0 → Class +1
If f(x) < 0 → Class -1

Step 7: Check Each Training Point

Point A(1,1)

f(x) = 1 + 1 - 6

f(x) = -4

Since value is negative:

Prediction = -1

Correct.

Point B(2,2)

f(x) = 2 + 2 - 6

f(x) = -2

Since value is negative:

Prediction = -1

Correct.

Point C(4,4)

f(x) = 4 + 4 - 6

f(x) = 2

Since value is positive:

Prediction = +1

Correct.

Point D(5,5)

f(x) = 5 + 5 - 6

f(x) = 4

Since value is positive:

Prediction = +1

Correct.

Step 8: Find Support Vectors

Support vectors are the closest points to the decision boundary.

Distance from point to line:

Distance = |w1x1 + w2x2 + b| / √(w1² + w2²)

Here:

w1 = 1
w2 = 1
b = -6

So:

Distance = |x1 + x2 - 6| / √(1² + 1²)

Distance = |x1 + x2 - 6| / √2

Distance of A(1,1)

Distance = |1 + 1 - 6| / √2

= 4 / √2

= 2.828

Distance of B(2,2)

Distance = |2 + 2 - 6| / √2

= 2 / √2

= 1.414

Distance of C(4,4)

Distance = |4 + 4 - 6| / √2

= 2 / √2

= 1.414

Distance of D(5,5)

Distance = |5 + 5 - 6| / √2

= 4 / √2

= 2.828

Minimum distance from the boundary is:

1.414

Points with minimum distance are:

B(2,2) and C(4,4)

Therefore:

Support Vectors = B(2,2), C(4,4)

Step 9: Margin Calculation

Margin width formula:

Margin Width = 2 / ||w||

Where:

||w|| = √(w1² + w2²)

Here:

w = (1,1)

So:

||w|| = √(1² + 1²)

||w|| = √2

||w|| = 1.414

Therefore:

Margin Width = 2 / 1.414

Margin Width = 1.414

So total margin width is:

1.414 units

Step 10: Predict New Data Points

New Point P(6,3)

f(x) = x1 + x2 - 6

f(6,3) = 6 + 3 - 6

= 3

Since value is positive:

Prediction = +1

New Point Q(1,2)

f(x) = x1 + x2 - 6

f(1,2) = 1 + 2 - 6

= -3

Since value is negative:

Prediction = -1

New Point R(3,3)

f(x) = 3 + 3 - 6

= 0

Since value is zero:

Point lies exactly on the decision boundary.

Final Result

Dataset:
A(1,1) → -1
B(2,2) → -1
C(4,4) → +1
D(5,5) → +1

Decision Boundary:
x1 + x2 - 6 = 0

Prediction Function:
f(x) = x1 + x2 - 6

Support Vectors:
B(2,2), C(4,4)

Weight Vector:
w = (1,1)

Bias:
b = -6

Margin Width:
1.414 units

Important Points

Linear SVM finds the best separating hyperplane.
The closest opposite-class points help decide the boundary.
The decision boundary passes through the middle region between closest classes.
Support vectors are closest points to the hyperplane.
In this example, B(2,2) and C(4,4) are support vectors.
The prediction is based on the sign of f(x).
Positive value gives class +1.
Negative value gives class -1.
Zero means the point lies exactly on the boundary.