What is a Naive Bayes Classifier?

Bayes Classifier is a “probabilistic” classification algorithm for machine learning.

A probabilistic model gives the probability distribution of an instance over a set of classes as opposed to a deterministic model that only output whether an instance belongs to a positive class or a negative class.

Let us assume we have two classes A & B for the classifier. A deterministic model would output an instance belonging to either A or B, but in the case of a probabilistic model, it would output an instance A having 80% probability belonging to class A and 60% probability belonging to class B.

This technique is based on the Bayes’ Theorem and on a naive assumption that all the features are independent of one another, hence called a Naive Bayes Classifier.

Bayes’ theorem describes the probability of an event, based on prior knowledge of conditions that might be related to the event. It simply follows the axioms of the conditional probability that is defined as the likelihood of an event occurring, based on the occurrence of previous events.

The formula for Bayes Theorem is as follows:

Let us say, you’re planning to buy a pet that should be kid-friendly, hypoallergic and can be trained to do some tricks.

For this, you have the following sample data of 145 animals consisting of 60 dogs, 55 birds, and 30 snakes.

Now, as per the Naive Bayes Algorithm, you have to calculate the probability of each of the features that you want against the set of animals that you have. e.g., We have to calculate the probability of animal being Kid-friendly, hypoallergic and able to do tricks, given the animal is a dog, which is nothing but P(Kid Friendly, Hypoallergic, Can do tricks | Dog).

Let’s calculate P(Kid Friendly| Dog): The probability of a dog being kid-friendly.

Our formula, in this case, would be,

P(Kid Friendly| Dog) = P(Dog|Kid Friendly). P(Kid Friendly) / P(Dog)

P(Kid Friendly) = Total count of Kid Friendly animals / Total count of animals = 45/145 = 0.3

Similarly,

P(Dog) = 60/145 = 0.4

P(Dog| Kid Friendly) = 30/45 = 0.6

P(Kid Friendly | Dog) = (0.6)(0.3)/(0.4) = 0.45.

In a similar fashion, we can calculate,

P(Hypoallergic| Dog) = 0.3

P(Can do tricks | Dog) = 0.5

So now, P(Kid Friendly, Hypoallergic, Can do tricks | Dog) = P(Kid Friendly|Dog) . P (Hypoallergic|Dog) . P(Can do tricks | Dog)

= (0.45)(0.3)(0.5) = 0.0675 =6%

Similarly, we can do the same calculation for Birds and Snakes.

P(Kid Friendly, Hypoallergic, Can do tricks | Bird) = 0.01 = 1%

P(Kid Friendly, Hypoallergic, Can do tricks | Snake) =0%

We can see that the Dog here has the highest probability of satisfying all our conditions (6%). So in this case, Dog would be a winner for our Naive Bayes classifier.

The advantage of a Naive Bayes Classifier is that it’s simple and easy to understand. It also needs less training data and is highly scalable.

There are various flavors of Naive Bayes that can work with discrete as well as continuous variables.

The main disadvantage is that it is based on a strong assumption that two features are independent of one another.

Conclusion: The Naive Bayes Technique is one of the most popular classification algorithms due to its speed and simplicity. It is mostly used in text classification and when the data is huge. There are various flavors of this classifier, namely, Gaussian, multinomial and Bernoulli, which I hope to cover in the next article.

Related posts:

We have always dreamed of traveling the world and decided to seize the moment for our “adventure-moon.” We’ll be sharing highlights as we explore Singapore, Vietnam, Cambodia, Laos, Thailand, Japan…

In the canon of true-crime cases, one tragedy stands out as a legendary, a perfect storm of murder and apathy: on March 13, 1964, bar manager Catherine “Kitty” Genovese was raped and stabbed to death…

In a world dominated by moral relativism, “what is truth?” According to Patrick Madrid, “the answer, from a Catholic perspective, is Jesus Christ… ‘I am the way, the truth, and the life.’” Morality…