Thanks for the quick reply. That was a very good detailed explanation and it solved the problem.
Here's what I did (just for the guys who has similar problem)
I have inherited NotificationObject (from Prism) in my viewmodel.
The NotificationObject has INotifyCollectionChanged implemented so we dont have to worry about it!
and then on Collection's Setter
this.RaisePropertyChanged(() => this.Books);