I think I am in total agreement with you, on all that. So, I've long been a fan of a discovery framework from the librarian Nancy Pearl. I came up with a similar framework when first working in book discovery, but she describes it better. haha. In short, readers enter books through several possible doorways - language, characters, story, and setting. Different people will enter into each book in different ways, but we do tend to have preferences. Some people may love how a book is written. Another may say the love who it's about. Or that they love spending time in the era of Jane Eyre.

The way I used to describe it is that, in order for a reader to find the best book of all time for them, it has to a) be about something they find interesting or compelling, b) it has to happen to characters they like or find interesting, and C) it has to be told through a language they love or find effortless. While you can be off on those and still have a good book, the chances of it being your favorite book are going to be lower if those are poorly aligned. And the chances of you even finishing the book are much, much lower if they're all wildly off for you.

From that framework, discovery is a matter of understanding how a reader enters books, what their ranges are (including how different and new something should be, not just the same), and then populating their bookshelf with books that are ideally suited for their doorways, and their level of adventurous reading.

There are a lot of characteristics about characters, story, language and setting that we can understand using computers. Not perfect, but much more than we currently can with social data. For example, all of those things change throughout a book. A character can appear more at the beginning, and less in the end. Or disappear from a series all together (The Wheel of Time had a substantial issue with this in the middle books, where the main characters essentially disappeared). Vampires can be combative vampires in chapter one, and romantic in chapter 23. Maybe the eb and flow of pacing is as important to one reader as the language?

There is no realistic way to get that granular of metadata for 99% of books using social recommendations. And I guess that's the point. There's a lot of potential that has never been fully utilized with these tools.