Who are the gatekeepers? When you send a manuscript to an agent or publisher first it has to get past a reader. Someone who will look it over and decide whether it is worthy of higher consideration. The people who are giving this task are generally secretaries, PA’s, younger writers looking to make a bit of money and interns. They have no great expertise in identifying a good novel/script and have their own personal preferences and prejudices.
If they come across something so amazing that even they can spot its value then they can hand it over to someone higher up the chain. Obviously there will be various manuscripts with potential that will be overlooked, but since there are plenty of writers with potential already on the books, it’s considered a risk worth taking.
Often it is seen as a way to gets a more junior member of staff up to speed on the standard of work out there. Everything does get read eventually but it soon becomes apparent that most of it is not very good and the enthusiasm with which a reader will approach work sent in will fade over time. To be honest the pitch in the query letter is just as much aimed at that slightly bored person who spends most of the day answering phones and making coffee as it is at the agent.
So having a well written, interesting, potentially great manuscript isn't enough to guarantee consideration by the person or company you sent it to. That's just the way things are. The person who will actually read it is probably around the same level of expertise as you (i.e. not very expert at all). In fact since they are in ‘the business’ they might have an elevated idea of their own position, making them a little more arrogant in their judgements. Whether it is a struggling writer who resents the vampire fantasies dominating the marketplace, preventing his magnus opus from selling, who roundly rejects your urban fantasy, or a university graduates with a Masters in classic romantic literature who considers your sci-fi soap opera to be ridiculous, there are numerous reasons for a story to get rejected that have nothing to do with the quality of the story itself.
You have to remember that even the novels that get supported all the way into print, the vast majority are massive failures commercially. The people who make the decisions about what the public want to read are wrong most of the time. What keeps the business afloat is anonymous low grade crap and the occasional hit that makes millions. Even the people at the top don't really know what will sell. Yes they were convinced book A was going to be a hit, and it was. But they were also convinced books X,Y and Z were going to be hits, and they weren’t.
So finding your way onto an agent's list or into a publisher’s catalogue is a bit of a lottery. The person whose hands your writing end up in has to be inclined towards whatever your style happens to be, they have to be in the right mood, and they have to have the ability to see potential if it's there. And then they pass it onto someone else who has to go through the whole process again. And so on and so on.
That's not to say there’s no point. There are lots of places to send your work, somebody will certainly read at least a bit of it, and if it ends up in the right hands, the fates might smile kindly on you. But it would certainly help if you make it as good as you possibly can. It would be a terrible shame if you managed to get in the right place at the right time and then blew it because you didn't apply enough spit and polish.