First, I've read some of the author's published short fiction, and I consider it quite good and tonally consistent, which is sometimes hard to achieve when starting out. Novels are a different beast than short stories, but it's for novels that agents are usually found; short fiction is almost never (but not never) represented unless it's a collection.
In this case, the author feels that work is a cross-genre work; that it defies easy categorization. The author's inclination at this point is to try to find an agent first, because an agent will better be able to market a book whose market is not clear to the author. We were at the store at the time that the author mentioned this, and they were on break, so there was no in depth discussion.
In passing, I made my usual face at a suggestion that strikes me as not quite right (and I apologize if this was either intimidating or seemed ill-tempered; I was thinking, and my thinking face is often … well. Ask andpuff). I suggested that if there was no easily identifiable market for the book, it would be a difficult sell to agents. But on reflection, that's not the whole of the difficulty.
I would, in this author's position, do a couple of things. First: I would ask my first readers for two types of input. The first, of course, about the work itself. But the second, about identifying books that are similar in nature or tone that they've read before. It's often very difficult to be objective about one's first novel, especially when one isn't thinking about a market when one starts writing it. I know of where I speak <rueful g>.
The comment I made in the store about agents stems from a couple of things.
Allow me to digress at this point. Agents start out the same way that any of us do. Green, and underpaid. They often "apprentice" to other agents before they build up a client list and head out on their own. Sometimes they don't head out on their own, but they still have to build that list; in that, they're not unlike editors. An agent's clout -- like an editor's -- depends on the strength of her list. The more bestsellers, or sure sellers they have on that list, the more clout they have. When they're just starting out, they start out with new authors, and they have a lot of hope, idealism and belief. They also, like the rest of us, have bills to pay. As they get better at what they do, they often find they have a particular strength or set of strengths in recognizing a particular type of talent/book, and this is why many agents end up specializing in specific genres. But a new agent hasn't got a lot of clout, and therefore, they can't make editors sit up, take notice, or return phone calls at blinding speeds. They can submit novels to publishing houses that will not consider unagented submissions, but if that's all they can do for you, it's a risk that you take.
You pay the agent. The agent is your employee. This is true. But it is also true that a top notch agent will often pass on authors with previous track records or publishing histories. For an agent, it's the early books that are often the most work; there are always exceptional stories, but for the most part, the first novels don't earn large advances, and the agent has to do more work to place the books. More work for less money. As an agent gathers experience with various editors, and a much more certain sense of the editorial tastes of the editors they've successfully dealt with, they begin to lack time. If your book is something that's already going to be a hard sell, unless they love it to death (and this does happen), they might feel they have to pass on it.
There are so many books and articles and internet information sources about what an agent is supposed to do for you that I won't attempt to summarize them all here. What I will say is this: The Agent is in business for herself, just as the publisher is in business for themselves. The bottom line drives a business. There are always compromises that are made, and sometimes, some of those will be about you. Knowing the market for your book is important -- both for you and your agent. What an agent will bring to the table is contacts. What an agent will also bring to the table is prior history. If you can sell your first novel without an agent, there's demonstrably a market for it. The very best scenario is that you do your agent research, find a house for your book, and then call the agent of your choice and ask them if they would care to read and negotiate the offer for the book.
This is the end of that digression. The reason that you want to know your market, in as much as you can, the reason that you want to be able to say "this is like Nina Kiriki Hoffman or Charles de Lint, or etc.," is not actually for the sake of getting an agent to take the book on. It's because you need that information in order to set about choosing an agent who's proven themselves in their handling of books that are to some approximation similar to yours.
There are many agents who specialize in sub-genres. Some are very good with mysteries, some are good with SF/F, some are good with romance; some are good with all of these things, and some are actively bad with some of them. As in, tell an NYT bestseller that her books are crap, bad.
When you're trying to find an agent, you want one who knows your field, and who can handle -- who has a proven track record handling -- the type of books that you write.
I realize that, from the perspective of many years, this is far easier for me to say than seems reasonable, but it's not just from my experience that I speak; a lot of writers will talk privately about their (often former) agents and their incompatibilities with them. It's actively hard to find an agent who is good if you don't know what it is you want them to sell. Or rather, it's hard to evaluate an agent's worth to your career in the long term without a sense of at least that.
If you can't look at your book and say it's like another published author's work, and if it's cross-genre, make a column list for each genre, and then bullet those things in the work that fit each genre.
If, for instance, the work is a contemporary work that has fantasy elements, there are many authors who do that successfully, and sell. There has to be a reason why your book isn't one of those. If it's a romance/fantasy cross, that's becoming a genre of its own. I can't, offhand, think of a cross-genre book that would have difficulty being presented in a marketable way unless it's a Western/something cross <g>. If it's contemporary in tone, but with subtle supernatural elements that are entirely emotional and relevant to the interior character, think SHADOW OF ASHLAND. If it's a sports book with ghosts, think SHOELESS JOE. Etc.
But do as much of this work as you can, because the pay-off will then be finding the right agent for the work.