I recognize that it's difficult to ask your beta readers to be more specific, because you already feeling in debt to them. After all they're doing you a favour. I'll mention a few points:
What you'd want is to make them more aware before reading. If they are avid readers, they are probably sensible enough to notice mistakes or problems, but they need to be in the right mind set to do that.
Generally, if I'm reading something for my own pleasure, I tend to skip over the nitty-gritty details and when I'm finished I do have only general impressions. When I'm reading something with the goal of giving feedback, I tend to be more alert.
So, you could tell them something of the like:
"I really need feedback on this one, so feel free to tell me anything you stumble upon"
"I'm not sure about the main character, can you read and tell me if it works?"
"So, here's the last chapter. Can you tell me if it flows nicely? Also don't be afraid to tell me if it's too boring to read."
What you would be really doing with this questions is leading them to focus on specific issues. It's efficiency may vary, since there's no guarantee that they will remember what you asked when they'll start reading, but it's worth a shot.
On the other hand, the raw feedback that you are receiving right now is important too, since it mimicks the mindset that a general audience would have.
As I stated in the sentence above: raw feedback from a "reader" point of view is really important. So, what you may want is to add to your beta-readers some writer. Writers are more in the "editing loop" and will be used to get in a more analytical mindset when reading.
I know, you just mentioned that you don't have a writing group yet, but it's not too late to find one. You could, alternatively, find some writer up to the task.
Ask for clarification
DPT already gave good examples of this. Basically, you want to have some follow-up questions ready. When a vague feedback comes in, you want to ask the why&what questions (without appearing defensive or confrontational, of course).
I understand the character's motivations, but I don't really connect, don't know why.
"I see, maybe its the character personality?"
"Was it boring following a character you couldn't connect with?"
"Do you think it's a matter of personal tastes, or there's something wrong with the novel?"
or, in the other example:
something's off, can't really put my finger on what
"Ah, I'm sorry. I hope it wasn't boring/slow/uninteresting to read."
"Hm, I see. Do you think it may be a problem of pacing?"
"Hm, I see. Do you find the narration a bit off?"
So basically you would be trying to make them be more specific. Chances are that someone will be able to answer.
More well-inclined readers will be willing to go back on their steps and finding what gave them that impression. Some others won't be able to give reasons behind their gut feelings, but that is what gut feelings are, by definition.