How to gain an information-edge by asking better questions
💌 The Roundup // 004
The Jungle Gym is a monthly newsletter full of ideas and resources to help you think clearer and work smarter.
🤝 When I meet great people, I often add them to this list to make it easier to stay in touch. If, for any reason, you'd prefer not to receive future issues, don't hesitate to hit the unsubscribe button below.
🔍 If you're not getting this email consistently, try checking your spam folder and marking this address as 'not spam.' If the newsletter isn't there, check your Promotions tab.
💌 If you're enjoying what you're reading, spread the word via email or Twitter.
📝 And, if you aren't yet a subscriber and want to sign up for future issues, just enter your email:
How to gain an information-edge by asking better questions
The quality of decisions you make directly depends on the quality of information you use to inform them. While some of the data you need to make those decisions may exist in print, that information tends to be public. That means it won’t give you much of an advantage in competitive domains like hiring, entrepreneurship, or investing. To gain a true information edge you’ll need to get your information from other people. To illustrate, consider the following typical Silicon Valley scenarios:
A job candidate wants to learn about a company’s recent performance so she can decide whether to accept an offer there.
A hiring manager needs to know whether a candidate is qualified for a role before he offers her a position.
A founder tries to learn best practices in a new function to assess whether a direct report is doing a good job
An investor wants to understand the dynamics of a market to determine whether to invest in a startup
In each of these situations, the information the decision-maker needs is locked in the brains of other people.
How does one access that information? By asking good questions.
What makes for a good question?
Since you were a baby you’ve been asking questions as a way to understand the world around you. Why did she say that? How do you spell Mississippi? Where do we go when we die? Questions are so second nature to us, that few have put the time in to learning how to ask good ones.
The quality of a question can largely be judged by the usefulness of the response it elicits. So, perhaps it’s best to start with some principles that make for a good response, and how you can phrase a question to increase your chances of getting one.
Characteristics of Useful Responses
On Topic
You want to make it as easy as possible for your respondents to surface the information you’re looking for in their memory. If your question is long or complicated, you’ll create too much cognitive load, and get a lower quality answer. To focus your subject ask questions that are:
Short – Avoid stuffing a question with too much upfront exposition. Often this can come in the form of qualifiers that make the question overly confusing. “How long have you been working at this company in the context of your function on this project?”
Properly framed – If you need to focus someone on a specific topic, start with a short framing statement like “I’m now going to ask you about your previous role…” that can help focus them on the topic you want to know about. With that out of the way, you can ask shorter follow up questions.
About only one thing – Asking about more than one topic in a single question will lead to confused and muddled answers as people try to gather information from multiple areas and weave it together. If you need to know about more than one thing, use two separate questions to ask.
Accurate
People have both a natural desire to tell others what they want to hear and to conceal information from those they don’t trust. If you want truthful answers, you’ll need to:
Avoid biasing your subject by indicating the kind of answer you’re looking for – All it takes is one extra word to indicate to the respondent what kind of answer you’re looking for. Even a simple question like: “How much will the market grow next year?” contains the assumption that the market will grow. Instead a question like: “Where will this market be one year from today?” will prompt the respondent for a more accurate answer.
Build rapport and trust with your subject – A subject that trusts you and likes you is more likely to divulge the information you want to know.
Narrative
When exploring a new topic, you’ll find there are lots of questions you won’t even know to ask. By getting subjects to offer longer narrative responses you’re more likely to pick up useful information that you wouldn’t have known to ask for. To get narrative responses, ask questions that:
Don’t prompt a yes/no answer – openers like “do you,” “could you,” or “can you” don’t lend themselves to elaboration.
Start with interrogatives – these are words like “why,” “when,” and “how” that will prompt longer answers.
Are followed up by asking “what else?” – instead of moving on after a question has been answered, probe deeper when you think the subject may still have more detail to share.
Of course, open-ended questions don’t always make sense. If you find yourself in a competitive situation, like a negotiation where the other party doesn’t want to reveal information, open-ended questions can leave them a way to dodge or lie by omission. In this case, yes/no questions may be more effective at getting them to open up.
You will rarely be asking single questions in isolation. More often you’ll have a sequence of questions that you’ll need to adapt on the fly based on the responses you get. Here are six common types of questions that tend to be useful for drawing out information.
Six types of good questions
Direct questions are useful when you know exactly what you want information about. These questions often start with an interrogative, contain one verb and one noun or pronoun.
Examples:
“How long have you worked in design?”
“Why did you join that company?”
“How large is the market?”
“Who is doing the most interesting research in your field?”
Control questions are ones that you know the answer to before asking. They can help you determine if the subject is lying, uninformed or not paying attention.
Example: When assessing someone’s level of expertise you might start a conversation by asking a question about the subject matter that you already know the answer to.
Repeat questions are designed to get at the same information in different ways to help determine the validity of an initial response.
Example: If a founder tells you his company is on a steep growth trajectory, you might ask him how many customers he’s signed in the past quarter, or his year-over-year revenue growth and look for inconsistencies.
Persistent questions help you get more complete answers by pulling out more details on a topic.
Example: if a candidate revealed a red flag in an interview, you might need to ask multiple questions about the incident in order to get to the truth:
“Why did you have a conflict with your previous manager?”
“What did your manager do that annoyed you most?”
“How did you handle your strained relationship with your manager?”
Summary questions allow you to feed information back to the respondent to confirm or disconfirm your interpretation of what they said.
Examples:
“If I’m getting this right, you’re saying the market is about to have explosive growth in Q1 of next year?”
“You’re saying that if I submit an application to the company, the CEO will personally look at it?”
“You really think that the CEO is about to resign?”
Non-pertinent questions aren’t asked to get information about the topics you care about. Instead, they serve other purposes like building rapport, putting the subject’s mind at ease or stalling while you decide what to ask next.
Example: If you’re doing a reference call for a candidate and find that the other person is uncomfortable answering your questions, you might try asking them about their weekend plans as an easy question to mitigate the tension.
Which of these questions you use will depend on the behavior and personality of your respondent. Here are some common types of respondents along with advice on how to handle them.
Respondent Types
Integrators will revisit their initial answer to your question until they’re satisfied that they’ve given the best possible answer.
How to handle: Since this type will often change their answers, you may want to try a repeat line of questioning to get the same information from another angle to ensure it lines up.
Dictators like to deliver definitive answers, often presenting their personal opinions as fact.
How to handle: Use a persistent line of questioning to probe the true depth of the subject’s expertise.
Commenters often give far more detail in their answers than a question requires, which can drag a line of questioning off track.
How to handle: Try framing your questions to narrow the scope of acceptable answers as much as possible.
Evaders avoid answering your questions because they may feel uncomfortable or have something to hide.
How to handle: Asking the question from another angle can make the subject more comfortable with providing an answer. If the person is shy, you may have more luck warming them up with a non-pertinent question before asking what you really want to know.
Of course, asking good questions is only half the battle. Unless you closely listen to what you hear, it will be hard to pull out useful insights.
Listening
For most of us, it’s hard to listen without getting distracted by our environment. From the buzzing of a phone to worries about what the other party thinks of us, it’s easy to lose track of the responses we’re hearing.
One technique to help focus the mind is active listening. While hearing is a passive activity, listening forces you to hear sound with deliberate intention. Some general principles of active listening:
Pay attention – not just to what the speaker is saying, but how they are saying it. Listen for pauses and words they use repeatedly. As you listen, look at the speaker directly and avoid getting distracted by your environment. While you may be tempted to think about what you want to say next, don’t pull your attention away from the speaker.
Show you’re listening – keep eye contact, open body language and a slight lean toward the speaker. Every so often, offer the speaker a signal that you’re still paying attention like a nod, a smile or short verbal affirmation. Sensing that you’re interested and attentive, the speaker will want to keep revealing information.
Provide feedback – as the speaker reveals new information, ask follow-up questions to clarify things that don’t make sense. Periodically you can ask summary questions to make sure you understand what the other person is saying, and show them you’re listening.
Here’s a quick demonstration so you can see these techniques in action:
When it comes to getting smart on new topics, perhaps the most important part of the process is making sure that you’re sitting in front of someone who actually has the knowledge you need.
Identifying Experts
When you’re new to a topic it can be challenging to determine the difference between an expert and a charlatan. Here are a few strategies that can help you find and vet potential experts:
Ladder up – You may not know an expert on a given topic, but you probably know someone who’s smarter than you on that topic. Start by having a conversation with that person to get some basics. At the end, ask them for the name of the smartest person they know on the topic. You only need to do this a few times before you’re sitting with a real expert. This strategy will also help ensure that by the time you’re sitting in front of a true expert you’re able to have a more fruitful conversation.
Vet the expert with mutual connections – Once you reach someone who seems to have legitimate expertise, try cross-checking them with mutual connections. While your mutual connections may not be able to gauge the depth of their knowledge, they can probably tell you if the person generally tells the truth.
Do some background research – In the age of Google, there’s a lot you can learn online about anyone. Do some homework to look for legitimate signals of expertise in that topic including where they’ve worked, who they know, what they’ve studied and the topics they’ve published on.
Test their knowledge –Use a mix of control, persistent and repeat questioning to test how much they know about a topic and spot inconsistencies in their answers.
Become a Conduit for Value
You’ve now got the foundations for a toolkit that will help you get useful data from anyone. To press your advantage, start thinking of yourself as a conduit for valuable information. Perhaps the expert in data science would love to know more about the latest trends in design systems, or the nutrition expert would appreciate learning about commodities markets. By becoming a broker of expertise, you’ll be able to cultivate relationships with experts. In turn, those experts will want to continue supplying you with valuable data that you can use to make better decisions on an ongoing basis.
End Notes:
A good deal of the research for this post came from this book.
Permalink for this post can be found here.
Recommended Content
The Chief of Staff role in Silicon Valley
(8 min read)
During business school, I was lucky enough to take a class taught by Andy Rachleff. Those who’ve encountered Andy will know he’s not shy about sharing his opinions. During class, the conversation turned to the role of Chief of Staff, and whether it’s a worthwhile path for building a career in Silicon Valley.
Andy’s assessment: “Chief of Staff is a bullshit title.”
Having witnessed a few folks attempt this role over the past few years, I think I’ve come to see it with a slightly more nuanced point of view.
This post, written by the former Chief of Staff at Opendoor, does a good job of describing the benefits as well as the potential drawbacks of the role:
Benefits:
Unique perspective across the company: you’ll have a view across the entire company unlike any other role except your executive’s, and may include things like joining board meetings, executive staff meetings, and supporting the vision and strategy setting for the company
Access to the executive team: you’ll help your executive manage their staff, so you’ll understand their priorities, working styles, strengths and weaknesses, and through this be able to learn from them
Ability to learn about and flex across various functions: you will likely do things like communications, recruiting, and board preparation, perhaps for the first time
Network: you’ll have more access to the network of your executive, including board members and their own personal network
Credibility and opportunity: having the trust of your executive gives you credibility in the eyes of your company and their network, and if you do a good job in the role, can lead to opportunities both within the company and externally
Potential Drawbacks:
Opportunity cost: by choosing to enter the Chief of Staff role, you’re opting out of the learning and growth along the trajectory you’re currently on for the period of time you’re in the Chief of Staff role
More isolated: moving from a traditional team setup to the Chief of Staff role means much less time with your peers and those you used to work with. Additionally, many Chiefs of Staff are the first ones to take on the role at their company, and therefore need to define for themselves who their mentors are and how they interface with the rest of the organization.
Broad not deep: you’ll likely manage several projects at once, instead of going deep on one thing. One-off urgent tasks inevitably crop up.
If you’re considering a Chief of Staff role, I’d suggest reading her post. As you do, keep in mind that:
Opendoor has been a super fast-growing company with a prestigious employer brand.
By all accounts I’ve heard, Eric Wu (the CEO she worked for) is incredibly smart and capable.
It sounds like she wasn’t expected to come in and do a bunch of admin duties for him.
I think success in a Chief of Staff role tends to be highly dependent on those three factors. For those seriously considering this path, a few more links:
Prime Advice – An entire blog written by and for Chiefs of Staff
CoS Tech Forum – A Medium publication directed at Chiefs of Staff who work in tech
Do you want to know about the Chief of Staff role? – A solid reading list
Why Books Don’t Work & Going Critical
(21 min read & 21 min read)
Think about the last non-fiction book you read. Not the one you’re reading now– the one before that. As an experiment, try summarizing the main argument. Was it easy? What about the arguments from each chapter? If you’re like almost everybody, you haven’t retained much of what you read.
That’s not necessarily your fault. While there are strategies that can improve your retention, books themselves may be a bad medium for information transfer. As Andy Matuschak puts it in his essay Why Books Don’t Work:
Books don’t work for the same reason that lectures don’t work: neither medium has any explicit theory of how people actually learn things, and as a result, both mediums accidentally (and mostly invisibly) evolved around a theory that’s plainly false.
If you want to understand a topic, you need to engage with it by summarizing, synthesizing and analyzing the information. Books, lectures, and videos don’t lend themselves to this behavior because they require almost no active participation from the learner.
It was in the wake of reading Matuschak’s question about what format might replace the book that I stumbled on this fantastic post by Kevin Simler. Playing around with the simulations in the post, I really helped solidify his ideas about network science and diffusion.
I’ve noticed similar results from other learning experiences that involved simulations. Whether it was the sales management simulation I did during business school or the Evolution of Trust Game, I’ve found that learnings tend to stick when you get to run trials and see the results. While I’m not yet certain how well I’ll retain the concepts from Simler’s post, I’m increasingly bullish on this medium to promote real learning.
You Have To Live It To Believe It & Generational Theory and Social Choices
(22 min read & 18 min watch)
Why do the lessons we learn from experience seem to stick so much better than ones we read in books? While I can imagine why natural selection favored those who can learn from pain, how much suffering could be avoided if reading about a mistake could allow us to bypass it completely? In his post, Morgan Housel articulates the challenges of cementing a lesson without first-hand experience:
The hardest part of studying history is that you know how the story ends, often before you begin researching a topic. And I don’t think you can un-remember that fact when reading about an event. Particularly difficult is attempting to put yourself in someone’s shoes and imagining their emotions when you know how the story ends but they, at the time, did not.
SEAL Team 6 gained fame after it killed Osama Bin Laden. Video games and movies were made glorifying the raid. But President Obama later said the odds placed on whether Bin Laden was actually in the target house were 50/50. Last year I heard one of the SEALS involved in the mission speak at a conference. He said, regardless of whether Bin Laden was in the house, the team felt the odds they’d all be killed in the mission were also 50/50. So here we have a 75% chance that the raid would have ended in disappointment or catastrophe. Which is not something the people creating glorified video games of an epic adventure think about. They highlight the badass success and glory, because that’s how the story ended. But no one knew that before or during the raid.
Of course, there are some lessons we’d rather not have to experience ourselves. Lessons like these may be best captured by spending time with others who’ve experienced something that we ourselves have not. Jordan Hall articulates this phenomenon in his explanation of Generational Theory.
In my case, I have secondhand experience of the Vietnam war. I have direct personal experience with human beings who were themselves deeply impacted at first-hand by the event. So, I have stories that were human individual personal stories, and, I have an embodied sense of the consequence of those stories in a physical way. But, I wasn’t there. I didn’t live it directly, and that difference makes a difference.
If it’s true that the five people we spend the most time with have a disproportionate effect on our own behavior, it’s worth understanding how their first-hand experiences may be shaping us, by proxy.
I sometimes wonder how well my own experience has prepared me to deal with the challenges ahead. Perhaps living in Silicon Valley has readied me to navigate the waves of technological disruption to come. Of course, it’s equally likely that my lack of experience with subsistence agriculture may leave me woefully unprepared for the geopolitical response to that next wave of automation. Hopefully, this isn’t a lesson I’ll have to learn the hard way.
The Chinese Businessman Paradox
(48 min read, for the whole series)
If you polled a group of Silicon Valley investors to ask what traits they look for in the founders they back, I’d guess that the following would rank high on the list:
Well-educated
Rational
Strategic thinker
Would it surprise them to learn that most successful Chinese businessmen seem to display the exact opposite characteristics? The author describes the successful ones he’s encountered as:
Relatively uneducated
Superstitious
Short-term planners
The series lays out his theories to explain this paradox.
Less trials, better lessons
During their early years, many successful Chinese businessmen forego school in favor of working in the family business. This means instead of learning the “theory and insight” approach to problem-solving that is favored by formal education, most learn to solve problems through trial and error. The author argues that this type of problem-solving is ideal for chaotic environments like the business climate in SE Asia.
Thinking deeply and then acting is optimal for fields where a body of knowledge exists. But in fields where little is known, or where things change too quickly for theory to find handholds, trial and error dominates as the superior problem solving strategy. In these fields, failure is an acceptable cost of learning.
My current theory is that business is one such field. The optimal strategy for learning in business is trial and error, because business changes too quickly for there to be immutable rules. Traditional Chinese businessmen are thus the product of trial and error. The children who run their parent’s businesses are themselves taught via this principle of learning.
He theorizes that the most successful Chinese businessmen do better because they require fewer trials (and thus fewer errors) to learn the right lessons.
Epistemic vs. Instrumental Rationality
As a non-religious person, I often wonder how those who hold, what I consider to be, superstitious beliefs are able to achieve success in other domains that require clear-eyed pragmatism. This anomaly seems to especially apply to Chinese businessmen, who appear to be a particularly superstitious group.
To explain how Chinese businessmen can at once be both superstitious and pragmatic, the author differentiates between two forms of rationality:
Epistemic rationality — how do you know that your beliefs are true?
Instrumental rationality — how do you make better decisions to achieve your goals?
While relatively uneducated Chinese businessmen tend to know little about the cognitive biases and mental models you might read about in Farnam Street, their hard-won lessons from trial and error lead to very pragmatic decision-making. The author believes that this instrumental rationality is what leads them to succeed in business.
Instrumental rationality is the thing that dominates when it comes to success in business and life. It suggests to me that if you’re instrumentally rational, you don’t need to optimize for correct and true beliefs to succeed. You merely need a small set of true beliefs, related to your field; these beliefs can be determined from trial and error itself.
From what I know about signaling theory, I’d suspect that deciding to hold what Kevin Simler calls “crony beliefs,” may, in fact, be a rational calculation. If the people you’re doing business with hold superstitious beliefs and are skeptical of those who don’t, it may be wise to signal that you’re a believer as well.
Opportunistic Thinking
The final post in the series argues that Chinese businessmen tend to be short-term thinkers because the markets they to operate in– mostly commodities– don’t tend to require strategic thinking. Rather they pursue a different type of decision-making that is more opportunistic:
I think the majority of traditional Chinese businessmen don’t think strategically. They think opportunistically. And that’s perfectly fine — by and large, the business domains they work in don’t demand it.
While I don’t believe we should all start thinking more like successful Chinese businessmen, there is a lot of value that can be captured from studying what exactly leads to success in the real world.
Community
Shout-outs
Hugh McFall has been publishing a great newsletter on the intersection of technology and capitalism.
Liz Fosslien who co-authored and illustrated a wonderful book on emotions at work
I’ve also had a lot of fun hosting some recent fireside chats with Adam Compain, Chris Bennett, Jake Saper, Joe Du Bey, Katie Hughes, Kyle Wilkinson, Mike Duboe, Naomi Pilosof Ionita, and Sam Chaudhary
*Header image credit: Olga Zalite