You’ll never guess how we answer this question: “Am I doing myself a disservice by being too idealistic in a corporate environment?”

A student writes:

I’m an undergrad, going into my 4th year. Over the course of my Business-Economics major and Gerontology minor, I’ve developed a burning interest in modeling and analysis and a smoldering distrust of most everything else in the field. I’m just finishing a summer internship I’ve spent the summer in the new Predictive Analytics group applying much of what I learned in Health Econometrics and Time-Series/Forecasting class.

I write because I found your blog because this summer gave me an intellectual crisis. This organization I worked for last summer isn’t concerned with robustness of analysis. They play very fast and loose with data cleaning, model interpretation, casuality, etc and focus most on “telling a story” and getting “buy-in” for their analyses. I’ve adopted some qualities they beat into me, including

I write because I’ll soon be choosing where to take a job for a few years before proceeding to get my MBA, and I’m hoping I can ask for your insight on two things:

1. Am I thinking about this wrong, in your opinion? Am I being too strongly negative in my reactions to how they approach analyses? Rather, am I doing myself a disservice by being too idealistic in a corporate environment?

2. Can you recommend some resources to begin to reconcile the in-fighting regarding which techniques to use? Blogs I’ve found have lots of Bayes-hating, for example, and I spent a half hour speaking to someone tore apart what I learned in school with some convincing arguments.

My reply:

I don’t have much corporate experience so I can’t comment on the general question of how these organizations feel about data. And don’t forget the story of Caesers casinos, whose leader was celebrated in the business press for his data-based rigor but was using all this statistical analysis to more effectively ruin the lives of addicts.

Regarding your particular experience, I imagine there is diversity within the company, and even if your immediate boss didn’t care about honest data processing, perhaps others there do.

For your second question, I think there are lots of methods that can solve statistics problems. I don’t see so much Bayes-hating anymore (at least, not like it used to be). I recommend learning lots of different techniques and you can see what works for you.

15 thoughts on “You’ll never guess how we answer this question: “Am I doing myself a disservice by being too idealistic in a corporate environment?”

  1. Student is being idealistic, but not overly-so. There are organizations out there that emphasize rigor (things like routine use of permutation testing, checking for leaks from the future, etc) and tell the client what they need to hear, even if it’s not what they want to hear. They put the “science” in Data Science: you spend a lot of time trying to break your model and if you can’t, then you declare that the model may be useful.

    I work for such an organization, but you have to look long and hard to find such places. “Data Science/Analytics” combines a bunch of disciplines including statistics, machine learning, programming, and consulting. It’s hard to get the balance right!

    To push back a little, I’d add that it’s easy to come out of technical schooling, having a technical/scientific attitude and not appreciate the “soft” skills that are necessary to succeed generally in a career, and specifically in client engagements.

    You do have to get client buy-in to get their cooperation: access to data, computing resources, subject matter experts, end users, etc. You need buy-in to get your results deployed and used. You need buy-in to get SME’s to validate your model, which is usually work added to their already-full plate. (And without this real-world validation, you haven’t accomplished anything real: all of your CV and other checks are secondary to actually predicting well.)

    You do have to tell a story that your clients can comprehend. Not the story about the cool stuff you did. (I’ll never forget the end-of-project briefing I did once where I was basically repeating a lot of what we’d told the fairly-technical clients at weekly status meetings… They didn’t understand several graphs — graphs that they’d nodded and “uh huh’d” when they were originally presented. Meaning they never actually understood. Image burned in my mind.)

    You often have to guide clients to come up with solid business questions to answer, rather than vagaries. All of these are consulting skills, and they are important.

    Many large consultancies have lots of this kind of expertise, but they lack statistical/scientific rigor. Lots of smart people who know about stats, and who can make cool technology (R, Hadoop, etc) work, but who don’t sweat the details. They don’t have the scientific mindset. And perhaps that’s Student’s experience. I’d personally avoid the name-brand, large management consultancies for that reason.

    In terms of particular techniques, I’ve gone back and forth. In one sense, I’m one of the few Bayesians in my organization. Sort of. Turns out that the Stan runs I’ve been doing take 2-3 hours to accomplish the Bayesian equivalent of 5-20 minutes of non-Bayesian approaches. As a different example, I was big on SVMs for a while — while there are several folks I tor with who discount them. But now I’ve turned away from them as well: my experience is that they’re slow and fidgety. I’ve also tinkered a lot with neural nets — not an expert by any means — and have also found them to be fidgety. Don’t get me wrong, we had great success with a hierarchical logistic regression last year, but I’m more of a Bayesian philosophically than in practice, and my overall experience is that most of the mainstream methods/philosophies are workable and it’s mainly a matter of your expertise, the problem at hand, the resources available, etc.

    So keep your rigorous attitude. Deeply learn and use good tools and techniques. Look for smaller organizations that prize a scientific approach. Know that consulting skills are important to add to your repertoire. Lead by example, not from a soap box. Look for an employer’s culture first, and other things (salary, reputation, mission) second. And good luck!

    • There are organizations out there that emphasize rigor (things like routine use of permutation testing, checking for leaks from the future, etc) and tell the client what they need to hear, even if it’s not what they want to hear. They put the “science” in Data Science: you spend a lot of time trying to break your model and if you can’t, then you declare that the model may be useful.

      I work for such an organization, but you have to look long and hard to find such places.

      Names please? You can’t just put this out there and not provide details.

  2. I have worked for a couple consulting firms, the federal government, two universities and I do a bit of small-contract consulting. There are many places that just want an answer, especially an answer that pleases the HiPPO (highest paid person in the room). If you care about getting the correct answer, don’t take such jobs because they will suck the soul out of you quickly. It can take some time to find, but there are many, many places that want a good answer (not necessarily the “correct” answer which would take too much time, but a “good” answer that can be had quickly enough). I echo Wayne’s sentiment above: look for culture first.

    Two additional points: it’s not just small firms that value correct analyses. Some large firms do, and some parts of some large firms do. A student I helped train in analytics got a job with Koch Industries a few years ago, and he just loves it; when he gets the “wrong” answer he is not told to deep-six it. He’s still there, still loving the job — all he does it crunch numbers. At the other end of the spectrum, a senior colleague was hired to direct research at a large, well-known organization that I shall not name. When the research team came up with answers that the client didn’t want to hear, he was ordered to change the result. He refused, and left with a nice severance package (he threatened to release the true report after the fake report was given to the client — so he got the good severance package).

  3. > They play very fast and loose with data … focus most on “telling a story” and getting “buy-in” for their analyses.

    That was my sense about the consulting firms I was interviewed by when completing my MBA (1983, when data was scarce in firms) and my motivation for going into clinical research instead. Turned out some of the clinical researchers I worked with or became familiar with their work were actually much worse…

    Finding the right group of folks to work with and being able to recognize when that is the case – is probably more important and challenging than anything else.

    Also, is there someone senior to you in the organization that you can get some feedback from on your views of what’s happening. Don’t complain or be accusatory but seek advice from them?

    • “Turned out some of the clinical researchers I worked with or became familiar with their work were actually much worse…”

      Yes, I suspect this problem may be lesser in industry than academia. I would consider the practices complained about in this earlier post as “standard operating procedure”:

      http://statmodeling.stat.columbia.edu/2015/12/20/once-i-was-told-to-try-every-possible-specification-of-a-dependent-variable-count-proportion-binary-indicator-you-name-it-in-a-regression-until-i-find-a-significant-relationship-that-is-it-no/

      You have to think about what your goal is (make money? do a good job?). If the latter, the people subscribing to that culture are literally obstacles in your way. Then it is a matter of either circumventing or engineering some solution to those obstacles. What’s the best way to do so? I haven’t solved that one yet.

    • In corporate there is a lot of repetition of the same thing, with incremental changes-region, product features, distribution channels, etc. Healthcare is struggling with being more quantitative because of government and cost concerns. It does not have a legacy culture of data summary and outcomes-they are in 1983 as Keith says. Research was either disease specific or population specific. I went to the health statistics policy conference and these are huge problems with some really sharp people getting their hands dirty. But a lot of the stakeholders come out of nursing and administration with fixed processes and a culture of inspection. So this student is confronting a world where 1) storytelling and buy-in is important, not just showy, and 2) complexity is huge but simplicity is often demanded. So you need to get people up a notch, such as basic logistic regression from simple tabulation, not necessarily to the level of an ivy league MPH program. So the student might prefer to go into academia, a think tank, or a more quantitative industry rather than a corporate healthcare vendor or provider.

  4. Unfortunately, I recognize that second paragraph from my decades in industry. There’s variation all over the place in turns of what we might summarize as ethics or hygiene, even within the same company, but I’m not going to contend that second paragraph doesn’t apply to lots of situations.

    How to find a compatible place? There are some indicators. Is there a lot of turnover? Bad sign. Do they have a set of clients they do long term, consistent business with? No is a bad sign, but even yes might mean they are good at feeding the client what they want to hear, not what they need to hear. Ask them for the two projects they worked on that had the most impact. That might provide a clue.

    As to techniques, bear in mind there’s a huge difference between academic and applied work. In academic work, there’s a lot of attention to the fanciness of the technique. In applied work, if you can answer the question with a 2×2 table, you’ll check for confounds, but present the 2×2 table — the clients will understand it and you’ll be able to get on to the next project on your list. One of the main purposes of more complicated analysis in applied work is to be sure you aren’t being deceived by the simpler analysis. In fact, a common problem with analysts is they get in over their head, doing more complicated analysis that they don’t understand, so the complicated analysis just gets in the way.

    Clean the darned data. Run a lot of simple summaries. These will make you popular.

    Use holdout samples. Replicate on different data, and try a different analytical method. These won’t make you popular, but are often essential to your feeling of personal integrity. The attitude toward validation in industry is much like Woody Hayes attitude toward the forward pass: “Three things can happen, and two of them are bad.”

    In applied research, validation has these outcomes, two bad from the point of view of the project:
    1. You get similar results, in which case you feel more confident (but have also used up time that could have been spent on the next project).
    2. You get different results, so you are back to square 1, and need even more time.
    3. You get results that aren’t quite the same, but aren’t quite different (e.g. different magnitude), leading the client to say “well, what is my price elasticity? -1.2 or -1.8?”, also resulting in more work.

    If you are frequent reader of this blog, you will know validation problems are a common occurrence in academic research as well, but it’s worse in applied work unless you work in a group that gets religious about it. Those two projects that I suggested you ask about above? See if they involve any attention to these issues.

    • Seconding this in general, but particularly the part about the the difference in what you present and what methods you use. Simpler presentations, and, where feasible, simpler techniques are always preferred; facility with advanced techniques as a check may be needed. and will be allowed in the best places.

  5. General advice about organizations: look at who is at the top, and where did they come from? If the CEO came from marketing, then how things will look to the customer will be the priority. If the CEO came from accounting, then the company probably pinches every penny. If the CEO was an engineer, then they probably value engineering. So look for companies whose very top position is held by someone who knows and respects data and you will probably be ok.

    Note that this works for countries too:
    US: Lawyer
    Germany: Chemistry PhD
    Russia: Secret police
    France: Bureaucrat

  6. I work for a large ($50 billion) company. In my work we have to consider the ROI on additional time spent on analytics. Another insight is that unlike academics spending time to create general knowledge, in my position we often are working to create enough knowledge to help someone make a decision. Sometimes there isn’t enough time to do a thorough analysis before a decision must be made.

    I have loved my time in industry and wouldn’t trade it for an academic career. I enjoy seeing the direct impact of my work in helping people running a huge company make informed decisions.

    • That’s a good point about impact. On the academic side, I would write up some research and then try to persuade some journal to publish it. It’s easy to feel a bit unwanted in this scenario.

      In an industry setting, you often have an anxious and ready audience that can’t wait until you tell them what the results are. Admittedly, after they trust you they often just read the topline, but I learned to treat that as a sign of their faith in me.

  7. I have worked in corporate environments. You will get a lot of benefit from reading about group dynamics, especially the theories concerning Bion’s ‘basic assumption groups’. These are three common failure modes that have analogues to infant developmental psychology — yes, organizaions really create entities that have the irrational drives of infants, and it good to understand just how stupid and paranoid organizations can be.

    https://en.wikipedia.org/wiki/Wilfred_Bion

    Also, it may be good for a Bayesian who is interested in neural nets to ask why such phenomena are scale invariant (fractal) — why do corporate group dynamics recapitulate Kleinian object relation theory and the Paranoid-Schizoid position, and why do both both recapitulate the trauma of a blastocyte attaching to the maternal fetal wall?

    It turns out the plot of 2001: A Space Odyssey got it wrong. Before we make an AI that seems normally human, we will *already have mastered* the paranoia states of infantile (and corporate) reasoning. They will be well known failure modes, unlikely to occur in production and easily diagnosed.

Leave a Reply

Your email address will not be published. Required fields are marked *