Anti-ScienceFeaturedFeminism

For-Profit Wage Comparison Website Overturns Feminist Dogma. Or Not.

This morning, reader “Max” sent us the following note via the Skepchick Contact Form:

http://finance.yahoo.com/blogs/the-exchange/3-myths-working-women-190653888.html

Women earn far less than men. On an aggregate basis this is true, but
Payscale controlled for a variety of factors and found that the pay
gap is far narrower among men and women who have similar levels of
experience, work in the same fields and locations, have the same
skills and certifications and are otherwise workplace clones. Among
non-managerial workers, Payscale found NO PAY GAP AT ALL, on an
apples-to-apples basis. At more senior levels, women do earn slightly
less than men, but the biggest gap [at executive level] was only 8.5%
— far lower than the frequently mentioned 23%. And that gap may exist
because top male executives put in more hours, travel more and spend
more time with clients than their female counterparts, which is hard
to measure among salaried employees.

http://www.payscale.com/data-packages/gender-wage-gap/

Thanks for sending this in Max. You see, I’ve been under the impression that there was a large gender pay gap of around 23% between men and women in the US and that the causes of this gap were largely because of overt sexism in hiring and promotion decisions (especially at the executive levels), institutionalized sexism in the structure and benefits of organizations, and cultural sexism that makes it more difficult for women to enter and compete in higher salaried fields such as science, mathematics and engineering. But, I am a skeptic and as a skeptic I am willing to change my beliefs based on evidence. The study Max sent in must contain a lot of convincing evidence in order to overturn the vast number of other studies looking at the gender pay gap. Let’s jump right in, shall we?

The first link Max posted was to a Yahoo! Finance article entitled “3 Myths About Working Women.” The points Max mentions all seem to come from this article. The article begins by letting us know that all these myths about women being paid less than men come from this American Association of University Women (AAUW) paper that uses labor force data to argue that there is a significant pay gap between men and women in the US. They go on to inform us that the prevalence of these myths has led to some terrible outcomes including “heated debate over workplace policies, family priorities and the changing nature of motherhood (and compelled a lot of men to simply keep their mouths shut).”

Oh no! The poor men being all silenced by those evil University Women and their false study using data from the “Bureau of Labor Statistics.” That doesn’t even sound like a real organization. Luckily there is some hope because Yahoo! Finance assures us that this new research will set the record straight. Rather than read Yahoo!’s summary of the data, I clicked the link to the source study to see the data myself.

The paper entitled “Women at Work: PayScale Redefines the Gender Wage Gap” is published by what I can only assume is a well-respected scientific publication PayScale.com. The top of the first page gives the following quote by their lead (and only) economist Katie Bardaro:

Unequal pay for equal work? Not really. Women earn less than men on average because they often fill jobs with a large societal benefit, but small monetary benefit. Instead of focusing the debate on the misbegotten gender wage gap, we should instead examine why women are absent from high-paying jobs and industries, like technology, engineering and executive positions.

Katie’s paper sounds so interesting. I bet she’s got a whole bunch of data to back up those claims. Right below Katie’s quote is a short summery of the data with a chart showing the wages of men and women.

"Uncontrolled" PayScale Chart showing large gender pay gap

Wow. From that chart it sure looks like men are paid a lot more than women for the same level jobs, but the caption tells me that it only looks this way because I’m looking at the chart blind. Ok PayScale, go ahead and open my eyes.

Controlled PayScale Chart showing little to no pay gap

Whoa! They totally moved that blue women’s wage line all the way up to the men’s line. Mind.Blown.

What is this? Some sort of dark data magic? Now that I’ve read the abstract, I’m ready to move onto the paper itself. I’m sure Katie does a good job at explaining exactly how they moved that lady line.

Hmmmm…This is a bit strange. Most of the links tend to go to pages asking me to give them money to tell me what my salary is supposed to be. I’m starting to think that maybe PayScale isn’t quite the academic journal I thought it was. In fact….wait a second. This can’t be. Those two charts and 3 sentences that I assumed were the abstract seem to be the entire main section of the article.

Underneath all the sections trying to convince me to give them money there is link to their methodology. I knew Katie wouldn’t let me down. I’m sure the methodology has all the information I’ll need to understand how they moved the lady line.

According to their Methodology, PayScale surveyed 13,500 U.S. working residents. Also ….  Nope. That’s it. That’s the methodology.

You know, I’m starting to think PayScale is just messing with me or something, but I have faith that Max never would have sent us this paper if it didn’t have some pretty good evidence in there. After a couple more minutes of clicking around the website and avoiding giving them money I think I may have found another section of the paper containing far more data than the three sentences we saw earlier.

This new section breaks down the median pay between men and women in their survey for 12 very specific professions: Software Architect, Pharmacist, Civil Engineer, Secondary School Teacher, Executive Chef, Social Worker, Chief Executive, Registered Nurse, Doctor/Physician, HR Administrator, Accountant and Secretary.

In all but two of these fields, PayScale shows men making more than women even after controls for experience and education are factored in. Even stranger, in four of the 12 professions the women controlled median salary is actually lower than their actual salary, implying that the women’s salary was less than the men’s even though they had more education and experience than their male counterparts.

This all seems to imply that there is a pay gap and women are being paid less than men for the exact same jobs. Sure it’s not the 23% pay gap mentioned in the AAUW paper, but that’s because the AAUW is looking at the entire US labor force while PayScale is merely comparing 12 extremely specific jobs. Most of these jobs, such as teachers, nurses, accountants, and social workers tend to be the types of positions that are unionized or otherwise have very specific pay structures that would make gender gaps in these positions far more unlikely.

Also, PayScale surveyed only 13,500 individuals. I wrote to the author Katie Bardaro on Twitter to see if I could get a little bit more information about the people surveyed.

She informed me that the study had 51% women and 49% men and that all jobs were considered, not just the 12 mentioned. This is a little worrying because 13,500 is really not that many people in a survey. Then, it’s being split up and analyzed in 12 distinct categories of which many individuals in the 13,500 likely do not fall in any of those specific positions.

For example, what percentage of people surveyed are executive chefs? What percentage are civil engineers? PayScale doesn’t seem to offer up this information and Katie didn’t reply to my follow-up questions, so we’re just going to have to guess. Lets try some back-of-the-envelope calculations with secondary school teachers.

The National Center for Education Statistics estimates there were about 3 million elementary and secondary school teachers in the US in 2011. But, this includes elementary school teachers. NCES also lists some older data showing that in 2008 about 40% of elementary and secondary school teachers taught secondary school. Assuming the proportion remains fairly constant since 2008, this would give us about 1.2 million secondary school teachers in the US.

The Bureau of Labor Statistics estimates that there are a total of 141 million people employed in the US labor force in 2011.

If one million of those are secondary school teachers, then secondary school teachers make up about 0.9% of the employed labor force. Since these are all estimates anyways, let just round that up to a generous 1%.

If PayScale’s survey got a good random cross section of the labor force, then 1% of the people surveyed are secondary school teachers. This would be a total of 350 135 individuals. PayScale does tell us that 56% of the secondary school teachers surveyed were women. That would give us 196 76 women and 154 59 men in PayScale’s survey that are secondary school teachers.

This seems to be quite a small study indeed and that is just for secondary school teachers, which are likely one of the more populous of the professions mentioned. I can only imagine that far less than 1% of the US employed labor force are executive chefs or pharmacists. Small groups like this are going to contain a lot of variation in salary. PayScale doesn’t give us any measure of variation so it’s impossible to put a range on the pay gap figures they mention.

For the secondary school teachers, PayScale found a 4% pay gap between men and women. Perhaps it is actually a 3-5% pay gap or a 2-6% pay gap or a -16% to 24% pay gap. PayScale just doesn’t give us enough information to know, though with the small number of people surveyed it is likely to be a large range.

A lot of the information in the Yahoo! Finance article that is mentioned by Max doesn’t seem to actually be in the PayScale study at all. Maybe there is a whole other page that I am missing but I swear I clicked around all over their website and didn’t see anything other than the main page, methodology and profession break-down. Either Yahoo! Finance has access to more data on the study than PayScale has released publicly or they are completely making up facts out of thing air and calling it journalism.

I can only conclude from all this that PayScale’s study, rather than being the game-changing new paper that Yahoo! Finance and Max make it out to be, is actually a concealed advertisement for compensation software put together by an extremely dodgy looking company  in order to trick terrible “news” outlets into writing about it.

Jamie Bernstein

Jamie Bernstein is a data, stats, policy and economics nerd who sometimes pretends she is a photographer. She is @uajamie on Twitter and Instagram. If you like my work here at Skepchick & Mad Art Lab, consider sending me a little sumthin' in my TipJar: @uajamie

Related Articles

46 Comments

  1. Great analysis. My immediate intuition would have been that 13,500 is a pretty good sample size (based on the more psychology-focused surveys I tend to see), so it was really interesting to see how badly that falls apart once you start considering how the labor force is actually partitioned up. Seems like you’d need a pretty big survey in order to get good estimates of gender-based wage gaps!

    1. Yeah. It almost seems like you’d need some kind of governmental bureau to really track all that stuff. You know, one for labor statistics. But what would we call it?

    2. I also really liked this analysis (not the least of reasons being the aplomb with which it is treated).
      Psych studies are usually focused on specific, discrete things, right? Like particular areas of brain function or physiology or behavior? I’m not too familiar with psych sample sizes except for the studies I participated in as part of my undergrad psych course.

      1. It varies greatly but assuming your general psych study conducted on college-age students at an American uni a sample of a couple hundred would be pretty good. Many have been reported with ~50 or so participants.

    3. In my non-Skepchick life I’m a big data scientist so I’m used to working with datasets that have millions of observations. If parsing it up leads me to groups of under 50,000 each, I know I’m in trouble and can’t do the analysis. The problem is that I’m dealing with data that has a lot of variation and effect sizes that are very tiny. The variation is so high that if I have groups of under 50,000 the results will be all over the place.

      Obviously I’m not working with survey data because surveying millions of individuals would be inordinately expensive. Surveys are great for many things and you don’t necessarily need a lot of people to get some interesting information. For example, in a survey like PayScale did, with 13,500 people, you might learn that there is a lot of evidence to suggest that men are paid more than women overall, that women tend to be in lower-paid professions than men and that the pay gap seems to increase as positions become higher level. This would be a totally legitimate conclusion from the PayScale paper. In other words, it’s fine to do a survey and make some general statements about trend.

      What you couldn’t say is exactly how big the gap is. There are a huge range of possible salaries and it tends to be quite variable from one person to the next. So, you’d need a much bigger dataset than 13,500 to know exactly what the difference in salary is between two groups. When you start to partition that into even smaller pieces it becomes even more problematic.

      This is one of the reasons that opinion polls for the general election tend to be so variable. They conduct each poll on a lot of people, but there is so much variation. If one candidate is a good amount ahead of the other, you’ll probably detect that in your poll. But, you’ll not be able to say exactly how far ahead. That’s why it’s ridiculous when polls are conducted a week apart and the polling organizations announce a 1pt drop due to a candidate’s gaffe that week. You’ll just never be able to see detail that fine with such a small dataset and the 1pt difference is likely due to random variation. And, don’t get me started on what happens when you try to partitian that by gender or ethnicity or political party).

      However, if you make your dataset bigger by putting all the opinion polls together into an aggregate, you’re going to come up with something that has much finer detail and it will be more stable. You’ll be able to see details in the aggregate that you could never see from any individual poll. This is what Nate Silver did. The genius was in the statistical power gained by having a bigger dataset.

      Obviously, many times surveys are all you can do and that’s fine as long as you aren’t making extremely specific claims. But, in terms of labor statistics it’s completely ridiculous to do a survey. Not only do you have to take salary data from surveys with a grain of salt (traditionally people lie or leave it blank — We have no idea how many of the 13,500 left it blank on PayScale’s survey but on many surveys it could be over 50%) There is ample data available to researchers on exactly how much individuals make. Businesses have to hand over salary (and gender & demographics) data of their employees to the government and the government anonymizes it and reports on it and allows its use in research. The availability of good quality and quantity of data is what has made labor economics such a data rich field. There is just no excuse for ignoring all this data and doing a survey instead.

  2. OK, regardless of the freaky, mess-up data, this quote bothered me a bit:

    Unequal pay for equal work? Not really. Women earn less than men on average because they often fill jobs with a large societal benefit, but small monetary benefit. Instead of focusing the debate on the misbegotten gender wage gap, we should instead examine why women are absent from high-paying jobs and industries, like technology, engineering and executive positions.

    Uh… If that’s true, then why are men payed more for jobs that have equally large social benefit or, even worse, are paid more for jobs that have less social benefit than the jobs women tend to engage in? I mean, doesn’t that imply that in a capitalist society such as ours, we literally place far less value upon women’s lives, work, and interests?

    And why is that, exactly?

    ….crickets….

    Hello?

    1. That bit jumped out at me, too.

      Women aren’t paid less, they just have jobs that pay less. WTF?

      Circular reasoning works because circular reasoning works.

    2. “Uh… If that’s true, then why are men payed more for jobs that have equally large social benefit or, even worse, are paid more for jobs that have less social benefit than the jobs women tend to engage in?”

      The claim “equally large social benefit” is completely subjective; people would even disagree on measurement criteria. Pricing for goods and services (labor included) is determined by a little thing called supply and demand. Price is not determined by what the object or service provides the buyer (the reason for the demand), it is determined by it’s availability, it’s supply. Pricing helps our economy allocate resources efficiently; consider it a signal.

      “I mean, doesn’t that imply that in a capitalist society such as ours, we literally place far less value upon women’s lives, work, and interests?”

      Yes, if what those women are producing is in very great supply, and has driven the price down. If everyone decides to grow carrots, then we will value carrots less, right? This is a good thing because it sends a signal to the labor market that we now value a different allocation of resources. The solution is that women should change their life, work and interests into those that provide a greater return.

  3. I’ve been seeing a lot of stuff like this from Yahoo! lately. The other day there was an article that stated how under the ACA, people were going to see a decrease in benefits. How so? Well, it turns out that the cheapest plans under ACA will have high deductibles. So, for those who can’t afford health insurance now, the plans available to them won’t be that good. An important point, but how does that translate into people losing benefits? I’m not sure. Going from no insurance to high-deductible insurance is still a step up and no where in the article did it state that anybody with lower deductible plans would lose them. But, the article continued on to claim how much worse insurance would be under ACA undeterred by the total lack of evidence for it.

    1. Someone on my facebook feed recently stated that a certain (not sure which) crack-pot right-wing website was being placed on the Yahoo! news page as an actual valid news site. Not even Fox news, something really, obviously NOT A NEWS SITE.

      Something is up with Yahoo!

      I wonder if they are just flailing about, trying to get page hits from anywhere and anyone.

  4. Awesome post, Jamie. Wonderful analysis.

    I think it’s also interesting to point out that the sample Payscale uses is self-selected–it’s all people who visit the website and take their survey, not a random sample. And you mention above that you get nervous when you have groups of less than 50,000. Well what do you know, the Bureau of Labor Statistics and the Census data showing the 23% pay gap number comes from a monthly survey of 50,000 people by the Census Bureau (right there on page 6 of the AAUW paper you linked to).

    1. I was wondering about that part too. I’ve done survey studies and usually a standard discussion in the methodology section would be to describe 1) where your questions came from (without this you could just have horrible misleading questions) and 2) what the response rate was, if it was a low percentage that’s already a big red flag. Website surveys that just have people volunteer this information would be laughed at, possibly without evening finishing the review of the paper.

    2. Will, Sorry for taking so long to respond. i think I missed your comment earlier.

      First of all, are we sure that it was a self-selected survey on their website? That’s what I sort of assumed, but I didn’t see anywhere where they said that that was how they did the survey. They might have a panel and selected a specific demographic make-up within their full panel or something. This wouldn’t have as much self-selection bias. I asked the study author on Twitter for more details, but she just sent me to PayScale.com and was otherwise totally unhelpful.

      My issue in my work with only 50,000 people is because I’m generally looking for effect sizes that are quiet small, often under 1% difference between two groups. In order to see something this tiny, I need lots and lots of people in order to increase the accuracy. If the true effect size is 0.9% but my accuracy is within 2 percentage points, I’m just never going to be able to see that 0.9%.

      However, even in this case, the AUWA study is not based on only 50,000 individuals. The Census data from the CPS survey (which is like a super census survey) is done on 50,000 people a month. However, it’s designed to be as accurage as possible with as few people at possible. This is the survey from which we calculate unemployment on a monthly basis. Often, CPS data is not looked at monthly, but aggregated over the year (600,000 individuals annually). Additionally, this is the source where AUWA gets some of their demographic and employment data. The wage data doesn’t come from the CPS but from BLS, which surveys businesses, firms and organizations on the wages and benefits of their employees along with some basic demographic data. This data is considered extremely accurate because it is reported on by the business and they legally have to provide accurate information about the wages of their employees.

      I don’t know much about the AUWA study other than skimming over the paper that Yahoo!News linked to, so I can’t really comment on exactly what it covers or how accurate it is, but I can say that the data they say they’re using (from the CPS and BLS) is the most accurate wage data available. It is also public data, so I really don’t understand why PayScale just didn’t use this data, which they also have access to, instead of conducting their own clearly inferior study.

      1. Oh, one more clarification. Just because AUWA is using good data doesn’t necessarily mean that it’s a good study or interpreted correctly. I’m familiar with lots of other labor economics studies (such as on the effects of unions or minimum wage laws) and usually economists will use BLS data to make an economic argument, then other economists will do their own studies using the EXACT same data to argue the opposite. The real arguments within labor economics in the US generally take place over the correct way to analyze and interpret the BLS data, not by collecting data from some other source that will inevitably be inferior.

  5. The number of supposed skeptics who think that all the science in an article is contained in the abstract and conclusion is pretty staggering. Comparing two similar studies to identify what led to differences doesn’t require super-foundational knowledge – “This study had a larger sample size and used a different methodology” should be rather evident.

    On the topic of the supposed “disappearing wage gap” which occurs when we control for experience and job title – the fact that women as a group tend to work lower-paying jobs than men (which is one reason the wage gap “disappears” when you only compare equal employees) does seem like it should be a big deal.

    1. The number of supposed skeptics who are willing to cherry-pick individual scientific articles to support their preconceptions is also staggering. We all know that lots of crap articles like this one get published — what matters is the larger pattern in the data, and what the consensus of experts who are equipped to evaluate the evidence seems to be. Of course sometimes the consensus of the field is wrong, but corrections to that generally come from within in response to new data. But most of the time, new studies that appear to dramatically overturn everything we think we know (“Hey, sexism is over, everyone!”) turn out to be wrong. Skeptics ought to understand this, and ought to view studies that seem to corroborate their pet theories with greater scrutiny, not less. Funny how that works — it’s almost like learning how to question the existence of Bigfoot doesn’t automatically make you a superior being.

  6. Er, I think you made a minor math error, 1% of 13,500 should be 135. That would then break down into approximately 69 women and 66 men. A far smaller sample size, meaning the data is that much more meaningless.

    1. Wow. That’s a ridiculously obvious error. I was a bit sleep deprived last night and must have mixed up the numbers in my head or something. Thanks for noticing! I fixed it in the post. The 51% women was for everyone surveyed, but for teachers there were actually 56% women, so the true estimates would be 76 women and 59 men in the sample.

      1. ok, did not catch that part, was going off of the totals. But apart from that, it is still a ridiculously tiny sample size, almost as bad as the “diet coke is as bad for your teeth as crystal meth and crack” study which consisted of a sample size of 3.

  7. I’m reading still, but I noticed you said that ~13,000 is a small number for a survey, which I have to disagree with. A conservative estimate of the standard error of the sample mean is found by dividing your standard deviation (of the sample) by the square root of the sample size; 13,000 gives a ±0.88% for any statistic; that’s a pretty tight boundary. Reducing further though because of the male/female division, we have ±1.15% for each statistic; still pretty good.

    Of course, where I can certainly see a problem (and again I’m still reading so you probably touch on this too) is the choice of careers, when there is a much larger population of careers in the sample. This reeks heavily of data mining – you have to form your hypothesis prior to gathering the data, not after. If your alpha value is 0.05, you have a pretty darned good chance of finding what you want if you narrow down your sample size to fit criteria that you hadn’t specified before. And that ruins the whole point of the test.

    But anyway, still reading…

    1. Aaaaand just read your discussion on the percentages of employee type relative to total workforce – spot on.

      1. Where are you getting the standard deviation of wage to calculated the standard error for their 13,000 sample? I didn’t do this myself because I didn’t see anywhere on the PayScale site where they mentioned a standard deviation.

        13,000 could be fine for looking at an overall difference, especially considering the large difference between the wages of men and women when looking at an overall difference, but yah, it definitely starts to get sketchy when split up into the smaller job categories.

        1. Jamie, SEM as a percentage depends only on root n, so richard did not need to have the sd (as I’m sure you know!)
          I thought richard’s comments were spot on. I too was nonplussed by your statement that 13000 is a small sample size!
          Of course as soon as you begin to split the data into too many groups the statistical power bleeds away all too rapidly

  8. I’ve been having a go through of the information from the breakdown page you linked to, and comparing it to the BLS’s data from their Occupational Outlook Handbook.
    http://www.bls.gov/ooh/

    Many, MANY problems arise regarding this “paper,” and to name a few:

    1) Right off the bat, as Will pointed out, if the study was based on voluntary interviews, there is no way to put any sort of confidence intervals on any of these numbers. The study should be scrapped before we even begin; non-response bias isn’t quantifiable, and isn’t possible to take into account.

    2) Even if the sampling was i.i.d., the BLS job breakdown places the total number of such jobs at about 8.7% of the workforce; that means that the sample would really only contain 1181 people of those jobs, and more specifically only 770 women and 411 men. For each of the twelve jobs, given the gender breakdown from the “paper,” we get (men, women) in the sample:
    Soft. Arch. – 83, 4
    Pharmacist – 13, 13
    Civil Engin – 22, 3
    Sec. School Teacher – 44, 56
    Exec. Chef – 9, 1
    Social Worker – 52, 10
    Chief Executive – 28, 7
    Registered Nurse – 29, 233
    Physician/Doctor – 43, 23
    HR Administrator – 1, 6
    Cert. Public Acc. – 76, 41
    Secretary – 12, 372

    These are ridiculous subsample sizes, of course.

    3) The percentage women in the entire sample space that falls under one of these jobs that experiences a negative “control” adjustment is 43%. This is astounding, since the claim is that the pay gap is supposed to be explained by this “control!” In 43% of these cases, it is increased by it.

    4) Similarly, the percentage of women that only receive a $200 or less “correction” upward (so that includes the above) is a staggering 92% of the sample. This is due to the inclusion of only one group, the Secretary group.

    5) We don’t have any decent means of estimating the standard deviations of any of these sample statistics (gendered wage averages, “control” applied for each group, so on), which would be necessary in order to estimate the wage+control means and standard deviations for each job for women. But we could, for a rough estimate, weight the “control” values for each job with the number of women in those jobs in the sample; and then find the standard deviation of that group, which gives us a mean “control” correction of +$887 (completely dominated by the “control” applied to chief executives, mind you all) and a standard deviation of – wait for it – $6894. This is also due to the absolutely massive correction applied to the chief executive pay.

    6) Everything is actually in medians anyway, NOT means – if the women above the median are only slightly above it but the men above the median are WAY above it, you’re not going to be able to see that from the median data. You’re going to miss that wage gap.

    The data and methodology are SO bad. SO, SO bad.

    1. Richard, omg I loved this post. I’m going to respond to each in order:

      1) Yah, not to mention that the fact that people lie about their wage on survey data is a renown problem and that it was a total waste of time to do a survey on this when there is really good public wage data already available from BLS.

      2) Good find on that BLS data! I did “secondary school teachers” for the example because I thought it would be the most generous and it was one of the easiest to find data on. Those are getting into some seriously small numbers to be making sweeping claims about.

      3) Yah, I mentioned in one line of my post that for 4 of the positions listed the “control salary” for the women was lower than their actual salary. If we assumed their data was actually good, it would mean that the women were getting paid less than the men even though they were more qualified. That’s pretty crazy.

      4) Interesting.

      5) Wow.

      6) Yah, I almost mentioned this in the post but felt it was getting too into the nitty gritty details and wanted to focus on some of the bigger more obvious issues.

      Seriously, you should have written this post because the calculations you did using that BLS data is fantastic and really brings out the terribleness of this study better than I ever could.

  9. Thanks for the analysis. I was mainly curious whether the 23% wage gap really was estimated without controlling for any factors, and that seems to be the case.
    Here’s what the AAUW said in 2007.
    http://www.aauw.org/research/behind-the-pay-gap/
    “According to Behind the Pay Gap, women one year out of college are paid 80 percent of what men are paid. Ten years after college graduation that number drops to 69 percent. After controlling for factors that affect earnings — like college major, job, and hours worked per week — women are still paid an unexplained 5 percent less than men one year after college graduation. This unexplained pay gap widens to 12 percent ten years after graduation.”
    So their controls narrowed the gap from 20-30 percent to 5-12 percent, while PayScale found an unexplained 8.5% gap at the executive level. Both studies look at median wages and aren’t published by a well-respected scientific publication as far as I can tell.

    Bias and poor controls tend to shift results away from the null hypothesis, not towards it.

    PayScale points out that 90% of HR admins are female, and that “Median pay for each gender is just about even – not surprising for a job whose responsibilities include knowing what everybody in a company is paid.” Of course HR admins know the wages of employees in other departments as well.
    Managers joke that if women did equal work for less pay, they’d hire only women.

    All this is reminiscent of the famous Berkeley gender bias case, a textbook example of Simpson’s paradox. Just substitute low-wage jobs for low-admission departments.

    Even if all the controls eliminate the wage gap within an industry and job title, that still leaves the “job gap.”
    That’s the conclusion of the PayScale researcher, who tweeted, “The gender wage gap is largely false. The real issue is the jobs gap; men are still filling highly paid positions at a disproportionate rate.”

    1. Hi Max,

      A couple comments. First of all, it is true (and I am certainly not denying) that when you control for factors such as education, the pay gap narrows. There are lots and lots of studies that show this, including the AUWA data. That doesn’t make the fact that when not controlling, the pay gap becomes huge. It means that not only are women getting paid less in the same jobs and with the same background as men, but that they’re not necessarily able to get or discouraged from entering the jobs which would pay them more. However, even if the PayScale “study” found exactly the same thing that other studies found, it wouldn’t change the fact that it’s a completely worthless study with a terrible methodology and that you cannot draw any conclusions from their results.

      Bias tends to shift results towards the direction of the bias. If the bias is toward the null, then that’s where it will move. It’s completely ridiculous to say that it only “shifts results away from the null hypothesis.”

      On the rest of your points are you saying that women are not paid less than men and that there is absolutely no bias toward paying women less? And, do you realize how crazy this sounds when literally everything, including the PayScale “study” which you seem to like so much, say the opposite?

      1. Hi Jamie,

        I’m not sure it’s common knowledge that “77 cents on the dollar” is not “for doing the same work as men” as Obama claimed in a campaign ad.
        http://www.politifact.com/truth-o-meter/statements/2012/jun/21/barack-obama/barack-obama-ad-says-women-are-paid-77-cents-dolla/

        By bias, I meant anything that skews results from truth, not just ideological bias. It’s just hard if not impossible to control for everything, and any hidden factor can create a false positive outcome. For example, the Yahoo Finance article pointed out that it’s hard to measure and control for the hours put in by salaried workers, who aren’t paid by the hour. So if the null hypothesis is true, then a study with better controls will have an outcome closer to null. See studies on homeopathy or parapsychology.

        I can see why the American Association of University Women may be biased against the null hypothesis, but I don’t see why PayScale would care either way.
        PayScale found what looks like at least a 70% wage gap without controls, which gives its data some credibility, and they were able to really narrow the gap with controls. Why would a small sample or poor methodology go in the direction of narrowing rather than widening the gap?

        Accounting for the wage gap does not account for the jobs gap. Take computer programming, where according to the Institute for Women’s Policy Research the wage gap is 5% even before controlling for anything, yet only 22% of the workers are female. Why is that? Do they hire only the most educated, most experienced women, but pay them the same as the average man? Then controlling for education and experience should WIDEN the gap, not narrow it. Why women don’t enter high-paying jobs is the subject for other studies.
        http://www.iwpr.org/publications/pubs/the-gender-wage-gap-by-occupation-updated-april-2011/at_download/file

        And again, the Berkeley gender bias case demonstrated this same paradox. There was a big admission rate gap overall, but not in any individual department. Girls just tended to apply to departments that had lower admission rates.

        1. Max, you wrote “I can see why the American Association of University Women may be biased against the null hypothesis, but I don’t see why PayScale would care either way.” Being a for-profit organization, it would seem to be to PayScale’s advantage to report results that are “surprising,” which will encourage potential customers to purchase access to their services for more complete information (as evidenced by Jamie’s experience trying to follow links in the article). So there is a possible incentive for them to have a bias in a particular direction, depending on what they think their target market will find surprising. In this case, based on their framing of the study, they appear to be invested in marketing the study as surprisingly overturning the conventional wisdom that there exists a gender-based wage gap. Which, incidentally, tacitly acknowledges that their finding is a minority opinion on the issue.

          1. Yeah, but had they found that women are not paid as much as their male peers, they could encourage women to sign up to find out how their salary stacks up.

          2. I agree, and in fact the smart thing to do would be to create multiple studies published under different “brands,” with various findings targeting various audiences. But that’s not really my point. My point is was just to raise one possibility for why PayScale would have a vested interest in a particular result. Regarding a for-profit survey as necessarily more objective than one by a non-profit advocacy organization strikes me as naive.

            Anyway, my point isn’t even especially important. Jamie has explained pretty convincingly that the PayScale study’s description of its methods illustrates that it lacks sufficient power to report the results that it claims. Whether or not it’s a biased result doesn’t matter: it’s not really a result.

      2. Here’s an example of a better-controlled, higher-quality study, focused on limited factors in gender disparity (hiring, starting salary, and mentoring) in one profession (academic scientific research):
        Science faculty’s subtle gender biases favor male students
        http://www.pnas.org/content/109/41/16474.full
        (This may have been discussed on Skepchick before; I’m not sure.)
        In brief, professional academic scientists were asked to assess a sample student job application for a lab manager position; half the scientists were given an application with a male name, half with a female name, but the applications were otherwise identical. The scientists rated the “male” application as more hireable, more mentorable, and attracting a higher starting salary. The “female” application was rated as more likeable. Of note, female scientists were no more egalitarian in their ratings than male scientists.

        1. Thanks. I’ve heard of this or similar studies, which were pretty convincing that there’s a real bias. This one found a 12% wage gap in the offered starting salary: $26,507.94 vs. $30,238.10. The AAUW study of actual salaries after controlling for various factors “found that a 7 percent difference in the earnings of male and female college graduates one year after graduation was still unexplained.”
          http://www.aauw.org/research/graduating-to-a-pay-gap/

    2. Managers joke that if women did equal work for less pay, they’d hire only women.

      By which you mean, women simply aren’t as good as men, and that’s why they’re paid less?

      1. If controlling for education and experience narrows the gap, then that suggests they have less education and experience. Or they accept a lower salary in exchange for something else like more flexible hours. Or they work part-time and get paid accordingly.

      2. I have actually read analyses that suggest that the fact that women do equal work for less pay partially accounts for the fact that during the current economic recession, men have a higher rate of unemployment than women do. So this “managers’ joke” may actually reflect economic reality.

        1. Did they look at individual sectors or across sectors? Because I would’ve thought it’s because the health and education sectors were less impacted than, say, construction and financial sectors.

          1. I don’t recall the details of the analysis that I read, so I can’t comment on its reliability; I bring it up simply to mention that the “joke” isn’t necessarily that far-fetched. Your question is certainly relevant, though, and it could be that the effect is due partially or entirely to differences in gender ratio across these sectors.

  10. Dear Jamie,
    First, let me applaud anyone who places skepticism above idealism. Bravo.
    Still…..
    Your analysis of the PayScale study implies that their conclusion (that the wage gap is much smaller than 23% or inexistent in cases) is wrong. I agree that the PayScale study is not particularly convincing (for reasons you mentioned), but the shrinking of the wage gap is an observation made by quite a few more reliable sources. Even the AAUW pamphlet you quoted grudgingly admits a tremendous shrinking of the gap takes place once you take individual choice into account:

    “In part, these pay gaps do reflect men’s and women’s choices, especially the
    choice of college major and the type of job pursued after graduation. … After accounting for college
    major, occupation, economic sector, hours worked, months unemployed
    since graduation, GPA, type of undergraduate institution, institution
    selectivity, age, geographical region, and marital status, Graduating to a Pay
    Gap found that a 7 percent difference in the earnings of male and female
    college graduates one year after graduation was still unexplained.”

    So even the pamphlet that begins by decrying the 23% wage gap, backs up and says, well, after we take into account equal jobs/experience/education, that gap is more like 7%. This number is backed up by a report funded by the US Labor Department ( http://www.consad.com/content/reports/Gender%20Wage%20Gap%20Final%20Report.pdf ) in 2009 finding a wage gap of 4.8-7.1 percent once individual choice differences were taken into account. Yes, there is *still* an unexplained gap, but it is clearly not anywhere close to 23%.

    2009, was several years ago, and the newest research (based on Census Bureau data) is showing the wage gap has actually reversed in some cases, specifically among women under 30 in certain major metros. The study has been all over, here is a blurb on it at the NYT:
    http://www.nytimes.com/2011/12/14/us/14iht-letter14.html?_r=0
    I have not been able to dig up the study itself and look at methodology, so I would remain skeptical.

    However, it makes sense. The mechanisms of capitalism will tend to favor decreased discrimination.

    1. You say that I am implying that when controlling for education and job position there is still a 23% gap between the wages of men and women. This is completely untrue and I implied no such thing. 23% is the raw difference and contains no controls. There are many, many studies within labor economics that show that the gap shrinks when controlling for job position. However, this post wasn’t about that. It was about the PayScale study and how their results tell you absolutely nothing about the wage gap. It wouldn’t matter if their result was exactly the same as all the other very good studies done on this issue, because their methodology was so terrible that their results would still have to be thrown out.

    2. msrenard, IN THEORY, the mechanisms of capitalism SHOULD favor decreased discrimination. In reality, it’s human beings who have their hands on the levers of the mechanisms of capitalism, and human beings can make foolish, biased and financially unsound decisions that are not punished by the market, or which are punished in a way that doesn’t discourage that behavior.

    3. msrenard – the CONSAD report did not look at the same type of sample as other studies, which tend to only focus on full-time, year round workers. Their sample from the CPS implicitly (and explicitly in fact, if you merely look at their tables 5/6, columns 2 and 3) include a larger number of part time workers that are female than are men, because that’s how the population statistic is, and for all intents and purposes the CPS is i.i.d. Maybe women do indeed “choose” to work part time more (even though the mere fact that they do doesn’t actually suggest that they “choose” such jobs), but you cannot say that the results in a study that oversample part time employees for one gender are really comparing equal jobs. You can’t explain a wage gap in the FTYR group by throwing in part time employees to the female sample.

      There are some other good points brought up here that aren’t really measurable using the CONSAD data (as in, we can’t tell how their results would be impacted by them because the study didn’t examine them), which should be taken into consideration I think:
      http://www.amptoons.com/blog/2010/11/26/how-the-consad-report-on-the-wage-gap-masks-sexism-instead-of-measuring-it/

  11. What perhaps gets looked over by some people too is that just because you can explain a wage gap in some way with a given controlling factor, that doesn’t mean that the sexism claim is false. If occupational segregation is the trump reason for the wage gap, rather than employers in each firm deciding “Hm, I’m just going to pay my female employees 77¢ to the male dollar,” that’s still a large problem. The causes of representation bias in occupations are not as in-your-face with their sexism because we have internalized societal gender roles, whereas we tend to have a better conception of “equal pay for equal work” and why that makes sense given what other values we hold as a society.

    I recall having a conversation with my parents about the gender wage gap about a year ago, and my father brought up the point too that companies hire women to higher positions in order to avoid the accusation of sexism. The “token female” as it were – and apparently this has happened in both of the industries where my parents work, auto and transportation.

    This is anecdotal of course. It should provide impetus all the same to better consider what motivating factors there are for what we observe, and what assumptions we take for granted about *reasons* for the wage gap, not just for the *causes*. And thus I hope everyone can see me coming back to my first statement.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Back to top button