Edit: When I first heard about Moz's spam tool it sounded like they were marketing it as a completely automated disavow creator tool. After reading the comments on the Moz announcement, especially those from Rand, it sounds like this is not their intention and that the goal is to let people use this tool in conjunction with a manual review.
Still, this is not clear when you go to use the tool. There is still a button that encourages you to download Moz's list of potential spam links to your site and create a file and submit it to Google. I think that if the presentation of the tool were changed, it could be something that might be somewhat helpful if used in conjunction with a thorough manual link audit.
Because the Mozscape index is geared towards finding good links and not bad ones, it's going to fall short when we are trying to look at a site's unnatural links. You'll see, in my tests below that for the sites I tested, very few unnatural links were found, when in reality these sites had thousands of them.
I think that there are probably ways that people can use this tool and get helpful information from it. But I want to reiterate that it should not be used as a replacement for a thorough manual link audit.
Original article starts here:
I like Moz. The Moz Q&A is one of my favorite places to hang out online. I have several sites that I track with their rank checker. And, MozCon, their yearly marketing conference is beyond awesome. So I was definitely interested when Moz came out with their new spam score tool, marketed as a way for webmasters to review the links pointing to their site and determine which ones could potentially be harmful in the eyes of Google’s algorithms.
Moz has worked for over a year to create this tool, and I can understand why it has taken so long. Last summer, I put aside 3-4 hours of every day to work on something similar. This project was borne out of frustration over the automated link auditing tools that currently exist. In three years of almost exclusively working with sites dealing with unnatural links and algorithm issues, I have seen a lot of disavow files that were created using these tools. They are horrendously inaccurate. They will always miss a large number of unnatural links and quite often they’ll tell you to disavow links from really good sites. In one case a site owner was told to disavow incredible natural mentions from the BBC, the Guardian and other amazing links.
I want to tell you about this tool that I created...and then ditched. I also want to show you what happened when I tested Moz’s spam tool.
I created an automated link auditing tool.
My goal was to take the link auditing rules that I had in my head and create an algorithm that would classify a site’s links. I actually created a spam score, just like Moz has done and each link in a site’s link profile was labelled according to its potential risk level. I then spent weeks running this against disavow files that I had created by manually reviewing every single link to a site, and looking for ways to improve. I created a massive list of factors that were used in every decision. That summer, most nights I would dream in PHP and often not be able to sleep because I was developing a new way to analyze the data.
In some cases, my tool was 90% accurate when compared to my manual link audits. At first I was happy with this. But, this was nowhere near good enough. If a site had links from 1000 domains, a 90% accuracy would mean that there were links from 100 domains that I was either disavowing in error or missing from my disavow. We know that disavowing good links is a bad idea. I also know that any site that I have ever seen that has recovered from Penguin or a manual linking penalty has had to be insanely thorough in their disavow work. 90% was not going to cut it.
What I did next was divide links into three categories:
- Almost definitely unnatural.
- Almost definitely natural.
- Suggest manual review.
The links in this last category would show which rules they triggered to make them appear possibly unnatural. They would then give helpful advice on how to manually determine whether they were natural.
But after months and months of working on this I could not get a system that I felt was accurate enough for me to happily release to the public. I determined that it is not possible to algorithmically make disavow decisions with acceptable accuracy. Really, when you think of it, this is what Google is trying to do with Penguin...and they have incredibly smart engineers working on the algo day and night and they STILL are having horrible problems trying to get it right.
What I ended up doing is removing the majority of the automated auditing programming, and instead, have the program run a site’s links past my blacklist/whitelist that contains over 17,000 domains. I add to this list every time I do a link audit. And even with this in place, I still skim over these links to make sure my disavow decision is correct. I mostly use this tool internally as it saves me hours when I am manually auditing a site’s links. If you are interested in having me create a spreadsheet for your site using my tool, you can get more information on it here.
When I had heard that Moz had created a similar tool and was releasing it to the public, you can see why I was quite interested.
So I tested it.
Testing Moz’s spam tool on a site with good links
The first site that I ran through the spam tool was my veterinary site, askavetquestion.com. If you want to check it out, don’t try to glean SEO tips from it! It was the first website that I created and it is loaded with site structure issues, canonical issues and more. I don’t spend much time on it, but I used to. I have answered thousands of veterinary questions on this site. (If you don’t know, I was a veterinarian for years before getting into SEO.)
When I first started learning SEO for this site I did make a few links that would be considered unnatural today. They were links in low quality directories and some low quality article type links. I’d say there are about 10 of these links. But, there are a large number of truly natural mentions to this site. I was interested to see what Moz’s tool said:
The good news is that I only have two links that have a Moz spam score higher than 8, which is their criteria to disavow. But is this good news? What about the unnatural links that I previously made? I’m not proud of these links. But remember, I was new in the game, and learning SEO at the time...and this was prior to Penguin. Links like this were not picked up by the tool:
Ouch. It hurts to see those links. Don’t worry, they’re disavowed. If I could remember my passwords to these sites they’d be completely removed from the web. My point is, though, that Moz didn’t pick them up. If was going to disavow links, this is exactly the type of link that I should be disavowing.
The other bad news is that the spam score 8 link that Moz suggested disavowing is a completely natural mention. Someone wrote a news article and linked to me. I played no part in obtaining this link. It was not self-made for SEO purposes. There’s no need to disavow this link.
Confusing spam scores
I am concerned about the fact that so many of my good links pointing to my site are being given a spam score of 7 which, in fairness, is not enough for them to suggest disavow. But, if I put myself in the shoes of a small business owner who is desperate to clean up my link profile, I’d probably go ahead and disavow a link that Moz gives a spam score of 6 or 7. Here are a few of them:
http://crossroadsvet.com - This is a small animal practice that linked naturally to my chocolate toxicity calculator. It’s a good link.
http://www.redogulous.com/is-trifexis-safe-for-dogs/ - This is a natural link to an article that I wrote about some concerns I had about a particular medication. Definitely a good link.
http://www.benny-the-yorkie.com/ - Another natural link, from a small blog, to an article that I wrote.
There are others like this as well.
Now, maybe I’m being too harsh on Moz. To be fair, these links would not have been added to the disavow because they were not a spam score of 8 or higher. But, I am concerned that small business owners are going to do exactly what I did and say, “Oh. Moz says this link has a possibility of being unnatural.” Many will likely disavow good links.
Let’s look at some penalized sites
Here’s where I was most interested to dig in. I have a LARGE database of sites that have unnatural link issues. Now, for most of them, their bad links are disavowed. But, the Moz tool would not know this. So, let’s compare how Moz’s spam tool fares up against a manual disavow.
This site had a manual penalty last year and was loaded with directory and article spam. On manual review I found over 300 domains containing unnatural links. We disavowed all of these on the domain level. We were not able to remove very many links. The site saw no improvement after removing their manual penalty, but after a Penguin refresh they jumped from beyond page 6 to bottom of page 1 for most of their search terms.
Here is what the Moz spam tool showed me:
Oh no. Moz has suggested that there is one link that is unnatural and needs to be disavowed. One!
This site is one that had a manual penalty and is still struggling to get out from under Penguin. The manual review for these links took a LONG time. It’s a well known brand and they have a large number of links. Unfortunately the good links were tarnished when the company hired an SEO company. The SEO company made thousands of very bad links including directory links, paid blogroll spam and the creation of microsites. Our disavow file contains over 3000 domains.
Moz’s spam tool came up with 13 for us to disavow.
This is a small business that hired a low cost SEO company. The company made a lot of crappy directory links for them. This garnered them a manual penalty. We were able to get some links removed, but not many. As such, we identified and disavowed links from over 1500 domains.
I feel that this is the type of site that is Moz’s target audience. It’s a small business that uses Moz for ranking information and for learning SEO advice.
Remember that the spam tool can’t see what we have disavowed, so this site owner should be seeing hundreds, if not over a thousand unnatural links from the spam tool. Here is what I see:
I didn’t crop that list. That’s all I got. 4 domains. All good links. Not a single bad link in the list.
I thought that perhaps I had put in the wrong variant, so I tried https, non-www, etc. but I could not get Moz to show me any spam links.
I debated on whether to write this article, because I like Moz and I have benefitted so much from my relationship with them. But I think that it is vitally important for people to know that this spam tool should not be used to create a valid, complete disavow file.
Now perhaps the goal is not to create complete disavow files, but it does not come across this way, especially when Moz gives you the option of downloading a file that you can create and send to Google:
The tool has disclaimers which are good. But I fear that businesses are going to suffer if they use it. Some are going to disavow good links which will harm their site. Others will think they are filing a full disavow when really they have not scratched the surface when it comes to finding unnatural links.
When I do my link audits I always gather links from Webmaster Tools, Ahrefs, Majestic and Moz’s tool Open Site Explorer. There are other sources that I will use if available as well. My point is that you can’t do a thorough link audit based on one data source. I like Open Site Explorer when it comes to seeing the new good links to a site. I like the UX and I think that many small business owners feel comfortable using it. However, it’s not good at digging into the far depths of the web where scuzzy links exist. And those are the links that are most likely to cause problems.
Can Moz’s tool be improved upon? Certainly. But will it ever be a tool that you can use to help you create a disavow file? I’m not sure if that will happen. I could see myself using this tool if incorporated into the Moz toolbar, as an added metric when I am manually auditing links. That would be cool.
Who knows, maybe Moz will add a way for you to upload your links from other sources and then create a tool that helps you manually review those links….kind of like what I was trying to do. If this is done though, I think that there is more refining that needs to happen.
I was pleased to receive an email from Rand Fishkin asking for feedback from myself, Alan Bleiweiss and Sha Menz after Rand saw us discussing our concerns on Moz's tool on Twitter. I had wondered for a moment if perhaps this tool was an April Fool's joke:
Oops. I did not mean to belittle the Moz team.
Rand got some good feedback from Alan and Sha and I. I think that there is some potential for this tool to be improved so that it is helpful. But it has to be very very clear that it is not meant to be a completely automated disavow creator.
Some additional thoughts
After re-reading the announcement of the tool, I see that Moz does say that the tool should not be used to completely create a disavow file:
This needs to be made REALLY clear upon use of the tool. Many small business owners are going to think that it's an automated disavow creator. Perhaps removing the "create a disavow and send it to Google" button would be a good idea?
Please don’t hate me Moz. I love you guys. But I felt that I needed to publish this so that I don’t see more businesses hurt by improper disavowing. Many will not heed the disclaimers and will trust that because this tool is from Moz that they can rely on it fully. The last thing that we want to do is cause more harm to businesses who are already struggling with Google rankings.
Google update newsletter
Want an update when Google makes a big algorithm change or other announcement? Sign up here!
You are very polite Marie. 🙂 This tool shouldn’t have been released as it clearly is not ready for primetime based on what your testing shows. Just goes to underline the difficulty in developing automated tools that can algorithmically understand a link as I have lots of respect for the guy behind the tool.
I haven’t tried the tool myself but am curious if the links that it missed on the sites that had previous link penalties were even in the moz index? As you know, it is a pretty thin data set in comparison to other third party tools. If the links were in the index and were still missed then that would be quite troubling.
Thanks Paul. I am thinking that if Moz made it more clear that the tool was not made to be used to replace manual reviews, it would be so much better.
I have not tested whether the links that are missing are not in the Moz index, but I am guessing that that’s what is indeed happening.
Thank you for writing it as Moz is always open to feedback – even “hard” feedback. We’re a big, diverse family with different points of views, and the Moz community has definitely benefited from your expertise and contributions We learn from each other, and I consider you a valuable part of the community. Discussion is always welcome, and even necessary. Looking forward to further conversations and collaborations with you!
Disclaimer: While I work for Moz, I can’t speak for them on this particular
subject as others who worked on this particular feature are better
suited. That said, I can tell you that we take feedback like this very seriously.
Thanks so much Cyrus. I know that Moz has worked hard on this tool and I certainly don’t want to belittle that work. I can see that I actually could use the tool in my auditing process.
Again, my concern is that it is made very obvious that the tool is not to be used in the place of a full manual link audit.
But then again, considering that that’s my main line of business, perhaps I am biased. 🙂
I ran it on several sites that I happen to know had a manual spam penalty, and one of them scored more than a 2/17.
This article provides the real insight of how ambiguous the Moz Spam Score can be.
Whilst the metric does provide a simplified metric for link quality grading I feel there are too many outlier URLS that score highly on this scale that are genuine decent quality URLs to have a link from.
Moz is considered gospel by a number of “part-time” SEO focused people who are often in influential positions in a business such as directors, CEOs etc, who tend to use this kind of metric to simplify things. This article proves things cannot always be simplified if you want to do best practice SEO which we know can sometimes be more of an art than a science.
Thank you so much for the informative post, recently we undertaken immigration site for SEO, after looking back its back links on MOZ OSE tool, we were worried of its spam score 7. After reading all your post I am now fully OK.
Thanks for sharing your test. I hope they have improved on the tool now?
I have heard that they have made big improvements, but haven’t been able to check it out just yet.
Hi Marie, do you know how often the software is updated? I redirected a domain but the DA has not pick up.
Things have changed a lot with Moz since I wrote this post. You should see updated DA metrics within 24 hours now I believe. But, Moz may not pick up every change that you do, so if things don’t change, it might just be a Moz issue. Google likely will still see the redirects even if the Moz stats don’t reflect the change.
Hi Marie Haynes,
A few days back MOZ changed the free tool to paid. But MOZ provides us a free usage of the tool for a monthly max usage of 10 times. We can check the spam score and ranking using the Moz toolbar. Today the toolbar is also not working. Any idea? In the future all the SEO tools will be paid. No free tools available. 🙁
Hi Marie, what is your view about the new link explorer from Moz?
I’ve played around with it a little and it’s definitely a bigger index. I think they’ve made some great improvements.
This is very interesting and helpful content in this post and i am very gratefull to you for sharing such a great content.