Do You Trust Woodworking Magazine Tool Reviews?

An insightful thread at FamilyWoodworking really got my wheels turning in regards to magazine tool reviews: Whats your view on Magazine Tool Reviews please vote in the poll. For the record, 61% of the Family does believe that woodworking magazine reviews have credibility while 32% do NOT (at the time I posted this).

>> 5 Problems with Trusting Magazine Power Tool Reviews:
These are problems that Family Woodworking members noted that crop up sometimes, not standing issues with every single magazine power tool review. I hold magazine reviews in high regard – as a writer/tool researcher for ToolCrib.com. Here are some of the problems mentioned in the discussion:

1) Missing Model Syndrome
Where there’s a new or even old model missing – like if someone left the Bosch Colt out of a trim router comparison.

2) Improper Category Groupings/Comparisons
Say if a hybrid saw ended up in a cabinet saw comparison.

3) Bias in Testing to Favor Advertisers
This is alleged of course. One kind of bias cited is when the tests themselves are skewed to favor the new feature set of a new tool. That happens to be advertised heavily in the magazine…

4) Won’t Say It’s a Total Dud
Reviewers seem reluctant to proclaim a no-buy status for tools, when often times they should. Some think this is out of fear of losing advertising dollars.

5) Tester Overload – Too Many Tools, Too Many Comparison Points
Sometimes there’s just too much data for one person to sort through. You can see this especially when it comes to the writing.

>> Problems with Trusting Forum Recommendations:
There are some inherent problems with trusting forum recommendations too – the same pretty much goes for user reviews in Amazon or Epinions or the like.

1) “Purchase Justification” Bias
This was well put in the Family Woodworking thread by Bill Satko: “It is hard to spend that money on a tool and be honest about it. There is a lot of justification that we need to do to feel good about the purchase. We also tend to be brand loyal. Just look at the Lie Nielson vs Vertias handplane posts in SMC & Knots.”

2) Haven’t Used All Tools in Category
How many people IRL (in real life) have used all the tools in a particular category? Even the seasoned veteran woodworker hasn’t used ALL the band saws out there. Maybe half of them, but never all of them.

>> What Does the Perfect Review Look Like?
There’s no perfect review except for YOUR review, but here are some of the ideas from the thread.

1) Consumer Reports
No ads = no bias. Still, they sometimes leave out important models, and can they really justify extending into the power tool space?

2) Multiple Unbiased Forum Reviewers
While this would be pricey for manufacturers, this idea from Chuck Thoits would give other consumers a pretty good direction to go in: “So the only way to fix that is for all the tool companies to send one of each of there tools to every member here with a list of tests. We test them and send the info to a central location to be compiled into one test result. You know. We had one hundred WW test these tools and the XYZ came out on top with 79% voting it best in its class. And here is what they said. XYZ cut faster had less hand fatigue blaa blaa blaa Tell me if you read a tool test like that you wouldn’t know which one you where going to look for in the store. ”

3) Multiple Editorial Reviews
This concept comes from Art Mulder, a sometimes writer for Canadian Home Woodworker: “That is one of the things I like about reviews in Canadian Home Workshop magazine — they have three reviewers. Each of the three tries out ALL the tools, puts them through the same paces, and then writes up their thoughts. They never totally agree. And you get different things like “I have small hands and this tool fit my hands better”, and so on.”

4) Aggregate Magazine Reviews, Forum Reviews and User Feedback
For the comparison reviews I write for ToolCrib I find all the magazine reviews I can, all the blog and web reviews as well as all the forum recommendations I can and stir them together in a very unscientific way… While I think these comparisons serve as a great starting point for purchase decision making, they could benefit from some objective process and some sort of ranking format (which I am working on by the way). MetaCritic and ConsumerSearch are two of my inspirations for this direction.

Here are a few of recent examples:
Top 16 Trim Routers: Bosch Colt vs. Makita vs. PC310 vs. Ridgid R2400 and MORE!
Top 10 Hybrid Table Saws: Craftsman vs. Grizzly vs. Steel City vs. Jet and MORE!
Best 14? Bandsaws: Grizzly vs. Rikon vs. Powermatic vs. Laguna and More

So… how do you make your power tool purchase decisions?

Read more at Family Woodworking: Whats your view on Magazine Tool Reviews?