How I test and evaluate audio equipment!

INTRODUCTION

Every Headphone IEM, Amp, DAP, and DAC that has come through my door for the past 2 years has been subjected to testing under as close a set of conditions that I could manage.  I do this because I go through many types of equipment, and I like to catalog them to hold better comparative ability in the future. 

I got sick of reading the subjective reviews on Head-Fi from people listening to X equipment with X variables, then making comparisons to other equipment and undergoing a completely different level of evaluation the next time around.  There was very simply little to no standardization of the process.  It got tired fast reading about how “this headphone has way more bass and is way more open than this etc.,” in most cases, people were recommending pieces off of a loose memory from the time that they owned said equipment with different songs and source equipment.  

The other peeve of mine and I’m sure once you have noticed, is audio reviewers flat out making up terminology and writing about a pair of headphones as if it was the twilight saga.  I am not a 12-year-old girl, and I’m sure most of the folk interested in high-end audio are not either. The Following is how I do it; it’s not necessarily the right method; I don’t think we will ever get to that stage, but I have found it works for me.  I hope you enjoy reading this, and please feel free to comment.

LET’S BEGIN

I had always kept loose listening notes from as far back as the release of Shure’s e2c IEM.  I wondered if I took just a few hours every time I got something new to use a standardized evaluation method to document its performance what sort of data I would accumulate.  The answer is a lot, far more than a hobby. It’s now an OCD-riddled obsession. I now have a massive catalog of reviews and listening notes, all performed under similar conditions and a compulsion to go through that process with every piece of gear I get.

I use Rhodia or Moleskine pads and either a Cross fountain or ballpoint. The one with the red Quiver pen holder is headphones; the orange earphones and my yellow one are for Amps Dacs DAPS.

I like to conduct my reviews in stages, as you can probably tell from my written reviews that everyone follows the same format.  This is in the hope that when someone is comparing one headphone to another that they can find a better ability to draw comparisons, and whilst this method is in no way perfect, it is an attempt at least to cut out some of the BS that I found so frustrating when I was searching for the perfect set of cans many years ago.

Stage 1:  Packaging

Firstly, I examine the packaging and presentation as I think this is important for 2 reasons; how does it make you feel? Does it add to the excitement? And second is how well packaged the product is to withstand delivery.

Stage 2:  Accessories

Next will be an evaluation of the accessories.  In this case, there is a flat base of expectation, and anything above gets marked higher.  For example, if I were buying a set of earphones, I would expect at least a 3 pack of varied size replacement tips.  The manufacturer may then add in bonuses such as additional tips, cable management solutions, and storage cases and can get extra points in this category for their quality (i.e., a food soft case does not equal a good hard case).

Stage 3: Build Quality / Aesthetics

After accessories are taken care of, I examine the build quality and style of the piece.  Does it feel sturdy? Are there any manufacturing defects? Does it feel premium or cheap?  How does it look? Are they eye-catching?  What environment would they fit into? How durable are jack points and strain reliefs? Etc.

Stage 4:  Fit

Now we are getting closer to the good part as the earphones are inserted or the IEM’s place in the ear.  Im about to take a look at the fit.  Are they comfortable? How is cable management? Length of the cable and how prone to microphonics it is.  In the case of headphones, I will look at the weight, the clamping force, and the pads’ comfort, among other things.

Stage 5:  Acclimatisation 

The product is then used for a minimum of 2 days in general use to remove any shock to the system and minimize perceived differences due to familiarity with a previous piece of equipment I was using.  Think of this as a palette cleansing of sorts before the main meal.

Stage 6: Critical listening and sound quality testing

Finally is the best part and the one to which I will go into greatest length on.  Until this point, my notebook is filled with around a page worth of scribble about the values and impressions gathered from the above sections.  The pages are with detailed testing notes as I work my way through the critical listening test playlist.

When listening, I always use the same tracks, without fail.  This has a few benefits, but most notable is that after listening to the same songs over and over and over, you begin to get an idea of how they should be performed.  If something is off, then it stands out like a sore thumb; I urge all enthusiasts to create a playlist specifically for this purpose.  The downside? Well, some of the songs you do get a bit sick of.

OK, let’s dive in.  I fire it up starting with classical, specifically Mozarts – Requiem (Colin Davis and LSO) and 1812 Overture by Tchaikovsky (Telarc Digital Canons).  Moving a little more modern now with Vanessa Mae – Storm and Ludovico Einaudi – Divenire.  The testing is so heavy in classical due to the large frequency range in the genre.  There is an instrument in there that will cover all frequencies well and is an excellent starting point for any evaluation.  There are wind instruments for highs, stringed for decay, and a lot is going on in orchestral tracks. To determine better separation learning to love classical is something I urge everyone to do both for the enjoyment and improve their technical listening ability.  

Vocals are up next, and for females, it’s Diana Krall –  and Nico’s – These Days (Krall managing the Upper and Nico the lower ranges of a female vocal).  For male Vocals Bocelli and …… wait for it…… Everlast – White Trash Beautiful (It has a wonderful grit and weight in the voice).

Electronic comes in Choas Royale – Anna Supernova, which is an excellent track to determine bass quantity and impact; sub-bass is inherently present throughout. Infected Mushroom – Pletzura is an extremely complicated track that goes through many movements and has lots of switching from right and left channels.

Rock? Led Zepplin – Whole Lotta Love for both the vocals and reproduction of Jimmy Pages iconic guitar riff.  Live rock I chose –  In this River by Black Label Society because of emotional vocals and environmental reasons, artifacts picked up from the crowd, and so on (plus the guitar solo is ridiculous, he’s playing a large part with his teeth).

Finally, I finish off with Amber Rubarth – Full moon in Paris (Chesky) for some binaural goodness.

I included the links to the youtube videos if you want to click, but this is only a guide. I urge you to grab the songs at least 320kbps mp3 or higher to get the full effect.

In all of these tracks, I listen for a range of factors from the following:   specific frequency response,  separation, decay, overall balance/tuning, separation, and many more.  I evaluate the highs, mids, and lows with specific sections I am familiar with in the above playlist, and through the entire test, I take scribbled notes that I come back to when writing my review to elaborate on.

WHATS NEXT?

So that’s it?  Not quite.  After the testing is done, I will use them to get more familiar with the product.  I take them out in public and on commutes to evaluate their fitness for purpose, and in the case of IEM’s each will come to the gym with me at least once to see how they perform. Why do I not take headphones to test at the gym? Well…It would help if you didn’t use headphones at the gym (a pet peeve).  Gym evaluation for the earphones doesn’t involve sitting around on a stationary bike etc.  I am a big fan of calisthenics, parallel bars, Muay Thai, and gymnastics, so the earphones are put through the absolute ringer.  Usually, cable over-the-ear earphones perform best in this field due to having less chance of popping out and having lowered microphonics.

In the Gym with the ProGuard P2+1  Expect every earphone to go through a few days of this.

During these next few days I then spend time listening to whatever I am into at that particular moment and will continue to make further observations on the performance and adding to the notebook.

The Ratings System

I rate each category out of 10. Again this is to find some level of quantification and comparability for the above categories between reviews.  Here is what the ratings are intended to mean.

1-3  Terrible, the lowest of the low

4-6  Average

7-8 Good

9-Very good almost top Tier

10-Top tier one of the best in class

Lets take for example the average category you could say that a 4 for example is just a step above Terrible but still passable whereas a 6 is still average but on the upper end and almost good.  This is meant purely as a guidance and I urge folks to read the words in each section rather than relying too heavily on a number. Rating headphones and earphones to me in minute decimals seems absurd very occasionally you will see a 9.5 when a headphone is just missing that special something to take it to the next level but there won’t be 4.3 6.1 etc.

Conclusion

As always the best way to test any headphones is to do it yourself and get some hands-on time with them.  Don’t get too hung up on incremental differences as the one thing I have learned is that the difference between a good set of headphones is very very small and usually down to tuning at the top end of the market.  We are talking at minute technical levels in this game, a game where the law of diminishing returns is ever-present the higher up a price scale you go. It’s fine to spend thousands extra to get to that next level but understand that expectations should be managed.

Beginners don’t get too hung up on the gear and don’t feel you need to go all-in at the beginning.  Progression is part of the fun of the hobby, experimenting and building up to your dream gear is a journey.  If you take your time building up you will have a greater appreciation for the nuances that make 1 piece of equipment technically better so take your time and develop your listening skills.

Always remember it should be about enjoying the music first.  Even with owning $1000’s worth of gear, I can still enjoy myself with a set of buds if needed and I would rather have good music I love on a crappy set of buds rather than the entire One Direction back catalog on a set of Sennheiser Orpheos.

Audiophile Onhttp://www.audiophileon.com
Audiophile On is a website dedicated to high-end audio products. With over 15 years as a reviewer, all articles are hand-written by one person to allow the comparison of products. Headphones, earphones, speakers, amplifiers, or DAC's we cover here.

Related Articles

6 COMMENTS

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Stay Connected

1,208FansLike
7,246FollowersFollow
2,535SubscribersSubscribe

Latest Articles