Developers Don't Polish, Testers Do.
Posted on Wed Feb 23 2011 00:53:18 GMT+0000 (UTC)
Last week I posted about some of my first experiences in the iOS App Store and where I felt my weaknesses were in grabbing more appeal - particularly in marketing and depth of content. As we discussed in the cocos2d forums, I forgot an important one many of us underestimate the need for on launch - polish. In my case, I had a particular polish issue with my menus and small buttons which didn't align perfectly with your finger location. This was frustrating even some of the most enthusiastic players.
So why did I let my game go into market with a slightly funky UI?
It can be misleading when we see almost every new app update itself several times to fix bugs after launch. It seems socially acceptable these days, almost, for an application to crash occasionally. When I was saying my product was almost ready, friends and family would argue "Well, you'll never get it perfect without dragging it on for months, besides, you can support it with updates after launch right? That's what everyone does."
These are only illusions of the App Store though, don't take it as an invitation to finish your product off in-market. I think I let this illusion led me to polish less than I should have.
I would also argue that polish is easier said than done due to its subjectivity - We all define polish a little differently based on our perceptions of what is important. There's only one clear way I see of fixing that.
The less user tests you put your game through before you launch, the more subjective 'polish' will be.
If you live in a cave and program your application from beginning to end without seeing the light of day, you would argue that you polished that application along the way, wouldn't you? After all, if you compare what you started with to what you got in the end, surely the end result is more polished.
Developers don't polish their own games. They can only change, improve or optimize them. Polish is something that needs to be consensually agreed upon by a larger number of people. Testing can reveal what changes/improvements/optimizations the developer should make to achieve greater polish.
I don't want to say that we can't look at our own games and see them for their faults and imperfections, but how much do we really want to rely on our own perceptions when we are about to try to sell at least thousands of copies of it to people? Isn't it possible our judgement is clouded by hours spent tweaking the project?
How would you feel as an end-user knowing you're being sold something that was built in a cave ?
My testing experiences with Pixel Fighters
I tested two ways. Most people I just sent a testing copy to and got feedback at their convenience. I also conducted a couple of tests silently watching over someone's shoulder (not giving game hints). Let me say, one over-the-shoulder test is worth at least 5 send out tests to me now. During send-out tests, some people would come back with "It's fun" and then I had to probe them for more. With an over-the-shoulder test, it is like a constant stream of feedback. After my first test, I had pages of complaints and fixes even though my subject wasn't saying much. I could see for myself where he struggled, what wasn't clear and how long it was taking.
Send out tests are still very important, especially for testing app usage over a longer period of time but make sure you do a few over-the-shoulder tests!
My testing weaknesses with Pixel Fighters
While I tested throughout my process, still I would say not enough not achieve greater polish. I've learned a lot about how to get even more out of it next time.
When you are very personally attached to what you develop, it's hard sometimes to take criticism in stride, especially when even you know its not finished yet. I think I let this discourage me too much from more thorough testing.
Its also sometimes difficult to accept too much criticism on an aspect you have spent a lot of time coding and toying with. In my case, it was my menus which had been quite a bit of work to figured out. I spent so much time using the in game menu I had built, I didn't even realize the funny way I got used to pointing at the option I wanted. People mentioned some awkward UI and I made an indirect attempt to improve it.
I also tested with too few people in most batches and I think this prevented me from having a predominant user opinion over my own thoughts. Sometimes I would take feedback from one person only before developing further. Worse yet, after some of my final testing, I continued to change/improve/optimize the game and then submitted to Apple. The absolute final product should have gone through an even more rigorous stage of testing but I spared it since the last round of testing had been recent enough.
Mistakes aside, testing was still great. Testing helped me push myself to constantly raise to bar on my game. There were also some major changes to my game that were inspired by friends, family and cocos2d testers.
I look forward to the next time I test because I know how much more I am going to get out of it.
"It's not Rocket Surgery"
If you are a small shop or independent developer like myself, it is unlikely you will have the budget or human resources to test as thoroughly and constantly as would be ideal. All we can do is our absolute best to catch the most glaring issues before entering the market. I strongly recommend Steve Krug's "Rocket Surgery Made Easy" if you have any doubts about your capability to user test as a one man band. While the book discusses testing for websites, it is all the same for iPhone apps.
No substitute for going live
In the end, the App Store is going to quickly whip your butt into better shape if you have any inclination to support your game (which you should). In no time it was echoed that my UI wasn't quite right and also a new one I hadn't heard in any testing - My game animations were too slow for some people's patience.
My second update is awaiting Apple's review, but the game's UI buttons and pace have significantly improved. I have to say, I'm already so much happier with my game than what I entered the market with that I'm actually a little embarrassed to have released it as it was in January. Its not that the game was unplayable or anything before, but even I enjoy playing it so much more with that extra polish I couldn't achieve alone.