With Feeling

August 21, 2009

Dapple — don’t do it

Filed under: iPhone apps — Tags: , , , , , — mechaferret @ 4:46 pm

Recently, I was browsing my feedly as usual a few weeks ago, and read this post by Jeff Atwood, about the wonders of the iPhone App Store. Since I’m not uninterested in developing iPhone apps¹, I read it with extra attention, and of course had to check out the cautionary links: one analyzing the price pressures caused by the App Store as a destructive race to the bottom, and the one right upfront in the comments about an iPhone app that isn’t making money, a game called Dapple.

The Dapple article proved quite fascinating, albeit depressing. Here was someone who had spent six months developing a novel game, and had gotten excellent reviews on popular sites, yet had only managed to sell a few hundred copies, at $1.99 each. He spent a good deal of time trying to analyze the problems and describing his next moves, both in that post and in a previous post Despite much thoughtful analysis and strategizing, it wasn’t clear what would actually work to make Dapple become a breakout app.

There was one obvious thing I could do, of course: buy the app.



I’m not a compulsive consumer of mobile phone games, although I have put in some time with Brickbreaker back when I had a Blackberry and Minesweeper on my iPhone, but it did look like the sort of game I would like: simple, low-dexterity, pattern-matching, with the additional twist of color mixing. The first time I played, it seemed very pretty but a little unpredictable: sometimes one click would remove four paint blotches, and sometimes it would clear half the screen. I put it away to check out more later.

One of my favorite places to play mobile phone games is in bed before I go to sleep, so I tried it out again a few nights later. One round, then another… I woke up at 5AM. My phone was lying next to me in bed instead of on the table where it is supposed to be. I had been so unwilling to stop playing that I had actually fallen asleep with my iPhone in hand.

It only got worse from there. I was meeting a friend around 8PM, stopped at home to pick up a few things, and decided to play “just one game”. Several games and fifty minutes later, I finally dragged myself away. I started seeing images of the colored paint dapples whenever I closed my eyes. Then I started seeing them when my eyes were open on any patterned surface: walls, floors, pieces of paper. Dapple had taken over my brain.

While I’ve gotten better at my Dapple playing lately (I don’t feel the need to play all the time, and I haven’t been late to any more events), I still often feel the ineluctable pull to just play a few games. So, my advice to the author of Dapple: don’t worry — given that you’ve written one of those games that’s more addictive than heroin, you will find your audience eventually.

And to those of you considering buying the game — unless you’re the sort of person who enjoys compulsive game playing — just don’t do it.

¹ Two questions about the iPhone SDK:

  1. Why, why, why does it only allow one app to execute at a time? As a long-time multithreaded, multiprocess developer, it’s an intolerable restriction. Even my old Blackberry let multiple apps run until I got around to killing them. And the push notification doesn’t do you any good if you just want to run in the background, because it requires active user confirmation to launch your app.
  2. Why does the SDK strip all EXIF information from files providing by UIImagePickerController? What’s the point of letting the user select a photo they took on the iPhone, which is nicely geotagged, if your app can’t get the geotag information? Why do you have to use the filesystem to get the full EXIF data?

It’s almost as if the iPhone is specifically designed to thwart any third-party use of its best features. Gee, that doesn’t sound like Apple at all….


August 14, 2009

Goldilocks and premature optimization

Filed under: Performance optimization — Tags: , , , , — mechaferret @ 1:54 am

Once upon a time there was a very famous computer scientist named Donald Knuth. He heard a quote from Tony Hoare and liked it so much that everyone now attributes it to him: “Premature optimization is the root of all evil.” Many people have run up against and been frustrated by misinterpretation or overapplication of this quote.  Some of them have written cogent analyses of what is wrong with the quote (such as this one, which also contains a nice selection of links to other articles on the topic).

Now, this quote is wrong for many reasons. It is dangerous for two reasons:

  1. People want to believe it, because performance optimization is not easy and it can only really be learned through experience. Or, as Charles Cook says in his post:

    A good software developer will do this automatically, having developed a feel for where performance issues will cause problems. An inexperienced developer will not bother, misguidedly believing that a bit of fine tuning at a later stage will fix any problems.

    The inexperienced developer is much happier if he can point to a quote to justify his beliefs, instead of knowing that he has a lot of time and work ahead before he’ll actually understand the nuances of performance.

  2. There is some truth to it. Premature optimization is a bad idea. It’s just that knowing when performance optimization is not premature is at least as hard as doing performance optimization, and is even more something that can only be learned through experience.

So, I hereby offer up several recent examples of performance optimization I’ve done that may be useful for helping developers tune their own optimization skills and get a sense for when optimization is not premature. Like all good fables, this post contains a full range of examples: too late, too early, and just right.

Too late

The Papa Bear example is a set of revenue reports for an internal web application.

Revenue could be classified by year, business period cycle, sales region, user affiliation, or type. The executive dashboard provides for a wide array of reports that can select by or aggregate over any of these, plus aggregate by month/week/day.

Tip #1: “aggregated” “reports” is the always a red flag that suggests that the need for performance optimization is just around the corner.

Revenue is contained in an orders table which is 50K rows. User affiliation is obtained through a join to the user table (100K rows) and the affiliate table (50k rows). The sales_region table is around 1000 rows. A given report often returns 1000 rows, one for each sales region.

Tip #2: Joining a table over 100K rows to a table over 50K rows is rarely something you want to do in real time.

The reports ran OK, if slowly, when first deployed without optimization. But by the time I got to this reporting system, some of the reports (like the “All regions” dashboard) were taking so long to run that Apache was timing them out; no one had seen them for several days.

Obviously, performance optimization was considered rather too late.

Fix: I created an aggregation table and pre-joined the revenue to its attribute fields in a classic star schema fact table design. Total database usage for the “All regions” dashboard went from 50+ seconds to under 3 seconds (there was still more to optimize in the revenue projections, but that was deemed not worth the effort to gain an additional few seconds).

Too early

The Mama Bear example owes full credit to me: I optimized this one way too early.

My company had its own custom phone call routing software, because it sold toll-free numbers to many different clients. An earlier version of the software had accessed the database every time a call came in to determine which actual number to connect the toll-free call with; that software was prone to long delays in connecting calls as it tried to access the database (on a different server), and even failures if the database wasn’t accessible. So when I rewrote the software I cached all the toll-free->local mappings in an in-memory hashtable, which both made lookup incredibly fast and allowed a routing server to remain up even if the database went down. So far, so good.

Then we added geo-targeted routing, in which a call to a toll-free number could be routed to one of several (or several hundred) numbers depending on the nominal region of the caller’s phone number, with a single fallback number to default to if the location didn’t have a specific geo-target. Obviously, I didn’t want to risk the same performance issues we’d had with the direct routing. So I cached all of the geo-targeted routings in memory as well.

Tip #3: When caching in memory to improve performance, make sure to take into consideration the size of what you are caching.

Tip #4: The problem you solved before might not be the problem you are solving now. Did all of the geotargets need to work flawlessly, or just the default number?

The new geo-targeted routing lookup certainly was fast. Except… it also had an enormous memory footprint. While less than 5% or so of our numbers had georouting, they took up 99% of the memory of the app. We very quickly ran up against the limitations of Java heap size on Linux and had to start doing the hacks that allowed for larger heaps. It also became a much more difficult problem to keep this much larger, double-hashed cache in sync with the actual routing as specified in the database. When I left, they were just looking into switching back to database lookup for geo-targeted routing, keeping only the default number in cache. Which, given the small number of customers using this service, would have been the right answer all along.

Just right

I’m quite proud of Baby Bear — it almost qualifies as “just-in-time” performance optimization.

We had a web site on which customers could search user profiles. They could search by a wide variety of things, including location, name, education, job history, specializations, and paragraph-long answers to four questions. Except for location, all of the searching was full-text: we took the text they entered into the search box and searched all of the fields for matches.

Tip #5: Full-text search can get slow when done directly in SQL.

We knew that, but we were in a hurry to launch. So we implemented the search directly in SQL. And at launch it was fine. Two months later, when we had 250 profiles and started our first marketing push, it was fine. At the end of the year, we had 1000 profiles, and it was starting to get a little slow, taking maybe 10 seconds to come back from all but location searches.

Tip #6: Always monitor performance and pay attention to anything that is slowing noticeably.

I paid attention. I assigned myself “Worry about profile search performance” as a to-do in Basecamp. Then I talked with one of the developers and had him change the search to use sphinx so that we could do our full-text search using a full-text search engine. Search time dropped from 10 seconds to well under a second. No one noticed. (It seems that 10 seconds was not quite long enough to be irritating.)

Six months later, we had an even bigger marketing push and got up to 7000 profiles. Given our previous benchmarks, search response time would have gone up to at least 70 seconds if we hadn’t made the change, which definitely would have been long enough to be irritating. But because we had switched to sphinx, search times… remained under a second.

Tip #7: The highest form of success in performance optimization is having no one notice that you did it, even as usage scales drastically.

In fact, Tip #7 may be the morale of this story. The first example wasn’t successful performance optimization because everyone noticed the dashboard timing out. The second example wasn’t successful because people noticed the excessive memory usage. The third example was successful because no one noticed anything at all.

And that’s the end of our story. All the programs perform better, all the readers are a little bit better at performance optimization, and everyone lived happily ever after….

August 7, 2009

It just feels right

Filed under: Software management — Tags: , , — mechaferret @ 1:12 am

I majored in mathematics because it was easy. A large part of why it was easy was because it was precise: formulas were either correct or they weren’t, proofs either worked or they had logical errors, and the professor’s judgment on that wouldn’t alter based on whether or not he knew and liked me, or even whether or not I showed up. So I took a lot of math courses, skipped a lot of classes in favor of seeing bands every night, and graduated with honors anyway. That would have been significantly harder if I’d been an English major. I continued on in this way through a PhD and a year of teaching.

Then I switched careers, changing software development from a hobby supplementing my impoverished grad-school lifestyle to a full-time job (and getting to live in Los Angeles instead of the Midwest in the process). I had pretty much assumed that software development would be as precise as mathematics, but easier. Much to my surprise, I discovered instead that software development is messy, and complex, and imprecise, and that the key to building working applications on time was building teams of people, with all of the emotion and indeterminacy that inevitably accompanies people-centric activities. And, even more to my surprise, that was the interesting part. I got up in the morning (late morning… I was still a developer) and went to work because I enjoyed putting together concepts, technologies, and people into something that produced the best possible software. I didn’t want to just do the things with the precise answers after all. I wanted to think about, build, and be part of a team.

One obvious consequence of that was that there were no longer precise answers to the questions we were trying to solve. How long would adding this feature take? Is it better to design the code as five classes or three? Not that these were unanswerable questions. They just didn’t have a single right answer, and even though some answers were right and others were wrong, it wasn’t always possible to explain exactly why. I thought the Asterisk migration would take about 3 months because it just feels right.¹

Of course, I’m hardly the first person to have made the mistake of thinking that, because software is based in hard science and has very exacting requirements (either the code compiles or it doesn’t, either the credit card gets charged the right amount or you don’t make money), it has precise answers, and its successful execution requires all logic and no feeling. In actuality, the predictability of a software development project is significantly greater than that of, say, producing a Camry clutch assembly, although it is not quite as unpredictable as writing a great American novel. In fact, in its volatility and complexity, it exactly resembles producing a scripted TV show: there’s a schedule you have to (try to) stick to, but a lot of highly variable parts go into that schedule, and sometimes you either have to skimp or run late to get it all working.

So, in the end, being a math major got me into a career as imprecise and intuition-driven as that of any English major, with an additional helping of dissonance from the expectations of many that it should not involve any of those traits. Then again, the challenge of overcoming the dissonance just makes software development that much more imprecise… and interesting…

¹ Points should be awarded here for using the title in the content… as usual, it’s not clear whether they are bonuses or demerits.

Create a free website or blog at WordPress.com.