Remember the old metaphysical question: “if a tree falls in the forest, and no one’s around to hear it, does it make a sound?”

Well, here’s an equally unanswerable question: “if someone designs a web site, and no one’s around to measure its performance, was it a success?” If you designed or manage that web site, your job depends on being able to answer that question. However, many user experience designers have historically avoided the question with the enigmatic and frustrating non-answer: “it depends.” Well, the time is overdue for us to take ownership of measuring the performance of our designs. We can start by understanding what performance means.

“Performance” implies that someone is observing the performance – the audience. And there’s an implication of critique, or measurement of the performance. In order to guide the evolution of the design to fulfill some strategy, the design’s performance must be measured.

Performance implies an audience.

The performance of any creative effort is subject to interpretation by the people observing the performance, and those people are called “the audience”. To reiterate the argument I made in Part 1, user experience designers aren’t creating art for self-expression. We’re creating art for use. My design is made to perform a particular function, made useful by having an audience that values the design’s function.

If I design a web application behind a firewall that no one ever finds or uses, what have I accomplished, other than self-expression? By definition, user experience design requires an other (the user) to experience the work. My design must be used in order to fulfill its destiny.

Performance implies measurement.

The audience is neither monolithic nor homogeneous. Each audience member will have their own unique set of values, motivations, cultural filters, and personality traits, which will affect how they view the performance. They will each bring their own criteria by which they will judge the performance, whether they know it or not.

In the movie theater, I might judge a performance based on the quality of the story, the set design, or the acting. A film critic might judge the character development, the narrative structure, or the impact on the world of filmmaking. A marketer might focus on the product placements, the brand opportunities, or the potential for a companion video game and a line of toys. Conversely, my mom might judge the performance on how comfortable the seats were, the temperature of the air conditioning, or the cleanliness of the restrooms.

Thankfully, there are some criteria that are common amongst audience members. We can group people together who have common attributes and common ways of judging performance. In the digital marketing business, they are called segments. In user experience design, these groups are called personas (I’ll save the discussion of the difference between marketing segments and personas for another day). For each audience segment or persona, the user experience designer can identify which criteria they will design for.

Who you perform for determines what you measure.

By choosing an audience to please, the designer must deal with the accompanying criteria that audience uses to judge performance. The criteria you choose to measure determines how you will design for performance.

For a typical web site, here are just a few examples of the different types of audiences and their different measures of performance:

You might notice that these last two are more cross-disciplinary and cross-audience. This is because they’re based on values, rather than on roles. Value-centered design (see Jess McMullin’s Boxes & Arrows article) and value-sensitive design (see Batya Friedman’s work at UW) are approaches to design that define success by how well the design aligns to commonly-held definitions of worth or ethics, respectively (I’m oversimplifying).

Some of the above approaches are better than others for different contexts. A user experience designer might gravitate toward a completely user-centered approach, but their job might depend on business- or politics-centered measurements. For a company whose entire business is conducted through their web site, depending solely on user-centered measurements of success might have disastrous consequences. Especially if the web server is crashing due to some poorly tested code.

As an online user experience designer, I must deliver a design that the users love, but that is also on time, within budget, functional, revenue-generating, and adheres to the values of the company and its customers. I also need to create an experience that will evolve as those values and the market landscape change over time.

How you perform determines how you EVOLVE.

Which brings us back around to evolution. Adapting a design to the changing environment will ensure its survival. Even if you don’t intentionally vary the design, your site or product WILL change over time. HOW it changes depends on how you measure its performance. What do you value? The user? The money? The environment? Design for the audience(s) you must satisfy to survive. Measure what matters to that audience. Keep what works, toss out what doesn’t, explore new approaches, measure again, and repeat.

Next up: Part 3 – Process & Deliverables

Evolution is really very simple: life begets life, with a little variation in each iteration (due to entropy). Some variants perform better than others, giving them an advantage in the face of predators and environmental factors. Over time and zillions of variations, specialization occurs.

Evolution in a nutshell

1. Live
2. Reproduce with variation
3. What performs well, survives
4. What survives, reproduces
5. Repeat

Evolution is not random

It’s important to note that the process of evolution is not random. If it were, we’d have 3-eyed frogs with 6 legs. The reason we don’t is that, if 3-eyed, 6-legged frogs did occur, they didn’t survive any better than the 2-eyed, 4-legged frogs. That’s not random. 2-eyed, 4-legged frogs are simply better at surviving.

What is random about evolution is the mutation and the environment in which this cycle of life and death take place. In nature, evolution is not intentionally guided, as far as we know. Catastrophic events and chance encounters with predators cannot be perfectly controlled in the wild, so evolution tends to take a wandering course.

Evolution with a purpose = breeding

If you wanted to intelligently guide the evolution of an organism, you’d need to look no further than your local AKC kennel or county fair. Humans have been breeding plants and animals to select for certain characteristics for millennia. Corn, rice, wheat, bananas, apples, chickens, cows, horses, goats, sheep, dogs, cats – practically every plant and animal we eat or live with has been bred intentionally. They didn’t evolve naturally – humans pollinated, inseminated, incubated, hybridized, and culled millions of generations of wild organisms to get the highly specialized cadre of useful, nutritious, cuddly, non-toxic, and benign farmyard animals and plants we eat and love.

What’s different between evolution and breeding is that most of the random variables have been removed. Farms are very controlled environments, with relatively few random predator attacks, competitors, or catastrophic weather events. This is because we humans protect our plants and animals with weeding, fencing, shelter, medicine, and a steady supply of food. The randomness of mutation remains but is minimized by selecting breeding pairs that resemble the desired characteristics as much as possible.

These domesticated breeds wouldn’t survive in the wild. Instead, they have helped us survive and become the dominant species on the planet. It was the transition to agriculture – intentional cultivation of plants and animals – that alleviated the need for humans to wipe entire herds of mastodons to feed their burgeoning population. In short, guided evolution allowed humans and their companion organisms to survive. It also allowed and all the beauty and variety of human culture, knowledge, and art to blossom. And it might – in a return to sustainable agriculture – be what allows us to stop catastrophic climate change and survive for another million years.

Art with a purpose = design

Art is self-expression. It’s an act of creation, or, in some cases, an act of destruction or deconstruction. Whatever art is, it rarely has a purpose beyond making visible/audible/legible/tangible the vision of the individual artist. As with the natural world, the world of art evolves based on predation, competition, and the sociopolitical weather. Like evolution, art is not random, but it isn’t necessarily guided either. At least, not guided sufficiently by factors outside the artist.

Design, on the other hand, is guided art. Design is art that is meant to be used by someone other than the artist. There are varying degrees to this – and probably libraries full of these that discuss this subject ad nauseam – but my operational definition of design is art with a purpose. User experience designers are in the business of creating systems that fulfill a purpose – namely, meeting the needs of users.

Evolutionary Design

The problem with the design agency world, where I work, is that design is often confused with art. We strive to create the One True Design, a work of art so perfect that it will be viewed as useful, usable, elegant, and beautiful to all. And it will make money, further the brand, and pacify business stakeholders and partners. We are so enamored with this idea that we structure our projects to conduct research, create a design concept, test that concept, and deliver the One True Design after a few rounds of revisions. Most designs fail the first time. Sometimes miserably. And if you bet the farm on that one design, you only get that one chance.

Here’s an alternative: forget the One True Design. It is an illusion. There are no perfect designs for all time, there are only appropriate designs for specific contexts. Just as orchids and tree frogs are perform well in their niches in the tropical rain forest but not the frozen tundra, certain designs only work well in certain environments. And those environments are subject to change over time. Thus, design must change along with the often catastrophic shifts in markets, customer needs, and business strategy.

At this moment in 2009, we are all facing catastrophic economic change. If we want to create user experiences that have a better chance of surviving the chaos of the market, it may help to adopt the same guided evolutionary approach in our design process.

Evolutionary design process

1. Strategize
2. Generate design variants
3. What performs well, survives
4. What survives, generates more variants
5. Repeat

This looks very similar to the guided evolutionary process (i.e. breeding), with one key difference: randomization. Unlike with breeding dogs or corn, a guided evolutionary approach to design could be nearly free of random factors. Designers can control the mutation in design variants, creating only those variants that have a chance of surviving. No 3-eyed, 6-legged frogs. But maybe a frog with wings. Or a frog with X-ray vision. Or a frog with the ability to digest pesticides. However, in order to understand whether our flying superfrog has left us better off than with a regular frog, we must measure its performance against some goal. Otherwise, we’re just wasting our time and torturing frogs.

Next: Part 2 – Performance-Driven Design

To the panelists, evolution was a thing to fear, as if we are as powerless as dinosaurs in the face of an approaching asteroid. I found myself wondering how we might embrace evolution as a tool for advancing the discipline or, more ambitious, develop an evolution-inspired approach to advancing the process of how IA is done. How would we re-conceive of the process of IA as the guided evolution of information spaces? Might we guide the evolution of the discipline itself, utilizing evolutionary principles to improve what how IA is done and ensure the survival of IA as a discipline?

That evening, I realized that the book I’m co-writing (more on this later) on the blending of analytics, optimization, and user experience already described this evolutionary approach. Early the next morning, I signed up for a session slot to co-present the idea with fellow ZAAZ user experience architect (and optimization test designer) Rachel Elkington. Our session title: “Darwin Does Design: Measuring & Optimizing the User Experience.”

You can view the slides of our presentation on SlideShare. Audio coming soon!

Stop thinking about designing the perfect system. Perfection is expensive, illusory, and unattainable. Perfection will get you laid off. Good-enough will allow you to survive until the next round.

Design sustainable systems. Create revenue, efficiency, and value feedback loops. Design iterative workflows and self-sustaining user flows that add value, increase engagement, and make the system smarter.

Just design something that works. If it can’t be built on time and under budget, you have failed as a designer. Every subsequent iteration is a chance to improve the quality and return-on-investment of your user experience. But there must be something in place to iterate on.

Understand how your users will evolve. Look beyond the first site visit or the first use. Look at the entire lifetime relationship you form with your users. Design for their first visit, their 10th visit, and their 100th visit. And know how to measure the value of each visit.

Measure the performance of your design. Measure what will keep you your job — revenue, customer satisfaction, efficiency. Measure pre-design. Measure post-design. Fix what’s broken and measure the performance of your fixes. If you can point to measurable improvements, you’ll keep your job.

Nature + Agriculture = Permaculture

Natural ecosystems evolved over hundreds of millennia, developing sustainable, complex webs of relationships over millions of generations. Humans gathered or hunted whatever edible organisms they could find from this emergent food web. However, supply was limited and unpredictable, which is why humans invented agriculture. Our ancestors replaced forests with fields, which were planted with carefully-selected species that provided the tastiest or most nourishing byproducts. Over time, these species were bred to maximize yield and reduce maintenance costs. Unfortunately, this led to a destruction of natural habitat and mass extinction of plants and animals that did not conform to the human ideal. Farms became brittle monocultures of one or two crops, denuding the land of nutrients.

There is a better way: permaculture. Permaculture is a human-cultivated, sustainable ecosystem that produces food by mimicking the balance and interactions between plants and animals found in nature. It basically involves planting a forest filled with a wide variety of native edible and beneficial species that enrich the soil and provide each other with nutrients, protection from parasites and disease. A permaculture plantation can yield food for humans and animals while surviving indefinitely — without the addition of artificial fertilizers, irrigation, or pesticides.

Permaculture for social media

What if we structured social media like a permaculture — capable of yielding revenue for a business while supporting a vibrant, self-organized, self-sustaining community indefinitely? What would that online community look like? How would we go about designing and structuring that permaculture for production AND longevity?

Here’s my unscientific and mostly untested back-of-the-paper-napkin wild-ass guess for an approach to applying permaculture practices to social media.

Step 1: Analyze the site

View your social media site as a plot of land. Each plot of land is different, with a unique mix of soils, wildlife, topography, microclimate, precipitation, sun profile, and people who live on or near that plot. The same principles apply to web sites. Each has a different organizational structure, political hierarchy, business model, content domain, audience, competitive landscape, and so on. In order to design a permaculture for your site and choose the appropriate elements for it, you must consider all of these factors.

1. Business model: Begin by reviewing how resources — money, usually — flow into the business and, as a result, the web site. What are all the sources of funding, staff, political will, and so on?
2. Content domain: What is your organization’s specific industry, subject matter expertise, or genre?
3. Audience: Who do you serve? Who do you sell to? Where do you sell? From what cultural point of reference do you speak from?
4. Competition: Who serves/sells to the same audience as you? Who offers the same product or expertise? Who has the same business model?
5. Seasonal factors: How does the environment for your organization change over time? Based on historical records, what periodic fluctuations can you expect on a monthly, quarterly, and yearly basis?

Step 2: Design the value web

Let’s return to the analogy of the natural ecosystem. In every self-sustaining community of organisms, there is a constant cycle of give and take. Predators eat prey while becoming prey themselves to some other predator. Each organism eats something and is eaten by something. Every organism’s byproducts becomes food, catalyst, insulation, structure, protection, or poison for another organism. The complex network of dependencies that emerges from an ecosystem is called a food web.

On a full-circle farm, the food web is simplified to be more manageable by humans. The sun feeds the grass which feed the sheep whose manure fertilizes the grass and attracts flies which lay eggs which hatch into maggots which are eaten by chickens whose manure fertilizes the grass and enriches the compost which nourishes the corn which is fed to pigs… and so on. In a permaculture, the food web is more complex. Fungi on the roots of a legume will enrich the soil with nitrogen, supporting nitrogen-hungry onions whose flowers produce an aroma that draws harmful insects away from the fruit tree whose fallen leaves prevent water from evaporating and block weed seeds from germinating… and so on.

In a social media system, the people and business entities that make up the network of dependencies don’t eat each other. Instead, they form a living food web where the unit of exchange is value, forming a value web.

In order to create highly targeted, self-sustainable, vibrant social system that also makes money, we must identify how each actor in the system gains value from and gives value back to the system. Consider how users, the content they contribute, and the affordances provided by the system act to create a living, vibrant community. For each user, determine their needs, what they produce, and how what they produce meets the needs of other types of users.

For each product offering, feature, function, target user type, or content type, answer:

• Who will find value in it?
• Who will use the output of it? For what purpose?
• What information goes into it? Where does that information come from?
• What part of it can be used directly? (e.g., revenue generation, brand awareness, data mining)
• What useful byproducts does it produce? (e.g., metadata, customer demographics, behavioral data)
• What waste products does it produce? (e.g., irrelevant content artifacts)
• What does it compete with?
• What is its lifecycle? How does it change over time?
• What is its life expectancy? How often does it need to be “re-planted”?

In gardening, a common practice is companion planting, where the gardener places two or three plants that have some sort of simple dependency relationship near each other. Over time, the output of each plant will nourish its companion.

Companion planting can be applied to social media as well. For example, a Twitter-style micro-blogging feed from a product design team could be used to seed topics for a Digg-style feature-voting discussion board about the product. Votes and comments harvested from the discussion group could then be used to provide feedback to the product design team. By designing multiple clusters of such value loops and then linking them together, a nascent value web could be created.

However, it’s not enough to just create a network of dependencies. The value web must be flexible enough to survive, even if one element of that web is removed. Each organism has multiple sources of nourishment and produces multiple byproducts for multiple organisms. Additionally, each organism has individual variations, even amongst members of its species. This is what makes natural ecosystems robust enough to survive — and even thrive on — storms, disease, or seasonal fires.

In the business world, this strategy is called diversification. However, this usually means diversifying investments in multiple markets and multiple products. What needs to be added to this strategy is an understanding of the different kinds of value your customers gain from your business. Consider the value a customer gets from not just the product, but also from friends and family, other customers, society (in the form of social cachet from using your product), their own sense of accomplishment, your customer service, bonuses and rewards, and so on. If there are enough different types of value from a diverse enough set of sources, you could replace one of those vlaue components without causing a collapse of the value web.

Step 3: Fill in the niches

In nature, certain roles must be filled and in balance for a permaculture to form and thrive. Might we construct a social permaculture online by identifying and designing for analogous user roles to those in nature? (Or am I taking the analogy too far?)

Step 4: Control Weeds

Some of the niches above will be filled naturally by “weedy” users. Weeds are essentially ANY plant growing in the wrong place. It could even be a very valuable plant, such as a saffron crocus or rare orchid. If it occurs in an improper context, it’s a weed. In a permaculture, weeds are naturally suppressed by having an abundance of the right kind of plant. If a community is filled with active moderators who diligently cull and suppress the irrelevant and harmful content, there will be little need for the business owner to actively weed.

Many social media sites simply start with an empty lot, letting their “plot of land” become overgrown with weedy users and their by-products — irrelevant content, off-topic flame wars, link farms, spam, and so on. Sometimes, by selective weeding and cultivation, these chaotic systems can be coaxed to some semblance of community. But it is very difficult, once a permaculture of weeds is established, to steer that community toward relevancy.

It’s far better to plan the social media permaculture and seed it with the right content and encourage the right users to participate. Identify and constrain the system to the audience you want to reach. Provide them with the right mix of functionality and interactions to encourage conversations and connections. Slowly add new elements until you get the right balance.

Step 5: Harvest, prune, and tend

The idea behind permaculture is to create a self-sustaining system that also produces food. In social media, you want to encourage community AND accomplish some business objective. How do you know the establishment of the community is helping you reach your goals? How do you know it’s making money? How do you know what’s working and what’s not? The answer: measurement.

Farming requires fastidious bookkeeping. What did I plant where? How did that plant react to the addition of the other plant? How did the late onset of summer affect yield? How does compost made of kitchen scraps and lawn clippings perform year-over-year compared to compost made with chicken manure? What’s the optimum distance between fruit trees so enough light reaches the understory to encourage the growth of fruiting shrubs?

The same goes for social media. Analytics must be collected throughout the lifetime of the site to understand the effect of seemingly minute changes to elements of the online community. You won’t be able to predict with 100% accuracy how your permaculture will develop over time. As a result, you’ll need to swap out underperforming technologies, keep an eye on content rot, prune back overgrown categories, re-target audiences, tune your messaging.

For the permaculture gardener, many of these optimization decisions require trial and error over decades. For a web site or online service, new features, designs, and content can be trialed and refined over a few weeks. Here are some of the most promising methods:

• Rapid, iterative testing & evaluation: prototype designs are tested with actual users, revised in real time or within a matter of days, and re-tested and revised repeatedly. This results in a relatively well-optimized pre-launch design.
• Beta launch: this is the pilot test. Users are more forgiving when informed that the site or service may not be stable. They are also more likely to return to see if improvements have been made since their last visit. Just don’t leave it in beta forever.
• Multivariate testing: in-flight testing of minor changes, run on a random selection of a small percentage of visitors.

The user experience of a permaculture

You may well wonder whether a permaculture feels any better to a user than a simple, straightforward, single-product site or service. I’m going to cop out and say, “it depends”. Any web site or product, no matter how complex, can be made to feel simple, given enough latitude in the design. Sure, a natural ecosystem would just look and feel like an overgrown jungle. But a truly useful, sustainable, and profitable permaculture by definition must have simple and aesthetically-pleasing pathways, fully-accessible harvest patches, and an easily-maintained structure. Likewise, a sustainable social media system must look and feel simple, approachable, and accessible, even though it may be supported by an extremely complex set of business rules and technologies.

We tried to apply user-centered design principles to our conference planning process — hopefully, that message comes through in the article.

I was struck by how similar – and educational – the garden design process is to the user experience design process. While reading these books, I’d often just substitute “garden” and “landscape” with “web site”, transforming the text into a comprehensive guide to creating sustainable and useful information systems.

The garden design process all starts with the vision and goals. Our goals for our garden include:

• low-maintenance structure
• self-sustaining, bootstrapping, balanced micro-ecosystem
• layout optimized for access & use (socializing, entertaining, harvesting)
• long-term, sustainable source of food

After that comes a site assessment that analyzes the specific space and context from many different perspectives: climate, microclimate, seasonal factors, flow, access, use, aesthetics, materials, soil, organisms, etc. A design can’t simply be copied from one site to another – the topology, wind, water, sunlight, and so on are extremely site and context-specific. For each perspective, the designer maps out the healthy and high-yield areas, the sick and risky areas, confounding factors, and so on. An overlay of all of these perspectives simultaneously shows which areas will support which types of features and which areas will need to be built up or re-designed.

From there, the designer brainstorms different approaches, blocking out on a conceptual bubble diagram how the new design will address the factors identified in the site assessment while still fitting within the vision. They then choose from a pattern library of well-tested, sustainable feature configurations, using some features to protect and support, while using others to add aesthetic and nutritional value. Again, not every pattern will work for every site. The designer must intelligently select the right pattern for the constraints and factors inherent to the context for which they’re designing.

The design is iteratively refined, considering all of the perspectives from the site assessment, until enough details have been resolved to begin selecting and placing individual plants. At the same time, the site can be prepped for installing the new design. As the new components are inserted into the landscape, the designer must consider how these will change over time. An edible forest garden must be able to sustainably evolve over decades, so the designer needs to take into account short-term factors as well as the long-term plan for their design.

Once the design is reasonably certain, implementation can begin. However, it doesn’t end when the last plant is placed or the last paving stone is installed. The design must be refined and the site tended over the months and years. Iteration and optimization are key to the long-term success of any design.

There are so many complex and competing factors to consider in a garden. As with a social networking web site, the community can’t really be controlled directly. The garden designer must attract beneficial organisms with certain layouts, features, and plants, while considering how the requirements and products of one organism might affect other organisms. If the structure is well-designed and plants well-selected, different organisms will nourish, protect, and police each other.

Sounds a lot like the user-centered design process, right? Well, it’s no mystery. Landscape architecture and garden design have histories that extend to the beginning of human civilization. There is deep wisdom to be gained from examining the process of creating and cultivating of healthy, sustainable forest gardens.

In web sites, the pathways in, out, and through the site determine how people use, gain value from, and give value to the site and the other people linked to it. The site will constantly evolve, powered by a constant influx and outflow of information, money, time, and so on. Like a yard gone to seed and overgrown with weeds, an unattended, poorly designed web site will stagnate and collect spammers and trolls. The web site designer must consider how to attract the right audiences, provide them with value, allow them to be productive, and make use of the products of their input to further improve the structure and value of the site.

I’ve posted a blog entry about the Blended Agenda Matrix and its history to the ZAAZ Blog.

Yesterday, it struck me that the users I design user interfaces for might just be those same people. This led me to realize that we in the user-centered design industry often tend to idealize our users, creating soft-focus glamorized personas that portray them as honest, hard-working, likeable people. I wonder, though… do we do the world a disservice by glossing over their flaws?

What would happen if we tried to understand our users as real human beings, warts and all? How would our design process change for people who are dishonest, lazy, disagreeable, and – heaven forbid – evil?

As user-centered professionals, it’s our job to promote and defend the needs of all users, right?

No, not really. It’s our job to design experiences that simultaneously accomplish the goals of our employer or client while meeting users’ needs as best we can. All user needs we meet must be within the subset that are correlated – directly or indirectly – with the business’ or organization’s needs.

In social networking sites and multiplayer online games, it becomes more crucial to understand the personality flaws of users. Every online community suffers from trolls and griefers who intentionally abuse other users and poison social systems with offending or annoying content or actions. They do this to gain attention, feel a sense of empowerment, get revenge on another user, or just entertain themselves. However, even “normal” users will game social systems to maximize their virtual wealth, improve their peer rating, gain attention, feel a sense of empowerment, etc. The design of any good online community will have checks and balances to prevent users from exploiting the system or abusing each other.

So what does this mean for the user-centered designer? I think it means that maybe we should drink our persona kool-aid with a grain of salt. Just as pharmacists include a list of side effects and contraindications for every drug they recommend, perhaps we should be detailing the weaknesses, negative traits, and potential errant behaviors of our personas. Or maybe we should create anti-personas for the trolls, griefers, or any other user we’d like to actively discourage.

What would follow from this would be designs that proactively inhibit or balance out negative behaviors. We would specify who we are designing against in addition to who we are designing for, thereby improving the focus on our true target audiences. We’d discover which audiences are supporting the bottom line of our clients. We’d create online communities that are self-regulating. Or maybe we’d just understand who our users really are: imperfect human beings with foibles and vices. Just like you and me.