How the software economy is driven by proprietary work

The source of proprietary "use value"
The roots of innovation spread far and wide. Some innovation appears random, the by-product of "pure research." The notebooks of Ramanujan, a mathematician who managed single-handedly to deduce all of western mathematics from his village in southern India before his invitation to Cambridge shortly before World War I, are still plumbed for new ideas. Ramunajun was a pure ideas person, in that many of his insights lacked at the time any apparent practical application. Topology is another area of mathematics that was considered "pure" by its early practitioners. Today, topology has grown in popularity as modern physics found ways to use it in theories of "Hyperspace".

Most innovation, however, results from attempts to solve real-world problems. Corporations lie at the heart of applied innovation, paying employees to figure out new ways to satisfy "real world demand" as represented by paying customers, either through new ideas or the application of "pure" innovation to customer needs.

Companies, however, are interested in taking market share from competitors. Money is at stake, and competition for it drives companies to differentiate themselves in ways that can create incompatibilities. To critics such as Raymond, this is a Bad Thing ™, and it is blamed for the fragmentation that occurred with proprietary Unix.

There are other ways to view this phenomenon, however. The proper manner to adapt technology to general consumer needs is not obvious. This is particularly the case with new technology, as no one has experience of its application. Companies, therefore, will offer implementations that differ in the details, all in an attempt to grow market share and earn revenue.

The motivations, to be sure, are self-interested. Regardless, large-scale parallel experimentation is one of the reasons market systems work better than planned systems. Parallel experimentation in the market for manufactured goods leads to the best means to allocate scarce resources (with "best" defined as the outcome which most satisfies human needs and wants). The same principle applies in software.

Why Proprietary Companies create so much innovation
Proprietary software companies, as noted by Raymond, rely on their ability to convince customers to upgrade to new versions. This is known as the "upgrade treadmill," and it is viewed by Raymond as an unmitigated Bad Thing ™.

Viewed a different way, the motivation of vested financial interests is a goad that drives proprietary companies to create new ideas. Given the disproportionate share of use value created by proprietary companies, this is apparently an effective motivator.

Second, proprietary companies are close to the primary source of information regarding human needs: their customers. Companies are best positioned to figure out what customers actually want. Open source, in contrast, is a product mostly reliant on voluntary contributions, and thus doesn't have the same access to these information wellsprings.

Third, proprietary software companies make money to a degree that open-source software companies don't, and it is this extra revenue that enables them to spend more energy on updating software as such. They can use this money to conduct applied research which produces new software ideas, or to conduct proper end-user studies to determine what interests them.

Lastly, developers are paid to orient themselves towards the interests of non-technical users. This is an important advantage, as volunteers are more likely to be interested in writing the kind of software that interests them, while paid workers have a vested interest in teasing out the needs of other people.

Ask yourself why so much open-source software is of the "infrastructure" variety ("plumbing" code, which in no way implies that such code isn't essential), or why Linux had (perhaps still has) such a difficult time making a consumer-friendly operating system. Developers who aren't paid to orient themselves towards the interests of regular consumers will tend to create software that matters to people like themselves.

Where open source makes most sense
Open source is most useful in areas of technology that are well understood. Proprietary companies have the advantage in new technology realms due to better information regarding real-world customer needs and their ability to experiment in parallel regarding the satisfaction of those needs.

Open source also makes most sense in "infrastructure." MOST contributors to open source do so free of charge (though some don't, a point I discuss in future instalments). This means that their work will tend to be oriented towards areas of interest to themselves. Infrastructure is of necessity oriented towards technical personnel, and thus the interests of the open-source contributors and end users (technical personnel) align closely.

The open-source market seems to have fallen into this pattern. Apache is the most popular Web server in existence. HTTP, however, is not a fast-evolving technology domain. The kinds of things required of HTTP are already well understood. Furthermore, the Apache Web server is targeted towards programmers and technicians. Within such an environment, the spontaneous contribution model offers almost as much as the parallel experimentation model.

Partial Agreement
Unlike Richard Stallman, Eric Raymond makes room for proprietary software in his vision of the future. As noted near the end of The Magic Cauldron:

"Applications, on the other hand, will have the most tendencies to remain closed. There will be circumstances under which the use value of an undisclosed algorithm or technology will be high enough (and the costs associated with unreliability will be low enough, and the risks associated with a supplier monopoly sufficiently tolerable) that consumers will continue to pay for close software" (NOTE: I'll deal with the points made in brackets in later instalments).

Eric used Id Software, maker of game software, as an example of a valid reason to release software under a proprietary licence. When it was first released, Doom was in Raymond's words utterly unique. It managed to create a 3-D universe on the low-powered processors of 1993 in ways that remained unfathomable for a number of months. Later, they released the source for the core rendering engine once the techniques used became well understood (among other reasons).

Doom was such interesting technology that it was right for "Id Software" to attempt to recoup its investment through proprietary licensing. Once the technology entered the realm of common knowledge, however, Id released the source code in order to benefit from the network effects and community improvements made possible by open source.

Some Parting Thoughts
Open source would rule the world if we'd reach a plateau in terms of the kinds of things we can do with software. At that point, software would cease to need much in the way of innovation, and the slow pace would leave plenty of time for most of the software world to be standardised by international standards bodies.

This simply isn't the case today. When I look towards the future, I see whole new categories of software bursting forth in response to new devices and new needs. In my opinion, developers have seen only the tip of a software iceberg.

Open source will continue to progress, growing into areas of technology as they become better understood. In areas of fast growth, proprietary software companies will continue to dominate and generate a disproportionate share of use value. This will be the case so long as proprietary companies, with their combination of close contact with real-world customers, stable sources of revenue and incentives to produce new ideas, proves the best way to tease out the intersection between technology and the general public.

John Carroll is a software engineer. He specialises in the design and development of distributed systems using Java and .Net. He is also the founder of Turtleneck Software.