Friday, 24 October 2014

GUIDED SELF-ORGANIZATION:Making the Invisible Hand Work

by Dirk Helbing [1] 

This is fourth in a series of blog posts that form chapters of my forthcoming book Digital Society. Last week's chapter was titled: CRYSTAL BALL AND MAGIC WAND:The Dangerous Promise of Big Data

In an increasingly complex and interdependent world, we are faced with situations that are barely predictable and quickly changing. And even if we had all the information and means at our disposal, we couldn’t hope to compute, let alone engineer, the most efficient or best state of the system: the computational requirements are just too massive. That’s why the complexity of such systems undermines the effectiveness of centralized planning and traditional optimization strategies. Such efforts might not only be ineffective but can make things even worse. At best we end up “fighting fires” – struggling to defend ourselves against the most disastrous outcomes.

If we’re to have any hope of managing complex systems and keeping them from collapse or crisis, we need a new approach. Whether or not the quote is apocryphal, what Einstein allegedly says holds true here: "We cannot solve our problems with the same kind of thinking that created them." What other options do we have? The answer is perhaps surprising: we have to step back from centralized top-down control, which often ends in failed brute-force attempts to impose a certain behavior. However, as I will now explain, we can find new ways of letting the system work for us.

This means that we should implement principles such as distributed bottom-up control and guided self-organization. What are these principles about and how do they work? Self-organization means that the interactions between the components of the system spontaneously lead to a collective, organized and orderly mode of behavior. That does not, however, guarantee that the state of the system is one we might find desirable, and that's why self-organization may need some "guidance."

Distributed control means that, if we wish to guide the system towards a certain desirable mode of behavior, we must do so by applying the guiding influences in many "local" parts of the system, rather than trying to impose a single global behavior on all the individual components at once. The way to do this is to help the system adapt locally to the desired state wherever it shows signs of deviating. This adaptation involves a careful and judicious choice of local interactions. Guided self-organization thus entails modifying the interactions between the system components where necessary while intervening as little and as gently as possible, relying on the system's capacity for self-organization to attain a desired state.

When and why are these approaches superior to conventional ones? For the sake of illustration, I will start with the example of the brain and then turn to systems such as traffic and supply chains. I will show that one can actually reduce traffic jams based on distributed real-time control, but it takes the right approach. In the remaining chapter of the book, we will explore whether and how these success principles might be extended from technological to economic and even social systems.

The miracle of self-organization

Our bodies represent perfect examples of the virtues of self-organization in generating emergent functions from the interactions of many components. The human brain, in particular, is made up of a trillion information-processing units, the neurons. Each of these is, on average, connected to about a thousand other neurons, and the resulting network exhibits properties that cannot be understood by looking at the single neurons: in our case, not just coordination, impulses and instincts, but also the mysterious phenomenon of consciousness. And yet, even though a brain is much more powerful than today's computers (which are designed in a top-down way and not self-organized), it consumes less energy than a typical light bulb! This shows how efficient the principle of self-organization can be.

In the previous chapter, we saw that the dynamical behavior of complex systems – how they evolve and change over time – is often dominated by the interactions between the system components. That’s why it is hard to predict the behaviors that will emerge – and why it is so hard to control complex systems. But this property of interaction-based self-organization is also one of the great advantages of complex systems, if we just learn to understand and manage them.

System behavior that emerges by self-organization of a complex system's components isn’t random, nor is it totally unpredictable. It tends to give rise to particular, stable kinds of states, called "attractors," because the system seems to be drawn towards them. For example, the figure below shows six typical traffic states, where each of the depicted congestion patterns is an attractor. In many cases, including freeway traffic, we can understand and predict these attractor states using simplified computer models of the interactions between the components (here: the cars). If the system is slightly perturbed, it will usually tend to return to the same attractor state. To some extent this makes the system resilient to perturbations. Large perturbations, however, will drive the system towards a different attractor: another kind of self-organized, collective behavior, for example, a congested traffic state rather than free traffic flow.

The Physics of Traffic

Contrary to what one might expect, traffic jams are not just vehicle queues that form behind bottlenecks. Traffic scientists were amazed when, beginning in the 1990s, they discovered the large variety and complexity of empirical congestion patterns. The crucial question is whether such patterns are understandable and predictable phenomena, such that we can find new ways to avoid congestion. In fact, there is now a theory that allows one to explain all traffic patterns as composites of elementary congestion patterns (see figure above). This theory can even predict the size and delay times caused by congestion patterns – it is now widely regarded as one of the great successes of complexity theory.

I started to develop this theory when I was working at the University of Stuttgart, Germany, together with Martin Treiber and others. We studied a model of freeway traffic in which each vehicle was represented by a computer “agent” – a driver-vehicle unit moving along the road in a particular direction with a preferred speed, which however would slow down whenever this was necessary to avoid a collision. Thus the model attempted to “build” a picture of traffic flow from the bottom up, based on simple interaction rules between the individual agents, the driver-vehicle units. Based on this model, we could run computer simulations to deduce the emergent outcomes resulting in different kinds of traffic situations.

For example, we simulated a multi-lane freeway with a bottleneck created by an on-ramp, where additional vehicles entered the freeway. At low vehicle densities, traffic flow recovered even from large perturbations in the flow such as a massive vehicle platoon. In sharp contrast, at medium densities even the slightest variation in the speed of a vehicle triggered a breakdown of the flow – the "phantom traffic jams" we discussed before. In between, however, there was a range of densities (called "bistable" or "metastable"), where small perturbations faded away, while perturbations larger than a certain size (the "critical amplitude") caused a traffic jam.

Interestingly, when varying the traffic flows on the freeway and the on-ramp in the presence of a small perturbation, we found all the empirical congestion patterns shown above. In essence, most traffic jams were caused by a combination of three elements: a bottleneck, high traffic flows, and a perturbation in the flow. Moreover, the different traffic states could be arranged in a so-called "phase diagram" (see below). The diagram schematically presents the flow conditions under which each type of pattern remains stable, and the boundaries that separate these regimes. Empirical observations nicely support this theoretical classification of possible traffic patterns.
A capacity drop, when traffic flows best!
Can we use this understanding to improve traffic flows? To overcome congestion, we must first recognize that the behavior of traffic flows can be counter-intuitive, as the "faster-is-slower effect" shows (see Information Box 1). Imagine a stretch of freeway joined by an on-ramp, on which the traffic density is relatively high but the flow is smooth and free of jams, and not prone to jam formation triggered by small disturbances. Suppose now we reduce the density of vehicles entering the considered freeway stretch for a short time. You might expect that traffic will flow even better. But it doesn’t. Instead, vehicles accelerate into the area of smaller density – and this behavior can trigger a traffic jam! Just when the entire road capacity is urgently needed, we find a breakdown of capacity, which can last for hours and can increase travel times by a factor of two, five, or ten. A breakdown may even be triggered by the perturbation created by a simple overtaking maneuver of trucks.

It's cynical that the traffic flow becomes unstable when the maximum throughput of vehicles is reached – that is, exactly in the most efficient state of operation from an “economic” point of view. To avoid traffic jams, therefore, we would have to stay sufficiently far away from this “maximally efficient” traffic state. But doesn’t this mean we must restrict ourselves to using the roads at considerably less efficiency than they are theoretically capable of? No it doesn’t, if we just build on guided self-organization.

Avoiding traffic jams

Traffic engineers have sought ways to improve traffic flows at least since the early days of computers. The classical "telematics" approach to reduce congestion is based on the concept of a traffic control center that collects information from a lot of traffic sensors, then centrally determines the best strategy and implements it in a top-down way – for instance, by introducing variable speed limits on motorways or using traffic lights at junctions. Recently, however, researchers and engineers have started to explore a different approach: decentralized and distributed concepts, relying on bottom-up self-organization. This can be enabled, for example, by car-to-car communication.

In fact, I have been involved in the development of a new traffic assistance system that can reduce congestion. From the slower-is-faster effect, we can learn that, in order to avoid or delay the breakdown of traffic flows and to use the full freeway capacity, it is important to smooth out perturbations of the vehicle flow. With this in mind, we have developed a special kind of adaptive cruise control (ACC) system, where distributed control attempts are made by a certain percentage (e.g. 30%) of ACC-equipped cars, while a traffic control center is not needed for this. The ACC system accelerates and decelerates a car automatically based on real-time data from a radar sensor, measuring the distance to the car in front and the relative velocity. Such radar-based ACC systems existed already before, but in contrast to conventional ACC systems, ours does not just aim to increase the driver’s comfort by eliminating sudden changes in speed. It also increases the stability and capacity of the traffic flow by taking into account what other nearby vehicles are doing, thereby supporting a favorable self-organization of the entire traffic flow. This is why we call it a traffic assistant system rather than a driver assistant system.

The distributed control approach of the underlying ACC system is inspired by fluids flows, which do not suffer from congestion: when we narrow a garden hose, the water simply flows faster through the bottleneck. To sustain the traffic flow, one can either increase the density or speed of vehicles, or both. The ACC system we developed with the Volkswagen company imitates the natural interactions and acceleration of driver-vehicle units, but in order to increase the vehicle flow where needed, it slightly reduces the time gap between successive vehicles. Additionally, our special ACC system increases the acceleration of vehicles out of the traffic jam to stabilize the flow.

In essence, we modify the driving parameters determining the acceleration and interactions of cars such that the traffic flow is increased and stabilized. The real-time measurement of distances and relative velocities by radar sensors allows the cars to adjust their speeds in a way that is superior to human drivers. This traffic assistant system, which I developed together with Martin Treiber, Arne Kesting, Martin Schönhof, Florian Kranke, and others, was also successfully tested under real traffic conditions.

Cars with collective intelligence

A key issue for the operation of the adaptive cruise control is to identify, where it needs to kick in and alter the way a vehicle is being driven. These locations can be figured out by connecting the cars into a communication network. Many new cars contain a lot of sensors that can be used to give them “collective intelligence.” They can perceive their driving state and features of their local environment (i.e. what nearby cars are doing), communicate with neighboring cars (through wireless inter-vehicle communication), make sense of the situation they are in (e.g. assess the surrounding traffic state), take autonomous decisions (e.g. adjust driving parameters such as the speed), and give advice to drivers (e.g. warn of a traffic jam behind the next curve). In a sense, such vehicles acquire also "social" abilities: they can coordinate their movements with those of others.

According to our computer simulations, even if only a small proportion of cars is equipped with such ACC systems, this can have a significant positive effect on the overall traffic situation. In contrast, most driver assistant systems today are still operating in a "selfish" way rather than creating better flow conditions for everyone. Our special, "social" solution approach, seeking to reach systemic benefits through collective effects of local interactions, is a central feature of what I call Socio-Inspired Technologies.

A simulation movie we have created illustrates how effective this approach can be (see While the ACC system is turned off, the traffic develops the familiar and annoying stop-and-go waves of congestion. When seen from a bird’s-eye view, it becomes evident that the congestion originates from small perturbations triggered by vehicles attempting to enter the freeway via an on-ramp. But once the ACC system is turned on, these stop-and-go waves vanish and traffic flows freely. In other words, modifying the interactions of vehicles based on real-time measurements allows us to produce coordinated and efficient flows in a self-organized way. Why? Because we have changed the interaction rules between cars based on real-time adaptive feedback, handing over responsibility to the autonomously driving system. With the impending advent of “driverless cars” such as those being introduced by Google, it’s clearer than ever that this sort of intervention is no fantasy at all.

Guided self-organization

So we see that self-organization may have favorable results (such as free traffic flows) or undesirable ones (such as congestion), depending on the nature of the interactions between the components of the system. Only a slight modification of these interactions can turn bad outcomes into good ones. Therefore, in complex dynamical systems, "interaction design" – also known as "mechanism design" – is the secret of success.

Self-organization based on modifications of interactions or institutional settings – so-called "guided self-organization" – utilizes the hidden forces acting in complex dynamical systems rather than opposing them. In a sense, the superiority of this approach is based on similar principles to those of Asian Martial Arts, where the forces created by the opponent are turned to one’s own advantage. Let’s have a look at another example: how best to coordinate traffic lights.

Self-organizing traffic lights

Relative to freeway flows, urban traffic flows incur additional challenges. Here the roads are connected into complex networks with many junctions, and the problem is mainly how to coordinate the traffic at all these intersections. When I began to study this difficult problem, my goal was to find an approach that would work not only when conditions are ideal but also when they are impaired or complicated, for example because of irregular road networks, accidents or building work. Given the large variability of urban traffic flows over the course of days and seasons, the best approach turned out to be one that adapts flexibly to the prevailing local travel demands, not one that is planned or optimized for "typical" (average) traffic flows. Rather than imposing a certain control scheme for switching traffic lights in a top-down way, as it is done by traffic control centers today, I concluded that it is better if the lights respond adaptively to the actual local traffic conditions. In this self-organizing traffic-light control, the actual traffic flows determine, in a bottom-up way and in real time, how the lights switch.

The local control approach was inspired by my previous experience with modeling pedestrian flows. These tend to show oscillating flow directions at bottlenecks, which look as if they were caused by “pedestrian traffic lights”, even though they are not. The oscillations are in fact created by changes in the crowd pressure on both sides of the bottleneck – first the crowd surges through the constriction in one direction, then in the other. This, it turns out, is a relatively efficient way of getting the people through the bottleneck. Could road intersections perhaps be understood as a similar kind of bottleneck, but with more flow directions? And could flows that respond similarly to the local traffic “pressure” perhaps generate efficient self-organized oscillations, which could in turn control the switching sequences of the traffic lights? Just at that time, a student named Stefan Lämmer knocked at my door and asked to write a PhD thesis in my team about this challenging problem. So we started to investigate this.

How to outsmart centralized control

How does self-organizing traffic light control work, and how successful is it? Let’s first look at how it is currently done. Many urban traffic authorities today use a top-down approach coordinated by some control center. Supercomputers try to identify the optimal solution, which is then implemented as if the traffic center were a "benevolent dictator." A typical solution creates "green waves" of synchronized lights. However, in large cities even supercomputers are unable to calculate the optimal solution in real time – it's too hard a computational problem, with just too many variables to track and calculate.

So the traffic-light control schemes, which are applied for certain time periods of the day and week, are usually optimized "offline." This optimization assumes representative (average) traffic flows at a certain day and time, or during events such as soccer matches. In the ideal case, these schemes are then additionally adapted to the actual traffic situation, for example by extending or shortening the green phases. However, at a given intersection the periodicity of the switching scheme (in what order the road sections get a green light) is usually kept the same. Within a particular control scheme, it’s mainly the length of the green times that is altered, while the order of switching just changes from one applied scheme to another.

Unfortunately, the efficiency of even the most sophisticated of these top-down optimization schemes is limited by the fact that the variability of traffic flows is so large that average traffic flows at a particular time and place are not representative for the traffic situation on any particular occasion at that time and place. The variation in the number of cars behind a red light and the fraction of vehicles turning right or going straight is more or less as big as the corresponding average values. This implies that a pre-planned traffic light control scheme isn't optimal at any time.

So let us compare this classical top-down approach carried out by a traffic control center with two alternative ways of controlling traffic lights based on the concept of self-organization (see illustration below). The first, called selfish self-organization, assumes that each intersection separately organizes its switching sequence to strictly minimize the travel times of the cars on the road sections approaching it. The second, called other-regarding self-organization, also tries to minimize the travel times of these cars, but aims before all else to clear the vehicle queues that exceed some critical length. Hence, this strategy also takes into account the implications for neighboring intersections.
How successful are the two self-organizing schemes compared to the centralized one? We’ll assume that at each intersection there are detectors that measure the outflows from its road sections and also the inflows into these road sections coming from the neighboring intersections (see illustration below). The information exchange between neighboring intersections allows short-term predictions of the arrival times of vehicles. The locally self-organizing traffic lights adapts to this prediction in a way that tries to keep vehicles moving and to minimize waiting times.

When the traffic flow is sufficiently far below the intersection capacity, both self-organization schemes produce well-coordinated traffic flows that are much more efficient than top-down control: the resulting queue lengths behind red traffic lights are much shorter (in the figure below, compare the violet dotted and blue solid line with the red dashed line). However, for selfish self-organization, the process of local optimization only generates good results below a certain traffic volume. Long before the maximum capacity utilization of an intersection is reached, the average queue length tends to get out of control, as some road sections with small traffic flows are not served frequently enough. This creates spillover effects – congestion at one junction leaks to its neighbors – and obstructs upstream traffic flows, so that congestion quickly spreads over large parts of the city in a cascade-like manner. The resulting state may be viewed as a congestion-related "tragedy of the commons," as the available intersection capacities are not anymore efficiently used. Due to this coordination failure between neighboring intersections, when the traffic volumes are high, today’s centralized traffic control can produce better flows than selfish self-organization, and that's actually the reason why we run traffic centers.
Yet by changing the way in which intersections respond to information about arriving vehicle flows, it becomes possible to outperform top-down optimization attempts over the whole range of traffic volumes that an intersection can handle (see the solid blue line). To achieve this, the rule of waiting time minimization must be combined with a second rule, which specifies that a vehicle queue must be cleared immediately whenever it reaches a critical length (that is, a certain percentage of the road section). This second rule avoids spill-over effects that would obstruct neighboring intersections and thereby establishes an "other-regarding” form of self-organization. Notice that at high traffic volumes, both the local travel time minimization (dotted violet line above) and the clearing of long queues (black dash-dotted line) perform badly in isolation, but when combined, they produce a superior way of coordination. One would not expect that two bad strategies in combination might produce the best results!

One advantageous feature of the self-organization approach is that it can use gaps that occur in the traffic as opportunities to serve other traffic flows. In that way, the coordination arising between neighboring traffic lights can spread over many intersections in a self-organized way. That’s how other-regarding self-organization can outsmart top-down control trying to optimize the system: it responds more flexibly to actual local needs, thanks to a coordinated real-time response.

Therefore, what will the role of traffic control centers be in the future? Will they be obsolete? Probably not. They will still be used to keep an overview of all urban traffic flows, to ensure information flows between distant parts of the city, and to implement political goals such as limiting the overall flows into the city center from the periphery.

A pilot study

After this promising study, Stefan Lämmer approached the public transport authority in Dresden to collaborate with them on traffic light control. The traffic center was using an adaptive state-of-the art control scheme based on "green waves." But although it was the best available on the market, they weren’t happy with it. In particular, they were struggling to manage the traffic around a busy railway station in the city center. There, the problem was that many public transport lines cut through a highly irregular road network, and the overall goal was to prioritize public transport rather than road traffic. However, if trams and buses were to be given a green light whenever they approached an intersection, this would destroy the green waves in the vehicle flows, and the resulting congestion would quickly spread, causing massive disruption over a huge area of the city.

When we applied our other-regarding self-organization scheme of traffic lights to the same kind of empirical inflow data that had been used to calibrate the current control scheme, we found a remarkable result. The waiting times were reduced for all modes of transport: considerably so for public transport and pedestrians, and somewhat also for vehicles. The roads were less congested, trams and buses were prioritized, and travel times became more predictable. In other words, everybody would benefit from the new approach (see figure below) – including the environment. It is just logical that the other-regarding self-organization approach is now being implemented at some traffic intersections in Dresden.
Lessons learned
From this example of traffic light control, we can draw a number of important conclusions. First, in complex systems with strongly variable and largely unpredictable dynamics, bottom-up self-organization can outperform top-down optimization by a central controller – even if that controller is kept informed by comprehensive and reliable data. Second, strictly local optimization may create a highly performing system under some conditions, but it tends to fail when interactions between the system components are strong and the optimization at each location is selfish. Third, an "other-regarding" approach that takes into account the situation of the interaction partners can achieve good coordination between neighbors and superior system performance.

In conclusion, a central controller will fail to manage a complex system because the computational demands needed to find the best solutions are overwhelming. Selfish local optimization, in contrast, ultimately fails because of a breakdown of coordination, when the system is used too much. However, an other-regarding self-organization approach based on local interactions can overcome both problems, producing resource-efficient solutions that are robust against unforeseen disturbances.
In many cities, there has recently been a trend towards replacing signal-controlled intersections with roundabouts, and towards changing urban spaces controlled by many traffic signs and rules in favor of designs that support voluntary, considerate interactions of road users and pedestrians. In other words, the self-organization approach is spreading.

As we will see in the chapters on Human Nature and the Economy 4.0, many of the conclusions we have drawn from traffic flows are relevant for socio-economic systems as well. These are also systems in which agents often have incompatible interests that cannot be satisfied at the same time... Production processes are an example for this as well.

Self-organizing production

Problems of coordinating flows appear also in man-made systems other than traffic and transportation. About ten years ago, together with Thomas Seidel and others, I began to study how production plants could be operated more efficiently, and be better designed. In the paper and packaging production plant we studied, we observed bottlenecks that occurred from time to time. When this happened, a jam of products waiting to be processed propagated upstream, while the shortfall in the number of finished products grew downstream (see illustration below). We noticed that there were quite a few analogies with traffic systems. For example, road sections are analogous to storage buffers where partly finished products can accumulate. Product-processing units are like road junctions, different product flows have different origins and destinations (like vehicles), production schedules function like traffic lights, cycle times are analogous to travel and delay times, full “buffer” sections suffer from congestion, and machine breakdowns are like accidents. However, modeling production is even more complicated than modeling traffic, as there are many different kinds of material flows.

Drawing on our experience with traffic models, we devised an agent-based model for these production flows. We focused again on how local interactions can govern and potentially assist the flow. We imagined equipping all machines and all products with a small "RFID" computer chip having memory and wireless short-range communication ability – a technology already widely implemented in other contexts, such as tagging of consumer goods. This would enable a product to communicate with other products and machines in the neighborhood (see figure below). For example, a product could signal that it was delayed and needed prioritized processing, requiring a kind of over-taking maneuver. Products could also select between alternative routes, and tell the machines what had to be done with them. They could cluster together with similar products to ensure efficient processing.

In the past, designing a good factory layout in a top-down way has been a complicated, time-consuming and expensive procedure. Bottom-up self-organization is again a superior approach. The above-described agent-based approach building on local interactions has a phenomenal advantage: it makes it easy to test different factory layouts without having to specify all the details of the fabrication plant. One just has to put the different elements of a factory together (such as machines and transportation units). The possible interactions are then specified automatically. The machines know immediately what to do with the products – because those products already bear with them the necessary instructions. Here too the local exchange of information between agents creates a collective, social intelligence. Given these favorable circumstances, it is easily possible to create and test many different factory layouts and to find, which are more efficient and more resilient to perturbations.

In the future one may even go a step further. If we consider that recessions are like traffic jams in the world economy, where capital or product flows are obstructed or delayed, couldn't real-time information about the world's supply networks be used to reduce economic disruptions? I actually think so. Therefore, if I had access to the data of the world-wide supply chains, I would be delighted to build an assistant system for global supplies that reduces cases of overproduction and situations, where resources are lacking.

Making the Invisible Hand work

We, therefore, see that vehicles and products can successfully self-organize if a number of conditions are fulfilled. First, the interacting system components are provided with real-time information. Second, there is prompt feedback – that is to say, appropriate rules of interaction – which ensures that this information elicits a suitable, adaptive response. (In later chapters, I will discuss in detail how such information can be gathered and how such interaction rules are determined.)

So, would a self-organizing society be possible? In fact, for hundreds of years, people have been inspired by the self-organization and social order in colonies of social insects such as ants, bees, or termites. For example, Bernard Mandeville’s The Fable of Bees (1714) argues that actions driven by private, even selfish motivations can create public benefits. A bee hive is an astonishingly differentiated and complex, well-coordinated social system, even though there is no hierarchical chain of command. No bee orchestrates the actions of the other bees. The queen bee simply lays eggs, and all other bees perform their respective roles without being told so. Adam Smith's "Invisible Hand" expresses a similar idea, namely that the actions of people, even if driven by the 'selfish' impulse of personal gain, would be invisibly coordinated in a way that automatically improves the state of the economy and the society. One might say that, behind this, there is often a believe in something like a divine order.

However, the recent global financial and economic crisis has questioned that complex systems would always produce the best possible outcomes by themselves. Phenomena such as traffic jams and crowd disasters suggest as well that a laissez faire approach that naively trusts into the "Invisible Hand" often fails. The same applies to failures of cooperation, which may result in the over-utilization of resources as discussed in the next chapter.

Nevertheless, whether the self-organization of a complex dynamical system ends in success or failure mainly depends on the interaction rules and institutional settings. I therefore claim that, three-hundred years after the principle of the Invisible Hand was postulated, we can finally make it work – based on real-time information and adaptive feedbacks to ensure the desired functionality. While the Internet of Things can provide us with the necessary real-time data, complexity science can inform us how choose the interaction rules and institutional settings such that the system would self-organize towards a desirable outcome.

Information technologies to assist social systems

Above, I have shown that self-organizing traffic lights can outperform the optimization attempts of a traffic control center. Furthermore, "mechanism design," which modifies local vehicle interactions by suitable driver assistant systems, can turn self-organization into a principle that helps to reduce rather than produce congestion. But these are technological systems.

Could we also design an assistance system for social behavior? In fact, we can! Sometimes, social mechanism design can be pretty challenging, but sometimes it's easy. Just imagine the task to share a cake in a fair way. If social norms allow the person who cuts the cake to take the first piece, this will often be bigger than the others. If he or she is to take last, the cake will probably be distributed in a much fairer way. Therefore, alternative sets of rules that are intended to serve the same goal (such as cake cutting), may result in completely different outcomes.

As Information Box 2 illustrates, it is not always easy to be fair. But details in the "institutional setting" – the specific "rules of the game" – can matter a lot. With the right set of interaction rules, we can, in fact, create a better world. The next chapter discusses, how the respective social mechanisms, which are part of our culture, can make a difference, and how one can build an assistant system to support cooperation in situations where it would otherwise be unlikely. Information and communication technologies are now offering entirely new opportunities!

INFORMATION BOX 1: Slower-is-faster effect

Let me illustrate with an example, how counter-intuitive the behavior of traffic flows can be (see picture above). When the traffic flow is sufficiently high, but still stable, a temporary reduction in the vehicle density (which locally allows drivers to move at a faster speed) can surprisingly cause a traffic jam. How does this "faster-is-slower effect" happen? First, the temporary perturbation of the vehicle density changes its shape, while traveling along the freeway. Then, it eventually causes a forwardly moving vehicle platoon, which grows in the course of time. Consequently, the perturbation of the traffic flow propagates downstream and eventually passes the location of the on-ramp. As the vehicle platoon is still moving forward, one would think that the perturbation will eventually leave the freeway stretch under consideration. But at a certain point in time, the vehicle platoon has grown so big that it suddenly changes its propagation direction, i.e. it starts to travel backward rather than downstream. This is called the "boomerang effect." The effect occurs, because vehicles in the cluster are temporarily stopped, when the vehicle platoon has reached a certain size. At the front of the cluster, vehicles are moving out of the traffic jam, while new vehicles join the traffic jam at the end. Altogether, this makes the traffic jam travel backwards, such that it eventually reaches the location of the on-ramp. When this happens, the inflow of cars via the on-ramp is perturbed so much that the upstream traffic flow breaks down. This causes a long vehicle queue, which continues to grow upstream. Therefore, even when the road capacity could theoretically handle the overall traffic flow, a perturbation in the traffic flow can cause a drop in the freeway capacity, which results from the interactions between cars. The effective capacity of the freeway is then given by the outflow from the traffic jam, which is about 30 percent below the maximum traffic flow on the freeway!

INFORMATION BOX 2: Fair supply in times of crises

In case of a shortage of resources that are required to satisfy our basic needs (such as food, water, and energy), it might be particularly important to share them in a fair way. Otherwise, violent conflicts for scarce resources might break out. But it is not always easy to be fair and requires suitable preparations. Together with Rui Carvalho, Lubos Buzna, and others, I have investigated this for cases such as gas supply through pipelines. There, we may visualize the percentages of pipeline use towards different destinations by a pie chart. It turns out that we must now cut several cakes (or pies) at the same time. Given multiple constraints by pipeline capacities, it is usually impossible to meet all goals and constraints at the same time. Therefore, one will often have to make compromises. Paradoxically, if overall less gas is transported due to non-deliveries from a source region of gas, fair sharing requires a re-routing of gas from other source regions. This will often lead to pipeline congestion problems, since the pipeline network was built for different origin-destination relationships. Nevertheless, an algorithm inspired by the Internet routing protocol can maximize fairness.

[1] Dear Reader,

Thank you for your interest in this chapter, which is thought to stimulate debate.

What you are seeing here is work in progress, a chapter of a book on the emerging Digital Society I am currently writing. My plan was to elaborate and polish this further, before I share this with anybody else. However, I often feel that it is more important to share my thoughts with the public now than trying to perfect the book first while keeping my analysis and insights for myself in times requiring new ideas.

So, please apologize if this does not look 100% ready. Updates will follow. Your critical thoughts and constructive feedback are very welcome. You can reach me via dhelbing (AT) or @dirkhelbing at twitter.

I hope these materials can serve as a stepping stone towards mastering the challenges ahead of us and towards developing an open and participatory information infrastructure for the Digital Society of the 21st century that would enable everyone to take better informed decisions and more effective actions.

I believe that our society is heading towards a tipping point, and that this creates the opportunity for a better future.

But it will take many of us to work it out. Let’s do this together!

Thank you very much, I wish you an enjoyable reading,

Dirk Helbing
PS: Special thanks go to the FuturICT community and to Philip Ball.

Friday, 17 October 2014

CRYSTAL BALL AND MAGIC WAND:The Dangerous Promise of Big Data

by Dirk Helbing

This is third in  a  series of blog posts that form chapters of my forthcoming book Digital Society. Last week's chapter was titled:  COMPLEXITY TIME BOMB: When systems get out of control

Data sets bigger than the largest library

The idea that we could solve the problems in the world, if we just had enough data about them, is intriguing. In fact, we are now entering an era of "Big Data" – masses of information, mostly in digital form, about all aspects of our lives, institutions and cultures. It will probably not be long before each newborn baby will have its genome sequenced at birth. Every purchase we make on the Internet releases data about our location, preferences, and finances that will be stored somewhere and quite possibly used without our consent. Cell phones disclose where we are, and private messages and conversations are not really private at all. Books are being digitized beyond the advent of printing, and are available in immense, searchable databases of words that are now being mined in "culturomics" studies that put history, society, art and cultural trends under the lens. Aggregated data can be used to reveal unexpected facts, such as flu epidemics being inferred from Google searches.

This avalanche of data is ever increasing: with the introduction of technologies such as Google Glass, people will have the option of documenting and archiving almost every aspect of their lives. Big Data such as credit-card transactions, communication and mobility data, public news, Google Earth imagery, comments and blogs, are creating an increasingly accurate digital picture of our physical and social world, including all its social and economic activities.

"Big Data" will change our world. The term, coined more than 15 years ago, means data sets so big that one can no longer cope with them with standard computational methods. To benefit from Big Data, we must learn to "drill" and "refine" data, i.e. to transform them into useful information and knowledge. The global data volume doubles every 12 months. Therefore, each year we produce as much data as in all previous years together.

These tremendous amounts of data relate to four important technological innovations: the Internet, which enables our global communication, the World Wide Web (WWW), a network of globally accessible websites that evolved after the invention of hypertext protocol (HTTP), the emergence of Social Media such as Facebook, Google+, Whatsup, or Twitter, which have created social communication networks, and the emergence of the "Internet of Things" (IoT), which allows sensors, smartphones, gadgets, and machines ("things") to connect to the Internet. Note that there are already more things connected to the Internet than humans.

Meanwhile, the data sets collected by companies such as ebay, Walmart or Facebook, reach the size of petabytes (1 million billion bytes)one hundred times the information content of the largest library in the world: the U.S. Library of Congress. The mining of Big Data opens up entirely new possibilities for the optimization of processes, the identification of interdependencies, and the support of decisions. However, Big Data also comes with new challenges, which are often characterized by four criteria: volume (the file sizes and number of records are huge), velocity (the data evaluation has often to be done in real-time), variety (the data are often very heterogeneous and unstructured), and veracity (the data are probably incomplete, not representative, and contain errors).

Gold rush for the 21st century's oil

When the social media portal WhatsApp with its 450 million users was recently sold to Facebook for $19 billion – almost half a billion dollars was made per employee. There’s no doubt that Big Data create tremendous business opportunities – not just because of its value for, say, marketing, but because the information itself is becoming monetarized.

Technology gurus preach that Big Data is becoming the new oil of the 21st century: a commodity that can be tapped for profit. With the virtual currency BitCoin temporarily becoming more valuable than gold, one can even literally say that data can be turned into value to an extent we only knew from fairy tales. Even though many sets of Big Data are proprietary, the consultancy company McKinsey recently estimated the potential value of Open Data alone to be 3 to 5 trillion dollars per year. If the worth of this publicly available information were to be evenly apportioned among the public itself, it would bring $500 to each person in the world.

The potential of Big Data spans all areas of social activity: from natural language processing to financial asset management, or to a smart management of cities that better balances energy consumption and production. It could enable better protection of our environment, risk detection and reduction, and the discovery of opportunities that would otherwise be missed. And it could make it possible to tailor medicine to patients, thereby increasing drug effectiveness, accelerating drug discovery and reducing side effects.

Big Data applications are now spreading very rapidly. They enable personalized services and products, open up entirely new possibilities to optimize production and distribution processes or services, allow us to run "smart cities," and reveal unexpected interconnections between our activities. Big Data also hold great potential for fostering evidence-based decision-making, particularly in big business and politics. Where is all of this leading us?

In this post, I explain what Big Data can and cannot do. In particular, I will show that it will never be enough on its own to avoid crises and catastrophes, or to solve all societal problems. Indeed, the suggestion sometimes heard – that Big Data is the key to the future – can be misleading in pretty dangerous ways. Most obviously, it could precipitate a descent into an authoritarian surveillance state where there is very little personal liberty or autonomy, and where no one can have any more secrets. But even if that were not to happen, Big Data could create a false sense that we can control our own destiny, if only we have enough data. Information is potentially useful, but it can only release its potential if it is coupled to a sound understanding of how complex social systems work.

Big Data fueling super-governments

The development of human civilization has depended on the establishment of mechanisms that promote cooperation and social order. One of these is based on the idea that everything we do is seen and judged by God. Bad deeds will be punished, while good ones will be rewarded. The age of information has inspired the dream that we might be able to know and control everything ourselves: to acquire God-like omniscience and omnipotence. There are now hopes and fears that such power lies within the reach of huge IT companies such as Google or Facebook and clandestine secret services such as the CIA or NSA. CIA Chief Technology Officer Ira "Gus" Hunt[2] has explained how easy it is for such institutions to gather a great deal of information about each of us:
"You're already a walking sensor platform… You are aware of the fact that somebody can know where you are at all times because you carry a mobile device, even if that mobile device is turned off. You know this, I hope? Yes? Well, you should… Since you can't connect dots you don't have, it drives us into a mode of, we fundamentally try to collect everything and hang on to it forever… It is really very nearly within our grasp to be able to compute on all human generated information." 
Could such a massive data-collection process be good for the world, helping to eliminate terrorism and crime? Or should we fear that the use of this information will undermine human rights and the basis of democratic societies and free economies? In this chapter, I explore the possibilities and limits of such an approach, for better or worse.

Imagine you are the president of a country, intending to ensure the welfare of all its people. What would you do? You might well wish to prevent wars and financial crises, economic recessions, crime, terror, and the spread of diseases. You may want people to be rich, happy, and healthy. You would like to avoid unhealthy drug consumption, corruption, and perhaps traffic jams as well. You would like to ensure a safe, reliable supply of food, water, and energy, and to keep the environment in good shape. In sum, you would like to create a prosperous, sustainable and resilient society.

What would it take to achieve all this? You would certainly have to take the right decisions and avoid those that would have harmful, unintended side effects. So you should know about alternatives for impending decisions, along with their opportunities and risks. For your country to thrive, you would have to avoid ideological, instinctive or traditional decisions in favor of evidence-based decisions. To have the evidence needed to inform this decision-making, you would need a lot of data about all quantifiable aspects of society, and excellent data analysts to interpret it. You might well decide to collect all the data you can get, just in case it might turn out to be useful one day to counter threats and crises, or to exploit opportunities that might arise.

Previously, rulers and governments were not in this position: they generally lacked the quality or quantity of data needed to take well-informed decisions. But that is now changing. Over the past several decades, the processing power of computers has exploded, roughly doubling every 18 months. The capacity for data storage is growing even faster. The amount of data is doubling every year. With the now emerging "Internet of Things," cell phones, computers and factories will be connected to the most mundane devices – coffee machines, fridges, shoes and clothes – creating an overwhelming stream of information that feeds an ocean of "Big Data."

Humans governed by computers?

The more that data is generated, stored and interpreted, the easier is it to find out about each individual’s behavior. Everyone's computer and everyone's device-encoded behavior (such as the record of our movements produced by the cell phones we carry) leaves a unique fingerprint, such that it is possible to know our interests, our thinking, our passions, and our feelings. Some companies analyze "consumer genes" to offer personalized products and services. They have already collected up to 3,000 personal data from almost a billion people in the world: names, contact data, incomes, consumer habits, health information, and more. This is pretty much everyone with a certain level of income and Internet connectivity.

Would it be beneficial if a well-intentioned government had access to all this data? It could help politicians and administrations to take better informed decisions: to reduce terrorism and crime, say, and to use energy more efficiently, protect our environment, improve traffic flows, avoid financial meltdowns, mitigate recessions, enhance our health system and education, and provide better fitting services to citizens.

Moreover, could a government use the information not only to understand but also to predict our behavior, and map out the course of our society? Could it optimize our social systems and take the best decisions for everyone?

In the past, we have used supercomputers for almost everything except understanding our economy, society, and politics. Every new car or airplane is designed, simulated, and tested on a computer. Increasingly, so are new drugs. Thus, why shouldn't we use computers to understand and guide our economy and society too? In fact, we are slowly moving towards that very situation. As a minor (yet revealing) example, since their early days computers have been used for traffic control. Today’s economic production and the management of supply chains would not be conceivable without computer control as well, and large airplanes are now controlled by a majority decision among several computers. Computers can already beat the best chess players, and about 70 percent of financial transactions are executed by trading computers in the meantime. IBM's Watson computer has started to take care of some customer hotlines, and computer-driven Google cars will soon move around without a driver, perhaps picking up the goods we ordered on the Web without us being present. In all these cases, computers already do a better job than humans. Why shouldn't they eventually make better policemen, administrators, lawyers, and politicians?

It no longer seems unreasonable, then, to imagine a gigantic computer program that could simulate the actions and interactions of all the humans in the world, perhaps even equipping these billions of agents with cognitive abilities and intelligence. If we fed these agents with our own personal data, would they behave as we do? In other words, would it be possible to create a virtual mirror world? And would machine learning eventually be able to build the computer agents so similar to us that they would take decisions undistinguishable from ours? Attempts to construct or at least envisage such a scheme are already underway. If they were realized, would they represent a kind of Crystal Ball with which we could predict the future of society?

The prospect might sound unnerving to some, but in principle the potential benefits aren’t hard to see. There are many huge problems that such a predictive capability might help to solve. The financial crisis has created global losses of at least 20 trillion US dollars. Crime and corruption consume about 2-5% of the gross domestic product (GDP) of all nations on earth – about 2 trillion US dollars each year. The lost output of the US economy as a result of the 9/11 terror attacks is estimated to be of the order of 90 billion dollars. A major influenza pandemic infecting 1% of the world’s population would cause losses of 1-2 trillion dollars per year. Cybercrime costs Europe alone 750 billion Euros a year. The negative economic impact of traffic congestion amounts to 7-8 billion British Pounds in the United Kingdom alone.

If a computer simulation of the entire global socio-economic system could produce just a 1 percent improvement in dealing with these problems, the benefits to society would be immense. And in fact, if experiences with managing smaller complex social systems this way are any guide, an improvement of 10-30 percent seems conceivable. Overall this would amount to savings of more than 1 trillion dollars annually. Even if we had to invest billions in creating such a system, the benefits could exceed the investments hundred-fold. Even if the success rates were significantly smaller, this would represent a substantial gain. It would be hard to see how any responsible politician could decline to support such an investment.

But would such a system work as one might hope? Is Big Data all a government needs to get our world under control?

Crystal Ball and Magic Wand

Recent studies using smartphone data and GPS traces suggest that more than 90 percent of the mobility of people – where they will be at a certain time – can be forecast, because of its repetitive nature. If other aspects of our behavior show the same degree of predictability, it’s not hard to imagine that the trajectory of society can indeed be mapped out in advance, with all that this entails for successful social planning. While some people might not like this prospect at all, many would perhaps appreciate a predictable life.

How far does this idea extend? If we have enough data about every aspect of life, could we become omniscient about the future? In order to achieve that, we would need to be able to manipulate people’s choices using the information provided to them. Personalized Internet searches, systems such as Google Now, and personalized advertising are already going in this direction.But given the overwhelming amount of data available, it needs to be filtered before to be useful. 

Such filtering will inevitably be done in the interests of those who do the filtering. For example, companies want potential customers to see their ads and buy their products. The better people’s characteristics are known, the easier it becomes to manipulate their choices. A recent, controversial Facebook experiment with 600 million users showed that it is possible to manipulate people’s feelings and mood. Therefore, it’s not hard to imagine that omniscience might indeed imply omnipotence: those who know everything could control everything. Let's call the hypothetical tool creating such power a "Magic Wand".

Assuming that we had a Magic Wand, could we take the right decisions for our society, or even for every individual? Many people might say that forecasting societal trends is different from forecasting the weather: the weather does not care about the forecast, but people will respond to it, and this will defeat the prophecy. That seems to imply that successful forecasting of societal developments would require that people don't know about the forecasts, while governments do. This again suggests that one would need a secretly operating authority advising the government about the right decisions, and that it would use the Magic Wand according to the evidence provided by the data-collecting Crystal Ball. Could such a scheme work?

A New World Order based on information? 

Our "wise king" or "benevolent dictator" would probably see the Crystal Ball and the Magic Wand as perfect tools to create social order. Singapore is sometimes seen as an approximation of such a system. The country has indeed been enormously successful in the past decades, but despite great advances and fast economic growth, people's satisfaction has decreased. Why?

A wise king would certainly sometimes have to interfere with our individual freedoms, if we would otherwise take decisions that would create more damage for the economy and society than benefits. This might end up in a situation where we would always have to execute what the government wants us to do, pretty much as if they were commands from God. If we were manipulated in our decision-making, this might even happen without our knowledge. Although the wise king would not be able to fulfill our wishes all the time, on average he might create better outcomes for everyone as long as do as we are told. Sure, this sounds dystopian, but let us nevertheless pursue the concept for a while to see whether it is feasible in principle. If we obediently followed the dictates of the wise king, could this improve the state of the world and turn it into a perfectly working clockwork?

Why top-down control is destined to fail

In short, it would not work. This kind of top-down management, even if guided by comprehensive information, is destined to fail. This book is, therefore, concerned with elaborating alternative and better ways of using data, which are compatible with constitutional rights and cultural values such as privacy. But let us first figure out what are the reasons why a well-working Crystal Ball and Magic Wand can't exist.

One of the problems is statistical in nature. To distinguish “good” from “bad” behavior, we need criteria that clearly separate the two. In general, however, reliable criteria of this sort don’t exist. We face the problems of false positive classifications (false alarms, so-called type I errors) and false negatives (type II errors, where the alarm is not triggered when it should be).

For example, imagine a population of 500 million people, among which there are 500 terrorists. Let’s assume that we can identify terrorists with an extremely impressive 99 percent accuracy. Then there are 1 percent false negatives (type II error), which means that 5 terrorists are not detected, while 495 will be discovered. It has been revealed that about 50 terror acts were prevented over the past 12 years or so, while a few, such as the one during the Boston marathon, were not prevented even though the terrorists were listed in some databases of suspects (in other words, they turned out to be false negatives).

How many false positives (false alarms) would the above numbers create? If the type I error is just 1 out of 10,000, there will be 50,000 wrong suspects, while if it is 1 in one thousand then there will be 500,000 wrong suspects. If it is 1 percent (which is entirely plausible), there will be 5 million false suspects! It has been reported that there are indeed between 1 and 8 million people on lists of suspects in the USA. If these figures are correct, this would mean that for every genuine terrorist, up to 10,000 innocent citizens would be wrongly categorized as potential terrorists. Since the 9/11 attacks, about 40,000 suspects have had to undergo special questioning and screening procedures at international airports, even though in 99 percent of these cases it was concluded that the suspects were innocent. And yet the effort needed to reach even this level of accuracy is considerable and costly: according to media reports, it involved around a million people who had a National Security Agency (NSA) clearance on the level of Edward Snowden. 

So, large-scale surveillance is not an effective means of fighting terrorism. This conclusion has, in fact, been reached by several independent empirical studies. Applying surveillance to the whole population is not sensible, for the same reasons why it is not generally useful to apply prevention-oriented medical tests or medical treatments to the entire population: since such mass screenings imply large numbers of false positives, millions of people might be wrongly treated, often with negative side effects on their health. Thus, for most diseases, patients should be tested only if they show worrying symptoms.

Besides these errors of first and second kind, one may face errors of third kind, namely inappropriate models for separating "good" from "bad" cases. For example, unsuitable risk models have been identified as one reason for the recent financial and economic crisis. The risks of many financial products turned out to be wrongly rated, creating immense losses. Adair Turner, head of the UK Financial Service Authority, has said that there is
“a strong belief ... that bad or rather over-simplistic and overconfident economics helped create the crisis. There was a dominant conventional wisdom that markets were always rational and self-equilibrating, that market completion by itself could ensure economic efficiency and stability, and that financial innovation and increased trading activity were therefore axiomatically beneficial.”

Limitations of the Crystal Ball

One might think that errors of first, second, and third kind could be overcome if only we had enough data. But is this true? There are a number of fundamental scientific factors that will impair the Crystal Ball’s functioning (see Information Box 1). The problem known as "Laplace's Demon" reflects on the history-dependence of future developments, and our inability to ever measure all the past information needed to predict the future, even if the world changed according to deterministic rules (that is, if there were no randomness). This is why we are still influenced by cultural inventions, ideas, and social norms that are hundreds or thousands of years old. 

Furthermore, turbulence and chaos are well-known properties of many complex dynamical systems. These factors imply that even the slightest change in the system at a certain point in time may fundamentally change the outcome over a sufficiently long period of time. The phenomenon, also named the "butterfly effect," is well-known to impose limits on the time horizon of weather forecasts.[3] In social systems as in the weather system, this extreme sensitivity to small but unpredictable disturbances arises from the complexity of the system: the existence of many inter dependencies between the component parts. 

Furthermore, we can determine the parameters of our models only with a finite accuracy. However, even small changes in these parameters may fundamentally change the outcome of the model. There is also a problem of ambiguity: the same information may have several different meanings depending on the respective context, and the particular interpretation we choose may influence the future course of the system. Beyond this, we also know that certain statements are fundamentally “undecidable” in the sense that there are questions that cannot be answered with formal logic. Lastly, too much information may reduce the quality of predictions because of over-fitting, spurious correlations, and herding effects. The Information Box at the end of this chapter elaborate these points in more detail.So one can say that Big Data is not the universal tool that it is often claimed to be.[4] Any attempt to predict the future will be limited to probabilistic and mostly short-term forecasts. It is therefore dangerous to suggest that a Crystal Ball could be built that would reliably predict the future. 

Limitations of the Magic Wand

If the Crystal Ball is cloudy, it doesn’t augur well for the Magic Wand that would depend on it. In fact, top-down control is still very ineffective, as the abundance of problems in our world shows. To control complex systems, i.e. to force them to behave in certain ways, we often do not understand them well enough and lack effective means. Therefore, in many cases attempting to control a complex dynamical system in a top down way undermines its functionality. The result is often a broken system, for example, an accident or crisis. 

An example of the failure of top-down control is the fact that even the most sophisticated technological control mechanisms for airplane flight safety increased it less efficiently than introducing a non-hierarchical culture of collaboration in the cockpit, when co-pilots were encouraged to question the decisions and actions of the pilot. In another example, the official report on the Fukushima nuclear disaster in Japan stresses that it was not primarily the earthquake and tsunami that were responsible for the nuclear meltdowns, but 

“our reflexive obedience; our reluctance to question authority; our devotion to ‘sticking with the program’; our groupism.”

In other words, the problem was too much top-down control. Attempts to control complex systems in a top-down way are also very expensive, and we find it increasingly hard to pay for them: most industrialized countries already have debt levels of at least 100 or 200 percent of their gross domestic product. But do we have any alternatives? In fact, the next chapters of this book will elaborate one.

Complexity is the greatest challenge, but also the greatest opportunity

There are further reasons why the concept of a "super-government", "wise king" or "benevolent dictator" can’t really work. These are related to the complexity of socio-economic systems. There are at least four kinds of complexity that matter: dynamic complexity, structural complexity, functional complexity and algorithmic complexity. The problem of complex dynamics has been addressed in the previous chapter. Here, I will focus on implications of structural, functional and algorithmic complexity. In fact, with a centralized super-computing approach we can only solve those optimization problems, which have sufficiently low algorithmic complexity. However, many problems are "NP-hard," i.e. so computationally demanding that they cannot be handled in real-time even by super-computing. This problem is particularly acute in systems that are characterized by a large variability. In such cases, top-down control cannot reach optimal results. In the next chapter, I will illustrate this by the example of traffic light control.

Given the quick increase in computing power, couldn’t we overcome this challenge in the future? The surprising answer is “no.” While the processing power doubles every 18 months (blue curve in the illustration above), the amount of data doubles every year (green curve above). This implies that we are heading from a situation in which we did not have enough data to take good decisions, to a situation where we can take evidence-based decisions. However, despite the rising processing power, we will be able to process a decreasing share of all the data existing in the world. Moreover, the lack of processing power will be quickly increasing. So we are moving to a situation where we can shed light on everything with a spotlight, but many things will remain unseen in the dark. This creates a new kind of problem: paying too much attention to some problems, while neglecting others. In fact, governments didn't see the financial crisis coming, they didn't see the Arab Spring coming, they didn't see the crisis in the Ukraine coming, and they didn't see the Islamic State (IS) fighters in Iraq coming. Thus, keeping a well-balanced overview of everything will become progressively more difficult. Instead, politics will be increasingly driven by problems that suddenly happen to gain public attention, i.e. made in a reactive rather than anticipatory way.

But let’s now have a look at the question of how the world is expected to change depending on its complexity. The possibility to network the components of our world creates ever more options. We have, in fact, a combinatorial number of possibilities to produce new systems and functionalities. If we have two kinds of objects, we can combine them to produce a third one. These three kinds of objects allow us to create six ones, and those already 720. This is mathematically reflected by a factorial function, which grows much faster than exponentially (see the red curve above). For example, we will soon have more devices communicating with the Internet than people. In about 10 years from now, 150 billion (!) things will be part of the Internet, forming the "Internet of Things." Thus, even if we realize just every thousandth or millionth of all combinatorial possibilities, the factorial curve will eventually overtake the exponential curves representing data volumes and computational power. It has probably overtaken both curves already some time ago.

In other words, attempts to optimize systems in a top-down way will become less and less effective – and cannot be done in real time. Paradoxically, as economic diversification and cultural evolution progress, a "big government", "super-government" or "benevolent dictator" would increasingly struggle to take good decisions, as it becomes more difficult to satisfy the diverse local expectations and demands. This means that centralized governance and top-down control are destined to fail. Given the situation in Afghanistan and Iraq, Syria, Ukraine, and the states experiencing the Arab Spring, given the financial, economic and public debt crisis, and given the quick spreading of the Ebola disease in Africa, have we perhaps lost control already? Are we fighting a hopeless battle against complexity?

Simplifying our world by homogenization and standardization would not fix the problem, as I will elaborate in the chapter on the Innovation Accelerator. It would undermine cultural evolution and innovation, thereby causing a failure to adjust to our ever-changing world. Thus, do we have alternatives? Actually, yes: rather than fighting the properties of complex systems, we can use them for us, if we learn to understand their nature. The fact that the complexity of our world has surpassed our capacity to grasp it, even with all the computers and information systems assisting us, does not mean that our world must end in chaos. While our current system is based on administration, planning, and optimization, our future world will be built on evolutionary principles and collective intelligence, i.e. intelligence surpassing that of the brightest people and best expert systems.

How to get there? 

In the next chapters, I will show how the choice of suitable local interaction mechanisms can, in fact, create desirable outcomes. Information and communication systems will enable us to let things happen in a favorable way. This is the path we should take, because we don't have better alternatives. The proposed approach will create more efficient socio-economic institutions and new opportunities for everyone: politics, business, science, and citizens alike. As a positive side effect, our society will become more resilient to the future challenges and shocks that we will surely face.


"Big Data" has great potential, in particular for better, evidence-based decision-making. But it is not a universal solution, as it is often suggested. In particular, data-driven approaches are notoriously bad at predicting systemic shifts, where the entire way of organizing or doing things change. Moreover, like any technology, Big Data can be seriously misused, posing a "dual use problem" (see the Information Box 2 below). Without suitable precautions – for example, the use of "data safes," decentralization, encryption, the logging of large-scale data-mining activities, the limitation of large processing volumes to qualified and responsible users, the accountability of Big Data users for damage created by them, and large fines in cases of damage, misuse, or discrimination – mining Big Data may create massive problems (intentionally or not). It is, therefore, crucial to design socio-technical systems in ways that promote their ethical use.

INFORMATION BOX 1: Limitations to Building a Crystal Ball

Sensitivity - When all the data in the world can't help

How close can computer-modeled behavior ever come to real human social behavior? To specify the parameters and starting conditions of a computer model, these are varied by calibration procedures until the difference between measurement data and model predictions becomes as small as possible. However, the best fitting model parameters are usually not the correct parameters. These parameters are typically within a certain "confidence interval." But if the parameters are randomly picked from the confidence interval, the model predictions may vary a lot. This problem is known as sensitivity.

Turbulence and chaos

Two further problems of somewhat similar nature are "chaos" and "turbulence." Rapid flows of gases or liquids produce swirly patterns – the characteristic forms of turbulence. In chaotically behaving systems, too, the motion becomes unpredictable after a certain time period. Even though the way a "deterministically chaotic" system evolves can be precisely stated in mathematical terms, without random elements, the slightest change in the starting conditions can eventually cause a completely different global state of the system. In such a case, no matter how accurately we measure the initial conditions of the system, we will effectively not be able to predict the later behavior.


Information can have different meanings. In many cases, the correct interpretation can be found only with some additional piece of information: the context. This contextualization is often difficult and not always available when needed. Different pieces of information can also be inconsistent, without any means of resolving the conflict.A typical problem in "data mining" challenges is that data might be plentiful, but inconsistent, incomplete, and not even representative. Moreover, a lot of it might be wrong, because of measurement errors, misinterpretations or application of wrong procedures, or manipulation.

Laplace's Demon and measurement problems

Laplace's Demon is a hypothetical being who could calculate all future states of the world, if he knew the exact positions and speeds of all particles and the physical laws governing their motion and interactions. Laplace's Demon cannot exist in reality, not least because of the fundamental limitation that measurements to determine all particle speeds would be impossible due to the restriction of special relativity: all velocities must be less than the speed of light. This would prevent one from gathering all the necessary data.
Information overload
Having a lot of data does not necessarily mean that we'll see the world more accurately. A typical problem is that of "over-fitting," where a model characterized by many parameters is fitted to the fine details of a data set in ways that are actually not meaningful. In such a case, a model with less parameters might provide better predictions. Spurious correlations are a somewhat similar problem: we tend to see patterns where they actually don't exist (see for some examples).Note that we are currently moving from a situation where we had too little data about the world to a situation where we have too much. It’s like moving from darkness, where we can't see enough, to a world flooded with light, in which we are blinded. We will need "digital sunglasses": information filters that will extract the relevant information for us. But as the gap between the data that exists and the data we can analyze increases, it might become harder to pay attention to those things that really matter. Although computer processing power doubles every 18 months, we will be able to process an ever decreasing fraction of all the data we possess, because the data storage capacity doubles every year. In other words, there will be increasing volumes of data that will never be touched.
When people feel insecure, they tend to follow decisions and actions of others. This produces undesirable herding effects. The economics Nobel laureates George Akerlof (*1940) and Robert Shiller (*1946) have called this behaviour "animal spirits," but in fact the idea of herding in economics goes back at least to the French mathematician Louis Bachelier (1870-1946). Bubbles and crashes in stock markets are examples of where herding can lead.
Randomness and innovation
Randomness is a ubiquitous feature of socio-economic systems. However, even though we would often like to reduce the risks it generates, we would be unwise to try to eliminate randomness completely. It is an important driver of creativity and innovation; predictability excludes positive surprises and cultural evolution. We will see later that some important and useful social mechanisms can only evolve in the presence of randomness. Although newly emerging behaviors are often costly in the beginning, when they are in a minority position, the random coincidence or accumulation of such behaviors in the same neighborhood can be very beneficial, and such behavior may then eventually succeed and spread.

INFORMATION BOX 2: Side effects of massive data collection

Like any technology, Big Data has not only great potential but also harmful side effects. Not all Big Data applications come with these problems, but they are not uncommon. What we need to identify, are those problems that can lead to major crises rather than just localized, small-scale defects.


In the past years, cybercrime has increased exponentially, costing Europe alone around 750 million EUR per year. Some of this has resulted from the undermining of security standards (for example those of financial transactions) for the purpose of surveillance. Other common problems are data theft or identity theft, data manipulation, and the fabrication of false evidence. These crimes are often committed by means of “Trojan horses”, computer codes that can steal passwords and PIN codes. Further problems are caused by computer viruses or worms that damage software or data.

Military risks

Because most of our critical infrastructures are now connected with other systems via information and communications networks, they have become pretty vulnerable to cyber attacks. In principle, malicious intruders can manipulate the production of chemicals, energy (including nuclear power stations), and communication and financial networks. Attacks are sometimes possible even if the computers controlling such critical infrastructures are not connected to the Internet. Given our dependence on electricity, information and money flows as well as other goods and services, this makes our societies vulnerable as never before. Coordinated cyber-attacks could be launched within microseconds and bring the functioning of our economy and societies to a halt.

The US government apparently reserves the right to respond to cyberwar with a nuclear counter-strike. We are now seeing a digital arms race for the most powerful information-based surveillance and manipulation technologies. It is doubtful whether governments will be able to prevent serious misuse of such powerful tools. Just imagine, a Crystal Ball or Magic Wand or other powerful digital tools would exist. Then, of course, everyone wanted to use them, including our enemies, and criminals as well. It is obvious that, sooner or later, these powerful tools would get into wrong hands and finally out of control. If we don't take suitable precautions, mining massive data may (intentionally or not) create problems of any scale – including digital weapons of mass destruction. Therefore, international efforts towards confidence-building and digital disarmament are crucial and urgent.

Economic risks

Cybercrime poses obvious risks to the economy, as do illicit access to sensitive business secrets and theft of intellectual property. Loss of customer trust in products can cause sales losses of the order of billions of dollars for some companies. Systems that would not work effectively without a sufficient level of trust include electronic banking, sensitive communication by email, eBusiness, eVoting, and social networking. Yet more than two thirds of all Germans say they do not trust government authorities and Big Data companies any longer to not misuse their personal data. More than 50 percent even feel threatened by the Internet. The success of the digital economy is further threatened by information pollution, for example, spam and undesired ads.
Social and societal risks
To contain "societal diseases" such as terrorism and organized crime, it often seems that surveillance is needed. However, the effectiveness of mass surveillance in improving the level of security is questionable and hotly debated: the evidence is missing or weak. At the same time, mass surveillance erodes the trustful relationship between citizens and the state. The perceived loss of privacy is also likely to promote conformism and to endanger diversity and useful criticism. Independent judgments and decision-making could be undermined. Excessive state control of the behavior of citizens would, therefore, impair our society’s ability to innovate and adapt.
For such reasons, the constitutions of many countries consider it of fundamental importance to protect privacy, informational self-determination, private communication, and the principle of assumed innocence without proof of guilt. These things are also considered to be essential for human dignity, and elementary preconditions for democracies to function well.
However, today the Internet lacks good mechanisms for forgetting, forgiveness, and re-integration. There are also concerns that the increasing use of Big Data could lead to greater discrimination, which in turn could promote increasing fragmentation of our society into subcultures. For example, it is believed that the spreading of social media has promoted the polarization of US society.
Political risks
It is often pointed out that leaking confidential political communication can undermine the success of sensitive negotiations. Moreover, if incumbent governments have better access to Big Data applications than parties in opposition, this could result in unfair competition and non-representative election outcomes. Last but not least, in the hands of extremist political groups or criminals, Big Data could become a dangerous tool for acquiring and exerting power.

[1] Dear Reader,
thank you for your interest in this chapter, which is thought to stimulate debate.
What you are seeing here is work in progress, a chapter of a book on the emerging Digital Society
I am currently writing. My plan was to elaborate and polish this further, before I share this with anybody else. However, I often feel that it is more important to share my thoughts with the public now than trying to perfect the book first while keeping my analysis and insights for myself in times requiring new ideas.
So, please apologize if this does not look 100% ready. Updates will follow. Your critical thoughts and constructive feedback are very welcome. You can reach me via dhelbing(AT) or @dirkhelbing at twitter.
I hope these materials can serve as a stepping stone towards mastering the challenges ahead of us and towards developing an open and participatory information infrastructure for the Digital Society of the 21st century that would enable everyone to take better informed decisions and more effective actions.
I believe that our society is heading towards a tipping point, and that this creates the opportunity for a better future.
But it will take many of us to work it out. Let’s do this together!
Thank you very much, I wish you an enjoyable reading,
Dirk Helbing
PS: Special thanks go to the FuturICT community and to Philip Ball.
[3] These prediction limits are not just a matter of getting enough measurement data and having a sufficiently powerful computer – one cannot get beyond a certain precision because of the physical nature of the underlying process.
[4] To convince me of the opposite, in analogy to the "Turing test" checking whether a computer can communicate undistinguishable from a human, one would have to demonstrate that a computer system passes the "Helbing test," i.e. finds all fundamental laws of physics discovered by scientists so far, just by mining the experimental data accumulated over time.