Category Killers of the Internet are Significantly Reducing Online Diversity

The number of distinctive sources and voices on the internet is proven to be in long-term decline, according to new research.

A paper entitled “Evolution of diversity and dominance of companies in online activity” has shown between 60 and 70 per cent of all attention on key social media platforms in different market segments is focused on just 10 popular domains.

In stark contrast, new competitors are struggling to survive against such dominant players, with just 3 per cent of online domains born in 2015 still active today, compared to nearly 40 per cent of those formed back in 2006.

The researchers say if these were organic lives, the infant mortality rate would be considered a crisis.

Paul X. McCarthy, a co-author of the paper and Adjunct Professor from UNSW Sydney’s Engineering faculty, said: “The internet started as a source of innovation, new ideas and inspiration, a technology that opens up the playing field. But it is now becoming a medium that actually stifles competition, promotes monopolies and the dominance of a small number of players.

The results indicate the end state of many new industries is likely to be more concentrated than in the analogue economy. With a winner-takes-most outcome for many.

It means that there’s not as much natural competition in established domains, for example in retail with Amazon or in music with Spotify. This may lead to non-competitive behaviour by market leaders such as price discrimination and the use of market power to control suppliers and stifle potential future rivals.

“That is why some see a new role for market regulators to step in here.”

The research team, which also included Dr. Marian-Andrei Rizoiu from UTS, Sina Eghbal from ANU, and Dr. Daniel Falster and Xian Gong from UNSW, performed a large-scale longitudinal study to quantify the distribution of attention given in the online environment to competing organisations.

They tallied the number of external links to an organisation’s main domain posted on two large social media channels, namely Reddit and Twitter, as a proxy for online attention towards an organisation.

More than 6 billion user comments were analysed from Reddit, dating back to 2006, while the Twitter data comprised 11.8 billion posts published since 2011. In total, a massive trove of 5.6Tb data was analysed from over a decade of global activity online—a data set more than four times the size of the original data from the Hubble Space Telescope.

And the results proved that over the long run, a small number of competitive giants are likely to dominate each functional market segment, such as search, retail and social media.

For example, the top 10 most popular domains mentioned on Reddit received around 35% of all links in 2006, which grew to 60% in 2019. On Twitter, the top 10 domains grew from 50% in 2011 to 70% in 2019.

The paper notes: “If there are too few competitors or a small number of players become too dominant within any economic sector, there emerges the potential for artificially high prices and constraints to supply. Even more importantly, in the long-term, this gives rise to constraints on innovation.”

Co-author Dr Marian-Andrei Rizoiu, who leads the Behavioural Data Science lab at the UTS Data Science Institute, said: “This research indicates the environment for new players online is becoming increasingly difficult. In this way, pine trees sterilise the ground under their branches to prevent other trees from competing with them, once they are established dominant players, online crowd, out competitors in their functional niche.

“Diversity of sources is in long term decline and although the worldwide web continues to grow, the attention is focused on fewer and few organisations.

“Attention online is a new form of currency. This new research illustrates that using a variety of analytic techniques and large-scale, global, longitudinal data, we can reveal patterns hitherto unseen on a global scale.”

The research team’s review of data going back 15 years allowed them to observe some specific dynamics that often play out when a new online market or function starts to flourish.

At first, there is a great burst of diverse businesses that appear and attempt to serve the market. After that comes a development phase when the number of competitors peaks and then starts to dwindle.

Finally, in the maturity phase, there is a significant reduction in diversity as users converge around at a single dominant organisation—such as Google in terms of web searching and Amazon for online shopping.

Co-author Dr Daniel Falster, who leads his own lab in Evolutionary Ecology in the School of Biological, Earth and Environmental Sciences at UNSW, said: “As with the natural environment, we can see the birth, growth and survival patterns of companies online when we look at them on a large enough scale.

“Now we can see consistent patterns of competitive dynamics that are becoming clearer and clearer. The loss of diversity is a cause for concern.”

Despite the findings, Professor McCarthy believes there are still opportunities for new global business to emerge and thrive.

“Businesses looking to grow and expand in the online world should be focused on innovation and collaboration—two salient features of many of the biggest winners in the digital economy,” he said.

“This new research—inspired in part by ecology—illustrates while ‘species diversity is in decline, there are also many new functional openings continually emerging in the jungle with global potential.