music-observatory | Digital Music Observatory

New spotifyr R Package Release

Thu, 17 Jun 2021 08:10:00 +0200

The package is an excellent starting to point for R newbies to try their hands on musicology analysis with a few keystrokes. And of course, it is an essential part of the research infrastructure of musicology worldwide in far more advanced applications.

I have been a very long-time user of Charlie Thomson’s spotifyr R package, which is probably the most used open-source music analytics software in the world. It provides programmatic access to the Spotify Web API, which contains access to the former Echo Nest quantitative musicology engine.

It is an essential part of the Digital Music Observatory’s streaming analysis and our Listen Local apps designed for our trustworthy AI experiments and independent artist services. I am extremely proud to announce that after a very thorough modernization of the package’s exception handling, documentation, and code dependencies that I did in the last week, the package has passed again the peer-review standards and it is back on CRAN.

Getting Started

You should start with the README and get your Spotify Web API access tokens to get started.

What Was the Beatles' Favorite Key?

library(spotifyr)
beatles <- get_artist_audio_features('the beatles')

library(dplyr)
library(purrr)
library(knitr)

beatles %>% 
    count(key_mode, sort = TRUE) %>% 
    head(5) %>% 
    kable()

key_mode	n
C major	104
D major	98
G major	82
A major	76
E major	62

Get your most recently played tracks

library(lubridate)

get_my_recently_played(limit = 5) %>% 
    mutate(
        artist.name = map_chr(track.artists, function(x) x$name[1]),
        played_at = as_datetime(played_at)
        ) %>% 
    select(
      all_of(c("track.name", "artist.name", "track.album.name", "played_at"))
      ) %>% 
    kable()

track.name	artist.name	track.album.name	played_at
A Case of You	Tristen	A Case of You	2021-06-14 09:54:44
Paper Cup	Real Estate	Paper Cup	2021-06-10 20:20:11
Wrong with You	Tristen	Wrong with You	2021-06-10 20:17:24
Animal - Edit	LUMP	Animal	2021-06-10 20:13:21
Streets Of Your Town	DOPE LEMON	Streets Of Your Town	2021-06-10 18:23:00

That’s about right…

Find Your All Time Favorite Artists

get_my_top_artists_or_tracks(type = 'artists', 
                             time_range = 'long_term', 
                             limit = 5) %>% 
    select(.data$name, .data$genres) %>% 
    rowwise %>% 
    mutate(genres = paste(.data$genres, collapse = ', ')) %>% 
    ungroup %>% 
    kable()

name	genres
Japanese Breakfast	art pop, bubblegrunge, eugene indie, indie pop, indie rock, philly indie
Haley Bonar	melancholia, stomp and holler
Balthazar	belgian indie, belgian rock, dutch indie, dutch rock, ghent indie
Buildings Breeding	indie fuzzpop
Angus & Julia Stone	australian indie folk, indie folk, stomp and holler

What could I say? I travelled Australia listening only to Angus & Julia Stone, the Buildings Breeding have been with me since I discovered them on my first iPod, in one of the first podcasts, the Indiefeed. I created my Kickstarter account back in 2010 to support Haley Bonar’s third album. And the year before I was very much into Japanese Breakfast and Balthazar.

Find your favorite tracks at the moment

get_my_top_artists_or_tracks(type = 'tracks', 
                             time_range = 'short_term', 
                             limit = 5) %>% 
    mutate(
        artist.name = map_chr(artists, function(x) x$name[1])
        ) %>% 
    select(name, artist.name, album.name) %>% 
    kable()

name	artist.name	album.name
Hot & Heavy	Lucy Dacus	Hot & Heavy
Sea Urchin	Mystic Braves	Sea Urchin
Human	Freedom Fry	Human
Hot Motion	Temples	Hot Motion
Animal - Edit	LUMP	Animal

What’s the most joyful Joy Division song?

Let’s take a look at the audio feature has to be valence, a measure of musical positivity.

joy <- get_artist_audio_features('joy division')

joy %>% 
    arrange(-valence) %>% 
    select(.data$track_name, .data$valence) %>% 
    head(5) %>% 
    kable()

track_name	valence
Passover - 2020 Digital Master	0.946
Passover - 2007 Remaster	0.941
Colony - 2020 Digital Master	0.829
Colony - 2007 Remaster	0.808
Atrocity Exhibition - 2020 Digital Master	0.790

Now if only there was some way to plot joy…

Joyplot of the emotional rollercoasters that are Joy Division’s albums

Sentify: A Shiny app

This app, powered by spotifyr, allows you to visualize the energy and valence (musical positivity) of all of Spotify’s artists and playlists.

Dope Stuff Other People Have Done with spotifyr

The coolest thing about making this package has definitely been seeing all the awesome stuff other people have done with it. Here are a few examples:

Exploring the Spotify API with R: A tutorial for beginners, by a beginner, Mia Smith

Blue Christmas: A data-driven search for the most depressing Christmas song, Caitlin Hudon

Sente-se triste quando ouve “Amar pelos dois”? Não é o único (Do you feel sad when you hear “Love for both?” You’re not alone), Rui Barros, Rádio Renascença

Using Data to Find the Angriest Death Grips Song, Evan Oppenheimer

Hierarchical clustering of David Bowie records, Alyssa Goldberg

tayloR, Simran Vatsa

Our Work with spotifyR

A key mission of our Digital Music Observatory, which is our modern, subjective approach on how the future European Music Observatory should look like, is to not only to provide high-quality data on the music economy, the diversity of music, and the audience of music, but also on metadata. The quality and availability, interoperability of metadata (information about how the data should be used) is key to build trustworthy AI systems. We rely on spotifyr to fulfil this mission.

*Join our open collaboration Music Data Observatory team as a data curator, developer or business developer.

EU Datathon 2021

Wed, 12 May 2021 18:09:00 +0200

Reprex, a Dutch start-up enterprise formed to utilize open source software and open data, is looking for partners in an agile, open collaboration to win at least one of the three EU Datathon Prizes. We are looking for policy partners, academic partners and a consultancy partner. Our project is based on agile, open collaboration with three types of contributors.

With our competing prototypes we want to show that we have a research automation technology that can find open data, process it and validate it into high-quality business, policy or scientific indicators, and release it with daily refreshments in a modern API.

We are looking for institutions to challenge us with their data problems, and sponsors to increase our capacity. Over then next 5 months, we need to find a sustainable business model for a high-quality and open alternative to other public data programs.

The EU Datathon 2021 Challenge

To take part, you should propose the development of an application that links and uses open datasets. - our data curator team
Your application … is also expected to find suitable new approaches and solutions to help Europe achieve important goals set by the European Commission through the use of open data.” - this application is developed by our technology contributors
Your application should showcase opportunities for concrete business models or social enterprises. - our service development team is working to make this happen!
We use open source software and open data. The applications are hosted on the cloud resources of Reprex, an early-stage technology startup currently building a viable, open-source, open-data business model to create reproducible research products.
We are working together with experts in the domain as curators (check out our guidelines if you want to join: Data Curators: Get Inspired!).
Our development team works on an open collaboration basis. Our indicator R packages, and our services are developed together with rOpenGov.

Mission statement

We want to win an EU Datathon prize by processing the vast, already-available governmental and scientific open data made usable for policy-makers, scientific researchers, and business researcher end-users.

“To take part, you should propose the development of an application that links and uses open datasets. Your application should showcase opportunities for concrete business models or social enterprises. It is also expected to find suitable new approaches and solutions to help Europe achieve important goals set by the European Commission through the use of open data.”

We aim to win at least one first prize in the EU Datathon 2021. We are contesting all three challenges, which are related to the EU’s official strategic policies for the coming decade.

Challenge 1: A European Grean Deel

Our Green Deal Data Observatory connects socio-economic and environmental data to help understanding and combating climate change.

Challenge 1: A European Green Deal, with a particular focus on the The European Climate Pact, the Organic Action Plan, and the New European Bauhaus, i.e., mitigation strategies.

Climate change and environmental degradation are an existential threat to Europe and the world. To overcome these challenges, the European Union created the European Green Deal strategic plan, which aims to make the EU’s economy sustainable by turning climate and environmental challenges into opportunities and making the transition just and inclusive for all.

Our Green Deal Data Observatory is a modern reimagination of existing ‘data observatories’; currently, there are over 70 permanent international data collection and dissemination points. One of our objectives is to understand why the dozens of the EU’s observatories do not use open data and reproducible research. We want to show that open governmental data, open science, and reproducible research can lead to a higher quality and faster data ecosystem that fosters growth for policy, business, and academic data users.

We provide high quality, tidy data through a modern API which enables data flows between public and proprietary databases. We believe that introducing Open Policy Analysis standards with open data, open-source software, and research automation, can help the Green Deal policymaking process. Our collaboration is open for individuals, citizens scientists, research institutes, NGOS, and companies.

Challenge 2: An economy that works for people

Our Economy Data Observatory will focus on competition, small and medium sized enterprizes and robotization.

Challenge 2: An economy that works for people, with a particular focus on the Single market strategy, and particular attention to the strategy’s goals of 1. Modernising our standards system, 2. Consolidating Europe’s intellectual property framework, and 3. Enabling the balanced development of the collaborative economy strategic goals.

Big data and automation create new inequalities and injustices and have the potential to create a jobless growth economy. Our Economy Data Observatory is a fully automated, open source, open data observatory that produces new indicators from open data sources and experimental big data sources, with authoritative copies and a modern API.

Our observatory monitors the European economy to protect consumers and small companies from unfair competition, both from data and knowledge monopolization and robotization. We take a critical Small and Medium-Sized Enterprises (SME)-, intellectual property, and competition policy point of view of automation, robotization, and the AI revolution on the service-oriented European social market economy.

We would like to create early-warning, risk, economic effect, and impact indicators that can be used in scientific, business, and policy contexts for professionals who are working on re-setting the European economy after a devastating pandemic in the age of AI. We are particularly interested in designing indicators that can be early warnings for killer acquisitions, algorithmic and offline discrimination against consumers based on nationality or place of residence, and signs of undermining key economic and competition policy goals. Our goal is to help small and medium-sized enterprises and start-ups to grow, and to furnish data that encourages the financial sector to provide loans and equity funds for their growth.

Challenge 3: A Europe fit for the digital age

Our Digital Music Observatory is not only a demo of the European Music Observatory, but a testing ground for data governance, Digital Servcies Act, and trustworthy AI problems.

Challenge 3: A Europe fit for the digital age, with a particular focus Artificial Intelligence, the European Data Strategy, the Digital Services Act, Digital Skills and Connectivity.

The Digital Music Observatory (DMO) is a fully automated, open source, open data observatory that creates public datasets to provide a comprehensive view of the European music industry. It provides high-quality and timely indicators in all four pillars of the planned official European Music Observatory as a modern, open source and largely open data-based, automated, API-supported alternative solution for this planned observatory. The insight and methodologies we are refining in the DMO are applicable and transferable to about 60 other data observatories funded by the EU which do not currently employ governmental or scientific open data.

Music is one of the most data-driven service industries where most sales are currently executed by AI-driven autonomous systems that influence market shares and intellectual property remuneration. We provide a template that enables making these AI-driven systems accountable and trustworthy, with the goal of re-balancing the legitimate interests of creators, distributors, and consumers. Within Europe, this new balance will be an important use case of the European Data Strategy and the Digital Services Act.

The DMO is a fully functional service that can serve as a testing ground of the European Data Strategy. It can showcase the ways in which the music industry is affected by the problems that the Digital Services Act and European Trustworthy AI initiatives attempt to regulate. It is being built in open collaboration with national music stakeholders, NGOs, academic institutions, and industry groups.

Our Product/Market Fit was validated in the world’s 2nd ranked university-backed incubator program, the Yes!Delft AI Validation Lab. We are currently developing this project with the help of the JUMP European Music Market Accelerator program.

Problem Statement

The EU has an 18-year-old open data regime and it makes public taxpayer-funded data in the values of tens of billions of euros per year; the Eurostat program alone handles 20,000 international data products, including at least 5,000 pan-European environmental indicators.

As open science principles gain increased acceptance, scientific researchers are making hundreds of thousands of valuable datasets public and available for replication every year.

The EU, the OECD, and UN institutions run around 100 data collection programs, so-called ‘data observatories’ that more or less avoid touching this data, and buy proprietary data instead. Annually, each observatory spends between 50 thousand and 3 million EUR on collecting untidy and proprietary data of inconsistent quality, while never even considering open data.

Our automated data observatories are modern reimaginations of the existing observatories that do not use open data and research automation.

The problem with the current EU data strategy is that while it produces enormous quantities of valuable open data, in the absence of common basic data science and documentation principles, it seems often cheaper to create new data than to put the existing open data into shape.

This is an absolute waste of resources and efforts. With a few R packages and our deep understanding of advanced data science techniques, we can create valuable datasets from unprocessed open data. In most domains, we are able to repurpose data originally created for other purposes at a historical cost of several billions of euros, converting these unused data assets into valuable datasets that can replace tens of millions’ worth of proprietary data.

What we want to achieve with this project – and we believe such an accomplishment would merit one of the first prizes - is to add value to a significant portion of pre-existing EU open data (for example, available on data.europa.eu/data) by re-processing and integrating them into a modern, tidy database with an API access, and to find a business model that emphasises a triangular use of data in 1. business, 2. science and 3. policy-making. Our mission is to modernize the concept of data observatories.

Our solution

We are empowering data curators with reproducible research solutions to create high-quality, rigorously tested original datasets from low quality, not validated, not tidy open data. We help them to design meaningful business, policy or scientific indicators and provide them with a software and API to keep the data up-to-date. We help them deposit a copy of the authoritative, uncompromised dataset onto Zenodo, the EU’s data repository, with a DOI or new DOI version.
We create a research workflow that periodically (daily, weekly, monthly, quarterly or annually) collects, corrects and re-processes the data. We use peer-reviewed statistical software and unit-tests to make sure that the data is sound.

Panning out gold from muddy open sources - with automation technology.

We add value with correcting open (and proprietary!) data problems that make open data hard to use, and proprietary, in-house data hard to re-use.
- regions corrects inconsistent geographical coding. Eurostat has no mandate to correct geographical coding, and member states do not historically adjust their data. With many thousands of parish, county, region, province, state boundary changes within states, regional and metropolitian area datasets are not usable without our software.
- iotables puts extremely complex national accounts data into actually useful environmental and economic impact indicators. Instead of working with each country separately, our standardized system can calculate direct and indirect effects, as well as multipliers for every European country that works in the European statistical framework (EU member states, EEA, UK, member candidates.)
- retroharmonize connects cross-sectional surveys with non-European countries, puts pan-European surveys into time series, and corrects regional subsamples. We are creating new indicators from Eurobarometer, Afrobarometer, Arab Barometer, and standardized CAP surveys, as well as other harmonized surveys. We help design surveys that can utilize data from already existing, openly available surveys.
We place the authoritative copy to a data repository (Zenodo or Dataverse), automatically document the data, and make it available in a modern API for SQL queries or CSV downloads.
We present the data with commentary and blog posts from our curators (see: Is Drought Risk Uninsurable? - solidarity and climate change in Belgium) and contributors on a semi-automatically refreshed, open source web portal.
We are perfecting the agile open collaboration model in a triangular setting, where corporate users, scientific researchers, public and non-governmental policy makers, and even citizen scientists can work around a single data ecoystem.
We are validating a business model that allows the commercial, scientific, and policy use of re-processed, high quality data products made from open and shared data.

Our Music Observatory in the Jump European Music Market Accelerator: Meet the 2021 Fellows and their Tutors

Thu, 04 Mar 2021 15:00:00 +0200

According to the announcement of JUMP, the European Music Market Accelerator, after a careful screening of all applications received, the selection committee composed of all JUMP board members has selected the most promising ideas and projects to be developed together with renowned tutors for this 2021 fellowship.

For nine months, the 20 fellows living in many European countries will develop their innovative projects, while receiving a comprehensive 360° training. In addition to specialised workshops by highly qualified experts, each fellow will receive one-on-one tutoring sessions from the most renowned music professionals coming from all over Europe.

The 20 selected projects cover a great variety of urgent needs faced within the music sector. They will:

help fostering social change with projects focusing on diversity in the industry, more fairness and transparency as well as raising awareness on timely issues.
enhance technological development with projects using blockchain, immersive sound and VR and AR.
build bridges between different key actors of the ecosystem.

Download the entire JUMP press release.

Reprex’s project, the automated Demo Music Observatory will be represented by Daniel Antal, co-founder of Reprex among other building bridges projects. This project offers a different approach to the planned European Music Observatory based on the principles of open collaboration, which allows contributions from small organizations and even individuals, and which provides higher levels of quality in terms of auditability, timeliness, transparency and general ease of use. Our open collaboration approach allows to power trustworthy, ethical AI systems like our Listen Local that we started out from Slovakia with the support of the Slovak Arts Council.

JUMP fellows building bridges between different key actors of the ecosystem.

Apart from our Demo Music Observatory the build bridges section Groovly with Martin Zenzerovich, From Play To Rec by Jeremy Dunne, Hajde Radio by Thibaut Boudaud, LowDee by Alex Davidson and ONO-HU! by Gina Akers.

Meet all the JUMP 2021 Fellows, including the technology and social change professionals!

Reprex is a start-up company based in the Netherlands and the United States that validated its early products in the Yes!Delft AI+Blockchain Lab in the Hague. In 2021 we joined the Dutch AI Coalition – NL AIC and requested membership in the European AI Alliance. Reprex is committed to applying reproducible in an open collaboration with our business, scientific, policy and civil society partners, and facilitate the use of open data and open-source software. Many fellows in the program are connected to other regions, like North America and Australia – because music is one of the most globalized industries and forms of art in the world! Reprex is a startup based in the Netherlands and the United States, and we are very excited to collaborate with our peers in new European territories, and in Canada and Australia.

Hope to meet you in these great events - maybe not only online!

Further links:

From Play to Rec on Facebook
HAJDE FR/EN

Music Streaming: Is It a Level Playing Field?

Tue, 23 Feb 2021 21:23:00 +0200

Our article, Music Streaming: Is It a Level Playing Field? is published in the February 2021 issue of CPI Antitrust Chronicle, which is fully devoted to competition policy issues in the music industry.

The dramatic growth of music streaming over recent years is potentially very positive. Streaming provides consumers with low cost, easy access to a wide range of music, while it provides music creators with low cost, easy access to a potentially wide audience. But many creators are unhappy about the major streaming platforms. They consider that they act in an unfair way, create an unlevel playing field and threaten long-term creativity in the music industry.

Our paper describes and assesses the basis for one element of these concerns, competition between recordings on streaming platforms. We argue that fair competition is restricted by the nature of the remuneration arrangements between creators and the streaming platforms, the role of playlists, and the strong negotiating power of the major labels. It concludes that urgent consideration should be given to a user-centric payment system, as well as greater transparency of the factors underpinning playlist creation and of negotiated agreements.

You can read the entire issue and the full text of our article on Competition Policy International in pdf.

Daniel Antal, co-founder of Reprex Was Selected into the 2021 Fellowship Program of the European Music Market Accelerator

Mon, 22 Feb 2021 21:23:00 +0200

Daniel Antal, co-founder of Reprex, and co-editor of Data & Lyrics, was selected into 2021 Fellowship program of JUMP, the European Music Market Accelerator. Jump provides a framework for music professionals to develop innovative business models, encouraging the music sector to work on a transnational level. The European Music Market Accelerator composed of MaMA Festival and Convention, UnConvention, MIL, Athens Music Week, Nouvelle Prague and Linecheck support him in the development of our two, interrelated projects over the next nine months.

Our Demo Music Observatory is a demo version of the European Music Observatory based on open data, open source, automated research in open collaboration with music stakeholders. We hope that we can further develop our business model and find new users, and help the recovery of the festival and live music segment.
Listen Local is our AI system that validated third party music AI, such as Spotify’s or YouTube’s recommendation systems, and provides trustworthy, accountable, transparent alternatives for the European music industry. We hope to expand our pilot project from Slovakia to several European countries in 2021.

Reprex is committed to applying reproducible in an open collaboration with our business, scientific, policy and civil society partners, and facilitate the use of open data and open-source software.

Ensuring the Visibility and Accessibility of European Creative Content on the World Market: The Need for Copyright Data Improvement in the Light of New Technologies

Sat, 13 Feb 2021 18:10:00 +0200

The majority of music sales in the world is driven by AI-algorithm powered robots that create personalized playlists, recommendations and help programming radio music streams or festival lineups. It is critically important that an artist’s work is documented, described in a way that the algorithm can work with it.

In our research paper – soon to be published – made for the Listen Local Initiative we found that 15% of Dutch, Estonian, Hungarian, or Slovak artists had no chance to be recommended, and they usually end up on Forgetify, an app that lists never-played songs of Spotify. In another project with rights management organizations, we found that about half of the rightsholders are at risk of not getting all their royalties from the platforms because of poor documentation.

But how come that distributors give streaming platforms songs that are not properly documented? What sort of information is missing for the European repertoire’s visibility? Reprex is exploring this problem in a practical cooperation with SOZA, the Slovak Performing and Mechanical Rights Society, and in an academic cooperation that involves leading researchers in the field. A manuscript co-authored Martin Senftleben, director of the Institute for Information Law in Amsterdam, and eminent researchers in copyright law and music economics, Reprex’s co-founder makes the case that Europe must invest public money to resolve this problem, because in the current scenario, the documentation costs of a song exceed the expected income from streaming platforms.

In the European Strategy for Data, the European Commission highlighted the EU’s ambition to acquire a leading role in the data economy. At the same time, the Commission conceded that the EU would have to increase its pools of quality data available for use and re-use. In the creative industries, this need for enhanced data quality and interoperability is particularly strong. Without data improvement, unprecedented opportunities for monetising the wide variety of EU creative and making this content available for new technologies, such as artificial intelligence training systems, will most probably be lost. The problem has a worldwide dimension. While the US have already taken steps to provide an integrated data space for music as of 1 January 2021, the EU is facing major obstacles not only in the field of music but also in other creative industry sectors. Weighing costs and benefits, there can be little doubt that new data improvement initiatives and sufficient investment in a better copyright data infrastructure should play a central role in EU copyright policy. A trade-off between data harmonisation and interoperability on the one hand, and transparency and accountability of content recommender systems on the other, could pave the way for successful new initiatives. Download the manuscript from SSRN

Our Slovak Demo Music Database project is a best example for this. We started systematically collect publicly available information from Slovak artists (in our write-in process) and ask them to give GDPR-protected further data (in our opt-in process) to create a comprehensive database that can help recommendation engines as well as market-targeting or educational AI apps.

We believe that one of the problems of current AI algorithms that they solely or almost only work with English language documentation, putting other, particularly small language repertoires at risk of being buried below well-documented music mainly arriving from the United States.

We are looking for rightsholders and their organizations, artists, researchers to work with us to find out how we can increase the visibility of European music.

Future Audience: Size Of Music Discovery Age Populations In Europe

Fri, 20 Nov 2020 15:10:00 +0200

There is various global and national research available on the music discovery demography. People usually discover new music in their young age as they are forming their own personal identity with their peer group. This process mainly happens in the school, where similarly aged, educated young people spend much of their day together.

The music discovery phase is strongly related to the time when most people get their music education: their skills to listen to music, play music or sing, and to compose new music. Investment into music education pays of in later age: people with more diverse music education, for example, with the ability to play music, tend to visit concerts more or discover more music in later life stages.

The annual growth (top chart) and total growth (lower chart) of the 15-24 years-old population in select European countries.

For practical reasons and supported by academic research, we chose the age group 15-24 as the key age group for music discovery. We placed the national population of the music discovery age group in the Music & Society pillar of the Demo Music Observatory.

The changes across Europe are dramatic: in Latvia and the Baltic states, the music discovery age population decreased by 60% since the countries (re-)gained independence in 1991. Musicians and their organizations must work hard in these countries on developing export strategies if they want to maintain the market and audience of their music.

Some European countries, for example, the Netherlands and Turkey have significantly increased their future audiences. In Turkey this is mainly related to population growth, in the Netherlands to immigration. Countries with a growing young population have a strong position for their music business.

Not all lost in the Baltics and other countries where the music discovery age population is shrinking. Much of the lost young population migrated to the United Kingdom and other parts of Europe. If artists and their association in these countries find a way to connect with these new diaspora, their market loss will be far less dramatic. Connecting to the diaspora is the most natural way to start exporting new music.

Get the data from the Demo Music Observatory.
Earlier this year we introduced some in-depth comparison of music audiences in the Chapter 2. Central European Music Industry Report.

Photo credit LauraMayM

Feasibility Study For The Establishment Of A European Music Observatory & The Demo Observatory

Mon, 16 Nov 2020 08:03:00 +0200

The Feasibility study for the establishment of a European Music Observatory was published on 13 November. Our private observatory, CEEMID was consulted in the creation of the Feasibility Study, and some of our recommendations found way into the consultant’s document. We created a Demo Music Observatory to provide a practical guidance on the decisions facing the European stakeholders, and to answer the questions that were left open in the Feasibility Study — particularly on data integration and the institutional model, where a wrong choice can lead to very long delivery time, quality control and budgeting.

We have been developing our Demo Music Observatory in the world’s 2nd ranked university-backed incubator program, the Yes!Delft AI Validation Lab since 15 September 2020. Our aim is to show a better organizational model, examples of research automation and other data integration innovation that can reduce the budgetary needs of the European Music Observatory by 80-90% and provide far more timely, accurate, and relevant service than most data observatories in Europe.

CEEMID has been creating a similar data observatory to the foreseen European Data Observatory, solely based on the contribution of about 60 European stakeholders. As the Feasibility Study suggests, we would be happy to transfer much of CEEMID’s content to the European Data Observatory, which could potentially fill up about 50-70% of the envisioned observatory. We are building our Demo Music Observatory based on the 2000 pan-European indicators collected by CEEMID since 2014.

Challenge Our Demo Observatory: Check out the Music Diversity & Circulation Pillar of our Demo Music Observatory. If you do not find what you are looking for, contact us — we will try to put the data there from our repositories.

Illusory data gap: active and music participation is available on EU level both for gender groups or four ethnic minorities – this is regularly featured in various European CAP surveys and in our national CAP surveys, too.

The Feasibility Study is based on perceived data gaps between data needs of the European stakeholders and data availability. We have shown earlier this year to the European stakeholders that much of these data gaps are illusory. We would like to give about 50 indicators with full documentation, automated, weekly, monthly, quarterly, or annual refreshment for free for all music industry users. We would like to challenge the stakeholders to formulate data requests to us and think together on the ways how could the European music industry build a better observatory faster and with less cost.

Challenge Our Demo Observatory: Check out the Music Economy Pillar of our Demo Music Observatory. If you do not find what you are looking for, contact us — we will try to put the data there from our repositories.

The Feasibility Study concludes that a “European Music Observatory would require a very significant allocation of funds, beyond what could be currently expected from the possible budget of the future Creative Europe programme”. While the Feasibility Study provide cost options, or any cost-benefit analysis, we are certain that this is an exaggeration. Most European data observatories operate with an annual 20,000-200,000-euro subsidy. We want to show with our Demo Music Observatory what can be achieved with an annual budget of 20,000 euros, 50,000 euros, 100,000 euros or 200,000 euros.

Challenge Our Demo Observatory: Check out the Music, Society and Citizenship Pillar of our Demo Music Observatory. If you do not find what you are looking for, contact us — we will try to put the data there from our repositories.

Creating An Automated Data Observatory

Fri, 11 Sep 2020 16:00:39 +0200

We are building data ecosystems, so called observatories, where scientific, business, policy and civic users can find factual information, data, evidence for their domain. Our open source, open data, open collaboration approach allows to connect various open and proprietary data sources, and our reproducible research workflows allow us to automate data collection, processing, publication, documentation and presentation.

Our scripts are checking data sources, such as Eurostat’s Eurobase, Spotify’s API and other music industry sources every day for new information, and process any data corrections or new disclosure, interpolate, backcast or forecast missing values, make currency translations and unit conversions. This is shown illustrated with an earlier post.

For direct access to the file visit this link.

In the video we show automated the creation of an observatory website with well-formatted, statistical data dissemination, a technical document in PDF and an ebook can be automated. In our view, our technology is particularly useful technology in business and scientific researech projects, where it is important that always the most timely and correct data is being analyzed, and remains automatically documented and cited. We are ready deploy public, collaborative, or private data observatories in short time.

Data processing costs can be as high as 80% for any in-house AI deployment project. We work mainly with organization that do not have in house data science team, and acquire their data anyway from outside the organization. In their case, this rate can be as high as 95%, meaning that getting and processing the data for deploying AI can be 20x more expensive than the AI solution itself.

AI solutions require a large amount of standardized, well processed data to learn from. We want to radically decrease the cost of data acquisition and processing for our users so that exploiting AI becomes in their reach. This is particularly important in one of our target industries, the music industries, where most of the global sales is algorithmic and AI-driven. Artists, bands, small labels, publishers, even small country national associations cannot remain competitive if they cannot participate in this technological revolution.

We started our operations on 1 September 2020 on the basis of CEEMID, a pan-European data observatory that created about 2000 music and creative industry indicators for its users. In the coming days, we are gradually opening up about 50 music industry and 50 broader creative industry indicators in a fully reproducible workflow, with daily re-freshed, re-processed, well-formatted and documented indicators for business and policy decisions.

We would like to validate this approach in one of the world’s most prestigious university-backed incubator programs, in the Yes!Delft AI/Blockchain Validation Lab.

Video credits

Data acquisition and processing: Daniel Antal, CFA and Marta Kołczyńska, PhD (survey data).
Documentation automation: Sandor Budai
Video art: Line Matson
Music: Moon Moon Moon.

Our Approach To A European Music Observatory

Wed, 26 Aug 2020 08:00:00 +0200

We are developing our Demo Music Observatory in the world’s 2nd ranked university-backed incubator program, the Yes!Delft AI Validation Lab. Our aim is to show a better organizational model, examples of research automation and other data integration innovation that can reduce the budgetary needs of the European Music Observatory by 80-90% and provide far more timely, accurate, and relevant service than most data observatories in Europe.

Background

Data is power, and big data creates injustice. Organizations that control large amounts of data, for example, the entire listening history of hundreds of millions of people in all major countries of the world, can train algorithms and robots that drive most of the music in the world. They can make your investment into a sound recording successful or doomed. They can circumvent or help a local content regulation, reinforce, or disable a national cultural policy goal. A country may introduce national artist quotas on radio, if all the youth will be personally recommended foreign songs in their music discovery age in the very same country.

We believe that answer of the EU and UNESCO with the creation of permanent data collection points, so-called data observatories is correct. However, after reviewing 62 data observatories, we are convinced that many data observatories are doomed to fail.

Our private observatory, CEEMID was consulted in the creation of the Feasibility study for the establishment of a European Music Observatory, and some of our recommendations found way into the consultant’s document. We created a Demo Music Observatory to provide a practical guidance on the decisions facing the European stakeholders, and to answer the questions that were left open in the Feasibility Study — particularly on data integration and the institutional model, where a wrong choice can lead to very long delivery time, quality control and budgeting.

Data Gaps

According to the feasibility study, 11% of the stakeholders consulted found that sufficient data was available for their work, 30% said that there could be more data but that does not affect their work, and 59% found that the lack of data affects their work.

We have shown in the Central & Eastern European Music Industry Report 2020, featured in the CCS Ecosystems: FLIPPING THE ODDS Conference – a two-day high-level stakeholder event jointly organized by Geothe-Institute and the DG Education and Culture of the European Commission with the Creative FLIP project.that much of the data gap is illusory: the fragmented nature of the music industry, the lack of in-house data expertise create bottlenecks in the use of the data. However, the data is present, and as it is stated in 148-149, we are willing to transfer thousands of indicators in all pillars to the future European Music Observatory.

Our Demo Music Observatory is based on the accepted music industry concept in the US and the EU (the three-income stream model) and its data maps that we could apply to emerging markets – for example, in the case of Croatia. A clear map of the industry and a clear map of the data reveals that much of the perceived gap is illusory. We would like to challenge the European music stakeholders to test us, request seemingly missing data that we will try to publish.

Illustration: our map to find to locate data in the Croatian music industry. Our Demo Music Observatory is based on the accepted music industry concept in the US and the EU (the three-income stream model) and its data maps.

This remains a problem with the Summary of Data Gaps on pages 128-131 that enumerate the perceived lack of data for the three pillars. We believe that these data gaps are in most cases illusory, and the remedies offered in the study are often misplaced.

Circulation of Repertoire & Cultural Diversity

Streaming activity: the Feasibility Study offers Nielsen’s Global Charts as a potential solution, even though we have shown that this is not the right approach.

The problem with Nielsen’s charts, or any chart provider’s chart, is that it covers less than 0.01% of the European repertoire, and it depicts a completely irrelevant picture for the 99.99% of the European stakeholders.

We believe that the right approach, followed by our Demo Music Observatory, is a reconciliation with the work on Big Data the European Statistical System (i.e. the partnership of Eurostat and the national statistical authorities.) (See our analysis of 20 advanced, emerging and future markets in Europe.)

Challenge Our Demo Observatory: Check out the Music Diversity & Circulation Pillar of our Demo Music Observatory. If you do not find what you are looking for, contact us — we will try to put the data there from our repositories.

People & Diversity

The Feasibility Study claims that “there is no real EU wide data on diversity in the music sector. Diversity should include the study of ethnic minorities, gender and vulnerable groups…”

However, active and music participation is available on EU level both for gender groups (see our chart) or four ethnic minorities – this is regularly featured in various European CAP surveys and in our national CAP surveys, too. Our peer-reviewed, open-source retroharmonize statistical software was developed exactly to create these data products.

Challenge Our Demo Observatory: Check out the Music, Society and Citizenship Pillar of our Demo Music Observatory. If you do not find what you are looking for, contact us — we will try to put the data there from our repositories.

Music Economy

Because music is mainly a market activity, apart from the mainly publicly funded classical (arts) music sector, and a part of the music (stage) infrastructure, a wide variety of data is available on the music economy in a consistent manner.

We are calling on all partners to challenge us, either on Music Economy or the other pillars of the music observatory. We will try to provide the requested data on a weekly, monthly, quarterly or annually refreshing basis in the highest possible quality – for the first 50 request for free. We would like to demonstrate that most of the data gap is illusory, and a collaborative, bottom-up approach can deliver a European Music Observatory faster, cheaper in a much higher quality.

Challenge Our Demo Observatory: Check out the Music Economy Pillar of our Demo Music Observatory. If you do not find what you are looking for, contact us — we will try to put the data there from our repositories.

Organization and institutional model

We believe that the biggest shortcoming of the Feasibility Study is a systematic overview of a wide range of organizational possibilities. The Feasibility Study is illustrated by examples from European Audiovisual Observatory as a good practice, investigates the possibility of integrating the music observatory into the EUIPO Observatory, and as an alternative, it considers the six product observatories of DG Agriculture & Rural Development. This approach provides a too narrow range of potential organizational alternatives.

We have reviewed a further 59 observatories, including ones that were not successful and ceased to exist, to understand why some observatories can increase their service level and why others fail. It is unclear what criteria make the European Audiovisual Observatory a good example, so it is impossible to say from this Feasibility Study if there are better examples – and we believe that there are. The EAO is one of the oldest, if not the oldest observatories. It was created before the EU was founded before the commercial internet launched, when far different legal, governance and data collection needs had to be addressed. We believe that this may be the costliest and most time-consuming model to replicate.

Bottom-up versus top-down

The Feasibility Study only considers hierarchical and centralized observatories, that operate on a top-down data integration, or rather, simple data collection strategy. We have created a large observatory with a very limited budget (which has a public layer, the Demo Music Observatory, and privately funded, much larger version, CEEMID) when we realized that the centralized, hierarchical, top-down approach is not suitable for a fragmented industry like the music industry. The example of CEEMID shows that a bottom-up and collaborative approach requires far less time, budget than a hierarchical approach, and can yield superior data quality. It is not surprising that most of the 62 observatories that we have studied are not hierarchical, and not operated by a centralized body but by a Consortium of various scientific, policy and industry stakeholders.

The European Music Observatory should fully exploit the EU’s open data regime. (See our examples: our first open data release) Our proposal about this were quoted in the Feasibility Study.
It should not re-collect data that already exists. Instead it should create policies, synergies and incentives that bring that data to light. The Feasibility Study partly embraces this idea.

Hierarchical versus decentralized observatory

We want big data to work for small venues, independent labels, startups, great and undiscovered artists. We believe that you cannot make a successful album launch, a market entry or introduce a successful cultural policy without large amounts of well processed and correctly analysed data. We want to create a Music Observatory that integrates the small data of many small bands, small labels, small venues, small countries, and mount correct the injustice. Make algorithms transparent and the competition fair.

It should not re-collect data that already exists. Instead it should create policies, synergies and incentives that bring that data to light. The Feasibility Study partly embraces this idea.
The observatory must allow a seamless data integration between open data, and highly confidential proprietary data. The Feasiblity Study wants to own and buy data, instead getting access to industry data.
The open collaboration principle must allow an easy opt-in and opt-out for small and larger, public and private organizations, and offer for full transparency for all users. The Feasibility Study offers a closed observatory without opt-in.

Value For Money, Budget

We believe that a bottom-up, collaborative approach of the industry stakeholders offers a much faster approach to growth, because it can mange conflicts of interests, build upon existing and trusted data assets. The annual budget of the European Audiovisual Observatory is about 3.7 million euros. Most of the European data observatories that we surveyed have an annual budget of 20,000-200,000 euros. In our view, a very comprehensive data observatory can be built from 1-3 million euros and maintained with a collaborative use of existing data assets and a subsidy of up to 200,000 euros a year.

Quality Control

It should be transparent, the indicator design should be based on open-source statistical software. Our proposal about this were quoted in the Feasibility Study.
The European Music Observatory must embrace evidence-based open Policy Analysis. This topic is not considered in the Feasibility Study.

Use Cases

We want to support the European music industry and our friends in North America, Australia, and all over the world to turn the tables with @ref(innovation) innovation coming from the open source community.

We want to help you to create alternative recommendation engines that actually recommend songs from your country, and we want to give you very clear export market targets with the help of AI. Our Listen Local Initiative is aiming to create recommendation engines for cities and regions, and make sure that local bands are recommended to local audiences and audiences in the regions where they will be touring after the Covid-19 pandemic.
We want your evidence to stand a fighting chance against large teams of professional legal and economics teams on the other side with proper valuations and damage claims. And we want to present all those hundreds and thousands of pages automatically, going through dozens and dozens of automated “unit-tests” until nobody can find errors.
We want you to be able to prove to your fans, the press, your economy minister that music in many countries has not been at the mercy of the taxpayer, but has been carrying far heavier tax burdens than manufacturers. We want to make your case that the music industry plays a vital role in the European economic recovery and job creation, because we can create economic impact assessments on GDP, employment, tax, import and export effects automatically.
Because music and culture are often managed at the level of cities, regions and communities, we want to give you all the data on sub-national levels, whether for regions, metropolitan areas or smaller divisions.

First CEEMID Regional Open Data Release

Thu, 16 Apr 2020 17:00:00 +0200

Internet use: participating in social networks (creating user profile, posting messages or other contributions to facebook, twitter, etc.) [percentage of individuals] shows the development of social media use in the past 10 years in very high detail. Social media is one of the most important ways to communicate with fans and find audiences for gigs, performances and to share new releases.

The chosen dataset is highlighting several new priorities of CEEMID. This indicator, which is based on a Eurostat statistical product, has a much higher data quality than the Eurostat release, it is easier to use, and goes beyond the detail of national statistics. It contains data on 411 territories in Europe, including countries, provinces and smaller regions, even some metropolitan areas.

The dataset is available on the European open research data repository, Zenodo. It will be updated annually.

The processing information is available in the dataset. First, we have used our open source software to correct the regional boundary changes during the decade (see Programatic Approach To Work With European Regional Data Panels).

When data was missing in the end of the dataset, we used the last observation carry forward, when in the beginning, the next observation carry backwards data. Gaps in the data were interpolated using linear interpolation. Whenever data was available on a larger (provincial level), we imputed it to the smaller regional units.

The data contains standard national / regional references according the European NUTS2013 standard for backwards compatibility. Further documentation will be available on our methodology on the documentation.ceemid.eu wiki website.

At last, the data was put into a table that complies with the tidy data principles for easier processing on computers and immediate use in spreadsheet applications.

Use this in your article? Further research? Collaboration? Let’s get in touch on LinkedIn, Twitter, or via danielantal.eu

CCS Ecosystems: Evidence-Driven CCI Policy & The Central & Eastern European Music Industry Report

Wed, 29 Jan 2020 16:00:00 +0100

CEEMID & Consolidated Independent presented and discussed with stakeholders the Central & Eastern European Music Industry Report 2020 as a case-study on national and comparative evidence-based policymaking in the cultural and creative sector on the CCS Ecosystems: FLIPPING THE ODDS Conference – a two-day high-level stakeholder event jointly organized by Geothe-Institute and the DG Education and Culture of the European Commission with the Creative FLIP project.

Daniel’s presentation was focused around the question on invisibility of the CCI sector in the economic, education and labour policymaking, and the low level of data use in fact-based cultural policy design, and the following policy design problems. Our CEE Music Industry Report 2020 created with the help and support of Consolidated Independent was used as a use case for the problems.

This blog post is an edited version of Daniel’s presentation and the subsequent panel discussion with the stakeholders and organizers.

The CEE Report builds on the results of the first Hungarian, Slovak, Croatian and Czech music industry reports are compared with Armenian, Austrian, Bulgarian, Lithuanian, Serbian and Slovenian data and findings.

Invisibility

The measurement problems and the ‘invisibility’ of creative and cultural industries is a consequence of the fact that these industries are predominantly comprised of microenterprises and individual, freelancing entrepreneurs. Even larger entities, such as collective management organizations can be classified as small enterprises. In the EU, small- and medium enterprises, and especially microenterprises usually have reduced reporting duties to the statistical authorities and tax authorities (which is the most important secondary source of government statistics in anonymized, aggregated form.) This means that the supply side is not visible for policymakers. The recommendation is to conduct sample-based surveys among these microenterprises, but very few member states do that.

Another problem is that on the demand side, not all member states create statistics. ESSNet-Culture created recommendations how to measure cultural access and participation, which is a broader term than cultural consumption, as it includes non-market forms, for example, liturgical music, amateur acting, and so on. The European Commission sometimes makes excellent, but not detailed CAP surveys, which are the most important sources of Eurostat’s relevant pan-European statistics.

CEEMID has been making such surveys, both on the supply side (in music and film) and in the audience side (in music, film, theatre, opera, and artistic activities) in several member states, and has plenty of experience with using the EU-mandated CAP surveys of 2007 and 2013. The CEE report relies on the latter date set in first part of the audience analysis. Our work was already presented in 2015 as a best use case in Creating better national cultural statistics with Eurobarometer datasets and ESSNet-Culture technical recommendations at GESIS, which maintains the Eurobarometer archives, on the Eurobarometer Symposium Four Decades of Surveying Europe – Perspectives on Academic Research with the European Commission’s Eurobarometer Surveys

For the EU the lowest hanging fruit would be to critically assess the past surveys, including the partial CAP survey made in 2017 for the cultural heritage part of culture, and the national surveys conducted by CEEMID and its partners, and to mandate an annual or biannual CAP survey within the Eurobarometer series, probably as a specially sampled “Special Eurobarometer”. This would very significantly reduce the costs compared to a country-to-country basis, and probably could be shared by other institutions, for example, with the European Audiovisual Observatory. This alone would greatly increase the demand / consumption side information on the services of creative and cultural enterprises.

In the panel discussion Consolidated Independent highlighted the problem of leaving sample-based statistics making to the industry itself, which leads to myopical incentives to create ‘policy-based evidence’ that is cherry picked by industry trade associations to fit their often not very well thought out policy agenda. In our view, the most important role of a future European Music Observatory would be to set standards to data collection and integration. The music industry is one of the most data-rich industries globally, but the publicly available, verified, trustworthy information is scarce both for policy-making and for business planning among microenterprises who have know specialized research know-how. Our already existing, representative national surveys should and could be extended to the whole Europe with the quality control and visibility offered by such an Observatory.

Industry organization

A very significant challenge with evidence-based policymaking, or, in fact, with any policy making is that the extremely fragmented nature of the industry makes national and EU-level consultations very difficult. Most of the workforce is neither a classical employer nor employee but works as a freelancer or a micro-entrepreneur. Their views are very rarely represented by trade unions or employee associations or chambers of commerce. Despite all efforts on EU level to make the social dialogue more representative for atypical and flexible work forms, the problems and the views of the creative and cultural sectors are largely unrepresented in policy debate.

A prime example of this problem is the very heavy over-taxation of the music industry that we have shown in the Hungarian and Slovak national music industry reports, and in other documents in Croatia. The problem is not only characteristic for the music industry, but generally all CCIs because they share similar economic properties: no real chance to use tax breaks, high level of labour input and high level of value added. While policy makers and granting authorities spend much time on trying to improve the financing of this sector, in fact, reducing over-taxation would make a much more imminent impact on the liquidity, profitability and growth opportunities of the sector.

However, in the making of economic and tax policies, the fragmented creative and cultural industries hardly can represent their special situation.

Policy issues

Apart from taxation issues, which significantly reduce the liquidity, profitability and growth opportunities of these sectors in many EU countries, particularly in the CEE, where corporate taxes are low but value-added taxes are high, there are other important policy problem that reduce the long-term growth capacity of the sector.

The very small enterprise size leads to missing strategic human resources (HR) and research and development (R&D) functions in almost all enterprises – including simple market research activities. Companies with 1-2 employees do not have specialized management and support functions, and thus they have a very strong handicap in HR and R&D.

A good example of this problem is the motion picture and TV industry. These industries were comprised of medium-sized enterprises in the 1980s with significant in-house education functions. The current structure of these industries, however, resembles music, with almost all enterprises staying below the 5-person threshold. In our experience, based on development needs assessments in the Hungarian motion picture and the Czech music industry, this creates an acute skills and labor shortage. Missing skills cannot be well replaced by recruitment, or with strategic HR development via life-long learning.

Similarly to very reduced opportunities to participate in, and create life-long learning schemes, there are very little chances to engage in market research and R&D. While the music industry, for example, is one of the most data-driven industries in the world, the microenterprise size does not allow that these enterprises commission market research or hire data scientists. This leads to very asymmetrical relationships with the main distributors of music and media content on platforms that are owned by global data companies.

Recommendations

Daniel made a few concrete recommendations in the panel.

Creating the European Music Observatory, and the creation of similar industry initiatives, and commissioning collaborative (multi-stakeholder) market research, as CEEMID works in many countries, can only replace the missing market research function in the extremely fragmented industries, such as motion picture production, the music industry, but to a certain degree architecture and design.
A low-hanging fruit would be a critical assessment of the previous Pan-European CAP surveys and the comparable national CAP surveys conducted by CEEMID and its partners following the ESSNet Culture recommendations and the inclusion of the CAP surveys in an annual or bi-annual form in the Eurobarometer (Special) series. A Special Eurobarometer may be more suitable because usually a non-proportional weighting of the sample gives the best value for money in these surveys.
Critically reviewing the participation of micro-enterprises in the current forms of social dialogue and looking for formal or informal ways to increase participation in policy making can gradually lead to more creative-friendly economic regulation and tax policies. The Hungarian Music Industry Report 2015 and the Slovak Music Industry Report 2019 contains very interesting case studies for this problem that we briefly summarize in the CEE Music Industry Report 2020.
Critically reviewing the few examples from mainly Nordic member states where collective agreements and other collaborative institutions allow microenterprises and freelancers to participate in life-long learning problems can help designing policies that will gradually reduce the huge skill gap of the CCIs in management know-how, data know-how, or renew their technical skills necessary in the production of their services and content. Our work with the Hungarian National Film Fund and surveys among more than a thousand film makers can give much insight into this problem.

Further links

Our research findings were earlier presented and discussed in Vienna, Prague, Budapest and Bratislava with stakeholders.

You can follow Creative FLIP project on Twitter.