Google recently released its MusicLM language model, which can create new music from text inputs, to the public last month after announcing it in January.
Meta, the parent company of Facebook, recently introduced its own AI-powered text-to-music generator called MusicGen. According to the company, the model was trained using 20,000 hours of licensed music, including 10,000 high-quality recordings and 390,000 instrumental tracks sourced from Shutterstock and Pond5.
However, digital and computing behemoths other than Meta and Google are also pursuing research in the field of AI music.
Rival Microsoft oversees a sizable research initiative focused on AI music. It’s named “Muzic,” and its academics’ work includes lyric generation, lyric-to-melody generation, songwriting, and more using AI.
Microsoft explains “Muzic” as an AI music project that uses deep learning and artificial intelligence to improve music understanding and creation.
Here’s a screenshot of their landing page’s diagram:
One of the initiatives under the ‘The Deep and Reinforcement Learning Group’ at Microsoft Research Asia (MSR Asia) in China is Muzic, which was founded in 2019.
“A world-class research lab” with locations in Beijing and Shanghai is what Microsoft Research Asia is referred to as. The tech giant claims that MSR Asia, which was founded in 1998, “conducts basic and applied research in areas central to Microsoft’s long-term strategy and future computing vision”.
The ‘The Deep and Reinforcement Learning Group’ also conducts research on Neural Machine Translation, Text-to-Speech models based on neural networks, and AI Music.
I’ll say it again: ‘Muzic’ has already created a sizable body of work in the field of AI Music.
Some of its notable projects are listed below:
DEEPRAPPER,
Of all the projects Muzic is working on, this one might make some owners of music rights throw up their coffee.
A ‘rap generator’ powered by AI called DeepRapper was created by Muzic researchers in 2021.
According to the paper explaining the creation and testing of the text-based model, DeepRapper is the first [AI] system to produce rap that contains both rhymes and beats.
Additionally, they state that “both objective and subjective evaluations demonstrate that DeepRapper generates creative and high-quality raps.” You may find the DeepRapper code that was made available on GitHub here.
The researchers state that “since there is no available rap dataset with rhythmic beats,” they created “a data mining pipeline to collect a largescale rap dataset, which includes a large amount of rap songs with aligned lyrics and rhythmic beats” in order to create the DeepRapper system.
They also created a “carefully modelled” “transformer-based autoregressive language model” that “carefully models” rhymes and rhythms.
Later on in the paper, they go into greater detail about how they created “a data mining pipeline [to] collect a large-scale rap dataset for rhythm modelling” (see figure below).
According to their explanation, “To mine a large-scale rap dataset, we first scrape a huge number of rap songs from the Web that contain both words and singing audios.
In order to verify that the lyric and audio can be aligned at the sentence level, which is helpful for our eventual word-level beat alignment, we additionally crawl the start and finish time of each lyric sentence corresponding to the audio.
They continued to mine data for the study after that. Additionally, they employed their “data mining pipeline to gather another two datasets: Since non-rap songs are more diverse than rap songs, the non-rap song dataset can be larger than the rap song dataset. The pure lyrics dataset can be even bigger than the non-rap song dataset.
The two datasets mentioned above were used to train the DeepRapper model during the “pre-training stage”. They mentioned that they “fine-tune our pre-trained model on the rap songs with aligned beats” after that.
The study’s authors write in their conclusion, “Both objective and subjective evaluations demonstrate that DeepRapper generates high-quality raps with good rhymes and rhythms.”
You can check out a few of the 5,000 samples they generated at random right here.
The samples were created in Mandarin, and the researchers translated them into English using Google Translate.
(The opening line of the examples provided? Let this song arrive to the night of medical insomnia since we have yellow skin and hot blood.
We can create another rap singing system to sing out the raps in accordance with the rhymes and rhythms, which we leave as future work, due to DeepRapper’s design, the report concludes.
It is now generally acknowledged that generative AI models are trained on enormous data sets, frequently downloaded from the internet.
The risk of infringement by those AI models of music with copyrights is a fact that music rights holders do not particularly enjoy. The Microsoft team’s open admission of how DeepRapper’s data is acquired, albeit for research purposes, is what makes this story noteworthy.
It’s interesting to note that Microsoft looks to be conducting study on rhymes and rapping on a global scale.
Microsoft has a US patent for a “Voice Synthesised Participatory Rhyming Chat Bot” that looks to be an altogether different tool from DeepRapper, in addition to the DeepRapper model described above, created by the Muzic team in China.
The developers of this “rap-bot” technology are another team of Microsoft researchers working in the US. It was awarded a patent in April 2021.
The chatbot’s various functions are listed in the file, which MBW was able to get. The chatbot, for instance, “may support rap battles” and “participate in the music creation process in a social way.”
Comment