Smart orchestra

From connected baton to orchestra 2.0

 Project genesis

This project was born after a complicated orchestra rehearsal due to the absence of musicians. The idea came to me that virtual instruments could compensate for the absence of some musicians, especially those on whom the others are based. But how do you synchronise virtual instruments during rehearsal?

The simpliest way is to start from the conductor who guides the musicians. By developing a connected baton able to detect his movements, it would be possible to transcribe and interpret this information.

Regarding the sound system, it is better to play the sound in the middle of the musicians so that they find the sound environment to which they are accustomed. The idea is to position bots at the location of each absentees, each bot then playing a specific voice.

 Solution

The implemented solution is based on a network of interconnected devices:

• A connected baton picks up and interprets the conductor's movements,
• A computer retrieves scores available online from Adimuse and distributes them to other devices,
• Bots (robot musicians) spread around the room play the voices assigned to them,
• The conductor controls the ssytem via a dedicated tablet : choosing the score to be played, assingning of voices to the bots, starting bar, …),
• A box ensures exchanges between all the devices,
• As the icing on the cake, musicians can use a web application to display their score and synchronize it with the conductor's movements. No more turning pages or counting bars.

 Overview

Interactive image. Move the mouse and click on the elements to freeze the explanation window.

• Conductor
• Connected baton
• Conductor tablet
• Indicator light
• Musicians
• Score tablet
• Bots
• System
• Adibox
• PC Scores
• Internet access
• Adimuse

More info... Hide detail...

Components of the connected orchestra

The conductor

The conductor beat as usual.

Its movements are captured by the connected baton to synchronize bots and musician's tablets

The conductor tablet allows him to set up the bots, select the piece to play and enter the starting bar.

The indicator light, positioned in his field of vision, informs him of the status of the système.

The connected baton

The connected baton captures and interprets the conductor's movements.

The extracted data is used to synchronize bots bots and musician's tablets.

Its physical characteristics (size, weight) are close to those of a classical baton so as not to change the conductor's habits.

The conductor tablet

The conductor tablet allows the conductor to change the piece and indicate the starting time.

The conductor can also control each of the bots (assigned voices, sound volume, etc.).

In the event of an anomaly, it also makes it possible to find detailed information on the status of the système components.

At the end, the conductor can obtain an evaluation of his performance, which is very useful in the context of learning.

The indicator light

Like the red "On Air" lamp in the recording studios, the indicator light inform the conductor of the state of the system : At rest, ready to start, song in progress.

In case of a system error (internet connection problem, too much wifi disturbances, ...) the bulb flashes. The conductor can then consult the details of the error on his tablet or on the Score PC.

The positioning of the bulb in the distance allows the conductor to access information at any time, and without having to focus his gaze too precisely on his tablet.

The musicians

The musicians play as usual, following the conductor's movements..

They can swap their paper scores for a tablet (called a tablet score). No need to turn the pages or count the bars, the scores scroll by themselves according to the progress of the piece and a cursor indicates precisely the note to be played.

When a musician is absent from a rehearsal, a bot can play his score so that the other musicians keep their bearings.

The tablet score

The tablet score allow the musician to have his score available with automatic scrolling according to the progress of the piece. No need to turn the pages or count the measurements!

The musician can take advantage of all the facilities for reading a score offered by Adimuse (choice of voices to display, precautionary alterations, display proportional to time, page/linear mode, display of unmastered fingerings, ...).

The Bots

Bots fill in the gaps related to absent musicians.

When a musician is absent from a rehearsal, a bot can play his score so that the other musicians keep their bearings.

Bots can be distributed to the locations of the musicians whose scores they play, to further limit disruption due to the absence of the latter.

Bots work autonomously. Once started, they obey the conductor via the conductor's tablet for the choice of the score and the starting position, and via the connected baton to follow the conductor movements (start, rhythm and stop).

The system

There are other elements involved in the process. Although they are less visible to users, they are still essential:

• The Adibox is the center of the system. It is the one that ensures the exchanges and synchronization between all the elements of the system.
• The PC score recovers the partitions on Adimuse. It transcodes them into formats adapted to each type of device and distributes them to the bots, the connected baton, the conductor tablet and the score tablest.
• The internet connection gives internet access to the PC scores which can then download the scores from Adimuse.
• Adimuse is the online sheet music editor from which the scores used here come from.

Adibox

The Adibox is the center of the système. It is the one that ensures the synchronization and exchanges between all the technical elements.

The Adibox works in complete autonomy. Plug it in, and connect to the WIFI network it creates.

The PC scores

The PC scores partitions recovers the partitions on Adimuse. It transcodes them into formats adapted to each type of device and distributes them to the bots, the connected baton, the conductor tablet and the score tablet.

At the beginning of the session, the conductor preloads the scores that will be played. The PC scores must remain on, and works independently for the rest of the session.

Internet access

The internet access gives internet access to the PC scores which can then download the sheet music to Adimuse.

The internet access can be an operator box or a connection sharing.

Adimuse

Adimuse is the online score editor where the scores used here come from.

The user can write their own scores or choose a score from those published by the Adimuse community.

Find out more d'informations on www.adimuse.com.

 Status

A prototype is now operational, and work is underway to make it even easier to use:

• Automatic detection and interconnection of the various components of the system,
• Self-adaptating baton : currently, the baton is based on pre-recorded parameters, specific to each conductor, to optimize the capture of movements. The idea is that the baton will be able to self-adapt these parameters according to the user.

 Openings

Integration of the visually impaired

The signals interpreted by the baton can be relayed to the visually impaired via a haptic device.

This development would make it possible to open the doors of the orchestra to visually impaired people. In this case, during performances, the connected baton can be used without the bots.




Orchestra conductors training

With a system like this, the apprentice conductor would be able to practise in complete autonomy while enjoying an interesting sound experience.

It could also be used to evaluate the conductor's performance and highlight areas for improvement.

 Project owner

• 
  Corentin Kerampran
  Engineering student in Computer Science at UTC. I'm passionate about music and enjoy applying my technical knowledge to my passion.