๐MusEmoji๐
LET YOUR FACE ๐คชPLAY THE MUSIC
Description
MusEmoji ยฉ๏ธ is an iOS app, which allows you to play a variety of MIDI musical instruments with your facial expressions
Inspiration & Motivation
We play music mostly with our fingers, hands, mouths and feet. There is a good reason why we don't use other parts of our bodies like ties, legs or butts for making music. The reason is that the hands with the fingers and the mouth with the lungs provide fine muscle control. One part of our bodies, which also has fine muscle control but is largely overlooked is our faces. The the more than 50+ facial muscles are capable making dozens of distinct facial expressions. Until recently, however it was almost impossible to harness these muscles and use them for music making. Fortunately, new technologies like the TrueDepth cameras built into some smart phones and Augmented Reality tool kits like ARKit in iOS and ARCore in the Android operation systems are finally making this possible. Unlike any other musical instruments the MusEmoji app is played with your facial expressions. From all possible facial expression, which you are capable of doing we have chosen only 12 distinct ones that can be easily done by anyone. We have mapped those expression to the notes in the musical octave.
The Software
The MusEmoji app runs on iPhones or iPads version X or higher. It uses the TrueDepth camera to do face tracking and maps some of the most common facial expressions to musical notes. The app is a MIDI controller & synthesizer. It features 12 different MIDI instruments. Developed by Brainventions, Inc. MusEmoji is a member of an entire family of similar apps (MusiGigZ) designed for Making Music in Motion. Check out Sk8SoniX (a.k.a. MusiSk8s) for ice and roller skating, MusiShuZ for dancing, MusiMoto for all kinds of movement activities like walking, jogging, bicycle riding, ice skating and even dancing.
MusEmoji App
The MusEmoji app currently runs only on iOS 12 and iPadOS 12 or higher and devices model X or higher, which have TrueDepth camera. This camera is necessary for Face Tracking. The app is available for download on the Apple AppStore.
ย
The MusEmoji app is localized in 20 of the most popular world languages including: ar ๐ฆ๐ช, cs ๐จ๐ฟ, de ๐ฉ๐ช, el ๐ฎ๐ฑ, en ๐ฌ๐ง๐บ๐ธ, es ๐ช๐ธ, fr ๐ซ๐ท, he ๐ฌ๐ท, hi ๐ฎ๐ณ, hu ๐ญ๐บ, id ๐ฎ๐ฉ, it ๐ฎ๐น, ja ๐ฏ๐ต, ko ๐ฐ๐ท, nl ๐ณ๐ฑ, po ๐ต๐ฑ, pt ๐ต๐น๐ง๐ท, ru ๐ท๐บ and zh ๐จ๐ณ. To change the app's language to one of your choice follow the instructions in the Help โน๏ธ section of the app.
ย
To play the app your face must be sufficiently lit up. The iPhone screen illumination, if turned up to the maximum, should be sufficient so you can play even while in bed in the middle of the night.
How to Play
For detailed instruction on how to play the MusEmoji app tap the โน๏ธ button on the top menu.
ย
If you want to see yourself as you play, tap the video ๐ฆ button. For proper functioning of the app make sure that your face is always in front and facing straight the iPhone TrueDepth camera. In this mode your face will be tracked and the background will be green ๐ฉ.
ย
If the camera does not see your face it will switch to an idle mode (red background ๐ฅ) and will try to gain your attention again after 2 minutes of inactivity.
ย
To change instruments scroll left or right the instruments scroll view and tap the instrument of your choice. Alternatively you can change instruments by tapping the instrument icon on your forehead when the video mode is enabled. There are 12 MIDI instruments in the app that you can choose form including boy and girl solfege (๐ป ๐น ๐ช ๐บ ๐ช ๐ฅ ๐ท ๐ธ ๐ฏ ๐ ๐ฉโ๐ค ๐จโ๐ค).
ย
You can play musical notes Soft (piano) or Loud (forte). To change the volume/loudness ๐ of the note you play tilt your head slightly up or down. NOTICE: The app is very sensitive to slight head moves and changes in the expressions.
ย
The default octave is C4 4๏ธโฃ.ย To change the octave in which a note is played turn your head slightly left for lower octave 3๏ธโฃ or right for higher octave 5๏ธโฃ.
Face Expressions to Notes Mapping
The Mapping
As seen on the screenshot the white piano keys and the corresponding facial expressions are marked with their solfege notations Do ๐, Re ๐คฅ, Mi ๐, Fa ๐ฎ, Sol ๐, La ๐, Ti ๐ต (C, D, E, F, G, A, B). The 5 black keys (diezes & bemoles a.k.a. sharps & flats) are labeled with the 5 vowels of the Roman alphabet A(C#) ๐, E(Eb) ๐ก, I(F#) ๐ , O(G#) ๐คฅ, U(Bb)๐.
ย
Why these Choices ๐
In developing the MusEmoji app we had fo find and test an optimal mapping between facial expressions and musical notes. We had several considerations. The staring point was to choose the neutral face ๐ when no notes are played. This is equivalent to having fingers off the piano keys. We decided to NOT use the eyes with their up/down & left/right motions so that they can be used for the purpose nature intended them - seeing, reading. We also excluded the eye blinks since they occur naturally (even though they can be used for playing repeated notes) . What was left is the rest of the facial muscles including mouth, nose, eye brows, cheeks and tongue. Our current mapping choices are shown on the screen shot.
ย
Duets and Harmonies ๐ญ
The MusEmoji app can track two faces ๐ฅ at the same time. Just stand next to your partner so that the TrueDepth front camera can see you both. You can play the same or different instruments in the same or different octaves. Give it a try and enjoy. This is a great way to practice playing harmonies.
Practice Melodies
To help you master playing music with your facial expressions the MusEmoji app includes views of several popular melodies. Tap the ๐ถ button on the menu and swipe left or right to choose a melody. Each melody can be displayed in 5 different notations (SCR - score, DRM - solfege, CDE - American, 123 - Asian, OJI - MusEmoji). You can also watch a video of the chosen melody played by an experienced MusEmoji player.
The MusEmoji Game ยฉ๏ธ
Designed to help you master your MusEmoji music making skills, the MusEmoji Game is inspired by the well known Piano Roll, Guitar Hero , Dance-Dance-Revolution and even the original Mickey Egg Catcher and BeerTender games. To play it you do not need any hardware. This is entirely a software-based game featured in the MusEmoji app and controlled by your face.
ย
To start the game tap the โญ button. All notes types, which comprise the selected melody are displayed on the outer red circle as the hours on a clock face โฐ. There are 12 possible notes -the black and white keys comprising an octave. When you tap the start ๐ฌ button the notes will start floating to the center of the circle in the order in which they appear in the musical score of the melody. When an emoji touches the inner red circle - your target - you have to make the corresponding facial expression and keep it until the next emoji in the sequence touches the inner red circle. That will cause the note mapped to this emoji to play.
ย
Tap the level button 1๏ธto change the speed of playing. There are 3 possible levels/speeds - each one 2x faster than the previous.
ย
If the melody requires that you play a note in a lower or higher octave you will be prompted by a ๐ปor ๐บ icon on the bottom. Turn your face to the left or right to change the current octave.
ย
ย
ย
That's it in a nutshell. Enjoy! ๐
Warnings & Benefits
This app is currently still under active development and there is no warranty
Warning โ ๏ธ
Prolonged usage of the MuEmoji app can cause fatigue of the facial muscles. The MusEmoji Game may be addictive. Practice self monitoring and restrain! ๐ ๐
Benefits ๐ ๐ฉโโ๏ธ ๐
Playing the MusEmoji app is a great way to exercise your facial muscles. Persistent and regular practice may have a rejuvenating effect since all major muscle groups in your face are used during playing. By providing deterministic audio feedback, the MusEmoji app is RECOMMENDED for individuals who are interested in improving the responsiveness, speed and accuracy of their facial expressions including actors and actresses. With the built-in complete Voice Control it is also RECOMMENDED for disabled individuals โฟ including quadriplegics who want to play musical instruments HANDS-FREE.
Disclaimer - Release of Liability
Play at your own risk! By accepting and following the instructions for playing the MusEmoji app, the you - the user - release the creators of MusEmoji Val Nenov and Brainventions, Inc. of any liability for potential injuries๐ฉธ including facial spasms โก, fatigue and addiction ๐คข.
Compare with EyeHarp
A similar project for making music by people with disabilities!
Privacy Policy
At Brainventions, we believe privacy is a fundamental human right.
Use of the TrueDepth Camera
The MusiFace aka MusEmoji app works only on iPhone X and higher and iPads with built-in TrueDepth camera. The app uses the camera to detect your facial expressions and map them to musical notes played by the built-in MIDI controller & synthesizer. The camera also detects your head orientation (straight, up, down, left, right) and uses this information to setup the octave number and sound volume of the played notes. The TrueDepth camera data is also used in one of the built-in games. It saves pictures of your face in your device's cache folder when you make the facial expressions prompted by the game. These pictures are shown back to you as a visual feedback at the end of each session and are deleted from the device when you start a new game session by pressing the START button. The MusiFace app DOES NOT export or make available any of these temporarily stored face pictures to any third-parties.
Your personal data belongs to you, not others
Whether youโre playing music with your MusEmoji app, recording the music you play, or sharing it with your friends, you can do it knowing that Brainventions doesnโt gather your personal information to sell to advertisers or other organizations.
Your features improve while your data stays private
When we use data to create better experiences for you, we work hard to do it in a way that doesnโt compromise your privacy.
Whether you store it or send it, your data is protected
The MusEmoji app provides a feature to save your performances as MIDI files. The data is stored on your smartphone. Only you have access to your data. You can of course decide to share it with us to be used as foundation for further app development and improvements.
Your apps play by your rules
The MusiVINGZ app requires access to the video camera on your smartphone so that you can video record yourself while you are playing. It superimposes the video to the main screen. MusiVINGZ also uses Bluetooth to communicate with the numeric keypads. When the app launches for the first time you are prompted to provide access to these peripherals. These settings are stored automatically but you can change them at any time.
In-App Advertisement
The MusEmoji app does not show any ads.
Contact
Ask questions, leave comments and suggestions