Toga.Sound implementation design

[proneon267]

I will be implementing the Toga.Sound class.

The following are the methods and members that Toga.Sound class will have:

• path - Path to the sound file
• data - Bytes representation object of the sound file in a supported format
• play() - Play the sound resource
• stop() - Stop the sound resource

Other than pause() and resume(), what other APIs do you all think should be present?

[freakboy3742]

Those APIs are an obvious surface interpretation of a Sound API. Some questions/suggestions/areas for further research:

• What's the functional difference between pause() and stop()? My immediate guess would be that pause() retains the "current position" in the audio file - is that supported by the underlying API

• In terms of additional features - some additional ideas which may (or may not) be plausible:

  • something to control "looping" - i.e., start a background track of music that just keeps playing; when it gets to the end, the track restarts.
  • something to control "mixing" - i.e., if I have multiple sounds playing, what are the relative volumes of those two tracks? This is independent of the volume on the device; I might want background music to play quietly, but my "laser zap" to be quite loud.

• Is there a need to differentiate the Sound from a Sound Player? I honestly don't know - I'm not especially familiar with audio APIs. Especially if we're looking at mixing, there might be a need to have a central point of control.

• How does this interact with streaming audio? "Play this file from disk" is going to be a different proposition to "Play this file streaming from a server".

However, the biggest question is a meta-question: Is there a need for this in Toga at all? Could this be done by adding iOS/Android support to an existing audio library (like PyAudio)? I haven't done any work with Audio so I have no idea what the "state of the art" is on Python - but I know we've had multiple people asking about using PyAudio on our support forums. If we can let this be someone else's problem, that's going to be preferable to rolling our own; but if there isn't a good option out there that plays well with async and binary packaging, we might need to consider building our own.

If we do need to build our own, it would be worth looking at the APIs provided by other platforms (including the native APIs we need to integrate with) to see what functionality they provide for audio - both in terms of determining common language and API patterns, and also to see what is even possible.

It might also be worth considering whether this needs to be in toga core - perhaps a standalone "toga-audio" package would make more sense. Every application needs buttons and dialogs; only a very specific set of applications need to be able to play sound.

[proneon267]

Honestly, I was aiming for a toga counterpart of https://doc.qt.io/qt-5/qsound.html.

For achieving that, I was planning on using:

• On windows: https://learn.microsoft.com/en-us/dotnet/api/system.media.soundplayer?view=dotnet-plat-ext-7.0
• On Linux: GStreamer or Gst
• On Mac and iOS: https://developer.apple.com/documentation/avfaudio/avaudioplayer?language=objc
• On Android: https://developer.android.com/reference/android/media/MediaPlayer

PortAudio doesn't seem to be widely used on iOS. Aside from it, there are many more libraries which officially support wide range of desktop and mobile platforms. Most notably, https://miniaud.io/index.html supports all of the platforms that toga targets.

I agree that adding a Sound abstraction will be more work for GUI framework, especially adding features that users might expect out of a high end audio library.
I am not sure if I should proceed with this, if this will become a burden on the project later on. What do you all think?

[freakboy3742]

FWIW: PortAudio may not have iOS support, but it uses CoreAudio for macOS, and iOS uses the same API. I don't know if there's any additional complication there, but at least at a surface level, it may not be too hard (those are famous last words, I know 😄)

I agree that having the ability to use sound is an important part of an app. They key point that Malcolm and I are making is that the goal of "can use sound in an app" doesn't necessarily mean that Toga needs a sound API. If we're able to say "if you need sound, use PyAudio" (or whatever), that makes our job a lot easier, because it's one less thing we have to maintain.

Qt isn't a great point of comparison here, because they're a commercial company, and they actively want to be a one-stop-shop for all your app development needs. From their perspective, if you use a non-Qt API for anything, that's an opportunity lost for them to sell products and services. We don't have that constraint - we're an open source project, and it isn't a failure of Toga if you can use some other library to achieve a key goal of app building.

Is this worth working on? Yes and No.

At some level, we're going to need to solve the Sound-in-apps problem. What we're advocating is that we make sure we've ruled out a communtity-shared approach for a good reason before we embark on "build a whole new sound API".

So - my suggestion for a way forward is to try and rule out PyAudio (and other existing desktop audio solutions) working on iOS and Android. That could be a "proof by implementation" (actually getting it working), or "proof by research" (e.g., finding documentation that proves a wrapper around CoreAudio won't do what we need iOS, or why AVFoundation is a better layer to manage audio than CoreAudio on macOS); but either way - we need to rule out those options before we start building a Toga-owned design.

[freakboy3742]

To that end - I will say that at a surface level miniaudio looks really promising... it covers all our supported platforms (including web), and it has existing Python bindings. I've never heard of it before, but if it works...

[proneon267]

I agree with you and Malcolm, we should recommend similar audio libraries like miniaudio to users in need of audio support.
Hence, I have decided to not move forward with this API implementation.
Thank you both for helping :)

[mhsmith]

Miniaudio looks like a good option, and by coincidence someone actually contributed it for Android just a couple of months ago (chaquo/chaquopy#896). But I've only tested its MP3 decoder, and I don't know whether its playback functionality works.