Build a mod

Audio and voice chat

Play your own audio, use the player's microphone and connect players with WebRTC to build proximity voice chat.

Advertisement

Mods can play their own audio, hear the player through their microphone, and connect players straight to each other with WebRTC. Together that is everything a proximity voice chat needs: the closer another player is, the louder you hear them.

LibraryWhat it doesPermission
audioplays sounds and streams of samplesnone
micthe player's microphonemicrophone
rtcvoice connections to other players (WebRTC)webrtc

Play a sound

audio.buffer turns raw samples into something you can play, and audio.play plays it. Samples are a string of bytes: by default signed 16-bit numbers, two bytes each, low byte first. This makes one second of a beep:

local function s16(v)
  local n = math.floor(v * 32767 + 0.5)
  if n < 0 then n = n + 65536 end
  return string.char(n % 256, n // 256)
end

local samples = {}
for i = 0, 47999 do
  samples[#samples + 1] = s16(0.3 * math.sin(i / 48000 * 440 * 2 * math.pi))
end
local beep, err = audio.buffer(table.concat(samples), { rate = 48000 })
if not beep then
  print("No audio here: " .. err)
end

commands.register("beep", function()
  if beep then
    audio.play(beep, { gain = 0.5 })
  end
end, "Play a beep")

audio.buffer(samples, options) gives back a buffer, or nil and a message:

OptionDefaultWhat it does
format"s16""s16" (16-bit), "f32" (32-bit float) or "u8" (8-bit), little endian
channels11 for mono, 2 for stereo (left and right samples take turns)
rate48000samples per second, from 8000 to 96000

A buffer has duration (seconds), frames, rate, channels and release(). All of a mod's buffers together can hold 32 MB of samples, about three minutes of mono audio, so release the ones you are done with.

audio.play(buffer, options) gives back a sound:

OptionDefaultWhat it does
gain1volume, from 0 to 4
pan0-1 is left, 1 is right
pitch1speed and pitch, from 0.25 to 4
loopfalseplay it again and again until you stop it
position{ x, y, z } in the world. You hear it from where your camera is
distance16with a position, how many blocks away it fades to silence
category"master"which volume slider applies, see below

A sound has playing, stop(), setGain(g), setPan(p), setPitch(p) and setPosition(x, y, z, distance). setPosition(nil) turns it back into plain stereo. Calling these on a sound that has ended does nothing.

Volume sliders

Players control your audio with the game's sound settings. Master always applies, and so does the slider of the category you pick: master, music, record, weather, block, hostile, neutral, player, ambient, voice or ui. Voices from other players use voice (the Voice/Speech slider) unless you choose another.

Sound files

To play .ogg files, put them in your zip under assets/<mod id>/sounds/ with a sounds.json, like a resource pack, and play them with sound.play("<mod id>:name"). See Blocks, items and mobs.

Stream samples as they arrive

audio.stream(options) plays samples that you push a little at a time, for example voice that arrives over your own connection, or sound you make up as the game runs:

local stream = audio.stream({ rate = 48000, latency = 0.1 })

on("tick", function()
  if stream.queued < 0.2 then
    stream:push(makeMoreSamples())
  end
end)

It takes format, channels and rate like audio.buffer, the options of audio.play except pitch and loop, and latency: how many seconds it waits before it starts playing, so a chunk that arrives late does not cause a gap (0.1, from 0.02 to 1).

stream:push(samples) queues up to one second at a time. It gives back false when two seconds are already waiting, so you never fill memory. stream.queued is how many seconds are waiting, and stream:close() stops it. A stream also has setGain, setPan and setPosition.

A mod can play 32 sounds and streams at the same time. audio.available() tells you whether audio works here at all.

The microphone

The microphone needs "microphone" in your mod's permissions. The browser asks the player the first time a site uses it. Zeus adds one more rule, the same one browsers use for file pickers: the microphone only starts right after the player clicks or presses a key. Start it from a button, a keybind or a command:

keys.bind({
  id = "talk",
  label = "Push to talk",
  default = "v",
  onPress = function()
    if not mic.active() then
      local ok, err = mic.start(function(started, why)
        if not started then ui.notify("Voice", why) end
      end)
      if not ok then ui.notify("Voice", err) end
    end
    mic.setMuted(false)
  end,
  onRelease = function()
    mic.setMuted(true)
  end,
})

mic.start(options, callback) gives back true when the browser was asked, or false and why not, like when there was no click or key press. Both arguments are optional. The callback runs once the browser answers: with true, nil, rate when the microphone is on, or false and why (the player said no, there is no microphone, another program is using it).

OptionDefaultWhat it does
echoCancellationtrueremoves the sound of your speakers from the microphone
noiseSuppressiontrueremoves background noise
autoGainControltrueevens out loud and quiet speaking
onFramea function that gets the newest samples about every 20 ms, while not muted

onFrame gets signed 16-bit mono samples at mic.rate() samples per second (usually 48000), in the same format as audio.buffer. You do not need it for voice chat, because WebRTC sends the microphone by itself.

FunctionWhat it does
mic.stop()turns the microphone off
mic.active()true while it is on
mic.setMuted(muted)while muted, other players hear silence and onFrame stops, but the microphone stays on
mic.muted()true while muted
mic.level()how loud the player is right now, from 0 to 1. It also works while muted
mic.rate()the samples per second of onFrame

mic.level() working while muted is what voice activation needs: unmute when the level goes over a threshold, mute again a moment after it drops.

While any mod has the microphone on, a red Microphone on label with the mod's name shows in the top left corner of the game. Mods cannot draw over it or hide it. The microphone turns off when your mod calls mic.stop(), reloads, is turned off or crashes.

Voice chat with WebRTC

WebRTC connects two players straight to each other and sends the microphone with low delay and good quality. It needs "webrtc" in your permissions, and "microphone" too to send the microphone.

Before two players can connect, they have to swap two messages: an offer from the player who calls, and an answer from the player who picks up. How they get to each other is up to you. Usually it is your own little server, through web requests.

local SERVER = "https://voice.example.com"
local voices = {}

local function sent(what)
  return function(body, err)
    if not body then print("Could not send the " .. what .. ": " .. err) end
  end
end

local function connect(name, offer)
  local peer, err = rtc.connect({
    iceServers = { { urls = "stun:stun.l.google.com:19302" } },
    mic = true,
    onAudio = function(voice) voices[name] = voice end,
    onState = function(state) print(name .. ": " .. state) end,
  })
  if not peer then
    print("Voice chat is not available: " .. err)
    return nil
  end
  if offer then
    -- They called us: answer, and send the answer back through the server.
    peer:answer(offer, function(answer, why)
      if not answer then
        print("Could not answer " .. name .. ": " .. why)
        return
      end
      http.post(SERVER .. "/answer", { from = player.name(), to = name, sdp = answer }, sent("answer"))
    end)
  else
    -- We call them: make an offer and send it through the server.
    peer:offer(function(sdp, why)
      if not sdp then
        print("Could not call " .. name .. ": " .. why)
        return
      end
      http.post(SERVER .. "/offer", { from = player.name(), to = name, sdp = sdp }, sent("offer"))
    end)
  end
  return peer
end

-- When the server hands the caller the answer:
--   peer:accept(answer, function(ok, err) end)

-- Louder up close, silent past 24 blocks:
on("tick", function()
  for _, p in ipairs(world.players()) do
    local voice = voices[p.name]
    if voice then
      voice:setPosition(p.x, p.y + 1.6, p.z, 24)
    end
  end
end)

rtc.connect(options) gives back a peer, or nil and a message:

OptionDefaultWhat it does
iceServersnoneSTUN and TURN servers that help players find each other, at most 8: { urls, username, credential }
relayOnlyfalsesend everything through your TURN server, so players never see each other's IP address
micfalsesend this mod's microphone whenever it is on and not muted
onAudioruns with a voice when the other player's audio arrives
onStateruns when the connection changes: connecting, connected, disconnected, failed, closed
category"voice"which volume slider applies to the voices

You can call mic.start before or after rtc.connect. A peer with mic = true starts sending as soon as the microphone is on.

PeerWhat it does
peer:offer(fn)makes an offer and calls fn(sdp), or fn(nil, error)
peer:answer(offer, fn)answers an offer and calls fn(sdp), or fn(nil, error)
peer:accept(answer, fn)uses the answer to finish connecting, then calls fn(true), or fn(false, error)
peer.statethe connection state, like "connected"
peer:close()hangs up

Offers and answers already list every address the player can be reached on (the peer waits up to 5 seconds to find them), so one message each way is enough. Offering or answering again on a connected peer changes the connection without hanging up.

A voice is one other player's audio. It plays on its own, even without onAudio. It has id, active, level() (how loud they are right now, for a talking icon), setGain(g), setPan(p), setPosition(x, y, z, distance), setPosition(nil) and stop().

A mod can have 32 connections and 64 voices at a time. Everything closes when your mod reloads or is turned off.

Tips

  • Use headphones. Echo cancellation may not hear audio that mods play, so with speakers the other player can hear themselves.
  • Players on strict networks (some schools and offices) may need a TURN server to connect. Some networks block WebRTC entirely.
  • Hang up with peer:close() when a player leaves, and connect again when they come back.

Privacy

Players should always know when a mod can hear them, so:

  • The install prompt lists what a mod asks for: microphone ("hear you through your microphone after you click or press a key") and webrtc ("connect straight to other players, who can then see your IP address").
  • The microphone only starts right after a click or key press, the browser asks the player first, and the Microphone on label shows the whole time, with "(muted)" while muted.
  • A direct connection shows each player's IP address to the other player, and the offer and answer show it to whoever passes them along. If you set relayOnly = true with a TURN server, the audio goes through the TURN server and only its owner sees the addresses.
  • Audio between players is always encrypted. WebRTC does not allow anything else.

WebAssembly mods

WebAssembly mods use the same functions through the SDK, for example zeus::call("rtc.connect", (zeus::map! { "mic" => true },)) in Rust. Samples arrive as bytes.