mediadevices

package module
v0.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 13, 2024 License: MIT Imports: 20 Imported by: 0

README


Pion MediaDevices

Go implementation of the MediaDevices API

Slack Widget GitHub Workflow Status Go Reference Coverage Status


mediadevices provides access to media input devices like cameras, microphones, and screen capture. It can also be used to encode your video/audio stream to various codec selections. mediadevices abstracts away the complexities of interacting with things like hardware and codecs allowing you to focus on building appilcations, interacting only with an amazingly simple, easy, and elegant API!

Install

go get -u github.com/ABConnectIO/mediadevices

Usage

The following snippet shows how to capture a camera stream and store a frame as a jpeg image:

package main

import (
	"image/jpeg"
	"os"

	"github.com/ABConnectIO/mediadevices"
	"github.com/ABConnectIO/mediadevices/pkg/prop"

	// This is required to register camera adapter
	_ "github.com/ABConnectIO/mediadevices/pkg/driver/camera" 
	// Note: If you don't have a camera or your adapters are not supported,
	//       you can always swap your adapters with our dummy adapters below.
	// _ "github.com/ABConnectIO/mediadevices/pkg/driver/videotest"
)

func main() {
	stream, _ := mediadevices.GetUserMedia(mediadevices.MediaStreamConstraints{
		Video: func(constraint *mediadevices.MediaTrackConstraints) {
			// Query for ideal resolutions
			constraint.Width = prop.Int(600)
			constraint.Height = prop.Int(400)
		},
	})

	// Since track can represent audio as well, we need to cast it to 
	// *mediadevices.VideoTrack to get video specific functionalities
	track := stream.GetVideoTracks()[0]
	videoTrack := track.(*mediadevices.VideoTrack)
	defer videoTrack.Close()

	// Create a new video reader to get the decoded frames. Release is used 
	// to return the buffer to hold frame back to the source so that the buffer 
	// can be reused for the next frames.
	videoReader := videoTrack.NewReader(false)
	frame, release, _ := videoReader.Read()
	defer release()

	// Since frame is the standard image.Image, it's compatible with Go standard 
	// library. For example, capturing the first frame and store it as a jpeg image.
	output, _ := os.Create("frame.jpg")
	jpeg.Encode(output, frame, nil)
}

More Examples

  • Webrtc - Use Webrtc to create a realtime peer-to-peer video call
  • Face Detection - Use a machine learning algorithm to detect faces in a camera stream
  • RTP Stream - Capture camera stream, encode it in H264/VP8/VP9, and send it to a RTP server
  • HTTP Broadcast - Broadcast camera stream through HTTP with MJPEG
  • Archive - Archive H264 encoded video stream from a camera

Available Media Inputs

Input Linux Mac Windows
Camera ✔️ ✔️ ✔️
Microphone ✔️ ✔️ ✔️
Screen ✔️ ✔️ ✔️

By default, there's no media input registered. This decision was made to allow you to play only what you need. Therefore, you need to import the associated packages for the media inputs. For example, if you want to use a camera, you need to import the camera package as a side effect:

import (
	...
	_ "github.com/ABConnectIO/mediadevices/pkg/driver/camera"
)

Available Codecs

In order to encode your video/audio, mediadevices needs to know what codecs that you want to use and their parameters. To do this, you need to import the associated packages for the codecs, and add them to the codec selector that you'll pass to GetUserMedia:

package main

import (
	"github.com/ABConnectIO/mediadevices"
	"github.com/ABConnectIO/mediadevices/pkg/codec/x264"      // This is required to use H264 video encoder
	_ "github.com/ABConnectIO/mediadevices/pkg/driver/camera" // This is required to register camera adapter
)

func main() {
	// configure codec specific parameters
	x264Params, _ := x264.NewParams()
	x264Params.Preset = x264.PresetMedium
	x264Params.BitRate = 1_000_000 // 1mbps

	codecSelector := mediadevices.NewCodecSelector(
		mediadevices.WithVideoEncoders(&x264Params),
	)
	
	mediaStream, _ := mediadevices.GetUserMedia(mediadevices.MediaStreamConstraints{
		Video: func(c *mediadevices.MediaTrackConstraints) {},
		Codec: codecSelector, // let GetUsermedia know available codecs
	})
}

Since mediadevices doesn't implement the video/audio codecs, it needs to call the codec libraries from the system through cgo. Therefore, you're required to install the codec libraries before you can use them in mediadevices. In the next section, it shows a list of available codecs, where the packages are defined (documentation linked), and installation instructions.

Note: we do not provide recommendations on choosing one codec or another as it is very complex and can be subjective.

Video Codecs

x264

A free software library and application for encoding video streams into the H.264/MPEG-4 AVC compression format.

mmal

A framework to enable H264 hardware encoding for Raspberry Pi or boards that use VideoCore GPUs.

openh264

A codec library which supports H.264 encoding and decoding. It is suitable for use in real time applications.

vpx

A free software video codec library from Google and the Alliance for Open Media that implements VP8/VP9 video coding formats.

vaapi

An open source API that allows applications such as VLC media player or GStreamer to use hardware video acceleration capabilities (currently support VP8/VP9).

Audio Codecs

opus

A totally open, royalty-free, highly versatile audio codec.

Benchmark

Result as of Nov 4, 2020 with Go 1.14 on a Raspberry pi 3, mediadevices can produce video, encode, send across network, and decode at 720p, 30 fps with < 500 ms latency.

The test was taken by capturing a camera stream, decoding the raw frames, encoding the video stream with mmal, and sending the stream through Webrtc.

FAQ

Failed to find the best driver that fits the constraints

mediadevices provides an automated driver discovery through GetUserMedia and GetDisplayMedia. The driver discover algorithm works something like:

  1. Open all registered drivers
  2. Get all properties (property describes what a driver is capable of, e.g. resolution, frame rate, etc.) from opened drivers
  3. Find the best property that meets the criteria

So, when mediadevices returns failed to find the best driver that fits the constraints error, one of the following conditions might have occured:

  • Driver was not imported as a side effect in your program, e.g. import _ github.com/ABConnectIO/mediadevices/pkg/driver/camera
  • Your constraint is too strict that there's no driver can fullfil your requirements. In this case, you can try to turn up the debug level by specifying the following environment variable: export PION_LOG_DEBUG=all to see what was too strict and tune that.
  • Your driver is not supported/implemented. In this case, you can either let us know (file an issue) and wait for the maintainers to implement it. Or, you can implement it yourself and register it through RegisterDriverAdapter
  • If trying to use import _ github.com/ABConnectIO/mediadevices/pkg/driver/screen note that you will need to use GetDisplayMedia instead of GetUserMedia

Failed to find vpx/x264/mmal/opus codecs

Since mediadevices uses cgo to access video/audio codecs, it needs to find these libraries from the system. To accomplish this, pkg-config is used for library discovery.

If you see the following error message at compile time:

# pkg-config --cflags  -- vpx
Package vpx was not found in the pkg-config search path.
Perhaps you should add the directory containing `vpx.pc'
to the PKG_CONFIG_PATH environment variable
No package 'vpx' found
pkg-config: exit status 1

There are 2 common problems:

  • The required codec library is not installed (vpx in this example). In this case, please refer to the available codecs.
  • Pkg-config fails to find the .pc files for this codec (reference). In this case, you need to find where the codec library's .pc is stored, and let pkg-config knows with: export PKG_CONFIG_PATH=/path/to/directory.

Roadmap

The library can be used with our WebRTC implementation. Please refer to that roadmap to track our major milestones.

Community

Pion has an active community on the Slack.

Follow the Pion Twitter for project updates and important WebRTC news.

We are always looking to support your projects. Please reach out if you have something to build! If you need commercial support or don't want to use public methods you can contact us at team@pion.ly

Contributing

Check out the contributing wiki to join the group of amazing people making this project possible

License

MIT License - see LICENSE for full text

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RegisterDriverAdapter 

func RegisterDriverAdapter(a driver.Adapter, info driver.Info) error

RegisterDriverAdapter allows user space level of driver registration

Types

type AudioSource 

type AudioSource interface {
	audio.Reader
	Source
}

AudioSource is a specific type of media source that emits a series of audio chunks

type AudioTrack 

type AudioTrack struct {
	*audio.Broadcaster
	// contains filtered or unexported fields
}

AudioTrack is a specific track type that contains audio source which allows multiple readers to access, and manipulate.

func (*AudioTrack) Bind 

func (track *AudioTrack) Bind(ctx webrtc.TrackLocalContext) (webrtc.RTPCodecParameters, error)

func (AudioTrack) Kind 

func (track AudioTrack) Kind() webrtc.RTPCodecType

Kind returns track's kind

func (*AudioTrack) NewEncodedIOReader 

func (track *AudioTrack) NewEncodedIOReader(codecName string) (io.ReadCloser, error)

func (*AudioTrack) NewEncodedReader 

func (track *AudioTrack) NewEncodedReader(codecName string) (EncodedReadCloser, error)

func (*AudioTrack) NewRTPReader 

func (track *AudioTrack) NewRTPReader(codecName string, ssrc uint32, mtu int) (RTPReadCloser, error)

func (AudioTrack) OnEnded 

func (track AudioTrack) OnEnded(handler func(error))

OnEnded sets an error handler. When a track has been created and started, if an error occurs, handler will get called with the error given to the parameter.

func (AudioTrack) RID 

func (track AudioTrack) RID() string

RID is only relevant if you wish to use Simulcast

func (AudioTrack) StreamID 

func (track AudioTrack) StreamID() string

func (*AudioTrack) Transform 

func (track *AudioTrack) Transform(fns ...audio.TransformFunc)

Transform transforms the underlying source by applying the given fns in serial order

func (*AudioTrack) Unbind 

func (track *AudioTrack) Unbind(ctx webrtc.TrackLocalContext) error

type CodecSelector 

type CodecSelector struct {
	// contains filtered or unexported fields
}

CodecSelector is a container of video and audio encoder builders, which later will be used for codec matching.

func NewCodecSelector 

func NewCodecSelector(opts ...CodecSelectorOption) *CodecSelector

NewCodecSelector constructs CodecSelector with given variadic options

func (*CodecSelector) Populate 

func (selector *CodecSelector) Populate(setting *webrtc.MediaEngine)

Populate lets the webrtc engine be aware of supported codecs that are contained in CodecSelector

type CodecSelectorOption 

type CodecSelectorOption func(*CodecSelector)

CodecSelectorOption is a type for specifying CodecSelector options

func WithAudioEncoders 

func WithAudioEncoders(encoders ...codec.AudioEncoderBuilder) CodecSelectorOption

WithVideoEncoders replace current audio codecs with listed encoders

func WithVideoEncoders 

func WithVideoEncoders(encoders ...codec.VideoEncoderBuilder) CodecSelectorOption

WithVideoEncoders replace current video codecs with listed encoders

type EncodedBuffer 

type EncodedBuffer struct {
	Data    []byte
	Samples uint32
}

type EncodedReadCloser 

type EncodedReadCloser interface {
	Read() (EncodedBuffer, func(), error)
	Close() error
	codec.Controllable
}

type MediaDeviceInfo 

type MediaDeviceInfo struct {
	DeviceID   string
	Kind       MediaDeviceType
	Label      string
	DeviceType driver.DeviceType
	Name       string
}

MediaDeviceInfo represents https://w3c.github.io/mediacapture-main/#dom-mediadeviceinfo

func EnumerateDevices 

func EnumerateDevices() []MediaDeviceInfo

type MediaDeviceType 

type MediaDeviceType int

MediaDeviceType enumerates type of media device. 

const (
	VideoInput MediaDeviceType = iota + 1
	AudioInput
	AudioOutput
)

MediaDeviceType definitions.

type MediaOption 

type MediaOption func(*MediaTrackConstraints)

type MediaStream 

type MediaStream interface {
	// GetAudioTracks implements https://w3c.github.io/mediacapture-main/#dom-mediastream-getaudiotracks
	GetAudioTracks() []Track
	// GetVideoTracks implements https://w3c.github.io/mediacapture-main/#dom-mediastream-getvideotracks
	GetVideoTracks() []Track
	// GetTracks implements https://w3c.github.io/mediacapture-main/#dom-mediastream-gettracks
	GetTracks() []Track
	// AddTrack implements https://w3c.github.io/mediacapture-main/#dom-mediastream-addtrack
	AddTrack(t Track)
	// RemoveTrack implements https://w3c.github.io/mediacapture-main/#dom-mediastream-removetrack
	RemoveTrack(t Track)
}

MediaStream is an interface that represents a collection of existing tracks.

func GetDisplayMedia 

func GetDisplayMedia(constraints MediaStreamConstraints) (MediaStream, error)

GetDisplayMedia prompts the user to select and grant permission to capture the contents of a display or portion thereof (such as a window) as a MediaStream. Reference: https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia

func GetUserMedia 

func GetUserMedia(constraints MediaStreamConstraints) (MediaStream, error)

GetUserMedia prompts the user for permission to use a media input which produces a MediaStream with tracks containing the requested types of media. Reference: https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia

func NewMediaStream 

func NewMediaStream(tracks ...Track) (MediaStream, error)

NewMediaStream creates a MediaStream interface that's defined in https://w3c.github.io/mediacapture-main/#dom-mediastream

type MediaStreamConstraints 

type MediaStreamConstraints struct {
	Audio MediaOption
	Video MediaOption
	Codec *CodecSelector
}

type MediaTrackConstraints 

type MediaTrackConstraints struct {
	prop.MediaConstraints
	// contains filtered or unexported fields
}

MediaTrackConstraints represents https://w3c.github.io/mediacapture-main/#dom-mediatrackconstraints

type RTPReadCloser 

type RTPReadCloser interface {
	Read() (pkts []*rtp.Packet, release func(), err error)
	Close() error
	codec.Controllable
}

type Source 

type Source interface {
	ID() string
	Close() error
}

Source is a generic representation of a media source

type Track 

type Track interface {
	Source
	// OnEnded registers a handler to receive an error from the media stream track.
	// If the error is already occured before registering, the handler will be
	// immediately called.
	OnEnded(func(error))
	Kind() webrtc.RTPCodecType
	// StreamID is the group this track belongs too. This must be unique
	StreamID() string
	// RID is the RTP Stearm ID for this track. This is only used for Simulcast
	RID() string
	// Bind binds the current track source to the given peer connection. In Pion/webrtc v3, the bind
	// call will happen automatically after the SDP negotiation. Users won't need to call this manually.
	Bind(webrtc.TrackLocalContext) (webrtc.RTPCodecParameters, error)
	// Unbind is the clean up operation that should be called after Bind. Similar to Bind, unbind will
	// be called automatically in Pion/webrtc v3.
	Unbind(webrtc.TrackLocalContext) error
	// NewRTPReader creates a new reader from the source. The reader will encode the source, and packetize
	// the encoded data in RTP format with given mtu size.
	//
	// Note: `mtu int` will be changed to `mtu uint16` in a future update.
	NewRTPReader(codecName string, ssrc uint32, mtu int) (RTPReadCloser, error)
	// NewEncodedReader creates a EncodedReadCloser that reads the encoded data in codecName format
	NewEncodedReader(codecName string) (EncodedReadCloser, error)
	// NewEncodedReader creates a new Go standard io.ReadCloser that reads the encoded data in codecName format
	NewEncodedIOReader(codecName string) (io.ReadCloser, error)
}

Track is an interface that represent MediaStreamTrack Reference: https://w3c.github.io/mediacapture-main/#mediastreamtrack

func NewAudioTrack 

func NewAudioTrack(source AudioSource, selector *CodecSelector) Track

NewAudioTrack constructs a new AudioTrack

func NewVideoTrack 

func NewVideoTrack(source VideoSource, selector *CodecSelector) Track

NewVideoTrack constructs a new VideoTrack

type VideoSource 

type VideoSource interface {
	video.Reader
	Source
}

VideoSource is a specific type of media source that emits a series of video frames

type VideoTrack 

type VideoTrack struct {
	*video.Broadcaster
	// contains filtered or unexported fields
}

VideoTrack is a specific track type that contains video source which allows multiple readers to access, and manipulate.

func (*VideoTrack) Bind 

func (track *VideoTrack) Bind(ctx webrtc.TrackLocalContext) (webrtc.RTPCodecParameters, error)

func (VideoTrack) Kind 

func (track VideoTrack) Kind() webrtc.RTPCodecType

Kind returns track's kind

func (*VideoTrack) NewEncodedIOReader 

func (track *VideoTrack) NewEncodedIOReader(codecName string) (io.ReadCloser, error)

func (*VideoTrack) NewEncodedReader 

func (track *VideoTrack) NewEncodedReader(codecName string) (EncodedReadCloser, error)

func (*VideoTrack) NewRTPReader 

func (track *VideoTrack) NewRTPReader(codecName string, ssrc uint32, mtu int) (RTPReadCloser, error)

func (VideoTrack) OnEnded 

func (track VideoTrack) OnEnded(handler func(error))

OnEnded sets an error handler. When a track has been created and started, if an error occurs, handler will get called with the error given to the parameter.

func (VideoTrack) RID 

func (track VideoTrack) RID() string

RID is only relevant if you wish to use Simulcast

func (*VideoTrack) SetShouldCopyFrames 

func (track *VideoTrack) SetShouldCopyFrames(shouldCopyFrames bool)

SetShouldCopyFrames enables frame copy for this track, sending each reader a different read buffer instead of sharing one.

func (*VideoTrack) ShouldCopyFrames 

func (track *VideoTrack) ShouldCopyFrames() bool

ShouldCopyFrames indicates if readers on this track should receive a clopy of the read buffer instead of sharing one.

func (VideoTrack) StreamID 

func (track VideoTrack) StreamID() string

func (*VideoTrack) Transform 

func (track *VideoTrack) Transform(fns ...video.TransformFunc)

Transform transforms the underlying source by applying the given fns in serial order

func (*VideoTrack) Unbind 

func (track *VideoTrack) Unbind(ctx webrtc.TrackLocalContext) error

Source Files

View all Source files

Directories

Path Synopsis
internal
codec
logging
pkg
avfoundation
Package avfoundation provides AVFoundation binding for Go
 Package avfoundation provides AVFoundation binding for Go
codec
codec/internal/codectest
Package codectest provides shared test for codec implementations.
 Package codectest provides shared test for codec implementations.
codec/mmal
Package mmal implements a hardware accelerated H264 encoder for raspberry pi.
 Package mmal implements a hardware accelerated H264 encoder for raspberry pi.
codec/openh264
codec/opus
codec/vaapi
Package vaapi implements hardware accelerated codecs.
 Package vaapi implements hardware accelerated codecs.
codec/vpx
Package vpx implements VP8 and VP9 encoder.
 Package vpx implements VP8 and VP9 encoder.
codec/x264
Package x264 implements H264 encoder.
 Package x264 implements H264 encoder.
driver
driver/audiotest
Package audiotest provides dummy audio driver for testing.
 Package audiotest provides dummy audio driver for testing.
driver/availability
driver/camera
Package camera provides a video camera driver.
 Package camera provides a video camera driver.
driver/microphone
driver/screen
driver/videotest
Package videotest provides dummy video driver for testing.
 Package videotest provides dummy video driver for testing.
driver/vncdriver
Package videotest provides vncDevice video driver for testing.
 Package videotest provides vncDevice video driver for testing.
driver/vncdriver/vnc
Package vnc implements a VNC client.
 Package vnc implements a VNC client.
frame
io
io/audio
io/video
prop
wave
Package wave implements a basic audio data library.
 Package wave implements a basic audio data library.
wave/mixer

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.