media

README

go-media

This module provides an interface for media services, including:

• Bindings in golang for ffmpeg 6;
• Opening media files, devices and network sockets for reading and writing;
• Retrieving metadata and artwork from audio and video media;
• Re-multiplexing media files from one format to another;
• Fingerprinting audio files to identify music.

Current Status

This module is currently in development and subject to change. If there are any specific features you are interested in, please see below "Contributing & Distribution" below.

Requirements

In order to build the examples, you'll need the library and header files for FFmpeg 6 installed.The chromaprint library is also required for fingerprinting audio files. On Macintosh with homebrew, for example:

brew install ffmpeg@6 chromaprint make

There are some examples in the cmd folder of the main repository on how to use the package. The various make targets are:

• make all will perform tests, build all examples and the backend API;
• make test will perform tests;
• make cmd will build example command-line tools into the build folder;
• make clean will remove all build artifacts.

There are also some targets to build a docker image:

• DOCKER_REGISTRY=docker.io/user make docker will build a docker image;
• DOCKER_REGISTRY=docker.io/user make docker-push will push the docker image to the registry.

For example,

git clone git@github.com:djthorpe/go-media.git
cd go-media
DOCKER_REGISTRY=ghcr.io/mutablelogic make docker

Examples

There are a variety of types of object needed as part of media processing. All examples require a Manager to be created, which is used to enumerate all supported formats and open media files and byte streams.

• Manager is the main entry point for the package. It is used to open media files and byte streams, and enumerate supported formats, codecs, pixel formats, etc.
• Media is a hardware device, file or byte stream. It contains metadata, artwork, and streams.
• Decoder is used to demultiplex media streams. Create a decoder and enumerate the streams which you'd like to demultiplex. Provide the audio and video parameters if you want to resample or reformat the streams.
• Encoder is used to multiplex media streams. Create an encoder and send the output of the decoder to reencode the streams.

Demultiplexing

import (
  media "github.com/mutablelogic/go-media"
)

func main() {
  manager, err := media.NewManager()
  if err != nil {
    log.Fatal(err)
  }

  // Open a media file for reading. The format of the file is guessed.
  // Alteratively, you can pass a format as the second argument. Further optional
  // arguments can be used to set the format options.
  file, err := manager.Open(os.Args[1], nil)
  if err != nil {
    log.Fatal(err)
  }
  defer file.Close()

  // Choose which streams to demultiplex - pass the stream parameters
  // to the decoder. If you don't want to resample or reformat the streams,
  // then you can pass nil as the function and all streams will be demultiplexed.
  decoder, err := file.Decoder(func (stream media.Stream) (media.Parameters, error) {
    return stream.Parameters(), nil
  }
  if err != nil {
    log.Fatal(err)
  }

  // Demuliplex the stream and receive the packets. If you don't want to
  // process the packets yourself, then you can pass nil as the function
  if err := decoder.Demux(context.Background(), func(_ media.Packet) error {
    // Each packet is specific to a stream. It can be processed here
    // to receive audio or video frames, then resize or resample them,
    // for example. Alternatively, you can pass the packet to an encoder
    // to remultiplex the streams without processing them.
    return nil
  }); err != nil {
    log.Fatal(err)  
  })
}

Decoding - Video Frames

This example shows you how to decode video frames from a media file into images.

import (
  media "github.com/mutablelogic/go-media"
)

func main() {
  manager, err := media.NewManager()
  if err != nil {
    log.Fatal(err)
  }

  media, err := manager.Open("etc/test/sample.mp4", nil)
  if err != nil {
    log.Fatal(err)
  }
  defer media.Close()

  // Create a decoder for the media file. Only video streams are decoded
  decoder, err := media.Decoder(func(stream Stream) (Parameters, error) {
    if stream.Type() == VIDEO {
      // Copy video
      return stream.Parameters(), nil
    } else {
      // Ignore other stream types
      return nil, nil
    }
  })
  if err != nil {
    log.Fatal(err)
  }

  // The frame function is called for each frame in the stream
  framefn := func(frame Frame) error {
    image, err := frame.Image()
    if err != nil {
      return err
    }
    // TODO: Do something with the image here....
    return nil
  }

  // decode frames from the stream
  if err := decoder.Decode(context.Background(), framefn); err != nil {
    log.Fatal(err)
  }
}

Encoding

TODO

Multiplexing

TODO

Retrieving Metadata and Artwork from a media file

Here is an example of opening a media file and retrieving metadata and artwork.

package main

import (
  "log"
  "os"

  media "github.com/mutablelogic/go-media"
  file "github.com/mutablelogic/go-media/pkg/file"
)

func main() {
  manager, err := media.NewManager()
  if err != nil {
    log.Fatal(err)
  }

  // Open a media file for reading. The format of the file is guessed.
  // Alteratively, you can pass a format as the second argument. Further optional
  // arguments can be used to set the format options.
  reader, err := manager.Open(os.Args[1], nil)
  if err != nil {
    log.Fatal(err)
  }
  defer reader.Close()

  // Retrieve all the metadata from the file, and display it. If you pass
  // keys to the Metadata function, then only entries with those keys will be
  // returned.
  for _, metadata := range reader.Metadata() {
    log.Print(metadata.Key(), " => ", metadata.Value())
  }

  // Retrieve artwork by using the MetaArtwork key. The value is of type []byte.
  // which needs to be converted to an image. There is a utility method to
  // detect the image type.
  for _, artwork := range reader.Metadata(media.MetaArtwork) {
    mimetype, ext, err := file.MimeType(artwork.Value().([]byte))
    if err != nil {
      log.Fatal(err)
    }
    log.Print("got artwork", mimetype, ext)
  }
}

Audio Fingerprinting

You can programmatically fingerprint audio files, compare fingerprints and identify music using the following packages:

• sys/chromaprint provides the implementation of the lower-level function calls to chromaprint. The documentation is here
• pkg/chromaprint provides the higher-level API for fingerprinting and identifying music. The documentation is here.

You'll need an API key in order to use the AcoustID service. You can get a key here.

Contributing & Distribution

This module is currently in development and subject to change.

Please do file feature requests and bugs here. The license is Apache 2 so feel free to redistribute. Redistributions in either source code or binary form must reproduce the copyright notice, and please link back to this repository for more information:

go-media https://github.com/mutablelogic/go-media/ Copyright (c) 2021-2024 David Thorpe, All rights reserved.

This software links to shared libraries of FFmpeg licensed under the LGPLv2.1.

References

• https://ffmpeg.org/doxygen/6.1/index.html
• https://pkg.go.dev/github.com/mutablelogic/go-media

Documentation

Overview

This is a package for reading, writing and inspecting media files. In order to operate on media, call NewManager() and then use the manager functions to determine capabilities and manage media files and devices.

Index

• Constants
• Variables
• type AudioParameters
• type Codec
• type Decoder
• type DecoderFunc
• type DecoderMapFunc
• type Device
• type Format
• type Frame
• type FrameFunc
• type LogFunc
• type Manager
  • func NewManager(opt ...Opt) (Manager, error)
• type Media
• type MediaType
  • func (v MediaType) FlagString() string
  • func (v MediaType) Is(f MediaType) bool
  • func (v MediaType) MarshalJSON() ([]byte, error)
  • func (v MediaType) String() string
• type Metadata
• type Opt
  • func OptForce() Opt
  • func OptLog(verbose bool, fn LogFunc) Opt
• type Packet
• type Parameters
• type Stream
• type VideoParameters

Constants

const (
	MetaArtwork  = "artwork"  // Metadata key for artwork, sets the value as []byte
	MetaDuration = "duration" // Metadata key for duration, sets the value as float64 (seconds)
)

Variables

var NativeEndian binary.ByteOrder

NativeEndian is the ByteOrder of the current system.

Types

type AudioParameters

type AudioParameters interface {
	// Return the channel layout
	ChannelLayout() string

	// Return the sample format
	SampleFormat() string

	// Return the sample rate (Hz)
	Samplerate() int
}

Audio parameters for encoding or decoding audio data.

type Codec

type Codec interface {
	// Return the codec name
	Name() string

	// Return the codec description
	Description() string

	// Return the codec type (AUDIO, VIDEO, SUBTITLE, DATA, INPUT, OUTPUT)
	Type() MediaType
}

Codec represents a codec for encoding or decoding media streams.

type Decoder

type Decoder interface {
	// Demultiplex media into packets. Pass a packet to a decoder function.
	// Stop when the context is cancelled or the end of the media stream is
	// reached.
	Demux(context.Context, DecoderFunc) error

	// Decode media into frames, and resample or resize the frame.
	// Stop when the context is cancelled or the end of the media stream is
	// reached.
	Decode(context.Context, FrameFunc) error
}

Decoder represents a demuliplexer and decoder for a media stream. You can call either Demux or Decode to process the media stream, but not both.

type DecoderFunc

type DecoderFunc func(Packet) error

DecoderFunc is a function that decodes a packet. Return io.EOF if you want to stop processing the packets early.

type DecoderMapFunc

type DecoderMapFunc func(Stream) (Parameters, error)

Return parameters if a the stream should be decoded and either resampled or resized. Return nil if you want to ignore the stream, or pass identical stream parameters (stream.Parameters()) if you want to copy the stream without any changes.

type Device

type Device interface {
	// Device name, format depends on the device
	Name() string

	// Description of the device
	Description() string

	// Flags indicating the type INPUT or OUTPUT, AUDIO or VIDEO
	Type() MediaType

	// Whether this is the default device
	Default() bool
}

Device represents a device for input or output of media streams.

type Format

type Format interface {
	// Name(s) of the format
	Name() []string

	// Description of the format
	Description() string

	// Extensions associated with the format, if a stream. Each extension
	// should have a preceeding period (ie, ".mp4")
	Extensions() []string

	// MimeTypes associated with the format
	MimeTypes() []string

	// Flags indicating the type. INPUT for a demuxer or source, OUTPUT for a muxer or
	// sink, DEVICE for a device, FILE for a file. Plus AUDIO, VIDEO, DATA, SUBTITLE.
	Type() MediaType
}

Format represents a container format for input or output of media streams. Use the manager object to get a list of supported formats.

type Frame

type Frame interface {
	Parameters

	// Number of samples in the frame, for an audio frame.
	NumSamples() int

	// Return stride for each plane, for a video frame.
	Stride(int) int

	// Return a frame plane as a byte slice.
	Bytes(int) []byte

	// Return a frame plane as int16 slice
	Int16(int) []int16

	// Return the presentation timestamp for the frame or
	// a negative number if not set
	Time() time.Duration

	// Return a frame as an image, which supports the following
	// pixel formats: AV_PIX_FMT_GRAY8, AV_PIX_FMT_RGBA,
	// AV_PIX_FMT_RGB24, AV_PIX_FMT_YUV420P
	Image() (image.Image, error)
}

Frame represents a frame of audio or video data.

type FrameFunc

type FrameFunc func(Frame) error

FrameFunc is a function that processes a frame of audio or video data. Return io.EOF if you want to stop processing the frames early.

type LogFunc

type LogFunc func(string)

Logging function which is used to log messages

type Manager

type Manager interface {
	// Open a media file or device for reading, from a path or url.
	// If a format is specified, then the format will be used to open
	// the file. Close the media object when done.
	Open(string, Format, ...string) (Media, error)

	// Open a media stream for reading.  If a format is
	// specified, then the format will be used to open the file. Close the
	// media object when done. It is the responsibility of the caller to
	// also close the reader when done.
	Read(io.Reader, Format, ...string) (Media, error)

	// Create a media file or device for writing, from a path. If a format is
	// specified, then the format will be used to create the file or else
	// the format is guessed from the path. If no parameters are provided,
	// then the default parameters for the format are used.
	Create(string, Format, []Metadata, ...Parameters) (Media, error)

	// Create a media stream for writing. The format will be used to
	// determine the formar type and one or more CodecParameters used to
	// create the streams. If no parameters are provided, then the
	// default parameters for the format are used. It is the responsibility
	// of the caller to also close the writer when done.
	Write(io.Writer, Format, []Metadata, ...Parameters) (Media, error)

	// Return supported input formats which match any filter, which can be
	// a name, extension (with preceeding period) or mimetype. The MediaType
	// can be NONE (for any) or combinations of DEVICE and STREAM.
	InputFormats(MediaType, ...string) []Format

	// Return supported output formats which match any filter, which can be
	// a name, extension (with preceeding period) or mimetype. The MediaType
	// can be NONE (for any) or combinations of DEVICE and STREAM.
	OutputFormats(MediaType, ...string) []Format

	// Return supported devices for a given format.
	// Not all devices may be supported on all platforms or listed
	// if the device does not support enumeration.
	Devices(Format) []Device

	// Return all supported channel layouts
	ChannelLayouts() []Metadata

	// Return all supported sample formats
	SampleFormats() []Metadata

	// Return all supported  pixel formats
	PixelFormats() []Metadata

	// Return all supported codecs
	Codecs() []Metadata

	// Return audio parameters for encoding
	// ChannelLayout, SampleFormat, Samplerate
	AudioParameters(string, string, int) (Parameters, error)

	// Return video parameters for encoding
	// Width, Height, PixelFormat
	VideoParameters(int, int, string) (Parameters, error)

	// Return version information for the media manager as a set of
	// metadata
	Version() []Metadata

	// Log error messages with arguments
	Errorf(string, ...any)

	// Log warning messages with arguments
	Warningf(string, ...any)

	// Log info messages  with arguments
	Infof(string, ...any)
}

Manager represents a manager for media formats and devices. Create a new manager object using the NewManager function.

func NewManager

func NewManager(opt ...Opt) (Manager, error)

type Media

type Media interface {
	io.Closer

	// Return a decoding context for the media stream, and
	// map the streams to decoders. If no function is provided
	// (ie, the argument is nil) then all streams are demultiplexed.
	Decoder(DecoderMapFunc) (Decoder, error)

	// Return INPUT for a demuxer or source, OUTPUT for a muxer or
	// sink, DEVICE for a device, FILE for a file or stream.
	Type() MediaType

	// Return the metadata for the media, filtering by keys if any
	// are included. Use the "artwork" key to return only artwork.
	Metadata(...string) []Metadata
}

Media represents a media stream, which can be input or output. A new media object is created using the Manager object

type MediaType

type MediaType uint32

Media type flags

const (
	NONE     MediaType = 0
	UNKNOWN  MediaType = (1 << iota) // Usually treated as DATA
	VIDEO                            // Video stream
	AUDIO                            // Audio stream
	DATA                             // Opaque data information usually continuous
	SUBTITLE                         // Subtitle stream
	INPUT                            // Demuxer
	OUTPUT                           // Muxer
	FILE                             // File or byte stream
	DEVICE                           // Device rather than stream
	CODEC                            // Codec

	// Set minimum and maximum values
	MIN = UNKNOWN
	MAX = CODEC

	// Convenience values
	ANY = NONE
)

func (MediaType) FlagString

func (v MediaType) FlagString() string

func (MediaType) Is

func (v MediaType) Is(f MediaType) bool

func (MediaType) MarshalJSON

func (v MediaType) MarshalJSON() ([]byte, error)

func (MediaType) String

func (v MediaType) String() string

type Metadata

type Metadata interface {
	// Return the metadata key for the entry
	Key() string

	// Return the metadata value for the entry
	Value() any
}

Metadata represents a metadata entry for a media stream.

type Opt

type Opt func(*opts) error

func OptForce

func OptForce() Opt

Force resampling and resizing on decode, even if the input and output parameters are the same

func OptLog

func OptLog(verbose bool, fn LogFunc) Opt

Set a logging function

type Packet

type Packet interface{}

Packet represents a packet of demultiplexed data. Currently this is quite opaque!

type Parameters

type Parameters interface {
	AudioParameters
	VideoParameters

	// Return the media type (AUDIO, VIDEO, SUBTITLE, DATA)
	Type() MediaType

	// Return number of planes for a specific PixelFormat
	// or SampleFormat and ChannelLayout combination
	NumPlanes() int
}

Parameters represents a set of parameters for encoding

type Stream

type Stream interface {
	// Return AUDIO, VIDEO, SUBTITLE or DATA
	Type() MediaType

	// Return the stream parameters
	Parameters() Parameters
}

Stream represents a audio, video, subtitle or data stream within a media file

type VideoParameters

type VideoParameters interface {
	// Return the width of the video frame
	Width() int

	// Return the height of the video frame
	Height() int

	// Return the pixel format
	PixelFormat() string
}

Video parameters for encoding or decoding video data.