tf2vpk

package module
v0.0.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 13, 2024 License: MIT Imports: 16 Imported by: 0

README

tf2vpk

Libraries and tools for working with Titanfall 2 VPKs.

Download

Features
  • Command-line utilities.
  • Extremely memory and CPU-efficient.
  • Can run anywhere Go can run, without a C compiler (with a performance penalty).
  • Tools to optimize VPKs without recompressing from scratch and preserving all metadata.
  • Full-featured VPK file listing.
  • Go library exposing most functionality.
  • Designed for r2 VPKs, should work with r1/r5 as well.
  • Extremely flexible.
  • Deterministic output for most commands.
  • Supports unpacking VPKs with full support for load/texture flags (it can generate either an optimized flags file with directory-based inheritance, or it can have one entry for every file in the source VPK).
  • WIP: Supports repacking VPKs with full support for ignoring files and setting load/texture flags (as either a new VPK, an update of an existing one, or an addition to an existing one), optionally reusing chunks where possible.
Examples
List a VPK

Any of the following commands will show a simple list of all files in the VPK.

tf2-vpklist /path/to/Titanfall2/vpk client_mp_angel_city.bsp.pak000

tf2-vpklist /path/to/Titanfall2/vpk/client_mp_angel_city.bsp.pak000_000.vpk

tf2-vpklist /path/to/Titanfall2/vpk/englishclient_mp_angel_city.bsp.pak000_dir.vpk
Create a new unpacked VPK

The following command will create a basic .vpkflags and .vpkignore file in the provided directory so it can be repacked later.

tf2-vpkunpack /path/to/empty/folder/for/vpk
Extract a single VPK to the current directory

Any of the following commands will extract the provided vpk into the provided directory, including a pre-configured .vpkflags and .vpkignore so it can be repacked later, preserving flags.

tf2-vpkunpack /path/to/empty/folder/for/vpk /path/to/Titanfall2/vpk client_mp_angel_city.bsp.pak000

tf2-vpkunpack /path/to/empty/folder/for/vpk /path/to/Titanfall2/vpk/client_mp_angel_city.bsp.pak000_000.vpk

tf2-vpkunpack /path/to/empty/folder/for/vpk /path/to/Titanfall2/vpk/englishclient_mp_angel_city.bsp.pak000_dir.vpk
Extract models from all VPKs

The following command will extract models from all VPKs, using tar.

for vpk in /path/to/Titanfall2/vpk/english*_dir.vpk
do
    tf2-vpk2tar "$vpk"  --exclude '/*' --include '/models' | tar xvf -
done

The tf2-vpk2tar command is a better option than tf2-vpkunpack if you don't care about repacking the VPK later and want to have finer-grained control over the output or integrate it with other tools.

Convert a VPK to SquashFS (Linux)

The following command directly converts a VPK to SquashFS so it can be mounted.

tf2-vpk2tar /path/to/Titanfall2/vpk/englishclient_mp_angel_city.bsp.pak000_dir.vpk | sqfstar client_mp_angel_city.bsp.pak000.squashfs
Optimize and remove unnecessary files from VPKs

The following command optimizes VPKs by removing unused files and regenerating the blocks.

tf2-vpkoptim /path/to/Titanfall2/vpk \
    --verbose \
    --merge \
    --exclude '/depot' \
    --exclude '*.bsp_lump' \
    --output /path/to/new/vpks

Remember to copy the enable.txt file from the original vpk folder to the new one, or the game won't work.

Optimize and remove unnecessary files from VPKs for the Northstar dedicated server

The following command also removes large files only used by the client.

tf2-vpkoptim /path/to/Titanfall2/vpk \
    --verbose \
    --merge \
    --exclude '*.vtf' \
    --exclude '/depot' \
    --exclude '*.bsp_lump' \
    --output /path/to/new/vpks

Remember to copy the enable.txt file from the original vpk folder to the new one, or the game won't work.

List a VPK verbosely and verify checksums

The following command verbosely lists all metadata from the VPK in human-readable form, while also reading and verifying the contents.

tf2-vpklist -lhft /path/to/Titanfall2/vpk/englishclient_mp_angel_city.bsp.pak000_dir.vpk

000 00000000000100000000000100000001 0000000000001000 A77AB754  81.18 %    4.6 kB    5.6 kB  materials/models/domestic/luggage_01_screen_col.vtf               OK # load=[00:VISIBLE 08:CACHE 20:TEXTURE_UNK2] texture=[03:DEFAULT]
000 00000000000100000000000100000001 0000000000001000 C2B89C0A  13.00 %   11.4 kB   87.7 kB  materials/models/humans/mri/subtle_scanline.vtf                   OK # load=[00:VISIBLE 08:CACHE 20:TEXTURE_UNK2] texture=[03:DEFAULT]
000 00000000000011000000000100000001 0000000000001000 7CB126EE  62.00 %  109.7 kB  176.9 kB  materials/world/cloud_masks/mp_angel_city_mask01_col.vtf          OK # load=[00:VISIBLE 08:CACHE 18:TEXTURE_UNK0 19:TEXTURE_UNK1] texture=[03:DEFAULT]
000 00000000000000000000000100000001 0000000000000000 56AEEFAC  65.45 %    180  B    275  B  materials/tools/toolsblock_los_n_clip.vmt                         OK # load=[00:VISIBLE 08:CACHE] texture=[]
000 00000000000000000000000100000001 0000000000000000 BF7AB187  69.89 %    253  B    362  B  materials/particle/smoke/smoke_puff_01_db128_nc2_nodoftsaa.vmt    OK # load=[00:VISIBLE 08:CACHE] texture=[]
000 00000000000000000000000100000001 0000000000000000 B25FE559  65.27 %    312  B    478  B  materials/models/domestic/luggage_01_screen.vmt                   OK # load=[00:VISIBLE 08:CACHE] texture=[]
000 00000000000000000000000100000001 0000000000000000 ED873D51  69.37 %    188  B    271  B  resource/overviews/mp_angel_city.txt                              OK # load=[00:VISIBLE 08:CACHE] texture=[]
006 00000000000000000000000100000001 0000000000000000 C7153AF9  62.04 %    201  B    324  B  scripts/levels/mp_angel_city.rson                                 OK # load=[00:VISIBLE 08:CACHE] texture=[]
000 00000000000000000000000100000001 0000000000000000 1F294071  51.54 %   50.7 kB   98.3 kB  materials/correction/mp_angel_city_dlc.raw                        OK # load=[00:VISIBLE 08:CACHE] texture=[]
000 00000000000000000000000100000001 0000000000000000 3052F29E  27.23 %    8.7 kB   32.0 kB  scripts/vscripts/client/cl_carrier.gnut                           OK # load=[00:VISIBLE 08:CACHE] texture=[]
000 00000000000000000000000100000001 0000000000000000 9D8E4973  85.79 %    157  B    183  B  scripts/vscripts/client/levels/cl_mp_angel_city.nut               OK # load=[00:VISIBLE 08:CACHE] texture=[]
000 00000000000000000000000100000001 0000000000000000 8FA3B7AC  26.12 %    672  B    2.6 kB  models/water/water_puddle_decal_01.mdl                            OK # load=[00:VISIBLE 08:CACHE] texture=[]
000 00000000000000000000000100000001 0000000000000000 FB46E776  28.99 %    754  B    2.6 kB  models/signs/market/angel_city_market_sign18.mdl                  OK # load=[00:VISIBLE 08:CACHE] texture=[]
Building

Pre-built binaries for the latest commit can be found here.

Note that building with CGO_ENABLED=1 results in a significant performance impact; tf2-vpk2tar is about 4-8x slower (even with parallelism).

Build with a C compiler
CGO_ENABLED=1 go build -trimpath -v -x ./cmd/tf2-vpk2tar
CGO_ENABLED=1 go build -trimpath -v -x ./cmd/tf2-vpkoptim
CGO_ENABLED=1 go build -trimpath -v -x ./cmd/tf2-vpklist
CGO_ENABLED=1 go build -trimpath -v -x ./cmd/tf2-vpkunpack
Build without a C compiler
CGO_ENABLED=0 go build -trimpath -v -x ./cmd/tf2-vpk2tar
CGO_ENABLED=0 go build -trimpath -v -x ./cmd/tf2-vpkoptim
CGO_ENABLED=0 go build -trimpath -v -x ./cmd/tf2-vpklist
CGO_ENABLED=0 go build -trimpath -v -x ./cmd/tf2-vpkunpack
Build for Windows using MinGW
CGO_ENABLED=1 GOOS=windows GOARCH=amd64 CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ HOST=x86_64-w64-mingw32 go build -trimpath -v -x ./cmd/tf2-vpk2tar
CGO_ENABLED=1 GOOS=windows GOARCH=amd64 CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ HOST=x86_64-w64-mingw32 go build -trimpath -v -x ./cmd/tf2-vpkoptim
CGO_ENABLED=1 GOOS=windows GOARCH=amd64 CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ HOST=x86_64-w64-mingw32 go build -trimpath -v -x ./cmd/tf2-vpklist
CGO_ENABLED=1 GOOS=windows GOARCH=amd64 CC=x86_64-w64-mingw32-gcc CXX=x86_64-w64-mingw32-g++ HOST=x86_64-w64-mingw32 go build -trimpath -v -x ./cmd/tf2-vpkunpack
Build for Windows without using MinGW
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -trimpath -v -x ./cmd/tf2-vpk2tar
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -trimpath -v -x ./cmd/tf2-vpkoptim
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -trimpath -v -x ./cmd/tf2-vpklist
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -trimpath -v -x ./cmd/tf2-vpkunpack
VPK unpacking/packing

TODO: more specific documentation

Documentation

Overview

Package tf2vpk manipulates Titanfall 2 VPKs.

Index

Constants

View Source 
const (
	ValvePakMagic                    uint32 = 0x55AA1234
	ValvePakVersionMajor             uint16 = 2
	ValvePakVersionMinor             uint16 = 3
	ValvePakMaxChunkUncompressedSize uint64 = 0x100000
)

Titanfall 2 VPK constants.

View Source 
const Ext = ".vpk"

Ext is the file extension of a VPK.

Variables

This section is empty.

Functions

func DescribeLoadFlags added in v0.0.2 

func DescribeLoadFlags(flags uint32) (s []string)

DescribeLoadFlags returns a human-readable slice of strings describing the provided load flags.

func DescribeTextureFlags added in v0.0.2 

func DescribeTextureFlags(flags uint16) (s []string)

DescribeTextureFlags returns a human-readable slice of strings describing the provided texture flags.

func JoinName added in v0.0.4 

func JoinName(prefix, name string, idx ValvePakIndex) (fn string)

JoinName generates a filename for a VPK.

func NewCRC 

func NewCRC() hash.Hash32

Types

type Reader 

type Reader struct {
	Root ValvePakDir
	// contains filtered or unexported fields
}

Reader reads Titanfall 2 VPKs.

func NewReader 

func NewReader(vpk ValvePak) (*Reader, error)

NewReader creates a new Reader reading from vpk.

func (*Reader) Close 

func (r *Reader) Close() error

Close cleans files opened by the Reader.

func (*Reader) Open 

func (r *Reader) Open(name string) (fs.File, error)

Open implements fs.FS.

func (*Reader) OpenBlockRaw 

func (r *Reader) OpenBlockRaw(n ValvePakIndex) (io.ReaderAt, error)

OpenBlockRaw opens a new reader reading the contents of a specific block.

func (*Reader) OpenChunkRaw 

func (r *Reader) OpenChunkRaw(f ValvePakFile, c ValvePakChunk) (io.Reader, error)

OpenChunkRaw returns a new reader reading the contents of a specific chunk.

func (*Reader) OpenFile 

func (r *Reader) OpenFile(f ValvePakFile) (io.Reader, error)

OpenFile returns a new reader reading the contents of a specific file. The checksum is verified at EOF.

func (*Reader) OpenFileParallel added in v0.0.2 

func (r *Reader) OpenFileParallel(f ValvePakFile, n int) (io.Reader, error)

OpenFileParallel is like OpenFile, but but decompresses chunks in parallel using n goroutines going no more than n compressed chunks ahead.

type ValvePak added in v0.0.4 

type ValvePak interface {
	// Open opens a reader for the provided index. It may also implement
	// io.Closer (this should be checked by the caller).
	Open(ValvePakIndex) (io.ReaderAt, error)

	// Create opens a writer writing to the provided index, truncating it if it
	// exists. It may also implement io.Closer (this should be checked by the
	// caller).
	Create(ValvePakIndex) (io.Writer, error)

	// Delete deletes all files associated with the VPK. If no files exist, nil
	// is returned.
	Delete() error
}

ValvePak provides access to VPK files.

func VPK added in v0.0.4 

func VPK(path, prefix, name string) ValvePak

VPK returns a ValvePak reading the provided path/prefix/name. It may or may not exist.

func VPKFromPath added in v0.0.4 

func VPKFromPath(filename, prefix string) (ValvePak, error)

VPKFromPath attempts to return a ValvePak from the provided path. It may or may not exist.

type ValvePakChunk 

type ValvePakChunk struct {
	LoadFlags        uint32 // note: these flags seem to be the same for all chunks in a ValvePakFile
	TextureFlags     uint16 // ^, and these ones only seem to be on VTF files
	Offset           uint64
	CompressedSize   uint64
	UncompressedSize uint64
}

ValvePakChunk is a file chunk (possibly shared) in a Titanfall 2 VPK.

func (ValvePakChunk) CreateReader 

func (c ValvePakChunk) CreateReader(r io.ReaderAt) (io.Reader, error)

CreateReader creates a new reader for the chunk.

func (ValvePakChunk) CreateReaderRaw 

func (c ValvePakChunk) CreateReaderRaw(r io.ReaderAt) (io.Reader, error)

CreateReader creates a new reader for the raw data of the chunk.

func (*ValvePakChunk) Deserialize 

func (c *ValvePakChunk) Deserialize(r io.Reader) error

Deserialize parses a ValvePakChunk from r.

func (ValvePakChunk) IsCompressed 

func (c ValvePakChunk) IsCompressed() bool

IsCompressed checks if a chunk is compressed.

func (ValvePakChunk) Serialize 

func (c ValvePakChunk) Serialize(w io.Writer) error

Serialize writes an encoded ValvePakChunk to w.

type ValvePakDir 

type ValvePakDir struct {
	Magic        uint32
	MajorVersion uint16
	MinorVersion uint16

	DataSize uint32
	File     []ValvePakFile
	// contains filtered or unexported fields
}

ValvePakDir is the root directory of a Titanfall 2 VPK, providing byte-for-byte identical serialization/deserialization and validation (it will refuse to read or write invalid structs).

func (ValvePakDir) ChunkOffset added in v0.0.4 

func (d ValvePakDir) ChunkOffset() (n uint32, err error)

ChunkOffset returns the starting offset of chunk data stored after the dir index (i.e., add this to the ValvePakChunk.Offset when reading a chunk for a file with ValvePakFile.Index == ValvePakIndexDir).

func (*ValvePakDir) Deserialize 

func (d *ValvePakDir) Deserialize(r io.Reader) error

Deserialize parses a ValvePakDir from r.

func (ValvePakDir) Serialize 

func (d ValvePakDir) Serialize(w io.Writer) error

Serialize writes an encoded ValvePakDir to w. The output should be identical byte-for-byte.

func (*ValvePakDir) SortFiles 

func (d *ValvePakDir) SortFiles() error

SortFiles sorts the files in an order suitable for the tree.

func (ValvePakDir) TreeSize 

func (d ValvePakDir) TreeSize() (uint32, error)

type ValvePakFile 

type ValvePakFile struct {
	Path         string
	CRC32        uint32
	PreloadBytes uint16
	Index        ValvePakIndex
	Chunk        []ValvePakChunk
}

ValvePakFile is a file in a Titanfall 2 VPK.

func (*ValvePakFile) CreateReader 

func (f *ValvePakFile) CreateReader(r io.ReaderAt) (io.Reader, error)

CreateReader creates a new reader for the file, checking the CRC32 at EOF.

func (*ValvePakFile) CreateReaderParallel added in v0.0.2 

func (f *ValvePakFile) CreateReaderParallel(r io.ReaderAt, n int) (io.Reader, error)

CreateReaderParallel is like CreateReader, but decompresses chunks in parallel using n goroutines going no more than n compressed chunks ahead.

func (*ValvePakFile) Deserialize 

func (f *ValvePakFile) Deserialize(r io.Reader, path string) error

Deserialize parses a ValvePakFile from r.

func (*ValvePakFile) LoadFlags added in v0.0.2 

func (f *ValvePakFile) LoadFlags() (uint32, error)

LoadFlags gets the load flags for the file.

func (ValvePakFile) Serialize 

func (f ValvePakFile) Serialize(w io.Writer) error

Serialize writes an encoded ValvePakFile to w.

func (*ValvePakFile) TextureFlags 

func (f *ValvePakFile) TextureFlags() (uint16, error)

TextureFlags gets the texture flags for the file.

type ValvePakIndex added in v0.0.4 

type ValvePakIndex uint16

ValvePakIndex is an VPK block index. 

const (
	ValvePakIndexDir ValvePakIndex = 0x7FFF // refers to data after the index in _dir.vpk
	ValvePakIndexEOF ValvePakIndex = 0xFFFF // not actually one
)

func SplitName added in v0.0.4 

func SplitName(fn, prefix string) (name string, idx ValvePakIndex, err error)

SplitName is the inverse of JoinName.

func (ValvePakIndex) GoString added in v0.0.4 

func (i ValvePakIndex) GoString() string

func (ValvePakIndex) String added in v0.0.4 

func (i ValvePakIndex) String() string

Source Files

View all Source files

Directories

Path Synopsis
cmd
tf2-vpk2tar
Command tf2-vpk2tar streams Titanfall 2 VPKs as a tar archive.
 Command tf2-vpk2tar streams Titanfall 2 VPKs as a tar archive.
tf2-vpklist
Command tf2-vpklist shows information about a VPK.
 Command tf2-vpklist shows information about a VPK.
tf2-vpkoptim
Command tf2-vpkoptim repacks and filters Titanfall 2 VPKs.
 Command tf2-vpkoptim repacks and filters Titanfall 2 VPKs.
tf2-vpkunpack
Command tf2-vpkunpack unpacks or initializes a new vpk for modification and repacking.
 Command tf2-vpkunpack unpacks or initializes a new vpk for modification and repacking.
Package vpkutil provides useful helpers for manipulating VPK files.
 Package vpkutil provides useful helpers for manipulating VPK files.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.