ibd2schema

package module
v0.0.0-...-df0a95e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 30, 2024 License: MIT Imports: 11 Imported by: 1

README

go-ibd2schema

Parse MySQL 8.0 InnoDB .ibd files to SDI and convert SDI to table schema

Overview

go-ibd2schema is a tool inspired by ibd2sdi and sdi2ddl, but with enhanced functionality and performance. It allows you to extract table schemas directly from MySQL 8.0 InnoDB .ibd files, offering a streamlined process for database schema analysis and migration.

Why we wrote this

We developed go-ibd2schema to address a specific need in managing large MySQL databases with numerous tables. Our primary goals were:

  1. To ensure consistency in table schemas when working with Xtrabackup results.
  2. To extract schema information from physical files (specifically .ibd files for MySQL 8.x) while streaming to storage.

While existing tools like ibd2sdi can extract SDI from .ibd files, they are limited to working with files on disk. go-ibd2schema extends this functionality by:

  • Allowing SDI extraction directly from memory
  • Combining the functionality of ibd2sdi and sdi2ddl into a single step
  • Converting the extracted SDI to DDL in one seamless process

This approach significantly improves efficiency when dealing with large-scale database operations, particularly in streaming or cloud environments where direct file access might be limited.

Features

  • Read .ibd files from both file system and data streams
  • Direct parsing of .ibd files to schema without intermediate steps
  • Support for fulltext index parsing
  • Support for spatial index parsing
  • Parsing of key parser configurations (e.g., /*150100 WITH PARSER ngram */)
  • Parsing of update_option in table definitions (e.g., ON UPDATE CURRENT_TIMESTAMP)
  • Improved performance compared to ibd2sdi + sdi2ddl combination

Comparison with ibd2sdi + sdi2ddl

Feature go-ibd2schema ibd2sdi + sdi2ddl
Read .ibd from file ✅ ✅
Read .ibd from data stream ✅ ❌
Direct schema parsing ✅ ❌ (two-step process)
Fulltext index support ✅ ✅ (ibd2sdi) / ❌ (sdi2ddl)
Spatial index support ✅ ✅ (ibd2sdi) / ❌ (sdi2ddl)
Key parser support ✅ ✅ (ibd2sdi) / ❌ (sdi2ddl)
update_option support ✅ ✅ (ibd2sdi) / ❌ (sdi2ddl)
Performance (file parsing) 1.2~1.4x 1x

Usage

Just give the parser filepath and file reader of the frm file.

func main() {
 filePath := os.Args[1]
 file, err := os.Open(filePath)
 if err != nil {
  panic(err)
 }

 ts, err := ibd2schema.NewTableSpace(file, os.Stdout)
 if err != nil {
  panic(err)
 }
 // dump ddl: only support file per table = On
 err = ts.DumpSchemas()
 if err != nil {
  fmt.Printf("%+v\n", err)
  os.Exit(-1)
 }
 for db, table := range ts.TableSchemas {
  fmt.Printf("Database: %s\n", db)
  fmt.Printf("Table DDL: %s\n", table.DDL)
 }
 // dump sdi
 err = ts.DumpSDIs()
 if err != nil {
  fmt.Printf("%+v\n", err)
  os.Exit(-1)
 }
 fmt.Println(string(pretty.Pretty(ts.SDIResult)))
}

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License.

Acknowledgements

This project was inspired by ibd2sdi and sdi2ddl. We thank the creators of these tools for their work in the MySQL ecosystem.

Documentation

Index

Constants

View Source 
const (
	KiB                    = 1024
	MiB                    = 1024 * KiB
	SIZE_OF_UINT32         = 4
	ULINT_UNDEFINED uint64 = ^uint64(0)
)

size

View Source 
const (
	/** Transaction ID type size in bytes. */
	DATA_TRX_ID_LEN uint32 = 6
	/** Rollback data pointer type size in bytes. */
	DATA_ROLL_PTR_LEN uint32 = 7
)
View Source 
const (
	/** Encryption magic bytes size */
	MAGIC_SIZE = 3
	/** Encryption key length */
	KEY_LEN = 32
	/** UUID of server instance, it's needed for composing master key name */
	SERVER_UUID_LEN = 36
	/** Encryption information total size: magic number + master_key_id +
	  key + iv + server_uuid + checksum */
	INFO_SIZE = (MAGIC_SIZE + SIZE_OF_UINT32 + (KEY_LEN * 2) + SERVER_UUID_LEN +
		SIZE_OF_UINT32)
	/** Maximum size of Encryption information considering all
	  formats v1, v2 & v3. */
	INFO_MAX_SIZE = INFO_SIZE + SIZE_OF_UINT32
)
View Source 
const (
	// The number of bytes required to store the file space header
	FIL_PAGE_DATA = 38
	// address size is 6 bytes
	FIL_ADDR_SIZE = 6
	/** page offset inside space */
	FIL_PAGE_OFFSET                  = 4
	FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID = 34
	// The physical size of a list base node in bytes
	FLST_BASE_NODE_SIZE = 4 + 2*FIL_ADDR_SIZE
	FLST_NODE_SIZE      = 2 * FIL_ADDR_SIZE // The physical size of a list node in bytes
	// page type offset in page
	FIL_PAGE_TYPE = 24
	/** File page types (values of FIL_PAGE_TYPE) @{ */
	/** B-tree node */
	FIL_PAGE_INDEX = 17855

	/** R-tree node */
	FIL_PAGE_RTREE = 17854

	/** Tablespace SDI Index page */
	FIL_PAGE_SDI = 17853

	/** This page type is unused. */
	FIL_PAGE_TYPE_UNUSED = 1

	/** Undo log page */
	FIL_PAGE_UNDO_LOG = 2

	/** Index node */
	FIL_PAGE_INODE = 3

	/** Insert buffer free list */
	FIL_PAGE_IBUF_FREE_LIST = 4

	/* File page types introduced in MySQL/InnoDB 5.1.7 */
	/** Freshly allocated page */
	FIL_PAGE_TYPE_ALLOCATED = 0

	/** Insert buffer bitmap */
	FIL_PAGE_IBUF_BITMAP = 5

	/** System page */
	FIL_PAGE_TYPE_SYS = 6

	/** Transaction system data */
	FIL_PAGE_TYPE_TRX_SYS = 7

	/** File space header */
	FIL_PAGE_TYPE_FSP_HDR = 8

	/** Extent descriptor page */
	FIL_PAGE_TYPE_XDES = 9

	/** Uncompressed BLOB page */
	FIL_PAGE_TYPE_BLOB = 10

	/** First compressed BLOB page */
	FIL_PAGE_TYPE_ZBLOB = 11

	/** Subsequent compressed BLOB page */
	FIL_PAGE_TYPE_ZBLOB2 = 12

	/*
	   * In old tablespaces, garbage in FIL_PAGE_TYPE is replaced with
	   this value when flushing pages.
	*/
	FIL_PAGE_TYPE_UNKNOWN = 13

	/** Compressed page */
	FIL_PAGE_COMPRESSED = 14

	/** Encrypted page */
	FIL_PAGE_ENCRYPTED = 15

	/** Compressed and Encrypted page */
	FIL_PAGE_COMPRESSED_AND_ENCRYPTED = 16

	/** Encrypted R-tree page */
	FIL_PAGE_ENCRYPTED_RTREE = 17

	/** Uncompressed SDI BLOB page */
	FIL_PAGE_SDI_BLOB = 18

	/** Compressed SDI BLOB page */
	FIL_PAGE_SDI_ZBLOB = 19

	/** Legacy doublewrite buffer page. */
	FIL_PAGE_TYPE_LEGACY_DBLWR = 20

	/** Rollback Segment Array page */
	FIL_PAGE_TYPE_RSEG_ARRAY = 21

	/** Index pages of uncompressed LOB */
	FIL_PAGE_TYPE_LOB_INDEX = 22

	/** Data pages of uncompressed LOB */
	FIL_PAGE_TYPE_LOB_DATA = 23

	/** The first page of an uncompressed LOB */
	FIL_PAGE_TYPE_LOB_FIRST = 24

	/** The first page of a compressed LOB */
	FIL_PAGE_TYPE_ZLOB_FIRST = 25

	/** Data pages of compressed LOB */
	FIL_PAGE_TYPE_ZLOB_DATA = 26

	/*
	   * Index pages of compressed LOB. This page contains an array of
	   z_index_entry_t objects.
	*/
	FIL_PAGE_TYPE_ZLOB_INDEX = 27

	/** Fragment pages of compressed LOB. */
	FIL_PAGE_TYPE_ZLOB_FRAG = 28

	/** Index pages of fragment pages (compressed LOB). */
	FIL_PAGE_TYPE_ZLOB_FRAG_ENTRY = 29

	/** Note the highest valid non-index page_type_t. */
	FIL_PAGE_TYPE_LAST = FIL_PAGE_TYPE_ZLOB_FRAG_ENTRY

	FIL_PAGE_DATA_END = 8

	/** if there is a 'natural' successor of the page, its offset. Otherwise
	FIL_NULL. B-tree index pages(FIL_PAGE_TYPE contains FIL_PAGE_INDEX) on the
	same PAGE_LEVEL are maintained as a doubly linked list via FIL_PAGE_PREV and
	FIL_PAGE_NEXT in the collation order of the smallest user record on each
	page. */
	FIL_PAGE_NEXT = 12

	FIL_NULL = math.MaxUint32
)

FIL file

View Source 
const (
	// Offset of the space header within a file page
	FSP_HEADER_OFFSET = FIL_PAGE_DATA
	// The number of bytes required to store SDI root page number(4) and SDI version(4) at Page 0
	FSP_SDI_HEADER_LEN = 8
	// fsp_space_t.flags, similar to dict_table_t::flags
	FSP_SPACE_FLAGS = 16
	// Number of flag bits used to indicate the tablespace page size
	FSP_FLAGS_WIDTH_PAGE_SSIZE uint32 = 4
	/** Zero relative shift position of the POST_ANTELOPE field */
	FSP_FLAGS_POS_POST_ANTELOPE uint32 = 0
	/** Width of the POST_ANTELOPE flag */
	FSP_FLAGS_WIDTH_POST_ANTELOPE uint32 = 1
	/** Number of flag bits used to indicate the tablespace zip page size */
	FSP_FLAGS_WIDTH_ZIP_SSIZE uint32 = 4
	/** Zero relative shift position of the ZIP_SSIZE field */
	FSP_FLAGS_POS_ZIP_SSIZE uint32 = FSP_FLAGS_POS_POST_ANTELOPE + FSP_FLAGS_WIDTH_POST_ANTELOPE
	// Zero relative shift position of the ATOMIC_BLOBS field
	FSP_FLAGS_POS_ATOMIC_BLOBS uint32 = FSP_FLAGS_POS_ZIP_SSIZE + FSP_FLAGS_WIDTH_ZIP_SSIZE
	/** Width of the SDI flag.  This flag indicates the presence of
	tablespace dictionary.*/
	FSP_FLAGS_WIDTH_SDI uint32 = 1
	/** Width of the encryption flag.  This flag indicates that the tablespace
	is a tablespace with encryption. */
	FSP_FLAGS_WIDTH_ENCRYPTION = 1
	/** Zero relative shift position of the start of the DATA_DIR bit */
	FSP_FLAGS_POS_DATA_DIR = FSP_FLAGS_POS_PAGE_SSIZE + FSP_FLAGS_WIDTH_PAGE_SSIZE
	/** Width of the DATA_DIR flag.  This flag indicates that the tablespace
	is found in a remote location, not the default data directory. */
	FSP_FLAGS_WIDTH_DATA_DIR = 1
	/** Zero relative shift position of the start of the SHARED bit */
	FSP_FLAGS_POS_SHARED = FSP_FLAGS_POS_DATA_DIR + FSP_FLAGS_WIDTH_DATA_DIR
	/** Width of the TEMPORARY flag.  This flag indicates that the tablespace
	is a temporary tablespace and everything in it is temporary, meaning that
	it is for a single client and should be deleted upon startup if it exists. */
	FSP_FLAGS_WIDTH_TEMPORARY = 1
	/** Width of the SHARED flag.  This flag indicates that the tablespace
	was created with CREATE TABLESPACE and can be shared by multiple tables. */
	FSP_FLAGS_WIDTH_SHARED = 1
	/** Zero relative shift position of the start of the TEMPORARY bit */
	FSP_FLAGS_POS_TEMPORARY = FSP_FLAGS_POS_SHARED + FSP_FLAGS_WIDTH_SHARED
	/** Zero relative shift position of the start of the ENCRYPTION bit */
	FSP_FLAGS_POS_ENCRYPTION = FSP_FLAGS_POS_TEMPORARY + FSP_FLAGS_WIDTH_TEMPORARY
	/** Zero relative shift position of the start of the SDI bits */
	FSP_FLAGS_POS_SDI = FSP_FLAGS_POS_ENCRYPTION + FSP_FLAGS_WIDTH_ENCRYPTION
	/** Width of the ATOMIC_BLOBS flag.  The ability to break up a long
	column into an in-record prefix and an externally stored part is available
	to ROW_FORMAT=REDUNDANT and ROW_FORMAT=COMPACT. */
	FSP_FLAGS_WIDTH_ATOMIC_BLOBS uint32 = 1
	// Zero relative shift position of the PAGE_SSIZE field
	FSP_FLAGS_POS_PAGE_SSIZE uint32 = FSP_FLAGS_POS_ATOMIC_BLOBS + FSP_FLAGS_WIDTH_ATOMIC_BLOBS
	// Bit mask of the PAGE_SSIZE field
	FSP_FLAGS_MASK_PAGE_SSIZE uint32 = (1<<FSP_FLAGS_WIDTH_PAGE_SSIZE - 1) << FSP_FLAGS_POS_PAGE_SSIZE
	/** Bit mask of the ZIP_SSIZE field */
	FSP_FLAGS_MASK_ZIP_SSIZE uint32 = (1<<FSP_FLAGS_WIDTH_ZIP_SSIZE - 1) << FSP_FLAGS_POS_ZIP_SSIZE
	/** Bit mask of the SDI field */
	FSP_FLAGS_MASK_SDI uint32 = (1<<FSP_FLAGS_WIDTH_SDI - 1) << FSP_FLAGS_POS_SDI
	/* File space header size */
	FSP_HEADER_SIZE = 32 + 5*FLST_BASE_NODE_SIZE

	/** On a page of any file segment, data
	may be put starting from this offset */
	FSEG_PAGE_DATA = FIL_PAGE_DATA
	/** Length of the file system  header, in bytes */
	FSEG_HEADER_SIZE = 10
)

FSP file space

View Source 
const (
	/** 8 bytes containing the length of the externally stored part of the LOB.
	  The 2 highest bits are reserved to the flags below. */
	BTR_EXTERN_LEN = 12
	/** page number where stored */
	BTR_EXTERN_PAGE_NO = 4
	LOB_HDR_PART_LEN   = 0
	LOB_HDR_SIZE       = 10
	LOB_PAGE_DATA      = FIL_PAGE_DATA + LOB_HDR_SIZE
)
View Source 
const (
	/** index page header starts at this offset */
	PAGE_HEADER = FSEG_PAGE_DATA
	/** start of data on the page */
	PAGE_DATA = PAGE_HEADER + 36 + 2*FSEG_HEADER_SIZE
	/** number of records in the heap, bit 15=flag: new-style compact page format */
	PAGE_N_HEAP = 4
	/** level of the node in an index tree; the leaf level is the level 0.
	This field should not be written to after page creation. */
	PAGE_LEVEL = 26
	/** index id where the page belongs. This field should not be written to after
	page creation. */
	PAGE_INDEX_ID = 28
	/** number of slots in page directory */
	PAGE_N_DIR_SLOTS = 0
	/** pointer to record heap top */
	PAGE_HEAP_TOP = 2
	/* Offset of the directory start down from the page end. We call the
	slot with the highest file address directory start, as it points to
	the first record in the list of records. */
	PAGE_DIR = FIL_PAGE_DATA_END
	/* We define a slot in the page directory as two bytes */
	PAGE_DIR_SLOT_SIZE = 2
	/** number of user records on the page */
	PAGE_N_RECS = 16
	/** First user record in creation (insertion) order, not necessarily collation
	  order; this record may have been deleted */
	PAGE_HEAP_NO_USER_LOW = 2
	/** Size of an compressed page directory entry */
	PAGE_ZIP_DIR_SLOT_SIZE = 2
	/** Mask of record offsets */
	PAGE_ZIP_DIR_SLOT_MASK uint32 = 0x3fff
	/** 'owned' flag */
	PAGE_ZIP_DIR_SLOT_OWNED = 0x4000
	/** offset of the page infimum record on a new-style compact page */
	PAGE_NEW_INFIMUM = (PAGE_DATA + REC_N_NEW_EXTRA_BYTES)
	/** offset of the page infimum record on an
	  old-style page */
	PAGE_OLD_INFIMUM = PAGE_DATA + 1 + REC_N_OLD_EXTRA_BYTES
	/** offset of the page supremum record on a new-style compact page */
	PAGE_NEW_SUPREMUM = PAGE_DATA + 2*REC_N_NEW_EXTRA_BYTES + 8
	/** offset of the page supremum record end on a new-style compact page */
	PAGE_NEW_SUPREMUM_END = PAGE_NEW_SUPREMUM + 8
	/** offset of the page supremum record on an
	old-style page */
	PAGE_OLD_SUPREMUM = PAGE_DATA + 2 + 2*REC_N_OLD_EXTRA_BYTES + 8
	/** Start offset of the area that will be compressed */
	PAGE_ZIP_START = PAGE_NEW_SUPREMUM_END
	/* The offset of the physically lower end of the directory, counted from
	page end, when the page is empty */
	PAGE_EMPTY_DIR_START = PAGE_DIR + 2*PAGE_DIR_SLOT_SIZE
)
View Source 
const (

	/* Number of extra bytes in a new-style record,
	in addition to the data and the offsets */
	REC_N_NEW_EXTRA_BYTES uint32 = 5
	// next record mask
	REC_NEXT       uint32 = 2
	REC_NEXT_MASK  uint32 = 0xFFFF
	REC_NEXT_SHIFT uint32 = 0
	/** Stored at rec origin minus 3rd byte. Only 3bits of 3rd byte are used for rec type. */
	REC_OFF_TYPE uint32 = 3
	/** Stored at rec_origin minus 2nd byte and length 2 bytes. */
	REC_OFF_NEXT = 2
	/** Length of TYPE field in record of SDI Index. */
	REC_DATA_TYPE_LEN = 4
	/** Length of ID field in record of SDI Index. */
	REC_DATA_ID_LEN = 8
	/* This is single byte bit-field */
	REC_OLD_INFO_BITS uint32 = 6 /* This is single byte bit-field */
	REC_NEW_INFO_BITS uint32 = 5 /* This is single byte bit-field */
	/** The deleted flag in info bits; when bit is set to 1, it means the record has
	been delete marked */
	REC_INFO_DELETED_FLAG uint32 = 0x20
	REC_INFO_BITS_SHIFT   uint32 = 0
	/* Number of extra bytes in an old-style record,
	in addition to the data and the offsets */
	REC_N_OLD_EXTRA_BYTES = 6
	/** SDI Index record Origin. */
	REC_ORIGIN uint32 = 0
	/** Offset of TYPE field in record (0). */
	REC_OFF_DATA_TYPE uint32 = REC_ORIGIN
	/** Offset of ID field in record (4). */
	REC_OFF_DATA_ID uint32 = REC_OFF_DATA_TYPE + REC_DATA_TYPE_LEN
	/** Offset of 6-byte trx id (12). */
	REC_OFF_DATA_TRX_ID uint32 = REC_OFF_DATA_ID + REC_DATA_ID_LEN
	/** 7-byte roll-ptr (18). */
	REC_OFF_DATA_ROLL_PTR uint32 = REC_OFF_DATA_TRX_ID + DATA_TRX_ID_LEN
	/** 4-byte un-compressed len (25) */
	REC_OFF_DATA_UNCOMP_LEN uint32 = REC_OFF_DATA_ROLL_PTR + DATA_ROLL_PTR_LEN
	/** Length of UNCOMPRESSED_LEN field in record of SDI Index. */
	REC_DATA_UNCOMP_LEN uint32 = 4
	/** 4-byte compressed len (29) */
	REC_OFF_DATA_COMP_LEN uint32 = REC_OFF_DATA_UNCOMP_LEN + REC_DATA_UNCOMP_LEN
	/** Length of SDI Index record header. */
	REC_MIN_HEADER_SIZE uint32 = REC_N_NEW_EXTRA_BYTES
	/** Length of COMPRESSED_LEN field in record of SDI Index. */
	REC_DATA_COMP_LEN uint32 = 4
	/** Variable length Data (33). */
	REC_OFF_DATA_VARCHAR uint32 = REC_OFF_DATA_COMP_LEN + REC_DATA_COMP_LEN
)
View Source 
const (
	/* Record status values */
	REC_STATUS_ORDINARY = iota
	REC_STATUS_NODE_PTR
	REC_STATUS_INFIMUM
	REC_STATUS_SUPREMUM
)
View Source 
const (
	/* Define the Min, Max, Default page sizes. */
	/** Minimum Page Size Shift (power of 2) */
	UNIV_PAGE_SIZE_SHIFT_MIN = 12
	/** Maximum Page Size Shift (power of 2) */
	UNIV_PAGE_SIZE_SHIFT_MAX = 16
	/** Original 16k InnoDB Page Size Shift, in case the default changes */
	UNIV_PAGE_SIZE_SHIFT_ORIG = 14
	/** Minimum page size InnoDB currently supports. */
	UNIV_PAGE_SIZE_MIN = 1 << UNIV_PAGE_SIZE_SHIFT_MIN
	/** Maximum page size InnoDB currently supports. */
	UNIV_PAGE_SIZE_MAX = 1 << UNIV_PAGE_SIZE_SHIFT_MAX
	/** Original 16k page size for InnoDB tablespaces. */
	UNIV_PAGE_SIZE_ORIG = 1 << UNIV_PAGE_SIZE_SHIFT_ORIG
	/** log2 of smallest compressed page size (1<<10 == 1024 bytes)
	Note: This must never change! */
	UNIV_ZIP_SIZE_SHIFT_MIN = 10
	/** log2 of largest compressed page size (1<<14 == 16384 bytes).
	A compressed page directory entry reserves 14 bits for the start offset
	and 2 bits for flags. This limits the uncompressed page size to 16k.
	Even though a 16k uncompressed page can theoretically be compressed
	into a larger compressed page, it is not a useful feature so we will
	limit both with this same constant. */
	UNIV_ZIP_SIZE_SHIFT_MAX = 14
	/** Smallest compressed page size */
	UNIV_ZIP_SIZE_MIN = 1 << UNIV_ZIP_SIZE_SHIFT_MIN
	/** Largest compressed page size */
	UNIV_ZIP_SIZE_MAX = 1 << UNIV_ZIP_SIZE_SHIFT_MAX
	/** Original 16k InnoDB Page Size as an ssize (log2 - 9) */
	UNIV_PAGE_SSIZE_ORIG = UNIV_PAGE_SIZE_SHIFT_ORIG - 9
)

universal page sizes

View Source 
const (
	/*                      EXTENT DESCRIPTOR
	                        =================
	File extent descriptor data structure: contains bits to tell which pages in
	the extent are free and which contain old tuple version to clean. */
	/*-------------------------------------*/
	// The identifier of the segment to which this extent belongs
	XDES_ID = 0
	// The list node data structure for the descriptors
	XDES_FLST_NODE = 8
	// contains state information of the extent
	XDES_STATE = FLST_NODE_SIZE + 8
	// Descriptor bitmap of the pages in the extent
	XDES_BITMAP = FLST_NODE_SIZE + 12
	/** How many bits are there per page */
	XDES_BITS_PER_PAGE = 2
	/** Offset of the descriptor array on a descriptor page */
	XDES_ARR_OFFSET = FSP_HEADER_OFFSET + FSP_HEADER_SIZE
)
View Source 
const (
	/** SDI BLOB not expected before the following page number.
	0 (tablespace header), 1 (tabespace bitmap), 2 (ibuf bitmap)
	3 (SDI Index root page) */
	SDI_BLOB_ALLOWED = 4
)

Variables

View Source 
var ColumnMembers = []string{
	`name`,
	`type`,
	`is_nullable`,
	`is_zerofill`,
	`is_unsigned`,
	`is_auto_increment`,
	`is_virtual`,
	`hidden`,
	`ordinal_position`,
	`char_length`,
	`numeric_precision`,
	`numeric_scale`,
	`numeric_scale_null`,
	`datetime_precision`,
	`datetime_precision_null`,
	`has_no_default`,
	`default_value_null`,
	`srs_id_null`,
	`srs_id`,
	`default_value`,
	`default_value_utf8_null`,
	`default_value_utf8`,
	`default_option`,
	`update_option`,
	`comment`,
	`generation_expression`,
	`generation_expression_utf8`,
	`options`,
	`se_private_data`,
	`engine_attribute`,
	`secondary_engine_attribute`,
	`column_key`,
	`column_type_utf8`,
	`elements`,
	`collation_id`,
	`is_explicit_collation`,
}
View Source 
var ForeignKeyElementMembers = []string{
	`referenced_column_name`,
	`column_opx`,
}
View Source 
var ForeignKeyMembers = []string{
	`name`,
	`elements`,
	`update_rule`,
	`delete_rule`,
	`referenced_table_schema_name`,
	`referenced_table_name`,
}
View Source 
var INFIMUM_DATA = []byte{
	0x69, 0x6e, 0x66, 0x69, 0x6d, 0x75, 0x6d, 0x00,
}

* Data bytes of an infimum record

View Source 
var INFIMUM_EXTRA = []byte{
	0x01,
	0x00, 0x02,
}
The infimum and supremum records are omitted from the compressed page.

On compress, we compare that the records are there, and on uncompress we restore the records. * Extra bytes of an infimum record

View Source 
var IndexElementMembers = []string{
	`length`,
	`hidden`,
	`column_opx`,
}
View Source 
var IndexMembers = []string{
	`name`,
	`type`,
	`hidden`,
	`elements`,
	`options`,
	`algorithm`,
	`is_algorithm_explicit`,
	`comment`,
}
View Source 
var SUPREMUM_EXTRA_DATA = []byte{

	0x00,
	0x0b,
	0x00,
	0x00,
	0x73,
	0x75,
	0x70,
	0x72,
	0x65,
	0x6d,
	0x75,
	0x6d,
}

* Extra bytes and data bytes of a supremum record

View Source 
var UNIV_PAGE_SIZE uint32

Universal page size

Functions

func CheckColumnMembers 

func CheckColumnMembers(column gjson.Result) error

func CheckForeignKeyElementMembers 

func CheckForeignKeyElementMembers(element gjson.Result) error

func CheckForeignKeyMembers 

func CheckForeignKeyMembers(fk gjson.Result) error

func CheckIndexElementMembers 

func CheckIndexElementMembers(element gjson.Result) error

func CheckIndexMembers 

func CheckIndexMembers(index gjson.Result) error

func CheckMember 

func CheckMember(column gjson.Result, member string) error

func FspExtentSize 

func FspExtentSize(size uint32) uint32

* File space extent size in pages page size | file space extent size ----------+----------------------- 

 4 KiB  | 256 pages = 1 MiB
 8 KiB  | 128 pages = 1 MiB
16 KiB  |  64 pages = 1 MiB
32 KiB  |  64 pages = 2 MiB
64 KiB  |  64 pages = 4 MiB

FSPExtentSize calculates the file space extent size in pages based on a given page size

func FspFlagsGetPageSsize 

func FspFlagsGetPageSsize(flags uint32) uint32

FspFlagsGetPageSsize returns the value of the PAGE_SSIZE field from the given flags.

func FspFlagsGetZipSsize 

func FspFlagsGetZipSsize(flags uint32) uint32

* FspFlagsGetZipSsize Return the value of the ZIP_SSIZE field

func FspFlagsHasSDI 

func FspFlagsHasSDI(flags uint32) uint32

* Return the value of the SDI field

func FspHeaderGetField 

func FspHeaderGetField(page []byte, field uint32) uint32

* Read a tablespace header field. @param[in] page first page of a tablespace @param[in] field the header field @return the contents of the header field

func FspHeaderGetFlags 

func FspHeaderGetFlags(page []byte) uint32

* Read the flags from the tablespace header page. @param[in] page first page of a tablespace @return the contents of FSP_SPACE_FLAGS

func FspHeaderGetSDIOffset 

func FspHeaderGetSDIOffset(pageSize *PageSize) (offset uint32)

func GetXdesSize 

func GetXdesSize(pageSize *PageSize) uint32

func ParseCollation 

func ParseCollation(ddObject gjson.Result) (ddl string, err error)

* Parse Collation @param[in] collation Table collation object @param[in,out] ddl DDL string @return False in case of errors

func ParseEngine 

func ParseEngine(ddlObject gjson.Result) (ddl string, err error)

* Parse the engine section of SDI JSON @param[in] dd_object Data Dictionary JSON object @param[in,out] ddl DDL string @return False in case of errors

func ParseForeignKeys 

func ParseForeignKeys(ddObject gjson.Result, columnCache ColumnCache, tableSchema string) (
	ddl string, err error)

func ParseIndexes 

func ParseIndexes(ddObject gjson.Result, columnCache ColumnCache) (
	ddl string, err error)

* Parse the indexes section of SDI JSON @param[in] dd_object Data Dictionary JSON object @param[in,out] ddl DDL string @return False in case of errors

func SetUnivPageSize 

func SetUnivPageSize(size uint32)

set UNIV_PAGE_SIZE

func UTBitsInBytes 

func UTBitsInBytes(bits uint32) uint32

UTBitsInBytes converts a given number of bits to the minimum number of bytes needed to store them

func XdesArrSize 

func XdesArrSize(pageSize *PageSize) uint32

* Calculates the descriptor array size. @param[in] page_size page size @return size of descriptor array

Types

type Collation 

type Collation struct {
	ID          int    `json:"id"`
	CharsetName string `json:"charset_name"`
	Name        string `json:"name"`
	IsDefault   bool   `json:"is_default"`
	Maxlen      int    `json:"maxlen"`
}

func GetCollationByID 

func GetCollationByID(id int) (*Collation, error)

GetCollationByID returns collations by given id.

type Column 

type Column struct {
	OrdinalPosition      int
	Name                 string
	Type                 ColumnType
	GenerationExpression string
	Hidden               HiddenType
	Size                 uint64
	Collation            *Collation
	GJson                gjson.Result
	DDL                  string
}

func NewColumn 

func NewColumn(c gjson.Result) (*Column, error)

func (*Column) SupportPrefixIndex 

func (c *Column) SupportPrefixIndex() bool
Check if column type support index prefix

@return True if the type supports index prefix

type ColumnCache 

type ColumnCache map[int]*Column

func ParseColumns 

func ParseColumns(ddObject gjson.Result) (
	ddl string, columnCache ColumnCache, err error)

func (ColumnCache) AddColumn 

func (cc ColumnCache) AddColumn(c gjson.Result) (column *Column, err error)

type ColumnType 

type ColumnType int

https://github.com/mysql/mysql-server/blob/trunk/sql/dd/types/column.h#L52 

const (
	CT_DECIMAL ColumnType = iota + 1
	CT_TINY
	CT_SHORT
	CT_LONG
	CT_FLOAT
	CT_DOUBLE
	CT_TYPE_NULL
	CT_TIMESTAMP
	CT_LONGLONG
	CT_INT24
	CT_DATE
	CT_TIME
	CT_DATETIME
	CT_YEAR
	CT_NEWDATE
	CT_VARCHAR
	CT_BIT
	CT_TIMESTAMP2
	CT_DATETIME2
	CT_TIME2
	CT_NEWDECIMAL
	CT_ENUM
	CT_SET
	CT_TINY_BLOB
	CT_MEDIUM_BLOB
	CT_LONG_BLOB
	CT_BLOB
	CT_VAR_STRING
	CT_STRING
	CT_GEOMETRY
	CT_JSON
)

type FKMatchOption 

type FKMatchOption int64
const (
	FK_OPTION_NONE FKMatchOption = iota + 1
	FK_OPTION_PARTIAL
	FK_OPTION_FULL
)

type FKRule 

type FKRule int64

https://github.com/mysql/mysql-server/blob/trunk/sql/dd/types/foreign_key.h 

const (
	FK_RULE_NO_ACTION FKRule = iota + 1
	FK_RULE_RESTRICT
	FK_RULE_CASCADE
	FK_RULE_SET_NULL
	FK_RULE_SET_DEFAULT
)

type ForeignKey 

type ForeignKey struct {
	Name                      string
	UpdateRule                FKRule
	DeleteRule                FKRule
	ReferenceNames            []string
	ReferencedTableSchemaName string
	ReferencedTableName       string
	Gjson                     gjson.Result
	DDL                       string
}

func NewForeignKey 

func NewForeignKey(fk gjson.Result) *ForeignKey

type ForeignKeyElement 

type ForeignKeyElement struct {
	ReferencedColumnName string
	ColumnOpx            int
}

func NewForeignKeyElement 

func NewForeignKeyElement(element gjson.Result) *ForeignKeyElement

type HiddenType 

type HiddenType int64
const (
	/// The column is visible (a normal column)
	HT_VISIBLE HiddenType = iota + 1
	/// The column is completely invisible to the server
	HT_HIDDEN_SE
	/// The column is visible to the server but hidden from the user.
	/// This is used for i.e. implementing functional indexes.
	HT_HIDDEN_SQL
	/// User table column marked as INVISIBLE by using the column visibility
	/// attribute. Column is hidden from the user unless it is explicitly
	/// referenced in the statement. Column is visible to the server.
	HT_HIDDEN_USER
)

type Index 

type Index struct {
	Name                string
	Type                IndexType
	Hidden              bool
	GJson               gjson.Result
	DDL                 string
	Algorithm           IndexAlgorithm
	IsAlgorithmExplicit bool
	Comment             string
}

func NewIndex 

func NewIndex(i gjson.Result) *Index

type IndexAlgorithm 

type IndexAlgorithm int64 // similar to ha_key_alg

const (
	IA_SE_SPECIFIC IndexAlgorithm = iota + 1
	IA_BTREE
	IA_RTREE
	IA_HASH
	IA_FULLTEXT
)

func (IndexAlgorithm) String 

func (ia IndexAlgorithm) String() string

type IndexElement 

type IndexElement struct {
	Length    int64
	Hidden    bool
	ColumnOpx int
}

func NewIndexElement 

func NewIndexElement(e gjson.Result) *IndexElement

type IndexType 

type IndexType int64

https://github.com/mysql/mysql-server/blob/trunk/sql/dd/types/index.h#L57 

const (
	IT_PRIMARY IndexType = iota + 1
	IT_UNIQUE
	IT_MULTIPLE
	IT_FULLTEXT
	IT_SPATIAL
)

func (IndexType) String 

func (it IndexType) String() string

type Page 

type Page struct {
	*PageSize
	PageNum          uint32
	NHeap            uint16
	NHeapBase        uint16
	NRecs            uint16
	NDense           uint16
	NSlots           uint16
	HeapTop          uint16
	OriginData       []byte
	UncompressedData []byte
	SlotOffset       uint32
	Slot             []byte
	Recs             []uint16
	PageLevel        uint16
	PageType         PageType
	NextPageNum      uint32
}

func NewPage 

func NewPage(pageNum uint32, pageSize *PageSize, originData []byte) (p *Page, err error)

func (*Page) Decompress 

func (p *Page) Decompress() (err error)

pageZipDecompressLow decompresses a page. This function should tolerate errors on the compressed page. Instead of causing assertions to fail, it will return false if an inconsistency is detected.

Returns true on success, false on failure.

func (*Page) DecompressDIR 

func (p *Page) DecompressDIR() (err error)

* Populate the sparse page directory from the dense directory. @return true on success, false on failure

func (*Page) GetHeapTop 

func (p *Page) GetHeapTop()

func (*Page) GetIsCompact 

func (p *Page) GetIsCompact()
  • Determine whether the page is in new-style compact format. @return nonzero if the page is in compact format, zero if it is in old-style format

func (*Page) GetNDens 

func (p *Page) GetNDens()
  • Gets the number of elements in the dense page directory, including deleted records (the free list). @return number of elements in the dense page directory

func (*Page) GetNHeap 

func (p *Page) GetNHeap()

func (*Page) GetNHeapBase 

func (p *Page) GetNHeapBase()

func (*Page) GetNRecs 

func (p *Page) GetNRecs()
  • Gets the number of user records on page (infimum and supremum records are not user records).

func (*Page) GetNSlotOffset 

func (p *Page) GetNSlotOffset() (offset uint32)
  • Gets pointer to nth directory slot. @return pointer to dir slot

func (*Page) GetNSlots 

func (p *Page) GetNSlots()
  • Gets the number of dir slots in directory. @return number of slots

func (*Page) GetNextPageNum 

func (p *Page) GetNextPageNum()

func (*Page) GetPageLevel 

func (p *Page) GetPageLevel()

* Determine whether the page is a B-tree leaf. @return true if the page is a B-tree leaf (PAGE_LEVEL = 0)

func (*Page) GetPageType 

func (p *Page) GetPageType()

func (*Page) GetRecType 

func (p *Page) GetRecType(recOffset uint32) (recType byte)

func (*Page) HeaderGetField 

func (p *Page) HeaderGetField(field int) uint16

func (*Page) IsEmpty 

func (p *Page) IsEmpty() bool

* Determine whether the page is empty. @return true if the page is empty (PAGE_N_RECS = 0)

func (*Page) RecCheck 

func (p *Page) RecCheck(recOffset uint32) (err error)

* Used to check the consistency of a record on a page. @return true if succeed

func (*Page) RecGetBitField_1 

func (p *Page) RecGetBitField_1(recOffset uint32, offs uint32, mask uint32, shift uint32) uint32

* Gets a bit field from within 1 byte. in: 

rec: pointer to record origin
offs: offset from the origin down
mask: mask used to filter bits
shift: shift right applied after masking

func (*Page) RecGetDeletedFlag 

func (p *Page) RecGetDeletedFlag(recOffset uint32) uint32

func (*Page) RecGetNextOffs 

func (p *Page) RecGetNextOffs(recOffset uint16) (nextRecOffset uint16, err error)

func (*Page) RecIsInfimum 

func (p *Page) RecIsInfimum(recOffset uint32) (isInfimum bool, err error)
  • true if the record is the infimum record on a page. @return true if the infimum record

func (*Page) RecIsInfimumLow 

func (p *Page) RecIsInfimumLow(recOffset uint32) (isInfimumLow bool, err error)
  • true if the record is the infimum record on a page. @return true if the infimum record

func (*Page) RecIsSupremum 

func (p *Page) RecIsSupremum(recOffset uint32) (isSupremum bool, err error)

* true if the record is the supremum record on a page. @return true if the supremum record

func (*Page) RecIsSupremumLow 

func (p *Page) RecIsSupremumLow(recOffset uint32) (isSupremumLow bool, err error)

* true if the record is the supremum record on a page. @return true if the supremum record

func (*Page) RecSetNextOffsNew 

func (p *Page) RecSetNextOffsNew(data []byte, currentOffset, nextOffset uint32) error
  • The following function is used to set the next record offset field of a new-style record.

func (*Page) ZipDirGet 

func (p *Page) ZipDirGet(slotID uint32) uint16

* Read a given slot in the dense page directory. @return record offset on the uncompressed page, possibly ORed with PAGE_ZIP_DIR_SLOT_DEL or PAGE_ZIP_DIR_SLOT_OWNED

type PageSize 

type PageSize struct {
	Physical      uint32
	PhysicalShift uint32
	Logical       uint32
	LogicalShift  uint32
	SSize         uint32
	IsCompressed  bool
	IsCompact     bool
}

* Page size descriptor. Contains the physical and logical page size, as well as whether the page is compressed or not.

func NewPageSize 

func NewPageSize(physical, logical uint32, isCompressed bool) (ps *PageSize, err error)

func NewPageSizeWithFlag 

func NewPageSizeWithFlag(fsp_flags uint32) (ps *PageSize, err error)

func (*PageSize) GetShift 

func (ps *PageSize) GetShift() (err error)

type PageType 

type PageType uint16

type SDI 

type SDI struct {
	Type                uint64
	ID                  uint64
	OriginData          []byte
	OriginDataLen       uint64
	UncompressedData    []byte
	UncompressedDataLen uint64
	DatabaseName        string
	TableSchema         *TableSchema
}

func (*SDI) DumpJson 

func (sdi *SDI) DumpJson() (result []byte)

func (*SDI) DumpTableSchema 

func (sdi *SDI) DumpTableSchema() (err error)

type TableSchema 

type TableSchema struct {
	Hidden HiddenType
	Name   string
	DDL    string
}

type TableSpace 

type TableSpace struct {
	Reader          io.Reader
	Buf             *bytes.Buffer
	Flags           uint32
	SpaceID         uint32
	FirstPageNum    uint32
	PageSize        *PageSize
	TablespaceFiles []*TablespaceFile
	SDIVersion      uint32
	SDIRootOffset   uint32
	SDIRootPageNum  uint32
	SDIRootPage     *Page
	SDIPages        []*Page
	SDIPagesMap     map[uint32]*Page
	LeafSDIPage     *Page
	SDIs            []*SDI
	CurPage         *Page
	SDIResult       []byte
	TableSchemas    map[string]*TableSchema
}

func NewTableSpace 

func NewTableSpace(r io.Reader) (ts *TableSpace, err error)

func (*TableSpace) CopyCompressedBlob 

func (ts *TableSpace) CopyCompressedBlob(
	firstBlobPageNum uint32, totalOffsetPageLength uint64, destBuf []byte) (err error)

* Read the compressed blob stored in off-pages to the buffer. @param[in] ts tablespace structure @param[in] first_blob_page_num first blob page number of the chain @param[in] total_off_page_length total Length of blob stored in record @param[in,out] dest_buf blob will be copied to this buffer @return 0 if blob is not read, else the total length of blob read from off-pages

func (*TableSpace) CopyUncompressedBlob 

func (ts *TableSpace) CopyUncompressedBlob(
	firstBlobPageNum uint32, totalOffsetPageLength uint64, destBuf []byte) (err error)

* Read the uncompressed blob stored in off-pages to the buffer. @param[in] ts tablespace structure @param[in] first_blob_page_num first blob page number of the chain @param[in] total_off_page_length total length of blob stored in record @param[in,out] dest_buf blob will be copied to this buffer @return 0 if blob is not read, else the total length of blob read from off-pages

func (*TableSpace) DumpAllRecsInLeafLevel 

func (ts *TableSpace) DumpAllRecsInLeafLevel() (err error)

func (*TableSpace) DumpSDIs 

func (ts *TableSpace) DumpSDIs() (err error)

func (*TableSpace) DumpSchemas 

func (ts *TableSpace) DumpSchemas() (err error)

func (*TableSpace) FetchPage 

func (ts *TableSpace) FetchPage(pageNum uint32) (page *Page, err error)

func (*TableSpace) FetchSDIPageAndLevel 

func (ts *TableSpace) FetchSDIPageAndLevel(pageNum uint32) (err error)

func (*TableSpace) FetchSDIPages 

func (ts *TableSpace) FetchSDIPages() (err error)

Read SDI pages

func (*TableSpace) GetFirstUserRec 

func (ts *TableSpace) GetFirstUserRec() (curRecOffset uint16, err error)

func (*TableSpace) GetNextRecOffset 

func (ts *TableSpace) GetNextRecOffset(recOffset uint16) (nextRecOffset uint16, err error)

func (*TableSpace) GetPageSize 

func (ts *TableSpace) GetPageSize() (err error)

func (*TableSpace) GetSDIRoot 

func (ts *TableSpace) GetSDIRoot()

func (*TableSpace) ParseFieldsInRec 

func (ts *TableSpace) ParseFieldsInRec(curRecOffset uint16) (err error)

* Extract SDI record fields @param[in] rec pointer to record @param[in,out] sdi_type sdi type @param[in,out] sdi_id sdi id @param[in,out] sdi_data sdi blob @param[in,out] sdi_data_len length of sdi blob @return DB_SUCCESS on success, else error code

func (*TableSpace) ReadToOffset 

func (ts *TableSpace) ReadToOffset(offset uint32) (err error)

type TablespaceFile 

type TablespaceFile struct {
	/** 0 in fil_per_table tablespaces, else the first page number in
	  subsequent data file in multi-file tablespace. */
	FirstPageNum uint32
	/** Total number of pages in a data file. */
	TotalNumOfPages uint32
	/** File handle of the data file. */
	File *os.File
}

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.