dataset

Documentation

Overview

Contains access/util functions to read parts of binary protobuf dataset files, JSON dataset metadata summaries, spectra, etc

Index

• func DecodeZeroRun(encodedSpectrum []int32) []int32
• func GetDataset(svcs *services.APIServices, s3Path string) (*protos.Experiment, error)
• func GetDetectorMetaValue(metaLabel string, detector *protos.Experiment_Location_DetectorSpectrum, ...) (protos.Experiment_MetaDataType, *protos.Experiment_Location_MetaDataItem, ...)
• func GetPMCsForLocationIndexes(locationIndexes []int32, dataset *protos.Experiment) ([]int, error)
• func MakeLocToPMCLookup(dataset *protos.Experiment, onlyWithNormalOrDwellSpectra bool) (map[int32]int32, error)
• func MakePMCBeamIndexLookup(dataset *protos.Experiment) map[int32]int32
• func MakePMCBeamLookup(dataset *protos.Experiment) map[int32]*protos.Experiment_Location_BeamLocation
• func MakePMCHasDwellLookup(dataset *protos.Experiment) (map[int32]bool, error)
• func ReadDatasetFile(path string) (*protos.Experiment, error)
• type APIDatasetSummary
• type DatasetConfig
• type MatchedAlignedImageMeta
• type SpectrumMetaValues
  • func GetSpectrumMeta(detector *protos.Experiment_Location_DetectorSpectrum, ...) (SpectrumMetaValues, error)
• type SummaryFileData
  • func ReadDataSetSummary(fs fileaccess.FileAccess, dataBucket string, datasetID string) (SummaryFileData, error)
  • func (s SummaryFileData) GetRTT() string

Functions

func DecodeZeroRun

func DecodeZeroRun(encodedSpectrum []int32) []int32

DecodeZeroRun - Decoding a zero-run-length encoded spectrum, where runs of 0's are replaced with a count and a 0

func GetDataset

func GetDataset(svcs *services.APIServices, s3Path string) (*protos.Experiment, error)

GetDataset - returns a dataset proto after downloading from S3

func GetDetectorMetaValue

func GetDetectorMetaValue(metaLabel string, detector *protos.Experiment_Location_DetectorSpectrum, dataset *protos.Experiment) (protos.Experiment_MetaDataType, *protos.Experiment_Location_MetaDataItem, error)

GetDetectorMetaValue - Has to do the same job as the function with the same name in PIXLISE DataSets.ts:

func GetPMCsForLocationIndexes

func GetPMCsForLocationIndexes(locationIndexes []int32, dataset *protos.Experiment) ([]int, error)

func MakeLocToPMCLookup

func MakeLocToPMCLookup(dataset *protos.Experiment, onlyWithNormalOrDwellSpectra bool) (map[int32]int32, error)

func MakePMCBeamIndexLookup

func MakePMCBeamIndexLookup(dataset *protos.Experiment) map[int32]int32

Maybe the name is not that clear, creates a lookup for PMC->index in beam ij array, eg if PMC6 has a context image and it's the first set of coordinates in the beam ij's, 6 will map to 0

func MakePMCBeamLookup

func MakePMCBeamLookup(dataset *protos.Experiment) map[int32]*protos.Experiment_Location_BeamLocation

func MakePMCHasDwellLookup

func MakePMCHasDwellLookup(dataset *protos.Experiment) (map[int32]bool, error)

func ReadDatasetFile

func ReadDatasetFile(path string) (*protos.Experiment, error)

ReadDatasetFile - reads a dataset proto from local file system

Types

type APIDatasetSummary

type APIDatasetSummary struct {
	*SummaryFileData

	DataSetLink      string `json:"dataset_link"`
	ContextImageLink string `json:"context_image_link"`
}

APIDatasetSummary - contains metadata fields for a given dataset This is returned from the API dataset listing endpoint. It's not private to that code because it's also used in the integration test code that tests it, and may be needed elsewhere in future

type DatasetConfig

type DatasetConfig struct {
	Datasets []SummaryFileData `json:"datasets"`
}

DatasetConfig is the container of the above This is the struct for dataset JSON files, as used by datasource updater lambda and dataset listing API endpoint

type MatchedAlignedImageMeta

type MatchedAlignedImageMeta struct {
	// PMC of the MCC image whose beam locations this image is matched with
	AlignedBeamPMC int32 `json:"aligned-beam-pmc"`

	// File name of the matched image - the one that was imported with an area matching the Aligned image
	MatchedImageName string `json:"matched-image"`

	// This is the x/y offset of the sub-image area where the Matched image matches the Aligned image
	// In other words, the top-left Aligned image pixel is at (XOffset, YOffset) in the matched image
	XOffset float32 `json:"x-offset"`
	YOffset float32 `json:"y-offset"`

	// The relative sizing of the sub-image area where the Matched image matches the Aligned image
	// In other words, if the Aligned image is 752x580 pixels, and the Matched image is much higher res
	// at 2000x3000, and within that a central area of 1600x1300, scale is (1600/752, 1300/580) = (2.13, 2.24)
	XScale float32 `json:"x-scale"`
	YScale float32 `json:"y-scale"`

	// Full path, no JSON field because this is only used internally during dataset conversion
	MatchedImageFullPath string `json:"-"`
}

MatchedAlignedImageMeta - metadata for an image that's transformed to match an AlignedImage (eg MCC)

type SpectrumMetaValues

type SpectrumMetaValues struct {
	SCLK     int32
	RealTime float32
	LiveTime float32
	XPerChan float32
	Offset   float32
	Detector string
	ReadType string
}

Intermediate representation which can be passed as an array to spectrum CSV writer. This is useful to be able to provide bulk sums!

func GetSpectrumMeta

func GetSpectrumMeta(detector *protos.Experiment_Location_DetectorSpectrum, dataset *protos.Experiment) (SpectrumMetaValues, error)

GetSpectrumMeta - Read meta columns saved in our bin file for a given spectrum These are named in pixlise-data-converter/importer/pixlfm/spectra.go

type SummaryFileData

type SummaryFileData struct {
	DatasetID           string      `json:"dataset_id"`
	Group               string      `json:"group"`
	DriveID             int32       `json:"drive_id"`
	SiteID              int32       `json:"site_id"`
	TargetID            string      `json:"target_id"`
	Site                string      `json:"site"`
	Target              string      `json:"target"`
	Title               string      `json:"title"`
	SOL                 string      `json:"sol"`
	RTT                 interface{} `json:"rtt,string"` // Unfortunately we stored it as int initially, so this has to accept files stored that way
	SCLK                int32       `json:"sclk"`
	ContextImage        string      `json:"context_image"`
	LocationCount       int         `json:"location_count"`
	DataFileSize        int         `json:"data_file_size"`
	ContextImages       int         `json:"context_images"`
	TIFFContextImages   int         `json:"tiff_context_images"`
	NormalSpectra       int         `json:"normal_spectra"`
	DwellSpectra        int         `json:"dwell_spectra"`
	BulkSpectra         int         `json:"bulk_spectra"`
	MaxSpectra          int         `json:"max_spectra"`
	PseudoIntensities   int         `json:"pseudo_intensities"`
	DetectorConfig      string      `json:"detector_config"`
	CreationUnixTimeSec int64       `json:"create_unixtime_sec"`
}

SummaryFileData - Structure of dataset summary JSON files

func ReadDataSetSummary

func ReadDataSetSummary(fs fileaccess.FileAccess, dataBucket string, datasetID string) (SummaryFileData, error)

func (SummaryFileData) GetRTT

func (s SummaryFileData) GetRTT() string