csfw

package module
v0.0.0-...-6f8fa1e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 2, 2016 License: Apache-2.0 Imports: 0 Imported by: 0

README

CoreStore library WIP = Work in Progress

Build Status wercker status Appveyor Build status GoDoc goreportcard

eCommerce library which is compatible to Magento 1 and 2 database schema.

Magento is a trademark of MAGENTO, INC..

Min. Go Version: 1.8

Usage

To properly use the CoreStore library some environment variables must be set before running go generate. (TODO)

Required settings

CS_DSN the environment variable for the MySQL connection.

$ export CS_DSN='magento1:magento1@tcp(localhost:3306)/magento1'
$ export CS_DSN='magento2:magento2@tcp(localhost:3306)/magento2'

$ go get github.com/corestoreio/csfw
$ export CS_DSN='see previous'
$ cd $GOPATH/src/github.com/corestoreio/csfw
$ go run codegen/tableToStruct/*.go

Testing

Setup two databases. One for Magento 1 and one for Magento 2 and fill them with the provided testdata.

Create a DSN env var CS_DSN and point it to Magento 1 database. Run the tests. Change the env var to let it point to Magento 2 database. Rerun the tests.

Finding Allocations

Side note: There is a new testing approach: TBDD = Test n' Benchmark Driven Development.

On the first run we got this result:

$ go test -run=😶 -bench=Benchmark_WithInitStoreByToken .
PASS
Benchmark_WithInitStoreByToken-4	  100000	     17297 ns/op	    9112 B/op	     203 allocs/op
ok  	github.com/corestoreio/csfw/store	2.569s

Quite shocking to use 203 allocs for just figuring out the current store view within a request.

Now compile your tests into an executable binary:

$ go test -c

This compilation reduces the noise in the below output trace log.

$ GODEBUG=allocfreetrace=1 ./store -test.run=😶 -test.bench=Benchmark_WithInitStoreByToken -test.benchtime=10ms 2>trace.log

Now open the trace.log file (around 26MB) and investigate all the allocations and refactor your code. Once finished you can achieve results like:

$ go test -run=NONE -bench=Benchmark_WithInitStoreByToken .
PASS
Benchmark_WithInitStoreByToken-4	 2000000	       826 ns/op	     128 B/op	       5 allocs/op
ok  	github.com/corestoreio/csfw/store	2.569s
Profiling
$ go test -cpuprofile=cpu.out -benchmem -memprofile=mem.out -run=NONE -bench=NameOfBenchmark -v
$ go tool pprof packageName.test cpu.out
Entering interactive mode (type "help" for commands)
(pprof) top5
560ms of 1540ms total (36.36%)
Showing top 5 nodes out of 112 (cum >= 60ms)
      flat  flat%   sum%        cum   cum%
     180ms 11.69% 11.69%      400ms 25.97%  runtime.mallocgc
  • flat is how much time is spent inside of a function.
  • cum shows how much time is spent in a function, and also in any code called by a function.

For memory profile:

Sample value selection option (for heap profiles):
  -inuse_space      Display in-use memory size
  -inuse_objects    Display in-use object counts
  -alloc_space      Display allocated memory size
  -alloc_objects    Display allocated object counts

$ go tool pprof -alloc_objects packageName.test mem.out
Bound Check Elimination

http://klauspost-talks.appspot.com/2016/go17-compiler.slide

$ go build -gcflags="-d=ssa/check_bce/debug=1" bounds.go
or
$ go test -gcflags="-d=ssa/check_bce/debug=1" .

Success - Check bounds outside the loop.

Running Benchmark

Assuming we have already an existing file called bm_baseline.txt.

$ go test -v -run=🤐 -bench=. -count=10 . > bm_baseline_new.txt

After running above command to generate the second benchmark statistics file we run:

$ benchstat bm_baseline.txt bm_baseline_new.txt

https://godoc.org/rsc.io/benchstat

Other development helpers

A preconfigured linter file lint has been included in this repoistory.

TODO

If you find an entry in the source like TODO(CS) that means you can ask CS to get more information about how to implement and what to fix if the context of the todo description isn't understandable.

  • Create Magento 1+2 modules to setup test database and test Magento system.

Contributing

Please have a look at the contribution guidelines.

Acknowledgements

Some packages have been fully refactored but the initial idea has been extracted from the geniuses listed below:

Name Package License
Steve Francia util/conv MIT Copyright (c) 2014
Jonathan Novak, Tyler Smith, Michal Bohuslávek dbr The MIT License (MIT) 2014
Martin Angers and Contributors. ctxthrottled The MIT License (MIT) 2014
Dave Cheney util/errors The MIT License (MIT) 2015
Jad Dittmar finance aka. money Copyright (c) 2011
Wenbin Xiao util/sqlparser Copyright 2015 BSD Style
Google Inc youtube/vitess\sqlparser Copyright 2012 BSD Style
Olivier Poitrey ctxmw.WithAccessLog & CORS Copyright (c) 2014-2015 MIT License
Dave Grijalva csjwt Copyright (c) 2012 MIT License
Uber Technologies, Inc. log Copyright (c) 2016 MIT License
2013 The Go Authors singleflight Copyright (c) 2013 BSD Style
Minio Cloud Storage, (C) 2016 Minio, Inc. blake2b-simd Apache License, Version 2.0
Ventu.io, Oleg Sklyar, contributors. util/shortid MIT License Copyright (c) 2016,
Carl Jackson (carl@avtok.com) (Goji) net/responseproxy Copyright (c) 2014, 2015, 2016
Greg Roseberry, 2014; Patrick O'Brien, 2016 util/null BSD Copyright (c) 2014, 2015, 2016
The Go-MySQL-Driver Authors util/null/time_mysql.go Mozilla Public License, v. 2.0, Copyright 2012
siddontang storage/binlogsync MIT Copyright (c) 2014
siddontang storage/myreplicator MIT Copyright (c) 2014

Licensing

CoreStore is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Cyrill Schumacher - PGP Key

Documentation

Overview

Package csfw contains the CoreStore library based on Magento's database structure. 99% compatible to Magento 1 and 2.

The package csfw contains at the moment only go:generate commands to build Go code.

Two skeleton projects (monolith and SOA) are already setup but of course empty.

Purpose

Why is someone trying to create a Magento database schema compatible online shop in Go?

Because performance :-)

Architecture

@todo 10km view.

Names

Generated SQL table structs start with the word "Table". The word "Slice" will be appended when there is a slice of structs.

Example for generated SQL table structs: 

type (
    // TableStoreSlice contains pointers to TableStore types
    TableStoreSlice []*TableStore
    // TableStore a type for the MySQL table core_store
    TableStore struct {
        ...
    }
)

Table indexes are iota constants and start with TableIndex[table name].

The word "collection" will be appended to a variable or function when that variable or function contains a materialized slice or handles it.

Event/Observer vs Publish/Subscribe

Basically those two are nearly the same. The only difference lies in its behaviour. Event/Observer can modify a type and block an operation while Publish/Subscribe runs asynchronously in its own Goroutine and cannot modify anything.

Required settings

CS_DSN the environment variable for the MySQL connection. 

$ export CS_DSN='magento1:magento1@tcp(localhost:3306)/magento1'
$ export CS_DSN='magento2:magento2@tcp(localhost:3306)/magento2'

$ go get github.com/corestoreio/csfw
$ export CS_DSN_TEST='see next section'
$ cd $GOPATH/src/github.com/corestoreio/csfw
$ go generate ./...

Testing

Setup two databases. One for Magento 1 and one for Magento 2 and fill them with the provided test data https://github.com/corestoreio/csfw/tree/master/testData

Create a DSN env var CS_DSN_TEST and point it to Magento 1 database. Run the tests. Change the env var to let it point to Magento 2 database. Rerun the tests. 

$ export CS_DSN_TEST='magento1:magento1@tcp(localhost:3306)/magento1'
$ export CS_DSN_TEST='magento2:magento2@tcp(localhost:3306)/magento2'

IDE

Currently using the IntelliJ IDEA Community Edition with the https://github.com/go-lang-plugin-org/go-lang-idea-plugin plugin.

At the moment Q2/2015: There are no official jar files for downloading so the go lang plugin will be compiled on a daily basis. The plugin works very well! Kudos to those developers!

IDEA has been configured with goimports, gofmt, golint, govet and ... with the file watcher plugin.

Why am I not using vim? Because I would only generate passwords ;-|.

Contributing

Please have a look at the contribution guidelines https://github.com/corestoreio/corestore/blob/master/CONTRIBUTING.md

Trademarks

Magento is a trademark of MAGENTO, INC. http://www.magentocommerce.com/license/

Source Files

View all Source files

Directories

Path Synopsis
Package backend provides common configuration and functions.
 Package backend provides common configuration and functions.
Package authorization enables management of access control list roles and rules.
 Package authorization enables management of access control list roles and rules.
xacml
Package xacml implements the OASIS/XACML standard for Policy-based authorization.
 Package xacml implements the OASIS/XACML standard for Policy-based authorization.
Package bundle handles the creation of catalog product bundles.
 Package bundle handles the creation of catalog product bundles.
Package catalog implements category tree and product features.
 Package catalog implements category tree and product features.
catattr
Package catattr handles all product and category related attributes.
 Package catattr handles all product and category related attributes.
catconfig
Package cataloginventory handles product inventory and indexing
 Package cataloginventory handles product inventory and indexing
Package codegen generates Go code and is only used in development.
 Package codegen generates Go code and is only used in development.
formhandler
formhandler generates code to map URL query string or URL encoded form data to structs and vice versa.
 formhandler generates code to map URL query string or URL encoded form data to structs and vice versa.
localization
package main generates i18n names of currencies, countries because not yet implemented in text/language TODO.
 package main generates i18n names of currencies, countries because not yet implemented in text/language TODO.
localization/gen
Package gen contains common code for the various code generation tools in the text repository.
 Package gen contains common code for the various code generation tools in the text repository.
materialization
package main materializes attributes, sets, groups, entity types, stores, websites, etc.
 package main materializes attributes, sets, groups, entity types, stores, websites, etc.
tableToStruct
package main generates Go structs, slices and function receivers from SQL tables.
 package main generates Go structs, slices and function receivers from SQL tables.
tableToStruct/_goast_generic
tableToStruct/_goast_generic_impl
tableToStruct/codecgen
codecgen generates codec.Selfer implementations for a set of types.
 codecgen generates codec.Selfer implementations for a set of types.
tableToStruct/tpl
Package config handles the configuration and its scopes.
 Package config handles the configuration and its scopes.
cfgmock
Package cfgmock provides mock types and functions for the config.Service type.
 Package cfgmock provides mock types and functions for the config.Service type.
cfgmodel
Package cfgmodel provides types to get/set values of a configuration.
 Package cfgmodel provides types to get/set values of a configuration.
cfgpath
Package cfgpath handles the configuration paths.
 Package cfgpath handles the configuration paths.
cfgsource
Package source provides a slice type for handling config sources to be used in config/models.
 Package source provides a slice type for handling config sources to be used in config/models.
element
Package element represents Magento system.xml configuration elements.
 Package element represents Magento system.xml configuration elements.
storage
Package storage defines the available configuration storage engines in its subpackages.
 Package storage defines the available configuration storage engines in its subpackages.
storage/boltdb
Package boltdb uses the bolt database for reading and writing configuration paths.
 Package boltdb uses the bolt database for reading and writing configuration paths.
storage/ccd
Package ccd = core_config_data uses the MySQL based table core_config_data for reading and writing configuration paths, scopes and values.
 Package ccd = core_config_data uses the MySQL based table core_config_data for reading and writing configuration paths, scopes and values.
storage/cfgbigcache
Package cfgbigcache uses the bigcache in-memory database for reading and writing configuration paths.
 Package cfgbigcache uses the bigcache in-memory database for reading and writing configuration paths.
storage/etcd
Package etcd uses etcd service for reading and writing configuration paths.
 Package etcd uses etcd service for reading and writing configuration paths.
Package customer handles a customer entity with its addresses.
 Package customer handles a customer entity with its addresses.
custattr
Package custattr handles all customer and address related attributes.
 Package custattr handles all customer and address related attributes.
custconfig
Package custconfig handles configuration structure and its models.
 Package custconfig handles configuration structure and its models.
Package directory provides features for currencies, currency rates, conversion of prices to a specified currency format, countries and regions.
 Package directory provides features for currencies, currency rates, conversion of prices to a specified currency format, countries and regions.
Package eav contains the logic for the Entity-Attribute-Value model based on the Magento database schema.
 Package eav contains the logic for the Entity-Attribute-Value model based on the Magento database schema.
Package mail provides functions and services for sending html or text emails via encrypted or unencrypted connections.
 Package mail provides functions and services for sending html or text emails via encrypted or unencrypted connections.
Package i18n supports string translations with variable substitution, CLDR pluralization, currency, formats, language, regions and timezones.
 Package i18n supports string translations with variable substitution, CLDR pluralization, currency, formats, language, regions and timezones.
Package locale provides locale specific services like formatting or translating.
 Package locale provides locale specific services like formatting or translating.
log
Package log contains BlackHole, StdLog, Log15 and the Logger interface.
 Package log contains BlackHole, StdLog, Log15 and the Logger interface.
http
Package http creates log fields for http Requests and Responses.
 Package http creates log fields for http Requests and Responses.
log15w
Package log15w provides a wrapper (w) to github.com/inconshreveable/log15.
 Package log15w provides a wrapper (w) to github.com/inconshreveable/log15.
logw
Package logw provides a wrapper (w) to Go's standard logger
 Package logw provides a wrapper (w) to Go's standard logger
zapw
Package zapw provides a wrapper (w) to github.com/uber-go/zap.
 Package zapw provides a wrapper (w) to github.com/uber-go/zap.
net
Package net provides additional network helper functions and in subpackages middleware.
 Package net provides additional network helper functions and in subpackages middleware.
accesslog
auth
Package auth provides authentication middleware.
 Package auth provides authentication middleware.
auth/backendauth
Package backendauth (TODO) defines the backend configuration options and element slices.
 Package backendauth (TODO) defines the backend configuration options and element slices.
compress
cors
Package cors provides a middleware for Cross-origin resource sharing (CORS).
 Package cors provides a middleware for Cross-origin resource sharing (CORS).
cors/backendcors
Package backendcors defines the backend configuration options and element slices.
 Package backendcors defines the backend configuration options and element slices.
cors/internal
csrf
Package csrf implements scope based Cross-Site Request Forgery protection.
 Package csrf implements scope based Cross-Site Request Forgery protection.
geoip
Package geoip detects the country by an IP address and provides alternative handlers.
 Package geoip detects the country by an IP address and provides alternative handlers.
geoip/backendgeoip
Package backendgeoip defines the backend configuration options and element slices.
 Package backendgeoip defines the backend configuration options and element slices.
geoip/maxmindfile
Package maxmindfile provides an OptionFactoryFunc for the backendgeopip package.
 Package maxmindfile provides an OptionFactoryFunc for the backendgeopip package.
geoip/maxmindwebservice
Package maxmindwebservice provides an OptionFactoryFunc for the backendgeopip package.
 Package maxmindwebservice provides an OptionFactoryFunc for the backendgeopip package.
httputil
Package httputil provides general functions for http handling (refactor) TODO(cs) => github.com/streadway/handy
 Package httputil provides general functions for http handling (refactor) TODO(cs) => github.com/streadway/handy
internal/scopedservice
Package scopedservice gets copied to specific net middleware packages - do not use.
 Package scopedservice gets copied to specific net middleware packages - do not use.
jwt
Package jwt provides a middleware for JSON web token authentication and runMode initialization.
 Package jwt provides a middleware for JSON web token authentication and runMode initialization.
jwt/backendjwt
Package backendjwt defines the backend configuration options and element slices.
 Package backendjwt defines the backend configuration options and element slices.
mw
Package mw provides a various middleware.
 Package mw provides a various middleware.
ratelimit
Package ratelimit implements scope based HTTP rate limiting.
 Package ratelimit implements scope based HTTP rate limiting.
ratelimit/backendratelimit
ratelimit/memstore
ratelimit/redigostore
request
Package request provides functions for http request handling.
 Package request provides functions for http request handling.
response
Package response provides functions for working with an http response.
 Package response provides functions for working with an http response.
responseproxy
Package responseproxy provides various proxy functions for extending http.ResponseWriter.
 Package responseproxy provides various proxy functions for extending http.ResponseWriter.
runmode
Package runmode defines store specific middleware to initialize the scope and its ID per request.
 Package runmode defines store specific middleware to initialize the scope and its ID per request.
secure
Package secure adds a middleware for quick security wins to response HTTP headers TODO(cs) https://github.com/unrolled/secure/blob/v1/secure.go
 Package secure adds a middleware for quick security wins to response HTTP headers TODO(cs) https://github.com/unrolled/secure/blob/v1/secure.go
signed
Package signed provides a middleware to sign responses and validate requests.
 Package signed provides a middleware to sign responses and validate requests.
signed/backendsigned
url
Package url parses program specific URLs and provides helper functions.
 Package url parses program specific URLs and provides helper functions.
Package payment defines a payment processing library which can connect to payment gateways.
 Package payment defines a payment processing library which can connect to payment gateways.
Package storage provides everything from MySQL, Redis, BoltDB, file, etc functions.
 Package storage provides everything from MySQL, Redis, BoltDB, file, etc functions.
binlogsync
containable
Package containable implements a container using a key with an expiration.
 Package containable implements a container using a key with an expiration.
csdb
Package csdb implements MySQL helper for tables, columns, statements, replication, validation and DB variables.
 Package csdb implements MySQL helper for tables, columns, statements, replication, validation and DB variables.
dbr
Package dbr has additions to Go's database/sql for super fast performance and convenience.
 Package dbr has additions to Go's database/sql for super fast performance and convenience.
money
Package money uses a fixed-length guard for precision arithmetic.
 Package money uses a fixed-length guard for precision arithmetic.
myreplicator
Package myreplicator handles the MySQL binary replication protocol.
 Package myreplicator handles the MySQL binary replication protocol.
text
Package text represents a []byte type for storing long text strings (Chars).
 Package text represents a []byte type for storing long text strings (Chars).
transcache
Package transcache transcodes arbitrary Go types to bytes and stores them in a cache reducing GC.
 Package transcache transcodes arbitrary Go types to bytes and stores them in a cache reducing GC.
transcache/tcbigcache
Package tcbigcache adapter for the transcache package to use an in-memory cache.
 Package tcbigcache adapter for the transcache package to use an in-memory cache.
transcache/tcboltdb
Package tcboltdb adapter for the boltdb to use a file based cache.
 Package tcboltdb adapter for the boltdb to use a file based cache.
transcache/tcredis
Package tcredis implements a wrapper for the Redis server.
 Package tcredis implements a wrapper for the Redis server.
Package store implements the handling of websites, groups and stores.
 Package store implements the handling of websites, groups and stores.
backendstore
scope
Package scope defines the configuration of scopes default, website, group and store.
 Package scope defines the configuration of scopes default, website, group and store.
storemock
Package mock implements mocking of the store.Service for tests.
 Package mock implements mocking of the store.Service for tests.
Package sync provides tools to deal with goroutines.
 Package sync provides tools to deal with goroutines.
singleflight
Package singleflight provides a duplicate function call suppression mechanism.
 Package singleflight provides a duplicate function call suppression mechanism.
suspend
Package suspend provides a complicated duplicate function call suppression mechanism.
 Package suspend provides a complicated duplicate function call suppression mechanism.
Package tax takes care of all kind of taxes.
 Package tax takes care of all kind of taxes.
Package user takes care of backend/API users.
 Package user takes care of backend/API users.
Package util should stay empty.
 Package util should stay empty.
bufferpool
Package bufferpool implements a pool of reusable auto-resizing buffers.
 Package bufferpool implements a pool of reusable auto-resizing buffers.
conv
Package conv - easy and safe casting from one type to another.
 Package conv - easy and safe casting from one type to another.
csjwt
Package csjwt handles JSON web tokens.
 Package csjwt handles JSON web tokens.
csjwt/cmd/jwt
A useful example app.
 A useful example app.
csjwt/jwtclaim
Package jwtclaim provides claim structs and maps for convenience.
 Package jwtclaim provides claim structs and maps for convenience.
csmath
Package csmath provides additional math helper functions.
 Package csmath provides additional math helper functions.
cstesting
Package cstesting provides testing helpers
 Package cstesting provides testing helpers
diff
package diff compares two strings and prints out all differences
 package diff compares two strings and prints out all differences
errors
Package errors provides simple error handling primitives.
 Package errors provides simple error handling primitives.
hashpool
Package hashpool implements a pool for reusable and registered hash.Hash types.
 Package hashpool implements a pool for reusable and registered hash.Hash types.
magento
Package magento contains helper functions to handle the different versions.
 Package magento contains helper functions to handle the different versions.
naughtystrings
Package naughtystrings is a collection of strings for testing that have a high probability of causing issues when used as user input.
 Package naughtystrings is a collection of strings for testing that have a high probability of causing issues when used as user input.
naughtystrings/internal
null
Package null contains SQL types that consider zero input and null input as separate values, with convenient support for JSON and text marshaling.
 Package null contains SQL types that consider zero input and null input as separate values, with convenient support for JSON and text marshaling.
null/convert
null/zero
Package zero contains SQL types that consider zero input and null input to be equivalent with convenient support for JSON and text marshaling.
 Package zero contains SQL types that consider zero input and null input to be equivalent with convenient support for JSON and text marshaling.
php
Package php provides compatibility functions between Go and PHP.
 Package php provides compatibility functions between Go and PHP.
php/phpdate
Package phpdate converts a PHP date format into Go format.
 Package phpdate converts a PHP date format into Go format.
php/phpserialize
php/phpsession
Package phpsession provides possibility to decode/encode php session data in php_binary format.
 Package phpsession provides possibility to decode/encode php session data in php_binary format.
shortid
Package shortid enables the generation of short, unique, non-sequential and by default URL friendly Ids.
 Package shortid enables the generation of short, unique, non-sequential and by default URL friendly Ids.
slices
Package slices contains functions for working on specific slice types.
 Package slices contains functions for working on specific slice types.
sqlbeautifier
package sqlbeautifier formats a SQL string to a more human readable format.
 package sqlbeautifier formats a SQL string to a more human readable format.
sqlparser
sqlparser/dependency/bson
Package bson implements encoding and decoding of BSON objects.
 Package bson implements encoding and decoding of BSON objects.
sqlparser/dependency/bytes2
Package bytes2 provides alternate implementations of functionality similar to go's bytes package.
 Package bytes2 provides alternate implementations of functionality similar to go's bytes package.
sqlparser/dependency/hack
Package hack gives you some efficient functionality at the cost of breaking some Go rules.
 Package hack gives you some efficient functionality at the cost of breaking some Go rules.
sqlparser/dependency/sqltypes
Package sqltypes implements interfaces and types that represent SQL values.
 Package sqltypes implements interfaces and types that represent SQL values.
translit
Package translit replaces characters in a string using a huge conversion table.
 Package translit replaces characters in a string using a huge conversion table.
validation
Package validation provides abstract functions and options to allow validation of any struct or slice without reflection or struct fields.
 Package validation provides abstract functions and options to allow validation of any struct or slice without reflection or struct fields.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.