README
¶
CoreStore library WIP = Work in Progress
eCommerce library which is compatible to Magento 1 and 2 database schema.
Magento is a trademark of MAGENTO, INC..
Min. Go Version: 1.8
Usage
To properly use the CoreStore library some environment variables must be set
before running go generate. (TODO)
Required settings
CS_DSN the environment variable for the MySQL connection.
$ export CS_DSN='magento1:magento1@tcp(localhost:3306)/magento1'
$ export CS_DSN='magento2:magento2@tcp(localhost:3306)/magento2'
$ go get github.com/corestoreio/csfw
$ export CS_DSN='see previous'
$ cd $GOPATH/src/github.com/corestoreio/csfw
$ go run codegen/tableToStruct/*.go
Testing
Setup two databases. One for Magento 1 and one for Magento 2 and fill them with the provided testdata.
Create a DSN env var CS_DSN and point it to Magento 1 database. Run the tests.
Change the env var to let it point to Magento 2 database. Rerun the tests.
Finding Allocations
Side note: There is a new testing approach: TBDD = Test n' Benchmark Driven Development.
On the first run we got this result:
$ go test -run=😶 -bench=Benchmark_WithInitStoreByToken .
PASS
Benchmark_WithInitStoreByToken-4 100000 17297 ns/op 9112 B/op 203 allocs/op
ok github.com/corestoreio/csfw/store 2.569s
Quite shocking to use 203 allocs for just figuring out the current store view within a request.
Now compile your tests into an executable binary:
$ go test -c
This compilation reduces the noise in the below output trace log.
$ GODEBUG=allocfreetrace=1 ./store -test.run=😶 -test.bench=Benchmark_WithInitStoreByToken -test.benchtime=10ms 2>trace.log
Now open the trace.log file (around 26MB) and investigate all the allocations and refactor your code. Once finished you can achieve results like:
$ go test -run=NONE -bench=Benchmark_WithInitStoreByToken .
PASS
Benchmark_WithInitStoreByToken-4 2000000 826 ns/op 128 B/op 5 allocs/op
ok github.com/corestoreio/csfw/store 2.569s
Profiling
$ go test -cpuprofile=cpu.out -benchmem -memprofile=mem.out -run=NONE -bench=NameOfBenchmark -v
$ go tool pprof packageName.test cpu.out
Entering interactive mode (type "help" for commands)
(pprof) top5
560ms of 1540ms total (36.36%)
Showing top 5 nodes out of 112 (cum >= 60ms)
flat flat% sum% cum cum%
180ms 11.69% 11.69% 400ms 25.97% runtime.mallocgc
flatis how much time is spent inside of a function.cumshows how much time is spent in a function, and also in any code called by a function.
For memory profile:
Sample value selection option (for heap profiles):
-inuse_space Display in-use memory size
-inuse_objects Display in-use object counts
-alloc_space Display allocated memory size
-alloc_objects Display allocated object counts
$ go tool pprof -alloc_objects packageName.test mem.out
Bound Check Elimination
http://klauspost-talks.appspot.com/2016/go17-compiler.slide
$ go build -gcflags="-d=ssa/check_bce/debug=1" bounds.go
or
$ go test -gcflags="-d=ssa/check_bce/debug=1" .
Success - Check bounds outside the loop.
Running Benchmark
Assuming we have already an existing file called bm_baseline.txt.
$ go test -v -run=🤐 -bench=. -count=10 . > bm_baseline_new.txt
After running above command to generate the second benchmark statistics file we run:
$ benchstat bm_baseline.txt bm_baseline_new.txt
https://godoc.org/rsc.io/benchstat
Other development helpers
A preconfigured linter file lint has been included in this repoistory.
TODO
If you find an entry in the source like TODO(CS) that means you can ask CS
to get more information about how to implement and what to fix if the context of
the todo description isn't understandable.
- Create Magento 1+2 modules to setup test database and test Magento system.
Contributing
Please have a look at the contribution guidelines.
Acknowledgements
Some packages have been fully refactored but the initial idea has been extracted from the geniuses listed below:
| Name | Package | License |
|---|---|---|
| Steve Francia | util/conv | MIT Copyright (c) 2014 |
| Jonathan Novak, Tyler Smith, Michal Bohuslávek | dbr | The MIT License (MIT) 2014 |
| Martin Angers and Contributors. | ctxthrottled | The MIT License (MIT) 2014 |
| Dave Cheney | util/errors | The MIT License (MIT) 2015 |
| Jad Dittmar | finance aka. money | Copyright (c) 2011 |
| Wenbin Xiao | util/sqlparser | Copyright 2015 BSD Style |
| Google Inc | youtube/vitess\sqlparser | Copyright 2012 BSD Style |
| Olivier Poitrey | ctxmw.WithAccessLog & CORS | Copyright (c) 2014-2015 MIT License |
| Dave Grijalva | csjwt | Copyright (c) 2012 MIT License |
| Uber Technologies, Inc. | log | Copyright (c) 2016 MIT License |
| 2013 The Go Authors | singleflight | Copyright (c) 2013 BSD Style |
| Minio Cloud Storage, (C) 2016 Minio, Inc. | blake2b-simd | Apache License, Version 2.0 |
| Ventu.io, Oleg Sklyar, contributors. | util/shortid | MIT License Copyright (c) 2016, |
| Carl Jackson (carl@avtok.com) (Goji) | net/responseproxy | Copyright (c) 2014, 2015, 2016 |
| Greg Roseberry, 2014; Patrick O'Brien, 2016 | util/null | BSD Copyright (c) 2014, 2015, 2016 |
| The Go-MySQL-Driver Authors | util/null/time_mysql.go | Mozilla Public License, v. 2.0, Copyright 2012 |
| siddontang | storage/binlogsync | MIT Copyright (c) 2014 |
| siddontang | storage/myreplicator | MIT Copyright (c) 2014 |
Licensing
CoreStore is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.
Copyright
Documentation
¶
Overview ¶
Package csfw contains the CoreStore library based on Magento's database structure. 99% compatible to Magento 1 and 2.
The package csfw contains at the moment only go:generate commands to build Go code.
Two skeleton projects (monolith and SOA) are already setup but of course empty.
Purpose ¶
Why is someone trying to create a Magento database schema compatible online shop in Go?
Because performance :-)
Architecture ¶
@todo 10km view.
Names ¶
Generated SQL table structs start with the word "Table". The word "Slice" will be appended when there is a slice of structs.
Example for generated SQL table structs:
type (
// TableStoreSlice contains pointers to TableStore types
TableStoreSlice []*TableStore
// TableStore a type for the MySQL table core_store
TableStore struct {
...
}
)
Table indexes are iota constants and start with TableIndex[table name].
The word "collection" will be appended to a variable or function when that variable or function contains a materialized slice or handles it.
Event/Observer vs Publish/Subscribe
Basically those two are nearly the same. The only difference lies in its behaviour. Event/Observer can modify a type and block an operation while Publish/Subscribe runs asynchronously in its own Goroutine and cannot modify anything.
Required settings ¶
CS_DSN the environment variable for the MySQL connection.
$ export CS_DSN='magento1:magento1@tcp(localhost:3306)/magento1' $ export CS_DSN='magento2:magento2@tcp(localhost:3306)/magento2' $ go get github.com/corestoreio/csfw $ export CS_DSN_TEST='see next section' $ cd $GOPATH/src/github.com/corestoreio/csfw $ go generate ./...
Testing ¶
Setup two databases. One for Magento 1 and one for Magento 2 and fill them with the provided test data https://github.com/corestoreio/csfw/tree/master/testData
Create a DSN env var CS_DSN_TEST and point it to Magento 1 database. Run the tests. Change the env var to let it point to Magento 2 database. Rerun the tests.
$ export CS_DSN_TEST='magento1:magento1@tcp(localhost:3306)/magento1' $ export CS_DSN_TEST='magento2:magento2@tcp(localhost:3306)/magento2'
IDE ¶
Currently using the IntelliJ IDEA Community Edition with the https://github.com/go-lang-plugin-org/go-lang-idea-plugin plugin.
At the moment Q2/2015: There are no official jar files for downloading so the go lang plugin will be compiled on a daily basis. The plugin works very well! Kudos to those developers!
IDEA has been configured with goimports, gofmt, golint, govet and ... with the file watcher plugin.
Why am I not using vim? Because I would only generate passwords ;-|.
Contributing ¶
Please have a look at the contribution guidelines https://github.com/corestoreio/corestore/blob/master/CONTRIBUTING.md
Trademarks ¶
Magento is a trademark of MAGENTO, INC. http://www.magentocommerce.com/license/
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package backend provides common configuration and functions.
|
Package backend provides common configuration and functions. |
|
Package authorization enables management of access control list roles and rules.
|
Package authorization enables management of access control list roles and rules. |
|
xacml
Package xacml implements the OASIS/XACML standard for Policy-based authorization.
|
Package xacml implements the OASIS/XACML standard for Policy-based authorization. |
|
Package bundle handles the creation of catalog product bundles.
|
Package bundle handles the creation of catalog product bundles. |
|
Package catalog implements category tree and product features.
|
Package catalog implements category tree and product features. |
|
catattr
Package catattr handles all product and category related attributes.
|
Package catattr handles all product and category related attributes. |
|
Package cataloginventory handles product inventory and indexing
|
Package cataloginventory handles product inventory and indexing |
|
Package codegen generates Go code and is only used in development.
|
Package codegen generates Go code and is only used in development. |
|
formhandler
formhandler generates code to map URL query string or URL encoded form data to structs and vice versa.
|
formhandler generates code to map URL query string or URL encoded form data to structs and vice versa. |
|
localization
package main generates i18n names of currencies, countries because not yet implemented in text/language TODO.
|
package main generates i18n names of currencies, countries because not yet implemented in text/language TODO. |
|
localization/gen
Package gen contains common code for the various code generation tools in the text repository.
|
Package gen contains common code for the various code generation tools in the text repository. |
|
materialization
package main materializes attributes, sets, groups, entity types, stores, websites, etc.
|
package main materializes attributes, sets, groups, entity types, stores, websites, etc. |
|
tableToStruct
package main generates Go structs, slices and function receivers from SQL tables.
|
package main generates Go structs, slices and function receivers from SQL tables. |
|
tableToStruct/codecgen
codecgen generates codec.Selfer implementations for a set of types.
|
codecgen generates codec.Selfer implementations for a set of types. |
|
Package config handles the configuration and its scopes.
|
Package config handles the configuration and its scopes. |
|
cfgmock
Package cfgmock provides mock types and functions for the config.Service type.
|
Package cfgmock provides mock types and functions for the config.Service type. |
|
cfgmodel
Package cfgmodel provides types to get/set values of a configuration.
|
Package cfgmodel provides types to get/set values of a configuration. |
|
cfgpath
Package cfgpath handles the configuration paths.
|
Package cfgpath handles the configuration paths. |
|
cfgsource
Package source provides a slice type for handling config sources to be used in config/models.
|
Package source provides a slice type for handling config sources to be used in config/models. |
|
element
Package element represents Magento system.xml configuration elements.
|
Package element represents Magento system.xml configuration elements. |
|
storage
Package storage defines the available configuration storage engines in its subpackages.
|
Package storage defines the available configuration storage engines in its subpackages. |
|
storage/boltdb
Package boltdb uses the bolt database for reading and writing configuration paths.
|
Package boltdb uses the bolt database for reading and writing configuration paths. |
|
storage/ccd
Package ccd = core_config_data uses the MySQL based table core_config_data for reading and writing configuration paths, scopes and values.
|
Package ccd = core_config_data uses the MySQL based table core_config_data for reading and writing configuration paths, scopes and values. |
|
storage/cfgbigcache
Package cfgbigcache uses the bigcache in-memory database for reading and writing configuration paths.
|
Package cfgbigcache uses the bigcache in-memory database for reading and writing configuration paths. |
|
storage/etcd
Package etcd uses etcd service for reading and writing configuration paths.
|
Package etcd uses etcd service for reading and writing configuration paths. |
|
Package customer handles a customer entity with its addresses.
|
Package customer handles a customer entity with its addresses. |
|
custattr
Package custattr handles all customer and address related attributes.
|
Package custattr handles all customer and address related attributes. |
|
custconfig
Package custconfig handles configuration structure and its models.
|
Package custconfig handles configuration structure and its models. |
|
Package directory provides features for currencies, currency rates, conversion of prices to a specified currency format, countries and regions.
|
Package directory provides features for currencies, currency rates, conversion of prices to a specified currency format, countries and regions. |
|
Package eav contains the logic for the Entity-Attribute-Value model based on the Magento database schema.
|
Package eav contains the logic for the Entity-Attribute-Value model based on the Magento database schema. |
|
Package mail provides functions and services for sending html or text emails via encrypted or unencrypted connections.
|
Package mail provides functions and services for sending html or text emails via encrypted or unencrypted connections. |
|
Package i18n supports string translations with variable substitution, CLDR pluralization, currency, formats, language, regions and timezones.
|
Package i18n supports string translations with variable substitution, CLDR pluralization, currency, formats, language, regions and timezones. |
|
Package locale provides locale specific services like formatting or translating.
|
Package locale provides locale specific services like formatting or translating. |
|
Package log contains BlackHole, StdLog, Log15 and the Logger interface.
|
Package log contains BlackHole, StdLog, Log15 and the Logger interface. |
|
http
Package http creates log fields for http Requests and Responses.
|
Package http creates log fields for http Requests and Responses. |
|
log15w
Package log15w provides a wrapper (w) to github.com/inconshreveable/log15.
|
Package log15w provides a wrapper (w) to github.com/inconshreveable/log15. |
|
logw
Package logw provides a wrapper (w) to Go's standard logger
|
Package logw provides a wrapper (w) to Go's standard logger |
|
zapw
Package zapw provides a wrapper (w) to github.com/uber-go/zap.
|
Package zapw provides a wrapper (w) to github.com/uber-go/zap. |
|
Package net provides additional network helper functions and in subpackages middleware.
|
Package net provides additional network helper functions and in subpackages middleware. |
|
auth
Package auth provides authentication middleware.
|
Package auth provides authentication middleware. |
|
auth/backendauth
Package backendauth (TODO) defines the backend configuration options and element slices.
|
Package backendauth (TODO) defines the backend configuration options and element slices. |
|
cors
Package cors provides a middleware for Cross-origin resource sharing (CORS).
|
Package cors provides a middleware for Cross-origin resource sharing (CORS). |
|
cors/backendcors
Package backendcors defines the backend configuration options and element slices.
|
Package backendcors defines the backend configuration options and element slices. |
|
csrf
Package csrf implements scope based Cross-Site Request Forgery protection.
|
Package csrf implements scope based Cross-Site Request Forgery protection. |
|
geoip
Package geoip detects the country by an IP address and provides alternative handlers.
|
Package geoip detects the country by an IP address and provides alternative handlers. |
|
geoip/backendgeoip
Package backendgeoip defines the backend configuration options and element slices.
|
Package backendgeoip defines the backend configuration options and element slices. |
|
geoip/maxmindfile
Package maxmindfile provides an OptionFactoryFunc for the backendgeopip package.
|
Package maxmindfile provides an OptionFactoryFunc for the backendgeopip package. |
|
geoip/maxmindwebservice
Package maxmindwebservice provides an OptionFactoryFunc for the backendgeopip package.
|
Package maxmindwebservice provides an OptionFactoryFunc for the backendgeopip package. |
|
httputil
Package httputil provides general functions for http handling (refactor) TODO(cs) => github.com/streadway/handy
|
Package httputil provides general functions for http handling (refactor) TODO(cs) => github.com/streadway/handy |
|
jwt
Package jwt provides a middleware for JSON web token authentication and runMode initialization.
|
Package jwt provides a middleware for JSON web token authentication and runMode initialization. |
|
jwt/backendjwt
Package backendjwt defines the backend configuration options and element slices.
|
Package backendjwt defines the backend configuration options and element slices. |
|
mw
Package mw provides a various middleware.
|
Package mw provides a various middleware. |
|
ratelimit
Package ratelimit implements scope based HTTP rate limiting.
|
Package ratelimit implements scope based HTTP rate limiting. |
|
request
Package request provides functions for http request handling.
|
Package request provides functions for http request handling. |
|
response
Package response provides functions for working with an http response.
|
Package response provides functions for working with an http response. |
|
responseproxy
Package responseproxy provides various proxy functions for extending http.ResponseWriter.
|
Package responseproxy provides various proxy functions for extending http.ResponseWriter. |
|
runmode
Package runmode defines store specific middleware to initialize the scope and its ID per request.
|
Package runmode defines store specific middleware to initialize the scope and its ID per request. |
|
secure
Package secure adds a middleware for quick security wins to response HTTP headers TODO(cs) https://github.com/unrolled/secure/blob/v1/secure.go
|
Package secure adds a middleware for quick security wins to response HTTP headers TODO(cs) https://github.com/unrolled/secure/blob/v1/secure.go |
|
signed
Package signed provides a middleware to sign responses and validate requests.
|
Package signed provides a middleware to sign responses and validate requests. |
|
url
Package url parses program specific URLs and provides helper functions.
|
Package url parses program specific URLs and provides helper functions. |
|
Package payment defines a payment processing library which can connect to payment gateways.
|
Package payment defines a payment processing library which can connect to payment gateways. |
|
Package storage provides everything from MySQL, Redis, BoltDB, file, etc functions.
|
Package storage provides everything from MySQL, Redis, BoltDB, file, etc functions. |
|
containable
Package containable implements a container using a key with an expiration.
|
Package containable implements a container using a key with an expiration. |
|
csdb
Package csdb implements MySQL helper for tables, columns, statements, replication, validation and DB variables.
|
Package csdb implements MySQL helper for tables, columns, statements, replication, validation and DB variables. |
|
dbr
Package dbr has additions to Go's database/sql for super fast performance and convenience.
|
Package dbr has additions to Go's database/sql for super fast performance and convenience. |
|
money
Package money uses a fixed-length guard for precision arithmetic.
|
Package money uses a fixed-length guard for precision arithmetic. |
|
myreplicator
Package myreplicator handles the MySQL binary replication protocol.
|
Package myreplicator handles the MySQL binary replication protocol. |
|
text
Package text represents a []byte type for storing long text strings (Chars).
|
Package text represents a []byte type for storing long text strings (Chars). |
|
transcache
Package transcache transcodes arbitrary Go types to bytes and stores them in a cache reducing GC.
|
Package transcache transcodes arbitrary Go types to bytes and stores them in a cache reducing GC. |
|
transcache/tcbigcache
Package tcbigcache adapter for the transcache package to use an in-memory cache.
|
Package tcbigcache adapter for the transcache package to use an in-memory cache. |
|
transcache/tcboltdb
Package tcboltdb adapter for the boltdb to use a file based cache.
|
Package tcboltdb adapter for the boltdb to use a file based cache. |
|
transcache/tcredis
Package tcredis implements a wrapper for the Redis server.
|
Package tcredis implements a wrapper for the Redis server. |
|
Package store implements the handling of websites, groups and stores.
|
Package store implements the handling of websites, groups and stores. |
|
scope
Package scope defines the configuration of scopes default, website, group and store.
|
Package scope defines the configuration of scopes default, website, group and store. |
|
storemock
Package mock implements mocking of the store.Service for tests.
|
Package mock implements mocking of the store.Service for tests. |
|
Package sync provides tools to deal with goroutines.
|
Package sync provides tools to deal with goroutines. |
|
singleflight
Package singleflight provides a duplicate function call suppression mechanism.
|
Package singleflight provides a duplicate function call suppression mechanism. |
|
suspend
Package suspend provides a complicated duplicate function call suppression mechanism.
|
Package suspend provides a complicated duplicate function call suppression mechanism. |
|
Package tax takes care of all kind of taxes.
|
Package tax takes care of all kind of taxes. |
|
Package user takes care of backend/API users.
|
Package user takes care of backend/API users. |
|
Package util should stay empty.
|
Package util should stay empty. |
|
bufferpool
Package bufferpool implements a pool of reusable auto-resizing buffers.
|
Package bufferpool implements a pool of reusable auto-resizing buffers. |
|
conv
Package conv - easy and safe casting from one type to another.
|
Package conv - easy and safe casting from one type to another. |
|
csjwt
Package csjwt handles JSON web tokens.
|
Package csjwt handles JSON web tokens. |
|
csjwt/cmd/jwt
A useful example app.
|
A useful example app. |
|
csjwt/jwtclaim
Package jwtclaim provides claim structs and maps for convenience.
|
Package jwtclaim provides claim structs and maps for convenience. |
|
csmath
Package csmath provides additional math helper functions.
|
Package csmath provides additional math helper functions. |
|
cstesting
Package cstesting provides testing helpers
|
Package cstesting provides testing helpers |
|
diff
package diff compares two strings and prints out all differences
|
package diff compares two strings and prints out all differences |
|
errors
Package errors provides simple error handling primitives.
|
Package errors provides simple error handling primitives. |
|
hashpool
Package hashpool implements a pool for reusable and registered hash.Hash types.
|
Package hashpool implements a pool for reusable and registered hash.Hash types. |
|
magento
Package magento contains helper functions to handle the different versions.
|
Package magento contains helper functions to handle the different versions. |
|
naughtystrings
Package naughtystrings is a collection of strings for testing that have a high probability of causing issues when used as user input.
|
Package naughtystrings is a collection of strings for testing that have a high probability of causing issues when used as user input. |
|
null
Package null contains SQL types that consider zero input and null input as separate values, with convenient support for JSON and text marshaling.
|
Package null contains SQL types that consider zero input and null input as separate values, with convenient support for JSON and text marshaling. |
|
null/zero
Package zero contains SQL types that consider zero input and null input to be equivalent with convenient support for JSON and text marshaling.
|
Package zero contains SQL types that consider zero input and null input to be equivalent with convenient support for JSON and text marshaling. |
|
php
Package php provides compatibility functions between Go and PHP.
|
Package php provides compatibility functions between Go and PHP. |
|
php/phpdate
Package phpdate converts a PHP date format into Go format.
|
Package phpdate converts a PHP date format into Go format. |
|
php/phpsession
Package phpsession provides possibility to decode/encode php session data in php_binary format.
|
Package phpsession provides possibility to decode/encode php session data in php_binary format. |
|
shortid
Package shortid enables the generation of short, unique, non-sequential and by default URL friendly Ids.
|
Package shortid enables the generation of short, unique, non-sequential and by default URL friendly Ids. |
|
slices
Package slices contains functions for working on specific slice types.
|
Package slices contains functions for working on specific slice types. |
|
sqlbeautifier
package sqlbeautifier formats a SQL string to a more human readable format.
|
package sqlbeautifier formats a SQL string to a more human readable format. |
|
sqlparser/dependency/bson
Package bson implements encoding and decoding of BSON objects.
|
Package bson implements encoding and decoding of BSON objects. |
|
sqlparser/dependency/bytes2
Package bytes2 provides alternate implementations of functionality similar to go's bytes package.
|
Package bytes2 provides alternate implementations of functionality similar to go's bytes package. |
|
sqlparser/dependency/hack
Package hack gives you some efficient functionality at the cost of breaking some Go rules.
|
Package hack gives you some efficient functionality at the cost of breaking some Go rules. |
|
sqlparser/dependency/sqltypes
Package sqltypes implements interfaces and types that represent SQL values.
|
Package sqltypes implements interfaces and types that represent SQL values. |
|
translit
Package translit replaces characters in a string using a huge conversion table.
|
Package translit replaces characters in a string using a huge conversion table. |
|
validation
Package validation provides abstract functions and options to allow validation of any struct or slice without reflection or struct fields.
|
Package validation provides abstract functions and options to allow validation of any struct or slice without reflection or struct fields. |
Click to show internal directories.
Click to hide internal directories.