cia_world_factbook_api

command module

v0.0.0-...-23920af Latest Latest Go to latest Published: Dec 2, 2020 License: MIT Imports: 11 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

Repository

github.com/iancoleman/cia_world_factbook_api

Links

Open Source Insights

README ¶

CIA World Factbook API

Converts the CIA World Factbook into a json data structure.

Data

Latest - approx 3 MB - updated monthly
Historical - approx 40 MB - last updated 2020-09-27
Html Archives - approx 400 MB - last updated 2020-09-27

Usage

If you just want the latest data, get it using the Latest link above.

If you also want to get the full historical data set, use the Historical link above.

If you want to parse the factbook html into json for yourself:

clone this repository to your local machine.
download the Html Archives above.
edit config.json with the paths to your downloaded html archives.
run go run parse_html_to_json.go to convert each country html to a json structure.
run go run create_weekly_json_files.go to combine each individual country into a week-by-week data file.

If you want to fetch the html files yourself and then parse them:

clone this repository to your local machine.
edit config.json with the paths to use for the downloaded html archives.
run python fetch.py to fetch the historical html files from archive.org (will take several days).
run go run parse_html_to_json.go to convert each country html to a json structure.
run go run create_weekly_json_files.go to combine each individual country into a week-by-week data file.

Tests

clone this repository to your local machine.
cd cwf/src/country
go test

Contributing

Contributions are most welcome.

Reporting Issues

Please report issues using the Issues tab at the top of this page.

Pull Requests

If you modify the code please submit a pull request for review.

Most of the parsing logic is in src/country in the files page.go and string_conversions.go.

If the parser is modified, please update the VERSION contant in country/page.go.

License

MIT - see LICENSE

Documentation ¶

There is no documentation for this package.

Source Files ¶

View all Source files

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
src
country
github.com/PuerkitoBio/goquery Package goquery implements features similar to jQuery, including the chainable syntax, to manipulate and query an HTML document.	Package goquery implements features similar to jQuery, including the chainable syntax, to manipulate and query an HTML document.
github.com/andybalholm/cascadia Package cascadia is an implementation of CSS selectors.	Package cascadia is an implementation of CSS selectors.
github.com/andybalholm/cascadia/fuzz
golang.org/x/net/bpf Package bpf implements marshaling and unmarshaling of programs for the Berkeley Packet Filter virtual machine, and provides a Go implementation of the virtual machine.	Package bpf implements marshaling and unmarshaling of programs for the Berkeley Packet Filter virtual machine, and provides a Go implementation of the virtual machine.
golang.org/x/net/context Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.	Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
golang.org/x/net/context/ctxhttp Package ctxhttp provides helper functions for performing context-aware HTTP requests.	Package ctxhttp provides helper functions for performing context-aware HTTP requests.
golang.org/x/net/dict Package dict implements the Dictionary Server Protocol as defined in RFC 2229.	Package dict implements the Dictionary Server Protocol as defined in RFC 2229.
golang.org/x/net/dns/dnsmessage Package dnsmessage provides a mostly RFC 1035 compliant implementation of DNS message packing and unpacking.	Package dnsmessage provides a mostly RFC 1035 compliant implementation of DNS message packing and unpacking.
golang.org/x/net/html Package html implements an HTML5-compliant tokenizer and parser.	Package html implements an HTML5-compliant tokenizer and parser.
golang.org/x/net/html/atom Package atom provides integer codes (also known as atoms) for a fixed set of frequently occurring HTML strings: tag names and attribute keys such as "p" and "id".	Package atom provides integer codes (also known as atoms) for a fixed set of frequently occurring HTML strings: tag names and attribute keys such as "p" and "id".
golang.org/x/net/html/charset Package charset provides common text encodings for HTML documents.	Package charset provides common text encodings for HTML documents.
golang.org/x/net/http2 Package http2 implements the HTTP/2 protocol.	Package http2 implements the HTTP/2 protocol.
golang.org/x/net/http2/h2i The h2i command is an interactive HTTP/2 console.	The h2i command is an interactive HTTP/2 console.
golang.org/x/net/http2/hpack Package hpack implements HPACK, a compression format for efficiently representing HTTP header fields in the context of HTTP/2.	Package hpack implements HPACK, a compression format for efficiently representing HTTP header fields in the context of HTTP/2.
golang.org/x/net/icmp Package icmp provides basic functions for the manipulation of messages used in the Internet Control Message Protocols, ICMPv4 and ICMPv6.	Package icmp provides basic functions for the manipulation of messages used in the Internet Control Message Protocols, ICMPv4 and ICMPv6.
golang.org/x/net/idna Package idna implements IDNA2008 (Internationalized Domain Names for Applications), defined in RFC 5890, RFC 5891, RFC 5892, RFC 5893 and RFC 5894.	Package idna implements IDNA2008 (Internationalized Domain Names for Applications), defined in RFC 5890, RFC 5891, RFC 5892, RFC 5893 and RFC 5894.
golang.org/x/net/internal/iana Package iana provides protocol number resources managed by the Internet Assigned Numbers Authority (IANA).	Package iana provides protocol number resources managed by the Internet Assigned Numbers Authority (IANA).
golang.org/x/net/internal/netreflect
golang.org/x/net/internal/nettest Package nettest provides utilities for network testing.	Package nettest provides utilities for network testing.
golang.org/x/net/internal/timeseries Package timeseries implements a time series structure for stats collection.	Package timeseries implements a time series structure for stats collection.
golang.org/x/net/ipv4 Package ipv4 implements IP-level socket options for the Internet Protocol version 4.	Package ipv4 implements IP-level socket options for the Internet Protocol version 4.
golang.org/x/net/ipv6 Package ipv6 implements IP-level socket options for the Internet Protocol version 6.	Package ipv6 implements IP-level socket options for the Internet Protocol version 6.
golang.org/x/net/lex/httplex Package httplex contains rules around lexical matters of various HTTP-related specifications.	Package httplex contains rules around lexical matters of various HTTP-related specifications.
golang.org/x/net/nettest Package nettest provides utilities for network testing.	Package nettest provides utilities for network testing.
golang.org/x/net/netutil Package netutil provides network utility functions, complementing the more common ones in the net package.	Package netutil provides network utility functions, complementing the more common ones in the net package.
golang.org/x/net/proxy Package proxy provides support for a variety of protocols to proxy network data.	Package proxy provides support for a variety of protocols to proxy network data.
golang.org/x/net/publicsuffix Package publicsuffix provides a public suffix list based on data from http://publicsuffix.org/.	Package publicsuffix provides a public suffix list based on data from http://publicsuffix.org/.
golang.org/x/net/route Package route provides basic functions for the manipulation of packet routing facilities on BSD variants.	Package route provides basic functions for the manipulation of packet routing facilities on BSD variants.
golang.org/x/net/trace Package trace implements tracing of requests and long-lived objects.	Package trace implements tracing of requests and long-lived objects.
golang.org/x/net/webdav Package webdav provides a WebDAV server implementation.	Package webdav provides a WebDAV server implementation.
golang.org/x/net/webdav/internal/xml Package xml implements a simple XML 1.0 parser that understands XML name spaces.	Package xml implements a simple XML 1.0 parser that understands XML name spaces.
golang.org/x/net/websocket Package websocket implements a client and server for the WebSocket protocol as specified in RFC 6455.	Package websocket implements a client and server for the WebSocket protocol as specified in RFC 6455.
golang.org/x/net/xsrftoken Package xsrftoken provides methods for generating and validating secure XSRF tokens.	Package xsrftoken provides methods for generating and validating secure XSRF tokens.
logger
orderedmap
scraper
strcase Package strcase converts strings to snake_case or CamelCase	Package strcase converts strings to snake_case or CamelCase