cia_world_factbook_api

command module
v0.0.0-...-23920af Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 2, 2020 License: MIT Imports: 11 Imported by: 0

README

CIA World Factbook API

Converts the CIA World Factbook into a json data structure.

Data

  • Latest - approx 3 MB - updated monthly
  • Historical - approx 40 MB - last updated 2020-09-27
  • Html Archives - approx 400 MB - last updated 2020-09-27

Usage

If you just want the latest data, get it using the Latest link above.

If you also want to get the full historical data set, use the Historical link above.

If you want to parse the factbook html into json for yourself:

  • clone this repository to your local machine.
  • download the Html Archives above.
  • edit config.json with the paths to your downloaded html archives.
  • run go run parse_html_to_json.go to convert each country html to a json structure.
  • run go run create_weekly_json_files.go to combine each individual country into a week-by-week data file.

If you want to fetch the html files yourself and then parse them:

  • clone this repository to your local machine.
  • edit config.json with the paths to use for the downloaded html archives.
  • run python fetch.py to fetch the historical html files from archive.org (will take several days).
  • run go run parse_html_to_json.go to convert each country html to a json structure.
  • run go run create_weekly_json_files.go to combine each individual country into a week-by-week data file.

Tests

  • clone this repository to your local machine.
  • cd cwf/src/country
  • go test

Contributing

Contributions are most welcome.

Reporting Issues

Please report issues using the Issues tab at the top of this page.

Pull Requests

If you modify the code please submit a pull request for review.

Most of the parsing logic is in src/country in the files page.go and string_conversions.go.

If the parser is modified, please update the VERSION contant in country/page.go.

License

MIT - see LICENSE

Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis
src
github.com/PuerkitoBio/goquery
Package goquery implements features similar to jQuery, including the chainable syntax, to manipulate and query an HTML document.
Package goquery implements features similar to jQuery, including the chainable syntax, to manipulate and query an HTML document.
github.com/andybalholm/cascadia
Package cascadia is an implementation of CSS selectors.
Package cascadia is an implementation of CSS selectors.
golang.org/x/net/bpf
Package bpf implements marshaling and unmarshaling of programs for the Berkeley Packet Filter virtual machine, and provides a Go implementation of the virtual machine.
Package bpf implements marshaling and unmarshaling of programs for the Berkeley Packet Filter virtual machine, and provides a Go implementation of the virtual machine.
golang.org/x/net/context
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
golang.org/x/net/context/ctxhttp
Package ctxhttp provides helper functions for performing context-aware HTTP requests.
Package ctxhttp provides helper functions for performing context-aware HTTP requests.
golang.org/x/net/dict
Package dict implements the Dictionary Server Protocol as defined in RFC 2229.
Package dict implements the Dictionary Server Protocol as defined in RFC 2229.
golang.org/x/net/dns/dnsmessage
Package dnsmessage provides a mostly RFC 1035 compliant implementation of DNS message packing and unpacking.
Package dnsmessage provides a mostly RFC 1035 compliant implementation of DNS message packing and unpacking.
golang.org/x/net/html
Package html implements an HTML5-compliant tokenizer and parser.
Package html implements an HTML5-compliant tokenizer and parser.
golang.org/x/net/html/atom
Package atom provides integer codes (also known as atoms) for a fixed set of frequently occurring HTML strings: tag names and attribute keys such as "p" and "id".
Package atom provides integer codes (also known as atoms) for a fixed set of frequently occurring HTML strings: tag names and attribute keys such as "p" and "id".
golang.org/x/net/html/charset
Package charset provides common text encodings for HTML documents.
Package charset provides common text encodings for HTML documents.
golang.org/x/net/http2
Package http2 implements the HTTP/2 protocol.
Package http2 implements the HTTP/2 protocol.
golang.org/x/net/http2/h2i
The h2i command is an interactive HTTP/2 console.
The h2i command is an interactive HTTP/2 console.
golang.org/x/net/http2/hpack
Package hpack implements HPACK, a compression format for efficiently representing HTTP header fields in the context of HTTP/2.
Package hpack implements HPACK, a compression format for efficiently representing HTTP header fields in the context of HTTP/2.
golang.org/x/net/icmp
Package icmp provides basic functions for the manipulation of messages used in the Internet Control Message Protocols, ICMPv4 and ICMPv6.
Package icmp provides basic functions for the manipulation of messages used in the Internet Control Message Protocols, ICMPv4 and ICMPv6.
golang.org/x/net/idna
Package idna implements IDNA2008 (Internationalized Domain Names for Applications), defined in RFC 5890, RFC 5891, RFC 5892, RFC 5893 and RFC 5894.
Package idna implements IDNA2008 (Internationalized Domain Names for Applications), defined in RFC 5890, RFC 5891, RFC 5892, RFC 5893 and RFC 5894.
golang.org/x/net/internal/iana
Package iana provides protocol number resources managed by the Internet Assigned Numbers Authority (IANA).
Package iana provides protocol number resources managed by the Internet Assigned Numbers Authority (IANA).
golang.org/x/net/internal/nettest
Package nettest provides utilities for network testing.
Package nettest provides utilities for network testing.
golang.org/x/net/internal/timeseries
Package timeseries implements a time series structure for stats collection.
Package timeseries implements a time series structure for stats collection.
golang.org/x/net/ipv4
Package ipv4 implements IP-level socket options for the Internet Protocol version 4.
Package ipv4 implements IP-level socket options for the Internet Protocol version 4.
golang.org/x/net/ipv6
Package ipv6 implements IP-level socket options for the Internet Protocol version 6.
Package ipv6 implements IP-level socket options for the Internet Protocol version 6.
golang.org/x/net/lex/httplex
Package httplex contains rules around lexical matters of various HTTP-related specifications.
Package httplex contains rules around lexical matters of various HTTP-related specifications.
golang.org/x/net/nettest
Package nettest provides utilities for network testing.
Package nettest provides utilities for network testing.
golang.org/x/net/netutil
Package netutil provides network utility functions, complementing the more common ones in the net package.
Package netutil provides network utility functions, complementing the more common ones in the net package.
golang.org/x/net/proxy
Package proxy provides support for a variety of protocols to proxy network data.
Package proxy provides support for a variety of protocols to proxy network data.
golang.org/x/net/publicsuffix
Package publicsuffix provides a public suffix list based on data from http://publicsuffix.org/.
Package publicsuffix provides a public suffix list based on data from http://publicsuffix.org/.
golang.org/x/net/route
Package route provides basic functions for the manipulation of packet routing facilities on BSD variants.
Package route provides basic functions for the manipulation of packet routing facilities on BSD variants.
golang.org/x/net/trace
Package trace implements tracing of requests and long-lived objects.
Package trace implements tracing of requests and long-lived objects.
golang.org/x/net/webdav
Package webdav provides a WebDAV server implementation.
Package webdav provides a WebDAV server implementation.
golang.org/x/net/webdav/internal/xml
Package xml implements a simple XML 1.0 parser that understands XML name spaces.
Package xml implements a simple XML 1.0 parser that understands XML name spaces.
golang.org/x/net/websocket
Package websocket implements a client and server for the WebSocket protocol as specified in RFC 6455.
Package websocket implements a client and server for the WebSocket protocol as specified in RFC 6455.
golang.org/x/net/xsrftoken
Package xsrftoken provides methods for generating and validating secure XSRF tokens.
Package xsrftoken provides methods for generating and validating secure XSRF tokens.
strcase
Package strcase converts strings to snake_case or CamelCase
Package strcase converts strings to snake_case or CamelCase

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL