goebaykleinanzeigen

README

goebaykleinanzeigen

Disclaimer

This project is in early development and will be buggy.

I am not associated with eBay Kleinanzeigen or eBay in any way.

Use this code at your own risk.

Introduction

goebaykleinanzeigen is an experimental library for scraping eBay Kleinanzeigens data written in go.

eBay Kleinanzeigen should not be confused with the main site eBay.

There is a runnable basic example for scraping car ads in the cmd/simple directory.

This library is written with the intend to scrape car listings but all categories should work and will be added in the future.

Why webscraping?

eBay Kleinanzeigen offers a private API which is only available to their official partners. If you need to use this library you are probably not a official partner.

Sadly this makes accessing eBay Kleinanzeigens data a little lot harder. Especially since we are limited to about 40 req/minute. And of course parsing HTML is pretty error prone.

eBay Kleinanzeigen URLs

In the following we will discuss the relevant URLs for our little scraper

The search URL

First we need to list all the ads.

The basic URL for this may look like this: https://www.ebay-kleinanzeigen.de/l3331r50. Here we can split up the parameter in 2 parts: l3331 and r50

• Where the 3331 in l3331 is some kind of internal location id for Berlin.
• Where the 50 in r50 is the distance from the center of the location in kilometers.

This is pretty boring and generic so lets specify a specific category in our URL: https://www.ebay-kleinanzeigen.de/l3331r50c216

• We added the parameter c216 to our URL! In this case c is probably a prefix for CategoryID and 216 is the ID for cars!

So now we only search for cars but this is still a little bit to generic if you ask me. We don't want to spend too much money on our new car so let's add a specific price range to our query: https://www.ebay-kleinanzeigen.de/preis:1000:4000/l3331r50c216

• Now we only search for cars in the specific price range from 1000 euro to 4000 euro (preis is the german word for price).

But we don't want a Volkswagen, we only want to search for BMWs. Luckily we can tell eBay to only look for BMWs! Our URL now looks like this: https://www.ebay-kleinanzeigen.de/preis:1000:4000/l3331r50c216+autos.marke_s:bmw.

• Options specific for a category are appended in the following format: +option_name:option_value

You don't really need to worry about this since the `SearchParam´ struct abstracts the query building away. The SearchParam for this example may look like this:

param := &goebaykleinanzeigen.SearchParam{
		Category:  goebaykleinanzeigen.Cars,
		Location: "3331",
		Radius:    goebaykleinanzeigen.FiftyKM,
		PriceFrom: 1000,
		PriceTo:   4000,
		SpecificParameter: map[goebaykleinanzeigen.ParamName]string{
			goebaykleinanzeigen.CarManufacturer: "bmw",
		},
	}

More specific parameters will be added to specificparams.go in the future.

The ad URL

This is pretty straight forward. The URL for an ad looks like this: https://www.ebay-kleinanzeigen.de/s-anzeige/{ADID}

LocationID for a Location

The internal ID for a location can be fetched by this URL (Berlin in this case): https://www.ebay-kleinanzeigen.de/s-ort-empfehlungen.json?query=Berlin

Ratelimit

Currently the max seems to be about 40 req/minute. This is currently not enforced by the library

Localisation

Keep in mind that eBay Kleinanzeigen is a German Site so everything will be in German.

Prices are integers in euro there are no cents on eBay Kleinanzeigen(? I did not confirm this).

Documentation

Overview

Package goebaykleinanzeigen exposes a webscraping interface for eBay Kleinanzeigen

Index

• type AdItem
• type AdListItem
• type AdListRepo
  • func NewAdListRepo(client *http.Client) *AdListRepo
  • func (al *AdListRepo) Fetch(ctx context.Context, param *SearchParam) (*AdListResponse, error)
• type AdListResponse
• type AdRepo
  • func NewAdRepo(client *http.Client) *AdRepo
  • func (ar *AdRepo) Fetch(ctx context.Context, url string) (*AdItem, error)
• type Category
• type Detail
• type LocationID
• type OfferType
• type ParamName
• type Provider
• type Radius
• type SearchParam
  • func ParamsFromJSON(data io.Reader) (*SearchParam, error)
  • func (sp *SearchParam) ToJSON(out io.Writer) error
• type Seller
• type SpecificParameter

Types

type AdItem

type AdItem struct {
	ID              string    `json:"id"`
	ListedSince     time.Time `json:"listed_since"`
	Title           string    `json:"title"`
	Price           int       `json:"price"`
	PriceNegotiable bool      `json:"price_negotiable"`
	Location        string    `json:"location"`
	ZipCode         string    `json:"zip_code"`
	Link            string    `json:"link"`
	Description     string    `json:"description"`
	Details         []*Detail `json:"details"`
	Extras          []string  `json:"extras"`
	Seller          *Seller   `json:"seller"`
}

AdItem represents a single AdItem

type AdListItem

type AdListItem struct {
	ID              string `json:"id"`
	Title           string `json:"title"`
	Price           int    `json:"price"`
	PriceNegotiable bool   `json:"price_negotiable"`
	Location        string `json:"location"`
	ZipCode         string `json:"zip_code"`
	Link            string `json:"link"`
}

AdListItem a single item in the returned list

type AdListRepo

type AdListRepo struct {
	// contains filtered or unexported fields
}

AdListRepo represents an AdListRepo The AdListRepo resolves a list of ads

func NewAdListRepo

func NewAdListRepo(client *http.Client) *AdListRepo

NewAdListRepo creates a new AdRepo, if client is nil, one will be created

func (*AdListRepo) Fetch

func (al *AdListRepo) Fetch(ctx context.Context, param *SearchParam) (*AdListResponse, error)

Fetch fetches a list of ads based on the provided param

type AdListResponse

type AdListResponse struct {
	Items      []*AdListItem
	IsLastPage bool
}

AdListResponse represents the response from the Fetch() call

type AdRepo

type AdRepo struct {
	// contains filtered or unexported fields
}

AdRepo represents an AdRepo The AdRepo resolves a single ad

func NewAdRepo

func NewAdRepo(client *http.Client) *AdRepo

NewAdRepo creates a new AdRepo, if client is nil, one will be created

func (*AdRepo) Fetch

func (ar *AdRepo) Fetch(ctx context.Context, url string) (*AdItem, error)

Fetch fetches a single AdItem

type Category

type Category string

Category represents a specific category id

const (
	Family         Category = "17"
	Music          Category = "73"
	HouseAndGarden Category = "80"
	Jobs           Category = "102"
	Pets           Category = "130"
	Fashion        Category = "153"
	Electronics    Category = "161"
	Hobby          Category = "185"
	Property       Category = "195"
	Boats          Category = "211"
	Cars           Category = "216"
	CarParts       Category = "223"
	Tickets        Category = "231"
	Education      Category = "235"
	Services       Category = "297"
)

type Detail

type Detail struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}

type LocationID

type LocationID string

LocationID represents a specifc location retrived from https://www.ebay-kleinanzeigen.de/s-ort-empfehlungen.json?query=Berlin for example

type OfferType

type OfferType string

OfferType represents the offer type

const (
	Offer  OfferType = "angebote"
	Wanted OfferType = "gesuche"
)

type ParamName

type ParamName string

ParamName represents a parameter name for a category specific parameter

const (
	CarManufacturer       ParamName = "autos.marke_s"
	CarModel              ParamName = "autos.model_s"
	CarKM                 ParamName = "autos.km_i"
	CarYearOfRegistration ParamName = "autos.ez_i"
	CarHP                 ParamName = "autos.power_i"
	CarTUEV               ParamName = "autos.tuevy_i"
)

type Provider

type Provider string

Provider represents a specific provider (private or commercial seller)

const (
	Private    Provider = "privat"
	Commercial Provider = "gewerblich"
)

type Radius

type Radius string

Radius represents the distance from the center of the location

const (
	WholePlace        Radius = ""
	FiveKM            Radius = "5"
	TenKM             Radius = "10"
	TwentyKM          Radius = "20"
	ThirtyKM          Radius = "30"
	FiftyKM           Radius = "50"
	OneHundredKM      Radius = "100"
	OneHundredFiftyKM Radius = "150"
	TwoHundredKM      Radius = "200"
)

type SearchParam

type SearchParam struct {
	Category          Category          `json:"category"`
	Provider          Provider          `json:"provider"`
	OfferType         OfferType         `json:"offer_type"`
	Location          LocationID        `json:"location"`
	Radius            Radius            `json:"radius"`
	SpecificParameter SpecificParameter `json:"specific_parameter"`
	// Which page should be scraped
	Page int `json:"page"`
	// Price from in euro
	PriceFrom int `json:"price_from"`
	// Price to in euro
	PriceTo int `json:"price_to"`
}

SearchParam holds all parameters for a search

func ParamsFromJSON

func ParamsFromJSON(data io.Reader) (*SearchParam, error)

ParamsFromJSON creates a SearchParam struct from JSON

func (*SearchParam) ToJSON

func (sp *SearchParam) ToJSON(out io.Writer) error

type Seller

type Seller struct {
	Name         string    `json:"name"`
	ActiveSince  time.Time `json:"active_since"`
	Friendliness string    `json:"friendliness"`
	Rating       string    `json:"rating"`
}

Seller represents the seller of an aditem

type SpecificParameter

type SpecificParameter map[ParamName]string

SpecificParameter represent specific parameters for each category