ScrapyCloudEx
An API wrapper for the ScrapyCloud API provided by ScraphingHub.com and documented here
The implementation is only partial at this time: among the endpoints listed here, the Collections
and Frontier
are not accessible through this wrapper (PRs welcome!).
This wrapper handles Http communication and reponse decoding for you (provided you install the dependencies below), but
will only request the json
format by default, and will only decoder json
responses. Other response types are passed
through, allowing you to process them if desired. If you prefer to use your own Http adapter or function to decode the
body, refer to the “custom implementations” section in the documentation.
Installation
The package can be installed by adding scrapy_cloud_ex
to your list of dependencies in mix.exs
:
def deps do
[
{:scrapy_cloud_ex, "~> 0.1.0"},
# Optional dependencies
# add this to use the default HttpAdapter
{:hackney, "~> 1.14"},
# add this to use the default Decoder
{:jason, "~> 1.1"}
]
end
For the “It Just Works” experience, simply add the above 3 dependencies to your project.
If you know what you’re doing and want to use custom implentations of http clients or response decoders,
feel free to include only :scrapy_cloud_ex
as a dependency and refer to the “custom implementations”
section of the documentation sidebar.
Basic Usage
To interact with the API endpoints, use the various modules prefixed with ScrapyCloudEx.Endpoints
.
iex(1)> ScrapyCloudEx.Endpoints.Storage.Items.get("API_KEY", "53/34/7", pagination: [count: 10])
Where "API_KEY"
is your API key (which you can find here)
and "53/34/7"
is a job id.
The docs can be found at https://hexdocs.pm/scrapy_cloud_ex.
License
Copyright 2018 David Sulc
This library is released under the Apache 2.0 License - see the LICENSE file.