# fastdownload


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

If you have datasets or other archives that you want to make available
to your users, and ensure they always have the latest versions and that
they are downloaded correctly, `fastdownload` can help.

## Install

Using pip:

    pip install fastdownload

…or using conda:

    conda install -c fastai fastdownload

## What’s this about?

The situation where you might want to use `fastdownload` is where you
have one or more URLs pointing at some archives you want to make
available, and you want to ensure that your users download those
archives correctly, have the latest version, and that it’s as easy as
possible for them to access the information in those archives.

Your user just calls a single method,
[`FastDownload.get`](https://fastdownload.fast.ai/core.html#fastdownload.get),
passing the URL required, and the URL will be downloaded and extracted
to the directories you choose. The path to the extracted file is
returned. If that URL has already been downloaded, then the cached
archive or contents will be used automatically. However, if that size or
hash of the archive is different to what it should be, then the user
will be informed, and a new version will be downloaded.

In the future, you may want to update one or more of your archives. When
you do so, `fastdownload` will ensure your users have the latest
version, by checking their downloaded archives against your updated file
size and hash information.

For instance, `fastai` uses `fastdownload` to provide access to datasets
for deep learning. `fastai` users can download and extract them with a
single command, using the return value to access the files. The files
are automatically placed in appropriate subdirectories of a `.fastai`
folder in the user’s homedir. If a dataset is updated, users are
informed the next time they use the dataset, and the latest version is
automatically downloaded and extracted for them.

## Usage: downloading files

When your users download an archive, `fastdownload` will automatically
save it to a directory, check if the size and hash matches, and extract
the contents. Minimal usage for downloading and extracting is:

``` python
from fastdownload import FastDownload
d = FastDownload()
path = d.get('https://...')
```

After this, `path` will contain the path where the extracted files are
located. By default, archives are saved to `{base}/archive`, and
extracted to `{base}/data`. `{base}` defaults to `~/.fastdownload`. If
there is more than one file or folder in the root of the downloaded
archive, then a new folder is created in `data` for the contents.

Instead of `get`, use `download` to download the URL without extracting
it, or `extract` to extract the URL without downloading it (assuming
it’s already been downloaded to the `archive` directory). All of these
methods accept a `force` parameter which will download/extract the
archive even if it’s already present.

You can change any or all of the `base`, `archive`, and `data` paths by
passing them to
[`FastDownload`](https://fastdownload.fast.ai/core.html#fastdownload):

``` python
d = FastDownload(base='~/.mypath', archive='downloaded', data='extracted')
```

You can remove the cached archive file and/or the extracted contents
with `rm`:

``` python
d.rm('https://...')
```

## Usage: making archives available to download

`fastdownload` will add a file `download_checks.py` to your Python
module which contains file sizes and hashes for your archives. The file
is located in the same directory as a module you choose, e.g.:

``` python
d = FastDownload(module=fastai.some_module)
```

Then use `update` to create or update the size and hash for a URL:

``` python
d.update('https://...')
```

You will now find there is a file called `download_checks.py` in the
same directory where `fastai.some_module` is located, which contains a
Python dict with the URL, size, and hash for this file. If you’ve
downloaded this file before to your `archive` path then it will be used,
instead of downloading a new copy. Use `get(force=True)` first to
download a new copy if even you have it in your archive.

## Config file

If there is a file called `config.ini` in your `base` directory, then
keys `archive` and `data` will be used as the default values for
[`FastDownload`](https://fastdownload.fast.ai/core.html#fastdownload).
The file should be in
[configparser](https://docs.python.org/3/library/configparser.html)
format. Here’s a sample `config.ini`:

    [DEFAULT]         
    archive = downloaded
    data = extracted

If there is no ini file present, one will be automatically created for
for you using the details you pass to
[`FastDownload`](https://fastdownload.fast.ai/core.html#fastdownload).

You can add any additional key/value pairs to the config file that you
want. When you call
[`FastDownload.get`](https://fastdownload.fast.ai/core.html#fastdownload.get)
pass `extract_key` to use a key other than `data` for choosing a
location to extract to.