Skip to content

Commit

Permalink
Merge branch 'develop'
Browse files Browse the repository at this point in the history
  • Loading branch information
juansensio committed Oct 24, 2024
2 parents c1915d9 + 95d3339 commit e220513
Show file tree
Hide file tree
Showing 25 changed files with 73 additions and 56 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@ You can learn more at the [EOTDL website](https://eotdl.com/) or start with our

The **Earth Observation Training Data Lab** (EOTDL) is a complete environment that allows you, among other things, to:

- Explore and download Training Datasets (TDS) for Earth Observation (EO) applications.
- Explore and stage Training Datasets (TDS) for Earth Observation (EO) applications.
- Create and upload your own TDS by combining and annotating EO data from different sources.
- Train Machine Learning (ML) models using the hosted TDS in the cloud with multi-GPU machines.
- Explore and download pre-trianed ML models for EO applications.
- Explore and stage pre-trianed ML models for EO applications.

## Why EOTDL?

Expand Down
4 changes: 2 additions & 2 deletions api/api/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,10 @@
description = """
The EOTDL API allows you, among other things, to:
* Explore and download Training Datasets (TDS) for Earth Observation (EO) applications.
* Explore and stage Training Datasets (TDS) for Earth Observation (EO) applications.
* Create and upload your own TDS by combining and annotating EO data from different sources.
* Train Machine Learning (ML) models using the hosted TDS in the cloud with multi-GPU machines.
* Explore and download pre-trianed ML models for EO applications.
* Explore and stage pre-trianed ML models for EO applications.
"""

app = FastAPI(
Expand Down
4 changes: 2 additions & 2 deletions eotdl/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@

This is the main library and CLI for the **Earth Observation Training Data Lab** (EOTDL), a complete environment that allows you, among other things, to:

- Explore and download Training Datasets (TDS) for Earth Observation (EO) applications.
- Explore and stage Training Datasets (TDS) for Earth Observation (EO) applications.
- Create and upload your own TDS by combining and annotating EO data from different sources.
- Train Machine Learning (ML) models using the hosted TDS in the cloud with multi-GPU machines.
- Explore and download pre-trianed ML models for EO applications.
- Explore and stage pre-trianed ML models for EO applications.

In our blog you will find tutorials to learn how leverage the EOTDL to create and use TDS and ML models for your own EO applications.

Expand Down
2 changes: 1 addition & 1 deletion tutorials/notebooks/03_q1_datasets.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -431,7 +431,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"After the ingestion, you can explore and download your dataset like shown in the previous tutorial."
"After the ingestion, you can explore and stage your dataset like shown in the previous tutorial."
]
},
{
Expand Down
2 changes: 1 addition & 1 deletion tutorials/notebooks/04_q2_datasets.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Once the metadata has been generated, you can ingest, explore and download a Q2 dataset as any other dataset.\n",
"Once the metadata has been generated, you can ingest, explore and stage a Q2 dataset as any other dataset.\n",
"\n",
"> Change the id of the catalog if you want to give it a specific name."
]
Expand Down
2 changes: 1 addition & 1 deletion tutorials/workshops/bids23/00_eotdl.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@
"source": [
"This is the scheduled agenda for the tutorial session:\n",
"\n",
"- [Explore and download datasets and models](01_exploring.ipynb)\n",
"- [Explore and stage datasets and models](01_exploring.ipynb)\n",
"- [Train models (locally or in the cloud)](02_training.ipynb)\n",
"- [Ingest datasets and models](03_ingesting.ipynb)\n",
"- [Create your own datasets](04_creating.ipynb)\n",
Expand Down
2 changes: 1 addition & 1 deletion tutorials/workshops/bids23/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
This is the scheduled agenda for the tutorial session:

- [Explore and download datasets and models](01_exploring.ipynb)
- [Explore and stage datasets and models](01_exploring.ipynb)
- [Train models (locally or in the cloud)](02_training.ipynb)
- [Ingest datasets and models](03_ingesting.ipynb)
- [Create your own datasets](04_creating.ipynb)
Expand Down
2 changes: 1 addition & 1 deletion tutorials/workshops/philab24/00_eotdl.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@
"source": [
"This is the scheduled agenda for the tutorial session:\n",
"\n",
"- [Explore and download datasets and models](01_exploring.ipynb)\n",
"- [Explore and stage datasets and models](01_exploring.ipynb)\n",
"- [Train models (locally or in the cloud)](02_training.ipynb)\n",
"- [Ingest datasets and models](03_ingesting.ipynb)\n",
"- [Create your own datasets](04_creating.ipynb)\n",
Expand Down
2 changes: 1 addition & 1 deletion tutorials/workshops/philab24/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This is the scheduled agenda for the tutorial session:

- [Explore and download datasets and models](01_exploring.ipynb)
- [Explore and stage datasets and models](01_exploring.ipynb)
- [Train models (locally or in the cloud)](02_training.ipynb)
- [Ingest datasets and models](03_ingesting.ipynb)
- [Create your own datasets](04_creating.ipynb)
Expand Down
18 changes: 14 additions & 4 deletions ui/src/routes/Popular.svelte
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
<script>
import Card from "$components/Card.svelte";
import Skeleton from "$components/Skeleton.svelte";
import {links, modelImagesOffset} from "$stores/images"
import { links, modelImagesOffset } from "$stores/images";
export let data;
export let title;
export let tags;
</script>

<div class="flex flex-col items-center w-full pt-10">
Expand All @@ -18,9 +17,20 @@
{#if data}
{#each data as item, i}
{#if title.includes("models")}
<Card data={item} {tags} img={links[(i+modelImagesOffset*2)%links.length]}/>
<Card
data={item}
{tags}
img={links[
(i + modelImagesOffset * 2) % links.length
]}
link="models"
/>
{:else}
<Card data={item} {tags} img={links[i%links.length]}/>
<Card
data={item}
{tags}
img={links[i % links.length]}
/>
{/if}
{/each}
{:else}
Expand Down
17 changes: 12 additions & 5 deletions ui/src/routes/Recent.svelte
Original file line number Diff line number Diff line change
@@ -1,12 +1,11 @@
<script>
import Card from "$components/Card.svelte";
import Skeleton from "$components/Skeleton.svelte";
import { links, modelImagesOffset } from "$stores/images"
import { links, modelImagesOffset } from "$stores/images";
export let data;
export let title;
export let tags;
</script>

<div class="flex flex-col items-center w-full">
Expand All @@ -31,11 +30,19 @@
{#if data}
{#each data as item, i}
{#if title.includes("models")}
<Card data={item} {tags} img={links[(i+modelImagesOffset)%links.length]}/>
<Card
data={item}
{tags}
img={links[(i + modelImagesOffset) % links.length]}
link="models"
/>
{:else}
<Card data={item} {tags} img={links[i%links.length]}/>
<Card
data={item}
{tags}
img={links[i % links.length]}
/>
{/if}

{/each}
{:else}
{#each [1, 2, 3] as _}
Expand Down
2 changes: 1 addition & 1 deletion ui/src/routes/blog/01_exploring.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Exploring and downloading Datasets and Models
date: '2023-11-04T01:00:00.000Z'
description: In this post you will learn how to explore and download the datasets and models hosted on EOTDL with the different tools of the environment
description: In this post you will learn how to explore and stage the datasets and models hosted on EOTDL with the different tools of the environment
tags: getting started, explore, download, datasets, models
link: https://github.com/earthpulse/eotdl/blob/main/tutorials/notebooks/01_exploring.ipynb
---
Expand Down
2 changes: 1 addition & 1 deletion ui/src/routes/blog/03_q1_datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,7 +251,7 @@ ingest_dataset('data/EuroSAT-RGB-small-STAC')
Done


After the ingestion, you can explore and download your dataset like shown in the previous tutorial.
After the ingestion, you can explore and stage your dataset like shown in the previous tutorial.


```python
Expand Down
2 changes: 1 addition & 1 deletion ui/src/routes/blog/04_q2_datasets.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ Remember, however, that the metrics will be computed automatically when ingestin

## Ingesting Q2 datasets

Once the metadata has been generated, you can ingest, explore and download a Q2 dataset as any other dataset.
Once the metadata has been generated, you can ingest, explore and stage a Q2 dataset as any other dataset.


```python
Expand Down
2 changes: 1 addition & 1 deletion ui/src/routes/datasets/[name]/+page.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,7 @@
</div>
<div class="flex flex-col gap-3 text-xs sm:mt-0 mt-16">
<hr class="sm:hidden" />
<p>Download the dataset with the CLI:</p>
<p>Stage the dataset with the CLI:</p>
<div class="relative">
<pre class="bg-gray-200 p-3 overflow-x-auto"><button
on:click={() =>
Expand Down
4 changes: 2 additions & 2 deletions ui/src/routes/docs/+layout.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
link: "datasets",
links: [
{ name: "Explore", link: "explore" },
{ name: "Download", link: "download" },
{ name: "Stage", link: "stage" },
{ name: "Ingest", link: "ingest" },
{ name: "Create", link: "create" },
{ name: "Labelling", link: "labelling" },
Expand All @@ -30,7 +30,7 @@
link: "models",
links: [
{ name: "Explore", link: "explore" },
{ name: "Download", link: "download" },
{ name: "Stage", link: "stage" },
{ name: "Train", link: "train" },
{ name: "Ingest", link: "ingest" },
],
Expand Down
4 changes: 2 additions & 2 deletions ui/src/routes/docs/+page.svx
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@
The **Earth Observation Training Data Lab**
(EOTDL) allows you, among other things, to:

- Explore and download Training Datasets (TDS) for Earth Observation (EO) applications.
- Explore and stage Training Datasets (TDS) for Earth Observation (EO) applications.
- Create and upload your own TDS by combining and annotating EO data from different sources.
- Train Machine Learning (ML) models using the hosted TDS in the cloud with multi-GPU machines.
- Explore and download pre-trained ML models for EO applications.
- Explore and stage pre-trained ML models for EO applications.

Here you will find information on how to use the platform and how to contribute to the project and get involved.

Expand Down
2 changes: 1 addition & 1 deletion ui/src/routes/docs/datasets/+page.svx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Datasets

The main feature that EOTDL offers is a repository of Training Datasets (TDS). Feel free to explore available datasets and download them for
The main feature that EOTDL offers is a repository of Training Datasets (TDS). Feel free to explore available datasets and stage them for
training your ML models.

EOTDL also offers tools to create new datasets, or improve existing ones, including data access functionality, data curation and engineering tools such as labelling and feature engineering.
2 changes: 1 addition & 1 deletion ui/src/routes/docs/datasets/create/+page.svx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

EOTDL offers the following functionality for creating datasets:

- Explore and download available satellite imagery
- Explore and stage available satellite imagery
- Generate STAC metadata for your dataset

You can combine the tools offered on EOTDL with your own to create a dataset.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,45 +9,45 @@
</script>


# Download datasets
# Stage datasets

Once you find a suitable dataset you can download it for training your models.
Once you find a suitable dataset you can stage it for training your models.

<!-- ## User interface

<UI><p class="flex flex-row gap-2">Inside the model's page, click on the download icon <Download color="gray" size={20} /> in the file that you want to download.</p></UI>
<UI><p class="flex flex-row gap-2">Inside the model's page, click on the stage icon <Download color="gray" size={20} /> in the file that you want to stage.</p></UI>

You'll need to be logged in to download datasets. -->
You'll need to be logged in to stage datasets. -->

## CLI

You can download a dataset with following CLI command:
You can stage a dataset with following CLI command:

<CLI><Code>eotdl datasets get "dataset name"</Code></CLI>

Your datasets will be downloaded to a default folder, but you can specify a different folder with the `--path` option or the `EOTDL_DOWNLOAD_PATH` environment variable. For example, to download the dataset to the current directory:
Your datasets will be staged to a default folder, but you can specify a different folder with the `--path` option or the `EOTDL_DOWNLOAD_PATH` environment variable. For example, to stage the dataset to the current directory:

<CLI><Code>eotdl datasets get "dataset name" --path .</Code></CLI>

<!-- For Q0 datasets, you can also specify which file to download.
<!-- For Q0 datasets, you can also specify which file to stage.

<CLI><Code>eotdl datasets get "dataset name" --file file-name</Code></CLI> -->

<!-- For Q1+ datasets, the previous command will only download the STAC metadata (which contains the links to all the assets). However, if you want to download the assets as well, you cann use the `--assets` option.
<!-- For Q1+ datasets, the previous command will only stage the STAC metadata (which contains the links to all the assets). However, if you want to stage the assets as well, you cann use the `--assets` option.

<CLI><Code>eotdl datasets get "dataset name" --assets</Code></CLI> -->

In order to overwrite a dataset that you already downloaded, you can use the `--force` option.
In order to overwrite a dataset that you already staged, you can use the `--force` option.

<CLI><Code>eotdl datasets get "dataset name" --force</Code></CLI>

If you know the specific version of the dataset to download, use the `--version` option.
If you know the specific version of the dataset to stage, use the `--version` option.

<CLI><Code>eotdl datasets get "dataset name" --version 1</Code></CLI>

## Library

You can download datasets using the following Python code:
You can stage datasets using the following Python code:

<Library><Code>
from eotdl.datasets import download_dataset<br/><br/>download_dataset("dataset-name")</Code></Library>
Expand All @@ -58,10 +58,10 @@ And use the same options seen before.

<!-- ## API

You can download a dataset file using the following API call:
You can stage a dataset file using the following API call:

<Api><Code>curl -X 'GET' \
'https://api.eotdl.com/datasets/dataset-id/download/file-name' \
'https://api.eotdl.com/datasets/dataset-id/stage/file-name' \
-H 'accept: application/json' \
-H 'Authorization: Bearer your-token'</Code></Api>

Expand Down
2 changes: 1 addition & 1 deletion ui/src/routes/docs/getting-started/quality/+page.svx
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

The datasets and models hosted on the EOTDL are categorized into for quality levels:

- **Q0**: datasets and models in the form of an archive with arbitary files without curation. This level is ideal for easy and fast upload/download of existing datasets and models.
- **Q0**: datasets and models in the form of an archive with arbitary files without curation. This level is ideal for easy and fast upload/stage of existing datasets and models.
- **Q1**: datasets and models with STAC metadata without QA. These datasets and models can leverage a limited set of EOTDL features.
- **Q2**: datasets and mmodels with STAC metadata and the EOTDL custom extensions, including automated QA. These can leverage the full potential of the EOTDL.
- **Q3**: Q2 datasets and models that are manually curated (or audited) by us. These are the most reliable and can be used for benchmarking.
Expand Down
2 changes: 1 addition & 1 deletion ui/src/routes/docs/models/+page.svx
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@

Besides working with training datasets, EOTDL is also a repositoy of Machine Learning models.

EOTDL offers tools to explore, download and train ML models with the datasets available in the platform.
EOTDL offers tools to explore, stage and train ML models with the datasets available in the platform.

You can also request a GPU-powered machine on the cloud to train your models via our [Discord](https://discord.gg/hYxc5AJB92).
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@
</script>


# Download models
# Stage models

In the same way, downloading models is similar to downloading datasets.
In the same way, staging models is similar to staging datasets.

<!-- ## User interface

Expand All @@ -21,11 +21,11 @@ You'll need to be logged in to download models. -->

## CLI

You can download a model with following CLI command:
You can stage a model with following CLI command:

<CLI><Code>eotdl models get "model-name"</Code></CLI>

Your models will be downloaded to a default folder, but you can specify a different folder with the `--path` option or the `EOTDL_DOWNLOAD_PATH` environment variable. For example, to download the model to the current directory:
Your models will be staged to a default folder, but you can specify a different folder with the `--path` option or the `EOTDL_DOWNLOAD_PATH` environment variable. For example, to stage the model to the current directory:

<CLI><Code>eotdl models get "model-name" --path .</Code></CLI>

Expand All @@ -37,17 +37,17 @@ Your models will be downloaded to a default folder, but you can specify a differ

<CLI><Code>eotdl models get "model name" --assets</Code></CLI> -->

In order to overwrite a model that you already downloaded, you can use the `--force` option.
In order to overwrite a model that you already staged, you can use the `--force` option.

<CLI><Code>eotdl models get "model-name" --force</Code></CLI>

If you know the specific version of the model to download, use the `--version` option.
If you know the specific version of the model to stage, use the `--version` option.

<CLI><Code>eotdl models get "model-name" --version 1</Code></CLI>

## Library

You can download models using the following Python code:
You can stage models using the following Python code:

<Library><Code>
from eotdl.models import download_model<br/><br/>download_model("model-name")</Code></Library>
Expand Down
2 changes: 1 addition & 1 deletion ui/src/routes/models/[name]/+page.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@
{/if} -->
</div>
<div class="flex flex-col gap-3 text-xs">
<p>Download the model with the CLI:</p>
<p>Stage the model with the CLI:</p>
<div class="relative">
<pre class="bg-gray-200 p-3 overflow-x-auto"><button
on:click={() =>
Expand Down
Loading

0 comments on commit e220513

Please sign in to comment.