Skip to content

Commit

Permalink
reorganization and content updates
Browse files Browse the repository at this point in the history
  • Loading branch information
russbiggs committed Oct 1, 2024
1 parent 236904c commit 1048fe8
Show file tree
Hide file tree
Showing 22 changed files with 199 additions and 320 deletions.
17 changes: 7 additions & 10 deletions astro.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -2,30 +2,26 @@ import { defineConfig } from 'astro/config';
import starlight from '@astrojs/starlight';
import starlightOpenAPI, { openAPISidebarGroups } from 'starlight-openapi';
import { shield } from '@kindspells/astro-shield'
import { resolve } from 'node:path'
import sitemap from '@astrojs/sitemap';

import robotsTxt from 'astro-robots-txt';

const rootDir = new URL('.', import.meta.url).pathname
const modulePath = resolve(rootDir, 'src', 'generated', 'sriHashes.mjs')

export default defineConfig({
site: 'https://openaq.github.io',
// base: 'docs.openaq.org',
site: 'https://docs.openaq.org',
integrations: [starlight({
title: 'OpenAQ Docs',
logo: {
light: './src/assets/logo.svg',
dark: './src/assets/logo-dark.svg',
},
favicon: '/images/favicon.svg',
favicon: '/favicon.svg',
head: [
{
tag: 'link',
attrs: {
rel: 'icon',
href: '/images/favicon.ico',
href: '/favicon.ico',
sizes: '16x16 32x32',
},
},
Expand All @@ -44,6 +40,7 @@ export default defineConfig({
{
base: 'api',
label: 'API Reference',
collapsed: true,
schema: 'https://api.openaq.org/openapi.json',
},
]),
Expand Down Expand Up @@ -127,8 +124,8 @@ export default defineConfig({
},
],
}),
// , shield({})
// , sitemap()
// , robotsTxt()
, shield({})
, sitemap()
, robotsTxt()
],
});
File renamed without changes.
File renamed without changes
2 changes: 2 additions & 0 deletions src/components/CustomCard.astro
Original file line number Diff line number Diff line change
Expand Up @@ -57,5 +57,7 @@ const { icon, title, href } = Astro.props;
.card .body {
margin: 0;
font-size: clamp(var(--sl-text-sm), calc(0.5rem + 1vw), var(--sl-text-body));
color: black;
text-decoration: inherit;
}
</style>
4 changes: 2 additions & 2 deletions src/content/docs/about/terms.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ data on the OpenAQ Platform, you acknowledge that, without limitation or
qualification, you have read, understood, and agreed to the Terms and any future
modifications to the Terms and to the collection and use of your information as
set forth in our
[Privacy Policy](https://docs.google.com/document/d/1h9rUHrup-QhLI1GXlOib-Ct5ZQlXdYaZb0l0oZjIMzg/edit?usp=sharing).
[Privacy Policy](https://openaq.org/privacy/).

OpenAQ reserves the right to modify these Terms without notice, in whole or in
part, at our sole discretion. If we change these Terms, we will post the changes
Expand Down Expand Up @@ -144,7 +144,7 @@ discontinuance of the OpenAQ API.

By using the OpenAQ Platform you acknowledge and agree to abide by the
principles in our
[Code of Conduct](https://github.com/openaq/openaq-info/blob/main/CODE-OF-CONDUCT.md).
[Code of Conduct](https://github.com/openaq/.github/blob/main/CODE_OF_CONDUCT.md).

### Violation of Terms of Use for Users

Expand Down
7 changes: 0 additions & 7 deletions src/content/docs/aws/about.mdx
Original file line number Diff line number Diff line change
@@ -1,12 +1,6 @@
---
title: About Open Data on AWS
description: A guide in my new Starlight docs site.
---
For larger data queries, OpenAQ data can be directly accessed as gzipped csv
(comma-separated values) files from the Open Data on AWS Program's S3 bucket.
Data are updated by location every 72 hours and organized to allow for
filtering by location, year and month.


## Bucket structure

Expand All @@ -33,7 +27,6 @@ import { FileTree } from '@astrojs/starlight/components';
</FileTree>



example file path:

```
Expand Down
40 changes: 24 additions & 16 deletions src/content/docs/aws/athena-guide.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,21 @@ title: Guide - Querying with AWS Athena

import { Aside } from '@astrojs/starlight/components';


AWS Athena provides a powerful SQL interface to query data from the OpenAQ
Archive S3 bucket.
Archive S3 bucket.

<Aside type="caution">
To use AWS Athena, an Amazon Web Services (AWS) account is required and is
subject to AWS pricing; see more about pricing on the
[Athena pricing page](https://aws.amazon.com/athena/pricing/)
To use AWS Athena, an Amazon Web Services (AWS) account is required and is
subject to AWS pricing; see more about pricing on the [Athena pricing
page](https://aws.amazon.com/athena/pricing/)
</Aside>

<Aside type="note">
This guide is not meant to replace a comprehensive tutorial of AWS Athena. We
recommend reading the official
[Athena documentation](https://docs.aws.amazon.com/athena/index.html).
This guide is not meant to replace a comprehensive tutorial of AWS Athena. We
recommend reading the official [Athena
documentation](https://docs.aws.amazon.com/athena/index.html).
</Aside>


To start create a table.

```sql
Expand All @@ -45,11 +43,17 @@ LOCATION
TBLPROPERTIES ('skip.header.line.count'='1')
```

The Athena table definition defines all the columns and partitions for the data file in the OpenAQ Archive. The Apache Hive Partitioning format help limit objects scans throughout the bucket, reducing cost and time for querying. Partitions are optional and are typically best optimized on a case by case basis. Read more about Athena partitioning on S3 on the [AWS docs](https://docs.aws.amazon.com/athena/latest/ug/partitions.html).
The Athena table definition defines all the columns and partitions for the data
file in the OpenAQ Archive. The Apache Hive Partitioning format help limit
objects scans throughout the bucket, reducing cost and time for querying.
Partitions are optional and are typically best optimized on a case by case
basis. Read more about Athena partitioning on S3 on the
[AWS docs](https://docs.aws.amazon.com/athena/latest/ug/partitions.html).

## Example

Loading all partitions can be very slow, so for this example we will partition manually for a single location and month that we are interested in:
Loading all partitions can be very slow, so for this example we will partition
manually for a single location and month that we are interested in:

```sql
ALTER TABLE openaqMeasurements ADD
Expand All @@ -62,19 +66,23 @@ ALTER TABLE openaqMeasurements ADD
PARTITION (year='2022',locationid='815');
```

Now with the table defined we can query the dataset using the SQL. To get all measurements for a single location (locationid 2178) in a single month (December 2022) we can use the following:
Now with the table defined we can query the dataset using the SQL. To get all
measurements for a single location (locationid 2178) in a single month
(December 2022) we can use the following:

```sql
SELECT
SELECT
locationid
, parameter
, value
, from_iso8601_timestamp(datetime) AS datetime
, from_iso8601_timestamp(datetime) AS datetime
FROM
openaqMeasurements WHERE locationid IN ('2178' , '827', '1644', '815') AND year='2022'
```

This will return all measurements for December 2022 in this following form:
Now due to the partitioning Athena will only scan the selected locations instead
of the entire bucket, reducing costs and time to compute. This will return all
measurements for December 2022 in this following form:

```csv
"parameter","value","datetime"
Expand All @@ -94,4 +102,4 @@ This will return all measurements for December 2022 in this following form:
"pm10","19.0","2022-12-12 14:00:00.000 -07:00"
"pm10","12.0","2022-12-12 15:00:00.000 -07:00"
...
```
```
20 changes: 9 additions & 11 deletions src/content/docs/aws/year-guide.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,15 @@
title: Guide - Download a year of data for a location.
---

This guide shows you how to download a relatively large amount of measurement data, one year of hourly measurements from a single location, from the Open Data on AWS bucket.
This guide shows you how to download a relatively large amount of measurement
data, one year of hourly measurements from a single location, from the Open Data
on AWS bucket.

import { Aside } from '@astrojs/starlight/components';



import { Aside, } from '@astrojs/starlight/components';



Run the following command to copy all data from [location 2178](https://explore.openaq.org/locations/2178) during the year 2020 to a local directory named `data`:
Run the following command to copy all data from
[location 2178](https://explore.openaq.org/locations/2178) during the year 2020
to a local directory named `data`:

```sh
aws s3 cp \
Expand All @@ -20,8 +19,7 @@ aws s3 cp \
s3://openaq-data-archive/records/csv.gz/locationid=2178/year=2020/ \
data
```

<Aside type="caution" title="Warning">
This will download over 350 files to your system, roughly 1.5 MB of data.
This will download over 350 files to your system, roughly 1.5 MB of data.
</Aside>


77 changes: 0 additions & 77 deletions src/content/docs/guides/find-locations-in-geojson.mdx

This file was deleted.

4 changes: 0 additions & 4 deletions src/content/docs/guides/get-latest-values.mdx

This file was deleted.

12 changes: 0 additions & 12 deletions src/content/docs/guides/query-locations.mdx

This file was deleted.

22 changes: 0 additions & 22 deletions src/content/docs/guides/query-measurements.mdx

This file was deleted.

4 changes: 2 additions & 2 deletions src/content/docs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ import CustomCard from '../../components/CustomCard.astro';
## Explore

<CardGrid stagger>
<CustomCard title="Learn about the OpenAQ API" href="/using-the-api/introduction" icon="open-book">

<CustomCard title="Get started with the OpenAQ API" href="/using-the-api/quick-start" icon="open-book">
Try our quick start guide to learn the basics of requesting data from OpenAQ.
</CustomCard>
<CustomCard title="Read the OpenAQ API OpenAPI Spec" href="https://api.openaq.org/docs" icon="seti:json">
See the Swagger reference documentation for the OpenAQ API.
Expand Down
2 changes: 1 addition & 1 deletion src/content/docs/resources/countries.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ air quality parameters available for each. The list only includes countries that
have monitoring locations from which OpenAQ ingests data, and the Country value
is derived from the coordinates provided by the upstream provider. OpenAQ uses
the Natural Earth 10m dataset for country boundaries, see this
[disclaimer](disclaimers/countries). OpenAQ uses the <a style="display: inline-flex;" href="https://www.iso.org/iso-3166-country-codes.html" target="_blank" rel="noopener noreferrer">ISO 3166-1 alpha 2<Icon name="external"/></a>
[disclaimer](/disclaimers/countries). OpenAQ uses the <a style="display: inline-flex;" href="https://www.iso.org/iso-3166-country-codes.html" target="_blank" rel="noopener noreferrer">ISO 3166-1 alpha 2<Icon name="external"/></a>
country codes, which may differ from that reported by the upstream provider.

## Purpose and Use
Expand Down
11 changes: 0 additions & 11 deletions src/content/docs/resources/latest.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -44,17 +44,6 @@ or
https://api.openaq.org/v3/parameters/2/latest
```

<Aside type="note">

The latest value does not represent the most recent measurement ingested by
OpenAQ for a given sensor, but instead represents the last measurement value in
the series. Due to the nature of how some upstream providers report data,
measurements are not necessarily reported in order. Polling latest resources for
measurement values does not guarantee complete data coverage as measurements may
be ingested out of their time series order and not show up as a "latest" value.

</Aside>

## Key Fields

- datetime: the time of the measurement, provided in UTC and local time. See
Expand Down
Loading

0 comments on commit 1048fe8

Please sign in to comment.