Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
04bee27
update discovery methods to more reliably add subresources to their p…
tipatterson-dev Oct 25, 2025
98e63c3
Add framework for future event bus functionality
tipatterson-dev Nov 10, 2025
95bdb6b
Implement CS API Part 3 pub/sub topics, expand public API surface, a…
tipatterson-dev Apr 2, 2026
d7e077e
updates for part 3 and datastores
tipatterson-dev Apr 11, 2026
ea73e22
Fix a longstanding time comparison issue, fix a deserialization probl…
tipatterson-dev Apr 14, 2026
5541ccd
fix incorrect serialization alias on VectorSchema
tipatterson-dev Apr 14, 2026
406bd26
Refactor event bus into events/ sub-package and wire lifecycle/stream…
tipatterson-dev Apr 28, 2026
2e15c29
Add SQLite-backed DataStore implementation
tipatterson-dev Apr 28, 2026
de14a46
add docker and local publish script devx improvements
tipatterson-dev Apr 30, 2026
5a4a970
Change swe components to use Literal[{type_name}]'s for the type fiel…
tipatterson-dev Apr 30, 2026
7e6ac05
Prefer AnyComponent type over SerializeAsAny to prevent loss of data …
tipatterson-dev Apr 30, 2026
c194e2b
remove unnecessary deep copy in event builder, update tests to preven…
tipatterson-dev Apr 30, 2026
853a028
update version, docs and remove a deprecated to_lower_camel call
tipatterson-dev Apr 30, 2026
1070198
add missing files for mkdocs, fix newline error for flake8
tipatterson-dev May 1, 2026
02ca719
add missing files for mkdocs, fix newline error for flake8
tipatterson-dev May 1, 2026
97cd5e2
Enforced the SWE Common 3 SoftNamedProperty rule: components carry an…
tipatterson-dev May 1, 2026
0d6ef64
bring data models more in line with SWECommon 3.0, add more validatio…
tipatterson-dev May 1, 2026
981e5a4
update readme to favor new mkdocs and publish scripts to supply the e…
tipatterson-dev May 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .github/workflows/docs_pages.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@ on: [ push, pull_request, workflow_dispatch ]
permissions:
contents: write

concurrency:
group: docs-${{ github.ref }}
cancel-in-progress: true

jobs:
build-docs:
runs-on: ubuntu-latest
Expand All @@ -19,9 +23,9 @@ jobs:
- name: Install dependencies
run: uv sync --all-extras

- name: Sphinx build
- name: Build MkDocs site
run: |
uv run sphinx-build -b html docs/source docs/build/html
uv run mkdocs build --strict

- name: Deploy documentation
uses: peaceiris/actions-gh-pages@v4
Expand Down
52 changes: 50 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,54 @@ recommendations for data, retrieving data in real time, archival streams, and ba

API Documentation available [here](https://botts-innovative-research.github.io/OSHConnect-Python/)

Links:
Links:
* [Architecture Doc](https://docs.google.com/document/d/1pIaeQw0ocU6ApNgqTVRZuSwjJAbhCcmweMq6RiVYEic/edit?usp=sharing)
* [UML Diagram](https://drive.google.com/file/d/1FVrnYiuAR8ykqfOUa1NuoMyZ1abXzMPw/view?usp=drive_link)
* [UML Diagram](https://drive.google.com/file/d/1FVrnYiuAR8ykqfOUa1NuoMyZ1abXzMPw/view?usp=drive_link)

## Generating the Docs

The documentation is built with [MkDocs](https://www.mkdocs.org/) using the
Material theme, [mkdocstrings](https://mkdocstrings.github.io/) for
auto-generated API reference from the source, and
[mermaid](https://mermaid.js.org/) for architecture diagrams. Markdown sources
live under `docs/markdown/`.

Install dev dependencies (including MkDocs and plugins):

```bash
uv sync
```

Build the HTML docs:

```bash
uv run mkdocs build
```

The output will be in `docs/build/html/`. Open `docs/build/html/index.html` in
a browser to view locally.

For a live-reloading preview while editing:

```bash
uv run mkdocs serve
```

Then visit http://127.0.0.1:8000.

To match what CI publishes (warnings become errors — useful when you've
touched docstrings):

```bash
uv run mkdocs build --strict
```

CI builds the site on every push and deploys `main` to GitHub Pages via
`.github/workflows/docs_pages.yaml`.

The legacy Sphinx setup under `docs/source/` is kept temporarily for
reference and builds to a separate output directory:

```bash
uv run sphinx-build -b html docs/source docs/build/sphinx
```
35 changes: 35 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# =============================================================================
# OSHConnect-Python — Local PyPI Server
#
# A lightweight pypiserver for publishing dev builds of oshconnect so that
# downstream projects (OCSASim, etc.) can `pip install` normally instead of
# pointing at raw wheel paths.
#
# Usage:
# docker compose up -d # start the local PyPI
# ./scripts/publish-local.sh # build wheel + upload to local PyPI
# docker compose down -v # tear down + remove packages volume
#
# Consumer side (e.g. OCSASim):
# pip install --index-url http://localhost:8090/simple/ oshconnect
# uv pip install --index-url http://localhost:8090/simple/ oshconnect
# =============================================================================

services:
pypi:
image: pypiserver/pypiserver:latest
container_name: local-pypi
command: run -a . -P . -o
ports:
- "8090:8080"
volumes:
- pypi-packages:/data/packages
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://localhost:8080/"]
interval: 5s
timeout: 3s
start_period: 5s
retries: 3

volumes:
pypi-packages:
90 changes: 90 additions & 0 deletions docs/markdown/api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# API Reference

All public symbols are re-exported from the top-level package and can be
imported directly:

```python
from oshconnect import OSHConnect, Node, Datastream, TimePeriod, ObservationFormat
```

Lower-level CS API utilities are available from the `oshconnect.csapi4py`
sub-package:

```python
from oshconnect.csapi4py import APIResourceTypes, MQTTCommClient, ConnectedSystemsRequestBuilder
```

---

## Core Application

::: oshconnect.oshconnectapi

---

## Streamable Resources

The primary objects for interacting with systems, datastreams, and control
streams on an OSH node. Includes `Node`, `System`, `Datastream`,
`ControlStream`, and supporting enums.

::: oshconnect.streamableresource

---

## Resource Data Models

Pydantic models that represent CS API resources returned from or sent to an
OSH server.

::: oshconnect.resource_datamodels

---

## SWE Schema Components

Builder classes for constructing datastream and command schemas using SWE
Common data types.

::: oshconnect.swe_components

::: oshconnect.schema_datamodels

---

## Event System

Pub/sub event bus for in-process notifications. Implement `IEventListener`
to receive events.

::: oshconnect.eventbus

::: oshconnect.events.core

::: oshconnect.events.builder

---

## Time Management

::: oshconnect.timemanagement

---

## CS API Integration (`csapi4py`)

### Constants and Enums

::: oshconnect.csapi4py.constants

### Request Builder

::: oshconnect.csapi4py.con_sys_api

### API Helper

::: oshconnect.csapi4py.default_api_helpers

### MQTT Client

::: oshconnect.csapi4py.mqtt
93 changes: 93 additions & 0 deletions docs/markdown/architecture.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Architecture

OSHConnect is structured around a small number of long-lived objects that mirror
the resource hierarchy of the OGC API – Connected Systems specification.

## Object hierarchy

```mermaid
graph TD
OSHConnect[OSHConnect<br/><i>application entry point</i>]
Node[Node<br/><i>connection to one OSH server</i>]
APIHelper[APIHelper<br/><i>CS API HTTP requests</i>]
Session[SessionManager<br/><i>OSHClientSession instances</i>]
MQTT[MQTTCommClient<br/><i>paho-mqtt wrapper</i>]
System[System<br/><i>sensor system</i>]
Datastream[Datastream<br/><i>output channel — observations</i>]
ControlStream[ControlStream<br/><i>input channel — commands &amp; status</i>]

OSHConnect --> Node
Node --> APIHelper
Node --> Session
Node --> MQTT
Node --> System
System --> Datastream
System --> ControlStream
```

## Key abstractions

- **`OSHConnect`** (`oshconnectapi.py`) — top-level class. Owns nodes and
provides `discover_systems()`, `discover_datastreams()`,
`save_config()` / `load_config()`, and `create_and_insert_system()`.
- **`Node`** (`streamableresource.py`) — wraps a server connection. Drives
discovery via `APIHelper` and owns the `MQTTCommClient`. All HTTP resource
creation goes through here.
- **`StreamableResource`** (`streamableresource.py`) — abstract base for
`System`, `Datastream`, and `ControlStream`. Manages MQTT
subscriptions/publications, WebSocket connections, and the inbound /
outbound message deques. Connection modes: `PUSH`, `PULL`, `BIDIRECTIONAL`.
- **`Datastream` / `ControlStream`** (`streamableresource.py`) — concrete
streamable resources. Datastreams publish observations; ControlStreams
publish commands and receive status updates. Both follow CS API Part 3
topic conventions (`:data`, `:status`, `:commands`).
- **`resource_datamodels.py`** — Pydantic models for the CS API resource types
(`SystemResource`, `DatastreamResource`, `ControlStreamResource`,
`ObservationResource`). These map directly to API request and response
bodies.
- **`swe_components.py`** — Pydantic models for SWE Common schema components
(`DataRecordSchema`, `QuantitySchema`, `VectorSchema`, etc.). Used to define
observation and command schemas when creating new datastreams.
- **`csapi4py/`** — sub-package that handles the CS API specifics: URL
construction (`endpoints.py`), request building (`con_sys_api.py`), enums
(`constants.py`), and MQTT topic conventions (`mqtt.py`).
- **`EventHandler`** (`eventbus.py`) — singleton pub/sub bus. Listeners
subscribe to event types (e.g. `NEW_OBSERVATION`) and topic strings; events
are dispatched asynchronously through an internal queue.
- **`timemanagement.py`** — `TimeInstant` (epoch / ISO-8601), `TimePeriod`,
`TemporalModes` (`REAL_TIME`, `ARCHIVE`, `BATCH`), and `TimeUtils`
conversions.

## Typical data flow

```mermaid
sequenceDiagram
autonumber
participant App as OSHConnect
participant N as Node
participant H as APIHelper
participant S as Server
participant DS as Datastream

App->>N: add_node()
App->>N: discover_systems()
N->>H: retrieve_resource(SYSTEM)
H->>S: HTTP GET /systems
S-->>H: JSON
H-->>N: System objects
App->>DS: discover_datastreams()
DS->>DS: initialize() — open MQTT/WebSocket
DS->>DS: start() — begin streaming
S-->>DS: observations → _inbound_deque
Note over App,DS: To insert: resource.insert_self() →<br/>APIHelper.create_resource() → POST →<br/>server returns Location header with new ID
```

## Dependencies

- **pydantic** — all resource and schema models. Bumping the minimum requires
confirming pre-built wheels exist for all supported Python versions
(3.12 – 3.14).
- **shapely** — geometry handling for spatial resources.
- **paho-mqtt** — MQTT streaming for CS API Part 3.
- **websockets** / **aiohttp** — WebSocket and async HTTP streaming.
- **requests** — synchronous HTTP for discovery and resource creation.
24 changes: 24 additions & 0 deletions docs/markdown/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# OSHConnect-Python

OSHConnect-Python is the Python member of the OSHConnect family of application
libraries. It provides a simple, straightforward way to interact with
OpenSensorHub (or any other OGC API – Connected Systems server).

It supports Parts 1, 2, and 3 (Pub/Sub) of the OGC Connected Systems API,
including:

- System, Datastream, and ControlStream discovery and management
- Real-time MQTT streaming using CS API Part 3 `:data` topic conventions
- Resource event topic subscriptions (CloudEvents lifecycle notifications)
- Batch retrieval and archival stream playback
- Configuration persistence (JSON save/load)
- SWE Common schema builders for defining datastream and command schemas

All major classes and utilities are importable directly from `oshconnect`.
Lower-level CS API utilities are available from `oshconnect.csapi4py`.

## Where to next

- [Architecture](architecture.md) — object hierarchy, data flow, and key abstractions
- [Tutorial](tutorial.md) — common workflows for connecting, discovering, streaming, and inserting resources
- [API Reference](api.md) — auto-generated reference for every public symbol
Loading
Loading