Nautical Graph Toolkit Setup Guide¶

This guide explains how to set up S-57 Electronic Navigational Chart (ENC) data for use with the Nautical Graph Toolkit notebooks.

Overview¶

The Nautical Graph Toolkit (formerly Maritime Module) supports three backend options for storing and querying S-57 ENC data:

1. PostGIS - PostgreSQL database with spatial extensions (recommended for large datasets and server deployment)
2. GeoPackage - Portable single-file database format (.gpkg)
3. SpatiaLite - SQLite database with spatial extensions (.sqlite)

Quick Comparison¶

Quick Decision Guide:

• Choose PostGIS if you have server infrastructure and need concurrent access or large datasets
• Choose GeoPackage if you need portability and wide tool compatibility
• Choose SpatiaLite if you want minimal setup and lightweight deployment

Prerequisites¶

Required Software¶

• Python 3.11 or higher
• GDAL 3.10.3 (exact version pinned)
• SQLite with RTREE support (see "SQLite RTREE Requirement" below)
• For PostGIS backend:
  • PostgreSQL 16+ with PostGIS extension
  • psycopg2 Python package

SQLite RTREE Requirement¶

Critical: GeoPackage and SpatiaLite backends require SQLite with RTREE (R-tree spatial indexing) support for spatial queries.

Automatic Solution:

This project includes sqlite in environment.yml, which provides SQLite with RTREE support enabled on all platforms.

How it works:

• Conda's sqlite package provides RTREE-enabled SQLite library
• Python's built-in sqlite3 module uses the Conda SQLite library when the Conda environment is activated
• The sqlite package is included in environment.yml for cross-platform compatibility

Verification:

To verify RTREE support is available:

import sqlite3

conn = sqlite3.connect(':memory:')
conn.execute('CREATE VIRTUAL TABLE test USING rtree(id, minx, maxx, miny, maxy)')
print("✓ RTREE support is available")
conn.close()

If RTREE is not available:

# Reinstall Conda environment to ensure sqlite is included
mamba env update -f environment.yml --prune
mamba activate nautical

Platform Compatibility:

• Linux: ✅ Tested (AMD64)
• macOS ARM (M1/M4): ✅ Tested
• macOS Intel: ⏸️ Expected to work (not tested, same as Linux AMD64)
• Windows 11: ✅ Tested

Note: pysqlite3-binary package has compatibility issues on macOS ARM and Windows. Conda's sqlite package is used instead for consistent cross-platform support.

Why RTREE is required:

• SpatiaLite uses RTREE for spatial indexing (10-100x performance improvement)
• GeoPackage format requires RTREE for geometry column indexing
• Without RTREE, spatial queries will fail with "no such module: rtree" error

Required Data¶

• S-57 ENC Files: Electronic Navigational Charts in .000 format
• ENC Catalog: Directory structure containing S-57 base files

Backend Setup Instructions¶

Option 1: PostGIS Backend¶

When to use:

• Large datasets (1000+ ENC files)
• Server deployment with concurrent access
• Advanced spatial queries and analysis
• Multi-user environments

Key Features:

• Database-side spatial operations for optimal performance
• ACID compliance with transactional integrity
• Concurrent multi-user access with connection pooling
• Advanced spatial indexing (GiST, BRIN)
• Server-side graph creation for massive datasets

Prerequisites:

• PostgreSQL 16+ with PostGIS 3.5+ extension installed
• Network access to PostgreSQL server
• .env file configured with database connection parameters
• Sufficient disk space (estimate: ~2-3x raw ENC file size)
• psycopg2 Python package

Setup Steps:

1. Start Docker PostGIS — Follow the Installation Guide: Docker PostGIS Setup for platform-specific instructions (Linux, macOS ARM, Windows).

2. Configure Environment Variables

   # .env file configuration
   # Copy from .env.example and customize:
   
   # PostgreSQL/PostGIS Database Configuration
   DB_NAME=enc_db
   DB_USER=postgres
   DB_PASSWORD=your_secure_password
   DB_HOST=localhost
   DB_PORT=5432
   
   # Optional: Mapbox Token for Interactive Maps (get from https://account.mapbox.com/)
   MAPBOX_TOKEN=your_mapbox_access_token_here
   

3. Import S-57 Data

   # Import S-57 data using the notebook
   
   # Run: docs/notebooks/import_s57.ipynb
   #   - Set backend = 'postgis'
   #   - Set methods = {'S57_Advanced': True}
   #   - Configure database schema name
   #   - Run all cells
   
   # The notebook includes detailed instructions and verification steps.
   # For script-based imports, see: docs/user-guides/scripts-guide.md
   

Required Schema Structure:

• Main schema: enc_west (or your custom schema name)
• Required layers: See "Required Layers" section below

Option 2: GeoPackage Backend¶

When to use:

• Moderate datasets (100-1000 ENC files)
• Portable single-file database needed
• Cross-platform compatibility required
• Desktop/laptop development

Key Features:

• Single-file portability - easy backup and sharing
• OGC standard format with wide tool support (QGIS, ArcGIS, etc.)
• Built-in spatial indexing (R-tree)
• No server required - works offline
• Cross-platform compatibility (Windows, Linux, macOS)

Prerequisites:

• GDAL 3.10.3 with GeoPackage driver
• Write permissions to output directory
• Sufficient disk space (estimate: ~1.5-2x raw ENC file size)
• pyogrio or fiona Python package for I/O

Setup Steps:

Import S-57 Data

# Import S-57 Data using the notebook
# Run: docs/notebooks/import_s57.ipynb
#   - Set backend = 'gpkg'
#   - Set output filename to 'enc_west.gpkg'
#   - Run all cells

# Output directory and file are created automatically
# Output: data/enc_west.gpkg
# For script-based imports, see: docs/user-guides/scripts-guide.md

File Location:

• Default: data/enc_west.gpkg
• Customizable via notebook configuration

Option 3: SpatiaLite Backend¶

When to use:

• Small to moderate datasets (< 500 ENC files)
• Lightweight deployment
• Minimal dependencies
• Testing and development

Key Features:

• Minimal footprint - SQLite-based with spatial extensions
• No server infrastructure required
• Fast read performance for smaller datasets
• Embedded database - zero configuration
• SQL spatial query support

Prerequisites:

• GDAL 3.10.3 with SQLite/SpatiaLite driver
• Write permissions to output directory
• Sufficient disk space (estimate: ~1.5-2x raw ENC file size)
• pyspatialite or sqlite3 Python package
• Note: May show harmless fiona warnings for certain S-57 field types

Setup Steps:

Import S-57 Data

# Import S-57 Data using the notebook
# Run: docs/notebooks/import_s57.ipynb
#   - Set backend = 'spatialite'
#   - Set output filename to 'enc_west.sqlite'
#   - Run all cells

# Output directory and file are created automatically
# Output: data/enc_west.sqlite
# For script-based imports, see: docs/user-guides/scripts-guide.md

File Location:

• Default: data/enc_west.sqlite
• Customizable via notebook configuration

Required Layers¶

After importing S-57 data, your backend must contain the following layers:

Essential Layers (Required for Basic Operations)¶

Navigation Layers (Required for Graph Creation)¶

Obstacle Layers (Required for Safety)¶

Additional Layers (Optional but Recommended)¶

Data Import Process¶

Note: Detailed import instructions will be added after completing the import_s57 notebook documentation.

Quick Start¶

1. Organize your S-57 files

   ENC_ROOT/
   ├── US1AK01M/
   │   └── US1AK01M.000
   ├── US2AK02M/
   │   └── US2AK02M.000
   └── ...
   

2. Run the import notebook

   • Run docs/notebooks/import_s57.ipynb for any backend (PostGIS, GeoPackage, or SpatiaLite)
   • Configure the notebook to select your desired backend

3. Verify the import

   # Verification after import
   
   # The import_s57.ipynb notebook includes verification steps at the end.
   
   # For comprehensive verification and testing:
   #
   #   docs/notebooks/enc_factory.ipynb - Quick verification
   #     - ENC summary and metadata
   #     - Layer feature counts
   #     - NOAA version checking
   #     - Bounding box visualization
   #
   #   docs/notebooks/import_deeptest.ipynb - Comprehensive integration testing
   #     - Full pipeline testing across all backends
   #     - Multi-backend validation and comparison
   #     - Data integrity and DSID stamping verification
   #     - Performance baseline tracking
   

Alternative: Download Pre-Imported Data¶

Skip the import process by downloading pre-processed ENC databases — see the Data Guide for the full catalogue, download steps, and use-case recommendations. Time saved: ~40-60 minutes.

Verification¶

After setup, verify your backend contains the required data:

PostGIS Verification¶

-- PostGIS Verification

-- For verification queries and testing:
--
--   docs/notebooks/enc_factory.ipynb - Quick verification
--     - Schema and table verification
--     - Feature counts by layer
--     - DSID stamping validation
--     - Spatial extent checking
--
--   docs/notebooks/import_deeptest.ipynb - Comprehensive testing
--     - Full S-57 pipeline testing
--     - Cross-backend consistency validation
--     - Integration test reports (JSON/CSV/TXT)

GeoPackage/SpatiaLite Verification¶

# GeoPackage/SpatiaLite Verification

# For verification and testing:
#
#   docs/notebooks/enc_factory.ipynb - Quick verification
#     - Supports all backends (PostGIS, GeoPackage, SpatiaLite)
#     - Layer summaries and feature counts
#     - Quick data quality checks
#
#   docs/notebooks/import_deeptest.ipynb - Comprehensive testing
#     - Full integration testing across all backends
#     - Cross-backend consistency validation
#     - Test reports with detailed results

Python Interpreter Path Verification¶

To verify your Python environment location (required for IDE configuration and Jupyter kernels):

# Activate environment first
mamba activate nautical

# Check Python executable path
python -c "import sys; print(f'Python executable: {sys.executable}')"

Expected output by platform:

• Windows: C:\Users\<YourUser>\.local\share\mamba\envs\nautical\python.exe
• Linux: /home/<user>/miniforge3/envs/nautical/bin/python
• macOS: /Users/<user>/miniforge3/envs/nautical/bin/python

Note the path from this command - you'll need it for IDE configuration and creating Jupyter kernels.

Jupyter Notebook Configuration (Optional)¶

If you plan to use Jupyter notebooks for interactive analysis, follow these steps to set up a Jupyter kernel for the nautical environment.

Creating a Jupyter Kernel¶

The Jupyter kernel allows notebooks to use your nautical environment directly:

# Activate environment
mamba activate nautical

# Create kernel (works on all platforms)
python -m ipykernel install --user --name nautical --display-name "Nautical Toolkit"

# Verify kernel installation
jupyter kernelspec list

Expected output:

Available kernels:
  nautical    C:\Users\<YourUser>\AppData\Roaming\jupyter\kernels\nautical    # Windows
  nautical    ~/.local/share/jupyter/kernels/nautical                         # Linux/macOS
  python3     ...

Using the Kernel in Jupyter¶

Jupyter Notebook:

mamba activate nautical
jupyter notebook

# In browser: Kernel → Change Kernel → Nautical Toolkit

Jupyter Lab:

mamba activate nautical
jupyter lab

# In browser: Select "Nautical Toolkit" when creating new notebooks

VS Code:

1. Open a .ipynb file
2. Click kernel selector (top-right corner)
3. Select "Nautical Toolkit" from the list

PyCharm Professional:

1. Open a notebook or Python file
2. Run → Edit Configurations → Jupyter Server (if using notebooks)
3. Select "existing" → Choose "nautical" kernel

IDE Python Interpreter Configuration¶

PyCharm:

1. File → Settings → Project: Nautical-Graph-Toolkit → Python Interpreter
2. Click gear icon → Add
3. Select "Conda Environment" → "Existing"
4. Paste the Python path from the verification command above (e.g., C:\Users\<YourUser>\.local\share\mamba\envs\nautical\python.exe)
5. Apply → OK

VS Code:

1. Open Command Palette (Ctrl+Shift+P / Cmd+Shift+P on macOS)
2. Type "Python: Select Interpreter"
3. Click "Enter interpreter path..."
4. Paste the Python path from the verification command above
5. Press Enter

Troubleshooting Jupyter Kernel Issues¶

If you encounter issues with Jupyter kernels, see TROUBLESHOOTING.md - Jupyter Kernel Issues for comprehensive solutions including:

• Kernel not found in Jupyter
• IDE shows wrong Python version
• Kernel dies immediately when starting

Configuration Files¶

Environment Variables (.env)¶

# PostGIS Configuration
DB_NAME=enc_db
DB_USER=your_username
DB_PASSWORD=your_secure_password
DB_HOST=localhost
DB_PORT=5432

# Optional: Mapbox Token for Interactive Maps
MAPBOX_TOKEN=your_mapbox_access_token_here

Graph Configuration (graph_config.yml)¶

The graph configuration file defines which layers to use for navigation:

# Graph Configuration (src/nautical_graph_toolkit/data/graph_config.yml)

# Graph Type Selection
graph_type: "h3"  # Options: "h3" (hexagonal) or "fine" (grid)

# Output Configuration
output_gpkg: "graphs/h3_graph.gpkg"

# Layer Configuration (by usage band and resolution)
layers:
  navigable:
    # Low-resolution layers (bands 1-2) for broad areas
    - { layer: "seaare", bands: [1, 2], resolution: 6 }

    # Medium-resolution layers (band 3) for approach areas
    - { layer: "seaare", bands: [3], resolution: 9 }
    - { layer: "fairwy", bands: [3], resolution: 9 }

    # High-resolution layers (bands 4-6) for harbors and channels
    - { layer: "drgare", bands: [3, 4, 5, 6], resolution: 11 }
    - { layer: "fairwy", bands: [4, 5, 6], resolution: 11 }
    - { layer: "tsslpt", bands: [3, 4, 5, 6], resolution: 11 }
    - { layer: "prcare", bands: [3, 4, 5, 6], resolution: 11 }

  obstacles:
    # Obstacle layers (all bands, no resolution needed)
    - { layer: "lndare", bands: [1, 2, 3, 4, 5, 6], resolution: null }
    - { layer: "slcons", bands: [1, 2, 3, 4, 5, 6], resolution: null }
    - { layer: "uwtroc", bands: [3, 4, 5, 6], resolution: null }
    - { layer: "obstrn", bands: [3, 4, 5, 6], resolution: null }

# Weighting Configuration (see config/workflow_config.yml for details)

Notebook-Specific Requirements¶

Basic Graph Creation Notebooks¶

Required for:

• graph_PostGIS_v2.ipynb
• graph_GeoPackage_v2.ipynb
• graph_SpatiaLite_v2.ipynb

Data Requirements:

• Imported S-57 data in chosen backend
• Minimum layers: seaare, lndare, fairwy, drgare, tsslpt, prcare
• Port data (included with package)

Fine Graph Creation Notebooks¶

Required for:

• graph_fine_GeoPackage_v2.ipynb
• graph_fine_PostGIS_v2.ipynb

Data Requirements:

• All basic graph requirements
• Additional obstacle layers: slcons, uwtroc, obstrn
• Higher resolution ENC data recommended

H3 Graph Creation Notebooks¶

Required for:

• graph_h3_GPKG_v2.ipynb (future)
• graph_h3_PostGIS_v2.ipynb (future)

Data Requirements:

• All basic graph requirements
• H3 Python package installed
• Sufficient memory for hexagon generation

Troubleshooting¶

Common Issues¶

Issue: "Layer not found" error

Solution: Verify layer exists in your backend using verification steps above

Issue: "Database connection failed" (PostGIS)

Solution: Check .env file credentials and verify PostgreSQL is running

Issue: "No ENCs found in boundary"

Solution: Verify ENC data was imported correctly and covers your area of interest

Issue: Import process is very slow

Solution:
- For PostGIS: Ensure proper indexing (see import notebook)
- For file backends: Use SSD storage
- Consider reducing number of ENCs for initial testing

Performance Optimization¶

PostGIS:

-- PostGIS Optimization

-- Spatial indexes are automatically created during import.
-- No additional optimization needed for typical notebook usage.

-- For production optimization, see:
--   docs/user-guides/scripts-guide.md
--   config/workflow_config.yml

GeoPackage/SpatiaLite:

# GeoPackage/SpatiaLite Optimization

# Spatial indexes are automatically created during import.

# Tips for better notebook performance:
# - Use fiona engine (default): geopandas.read_file(..., engine="fiona")
# - Filter with bbox parameter to load only needed data
# - Store files on SSD for best performance
# - For large datasets (>500 ENCs), consider PostGIS backend

# For production optimizations, see: docs/user-guides/scripts-guide.md

Next Steps¶

After completing setup:

1. Start with basic graph notebooks to verify your data is working
2. Review the graph configuration (src/nautical_graph_toolkit/data/graph_config.yml)
3. Explore advanced features in fine graph and H3 graph notebooks
4. Customize port data if needed (src/nautical_graph_toolkit/data/custom_ports.csv)

Additional Resources¶

• S-57 Standard Documentation: https://iho.int/en/standards/s-57-ecdis
• NOAA ENC Download: https://charts.noaa.gov/ENCs/ENCs.shtml
• PostGIS Documentation: https://postgis.net/documentation/
• GeoPackage Specification: https://www.geopackage.org/
• SpatiaLite Documentation: https://www.gaia-gis.it/fossil/libspatialite/

Support¶

For issues or questions:

1. Check the troubleshooting section above
2. Review the relevant notebook documentation
3. Open an issue on the project repository