Maritime Graph Workflow - PostGIS Backend¶

Overview¶

The Maritime Graph Workflow is a comprehensive Python script that orchestrates a complete maritime navigation graph pipeline using PostGIS as the backend. It automates the entire process from raw S-57 Electronic Navigational Chart (ENC) data to optimized, weighted navigation graphs ready for routing.

What It Does¶

The workflow performs four major steps:

1. Base Graph Creation (0.3 NM resolution)

   • Defines geographic area of interest between two ports
   • Filters ENC charts to the relevant region
   • Creates navigable water grid from S-57 layers
   • Generates initial graph structure
   • Computes baseline route

2. Fine/H3 Graph Creation (0.02-0.3 NM or hexagonal)

   • Focuses on route buffer around base route
   • Creates high-resolution graph for detailed routing
   • Two modes: regular grid ("fine") or hexagonal ("h3")
   • Supports multi-resolution optimization

3. Graph Weighting (dynamic weight calculation)

   • Converts graph to directed edges
   • Enriches edges with S-57 feature attributes
   • Applies three-tier weighting system:
     • Static weights: Distance-based penalties/bonuses from geographic features
     • Directional weights: Traffic flow alignment rewards/penalties
     • Dynamic weights: Vessel-specific constraints (draft, height)
   • Creates final routing weights

4. Pathfinding & Export

   • Loads weighted graph
   • Calculates optimal route using A* algorithm
   • Exports route to GeoJSON for visualization
   • Optional: exports weighted graph to GeoPackage

Prerequisites¶

Required Software¶

• Python 3.11+
• PostgreSQL with PostGIS extension
• GDAL 3.10.3 (for S-57 conversion)
• All dependencies listed in environment.yml and requirements.in

Required Data¶

• S-57 ENC charts in PostGIS database
• Two ports defined (World Port Index or custom)
• .env file with database credentials

Database Setup¶

Start Docker PostGIS following the Installation Guide, then verify:

# Check connection
psql -h localhost -U postgres -d enc_db -c "SELECT version();"

# Check S-57 schema exists
psql -h localhost -U postgres -d enc_db -c "SELECT * FROM information_schema.schemata WHERE schema_name = 'enc_west';"

Installation & Setup¶

1. Clone/Download the Project¶

cd ~/Nautical-Graph-Toolkit  # or wherever you cloned the project

2. Install Dependencies¶

mamba env update -f environment.yml --prune
pip install uv
uv pip compile requirements.in -o requirements.txt  # Optional: skip to use tested snapshot
uv pip install --no-deps -r requirements.txt
uv pip install -e .

3. Configure Database Credentials¶

Edit .env file:

# .env
DB_NAME="enc_db"
DB_USER="postgres"
DB_PASSWORD="your_secure_password"
DB_HOST="127.0.0.1"
DB_PORT="5432"
MAPBOX_TOKEN="your_mapbox_token"

4. Review Workflow Configuration¶

The script uses two configuration files:

• config/workflow_config.yml - Workflow orchestration (ports, AOI, steps)
• src/nautical_graph_toolkit/data/graph_config.yml - Graph parameters (layers, weights, H3)

Configuration Guide¶

workflow_config.yml¶

Workflow Control¶

workflow:
  run_base_graph: true       # Step 1: Create base graph
  run_fine_graph: true       # Step 2: Create fine/H3 graph
  run_weighting: true        # Step 3: Apply weighting
  run_pathfinding: true      # Step 4: Calculate routes

Base Graph Configuration¶

base_graph:
  departure_port: "Los Angeles"
  arrival_port: "San Francisco"
  expansion_nm: 24.0         # Buffer around ports (NM)
  spacing_nm: 0.3            # Node spacing for base graph
  layer_table: "seaare"      # Primary navigable layer

Fine/H3 Graph Configuration¶

fine_graph:
  # Naming: Graphs auto-generated as {mode}_graph_{name_suffix}
  #   - Undirected: h3_graph_20 or fine_graph_20
  #   - Weighted: h3_graph_wt_20 or fine_graph_wt_20
  mode: "h3"                 # "fine" (grid) or "h3" (hexagonal)
  name_suffix: "20"          # Change to customize all graph names

  buffer_size_nm: 24.0       # Buffer around base route

  # Fine grid specific (if mode: "fine")
  fine_spacing_nm: 0.02      # Dense grid spacing
  fine_bridge_components: true

  # Output options
  save_gpkg: true            # Save to GeoPackage
  save_postgis: false        # Save to PostGIS

Weighting Configuration¶

weighting:
  # IMPORTANT: Graph names are automatically constructed from fine_graph config:
  #   - source (undirected): {mode}_graph_{name_suffix}
  #   - target (directed): {mode}_graph_wt_{name_suffix}
  # Examples (fine_graph.mode="h3", fine_graph.name_suffix="20"):
  #   - source: h3_graph_20
  #   - target: h3_graph_wt_20
  # Do NOT manually set these; they are auto-generated.

  # Weight manager class: "weights" (standard) or "weights-open" (ML-optimized)
  weights_class: "weights"

  steps:
    convert_to_directed: true
    enrich_features: true
    apply_static_weights: true
    apply_directional_weights: true
    apply_dynamic_weights: true

  # Vessel parameters (determine navigable water and clearance requirements)
  vessel:
    draft: 7.5                    # Depth of hull below waterline (meters)
    height: 30.0                  # Height above waterline (meters)
    beam: 25.0                    # Vessel width (meters)
    length: 150.0                 # Vessel length (meters)
    vessel_type: "cargo"          # cargo, tanker, passenger, fishing
    ukc_safety_margin: 2.0        # Under-keel clearance buffer (meters)
    ver_clearance_margin: 5.0     # Vertical clearance buffer (meters)
    hor_clearance_margin: 20.0    # Horizontal clearance buffer (meters)
    compliance_zone: null         # null = use defaults, or [2.5, 1.8, 1.3]

  environment:
    weather_factor: 1.2           # 1.0=good, >1.0=poor (weight multiplier)
    visibility_factor: 1.1        # 1.0=good, >1.0=poor (weight multiplier)
    time_of_day: "day"            # "day" or "night"

  # Feature enrichment options
  enrichment:
    include_sources: false        # Store ENC source chart names in edge attributes
    soundg_buffer_meters: 30      # Radius for sounding point spatial queries (meters)
    work_mem: "512MB"             # PostGIS work memory for spatial joins

  # Static weights processing
  static_weights_usage_bands: [3, 4, 5]  # ENC chart bands contributing to weighting
  buffer_method: "auto"           # "auto", "degrees", or "geodesic"
  aggr_mode: exp                  # null (graph_config default), "max", or "exp"

  # Buffer zone visualization (environmental/regulatory coastal boundaries)
  buffer_zones:
    enabled: false                # Enable buffer zone classification
    save_buffer_zones: false      # Persist buffer zone geometries as separate tables

Pathfinding Configuration¶

pathfinding:
  departure_port: "SF Pilot"
  departure_coords: {lon: -122.780, lat: 37.006}
  arrival_port: "San Francisco Arrival"
  arrival_coords: {lon: -122.400, lat: 37.805}
  weight_key: "adjusted_weight"

  # A* algorithm selection
  astar_impl: "AstarMaritime"      # "Astar", "AstarImproved", or "AstarMaritime"

  # AstarMaritime-specific parameters
  corridor_buffer_nm: 5.0          # Corridor width for Pass-2 optimization (NM)
  include_tss: true                # Force-include TSS lanes in corridor
  tss_bbox_extend_factor: 0.5      # Extend TSS detection bbox

  # General pathfinding parameters
  min_cost_factor: 1.0             # Minimum cost threshold
  collect_edge_stats: true         # Collect detailed edge statistics

  # Output options
  route_filename_template: "detailed_route_{draft}m_draft.geojson"
  export_weighted_graph: false     # Export weighted graph to GeoPackage

graph_config.yml¶

This file (in src/nautical_graph_toolkit/data/) defines graph generation parameters:

• Navigable layers: Which S-57 features define safe water (seaare, fairwy, etc.)
• Obstacle layers: Which features are hazards (lndare, slcons, etc.)
• H3 settings: Hexagon resolution mapping, connectivity rules
• Weight settings: Vessel types, static layer configurations, directional weights

Typically no changes needed, but can be customized for specialized use cases.

Usage¶

Basic Commands¶

Full Pipeline (All Steps)¶

python scripts/maritime_graph_postgis_workflow.py

Skip Base Graph (Already Created)¶

python scripts/maritime_graph_postgis_workflow.py --skip-base

Use Fine Grid Instead of H3¶

python scripts/maritime_graph_postgis_workflow.py --graph-mode fine

Custom Configuration File¶

python scripts/maritime_graph_postgis_workflow.py --config custom_workflow_config.yml

Override Vessel Draft¶

python scripts/maritime_graph_postgis_workflow.py --vessel-draft 10.5

Dry Run (Validate Only, No Execution)¶

python scripts/maritime_graph_postgis_workflow.py --dry-run

Debug Logging (Verbose Console Output)¶

# INFO mode (default): Clean logs, ~1MB per log file
python scripts/maritime_graph_postgis_workflow.py --log-level INFO

# DEBUG mode: Comprehensive debugging, ~5-10MB per log file
# Third-party verbose logging (Fiona, GDAL) automatically suppressed
python scripts/maritime_graph_postgis_workflow.py --log-level DEBUG

Note: Log files now include:

• Automatic rotation: Max 50MB (INFO) or 500MB (DEBUG) per file, 3 backups
• Third-party suppression: Fiona/GDAL DEBUG logs filtered out (99% size reduction)
• Project-level logs: Full debug info for nautical_graph_toolkit modules

Command-Line Options¶

--config PATH                       Path to workflow config YAML
--output-dir PATH                   Output directory (default: auto-generated output/workflow_{graph}_{timestamp}/)
--graph-mode {fine,h3}              Override graph mode
--name-suffix SUFFIX                Override fine_graph.name_suffix (affects graph and output directory names)
--skip-base                         Skip base graph creation
--skip-fine                         Skip fine/H3 graph creation
--skip-weighting                    Skip weighting steps
--skip-pathfinding                  Skip final pathfinding
--vessel-draft FLOAT                Override vessel draft (meters)
--weights-class {weights,weights-open}  Weight manager class (default: "weights")
--log-level {INFO,DEBUG}            Console logging level
--dry-run                           Validate config without execution

Example Workflows¶

Scenario 1: Full Pipeline (Default)¶

python scripts/maritime_graph_postgis_workflow.py

# Expected time: 45-60 minutes
# Output directory: output/workflow_{graph}_{timestamp}/ (auto-generated)
#   - PostGIS tables: base_graph_PG, h3_graph_pg_6_11, h3_graph_directed_pg_6_11_v3
#   - GeoPackage files: base_graph_PG.gpkg, h3_graph_PG_6_11.gpkg
#   - Routes: GeoJSON files with detailed route segments
#   - Logs: maritime_workflow_20251027_142310.log
#   - Benchmarks: benchmark_graph_*.csv

Scenario 2: Resume from Fine Graph (Skip Base)¶

python scripts/maritime_graph_postgis_workflow.py --skip-base

# Use when base graph already exists in PostGIS
# Expected time: 20-30 minutes

Scenario 3: Fine Grid Mode (Regular Grid)¶

python scripts/maritime_graph_postgis_workflow.py --graph-mode fine

# Creates regular rectangular grid instead of hexagonal
# Faster processing, suitable for most use cases
# Expected time: 15-25 minutes

Scenario 4: Custom Vessel Specifications¶

python scripts/maritime_graph_postgis_workflow.py \
  --vessel-draft 12.0 \
  --skip-base

# Routes optimized for vessel with 12m draft
# Different shallow areas may be avoided

Scenario 5: Debug & Testing¶

# Validate configuration
python scripts/maritime_graph_postgis_workflow.py --dry-run

# Run with verbose logging
python scripts/maritime_graph_postgis_workflow.py --log-level DEBUG

# Run only weighting and pathfinding steps
python scripts/maritime_graph_postgis_workflow.py --skip-base --skip-fine

Scenario 6: ML-Optimized Weights (WeightsOpen)¶

python scripts/maritime_graph_postgis_workflow.py \
  --weights-class weights-open \
  --skip-base

# Uses WeightsOpen class which produces per-layer flat tracking columns
# (wt_{layer_name}, wt_{layer_name}_n) suitable for GNN/PyTorch consumption
# Output includes additional feature columns: ft_sounding_wrecks, ft_sounding_obstrn

Output Files¶

Database Tables (PostGIS)¶

Step 1: Base Graph¶

graph.base_graph_PG_nodes      - Node geometries
graph.base_graph_PG_edges      - Edge geometries and attributes
routes.base_routes             - Baseline route

Step 2: Fine/H3 Graph¶

graph.{mode}_graph_{suffix}_nodes   - High-resolution node geometries
graph.{mode}_graph_{suffix}_edges   - High-resolution edge geometries
                                      (e.g., h3_graph_20_nodes, h3_graph_20_edges)

Note: Names automatically constructed from config: {mode}_graph_{name_suffix}

• Example: fine_graph.mode="h3" + fine_graph.name_suffix="20" → h3_graph_20

Step 3: Weighted Graph¶

graph.{mode}_graph_wt_{suffix}_nodes    - Directed node geometries
graph.{mode}_graph_wt_{suffix}_edges    - Directed edges with weights:
                                          (e.g., h3_graph_wt_20_nodes, h3_graph_wt_20_edges)
  - weight: Original distance (NM)
  - adjusted_weight: Final routing weight
  - wt_static_blocking: Hazard penalties
  - wt_static_penalty: Warning penalties
  - wt_static_bonus: Preferred route bonuses
  - wt_dir: Traffic flow alignment weight
  - ft_*: S-57 feature attributes (depth, clearance, etc.)

GeoPackage Files¶

output/workflow_{graph}_{timestamp}/base_graph_PG.gpkg
output/workflow_{graph}_{timestamp}/h3_graph_PG_6_11.gpkg
output/workflow_{graph}_{timestamp}/h3_graph_directed_pg_6_11_v3.gpkg

• Auto-generated timestamped output directories (e.g., output/workflow_h3_20_20260416_142310/)
• Use --output-dir to override with a custom path
• Portable offline format, can be opened in QGIS, ArcGIS, etc.
• Contains nodes and edges layers with all attributes

Route Files¶

output/workflow_{graph}_{timestamp}/detailed_route_7.5m_draft.geojson

• GeoJSON format with route segments
• Each segment includes:
  • Geometry (line segment)
  • Edge attributes (weight, distance, features)
  • Cumulative distance and weight

Log Files¶

docs/logs/maritime_workflow_20251027_142310.log
docs/logs/maritime_workflow_20251027_142310.log.1  # Rotated backup (if exceeded size)
docs/logs/maritime_workflow_20251027_142310.log.2  # Rotated backup
docs/logs/maritime_workflow_20251027_142310.log.3  # Rotated backup

• Timestamped log file with automatic rotation
• Size limits: 50MB (INFO mode) or 500MB (DEBUG mode) per file
• Backup count: Keeps 3 old log files automatically
• Contains all SQL queries and operations (third-party DEBUG logs suppressed)
• Full stack traces for errors
• Useful for debugging and performance analysis
• 99% smaller than previous versions due to third-party log suppression

Benchmark Files¶

output/workflow_{graph}_{timestamp}/benchmark_graph_base.csv
output/workflow_{graph}_{timestamp}/benchmark_graph_fine.csv
output/workflow_{graph}_{timestamp}/benchmark_graph_weighted_directed.csv

• Performance metrics in CSV format
• Columns: timestamp, node_count, edge_count, timing for each step
• Used to track performance across runs

Performance Expectations¶

Typical Execution Times (Los Angeles - San Francisco)¶

Latest Performance Metrics: Comprehensive benchmark across three graph modes (47 S-57 ENCs)

Performance Breakdown Analysis¶

Time Distribution by Step:

Key Insights:

• 🚀 Database-side operations: PostGIS spatial indexing provides 2.0-4.2× speedup vs GeoPackage
• 📊 Weighting efficiency: Database-side spatial queries dramatically reduce enrichment time
• ⚡ Best for production: Optimal performance for large-scale deployments (>500K nodes)
• 🎯 Recommended mode: FINE 0.1nm provides best balance (21.3 min, detailed routes)
• 📈 Scaling: Weighting scales superlinearly (4× nodes → 4.7× weighting time)

PostGIS Performance Advantages¶

Why PostGIS Outperforms GeoPackage:

• Server-based spatial indexing: R-tree indexes optimized for large datasets
• Database-side operations: Geometry operations avoid Python/file I/O overhead
• Concurrent queries: Parallel edge enrichment processing
• Memory management: Better handling of multi-million edge graphs
• Query optimization: PostgreSQL query planner optimizes complex spatial joins

Recommended Configurations¶

Mode Selection Guide:

• FINE 0.2nm: Best for rapid iteration, testing workflow changes, demonstrations
• FINE 0.1nm: Production sweet spot - detailed enough for safe routing, fast enough for regular updates
• H3 Hexagonal: When you need multi-resolution capabilities or uniform cell sizes for analysis

Performance Scaling Analysis¶

Time per Million Nodes:

• FINE 0.2nm: 9.54 ms/node (smallest graph, less efficient)
• FINE 0.1nm: 6.92 ms/node (most efficient)
• H3 Hexagonal: 7.15 ms/node (good efficiency at scale)

Weighting Step Scaling:

• FINE 0.2nm → 0.1nm: 4× nodes → 4.7× weighting time
• FINE 0.1nm → H3: 4.8× nodes → 6.4× weighting time
• Conclusion: Superlinear scaling, but PostGIS handles it efficiently

When to Use PostGIS vs GeoPackage¶

Choose PostGIS when:

• ✅ Production deployment with server infrastructure
• ✅ Multi-user environment (concurrent route calculations)
• ✅ Large datasets (>500K nodes, frequent updates)
• ✅ Time-critical workflows (weighting speed matters)
• ✅ Professional deployment (reliability, scalability)

Choose GeoPackage when:

• ✅ Single-user or testing environment
• ✅ Portable/offline operation required
• ✅ No server infrastructure available
• ✅ Moderate dataset size (<500K nodes)
• ✅ File-based sharing needs (USB, cloud storage)

Weighting Performance Breakdown¶

Real Metrics (FINE 0.1nm - 184,637 nodes → 1,460,324 edges):

Optimization Strategies:

• Use --skip-base to resume from fine graph (saves ~3.2 min)
• Use --skip-base --skip-fine to resume from weighting (saves ~5 min)
• FINE 0.2nm mode if weighting time is critical constraint
• Database tuning: Ensure PostGIS spatial indexes exist on all geometry columns

Performance Tips¶

• First run: Base graph creation is expensive but runs once
• Resume workflow: Use --skip-base to skip existing base graph (saves 3-5 min)
• Incremental updates: Use S57Updater to refresh only changed ENCs
• Fine grid mode: Use FINE instead of H3 when speed matters (4-6× faster Fine step)
• Database tuning: Verify spatial indexes: SELECT * FROM pg_indexes WHERE tablename LIKE '%graph%';
• Smaller area: Use slice buffer to reduce geographic scope for testing

Troubleshooting¶

Common Issues¶

1. Database Connection Error¶

Error: Failed to initialize database: could not connect to server

Solution:

• Check PostgreSQL is running: sudo systemctl status postgresql
• Verify credentials in .env file
• Test connection: psql -h localhost -U postgres -d ENC_db

2. Missing Schema or Tables¶

Error: schema "enc_west" does not exist

Solution:

• S-57 data not loaded into PostGIS
• Convert S-57 ENCs first: See docs/getting-started/setup.md
• Verify schema name in config/workflow_config.yml

3. Port Not Found¶

Error: Could not find departure or arrival port

Solution:

• Check port names in config (must be in World Port Index or custom ports)
• List available ports: Query port_data.csv
• Add custom port in config with explicit coordinates

4. Out of Memory Error¶

MemoryError during graph creation

Solution:

• Reduce fine grid spacing in config
• Use H3 mode (more memory-efficient than fine grid)
• Slice buffer to smaller area
• Increase system RAM or use smaller area of interest

5. Graph Not Connected¶

Warning: H3 graph is not connected. Selecting the largest component.

Solution:

• Normal warning for multi-resolution graphs
• Pathfinding may fail if start/end in different components
• Try different vessel parameters or smaller area

Debugging Steps¶

1. Run dry-run first:

   python scripts/maritime_graph_postgis_workflow.py --dry-run
   

2. Check logs:

   tail -f docs/logs/maritime_workflow_*.log
   

3. Verify PostGIS setup:

   psql -d enc_db -c "SELECT postgis_version();"
   

4. Test with verbose logging:

   python scripts/maritime_graph_postgis_workflow.py --log-level DEBUG
   

5. Check intermediate outputs:

   • Base graph in PostGIS: SELECT COUNT(*) FROM graph.base_graph_PG_nodes;
   • Routes saved: SELECT COUNT(*) FROM routes.base_routes;

Advanced Topics¶

Custom Graph Configurations¶

Edit src/nautical_graph_toolkit/data/graph_config.yml to customize:

• Layer definitions: Add/remove navigable or obstacle layers
• Weight settings: Adjust static layer weights and factors
• H3 settings: Change hexagon resolution ranges
• Directional weights: Modify angle bands and weight factors

Example:

layers:
  navigable:
    - {layer: "seaare", bands: [1, 2, 3], resolution: 6}
    - {layer: "fairwy", bands: "all", resolution: 11}
  obstacles:
    - {layer: "lndare", bands: "all", resolution: null}

Resuming Partial Pipelines¶

The workflow can resume from any intermediate step:

# Create only weighted graph (skip graph creation)
python scripts/maritime_graph_postgis_workflow.py --skip-base --skip-fine

# Recalculate weights (graph already exists)
python scripts/maritime_graph_postgis_workflow.py --skip-base --skip-fine

Using Custom Port Coordinates¶

Override port definitions in config/workflow_config.yml:

base_graph:
  departure_port: "Custom Port"
  departure_coords: {lon: -122.789, lat: 37.005}
  arrival_port: "Custom Destination"
  arrival_coords: {lon: -122.400, lat: 37.805}

Exporting for External Analysis¶

The workflow generates exportable formats:

• GeoPackage: Open in QGIS for visualization/analysis
• GeoJSON: Import to web mapping libraries (Leaflet, Mapbox)
• PostGIS: Query with SQL for custom analysis
• CSV: Benchmark data for performance tracking

Performance Benchmarking¶

The script automatically generates benchmark CSVs:

# View benchmarks (output directory is printed at workflow startup)
cat output/workflow_h3_20_20260416_142310/benchmark_graph_base.csv
cat output/workflow_h3_20_20260416_142310/benchmark_graph_fine.csv

Compare across runs:

# Run new pipeline
python scripts/maritime_graph_postgis_workflow.py

# Analyze performance trends (replace with your output directory)
python -c "
import pandas as pd
df = pd.read_csv('output/workflow_h3_20_20260416_142310/benchmark_graph_fine.csv')
print(df[['timestamp', 'node_count', 'edge_count', 'total_pipeline_sec']])
"

Support & Documentation¶

Related Files¶

• Script: scripts/maritime_graph_postgis_workflow.py
• Configuration: config/workflow_config.yml
• Graph Config: src/nautical_graph_toolkit/data/graph_config.yml
• Setup Guide: docs/getting-started/setup.md
• Troubleshooting: docs/reference/troubleshooting.md

Jupyter Notebooks (Reference)¶

• Base graph creation: docs/notebooks/graph_PostGIS_v2.ipynb
• Fine graph creation: docs/notebooks/graph_fine_PostGIS_v2.ipynb
• Weighted graph: docs/notebooks/graph_weighted_directed_postgis_v2.ipynb