NSAI Pilot

NSAI Pilot

CI StatusHex.pmDocumentationElixirLicense

Interactive CLI and REPL for the NSAI ecosystem

Pilot provides a unified command-line interface for interacting with North Shore AI's ecosystem of services, including job management, experiment running, service monitoring, dataset operations, and metrics querying.

Features

Installation

Prerequisites

Build from Source

# Clone the repository
git clone https://github.com/North-Shore-AI/pilot.git
cd pilot

# Get dependencies
mix deps.get

# Build the escript
mix escript.build

# (Optional) Install globally
sudo mv pilot /usr/local/bin/

Configuration

Pilot looks for configuration in ~/.pilot/config.yaml:

default_tenant: "cns"
services:
  work: http://localhost:4000
  registry: http://localhost:4001
output_format: table
http_timeout: 30000

You can also use environment variables to override configuration:

export PILOT_TENANT=my_tenant
export PILOT_WORK_URL=http://work.example.com
export PILOT_FORMAT=json

Usage

Command Line Interface

Job Management

# List all jobs
pilot jobs list

# Filter by status
pilot jobs list --status running

# Submit a job
pilot jobs submit --type training --config job_config.json

# Check job status
pilot jobs status job_123

# Cancel a job
pilot jobs cancel job_123

Experiments

# Run a proposer experiment
pilot experiments run proposer --dataset scifact

# Run an antagonist experiment
pilot experiments run antagonist --dataset fever

# Check experiment status
pilot experiments status exp_456

# Compare two experiments
pilot experiments compare exp_456 exp_789

Services

# List all services
pilot services list

# Check service health
pilot services health

# View service logs
pilot services logs work --tail 100

# Follow logs in real-time
pilot services logs work --follow

Datasets

# List available datasets
pilot datasets list

# Get dataset information
pilot datasets info scifact

# Download a dataset
pilot datasets download fever

Metrics

# Query metrics
pilot metrics query --metric entailment --model llama-3.1

# Open metrics dashboard
pilot metrics dashboard

Interactive REPL

Start the REPL for interactive use:

pilot --repl

In the REPL, you can:

# List jobs
pilot> jobs.list

# Check service health
pilot> services.health

# List datasets
pilot> datasets.list

# Execute Elixir code directly
pilot> 1 + 1
2

pilot> Enum.map([1, 2, 3], &(&1 * 2))
[2, 4, 6]

# Use the HTTP client
pilot> Client.get(:work, "/api/jobs")
{:ok, [...]}

# View command history
pilot> history

# Show configuration
pilot> config

# Exit
pilot> exit

Output Formats

Control output format with the --format flag:

# Table format (default, human-readable)
pilot jobs list --format table

# JSON format (machine-readable)
pilot jobs list --format json

# YAML format
pilot jobs list --format yaml

Shell Completion

Bash

# Add to ~/.bashrc
source /path/to/pilot/priv/completions/pilot.bash

Zsh

# Add to ~/.zshrc
fpath=(/path/to/pilot/priv/completions $fpath)
autoload -U compinit && compinit

Architecture

Pilot is structured as follows:

lib/pilot/
├── cli.ex                 # Main CLI entry point
├── application.ex         # OTP application
├── config.ex             # Configuration management
├── output.ex             # Output formatting
├── client.ex             # HTTP client
├── repl.ex               # Interactive REPL
└── commands/             # Command implementations
    ├── jobs.ex
    ├── experiments.ex
    ├── services.ex
    ├── datasets.ex
    └── metrics.ex

Key Components

Development

Running Tests

# Run all tests
mix test

# Run with coverage
mix test --cover

# Run specific test file
mix test test/pilot/config_test.exs

Code Quality

# Format code
mix format

# Run static analysis
mix credo --strict

# Run dialyzer (first run will be slow)
mix dialyzer

Building

# Compile
mix compile

# Build escript
mix escript.build

# Clean build artifacts
mix clean

Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Integration with NSAI Ecosystem

Pilot integrates with the following NSAI services:

Examples

Submit a Training Job

# Create job configuration
cat > train_job.json <<EOF
{
  "model": "llama-3.1-8b",
  "dataset": "scifact",
  "epochs": 3,
  "learning_rate": 0.0001
}
EOF

# Submit job
pilot jobs submit --type training --config train_job.json

# Monitor job
pilot jobs status job_123

Run Complete CNS Pipeline

# Run Proposer
pilot experiments run proposer --dataset scifact

# Run Antagonist on results
pilot experiments run antagonist --dataset scifact

# Run Synthesizer
pilot experiments run synthesizer --dataset scifact

# Compare results
pilot experiments compare exp_1 exp_3

Monitor Services

# Check all services
pilot services health

# Follow logs for debugging
pilot services logs work --follow

Troubleshooting

Connection Errors

If you see connection errors, verify that services are running:

pilot services health

Check your configuration:

pilot --repl
pilot> config

Permission Denied

If you get permission errors when installing globally:

# Install to user directory instead
mkdir -p ~/.local/bin
mv pilot ~/.local/bin/

# Add to PATH in ~/.bashrc or ~/.zshrc
export PATH="$HOME/.local/bin:$PATH"

License

MIT License - see LICENSE file for details

Authors

North Shore AI Team

Links