JwstCli

A cli wrapper and downloader built around jwstapi.com

Author

raf+jwst@dreamthought.com

Warning and Context

This is a hobby project which is being used to learn elixir's OTP, whilst scratching an itch to explore the new jwstapi.com.
This API is to accompany dreamily staring into nebuli.
Use with caution.

Future work

JwstCli.Api may get factored out for reuse as a stand alone library, which wraps HTTPoison.

Tests

Will grow as more API endpoints and the project matures.

Installation

Local

Clone
mix release

Hex (not yet set up)

If available in Hex, the package can be installed by adding jwst_cli to your list of dependencies in mix.exs:

def deps do
  [
    {:jwst_cli, "~> 0.1.0"}
  ]
end

Documentation

Generating

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/jwst_cli.

Prod

Let's not go there yet. But do contribute back if you do.

Execution

Set JWST_API_KEY in your shell. Eg. export JWST_API_KEY=abcdKitten1234
Run mix release
As below, start the release in start_iex, or with daemon and remote

Set JWST_API_KEY

You can obtain a new API key from jwstapi.com

This provided from the environment and baked into the release at build time. It may be customised per environment using (./config/config.exs)[./config/config.exs] and sibling overides.

Eg.

  export JWST_API_KEY=abcdKitten1234

Buiding the Release

We simply run mix release

  mix release
Compiling 5 files (.ex)
warning: module attribute @me was set but never used
  lib/jwst_cli/repl/executor.ex:9

Generated jwst_cli app
Release jwst_cli-0.1.0 already exists. Overwrite? [Yn] y
* assembling jwst_cli-0.1.0 on MIX_ENV=dev
* skipping runtime configuration (config/runtime.exs not found)
* skipping elixir.bat for windows (bin/elixir.bat not found in the Elixir installation)
* skipping iex.bat for windows (bin/iex.bat not found in the Elixir installation)

Release created at _build/dev/rel/jwst_cli!

Connecting to the App to query JWST API

As mentioned there are two methods

Running as a deamon and connecting remotely

This scenario is better suited to a long running scenario where you may want to connect remotely.

Start the app as a background daemon

This will leave the GenServer (with supervisor) running in the background

   _build/dev/rel/jwst_cli/bin/jwst_cli daemon

This does not produce any output.

Connect Remotely

Very similar but you pass in remote. This pattern allows you abort each iex session, but still connect back to the same daemon to query the API.

Eg. The following connects and queries the list of JWST program id's:

 _build/dev/rel/jwst_cli/bin/jwst_cli remote
Erlang/OTP 25 [erts-13.0.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]

Interactive Elixir (1.13.2) - press Ctrl+C to exit (type h() ENTER for help)
 
iex(jwst_cli@feynman)1> GenServer.call(JwstCli.Repl.Executor, {:execute, "get_program_list"})
[2731, 2732, 2733, 2734]

start_iex

This creates a blocking process which will terminate on exit from iex.

It also produces greater visibility of logging output from the GenServer.

Eg.

  _build/dev/rel/jwst_cli/bin/jwst_cli start_iex
Erlang/OTP 25 [erts-13.0.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]


10:16:22.183 [info]  Using Children

10:16:22.183 [info]  [{JwstCli.Repl.Executor, [%{api_key: "**************"}]}]

10:16:22.183 [info]  [%{api_key: "***************"}]

10:16:22.183 [info]  Start on #PID<0.905.0>

10:16:22.183 [info]  [active: 1, specs: 1, supervisors: 0, workers: 1]
Interactive Elixir (1.13.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(jwst_cli@feynman)1> GenServer.call(JwstCli.Repl.Executor, {:execute, "get_program_list"})
10:16:49.465 [info]  "get_program_list"
10:16:49.466 [debug] Invoking /program/list with *********
10:16:50.605 [info]  [2731, 2732, 2733, 2734]
[2731, 2732, 2733, 2734]``

GenServer calls for Querying the API

Once you're in a session, you can query the GenServer by using call with the following general form:

iex> GenServer.call(JwstCli.Repl.Executor, {:execute, "<name of operation>"})

Operations with names ending in _raw will typically return the raw content of the equivalent remote resources as JSON (wrapped in a JASON.Response).

See below for you choice of operations

get_program_list

Returns all JWST program ids as an array

Eg.

iex> GenServer.call(JwstCli.Repl.Executor, {:execute, "get_program_list"})
[2731, 2732, 2733, 2734]

get_program_list_raw

Returns all JWST program ids contained in the response from /program/list defined by the JWST API

Eg.

GenServer.call(JwstCli.Repl.Executor, {:execute, "get_program_list_raw"})

get_recent_jpg

Returns last 30 jpgs defined by the JWST API

Eg.

GenServer.call(JwstCli.Repl.Executor, {:execute, "get_recent_jpg"})
iex(jwst_cli@feynman)6> GenServer.call(JwstCli.Repl.Executor, {:execute, "get_recent_jpg"})

get_all_jpg

Returns all JWST jpegs URLs from JWST defined by the JWST API

Eg.

GenServer.call(JwstCli.Repl.Executor, {:execute, "get_all_jpg"})
iex(jwst_cli@feynman)6> GenServer.call(JwstCli.Repl.Executor, {:execute, "get_all_jpg"})

get_all_jpg_raw

Returns all JWST jpegs from JWST defined by the JWST API

Eg.

GenServer.call(JwstCli.Repl.Executor, {:execute, "get_all_jpg_raw"})
iex(jwst_cli@feynman)6> GenServer.call(JwstCli.Repl.Executor, {:execute, "get_all_jpg_raw"})

{:ok,
 %HTTPoison.Response{
   body: "{\"statusCode\":200,\"body\":[{\"id\":\"jw02731002001_02107_00004_mirimage_o002_crf_thumb.jpg\",\"observation_id\":\"jw02731002001_02107_00004_mirimage_o002\",\"program\":2731,\"details\":{\"mission\":\"JWST\",\"instruments\":[{\"instrument\":\"FGS\"},{\"instrument\":\"NIRCam\"},{\"instrument\":\"NIRISS\"},{\"instrument\":\"NIRSpec\"},{\"instrument\":\"MIRI\"}],\"suffix\":\"_thumb\",\"description\":\"thumbnail image of the FITS data product\"},\"file_type\":\"jpg\",\"thumbnail\":\"\",\"location\":\"https://stpubdata-jwst.stsci.edu/ero/jw02731/jw02731002001/jw02731002001_02107_00004_mirimage_o002_crf_thumb.jpg\"], ....
   headers: [...],
   request: %HTTPoison.Request{ .... },
   request_url: ....,
   status_code: 200
 }}

get_all_fits_raw

Returns all JWST FITS imagse from JWST defined by the JWST API

Eg.

GenServer.call(JwstCli.Repl.Executor, {:execute, "get_all_fits_raw"})
iex(jwst_cli@feynman)6> GenServer.call(JwstCli.Repl.Executor, {:execute, "get_all_fits_raw"})

{:ok,
 %HTTPoison.Response{
   body: "{\"statusCode\":200,\"body\":[{\"id\":\"jw02731001003_02105_00001_nrca1_i2d.fits\",\"observation_id\":\"jw02731001003_02105_00001_nrca1\",\"program\":2731,\"details\":{\"mission\":\"JWST\",\"instruments\":[{\"instrument\":\"FGS\"},{\"instrument\":\"NIRCam\"},{\"instrument\":\"NIRISS\"},{\"instrument\":\"NIRSpec\"},{\"instrument\":\"MIRI\"}],\"suffix\":\"_i2d\",\"description\":\"exposure/target (L2b/L3): rectified 2D image\"},\"file_type\":\"fits\",\"thumbnail\":\"\",\"location\":\"https://stpubdata-jwst.stsci.edu/ero/jw02731/jw02731001003/jw02731001003_02105_00001_nrca1_i2d.fits\",...
   headers: [...],
   request: %HTTPoison.Request{ .... },
   request_url: ....,
   status_code: 200
 }}

get_all_ecsv_raw

Returns all ecsv file url's from JWST defined by the JWST API

Eg.

GenServer.call(JwstCli.Repl.Executor, {:execute, "get_all_ecsv_raw"})
iex(jwst_cli@feynman)6> GenServer.call(JwstCli.Repl.Executor, {:execute, "get_all_ecsv_raw"})

{:ok,
 %HTTPoison.Response{
   body: "{\"statusCode\":200,\"body\":[{\"id\":\"jw02731-o001_t017_nircam_clear-f444w_cat.ecsv\",\"observation_id\":\"jw02731-o001_t017_nircam_clear-f444w\",\"program\":2731,\"details\":{\"mission\":\"JWST\",\"instruments\":[{\"instrument\":\"FGS\"},{\"instrument\":\"NIRISS\"}],\"suffix\":\"_cat\",\"description\":\"target (L3) : source catalog\"},\"file_type\":\"ecsv\",\"thumbnail\":\"\",\"location\":\"https://stpubdata-jwst.stsci.edu/ero/jw02731/L3/t/jw02731-o001_t017_nircam_clear-f444w_cat.ecsv\"},{\"id\":\"jw02731-o001_t017_nircam_f444w-f470n_cat.ecsv\",\"observation_id\":\"jw02731-o001_t017_nircam_f444w-f470n\",\"program\":2731,\"details\":{\"mission\":\"JWST\",\"instruments\":[{\"instrument\":\"FGS\"},{\"instrument\":\"NIRISS\"}],\"suffix\":\"_cat\",\"description\":\"target (L3) : source catalog\"},\"file_type\":\"ecsv\",\"thumbnail\":\"\",\"location\":\"https://stpubdata-jwst.stsci.edu/ero/jw02731/L3/t/jw02731-o001_t017_nircam_f444w-f470n_cat.ecsv\",...
   headers: [...],
   request: %HTTPoison.Request{ .... },
   request_url: ....,
   status_code: 200
 }}

get_all_json_raw

Returns all json file url's from JWST defined by the JWST API

Eg.

GenServer.call(JwstCli.Repl.Executor, {:execute, "get_all_json_raw"})
iex(jwst_cli@feynman)6> GenServer.call(JwstCli.Repl.Executor, {:execute, "get_all_json_raw"})

{:ok,
 %HTTPoison.Response{
   body: "{\"statusCode\":200,\"body\":[{\"id\":\"jw02731-o001_20220712t165318_image2_317_asn.json\",\"observation_id\":\"jw02731-o001_20220712t165318_image2_317\",\"program\":2731,\"details\":{\"mission\":\"JWST\",\"instruments\":[{\"instrument\":\"FGS\"},{\"instrument\":\"NIRCam\"},{\"instrument\":\"NIRISS\"},{\"instrument\":\"NIRSpec\"},{\"instrument\":\"MIRI\"}],\"suffix\":\"_asn\",\"description\":\"source/target (L3) : association generator\"},\"file_type\":\"json\",\"thumbnail\":\"\",\"location\":\"https://stpubdata-jwst.stsci.edu/ero/jw02731/asn/jw02731-o001_20220712t165318_image2_317_asn.json\"}, ...
   headers: [...],
   request: %HTTPoison.Request{ .... },
   request_url: ....,
   status_code: 200
 }}

Log Levels

The app is currently VERY chatting. You may want to reduce the default log levels; this is on my TODO.

In iex:

iex> Logger.configure(level: :error)

In the config/config.exs

config :logger, level: :error

Stand alone session (WIP)

Buiding

This is a work in progres and will ulimately provide a more powerful standalone repl. Run the following to generate the jwst_cli executable in the parent directory.

mix escript.build

Still in progress - this will be a cli app to query the GenServer session as a connected node. You could potentially run this by starting iex with elixir --cookie <common value> and -s script I've yet to document this properly.

Thanks

Thank you to Kyle Redelinghuys who has provided the JWST API