Running the SDP stand-alone
Before running the SDP, your local development environment needs to be set up. This can either be a local Kubernetes instance running in Minikube or by using remote access to a Kubernetes cluster. Details can be found in the requirements section.
In this page, we describe how you can install, test, interact with, and uninstall SDP using helm, kubectl, the SDP CLI, and the itango console.
The ska-sdp-integration repository provides a Makefile, which simplifies some of these steps, including installing, uninstalling the SDP and running integration tests:
Create the namespace for SDP processing
SDP requires two namespaces, one for its control system and another for deploying its
processing scripts and their execution engines.
If you are using a local environment, you may use the default namespace for the control
system, but you will have to create a new namespace to run the processing scripts.
Below, when we provide commands for local development, this namespace is assumed to
be called sdp. In examples using remote Kubernetes the placeholder is <sdp-namespace>:
$ kubectl create namespace sdp
For remote Kubernetes clusters namespaces may be pre-assigned and, if so, we will show the modifications to allow commands to use such namespaces.
Deploying the SDP
Releases of the SDP Helm chart are published in the SKA artefact repository. To install the released version, you need to add this chart repository to helm:
$ helm repo add ska https://artefact.skao.int/repository/helm-internal
If you already have the repository, you can update it, in order to gain access to latest chart versions, using:
$ helm repo update
The chart can be installed with the command (assuming the release name is test):
$ helm install test ska/ska-sdp
Or with non-default namespace names:
$ helm install test ska/ska-sdp -n <namespace> --set helmdeploy.namespace=<sdp-namespace>
You can watch the deployment in progress using kubectl:
$ kubectl get pod -n <namespace> --watch
or the k9s terminal-based UI (recommended):
$ k9s
Wait until all the pods are running:
default databaseds-tango-base-test-0 ● 1/1 0 Running 172.17.0.12 m01 119s
default ska-sdp-console-0 ● 1/1 0 Running 172.17.0.15 m01 119s
default ska-sdp-etcd-0 ● 1/1 0 Running 172.17.0.6 m01 119s
default ska-sdp-helmdeploy-0 ● 1/1 0 Running 172.17.0.14 m01 119s
default ska-sdp-lmc-config-6vbtr ● 0/1 0 Completed 172.17.0.11 m01 119s
default ska-sdp-lmc-controller-0 ● 1/1 0 Running 172.17.0.9 m01 119s
default ska-sdp-lmc-subarray-01-0 ● 1/1 0 Running 172.17.0.10 m01 119s
default ska-sdp-proccontrol-0 ● 1/1 0 Running 172.17.0.4 m01 119s
default ska-sdp-script-config-2hpdn ● 0/1 0 Completed 172.17.0.5 m01 119s
default ska-tango-base-tangodb-0 ● 1/1 0 Running 172.17.0.8 m01 119s
The two pods with config in their name will vanish about 30 seconds after they complete.
You can check the logs of pods to verify that they are doing okay:
$ kubectl logs <pod_name> -n <namespace>
or for a non-default namespace:
$ kubectl logs <pod_name> -n <namespace>
For example:
$ kubectl logs ska-sdp-lmc-subarray-01-0
...
1|2021-05-25T11:32:53.161Z|INFO|MainThread|init_device|subarray.py#92|tango-device:test-sdp/subarray/01|SDP Subarray initialising
...
1|2021-05-25T11:32:53.185Z|INFO|MainThread|init_device|subarray.py#127|tango-device:test-sdp/subarray/01|SDP Subarray initialised
...
$ kubectl logs ska-sdp-proccontrol-0
1|2021-05-25T11:32:32.423Z|INFO|MainThread|main_loop|processing_controller.py#180||Connecting to config DB
1|2021-05-25T11:32:32.455Z|INFO|MainThread|main_loop|processing_controller.py#183||Starting main loop
1|2021-05-25T11:32:32.566Z|INFO|MainThread|main_loop|processing_controller.py#190||processing block ids []
...
If it looks like this, there is a good chance everything has been deployed correctly.
Testing it out
Connecting to the configuration database
The ska-sdp chart deploys a ‘console’ pod which enables you to interact with the
configuration database. You can start a shell in the pod by doing:
$ kubectl exec -it ska-sdp-console-0 -n <namespace> -- bash
This will allow you to use the ska-sdp command:
# ska-sdp list -a
Keys with prefix /:
/lmc/controller
/lmc/subarray/01
/script/batch:test-batch:0.3.0
...
Which shows that the configuration contains the state of the Tango devices and the processing script definitions.
Details about the existing commands of the ska-sdp utility can be found in the
CLI to interact with SDP
section in the SDP Configuration Library documentation.
Starting a processing script
Next, we can add a processing block to the configuration:
# ska-sdp create pb <script-kind>:<script-name>:<script-version>
For example
# ska-sdp create pb batch:test-dask:0.3.0
Processing block created with pb_id: pb-sdpcli-20221011-00000
The processing block is created with the /pb prefix in the
configuration:
# ska-sdp list -v pb
Keys with prefix /pb:
/pb/pb-sdpcli-20221011-00000 = {
"dependencies": [],
"eb_id": null,
"parameters": {},
"pb_id": "pb-sdpcli-20221011-00000",
"script": {
"kind": "batch",
"name": "test-dask",
"version": "0.3.0"
}
}
/pb/pb-sdpcli-20221011-00000/owner = {
"command": [
"test_dask.py"
],
"hostname": "proc-pb-sdpcli-20221011-00000-script--1-qqvgw",
"pid": 1
}
/pb/pb-sdpcli-20221011-00000/state = {
"deployments": {
"proc-pb-sdpcli-20221011-00000-dask-1": "RUNNING",
"proc-pb-sdpcli-20221011-00000-dask-2": "RUNNING"
},
"last_updated": "2022-10-11 08:20:34",
"resources_available": true,
"status": "RUNNING"
}
The processing block is detected by the processing controller which deploys the
script. The script in turn deploys the execution engines (in this case, Dask).
The deployments are requested by creating entries with /deploy prefix in
the configuration database, where they are detected by the Helm deployer
which actually makes the deployments:
# ska-sdp list -v deployment
Keys with prefix /deploy:
/deploy/proc-pb-sdpcli-20221011-00000-dask-1 = {
"args": {
"chart": "dask",
"values": {
"image": "artefact.skao.int/ska-sdp-script-test-dask:0.3.0",
"worker.replicas": 2
}
},
"dpl_id": "proc-pb-sdpcli-20221011-00000-dask-1",
"kind": "helm"
}
/deploy/proc-pb-sdpcli-20221011-00000-dask-1/state = {
"pods": {
"proc-pb-sdpcli-20221011-00000-dask-1-scheduler-7d6f5f9749-vr6dw": "Running",
"proc-pb-sdpcli-20221011-00000-dask-1-worker-5744899988-hmr5q": "Running",
"proc-pb-sdpcli-20221011-00000-dask-1-worker-5744899988-sqnf6": "Running"
}
}
/deploy/proc-pb-sdpcli-20221011-00000-dask-2 = {
"args": {
"chart": "dask",
"values": {
"image": "artefact.skao.int/ska-sdp-script-test-dask:0.3.0",
"worker.replicas": 2
}
},
"dpl_id": "proc-pb-sdpcli-20221011-00000-dask-2",
"kind": "helm"
}
/deploy/proc-pb-sdpcli-20221011-00000-dask-2/state = {
"pods": {
"proc-pb-sdpcli-20221011-00000-dask-2-scheduler-65cc58cf4f-8bm9r": "Running",
"proc-pb-sdpcli-20221011-00000-dask-2-worker-79694dbf85-j7nfb": "Running",
"proc-pb-sdpcli-20221011-00000-dask-2-worker-79694dbf85-njw6c": "Running"
}
}
/deploy/proc-pb-sdpcli-20221011-00000-script = {
"args": {
"chart": "script",
"values": {
"env": [
{
"name": "SDP_CONFIG_HOST",
"value": "ska-sdp-etcd-client.dp-orca"
},
{
"name": "SDP_HELM_NAMESPACE",
"value": "dp-orca-p"
},
{
"name": "SDP_PB_ID",
"value": "pb-sdpcli-20221011-00000"
}
],
"image": "artefact.skao.int/ska-sdp-script-test-dask:0.3.0"
}
},
"dpl_id": "proc-pb-sdpcli-20221011-00000-script",
"kind": "helm"
}
/deploy/proc-pb-sdpcli-20221011-00000-script/state = {
"pods": {
"proc-pb-sdpcli-20221011-00000-script--1-r4p9c": "Running"
}
}
The deployments associated with the processing block have been created
in the sdp namespace. You can list the running pods using kubectl
on the host (exit the console pod):
$ kubectl get pod -n <sdp-namespace>
NAME READY STATUS RESTARTS AGE
proc-pb-sdpcli-20221011-00000-dask-1-scheduler-7d6f5f9749-vr6dw 1/1 Running 0 9s
proc-pb-sdpcli-20221011-00000-dask-1-worker-5744899988-hmr5q 1/1 Running 0 9s
proc-pb-sdpcli-20221011-00000-dask-1-worker-5744899988-sqnf6 1/1 Running 0 9s
proc-pb-sdpcli-20221011-00000-dask-2-scheduler-65cc58cf4f-8bm9r 1/1 Running 0 10s
proc-pb-sdpcli-20221011-00000-dask-2-worker-79694dbf85-j7nfb 1/1 Running 0 10s
proc-pb-sdpcli-20221011-00000-dask-2-worker-79694dbf85-njw6c 1/1 Running 0 10s
proc-pb-sdpcli-20221011-00000-script--1-r4p9c 1/1 Running 0 14s
Cleaning up
Finally, let us remove the processing block from the configuration (in the SDP console shell):
# ska-sdp delete pb pb-sdpcli-20221011-00000
/pb/pb-sdpcli-20221011-00000
/pb/pb-sdpcli-20221011-00000/state
Deleted above keys with prefix /pb/pb-sdpcli-20221011-00000.
If you re-run the commands from the last section you will notice that this correctly causes all changes to the cluster configuration to be undone as well.
Accessing the Tango interface
By default, the ska-sdp chart does not deploy the iTango shell pod from the
ska-tango-base chart. To enable it, you can upgrade the release with:
$ helm upgrade test ska/ska-sdp --set ska-tango-base.itango.enabled=true
This command will need to be modified for non-default namespaces:
$ helm upgrade test ska/ska-sdp -n <namespace> --set helmdeploy.namespace=<sdp-namespace>,ska-tango-base.itango.enabled=true
Then you can start an iTango session with:
$ kubectl exec -it ska-tango-base-itango-console -n <namespace> -- itango3
You should be able to list the Tango devices:
In [1]: lsdev
Device Alias Server Class
---------------------------------------- ------------------------- ------------------------- --------------------
test-sdp/control/0 SDPController/0 SDPController
test-sdp/subarray/01 SDPSubarray/01 SDPSubarray
sys/access_control/1 TangoAccessControl/1 TangoAccessControl
sys/database/2 DataBaseds/2 DataBase
sys/rest/0 TangoRestServer/rest TangoRestServer
sys/tg_test/1 TangoTest/test TangoTest
This interface allows direct interaction with the devices, such as querying and changing attributes and issuing Tango commands to control SDP processing.
In [2]: d = DeviceProxy('test-sdp/subarray/01')
In [3]: d.state()
Out[3]: tango._tango.DevState.OFF
In [4]: d.On()
In [5]: d.state()
Out[5]: tango._tango.DevState.ON
In [6]: d.obsState
Out[6]: <obsState.EMPTY: 0>
To start processing on SDP, you will need a configuration string, which provides the set up, the request for resources, and the request of what processing script to run. You can find an example string in the Tango Jupyter notebook (the configuration string can be found here, we recommend that you copy it from the display-version of the notebook and not the raw file).
Save the copied configuration string (which is provided as JSON in the example notebook) as a python string. Make sure you update the execution block and processing block IDs (if you run this multiple times, you will need to increment the number at the end):
import json
EXECUTION_BLOCK_ID = f"eb-test-20221012-00001"
PROCESSING_BLOCK_ID_REALTIME = f"pb-testrealtime-20221012-00001"
PROCESSING_BLOCK_ID_BATCH = f"pb-testbatch-20221012-00001"
config = json.dumps(<copied-json-string>)
In addition more details about the SDP Subarray Tango commands and examples of valid control strings can be found in the Subarray section of SDP LMC.
The Telescope Model library provides detailed explanation of configurations strings for each SDP command.
Below, we provide the steps from assigning resources to the processing, through configure and scan, to releasing the resources and ending the process.
In [8}: d.AssignResources(config)
In [9]: d.obsState
Out[9]: <obsState.IDLE: 0>
In [10]: d.Configure('{"interface": "https://schema.skao.int/ska-sdp-configure/0.4", "scan_type": "target:a"}')
In [11]: d.obsState
Out[11]: <obsState.READY: 2>
In [12]: d.Scan('{"interface": "https://schema.skao.int/ska-sdp-scan/0.4", "scan_id": 1}')
In [13]: d.obsState
Out[13]: <obsState.SCANNING: 3>
In [14]: d.EndScan()
In [15]: d.obsState
Out[15]: <obsState.READY: 2>
In [16]: d.End()
In [17]: d.obsState
Out[17]: <obsState.IDLE: 0>
In [18]: d.ReleaseResources('{ "interface": "https://schema.skao.int/ska-sdp-releaseres/0.4", '
'"resources": {"receptors": ["SKA001", "SKA002", "SKA003", "SKA004"]}}')
In [19]: d.obsState
Out[19]: <obsState.EMPTY: 0>
In [20]: d.Off()
In [21]: d.state()
Out[21]: tango._tango.DevState.OFF
ReleaseResources takes the list of resources (“receptors”) to be released. If you
want to release all of them at once, you may use d.ReleaseAllResources().
Accessing the Taranta dashboard
By default, the ska-sdp chart does not deploy the Taranta dashboard. To enable it, you can
upgrade the deployment with:
$ helm upgrade test ska/ska-sdp --set ska-tango-taranta.enabled=true --set ska-tango-tangogql.enabled=true
After the upgrade you should be able to access the Taranta dashboard in the namespace it was deployed at this URL:
http://<ingress_host>/<namespace>/taranta/