Skip to main content

Install the On-Prem Worker

BigPanda's L1 Agent On-Prem Worker is a lightweight containerized agent that runs on your infrastructure to execute runbook automation steps. It connects to BigPanda using mutual TLS (mTLS) authentication and executes pre-registered workflows on demand.

This guide covers a first install: download the image, configure it to route through your corporate HTTP CONNECT proxy, run it, and confirm it reaches a healthy state. Workflow-specific configuration is added in a follow-up step coordinated with BigPanda.

For architecture, security, and FAQs, see the On-Prem Worker parent page.

Supported versions

These instructions apply to worker v13.9.0 and later. New installs use the latest version; earlier versions can no longer be installed.

Worker version

Status

Install guide

Changes

v13.10.0

Latest. Use for new installs.

v13.9.0 and later

Adds SSH targets (ADR_SSH_TARGETS).

v13.9.0

Supported

v13.9.0 and later

Adds the oauth2 auth type for ADR_HTTP_INTEGRATIONS.

v13.8.0

Existing deployments only, closed to new installs

Not available. See Updating to a new version.

No configuration changes documented.

v13.7.2

Existing deployments only, closed to new installs

Not available. See Updating to a new version.

Adds the none, query and cookie auth types, and an optional scheme for bearer. Adds the per-entry fields api_version, allowed_path_prefixes, allowed_methods, body_format, extra_headers and timeout_ms, a maximum of 50 entries, and name rules with reserved names. Invalid entries are dropped and logged; the rest keep working.

v13.6.3

Existing deployments only, closed to new installs

Not available. See Updating to a new version.

Adds HTTP CONNECT proxy support (TEMPORAL_PROXY_URL) and HTTP integrations (ADR_HTTP_INTEGRATIONS) with bearer, basic and header auth. Certificate files are renamed to tls.crt, tls.key and ca.crt.

To check the version you are running:

docker images | grep adr-temporal-worker

v13.9.0 and later

The worker ships as a Docker container image. BigPanda provides a presigned download URL during onboarding.

Install the worker

Before you start

Make sure you have the following ready:

  • Docker installed on the target host (Linux x86_64)

  • The mTLS certificate bundle from BigPanda (provided during onboarding): your client certificate (tls.crt), its private key (tls.key), and BigPanda's CA certificate (ca.crt).

  • A corporate HTTP CONNECT proxy allowlisted for CONNECT temporal.bigpanda.io:7233, with TLS inspection (SSL decryption) disabled for that destination. TLS-inspecting proxies break the mTLS chain. Major enterprise proxies (Zscaler, Palo Alto, Squid, Forcepoint, BlueCoat, Cloudflare Gateway) support per-destination decryption exclusions.

  • Outbound network access from the host to your proxy, and from the proxy to temporal.bigpanda.io:7233.

Deploy the worker with Docker

Download and load the image

BigPanda shares a presigned S3 URL (valid for 24 hours) for the version tag you are installing. Download the tarball, replacing <version> with that tag (for example, v13.10.0):

curl -L '<URL>' -o adr-temporal-worker-<version>.tar.gz

If your host reaches S3 through the corporate proxy, prefix with HTTPS_PROXY:

HTTPS_PROXY=http://<host>:<port> \
  curl -L '<URL>' -o adr-temporal-worker-<version>.tar.gz

Expected file size is approximately <SIZE> MB (<BYTES> bytes). Verify, then load:

ls -la adr-temporal-worker-<version>.tar.gz
docker load -i adr-temporal-worker-<version>.tar.gz

Expected output: Loaded image: adr-temporal-worker:latest. Confirm:

docker images | grep adr-temporal-worker
Install certificates

Place the three files BigPanda sent into a certs/ directory next to where you will create the .env file:

certs/
├── tls.crt   # Client certificate (identifies your organization)
├── tls.key   # Client private key (keep secure, do not share)
└── ca.crt    # BigPanda CA certificate (validates the Temporal server)
Create a configuration file

Create .env in the same directory as certs/:

# --- Connection (provided by BigPanda) ---
TEMPORAL_ADDRESS=temporal.bigpanda.io:7233
TEMPORAL_TLS_CERT_PATH=/certs/tls.crt
TEMPORAL_TLS_KEY_PATH=/certs/tls.key
TEMPORAL_TLS_CA_PATH=/certs/ca.crt

# --- Corporate egress proxy (required — provided by your network team) ---
# Format: http://[user:pass@]host:port (HTTP CONNECT only)
TEMPORAL_PROXY_URL=http://<host>:<port>
# With basic auth (URL-encode '@' as %40, ':' as %3A in user/pass):
# TEMPORAL_PROXY_URL=http://<user>:<pass>@<host>:<port>

# --- HTTP integrations (provided by BigPanda during onboarding) ---
# JSON array of the HTTP endpoints the worker may call. Each integration's
# credentials are supplied via its own secret_env variable, set separately.
ADR_HTTP_INTEGRATIONS=[ ... see Configuration reference ... ]

# --- SSH targets (v13.10.0 and later; provided by BigPanda during onboarding) ---
# JSON array of the hosts the worker may run commands on. Each target's private
# key is supplied via its own private_key_env variable, set separately.
ADR_SSH_TARGETS=[ ... see Configuration reference ... ]

# --- Operational (optional) ---
LOG_LEVEL=info
METRICS_PORT=8080
Run the worker
docker run -d --name adr-temporal-worker \
  --env-file .env \
  -v $(pwd)/certs:/certs:ro \
  --restart unless-stopped \
  adr-temporal-worker:latest
Verify the deployment

Confirm the container is running and healthy:

docker ps | grep adr-temporal-worker
docker logs --tail 50 adr-temporal-worker
GET http://localhost:8080/health   → {"status":"ok","startedAt":"..."}
GET http://localhost:8080/ready    → {"status":"ready"}

Both endpoints return 200 once the process is up, independently of how many integrations are configured. The image is node:*-slim: it ships node but not curl or wget, so a container health check must use a node one-liner:

node -e "require('http').get('http://localhost:8080/health',r=>process.exit(r.statusCode===200?0:1)).on('error',()=>process.exit(1))"

Deploy the worker with Kubernetes

If you are deploying to Kubernetes, use the following manifests as a starting point.

  1. Create a Secret for the certificates:

    apiVersion: v1
    kind: Secret
    metadata:
      name: adr-temporal-worker-certs
      namespace: bigpanda
    type: Opaque
    data:
      tls.crt: <base64-encoded-cert>
      tls.key: <base64-encoded-key>
      ca.crt: <base64-encoded-ca>
  2. Create a Secret for the monitoring API key, for each secret_env or client_secret_env named in ADR_HTTP_INTEGRATIONS, and for each private_key_env named in ADR_SSH_TARGETS:

    apiVersion: v1
    kind: Secret
    metadata:
      name: adr-temporal-worker-secrets
      namespace: bigpanda
    type: Opaque
    stringData:
      DEVICE_API_KEY: "<your-monitoring-platform-api-key>"
      EXAMPLE_API_CLIENT_SECRET: "<oauth client secret>"            # client_secret_env
      SSH_KEY_APP_SERVERS: "<private key, base64 on one line>"     # private_key_env (v13.10.0 and later)
  3. Create the deployment:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: adr-temporal-worker
      namespace: bigpanda
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: adr-temporal-worker
      template:
        metadata:
          labels:
            app: adr-temporal-worker
        spec:
          containers:
            - name: worker
              image: adr-temporal-worker:latest
              ports:
                - containerPort: 8080
              envFrom:
                - secretRef:
                    name: adr-temporal-worker-secrets
              env:
                - name: TEMPORAL_ADDRESS
                  value: "temporal.bigpanda.io:7233"
                - name: TEMPORAL_TLS_CERT_PATH
                  value: "/certs/tls.crt"
                - name: TEMPORAL_TLS_KEY_PATH
                  value: "/certs/tls.key"
                - name: TEMPORAL_TLS_CA_PATH
                  value: "/certs/ca.crt"
                - name: TEMPORAL_PROXY_URL
                  value: "http://<host>:<port>"
                - name: ADR_HTTP_INTEGRATIONS
                  value: '[ ... see Configuration reference ... ]'
                - name: ADR_SSH_TARGETS
                  value: '[ ... see Configuration reference ... ]'
              volumeMounts:
                - name: certs
                  mountPath: /certs
                  readOnly: true
              resources:
                requests:
                  memory: "256Mi"
                  cpu: "100m"
                limits:
                  memory: "512Mi"
                  cpu: "500m"
              livenessProbe:
                httpGet:
                  path: /health
                  port: 8080
                initialDelaySeconds: 30
              readinessProbe:
                httpGet:
                  path: /ready
                  port: 8080
                initialDelaySeconds: 15
          volumes:
            - name: certs
              secret:
                secretName: adr-temporal-worker-certs

Configuration reference

Connection

Variable

Description

Value

TEMPORAL_ADDRESS

BigPanda Temporal Server address

temporal.bigpanda.io:7233

TEMPORAL_TLS_CERT_PATH

Path in the container to the client certificate

/certs/tls.crt

TEMPORAL_TLS_KEY_PATH

Path in the container to the client private key

/certs/tls.key

TEMPORAL_TLS_CA_PATH

Path in the container to the BigPanda CA certificate

/certs/ca.crt

ADR_HTTP_INTEGRATIONS

JSON array of HTTP integrations the worker may call

See below

ADR_SSH_TARGETS

JSON array of SSH targets the worker may run commands on (v13.10.0 and later)

See below

Proxy

Proxy data should be provided by your network team.

Variable

Description

Example

TEMPORAL_PROXY_URL

HTTP CONNECT proxy URL. Format http://[user:pass@]host:port

http://corpproxy.yourdomain.com:8080

Proxy scope

Jira, Confluence, api.bigpanda.io, Itential, and ADR_HTTP_INTEGRATIONS traffic, including OAuth token endpoints, is not routed through TEMPORAL_PROXY_URL. Contact BigPanda support if that traffic must also use a proxy.

Optional

Variable

Default

Description

LOG_LEVEL

info

Log level (debug, info, warn, error)

METRICS_PORT

8080

Port for the health and ready endpoints

NODE_EXTRA_CA_CERTS

—

CA bundle for internally signed APIs or token endpoints

NODE_OPTIONS

—

--tls-cipher-list=DEFAULT@SECLEVEL=1. Stopgap for key too weak TLS errors. Applies to every outbound connection; keys under 1024 bits still fail.

ADR_HTTP_INTEGRATIONS

Variable

Required

Description

ADR_HTTP_INTEGRATIONS

No

JSON array of HTTP integrations. See the fields and authentication types below.

Targets of secret_env, client_secret_env, private_key_env

Conditional

One variable per credential named in the array

ADR_HTTP_INTEGRATIONS is a JSON array of up to 50 entries, one per API the worker may call on behalf of runbooks. Adding an integration is configuration only; no software update is needed.

Credentials never go in the JSON

Every auth block names an environment variable (secret_env, or client_secret_env / private_key_env for oauth2) that holds the credential. Set those variables through your runtime's secret mechanism. Query-string secrets are masked in worker debug output.

A bad entry costs only that entry

Each element is validated individually at startup. Invalid entries are dropped and logged at error as http integration rejected with the reason; the rest keep working. A runbook step that names a dropped integration fails with unknown integration '<name>'.

Fields

Field

Required

Description

name

Yes

Unique lowercase ID, ^[a-z0-9][a-z0-9_-]*$, maximum 64 characters. Must not reuse a built-in tool name: itential, check_device_connectivity, guardian, ping_device.

type

Yes

Tool kind, free text. For example grafana, jira, splunk, servicenow, http.

base_url

Yes

Absolute http(s) URL with no embedded credentials. All requests are pinned under it.

api_version

No

Free text that helps runbooks target the right API. For example, Jira 2 or 3.

description

No

One line describing this instance.

auth

Yes

Credential definition. See Authentication types.

allowed_path_prefixes

No

Restricts callable paths, for example ["/api/"]. Maximum 20, each starting with /. Default: any path under base_url.

allowed_methods

No

Non-empty subset of GET, POST, PUT, PATCH, DELETE. Default: all five. ["GET"] makes the integration read-only.

body_format

No

json (default) or form (application/x-www-form-urlencoded; the body must be a JSON object).

extra_headers

No

Constant, non-secret headers sent on every request. Reserved names are refused: Authorization, Cookie, Proxy-Authorization, Host, X-Forwarded-*.

timeout_ms

No

Per-integration request timeout. Default and maximum: 30000.

Authentication types

Choose by how the target API expects credentials.

Your API expects

auth.type

Fields

Example

Authorization: Bearer <token>

bearer

secret_env

Grafana

Authorization: <word> <token>

bearer + scheme

secret_env, scheme

Dynatrace Api-Token

Authorization: Basic base64(user:token)

basic

secret_env (holds the base64 value)

Jira Cloud, ServiceNow

One or more custom secret headers

header

headers[] of name + secret_env

Datadog

Authorization: <raw token>, no scheme word

header

Single Authorization entry

Linear

Token in the query string

query

param_name, secret_env

Itential static token

Session cookie

cookie

secret_env (full Cookie value)

—

Client ID + secret or key, and a token URL

oauth2

See oauth2

Any client-credentials provider

Nothing

none

—

Public APIs

oauth2

Only the client_credentials grant is supported. Exactly one of client_secret_env (with client_auth post or basic) or private_key_env (with private_key_jwt) is required.

Field

Required

Description

grant_type

Yes

client_credentials

token_url

Yes

Absolute https URL with no user:pass@, query, or fragment. Maximum 512 characters.

client_id

Yes

Not a secret

client_secret_env

Yes, unless private_key_jwt

Variable holding the client secret

client_auth

No

post (default), basic, or private_key_jwt

scope, audience, resource

No

As the provider requires

private_key_env

private_key_jwt only

Variable holding an unencrypted PEM, or its base64 on one line

assertion_alg

No

RS256 (default, RSA ≥2048) or ES256 (EC P-256)

key_id

No

JWT kid header

Token lifecycle

Tokens are cached in memory per process and renewed 30 seconds before expiry (halfway through for tokens shorter than 60 seconds). A missing expires_in is treated as 3600 seconds. The token request may take up to 10 seconds of the call's budget. A 401 on a token older than 60 seconds results in the call failing without retry. Credentials are read at startup: to rotate, update the variable and restart the worker.

Example
ADR_HTTP_INTEGRATIONS=[
  {
    "name": "grafana-eu-1",
    "type": "grafana",
    "base_url": "https://grafana-eu.example.internal",
    "api_version": "10.x",
    "auth": { "type": "bearer", "secret_env": "GRAFANA_EU_1_TOKEN" },
    "allowed_path_prefixes": ["/api/"]
  },
  {
    "name": "servicenow-prod",
    "type": "servicenow",
    "base_url": "https://acme.service-now.com",
    "auth": { "type": "basic", "secret_env": "SNOW_BASIC" },
    "allowed_methods": ["GET"],
    "extra_headers": { "X-Tenant-Id": "acme-prod" }
  },
  {
    "name": "example-api",
    "type": "http",
    "base_url": "https://api.example.internal",
    "auth": {
      "type": "oauth2",
      "grant_type": "client_credentials",
      "token_url": "https://auth.example.internal/oauth/token",
      "client_id": "svc-bigpanda",
      "client_secret_env": "EXAMPLE_API_CLIENT_SECRET"
    }
  }
]

One line under --env-file

docker run --env-file does not support multi-line values. Collapse the array to one line, or use docker compose or Kubernetes, where multi-line values work.

ADR_SSH_TARGETS

v13.10.0 and later.

A JSON array with one object per group of hosts the worker may run commands on. Per-entry fields: name (unique identifier), hosts (the machines this target may reach), optional description and port, username (the account the worker authenticates as), private_key_env, host_key_sha256 (the SHA256 fingerprint of each host's SSH host key), allowed_commands (the executables that may run), and allowed_path_prefixes (the directories their arguments may address). Optional max_output_bytes and timeout_ms default to 262144 and 30000.

private_key_env names an environment variable holding the actual key: an OpenSSH private key, or that key base64-encoded onto one line, defined alongside the worker's other environment variables or injected from your secret store. Never put a raw key in ADR_SSH_TARGETS itself.

Every host must have a host_key_sha256 entry. Collect each one with ssh-keyscan -t ed25519 <host> | ssh-keygen -lf -. A command is checked against allowed_commands and allowed_path_prefixes before any connection is opened, so anything outside them never reaches the host.

ADR_SSH_TARGETS=[
  {
    "name": "app-servers",
    "description": "Application servers - log reads",
    "hosts": ["host01.example.com", "host02.example.com"],
    "port": 22,
    "username": "svc_bigpanda",
    "private_key_env": "SSH_KEY_APP_SERVERS",
    "host_key_sha256": {
      "host01.example.com": "SHA256:6oPWAdTwUzje1PKJ21yzq/09O+CT71Gm/1ze8GzXx/k",
      "host02.example.com": "SHA256:kJ2hQ1vZ8nR4xT7wY0mB5cF3dG6sL9pA2eN1uI8oK4M"
    },
    "allowed_commands": ["id", "ls", "cat", "tail", "head"],
    "allowed_path_prefixes": ["/var/log/app/"]
  }
]

Itential

Variable

Required

Description

ITENTIAL_URL

Conditional

Operations Manager base URL

ITENTIAL_API_CLIENT_ID, ITENTIAL_API_CLIENT_SECRET

Conditional

OAuth service-account credentials. The worker refreshes the token itself.

ITENTIAL_API_TOKEN

No

Legacy static token. Use only where OAuth is unavailable.

ITENTIAL_SKIP_TLS_VERIFY

No

true skips certificate verification for Itential requests only. Prefer NODE_EXTRA_CA_CERTS.

Fetch and push

Variable

Required

Description

JIRA_AUTH

For Jira projects

Jira API token (Cloud) or PAT (Data Center) with read access to fetched projects

CONFLUENCE_AUTH

For Confluence spaces

Confluence API token or PAT with read access to target spaces

UDC_API_KEY

For UDC connections

BigPanda API key (Integrations > API > Create API Key) for pushing to api.bigpanda.io

Troubleshooting

Symptom

Cause

Fix

Error message: TEMPORAL_PROXY_URL must be a valid URL

Malformed value

Format is http://[user:pass@]host:port — port required

Error message: TEMPORAL_PROXY_URL scheme must be 'http'

Used https://

Use http://. The proxy hop is HTTP CONNECT; tunneled traffic is still TLS end-to-end

Error message: TEMPORAL_PROXY_URL must include a port

No port specified

Specify the port (for example :8080, :3128)

Error message: username but no password (or vice versa)

Half the credentials

Provide both or neither

Error message: username may not contain ':' or '@'

Reserved character in credential

URL-encode it (@ → %40, : → %3A)

Error message: TLS cert at /certs/tls.crt has no CN in subject

Wrong cert file at that path

Verify the file BigPanda sent is at the path the env var points to

Error message: ENOENT: no such file or directory ... /certs/tls.crt

Cert files not mounted

Verify -v $(pwd)/certs:/certs:ro and that all three files exist in ./certs/

Routing… proxy but never reaches RUNNING; eventually DEADLINE_EXCEEDED

Proxy not allowlisted for CONNECT temporal.bigpanda.io:7233, or performing TLS inspection

Confirm the destination is allowed CONNECT pass-through with no TLS decryption

TLS handshake failed / unable to verify the first certificate / self-signed certificate in chain

Missing or incorrect TEMPORAL_TLS_CA_PATH, or proxy intercepting TLS

Verify ca.crt matches what BigPanda sent and the path is correct; confirm the decryption exclusion is in place

Failed to connect to proxy

Proxy host/port unreachable from the worker host

Verify the proxy URL and that the host can reach it (curl -x http://<proxy-host>:<port> ...)

Container restarts repeatedly

Persistent startup or connection failure

Check docker logs --tail 200 adr-temporal-worker and match the error above

Error message: secret env var <VAR> is not set on this worker

A credential named by an integration or SSH target (v13.10.0 and later) is missing

Add the variable to .env (or your Secret) and restart

http integration rejected …

One entry in ADR_HTTP_INTEGRATIONS failed validation

That entry is dropped; the rest keep working. The message names the reason.

Error message: Itential auth not configured

Itential credentials are missing

Set ITENTIAL_API_CLIENT_ID and ITENTIAL_API_CLIENT_SECRET, or ITENTIAL_API_TOKEN. See Itential.

Error message: JIRA_AUTH env var is required

JIRA_AUTH is not set

Set JIRA_AUTH. See Fetch and push.

Error message: UDC_API_KEY env var is required

UDC_API_KEY is not set

Set UDC_API_KEY. See Fetch and push.

HTTP integration validation errors

Logged as http integration rejected with the reason. The entry is dropped; other entries keep working.

Reason

Fix

extra_headers may not set reserved header '<name>'

Remove it. Authentication belongs in auth.

token_url must be https with no embedded credentials, query or fragment

Use the plain https token endpoint

client_auth 'post' requires client_secret_env

Add "client_auth": "private_key_jwt" or supply client_secret_env

either client_secret_env or private_key_env, not both

Keep only the one your client_auth uses

Unrecognized key: "client_secret"

Move the secret to a variable referenced by client_secret_env

Runtime errors

Message

Meaning

unknown integration '<name>'

A runbook step names a dropped or unconfigured integration

<name>.request: path '<path>' must start with one of: <prefixes>

The path is outside allowed_path_prefixes; not retried. Mind the trailing slash: "/api/" refuses /apifoo, "/api" allows it.

generic-http.request: upstream returned error status http_status: 401

The API refused the credential or token. Check audience, resource, and scope. 403 means the account lacks a permission.

generic-http.request: failed

The token request failed. See OAuth 2.0 token errors (OAuth2TokenError).

Writes are never retried blind

For every auth type, a POST or PATCH that fails in transport with no response is not retried: it may already have been applied.

OAuth 2.0 token errors

An OAuth2TokenError message names the token URL and client_id, then the provider's error code and description, capped at 300 characters with the secret redacted. The raw response body and HTTP status are omitted because providers can echo credentials.

Provider error

Retried

Fix

invalid_client

No

Wrong ID or secret, or try "client_auth": "basic". Also emitted for an HTTP 200 with a {code, message} error body.

invalid_scope, unauthorized_client, invalid_grant, invalid_request, unsupported_grant_type

No

Fix the client's grants or scopes in the provider console

"response" is not a conform Token Endpoint response

Yes

token_url points at the wrong path, or the provider returned a 5xx

error:1E08010C:DECODER routines::unsupported

—

The private key is not a valid unencrypted PEM, or its base64 is broken

Network error, timeout, 5xx

Yes

—

TLS errors

Message

Fix

unable to verify the first certificate / self-signed certificate in certificate chain

Set NODE_EXTRA_CA_CERTS. Itential only: ITENTIAL_SKIP_TLS_VERIFY=true as a quick workaround.

EE certificate key too weak / CA certificate key too weak

Reissue with ≥2048-bit RSA or ≥224-bit ECC. Stopgap: NODE_OPTIONS=--tls-cipher-list=DEFAULT@SECLEVEL=1; keys under 1024 bits still fail.

Itential unreachable at … ENOTFOUND / ECONNREFUSED / The operation was aborted

Wrong ITENTIAL_URL or DNS; Itential down or port 443 blocked; /status took more than 10 seconds.

Useful commands

docker logs -f adr-temporal-worker          # Stream live logs
docker logs --tail 100 adr-temporal-worker  # Last 100 lines
docker restart adr-temporal-worker          # Restart worker
docker stop adr-temporal-worker             # Stop worker
docker rm adr-temporal-worker               # Remove stopped worker
docker images | grep adr-temporal-worker    # Confirm image version

Updating to a new version

When BigPanda releases a new tag, you receive a fresh presigned S3 URL. Your .env and certificates do not need to change:

docker stop adr-temporal-worker && docker rm adr-temporal-worker
# then re-run Download and load the image, and Run the worker

Follow the install guide for the target version; configuration may differ between versions.

Support

Contact BigPanda support for certificate renewal or reissue, changes to what data is fetched, and connectivity or authentication issues.