openshift-operator

README

Pachyderm Operator

The pachyderm operator is an application responsible for installing, watching and managing Pachyderm resources in your Openshift cluster.

Pachyderm is the data foundation for machine learning. Pachyderm provides industry leading data versioning, pipelines and lineage that allow data science teams to automate the machine learning lifecycle and optimize their machine learning operations (MLOps).

Usage

1. Using AWS S3 for Pachd storage

• Create a secret which contains the AWS S3 storage information

$ oc create secret generic pachyderm-aws --from-literal access-id=ABCDEFGHIJKLMNOPQR --from-literal access-secret=dkhfjdshfj/fjkdshfiuUjmfhdsjkhfjdhs/KLhdfuiseh --from-literal bucket=pachyderm-bucket --from-literal region=us-east-1`

secret/pachyderm-aws created
$

• Create a Pachyderm custom resource in the same namespace

$ cat <<EOF> pachyderm-cr.yaml
apiVersion: aiml.pachyderm.com/v1beta1
kind: Pachyderm
metadata:
  name: pachyderm-sample
  namespace: pachyderm-test
spec:
  console:
    disable: true
  pachd:
    metrics:
      disable: false
    storage:
      amazon:
        credentialSecretName: pachyderm-aws
      backend: AMAZON
EOF
$ oc create -f pachyderm-cr.yaml
pachyderm.aiml.pachyderm.com/pachyderm-sample created
$ 

• Ensure pachyderm is up and running

$ oc get pachyderm pachyderm-sample -o yaml | yq e '.status' -
phase: Running
$   

2. User-provided postgresql database

• Set postgresql to disabled in pachyderm.spec.postgresql

• Provide postgresql instance information in pachyderm.spec.pachd.postgresql

• Create a k8s secret to hold the postgresql password. It should have a key postgres-password