README
¶
go-k8s-process
go-k8s-process
This is the service that will process/rebuild the file, as per the projet defined in this repo These pods are controlled the rebuild pods controller.
Dev section
- the k8-process accept a rabbitmq header "content-management-policy" that is expected to contain the content management policy in json format
if the content-management-policy header is empty , the k8-process will use its default content management policy
if the content-managment-policy json content is malformed , the k8-process will return "INTERNAL ERROR"
the k8-process add additional values to the rabbitmq header wich contain additional inforamtions :
- "rebuild-sdk-version" which is the sdk version
- "rebuild-processing-status" which is the status of the rebuild processing operations , there is five states values :
"UNPROCESSABLE" if the file can't be processed for example not supported file type
"CLEAN" if the original file is clean so it remain unchanged
"CLEANED" if the file is cleaned by the rebuil engine
"INTERNALL ERROR" if there is error in the k8-process or the rebuilt ClI while procesing the file,
"SDK EXPIRED" if the rebuilt SDK expired
- "clean-presigned-url" which is the rebuilt file presigned url
- "report-presigned-url" which is the report file presigned url
- "gwlog-presigned-url" which is the glassWall log file presgined url
- "log-pesined-url" which is the log file presgined url
Steps of processing
- Listens on the processing request queue to get files pending for a rebuild
- Get the file on queue
- Copy it to local folder
- Run the rebuild service
- Get the rebuild result
- Upload the result file to minio
- Notifies the queue
- Exit
Sdk-rebuild-eval version
1.233
Docker build for local testing
- To build for local testing
git clone https://github.com/k8-proxy/go-k8s-process.git
cd go-k8s-process
git submodule update --init
go build -o go-k8s-process ./cmd
export ADAPTATION_REQUEST_QUEUE_HOSTNAME='<rabbit-host>' \
ADAPTATION_REQUEST_QUEUE_PORT='<rabbit-port>' \
MESSAGE_BROKER_USER='<rabbit-user>' \
MESSAGE_BROKER_PASSWORD='<rabbit-password>' \
MINIO_ENDPOINT='<minio-endpoint>' \
MINIO_ACCESS_KEY='<minio-access>' \
MINIO_SECRET_KEY='<minio-secret>' \
MINIO_SOURCE_BUCKET='<bucket-to-upload-file>' \
GWCLI='<cli-path>' \
INICONFIG='<ini-config-path>' \
XMLCONFIG='<xml-config-path>'
Installing the Glasswall Rebuild SDK
Copy the libglasswall.classic.so shared library into the /usr/lib folder.
cp ./sdk-rebuild-eval/libs/rebuild/linux/libglasswall.classic.so /usr/lib
Inform System about the Glasswall Rebuild SDK
Once in place the library needs to be registered to make it accessible. Create a glasswall.classic.conf file, with the installed location
echo "/usr/lib" > glasswall.classic.conf
Update the etc directory
sudo cp glasswall.classic.conf /etc/ld.so.conf.d
Run ldconfig to configure the dynamic linker run-time bindings
sudo ldconfig
Check that the Glasswall library has been installed
sudo ldconfig -p | grep glasswall.classic
Remove the .conf file
rm glasswall.classic.conf
Docker build
- To build the docker image
git clone https://github.com/k8-proxy/go-k8s-process.git
cd go-k8s-process
git submodule update --init
docker build -t <docker_image_name> .
- To run the container First make sure that you have rabbitmq minio and jaeger running, then run the command bellow note to activate Jaeger trace set JAEGER_AGENT_ON=true
docker run -e ADAPTATION_REQUEST_QUEUE_HOSTNAME='<rabbit-host>' \
-e ADAPTATION_REQUEST_QUEUE_PORT='<rabbit-port>' \
-e MESSAGE_BROKER_USER='<rabbit-user>' \
-e MESSAGE_BROKER_PASSWORD='<rabbit-password>' \
-e MINIO_ENDPOINT='<minio-endpoint>' \
-e MINIO_ACCESS_KEY='<minio-access>' \
-e MINIO_SECRET_KEY='<minio-secret>' \
-e MINIO_SOURCE_BUCKET='<bucket-to-upload-file>' \
-e JAEGER_AGENT_HOST='<jaeger-host>' \
-e JAEGER_AGENT_PORT='<jaeger-port>' \
-e JAEGER_AGENT_ON=true \
--name <docker_container_name> <docker_image_name>
Testing steps
-
Run srv1 as per https://github.com/k8-proxy/go-k8s-srv1
-
Run the container as mentionned above
-
Publish data reference to rabbitMq on queue name : adaptation-request-queue with the following data(table) :
- file-id : An ID for the file
- source-file-location : The full path to the file
- rebuilt-file-location : A full path representing the location where the rebuilt file will go to
- Check your container logs to see the processing
docker logs <container name>
known issues
the rebuild CLI have some problems working on alpine , maybe there is missing dependencies or directory structrure , so instead we use ubuntu base image for the moment
Rebuild flow to implement
Directories