This is a sample code for an Oracle Functions written in Golang that can be used in an OCI Data Integration (DI) Pipeline to zip files generated from an OCI DI Dataflow for Fusion SaaS data load.
For more information please refer to the blog post.
Oracle Functions is a fully managed, multi-tenant, highly scalable, on-demand, Functions-as-a-Service platform. It is built on enterprise-grade Oracle Cloud Infrastructure and powered by the Fn Project open source engine. Use Oracle Functions (sometimes abbreviated to just Functions) when you want to focus on writing code to meet business needs.
Data Integration is a fully managed, multi-tenant service that helps data engineers and ETL developers with common extract, transform, and load (ETL) tasks such as ingesting data from a variety of data assets; cleansing, transforming, and reshaping that data; and efficiently loading it to target data assets.
The Oracle API Gateway service enables you to publish APIs with private endpoints that are accessible from within your network, and which you can expose with public IP addresses if you want them to accept internet traffic. The endpoints support API validation, request and response transformation, CORS, authentication and authorization, and request limiting.
The function will read the transformed files from the requested bucket, zip the files and upload to the requested bucket.
If the upload process is completed with success the original file is deleted.
Ensure Oracle Cloud Functions is installed and that you can deploy a simple Golang based cloud function. If you haven't already installed and configured Oracle Cloud Functions then we can recommend going through the Oracle Cloud Functions quickstart as this will not only help you setup and configure your environment but also show you how to deploy some sample Oracle Cloud Functions.
fn deploy --app <app-name>
The expected request payload example :
{
"namespace": "mynamespace",
"sourceBucketName": "erp-transformed",
"targetBucketName": "erp-zip",
"prefix": "ArUpdCustomers"
}
namespace : bucket namespace
sourceBucketName : Transformed file source bucket name
targetBucketName : Target bucket name where the zip files are going to be uploaded
prefix : Folder name where the files are going to be read and copied from
echo -n '{"namespace":"mynamespace","sourceBucketName":"erp-transformed","targetBucketName":"erp-zip","prefix":"ArUpdateCustomer"}' | fn invoke <app-name> zipfiles
Response example :
{
"ProcessedWithSuccess": [
"customer-import/part-00000-2374e489-0ffd-47d1-8eb8-9eb623d65986-c000.csv"
],
"ProcessedWithError": [
{
"Name": "customer-import/part-objecterror.csv",
"Error": "file error deleting object from original storage"
}
]
}
Oracle takes security seriously and has a dedicated response team for reporting security vulnerabilities and to answer any security and vulnerability related questions.
We welcome all contributions to this sample and have a contribution guide for you to follow if you'd like to contribute.
If you need help with this sample, please log an issue within this repository and the code owners will help out where we can.
Developers choosing to distribute a binary implementation of this project are responsible for obtaining and providing all required licenses and copyright notices for the third-party code used in order to ensure compliance with their respective open source licenses.
Copyright (c) 2022, 2024 Oracle and/or its affiliates.
Released under the Universal Permissive License v1.0 as shown at https://oss.oracle.com/licenses/upl/.