> For the complete documentation index, see [llms.txt](https://datafibers-community.gitbook.io/datafibers-complete-guide/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://datafibers-community.gitbook.io/datafibers-complete-guide/architecture-design.md). # Architecture Design DataFibers application supports operations, such as ***L***ist, ***A***dd, ***U***pdate, and ***D***elete operations, on DF Connects or Transforms. In short, we call it **LAUD** operation. Below are DataFibers sequence diagram for common LAUD operations. * Connect LAUD: Sequence for connect operations. * Transform LAUD: Sequence for transform operations. * Schema Registry LAU: Sequence for schema operations. * Installed L: Sequence for list of installed connects/transforms. * Status Sync.: Sequence for synchronizing status. * Stream Back: Sequence to stream back the batch result to queue. ## Connect LAUD The Connect performs LAUD operation as follows. ![below is the PlantUML code](https://774911537-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHJwcTcVHSoKoQU8lwJ%2F-LmqYjCX3JhIq1xha5b2%2F-LmqYn_D9aZHKY1XvS9G%2FZP713e8m38RlVOfUEEu5Y34eX8jwqN0c2qn3n7LkGsECxsuXa0pYmTlM__kxzb_YlBI-kxK0il9ayQYSBzEsKTe311SKQypoygJiewaKnWpUEvRH4-utHOOfQQqgZn1GxDd9yrMManXZJlOfRJttJBrWJl-m3xQX-FF1bH3_M7WlGzgfvR07qXVAjZ4MffPUCMV8DHpB2s7KKCo22D2fPHXQFyyYjH6RxV-EKEdQj5BtNu7H.png?alt=media\&token=5ca7592b-d5a4-47de-9d87-dd3e48dc3c0f) ``` @startuml activate DF_Service database MongoDB activate RAS DF_Service -> MongoDB : Request list of active Connect|Transform note right: Use vertx.setPeriodic to \ncheck every 10 sec. DF_Service <-- MongoDB : Response list of active Connect|Transform DF_Service -> RAS: Loop to request status for each active Connect|Transform RAS --> DF_Service : Response status for each active Connect|Transform MongoDB <- DF_Service : Request to check/update the status note right: if the status has no change, \ngo to next Connect|Transform MongoDB --> DF_Service : Response Connect|Transform status updated @enduml ``` ## Transform LAUD The Connect performs LAUD operation as follows. ![](https://774911537-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHJwcTcVHSoKoQU8lwJ%2F-LmqZYCpGqLozY0Vf-5M%2F-LmqZaNPS0BTVsc8OL84%2F2.png?alt=media\&token=0208bdca-b7fe-4186-a6bd-8d06f781921e) ``` @startuml actor Rest_Client activate DF_Service database MongoDB activate Kafka_Connect Rest_Client -> DF_Service : Request to LAUD Connect DF_Service -> Kafka_Connect : Forward LAUD Connect Request Kafka_Connect -> Kafka_Connect : LAUD Kafka Connect DF_Service <-- Kafka_Connect : LAUD Kafka Connect Response DF_Service -> MongoDB: Update Repository MongoDB --> DF_Service: Update Response Rest_Client <-- DF_Service : Response LAUD Connect @enduml ``` ## Schema Registry LAU Schema registry is only support ***list***, ***add***, and ***update*** operations because of Confluent API limitation. ![](https://774911537-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHJwcTcVHSoKoQU8lwJ%2F-LmqZYCpGqLozY0Vf-5M%2F-Lmq_6kiciktzplTHoCF%2F3.png?alt=media\&token=02be602c-2d2c-4c00-a658-847ae9492fb9) ``` @startuml actor Rest_Client activate DF_Service activate Schema_Registry Rest_Client -> DF_Service : Request to LAU Avro Schema DF_Service -> Schema_Registry : Forward LAU Avro Schema Request Schema_Registry -> Schema_Registry : LAU Avro Schema DF_Service <-- Schema_Registry: LAU Transform Response Rest_Client <-- DF_Service : Response LAU Avro Schema @enduml ``` ## Install L Installed connect or transform view only supports a read-only ***list*** operation as follows. ![](https://774911537-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHJwcTcVHSoKoQU8lwJ%2F-LmqZYCpGqLozY0Vf-5M%2F-Lmq_aiRFQNWIb4EEbKb%2F4.png?alt=media\&token=51b69515-ddb3-49ab-9a10-0e1f0fd27362) ``` actor Rest_Client activate DF_Service activate Kafka_Connect Rest_Client -> DF_Service : Request to list installed DF_Service -> Kafka_Connect : Forward list installed Request Kafka_Connect -> Kafka_Connect : Get list installed DF_Service <-- Kafka_Connect: list installed Response Rest_Client <-- DF_Service : Response list installed ``` ## Status Sync. There are following functions to get status update regularly. ![](https://774911537-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHJwcTcVHSoKoQU8lwJ%2F-LmqZYCpGqLozY0Vf-5M%2F-Lmq_njthMNhlrzqNEl5%2FZPB1JiCm44Jl_ehz05BW6bMeGCMfIAY56n8opYQssku6UnCLYOzdqn8r0HMu-EBPDpDhpvBOACrsOwoJq5f1cD-zh32smQ4fhDXNcn3kcIgUtnn5o-kLCKSbP5Uz1d9Oubk3IM0JzE0IzbC8jqo4JZuUewLKSjmQOaN5K7d9uKbjMeoocoIK1uo1Y-1069x9UNHhGBrzXyiBIEWcG-zfbetCKysai3_Tn_6rKGuBvhhpZ5yD.png?alt=media\&token=06287e82-6aba-4549-aade-0b5f61c57e38) ``` activate DF_Service database MongoDB activate RAS DF_Service -> MongoDB : Request list of active Connect|Transform note right: Use vertx.setPeriodic to \ncheck every 10 sec. DF_Service <-- MongoDB : Response list of active Connect|Transform DF_Service -> RAS: Loop to request status for each active Connect|Transform RAS --> DF_Service : Response status for each active Connect|Transform MongoDB <- DF_Service : Request to check/update the status note right: if the status has no change, \ngo to next Connect|Transform MongoDB --> DF_Service : Response Connect|Transform status updated ``` ### Connect|Transform Sync. DataFibers application launches a background daemon for regular synchronizing the connect/transform status in the repository (MongoDB) against Kafka/Flink **R**est **A**pi **S**ervices (RAS). ### Metadata Sync. DataFibers application does not involve into the metadata, which keeps track of files processed by connects or transforms, synchronize path directly. For instead, it creates the metadata schema, and then launches the MongoDBSink to ship data into repository. ![](https://774911537-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHJwcTcVHSoKoQU8lwJ%2F-LmqZYCpGqLozY0Vf-5M%2F-Lmq_zoBkPqgYVVYmfr0%2F6.png?alt=media\&token=f84da68a-66c1-4ad7-9da2-22e36e76d588) ``` activate Connect activate Kafka activate Kafka_Connect database MongoDB Connect -> Kafka: Send metadata note right: Send to df_meta topic Kafka <- Kafka_Connect : Fetch data form df_meta note left: MongoDBSink Daemon Kafka --> Kafka_Connect : Response metadata from the topic Kafka_Connect -> MongoDB: Sink metadata Kafka_Connect <-- MongoDB: Response sink done ``` ### Startup Sync. When DF starts, it first imports all available connect from repository and synchronize their status with Kafka connect. > We do not create start-up synchronize for transforms since we are not able to rebuilt the task configurations if we import the active transforms from Flink. ![](https://774911537-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHJwcTcVHSoKoQU8lwJ%2F-LmqZYCpGqLozY0Vf-5M%2F-LmqaDX-wxlgTo2KnCiQ%2F7.png?alt=media\&token=2eb43471-8f8a-46ae-b909-2b25906dd462) ``` activate DF_Service database MongoDB activate Kafka_Connect DF_Service -> MongoDB: Request list of connect DF_Service <- MongoDB : Response list active connect name DF_Service -> Kafka_Connect : Request status from the connect note left: Foreach connect returned Kafka_Connect -> Kafka_Connect: Get connect status DF_Service <-- Kafka_Connect: Response connect status DF_Service -> MongoDB: Request compare/update the connect status DF_Service <-- MongoDB : Response status updated ``` ## Stream Back. When batch processing is complete, DF supports to stream back the result set to the queue for further consuming or transformation. It first exports the result set to a Json file. Then, it leverages file connector to send the file content to the queue when df status sync. daemon detects the batch file export is done. Once succeeded, the stream back connector will be deleted. ![](https://774911537-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LHJwcTcVHSoKoQU8lwJ%2F-LmqZYCpGqLozY0Vf-5M%2F-LmqaQ9yg9E_egsAP7J2%2F8.png?alt=media\&token=eb69d141-a881-4a4d-8432-658765ccfba4) ``` activate DF_Service activate DF_Status_Sync_Service activate Kafka_Connect DF_Service -> Batch_Transform : Request batch transform Batch_Transform -> Batch_Transform: Transform and export to json DF_Status_Sync_Service <-- Batch_Transform: Response complete and change to streaming status DF_Status_Sync_Service -> Kafka_Connect: Request stream the json to queue DF_Status_Sync_Service <-- Kafka_Connect : Stream back is completed and delete the connect ```