APPSeCONNECT executes the runtime engine to bridge the data between applications. The transactions are important and hence we keep track of the transactions that flow from the applications. The importance of traceability lies in the fact that there can be discrepancies during the data transfer. In such a case, we can exactly pinpoint to the developer why this has happened and thereby reduce extended man-hours to fix any issue in the integration. In the case of hosted agent scenario, we require hard disk space in the local drive to store these transactional data. We have a number of databases in the local drive (where APPSeCONNECT is installed) that keeps track of the Sync Info, Snapshots, etc, such that when it is requested by the cloud portal, it can be easily shown in the Portal. But databases tend to become bulky over time because of the huge number of transactions going through the platform regularly. As it costs you disk storage, we cannot keep the data indefinitely. APPSeCONNECT has a build-in capability for data cleanup that is more than 15 days old.
What are the pre-requisites for Data Cleanup?
During the implementation phase, the developer is developing the integration and hence requires to stop the auto cleanup feature. Hence, if the project is in the implementation phase, we do not execute our cleanup job regularly. Your project needs to be in the Live stage to activate the cleanup.
On the premise, it needs 3 times the size of the databases, to ensure our application can properly clean up the data.
Connect all your business applications under one single platform to automate the business process and increase your productivity and efficiency!
What does Data Cleanup Do?
A cleanup rule, as the name suggests, deletes all the data present in the databases beyond the stipulated 15 days. It also clears the log files and metadata caches created during the execution of the integration. The module will start, clean up the data in the databases and the file system, de-fragment the memory used by our tool, and finally start the services.
The cleanup job will execute every 24 hours at midnight in the customer’s environment, such that it does not affect business operations much.
How does Data Cleanup Work?
Data cleanup is a rule defined on our cloud platform. The rule is generic in nature and works by default in the environment. Once the project is set to Live, the rule gets activated. You need to download the rule from the portal to activate it. Once everything is set up, you will see a log entry in the Feeds section of the portal.
You can also go to the environment to see the logs. The logs will ensure whether there are any issues while executing the cleanup job or not. To ensure you have proper knowledge about the execution of the rule, just go to Deploy -> Environments in the project and select your environment. Select Properties Tab on top to get the Agent Logs.
When you click on the button, you will be presented with the logs on cleanup. You can also view the ScheduleRules.txt, to see the detailed steps regarding the execution of the rule.
Here you can clearly see, that the token ClearRuleForResyncLogAndFiles is evaluated to False, which means no clean up is required for a particular time span of 15 days. Here if the process faces any issues, the log will indicate those too.
If you see any errors in the files, please create a support ticket to let us know about it.
Connect all your business applications under one single platform to automate the business process and increase your productivity and efficiency!
What are the steps performed in the cleanup rule?
To understand it better, we encourage you to know the details of the rule. Now, let us take an example of an imaginary environment where the combined database size is 3 GB.
Now to clean up this data, the process does the following steps :
- Backs up the existing databases to a separate folder, such that if somehow the clean up fails, it can revert it back to the original database.
- Executes scripts on the databases which can delete the data from the tables.
- Executes scripts on the file system which can delete all unnecessary log files created during execution.
- Executes defragmentation of databases, which can shrink the data size. Now here, the database copies the existing data present after deletion to a separate Database and deletes the existing ones.
- Finally, it deletes the backup and sends a completion notification to the server.
Now as you can see, if 3GB is the size of the database, we need 9GB of storage available to execute the cleanup job. Here is the calculation :
- 3 GB is required for doing the backup of the data before executing the transaction
- 3 GB is required to defragment the data in the databases.
- 3 GB is the actual storage.
You might argue, that if the data size is changed from 3 GB to something less, do we still need it. For example, if after deletion the size is changed to 2 GB. Yes, in such a case, you don’t need 9 GB, but require 6 GB. For simplicity, we say you need 3 times the size of existing databases to execute the cleanup rules.
Conclusion
The cleanup rule is very important for any environment as it is directly proportional to the performance of the environment. Just like the clean up your operating system does regularly, the clean up is required by APPSeCONNECT too. Knowing the steps better gives you an added advantage to troubleshoot the issues easily and effectively.
Thank you.
Connect all your business applications under one single platform to automate the business process and increase your productivity and efficiency!
