Logging

One of the features Utopia has is the extensive logging it provides out the box. These can be extremely helpful when it come to fault finding or performance managing.

The logs can be seen via dashboard at the Utopia UI, Hangfire and Elmah

Below are the basic guidelines of where to find the log files which feed the UI's as well as highlight extra log details.

MongoDB:

Connector Logging - Logging the behaviour on each connector run. you can view the raw content that was collected or received, the times collected and process, batch information etc. These can be found by CSC_{{ connectorId }} (CollectionSourceConnector) or CPC_{{ connectorId }} (CollectionPublisherConnector).

Transaction Logging - These are a summary of the documents movements and data, these are used to populate the transaction page data in the UI too. You can identify these by transactions_{{ organisationId }}

Document Data - The data saved by the documents flowing through the system. The snap shots of the data and each collection. (data-in, transform, etc). You can identify these as guids with no prefix which are simply {{ collectionId }}.

MySQL (SQL, which ever table based DB you have selected):

Processing stats - Utopia records all each time a connector runs, how many docs it finds, how long it processed for etc. you can use this table to identify which processes are using your resources and working the hardest. This table can be found at microservices.ProcessStatistics

Connector logging - Here you can view any logging activities created by the Listeners (API's), so every time someone hits the end point a log is created. This activity is saved here and if you have an S3 setup for the cluster, it will also save the request details into S3. This table can be found at microservices.ConnectorLogging

Persistent Grants table - system autorisation token generation and management is logged in this table. If a system or user logs in and creates a token, this will hold its validity and details. This table is found at identity.PersistedGrants

S3 database (optional)

If you have a S3 setup for maintenance the following logs will be pulled out of Utopia and stored in S3 for reference (If the maintenance toggle is selected in the Utopia UI):

Collection Connector Logs - CSC_{{ }} & CPC_{{ }} Transaction Logs - transactions_{{ }} Documents - {{ }} Listener Logs - as described above, S3 file location found in microservices.ConnectorLogging

You can find the maintenance schedules and configs for thse in the microservices deployment:

            - name: DocumentMaintenance__Enabled
              value: 'true'
            - name: DocumentMaintenance__Dispatcher__Cron
              value: 0 09 * * *
            - name: DocumentMaintenance__Backup__Months
              value: '2'
            - name: SystemMaintenance__Enabled
              value: 'true'
            - name: SystemMaintenance__BackupStatistics__Days
              value: '90'
            - name: SystemMaintenance__Dispatcher__Cron
              value: 15 7 * * *
            - name: SystemMaintenance__Backup__Hours
              value: '72'

If you are unsure how to understand the cron numbering please take a look here