4. Known problems¶
4.1. Status of last updates¶
When restoring a GCenter, information related to the signature update status on GCaps is not restored. The status will update when the GCap retrieves a new rule file.
Workaround : Regenerate the rule file in Sigflow Manager.
4.2. Pairing to a GCAP not possible if no gateway indicated for VPN interface¶
Pairing between a GCap and a GCenter will fail if no default gateway is indicated when configuring interface mgmt0 of the Gcenter. GCap returns following error message during pairing : “Can’t connect to <Gcenter IP>”.
This happens even if the GCap and the GCenter belong to the same subnetwork and therefore no default gateway should be mandatory.
Workaround: Indicate a default gateway, whatever it is.
4.3. Pairing to a GCap not possible after GCenter network configuration has been changed¶
Following a modification of network configuration in the GCenter VPN interface (e.g. : IP, subnet, FQDN), pairing back to a previously paired GCap might not be possible. GCap returns following error message during pairing : “pairing not established”.
Workaround: Please contact Gatewatcher support.
4.4. LastInfoSec rules¶
Inconsistency between the LIS rules and the generated file, as the rules with the hashes are missing.
Workaround: No solution.
4.5. GUM error management in Local or Online mode¶
The GCenter webui does not display errors related to a malfunction of the GUM module (GATEWATCHER Update Manager).
Workaround: Error message available in the GCenter logs.
4.6. Double booting¶
As of version 2.5.3.100, dual booting is no longer supported.
Workaround: No solution.
4.7. Machine Learning engine and CIE editing¶
The GATEWATCHER tables of the Machine Learning engine do not take into account the license restriction if the GCenter is a CIE edition.
Workaround: No solution.
4.8. Self-signed certificate¶
An error may appear from the GCenter WebUI when applying self-signed certificates. This does not affect the proper operation of encrypted authentication.
Workaround: No solution.
4.9. Netdata export - temporary lack of information¶
When repeatedly enabling/disabling the netdata export, the monitoring information related to the detection probes may become momentarily unavailable for a period of 5 to 20 minutes.
Workaround: No solution.
4.10. Netdata export - Incompatibility with Netdata versions higher than 1.19¶
Exporting GCAP/GCENTER monitoring statistics to an external Netdata is only compatible with a Netdata server whose version is equal or lower than 1.19. In higher versions, the data is exported and can be queried within the external Netdata. However, an error in the graphic interface occurs and it is impossible to view the data. This does not affect GCenter, only the external Netdata server.
Workaround: Use an external Netdata with version 1.19.
4.11. GUM - Configuration Frequency is not preserved after upgrade¶
When upgrading from 2.5.3.100 to 2.5.3.101, the GUM update configuration panel exhibits a display problem when viewed for the first time.
Workaround: Refresh the page.
4.12. GScan - Edition Critical Infrastructure Edition (CIE)¶
The GScan functionality does not take into account the license restriction if the GCenter is a CIE edition.
Workaround: No solution.
4.13. Sigflow Manager - Transform Category¶
Applying a Transform category raises a 500 error if no ruleset is available on GCenter.
Workaround: Creating a ruleset.
4.14. Sigflow Manager - Error 500 when adding a rule to a custom source¶
Adding a rule raises a 500 error if the following conditions are present:
The rule is added by editing a custom source;
The rule already exists in another custom source (same SID).
Workaround: Change the rule’s SID that is to be added in order to avoid the SID conflict.
4.15. Sigflow Manager - Inconsistency in the display of the number of categories and rules of a category¶
The Sigflow homepage > Sources shows the number of categories and rules contained in each source. It is possible that the information displayed is inconsistent with the sources’ actual content. This situation may occur after editing a custom source or an update.
Workaround: No workaround.
4.16. DGA - Field not present¶
The absence of the dga_probability field in the events will be done if the following conditions are met:
The activation of logging on DNS event types;
Activation of the DGA Detection Machine Learning module;
A heavy DNS network load.
Workaround: No solution.
4.17. Third Party - Intelligence¶
Interconnection configuration with intelligence raises a 500 error if the token is incorrect.
Workaround: Apply a valid token.
4.18. Kibana - Inaccessible tables¶
KIBANA tables may not be displayed after a restart on GCenter and/or the WEB interface. The error message displayed is “Elastic did not load properly. Check the server output for more information”
Important
This problem is most likely to occur when starting the GCenter for the first time after an upgrade.
Workaround: The bug is corrected with the following manipulation: In the console, via gcenter-setup: GApps Management > Restart a GApp > WebUI Service #2
4.19. Kibana - inconsistency between timestamp fields¶
In Kibana, malware logs have three timestamps:
timestamp_detected: the point in time when the file was captured by the GCap;
timestamp_analysis: the time the file is processed by the GCenter;
and timestamp_last_malcore_analysis: in the event that the file had already been analysed, the time of its last analysis.
There is a format inconsistency in Kibana in the display of these fields, causing the first two timestamps to be reported at the local time configured in Kibana, while the third is displayed in UTC.
This can give the impression that timestamp_last_malcore_analysis is inconsistent with the other timestamps.
By default, Kibana takes the time zone of the browser used for the consultation. The discrepancy observed will therefore be equal to the difference between the browser’s time zone and UTC.
Workaround: As administrator, go to Kibana > Stack management > Index pattern > malware* > refresh field list > refresh (top right). This will eliminate the problem.
4.20. Kibana - “Not ready yet”¶
In a few special cases, a failure of the log rotation system can lead to the /var/log/ partition being saturated. This results in a “not ready yet” error message in Kibana.
Workaround: Connect via SSH to the GCenter and go to the GApps Management > Restart a GApp menu. Then proceed to restart the GApps in this order: - Threat analysis and retroactive orchestrator Service - DGA Engine - Kibana Service If this procedure does not work, contact Gatewatcher support.
4.21. Kibana - Maps GeoIP¶
Viewing GeoIP information within Kibana dashboards requires internet access in order for the map backgrounds to be downloaded.
Workaround: No solution.
4.22. Kibana - UPGRADE¶
When upgrading from version 2.5.3.100 to version 2.5.3.101, a corruption of .kibana indexes may occur, making KIBANA dashboards unavailable and displaying following message : “Kibana server is not ready yet”.
Workaround: Apply HF2 upgrade of GCenter version 101.
Should the issue happen again, the GCenter-Setup menu may be used to clean the .kibana index. To do so, in the GCenter GUI (via SSH or console), GApps Management > Reset a GApp > Wipe Elasticsearch index. Using this menu will erase all custom Kibana dashboards, but will preserve business data.
4.23. Malcore - Profiles not saved post upgrade¶
When upgrading from 2.5.3.100-hfx to 2.5.3.101, the profile configuration via Malcore Management is not maintained.
In particular, the parameter concerning the maximum size of files analysed by malcore, if it was at its default value in v100, it will be reduced to 2Mb.
Potentially, files rebuilt by GCap would no longer be scanned. This would result in the value File size exceeded in the total_found field of the malcore logs.
Workaround: Reapply the Malcore Management profile configuration and apply the recommended value of 100 for the parameter. Maximum size of scanned files (in MB) in the Default profile accessible from the Administrator/Malcore Management/Profile/Default menu.
4.24. Malcore - not working after a version update¶
Updates can be configured to take place automatically via GUM, which is accessible from Administrator/GUM/Config/Enabled.
If automatic updates are enabled and an upgrade is made from 2.5.3.100 to 2.5.3.101, Malcore will not be operational.
Workaround: From within Administrator/GCenter/Configuration/Proxy settings, the proxy must be enabled/disabled via the Enable web proxy setting, then reset to its original status. That is to say, if it was enabled, untick it (unticked, then save) and re-enable it (tick then save). Conversely, if it was disabled, enable it with any configuration and then disable it.
Important
The proxy server suffers from the bug Proxy - Error 500 when resolving name is impossible: Also, if a temporary configuration is needed to enable/disable the proxy, use an IP or a domain name reachable by the GCenter. You can use the GCenter IP address itself or 127.0.0.1 for example.
4.25. Malcore Management - GScan Profile¶
The Number of files option in Malcore Management’s GScan profile enables an alert to be issued based on the number of files in the archive. This feature is not operational.
Workaround: No workaround.
4.26. Malcore - Incorrect healthcheck status in Critical Infrastructure Edition (CIE) license¶
The health status of the malcore engine may be incorrectly displayed on the home page in global status/healthcheck. This can happen if the GCenter is running with the CIE license. The healthcheck may then display Malware Analysis engine has one or more issues, even if the engine is functional.
Workaround: No solution.
4.28. Malcore - File analysis¶
When malcore scans a file the first time, it generates a log with a replica=false field.
If that file is viewed again before the end of the file_resend_interval period, malcore does not rescan the file. It creates a log with a replica=true field. These scans of previously observed files are called replicas. (The file_resend_interval is configurable in the Sigflow/GCap Profiles settings of the GCap in Base variables and defaults to 24 hours).
Malcore ensures that each file is analysed at least once. In rare cases, and when a large number of replicas are observed, it is possible that some replica=true scan logs are not generated. Or on the contrary, they are observed in several copies. In the latter case, the try_count field - which is internal to the GCenter operation - is incremented between each copy.
Workaround: If a file with a large number of replica=true occurrences is to be found with certainty:
Find the SHA256 of the file in question in the data produced by malcore (field SHA256);
Filter the file reconstruction metadata of sigflow with this SHA (field fileinfo.sha256).
4.29. Malcore - No flow_id¶
In rare cases, the flow_id field of a malcore alert may not appear. Correlation with the metadata for this malcore event can be made using the SHA256 and timestamp_detected of the malcore alert.
Workaround: No solution.
4.30. Malcore - Black/White list configuration and application¶
It is possible to add White/Black lists of files that malcore should not scan.
The configuration of this feature can be found in Gcenter/Malcore Management/Whitelist(Blacklist).
The white/blacklisting of a file is not immediately effective. It becomes active after a period of time less than or equal to the file_resend_interval configured on the GCap observing the file. (The file_resend_interval is configurable in the Sigflow/GCap Profiles settings of the GCap in Base variables and defaults to 24 hours).
Workaround: No solution.
4.31. Malcore - Duplicate Analysis¶
Malcore analysis duplicates can occur during elasticsearch database shrinking operations. These operations take place every day at 02:00 UTC. They aim at optimising the memory consumption of elasticsearch by reducing the number of shards per index.
Workaround: No solution.
4.32. Malcore - Engine crash due to an overload¶
The malcore engine can become unstable if it is under extreme load and hundreds of thousands of files are waiting to be processed. This results in a total blockage of the engine (no more analysis) or a very significant reduction in the number of analyses being performed.
Workaround: In Gcenter-setup: Gapps Management > Reset a GApp > Reset Malcore Engine.
4.33. Malcore - Analysis engine saturation¶
If files are rebuilt faster than they are processed by malcore, a queue is created in the GCenter, causing a saturation of AV engines and a loss of real-time malcore alerts.
Workarounds:
Reducing the quantity of files being rebuilt and analysed by malcore, in the Operators > Sigflow > GCAP Profiles > Files rules management menu.
Check if the file_resend_interval value is at least 24h (default value) in the Operators > Sigflow > GCAP Profiles menu.
Upgrade to version 2.5.3.101 Hotfix 2 which significantly improves performances.
4.34. Malcore - Service stop due to saturation¶
In rare cases when the queue is filled with several thousand files of important sizes, service may slow down or stop.
Workaround: Apply HF2 upgrade of GCenter version 101.
4.35. Malcore - AV engine deactivation¶
Malcore is composed of 16 detection engines. One engine causes malfunctions. It has been deactivated since version 2.5.3.101-HF2.
This can be seen in the total_found field of Malcore logs, which is now XX/15 instead of XX/16.
Workaround: Apply HF3 upgrade of GCenter version 101.
4.36. Malcore - Log export with flow_id=0¶
In rare cases, the flow_id field of Malcore logs is not defined, which prevents them from being exported.
Workaround: Apply HF2 upgrade of GCenter version 101. Logs without flow_id are exported with value flow_id=0.
4.37. Malcore - Inconsistency between webui healthcheck and updates status¶
On the GCenter administrators home page, there is an inconsistency between the “Update Status” of the “Global Status” panel, and the “Malcore Update Status” panel. These two panels alert users when updates are more than 7 days old. While the former checks if updates are strictly more than 7 days old, the latter checks if updates are 7 or more days old.
Workaround: No solution.
4.38. Malcore - Error code 3¶
In some cases the files are not scanned by Malcore and store in /data/extraction/3. This code indicates that the file could not be analyzed by Malcore.
Workaround: Apply HF3 upgrade of GCenter version 101.
4.39. Malcore - Error code 10¶
In some cases the files are not scanned by Malcore and store in /data/extraction/10. This code indicates that no engine was not up to scan the file. This can append in CIE while updating Malcore engine.
Workaround: Apply HF3 upgrade of GCenter version 101. This upgrade limits the number of files with error code 10.
4.40. Malcore error filling app_proto field¶
In malcore logs, the app_proto field indicates which protocol transported the analysed file.
If a file is transported by different protocols (e.g. HTTP then SMTP) within the file_resend_interval (which can be configured in Operator > GCap profiles > Base variables > File resend interval) :
a first log with fields raplica=false and app_proto=HTTP will be produced;
a second log with fields replace=true will be produced. The app_proto field remains at HTTP, while it should have been set at SMTP.
Workaround: Use the flow_id field to cross-check malcore alerts and metadata.
4.41. API - Authentication parameter¶
Requests to the GCenter API use the API-KEY keyword to provide the authentication token as a parameter.
In swagger (https://HOSTNAME-GCENTER/docs/swagger/) the example of generated requests use the keyword apikey.
Workaround: In swagger queries, replace the keyword apikey with API-KEY.
4.42. API - endpoint /api/alerts not working¶
The /api/alerts endpoint of the GCenter API is not working:
When using descending date sorting, a 500 error is returned if the page parameter is not set or equals 1.
The page parameter determines the number of results returned instead of the specified page.
The page_size parameter is not taken into account.
Workaround: View alerts via the API by using:
The /api/alerts/clusters endpoint: alerts are then grouped by IP address and time period;
Or directly by making an elasticsearch request via the API endpoint api/data/es/search.
4.43. Payload and Payload printable options - Drop events¶
Under certain circumstances, and due to poor processing of the JSON format by suricata, the GCap may generate alert logs larger than 65kb. These logs are then lost. They may cause the subsequent log to be malformed in some way.
This anomaly can occur if suricata inserts a large amount of data into the logs, especially when the following options are enabled simultaneously:
payload,
payload_printable,
packet,
http_body
and http_body_printable.
These options can be set in Sigflow/GCap Profiles of the GCap in Base variables.
Workaround: Do not simultaneously enable the payload, payload_printable, packet, http_body, and http_body_printable options.
4.44. Proxy - Error 500 if unable to resolve name¶
If the proxy specified in Configuration/Proxy Configuration cannot be resolved by the DNS server configured for the GCenter, then this produces two errors:
A 500 error in the proxy configuration page (/configuration/proxy_settings/);
An error in the GUM configuration menu (/gum/configuration).
Workaround:
Use a proxy that can be resolved by the configured DNS;
Or set the IP address of the proxy server directly. In which case there is no need for DNS resolution.
4.45. Gcenter-setup - error message¶
When running gcenter-setup, the following error message may appear:
`Could not connect to home directory /nonexistent: No such file or directory`.
This does not interfere with GCenter’s operation in any way.
Workaround: No solution.
4.46. LDAP Configuration - TLS¶
User management can be achieved by connecting the GCenter to an Active Directory or other solution running LDAP via the Accounts/LDAP configuration menu.
If an LDAP server is used with TLS settings, the status visible in the LDAP interconnection status control panel may indicate an error although the configuration is operational. The error displayed is as follows:
`Cannot connect to LDAP with current settings: {'desc': "Can't contact LDAP server",'errno': 115, 'info': '(unknown error code)'}`.
Workaround: No solution.
4.47. LDAP with SSL or STARTTLS¶
If LDAP is configured with SSL or STARTTLS and uses a certificate to validate the server, it may disappear when changing the configuration via the GCenter WebUI. However, it is well preserved and used.
Workaround: No solution.
4.48. LDAP - GCenter doesn’t close connections¶
When an LDAP server is used to authentify users, the GCenter doesn’t close established connections with the LDAP server, and reopens a new one for each query.
Workaround: No solution.
4.50. Syslog Export - False Enrichment¶
When malcore alerts are exported at the syslog level, a number of enrichments occur in the http and smtp fields:
smtp.mail_from,
smtp.rcpt_to,
email.from, email.to,
email.cc,
email.bcc,
email.in_reply_to,
http.hostname,
http.url,
http.http_refer,
http.http_user_agent.
These enrichments occur between the SHA256 of the file analysed by malcore, and the most recent metadata from sigflow relating to the same SHA256.
If the file is viewed a significant number of times, and if the analyses produced by the GCenter are performed with a high latency due to a heavy load, it is possible that the most recent metadata does not correspond to the analysed file in question. Under these conditions, the enrichment may be false.
Workaround: At a SIEM level, use the flow_id field to correlate malcore alerts with sigflow metadata.
4.51. Syslog export: no malcore analysis of “unknown” files¶
A bug affecting the suricata engine under very specific conditions can result in the appearance of so-called unknown files. This means that the metadata could not be recovered by suricata. See detailed description of conditions here.
The malcore analyses for these files are viewable in Kibana, but are not exported via syslog.
Workaround: In addition to the query via Kibana, it is possible to perform an elasticsearch query via the GCenter API.
4.52. Syslog export: behaviour during saturations¶
If the throughput of the logs to be exported is such that it saturates the syslog export, gcenter will first process the metadata events in a best-effort manner (possible losses) in order to preserve the export of sigflow, malcore, and codebreaker alerts.
Workaround: Apply HF2 upgrade of GCenter version 101, which ensures at-least once delivery of all syslog exported messages.
4.53. Syslog export - maximum size of exported logs¶
The syslog export truncates logs larger than 65kb.
Workaround: No solution.
4.54. Syslog export - Exceptions in log formats¶
Minor inconsistencies may exist in malcore (malware index) logs when exported.
The following fields can be of integer type, without quotes around the field value, or of string type with quotes:
src_port,
dest_port,
detail_scan_time.
For example:
“src_port”: “25”
or “src_port”: 25.
Workaround: No solution.
4.55. Syslog export - duplicate sigflow alerts¶
In the syslog export, logs of type “sigflow alert” (type=suricata AND event_type=alert) are sent twice.
Workaround: Apply Hotfix 1. For MPL clients (no HF application possible) a procedure can be provided by gatewatcher support.
4.56. Wrong redirect between Trackwatch logs and dasbboard Syslog¶
Upon clicking on Administrator > GCenter > Trackwatch logs, the user is redirected in the “Tactical” dashboard instead of the “Syslog” dashboard. Workaround: No solution.
4.57. Exception caused by the megaraid driver¶
A problem with the megaraid driver built into the OS can cause the GCenter to freeze, requiring it to restart. A patch is currently under development.
Workaround: Apply HF2 upgrade of GCenter version 101.
4.58. Blocked Powershell analysis¶
The number of pending powershell analysis on the home page (Live Critical Indicators) may seem high and rise quickly, therefore giving the impression that the engin is blocked.
The reality is, the engine is working and analysis is still taking place. The counter has a coherence issue : some files that cannot be analysed by the module remain in the queue
This is not a blocking issue : Files are stored during the GCenter’s retention period (Configuration > Global settings > Data retention (in days)). They are then deleted.
Workaround: Apply HF3 upgrade of GCenter version 101.
If you are in version 2.5.3.101-HF2 or below, to set Powershell’s Live Critical Indicators back to 0, one may contact support to get the procedure.
4.59. Cold or hot data handling exception¶
Data indexed in elasticsearch is handled following a hot-cold architecture. Recent data is on a SSD, while slower data is moved on slower drives.
This data management is not working properly, leading to the saturation of some drives, and the malfunction of several parts of the GCenter (log indexing, malcore alert creation, etc.)
Workaround: Check the occupancy rate on the /es partition : this can be done in two different ways :
As an administrator, on the GCenter homepage, the occupancy rate is provided in Live Critical Indicators, at the Elasticsearch index size (used) row
As an administrator, in Administrators > GCenter > Monitor > BASIC HOST STATS
If the occupancy rate is lower than 65%:
Upgrade to version 2.5.3.101 Hotfix 2, which corrects the issue for all new data.
If the occupancy rate is higher 65%:
The hotfix doesn’t have a retroactive effect on former data, it is highly recommended to erase some data before upgrading.
To do so, please go to Administrators > GCenter > Data Management > Data deletion, and check the Malcore, Codebreaker, Sigflow and Syslog boxes.
Using the date selector, select the oldest data available on the GCenter. The retention period is 15 days by default, and can be configured in Administrators > GCenter > COnfiguration > Global settings > Data retention in days. In the default case, start by erasing data older than 15 days. Then 14 days, and so on until the /es occupancy rate is lower than 65%.
Upgrade to version 2.5.3.101 Hotfix 2, which corrects the issue for all new data.
4.60. Filebeat instability¶
During the massive injection of data from the GCap, Filebeat could report processing errors
Workaround: Apply HF3 upgrade of GCenter version 101.
4.61. Default accounts reactivation¶
When configuring a GCenter, operator and administator accounts must be disabled and/or have their password changed.
Upon upgrading, these accounts are reactivated and their values are set back to default.
Workaround: Disabling these accounts or changing their passwords is necessary.
4.62. GSCAN - maximum analyzed file size¶
The maximum size of files analyzed by GScan is 10Mo, even if configured higher in Administrators > Gcenter > Malcore management > Profiles > GScan profile settings > Maximum size of scanned files.
Workaround: No solution.
4.63. CIP/ENIP protocol activation by default¶
Parsing of CIP/ENIP protocol is enabled by default and may not be disabled in the Gcenter interface.
Workaround: No solution.
4.64. Display bug to add IPs in external_net¶
In Operators > Sigflow > Gcap profiles > Network variables, when one tries to add an EXTERNAL_NET of type list with a mask different from /24, a display bug prevents from adding the network.
Workaround: Refresh the page.
4.65. IPsec - Can not establish IPsec link in NAT network¶
IPsec link can’t be established between a GCenter and a GCap in NAT network due to IPtables rules in GCenter.
Workaround: Apply HF3 upgrade of GCenter version 101.