Learn more about activating a Broker VM with a Files and Folders Collector applet.
The Broker VM provides a Files and Folders Collector applet that enables you to monitor and collect logs from files and folders in a network share for a Windows or Linux directory, directly to your log repository for query and visualization purposes. The Files and Folders collector applet only starts to collect files that are more than 256 bytes and is only supported with a Network File System version 4 (NFSv4). After you activate the Files and Folders Collector applet, you can collect files as datasets (<Vendor>_<Product>_raw) by defining the following.
Details of the folder path on the network share containing the files that you want to monitor and upload to Cortex XSIAM.
Settings related to the list of files to monitor and upload to Cortex XSIAM, where the log format is either Raw (default), JSON, CSV, TSV, PSV, CEF, LEEF, Corelight, or Cisco.
Note
Cortex XSIAM only supports ingestion of files encoded in UTF-8 format.
Danger
Before activating the Files and Folders Collector applet, review and perform the following:
Know the complete path to the files and folders that you want Cortex XSIAM to monitor.
Ensure that the user permissions for the network share include the ability to rename and delete files in the folder that you want to configure collection.
Select Settings → Configurations → Data Broker → Broker VMs.
Do one of the following:
On the Brokers tab, find the Broker VM, and in the APPS column, left-click Add → Files and Folder Collector.
On the Clusters tab, find the Broker VM, and in the APPS column, left-click Add → Files and Folder Collector.
Configure the Files and Folder Collector settings.
Field
Description
Folder Path
Specify the path to the files and folders that you want Cortex XSIAM to monitor continuously to collect the files. The following formats are available based on the type of machine you are using:
Windows:
\\<hostname>\<shared_folder>orsmb://<hostname>/<shared_folder>Linux:
/<srv>/<shared_folder>ornfs://<srv>/<shared_folder>Note
When using the Linux file share, including the Linux share with nfs, a Username and Password is not required, so these fields are grayed out in the screen.
Recursive
Select this checkbox to configure the Files and Folders Collector applet to recursively examine any subfolders for new files as long as the folders are readable. This is not configured by default.
Username
Specify the username to access the shared resource using a User Principal Name (UPN) format.
Password
Specify the password to access the shared resource.
Test Connection
Select to validate the connection and permissions.
Field
Description
Mode
Select the mode to use for collecting data. The settings displayed change depending on your selection.
Tail: Continuously monitors the files for new data (default). The collector adds the new data from the files to the dataset.
Batch: Reads the files automatically at user determined intervals, updates the lookup datasets, and then renames or deletes the uploaded source files. Renaming or deleting the read source files ensures that the collector always reads the most up-to-date file. Depending on the Storage Method, the collector can Append the new data from the files to the dataset or completely Replace the data in the dataset.
Note
In Batch mode, the Files and Folders Collector supports collecting logs from a network share for a maximum file size of 500 MB.
Collect Every
This option is only displayed in Batch Mode. Specify the execution frequency of collection by designating a number and then selecting the unit as either Minutes, Hours, or Days.
After Files Uploaded
This option is only displayed in Batch Mode. Select what to do with the files after they are uploaded to the Cortex XSIAM server. You can Rename files with a suffix (default) or you can Delete files. When renaming, the suffix is added to the end of the original file name using the format
<file name>.<suffix>, which becomes the new name of the file.Include
Specify the files and folders that must match to be monitored by Cortex XSIAM. Multiple values are allowed with commas separating the values and are case-sensitive.
Allowed wildcard:
'?' matches a single alphabet character in a specific position.
'*' matches any character or set of characters, including no character.
Example 64.log*.jsonlog*.jsonincludes any JSON file starting with 'log'.Exclude (Optional)
Specify the files and folders that must match to not be monitored by Cortex XSIAM . Multiple values are allowed with commas separating the values.
Allowed wildcard:
'?' matches a single alphabet character in a specific position.
'*' matches any character or set of characters, including no character.
Example 65.*.backupexcludes any file ending with '.backup'.Log Format
Select the Log Format from the list as either Raw (default), JSON, CSV, TSV, PSV, CEF, LEEF, Corelight, or Cisco. This setting defines the parser used to parse all the processed files as defined in the Include and Exclude fields, regardless of the file names and extension. For example, if the Include field is set
*and the Log Format is JSON, all files (even those namedfile.log) in the specified folder are processed by the Files and Folders Collector as JSON, and any entry that does not comply with the JSON format are dropped.Note
When uploading JSON files, Cortex XSIAM only parses the first level of nesting and only supports single line JSON format, such that every new line means a separate entry.
# of Lines to Skip (Optional)
Specify the number of lines to skip at the beginning of the file. This is set to 0 by default.
Note
Use this option only in cases where your files contain some sort of "header" lines, such as a general description, an introduction, a disclaimer, or similar, and you want to skip ingesting them.
Field
Description
Storage Method
This option is only displayed in Batch Mode. Specify whether to Append the read data to the dataset, or to Replace all the data in the dataset with the newly read data.
Append: This mode is useful for log files where you want to keep all the log info from before.
Replace: This mode is useful for adding inventory data from CSV and JSON files which include properties, for example, a list of machines, a list of users, or a mapping of endpoints to users to create a lookup dataset. In each data collection cycle, the new data completely replaces the existing data in the dataset. You can use the records from the lookup datasets for correlation and enrichment through parsing rules, correlation rules, and queries.
Note
When the storing method is Replace, the maximum size for the total data to be imported into a lookup dataset is 30 MB each time the data is fetched.
The inventory data ingested using the Files and Folders collector is counted towards license utilization.
When you use a JOINT function with a lookup table in a query or correlation rule, make sure you configure the conflict strategy to point to the raw dataset. This ensures that the system fields are taken from the raw dataset and not from the lookup table.
Target Dataset
This option is only displayed in Batch Mode when the storing method is Replace. Select the name of an existing Lookup dataset or create a new Lookup dataset by specifying the name.
When you create a new target dataset name, specify a name that will be more meaningful for your users when they query the dataset. For example, if the original file name is
accssusr.csv, you can save the dataset asaccess_per_users.Dataset names can contain special characters from different languages, numbers (
0-9) and underscores (_). You can create dataset names using uppercase characters, but in queries, dataset names are always treated as if they are lowercase.Note
You can't specify a file name that's the same as a system file name.
The name of a dataset created from a tsv file must always include the extension. If the original file name is mrkdptusrsnov23.tsv, you can name save the dataset with the name marketing_dept_users_Nov_2023.tsv.
Vendor and Product
Specify the Vendor and Product for the type of data being collected. The vendor and product are used to define the name of your Cortex Query Language (XQL) dataset (
<Vendor>_<Product>_raw).Note
The Vendor and Product defaults to Auto-Detect when the Log Format is set to CEF or LEEF.
Select Generate Preview to display up to 10 rows from the first file and Preview the results. The Preview works based on the Files and Folders Collector settings, which means that if all the files that were configured to be monitored were already processed, then the Preview returns no records.
(Optional) Click Add Connection to define another Files and Folders connection for collecting logs from files and folders in a shared resource.
(Optional) Other available options.
As needed, you can return to your Files and Folders Collector settings to manage your connections. Here are the actions available to you:
Edit the connection name by hovering over the default Collection name, and selecting the edit icon to edit the text.
Disable/Enable a connection by hovering over the top area of the connection section, on the opposite side of the connection name, and selecting the applicable button.
Delete a connection by hovering over the top area of the connection section, on the opposite side of the connection name, and selecting the delete icon. You can only delete a connection when you have more than one connection configured. Otherwise, this icon is not displayed.
Activate the Files and Folders Collector applet.
After a successful activation, the APPS field displays File with a green dot indicating a successful connection.
(Optional) To view metrics about the Files and Folders, left-click the File connection in the APPS field for your Broker VM.
Cortex XSIAM displays Resources, including the amount of CPU, Memory, and Disk space the applet is using.
Manage the Files and Folders Collector.
After you activate the Files and Folders Collector, you can make additional changes as needed. To modify a configuration, left-click the File connection in the APPS column to display the Files and Folder Collector settings, and select:
Configure to redefine the Files and Folders Collector configurations.
Deactivate to disable the Files and Folders Collector.