Filebeat multiline filter doesn't work with txt file-CodePudding

I use the filebeat to collect data from .txt file.I'm trying to use Filebeat multiline capabilities to combine log lines into one entry using the following Filebeat configuration:

filebeat.inputs:

- type: filestream
  enabled: true
  multiline.pattern: '^[0-9]{2}\/[0-9]{2}\/[0-9]{4}'
  multiline.negate: true
  multiline.match: after
  paths:
    - .\My.log

output.logstash:
  hosts: ["localhost:5044"]

This is an example of logs. I want to merge stack trace logs.

18/11/2021 19:17:25,717 [96] ERROR B2XPPA.Web.UI.Utilities.GlobalExceptionFilter -
System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at B2X.AddressRestfulAPIClient.RestfulWebAPIClient.GetDetails(String url)
   at B2X.AddressValidation.AddressValidation.GetEircodeAddress(String Eircodeid)
   at B2XPPA.Web.UI.Services.DropdownsPPAService.GetEircodeAddress(String eirCodeId)
   at B2XPPA.Web.UI.Models.Populators.PPADetailsPopulator.Populate(IViewModel viewModel, Quote quote)
   at B2X.Services.ApplicationService.Application`1.PopulateDetailsView(DetailsViewModel viewModel, String referenceNumberInController, Quote quote)
   at B2XPPA.Web.UI.Controllers.QuoteController.Create(QuoteDetailsViewModel viewModel)
   at lambda_method(Closure , ControllerBase , Object[] )
   at System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters)
   at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)

However, all the logs in Kibana continue to be separated at each new line, and no multiline formatting is happening at all. Can anyone help me out with this?

CodePudding user response：

Your multiline pattern is not matching anything.

The pattern ^[0-9]{4}-[0-9]{2}-[0-9]{2} expects that your line to start with dddd-dd-dd, where d is a digit between 0 and 9, this is normally used when your date is something like 2022-01-22

But your line starts with the following pattern dd/dd/dddd, so you would need to change your multiline pattern to match the start of your lines.

This pattern ^[0-9]{2}\/[0-9]{2}\/[0-9]{4} would match lines starting with dates like the one you have, for example 18/11/2021.

CodePudding user response：

I've found out what was the problem. I used filestream as a filebeat input. multiline doesn't work with filestream. I've changed the type to log then everything works fine.

filebeat.inputs:
 multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
 multiline.negate: true
 multiline.match: after
- type: log
  enabled: true

  paths:
    - .\My.log

output.logstash:
  hosts: ["localhost:5044"]