Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should be able to monitor log files like the original Fail2Ban, not just Windows Events #33

Open
JennaScvl opened this issue Jun 12, 2024 · 11 comments
Labels
enhancement New feature or request

Comments

@JennaScvl
Copy link

I spent 3 hours trying to figure out why it wasn't loading when it turned out to be that it just can't monitor logs.
For example I'd been trying to use this as one of my rules
{
"logName": "F:/xampp/apache/logs/error.log",
"eventId": 0,
"ipAddressPattern": "\[client (?\d \.\d \.\d \.\d ):\d \]",
"failurePattern": "AH00124"
}
and it just wouldn't load. The idea was to block an exploit attempt I keep seeing pop up in my Apache error.log
Also
{
"logName": "F:/xampp/apache/logs/modsec_audit.log",
"eventId": 0,
"ipAddressPattern": ""client_ip":"(?\d \.\d \.\d \.\d )"",
"failurePattern": ""status":"218""
}
This one is to monitor modsecurity's audit log to block certain common exploit attempts I've seen come in.
But it just can't monitor log files like the original can, apparently.

@Aldaviva Aldaviva added the enhancement New feature or request label Jun 14, 2024
@jmoeller-ua
Copy link

jmoeller-ua commented Jul 10, 2024

Just wanted to 1 this request. I have a similar challenge with IIS logs. I can have the IIS put its logs into the Event Viewer, but the IP address and the rest of the filtering criteria I'd need to account for are in different EventData Data fields, so I can't use Fail2Ban4Win for this particular challenge. It'd work if I could trigger off of additional fields in an EventLogSelector, or if I could work with the logfile directly, like @JennaScvl is requesting.

@Aldaviva
Copy link
Owner

Hi @jmoeller-ua,
Feel free to export and upload an .evtx file for the IIS event(s), and I will try to take a look at them and see if they can be selected.
Thanks.

@jmoeller-ua
Copy link

jmoeller-ua commented Jul 10, 2024

An example entry looks like this:

date 2024-07-10 time 23:18:54 s-sitename W3SVC1 s-computername <my_host> s-ip <my_ip> cs-method POST cs-uri-stem <my_url> cs-uri-query - s-port 443 cs-username - c-ip <remote_ip> cs-version - cs(User-Agent) Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 Edg/115.0.1901.203 cs(Cookie) - cs(Referer) - cs-host - sc-status 200 sc-substatus 0 sc-win32-status 0 sc-bytes 15322 cs-bytes 422 time-taken 351

The Event Viewer puts each element neatly and correctly into the EventData hashtable, but since I need to reference elements other than just c-ip to know if it's actually a block-worthy event (cs(User-Agent), cs-method, etc). If I could work with the entire message via regexp, just like a line in a logfile, I could be off and running.

Unhelpfully, the IIS logger puts in all events with the same event ID of 6200. But I guess it makes sense for a web log that it wouldn't be able to discern outcomes.

@Aldaviva
Copy link
Owner

That wasn't an .evtx file, but I figured out how to generate my own:

iis-access-log.evtx

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
        <Provider Name="Microsoft-Windows-IIS-Logging" Guid="{7e8ad27f-b271-4ea2-a783-a47bde29143b}" />
        <EventID>6200</EventID>
        <Version>0</Version>
        <Level>4</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2024-07-10T23:39:12.6648021Z" />
        <EventRecordID>1</EventRecordID>
        <Correlation />
        <Execution ProcessID="10412" ThreadID="10788" />
        <Channel>Microsoft-IIS-Logging/Logs</Channel>
        <Computer>Thor.aldaviva.com</Computer>
        <Security UserID="S-1-5-18" />
    </System>
    <EventData>
        <Data Name="EnabledFieldsFlags">2478079</Data>
        <Data Name="date">2024-07-10</Data>
        <Data Name="time">23:39:09</Data>
        <Data Name="c-ip">73.202.30.161</Data>
        <Data Name="cs-username">-</Data>
        <Data Name="s-sitename">W3SVC1</Data>
        <Data Name="s-computername">Thor</Data>
        <Data Name="s-ip">192.168.1.17</Data>
        <Data Name="cs-method">GET</Data>
        <Data Name="cs-uri-stem">/health-check.txt</Data>
        <Data Name="cs-uri-query">-</Data>
        <Data Name="sc-status">304</Data>
        <Data Name="sc-win32-status">0</Data>
        <Data Name="sc-bytes">0</Data>
        <Data Name="cs-bytes">766</Data>
        <Data Name="time-taken">103</Data>
        <Data Name="s-port">443</Data>
        <Data Name="csUser-Agent">Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36</Data>
        <Data Name="csCookie">-</Data>
        <Data Name="csReferer">-</Data>
        <Data Name="cs-version">-</Data>
        <Data Name="cs-host">-</Data>
        <Data Name="sc-substatus">0</Data>
        <Data Name="CustomFields" />
    </EventData>
</Event>

This can be enabled by going to the Logging feature page for a site in IIS Manager, configuring it to use W3C format and log ETW events, and enabling and watching Event Viewer > Application and Service Logs > Microsoft > Windows > IIS-Logging > Logs.


Given this structure, it seems like allowing an additional optional configuration property to be specified in the EventLogSelector to act as a predicate in the XPath query generated by selectorToQuery would allow you to filter based on the user agent string, request method, response status code, and other data element values.

@jmoeller-ua
Copy link

That wasn't an .evtx file,

Indeed, I just wanted to strip out the identifying information. I had set it up just as you had mentioned.

Given this structure, it seems like allowing an additional optional configuration property to be specified in the EventLogSelector to act as a predicate in the XPath query generated by selectorToQuery would allow you to filter based on the user agent string, request method, response status code, and other data element values.

Yes, that's sort of what I was imagining, additional optional configuration items that I could refer to put a specific event in scope of Fail2Ban4Win. Would love to see it if you think it'd be a reasonable feature to implement.

@Aldaviva
Copy link
Owner

Right, those ETW instructions were for me and anyone else who looks at this issue in the future, because I had to figure it out and was worried I'd forget next time I have to do it (when updating unit tests, adding new dependent features, etc).

I believe adding the additional XPath predicate to the configuration is a reasonable feature to add. I will try to find the time soon to try it out and send you a development snapshot build to see if it fits your use case too before releasing it.

@Aldaviva
Copy link
Owner

Aldaviva commented Jul 11, 2024

@jmoeller-ua: give this a try and let me know how well it fits your use case

Developer snapshot executable

Fail2Ban4Win.zip
(minimally tested so far)

Configuration

There is now a new optional property of EventLogSelector configuration objects with the key eventPredicate and the value of a string. The value is an XPath 1.0 predicate expression, which must be surrounded by square brackets and is evaluated by ETW against the top-level Event object of the ETW log record element.

Multiple predicates can be constructed with XPath and and or operators, or by concatenating multiple predicates like [predicate1][predicate2] instead of using the and operator. Note that functions like contains() and starts-with() are unfortunately not supported by Windows ETW.

Example

This selector should select only IIS access logs where the response status code is 403.

{
    "logName": "Microsoft-Windows-IIS-Logging/Logs",
    "source": "IIS-Logging",
    "eventId": 6200,
    "ipAddressEventDataName": "c-ip",
    "eventPredicate": "[EventData/Data[@Name='sc-status']=403]"
}

This will result in Fail2Ban4Win using an effective XPath expression of

*[System/EventID=6200][System/Provider/@Name=\"IIS-Logging\"][EventData/Data[@Name='sc-status']=403]

Here is another example that matches requests with both status code 304 and request method GET.

{
    "eventPredicate": "[EventData[Data[@Name='sc-status']='304'][Data[@Name='cs-method']='GET']]"
    /* other properties are the same */
}

Here is an example of an or expression to match two different status codes.

{
    "eventPredicate": "[EventData/Data[@Name='sc-status']='304' or EventData/Data[@Name='sc-status']='403']"
    /* other properties are the same */
}

@jmoeller-ua
Copy link

jmoeller-ua commented Jul 11, 2024 via email

@jmoeller-ua
Copy link

@jmoeller-ua: give this a try and let me know how well it fits your use case

Developer snapshot executable

Fail2Ban4Win.zip (minimally tested so far)

It works great, this is amazing. I can't believe how fast you did this!

@Aldaviva
Copy link
Owner

Thanks! Great to hear it's working for you.

This was a pretty straightforward change because it leveraged Windows' existing ETW XPath filtering, which this project was already using. By adding the one new optional, backwards-compatible configuration property, it augmented the existing Event ID and Source filtering that this project had already been set up to use. So it was a small change that did not reinvent the wheel.

@Aldaviva
Copy link
Owner

I'm splitting the eventPredicate feature that we've been discussing here into a different issue, #34, because it resulted in a different implementation than the topic of this issue, tailing text log files. Any further eventPredicate discussions can happen in #34, while this issue can remain for text log files.

@Aldaviva Aldaviva removed this from the 1.3.0 milestone Jul 15, 2024
@Aldaviva Aldaviva removed their assignment Jul 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants