[Feature Request] Comprehensive Access Logging for Enhanced Monitoring and Security**

[AmIBeingObtuse]

Feature Request: Comprehensive Access Logging for Enhanced Monitoring and Security

This feature is already in nginx as basic access logging. Not sure how easy it would be to get that basic level over here in Zoraxy in the meantime before the majority of this feature request is implemented.

Description:

I propose adding a robust access logging feature to Zoraxy to provide detailed insights into incoming connections. This feature would generate structured logs capturing essential information for each connection, enabling integration with various monitoring and security tools.

Specific Logging Data Points:

• Timestamp: Precise time of connection initiation.
• Source IP: IP address of the incoming connection.
• Requested Host/URL: Target host/URL requested by the client.
• HTTP Method: GET, POST, PUT, etc.
• HTTP Status Code: Response code indicating success/failure.
• Data Transferred: Bytes sent and received during the connection.
• Connection Duration: Time taken to complete the request/response cycle.
• User Agent (Optional): Client's browser/application identifier.

Log Format:

The logs should be output in a standard, machine-readable format (e.g., JSON, CSV) to facilitate parsing and integration with external tools.

Integration Examples:

• Grafana: Visualize connection trends, peak usage times, and traffic patterns.
• Fail2ban: Detect and block malicious or abusive IPs based on log patterns.
• Network Monitoring Tools: Monitor bandwidth usage, identify top clients, and detect anomalies.
• Custom Scripts/Applications: Process logs for specific analysis or actions (e.g., rate limiting, alerts).

Benefits:

• Enhanced Security: Early detection of attacks and suspicious activity.
• Performance Optimization: Identify bottlenecks and optimize resource allocation.
• Usage Analysis: Gain insights into user behaviour and popular content.
• Compliance: Generate audit logs for regulatory purposes.

Additional Considerations (Optional):

• Log Rotation/Archiving: Implement mechanisms to manage log size and prevent storage issues.
• Filtering/Sampling: Allow selective logging based on specific criteria (e.g., IP ranges, URLs).
• Real-time Log Streaming: Provide an option to stream logs to external systems for immediate analysis.

Implementation:

This feature could be implemented as a configurable module within Zoraxy, allowing users to customize the logging level and output format according to their needs.

And anything in between. I don't expect this feature to be fully maxed out in the beginning but it would go a long way to helping with security.

[a0ndr]

I think this would be complicated and it will degrade proxying performance, but it would be really useful in a network where security matters. +1

[AmIBeingObtuse]

The thing is, it's a basic feature in every reverse proxy (access logs). That's all the feature needs to be. I can't see that degrading any performance. Just basic information for every connection made to it.

…

On Sat, 8 Jun 2024, 15:31 a0ndr, ***@***.***> wrote: I think this would be complicated and it will degrade proxying performance, but it would be really useful in a network where security matters. +1 — Reply to this email directly, view it on GitHub <#190 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2PH7LIVGTBWQFQOKV7YUBDZGMIVBAVCNFSM6AAAAABI7RT6L2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOMJUGE3DI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[tobychui]

A log file implementation will degrade the performance. I do agree this feature might help with security or debugging, that is why I am thinking how to make it less affecting to other service in Zoraxy.

In simple words, as everything in Zoraxy is highly concurrent, and you only got 1 log file, every log message need to be queued in order to be written into the file. Usually in cases like this I will just introduce a simple mutex design but a mutex will block the incoming request. Even if I run a mutex in go-routine, if your site have tons of traffic where the log writing speed is slower than the incoming traffic, this might results in huge memory consumption / crash.

[a0ndr]

What about something like a loadbalancer for instances where the throughput is large? The user would be able to make sub-instances on the same machine or somewhere else and every one would have it's own log

On 2024-06-09 04:48, Toby Chui wrote: A log file implementation will degrade the performance. I do agree this feature might help with security or debugging, that is why I am thinking how to make it less affecting to other service in Zoraxy. In simple words, as everything in Zoraxy is highly concurrent, and you only got 1 log file, every log message need to be queued in order to be written into the file. Usually in cases like this I will just introduce a simple mutex design but a mutex will block the incoming request. Even if I run a mutex in go-routine, if your site have tons of traffic where the log writing speed is slower than the incoming traffic, this might results in huge memory consumption / crash. -- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you commented.Message ID: ***@***.***>

Links: ------ [1] #190 (reply in thread) [2] https://github.com/notifications/unsubscribe-auth/BG77FQHH7GLJWN4LYIMFJCTZGO66LAVCNFSM6AAAAABI7RT6L2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOMJXGEYTI

[tobychui]

The load balancer is currently work in progress in v3.0.8 and it is integrated into the same instance (for now).

Though, this is a good suggestion that the control plane should be separated from the data plane which most of the professional grade reverse proxy and networking solutions are designed as such. The issue is, that would involve complex setup process (and hence UX design) that might make beginner hard to catch up with this software.

Back to the log issue, my idea now is to use a write queue and buffer zone where when enough log data are stored in memory, they will get flushed into a file or database. But i still have no idea what database to use. Any input will be welcomed 👍🏻

[a0ndr]

Good idea, I _might _try to implement something (even tho I don't know Go) if I'll have time. It would be useful for debugging purposes too.Message ID: ***@***.***>