Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix definitions of combined/common log formats #1821

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jjlin
Copy link
Contributor

@jjlin jjlin commented Jun 23, 2020

These formats have two initial fields to ignore (the RFC 1413 identity and
the HTTP authenticated user).

These formats have two initial fields to ignore (the RFC 1413 identity and
the HTTP authenticated user).
@jjlin
Copy link
Contributor Author

jjlin commented Jun 23, 2020

I'm actually pretty surprised a bug like this would remain in place for such a long time, so maybe I'm just missing something.

I'm using the combined log format as documented at https://httpd.apache.org/docs/2.4/logs.html#combined. I noticed that my transmit stats were way off; for example, downloading a multi-GB file wouldn't increase the Tx. Amount value at all. This change fixes that for me.

@allinurl
Copy link
Owner

There's no need for the extra %^. One ignore will look for the next character, so %h %^[ will skip everything until it finds [. Can you post your access log, I can take a look at what format you will need.

@jjlin
Copy link
Contributor Author

jjlin commented Jun 28, 2020

Ah, I see. The description of %^ in https://goaccess.io/man#custom-log doesn't seem to mention what constitutes a "field", and there are various other predefined log formats in goaccess that use adjacent %^ specifiers. I think being explicit about the fact that there are actually two fields would make things clearer.

In any case, my format is the combined log format as I mentioned. Here's an example I pulled out (with the IP address modified).

1.2.3.4 - - [28/Jun/2020:14:40:14 -0700] "GET /centos/7/os/x86_64/repodata/repomd.xml HTTP/1.1" 200 3736 "-" "urlgrabber/3.10 yum/3.4.3"

I use these definitions in httpd.conf:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
CustomLog "|bin/rotatelogs -f -n 30 -L logs/access_log logs/daily/access_log 86400" combined

(Note that LogFormat is the combined format documented at https://httpd.apache.org/docs/2.4/logs.html#combined.)

Does goaccess have any way to log entries that it thinks are malformed?

@allinurl allinurl force-pushed the master branch 2 times, most recently from 0550432 to b1332f5 Compare December 22, 2023 22:26
@allinurl allinurl force-pushed the master branch 5 times, most recently from 3d5333b to 6c68a6e Compare January 11, 2024 01:36
@allinurl allinurl force-pushed the master branch 7 times, most recently from 31e2ac1 to c6199f6 Compare May 14, 2024 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants