# Using Access Logs

{% hint style="warning" %}
Web server logging rarely includes HTTP POST payloads, which makes this approach unsuitable if you expect inbound HTTP payloads in your traffic. 
{% endhint %}

## Common Log Format (Apache)

The Common Log Format is the standard Apache logging format for web server access logging. This format contains the following fields:

1. The IP address of the client (remote host) which made the request to the server.
2. The [RFC 1413](https://tools.ietf.org/html/rfc1413) [identity](https://en.wikipedia.org/wiki/Ident_Protocol) of the client. Usually "-".
3. The UserId of the person requesting the document. Usually "-" unless .htaccess has requested authentication.
4. The date, time, and time zone that the request was received, by default in [strftime](https://en.wikipedia.org/wiki/Strftime) format %d/%b/%Y:%H:%M:%S %z.
5. The request line from the client. The method GET, /apache\_pb.gif the resource requested, and HTTP/1.0 the [HTTP protocol](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol).
6. The [HTTP status code](https://en.wikipedia.org/wiki/HTTP_status_code) returned to the client. 2xx is a successful response, 3xx a redirection, 4xx a client error, and 5xx a server error.
7. The size of the object returned to the client, measured in [bytes](https://en.wikipedia.org/wiki/Byte).

```
127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
```

To convert this file into a format that can be ingested by a running Spherical Defense instance, complete the following steps:

1. SSH into your Spherical Defense instance.
2. Create a new file in the \~/ directory called `apache_log_converter.py`
3. Copy the following script into the file:

```python
import argparse
import json
import re
parser = argparse.ArgumentParser(description="Convert Apache logs to JSON")
parser.add_argument("-s", "--source", type=str, required=True)
parser.add_argument("-t", "--target", type=str, required=True)
regex = re.compile(
    '([^ ]*) ([^ ]*) ([^ ]*) \[([^]]*)\] "([^"]*)" ([^ ]*) ([^ ]*)'
    )
def parse_line(line):
    m = regex.match(line)
    if not m:
        return "{}"
    host, ignore, user, date, request, status, size = m.groups()
    method_items = list(map(lambda x: x.replace("\"", ""), request.split(" ")))
    if len(method_items) == 2:
        method_items.append("-")
    _dict = {"host":host, "user":user, "timestamp":date, "method":method_items[0], "path":method_items[1], "protocol": method_items[2], "code":status, "size":size}
    return json.dumps(_dict)
def translate_file(source, target):
    with open(source, 'r') as _in, open(target, 'w+') as _out:
        for line in _in:
            _out.write(parse_line(line))
            _out.write("\n")
if __name__ == '__main__':
    args = parser.parse_args()
    translate_file(args.source, args.target)
    print("Converted {} to {}".format(args.source, args.target))

```

   4\. Ensure that your Apache logs are in the same directory as the above script.\
&#x20;  5\. Run the following command: `python3 apache_log_converter.py -s <SOURCE FILENAME> -t <TARGET FILENAME>`

&#x20;  6\. After you have generated a file that can be ingested by the Spherical Defense data ingestor, go back to    the AMI quickstart below to continue the on-boarding process.

{% content-ref url="/pages/-LuZVMQkwL0lAVGQngYg" %}
[Amazon Machine Image](/guide/quickstart/amazon-machine-image.md)
{% endcontent-ref %}

## Combined Log Format (NGINX)

The combined log format is the same as the above, but with two extra fields included:&#x20;

* A referrer URL
* A user-agent

```
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)"
```

## IIS Log Format

```
2010-05-02 22:48:39 206.175.82.5 - 208.201.133.173 GET /global/images/navlineboards.gif - 200 540 324 157 HTTP/1.0 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+95) USERID=CustomerA;+IMPID=01234 http://www.loganalyzer.net
```

## **W3C Extended Log Format**

```
2010-05-02 15:42:15 - 40.89.255.10  34.14.255.10 80 GET /default.htm 200 - HTTP/1.0 Mozilla/4.0  (compatible: MSIE+5.5+Windows+2000+Server)
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://guide.sphericaldefence.com/guide/quickstart/amazon-machine-image/using-existing-logs.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.