# Summary This page describes what Spherical is, and what it can detect. ## What is Spherical Defense? Spherical Defense is an API security solution that uses **deep unsupervised learning** to protect your APIs. The product is deployed onto AWS, and integrates with your API gateway to mirror your inbound API traffic. The running Spherical Defense instance ingests this traffic, and builds an internal model of normal API behavior. After sufficient training, it will mount a model for evaluation. Every subsequent API request will be classified as either **normal** or **anomalous**, depending on whether or not it is a threat. Additional information is included in the event, such as the part of the API request which is most suspicious, and a **score** which enables you to see the worst potential threats. Any likely threats are filtered as **events** which can be consumed by SIEM solutions such as **Splunk**. {% hint style="info" %} Spherical is an analytics solution to help you detect when your APIs are under attack. {% endhint %} Spherical has a three-stage life cycle. {% tabs %} {% tab title="Listen" %} Once you have deployed your Spherical instance, it will immediately start listening for API traffic. It will stay in this mode for only as long as there is insufficient data to train the first security model. After receiving roughly 160,000 requests, it will move to the next stage. {% endtab %} {% tab title="Train" %} After sufficient data has been received, the system moves into training mode. This mode will result in a trained security model after roughly 6 hours, which will then be mounted for evaluation. As new data is received, the Spherical instance will train more models to account for natural changes in your API traffic over time. {% endtab %} {% tab title="Secure" %} Once the first security model has been trained, it is mounted for evaluation. This means that every subsequent API request that is received by the system is given a **classification** (either benign or anomalous), and a **score**. If you have integrated with an outbound service, these events will be filtered back. {% endtab %} {% endtabs %} ## What can Spherical detect? Spherical Defense can protect your APIs from malicious injection, mis-configuration, and generic misuse. Some examples of attacks that we can detect are as follows: #### **E**xcessive Data Exposure Exposing more object-level data than necessary over API endpoints #### **Malicious Injection** Passing malicious instructions to databases and other services via the API. These include things like SQL injection. #### **Improper Assets Management** Exposing debug, administration and obsolete API endpoints. **Sensitive Information Transmission** Users passing personally identifiable information into the wrong field, resulting in a GDPR breach. **Mass Assignment** Accepting an unauthorized object update request. **Authorized Stateful Attacks** Authorized users attempting to subvert application state. These include things like *Replay Attacks.* **ML Attack Tools** Adversarial API fuzzing can be trained to subvert existing security systems.. ## How does it work? Spherical Defense applies semi-supervised learning to the task of application-level threat detection. We train an advanced model to recreate your data and thus, learn the underlaying structure, syntax and semantics. With this insight it is then able to easily distinguish between benign and malicious requests. # Frequently Asked Questions Here you can read the answers to some commonly asked questions. ## Why aren't Web Application Firewalls good enough? Web Application Firewalls (WAFs) do not work for API traffic. Requests that are sent over APIs often contain complex application-specific data which are hard to codify with rules - especially if you are using a legacy solution or ruleset. Furthermore, as the pace of development and continuous releases have become the industry standard, the cost of maintaining WAFs has sky-rocketed. If you are aiming to protect complex web applications or APIsour solution is the best fit for you. ## What is the detection accuracy? Spherical has a false positive rate of as low as **0%** in structured APIs, to as high as **0.5%** in highly irregular API traffic (including requests with natural language components - like a chat app). In the **CSIC2010** WAF Benchmark, Spherical performs extremely favourably against off the shelf Web Application Firewall Solutions, including ModSecurity (core ruleset) - as seen below. | Solution | False Negatives | False Positives | | ----------------- | --------------- | --------------- | | ModSecurity CRS | 23.93% | 65.68% | | Spherical Defence | 3.15% | 0.5% | ## How much does it cost? For a limited time, Spherical costs **$1 per hour** for a smaller instance, and **$2 per hour** for a larger instance. ## How can I get up and running? If you are using Amazon Web Services, you can get up and running within **1 minute.** If you also already have a data stream ready, you can get a trained security model within **4 hours**. To do this, follow these instructions: {% content-ref url="/pages/-LuDOHvyAYX9LFfHCDNE" %} [Quickstart](/guide/quickstart) {% endcontent-ref %} ## Do you support Azure or Google Cloud? Unfortunately not at the moment, although these deployment options are under active development. ## What are the limitations of the Spherical security system? Spherical works best on APIs that have a **regular internal structure**. The less structure there is in a request, the less effective Spherical will be at detecting attacks in it. For example, the following request will be hard to learn from, because there is minimal internal structure: ```javascript { "encoded" : "TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5IGJ5IGhpcy=" } ``` Additionally, Spherical Defense is also less effective in the case of API traffic with highly imbalanced endpoints. If one endpoint is hit a million times less frequently than another, it will be less effectively protected. An example of an API request that Spherical Defense will learn well from is as follows: ```javascript { "method": "POST", "url": "/api/payment", "body": { "expiry": "11/2020", "number": "3112893758824764", "cvv": "718" }, "headers": { "Content-Length": "311", "Accept-Language": "en-US,en;q=0.8", "Content-Type": "application/json,*/*;q=0.1", "Accept": "application/json", "Accept-Encoding": "deflate", "X-Forwarded-Port": "9490", "Cookies": [ { "vk": "O226kQr8-WC5P-gVll-bkRc-1aGv3bLW" } ] } } ``` # Quickstart This guide shows you how to get up and running with Spherical on Amazon Web Services. ## Amazon Machine Image To get started with Spherical Defense, you must first install the AMI onto your AWS infrastructure. This can be done in **3 minutes**. {% content-ref url="/pages/-LuZVMQkwL0lAVGQngYg" %} [Amazon Machine Image](/guide/quickstart/amazon-machine-image) {% endcontent-ref %} ## CloudFormation In order to integrate with your existing workflow, it is necessary to set up other AWS services to support the existing . The below instructions will help you set up a Spherical stack, including an externally facing **AWS API Gateway** for data ingestion. {% content-ref url="/pages/-LuZVQ884KCXfbOsvRBb" %} [CloudFormation](/guide/quickstart/cloudformation) {% endcontent-ref %} ## Docker Compose If you want total control of your hardware, you can also deploy using Docker after obtaining a license agreement. Please contact the team at **** to get access to our Docker repositories. #### # Amazon Machine Image This page shows how you can get up and running with an Amazon Machine Image You can use Amazon Machine Images to get the Spherical Defense service up and running, and then craft your own integrations around it. This installation method gives you the greatest flexibility, and is useful when you wish to train a security model without having to ingest traffic over the standard HTTP interface. This deployment approach is **free for 7 days**, before reverting to a **$1 per hour** pricing model. ## Installation It is straightforward to install a Spherical AMI onto your infrastructure. 1. Go to the Spherical Defense listing [here](https://aws.amazon.com/marketplace/pp/B08232Y26F). 2. Click 'Continue to Subscribe'. 3. At this stage you can view our End User License Agreement, before clicking 'Continue to Configuration'. 4. Select your fulfilment options, and click 'Continue to Launch'. 5. On the launch screen, you will be presented with several options: 1. From 'Choose Action' - select '**Launch from Website'.** 2. From 'EC2 Instance Type' - select '**G3.4xlarge**'. 3. From 'VPC Settings' - select a VPC for the Spherical app, and then either select an existing subnet within this VPC that you already have in place for other service, or let AWS create you one automatically for the deployment. 4. From 'Security Group Settings' - either create your own, or use an existing one. \ **Ensure that ports 9000 and 8080 are open.** Port 8080 is the management port, and needs to be exposed to your local machine. Port 9000 is the ingestion port, and must be exposed to your API gateway VPC. 5. From 'Key Pair Settings' - select a key pair that you've previously used to access your AWS instances. 6. Click 'Launch'. 7. In the region that you launched in, go to the AWS EC2 console, and verify that the instance launched correctly. ## Data Ingestion A running Spherical instance needs data in order to build a machine learning model to protect you. There are two ways to pass in this data. For longer term protection, a **live** stream of traffic is necessary - ideally from an API gateway. However, to get up and running quickly you can use an existing dataset. The format required when ingesting a static dataset is a sequence of JSON objects, separated by a new line. See the following example: ``` {"method": "POST", "url": "/api/payment", "body": {"expiry": "11/2013", "number": "3112893758824764", "cvv": "718"}, "headers": {"Content-Length": "311", "Accept-Language": "en-US,en;q=0.8", "Content-Type": "application/json,*/*;q=0.1", "Accept": "application/json", "Accept-Encoding": "deflate", "X-Forwarded-Port": "9490", "Cookies": [{"vk": "O226kQr8-WC5P-gVll-bkRc-1aGv3bLW"}]}} {"method": "DELETE", "url": "/orders/vHgPXnL7C6U4kWPE95ZEQlDKW0ESNTeCk", "query": {"order_id": "vHgPXnL7C6U4kWPE95ZEQlDKW0ESNTeCk"}, "headers": {"Content-Length": "294", "Accept-Language": "en-US,en;q=0.8", "Content-Type": "application/json,*/*;q=0.1", "Accept": "application/json", "Accept-Encoding": "deflate", "X-Forwarded-Port": "7663", "Cookies": [{"vk": "VCkYuJST-ruJX-ghM0-bTFf-GpmnsVBo"}]}} {"method": "POST", "url": "/deposits/payment-method", "body": {"amount": "9948453.498402445", "currency": "Gbi", "payment_method_id": "RjDyGDw3W3DxvEJzhb9iFCLcchC0ugoEp9"}, "headers": {"Content-Length": "205", "Accept-Language": "en-US,en;q=0.8", "Content-Type": "application/json,*/*;q=0.1", "Accept": "application/json", "Accept-Encoding": "deflate", "X-Forwarded-Port": "6299", "Cookies": [{"vk": "a4Ho12tU-Kn2l-iKEC-jVGt-kEfpNXyD"}]}} {"method": "POST", "url": "/deposits/account", "body": {"amount": "5595519.921778078", "currency": "tRy", "account_id": "HUVTS4iLTvajacsLEmOEwzFwfbOb0jDjbZdH"}, "headers": {"Content-Length": "291", "Accept-Language": "en-US,en;q=0.8", "Content-Type": "application/json,*/*;q=0.1", "Accept": "application/json", "Accept-Encoding": "deflate", "X-Forwarded-Port": "5453", "Cookies": [{"vk": "B1fWiRp2-PmKS-hb7Q-aPRN-8BQyN8y4"}]}} ... {"method": "GET", "url": "/accounts/7124518/holds", "query": {"id": "7124518", "account_id": "7UBMT1IhCLyFFnQ2wdkjhGPipTq", "created_at": "a7PdLpBZDmvlhrrT6EGgl", "updated_at": "rCDH3e5ng05b4XvQW2Hx490zGlp", "amount": "4730830", "type": "qgSQgZBZoN", "ref": "WzMqzFrJem37GEt7wxlFdLn6twIet6apAUbgby"}, "headers": {"Content-Length": "227", "Accept-Language": "en-US,en;q=0.8", "Content-Type": "application/json,*/*;q=0.1", "Accept": "application/json", "Accept-Encoding": "deflate", "X-Forwarded-Port": "5367", "Cookies": [{"vk": "A7bpIqdo-PMUP-lUZa-Zkuk-xweqniPM"}]}} ``` 1. Use a program such as SCP to transfer a static dataset file onto the running instance. \ If you already have existing logs in a different format, first [you have to convert them into the correct format.](/guide/quickstart/amazon-machine-image/using-existing-logs) 2. Once the dataset is on the filesystem, execute the **ingestion script** in the 'ubuntu' user root directory as shown below, after ensuring there are execution permissions in place. {% hint style="info" %} If you don't have a dataset to hand, use the demo file provided at the bottom of this section. {% endhint %} ``` cd ~/ chmod +x charon-upload.sh ./charon-upload continuous CharacterDatabase ``` {% hint style="warning" %} Ensure you use the **absolute path** to the log file or the ingestion process won't work. {% endhint %} This script will take the provided file, ingest it into the internal database, and start training a security model. {% file src="/files/-LugfBb8SR350zhMba8a" %} Demo dataset {% endfile %} ## Training Progress `GET` `https://:8080/collections/:collection/progress` Once you have ingested a dataset, it's important to ensure that the training process has begun. This can be done by accessing this API endpoint.\ \ This method enables you to see the progress of a security model that is being trained on a specific *collection*. #### Path Parameters | Name | Type | Description | | ---------- | ------ | --------------------------------------------------- | | Collection | string | The name of the collection. Default to 'continuous' | {% tabs %} {% tab title="200 This response gives information about the data ingestion process, i.e whether enough data has been seen by the system, as well as: 1. The predicted accuracy of a running security model. 2. The parameters of the trained security model. 3. The number of different security models that have been trained and evaluated. " %} ```javascript { "populationProgress": { "generation": 0, "numberOfIterations": 3000, "progress": 0.9951258086449938, "parents": 12, "popSize": 20, "active": true, "bestGenome": { "score": 0.9951258086449938, "genome": { "weight_decay": 0.00026861, "hidden_size": 105, "embedding_size": 201, "learning_rate": 0.02487348 }, "generation": 0 }, "elites": [] }, "trainingDataState": { "readyToTrain": true, "percentageOfDataConsumedForTraining": 1, "minimumTrainingBatches": 500, "keyDatabasePresent": true, "tokenDatabasePresent": true, "statsPresent": true, "dataSetsCounts": { "trainingDataCount": 8748, "attackDataCount": 0, "classifierFitDataCount": 0, "validationTreeDataCount": 0 } } } ``` {% endtab %} {% endtabs %} ## Evaluate Security Model `POST` `https://:9000/evaluate` Once a security model has been trained, you can start evaluating it by sending JSON objects in the body of the request. #### Request Body | Name | Type | Description | | ---- | ------ | --------------------------------------------------- | | | object | A JSON object to be evaluated by the security model | {% tabs %} {% tab title="200 This response contains event information regarding the classified request object. " %} ```javascript { "id": "5dde6f69a7b11b000696952f", "collection": "continuous", "timestamp": 1574858601646, "score": 0.89, "mostSuspiciousField": ".body.names.first:felipa", "request": "{\"method\":\"PUT\",\"url\":\"/api/Order\",\"id\":\"HVRwQMpb-QBWl-fRj7-aC0u-9wP4qS4H\",\"body\":{\"names\":{\"first\":\"felipa\",\"second\":\"wiggins\"},\"age\":\"67\",\"address\":\"venusst\",\"state\":\"CA\",\"city\":\"San Francisco\"},\"origin\":\"108.119.5.22\",\"status\":\"200\",\"headers\":{\"Content-Length\":\"283\",\"Accept-Language\":\"en-US,en;q=0.8\",\"Content-Type\":\"aplication/json,*/*;q=0.1\",\"Accept\":\"application/json\",\"Accept-Encoding\":\"deflate\",\"X-Forwarded-Port\":\"9697\",\"Host\":\"wwi.microsoft.org\",\"Cookies\":[{\"ASP.NET_SessionID\":\"HVRwQMpb-QBWl-fRj7-aC0u-9wP4qS4H\"},{\"vk\":\"HVRwQMpb-QBWl-fRj7-aC0u-9wP4qS4H\"}]}}", "isAnomaly": false } ``` {% endtab %} {% endtabs %} # Using Access Logs This page describes how to convert your existing access logs into a format that Spherical Defense can ingest and learn from. {% hint style="warning" %} Web server logging rarely includes HTTP POST payloads, which makes this approach unsuitable if you expect inbound HTTP payloads in your traffic. {% endhint %} ## Common Log Format (Apache) The Common Log Format is the standard Apache logging format for web server access logging. This format contains the following fields: 1. The IP address of the client (remote host) which made the request to the server. 2. The [RFC 1413](https://tools.ietf.org/html/rfc1413) [identity](https://en.wikipedia.org/wiki/Ident_Protocol) of the client. Usually "-". 3. The UserId of the person requesting the document. Usually "-" unless .htaccess has requested authentication. 4. The date, time, and time zone that the request was received, by default in [strftime](https://en.wikipedia.org/wiki/Strftime) format %d/%b/%Y:%H:%M:%S %z. 5. The request line from the client. The method GET, /apache\_pb.gif the resource requested, and HTTP/1.0 the [HTTP protocol](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol). 6. The [HTTP status code](https://en.wikipedia.org/wiki/HTTP_status_code) returned to the client. 2xx is a successful response, 3xx a redirection, 4xx a client error, and 5xx a server error. 7. The size of the object returned to the client, measured in [bytes](https://en.wikipedia.org/wiki/Byte). ``` 127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 ``` To convert this file into a format that can be ingested by a running Spherical Defense instance, complete the following steps: 1. SSH into your Spherical Defense instance. 2. Create a new file in the \~/ directory called `apache_log_converter.py` 3. Copy the following script into the file: ```python import argparse import json import re parser = argparse.ArgumentParser(description="Convert Apache logs to JSON") parser.add_argument("-s", "--source", type=str, required=True) parser.add_argument("-t", "--target", type=str, required=True) regex = re.compile( '([^ ]*) ([^ ]*) ([^ ]*) \[([^]]*)\] "([^"]*)" ([^ ]*) ([^ ]*)' ) def parse_line(line): m = regex.match(line) if not m: return "{}" host, ignore, user, date, request, status, size = m.groups() method_items = list(map(lambda x: x.replace("\"", ""), request.split(" "))) if len(method_items) == 2: method_items.append("-") _dict = {"host":host, "user":user, "timestamp":date, "method":method_items[0], "path":method_items[1], "protocol": method_items[2], "code":status, "size":size} return json.dumps(_dict) def translate_file(source, target): with open(source, 'r') as _in, open(target, 'w+') as _out: for line in _in: _out.write(parse_line(line)) _out.write("\n") if __name__ == '__main__': args = parser.parse_args() translate_file(args.source, args.target) print("Converted {} to {}".format(args.source, args.target)) ``` 4\. Ensure that your Apache logs are in the same directory as the above script.\ 5\. Run the following command: `python3 apache_log_converter.py -s -t ` 6\. After you have generated a file that can be ingested by the Spherical Defense data ingestor, go back to the AMI quickstart below to continue the on-boarding process. {% content-ref url="/pages/-LuZVMQkwL0lAVGQngYg" %} [Amazon Machine Image](/guide/quickstart/amazon-machine-image) {% endcontent-ref %} ## Combined Log Format (NGINX) The combined log format is the same as the above, but with two extra fields included: * A referrer URL * A user-agent ``` 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en] (Win98; I ;Nav)" ``` ## IIS Log Format ``` 2010-05-02 22:48:39 206.175.82.5 - 208.201.133.173 GET /global/images/navlineboards.gif - 200 540 324 157 HTTP/1.0 Mozilla/4.0+(compatible;+MSIE+4.01;+Windows+95) USERID=CustomerA;+IMPID=01234 http://www.loganalyzer.net ``` ## **W3C Extended Log Format** ``` 2010-05-02 15:42:15 - 40.89.255.10 34.14.255.10 80 GET /default.htm 200 - HTTP/1.0 Mozilla/4.0 (compatible: MSIE+5.5+Windows+2000+Server) ``` # CloudFormation This page shows you how you can get up and running with AWS CloudFormation The simplest installation route is through **CloudFormation**, which enables you to get up and running with (almost) a single click. The following link will set up a Spherical stack, including an externally facing **AWS API Gateway** for data ingestion. [Launch a CloudFormation Spherical Stack](https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/create/review?templateURL=https://spherical-integrations-virgin.s3.amazonaws.com/SphericalDefenceIntegrationHarness.cloudformation.yaml\&stackName=SphericalDefence) You can choose to run the system in front of an underlying service as a **reverse proxy**, or alternatively in offline mode without any redirection of requests to another service. Offline mode is usually a much better way of getting started. Additional benefits of using **CloudFormation** as your deployment option enables easy integration with a number of services. ![The stack that is created on your AWS infrastructure looks like this. ](/files/-LuIRrOisdn1kWm7X7Nv) Spherical integrates with services in two ways, **outbound** and **inbound.** Inbound integrations help get data into a Spherical instance. Outbound integrations help get events out of a Spherical instance for your security team to view. See how to integrate Spherical into your workflow with the following link: {% content-ref url="/pages/-LuDOkA1CAQq-zzNHAGU" %} [Integrations](/guide/integrations) {% endcontent-ref %} #### API Gateway The API Gateway is the component that allows you to process API traffic coming in from the outside world. Spherical can be configured with most API Gateways that enable *proxy integrations.* This means that the API Gateway will pass forward the entirety of every request that it receives. #### Proxy Lambda This component is used to enable Spherical to get a **copy** of all traffic moving from your API Gateway, and your backend API service. It is very **lightweight**, and merely loads the HTTP/S request into memory, before sending a copy to the **Broker Lambda** and the original onto the underlying service. #### Broker Lambda This component handles the **outbound integrations** to alerting services like Splunk. It either integrates with the Proxy Lambda directly (when using the AWS API Gateway), or alternatively with the Gateway itself (when using Kong or Apigee). This Lambda forwards API requests to the Spherical instance, and does something with the response. # User Interface Spherical Defense comes with a web user interface included. Here are the usage instructions. The user interface is hosted on port 80 of your Spherical Defense deployment, which means you can access it from your browser by simply entering the address of the server in your browser. ![Spherical Defense user interface.](/files/-LyPDo25B9-4KIBPQxwO) The application is easy to use, and provides information about the running system and also the API requests that is has analysed. ![System status, and model training information panel.](/files/-LyPEuvKhNd9QhjXYXi5) | Element | Detail | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------ | | Collecting | This task is shown with tick if sufficient traffic has been ingested to begin training. | | Processing | This task is shown with a tick if the pre-training processing has completed. | | Training | This task is shown with a tick if the system is current training a security model. | | Ready | This task is shown with a tick if at least one security model has been trained, and new requests are being evaluated. | | Training Requests | The number of requests that have been received and are being incorporated for training subsequent models. | | Models | The number of security models that have been trained. The one with the highest detection accuracy is preserved for evaluation. | | Accuracy | The estimated detection accuracy of the best available model. | | Events | The number of security events that have been received. | ![Historical API requests over the last 30 days, including the number of detected anomalies (red).](/files/-LyPFNGtr5ntWgsF3eGA) ![You can see security alerts by selecting a date range in the last 30 days. ](/files/-LyPFwqj_Ba4QxCubr3f) | | | - | ![Security alerts will be shown in this table, which can searched and sorted. ](/files/-LyPHanGW4T3m0hc8dsG) | Column | Detail | | ------------- | --------------------------------------------------------------------------------------------------------------------- | | Score |

How anomalous this event is.

This ranges between 100 (very anomalous) and 0 (only slightly anomalous).

| | Timestamp | When the event occurred. | | Anomaly Focus | The part of the request which triggered the event. | | **›** | Show the entirety of the request. | ![The API request that triggered the event can be explored by clicking on the right-facing chevron.](/files/-LyPJvIB8KzOj2D-USMd) # Integrations This group shows how you can integrate different business solutions into your Spherical deployment. ## Inbound Getting data **into** a running Spherical instance is a straightforward process. It can be done by connecting an API gateway by following the relevant instructions on the following pages: {% content-ref url="/pages/-LuDP1H-2TzOtKcRubWQ" %} [Apigee](/guide/integrations/apigee) {% endcontent-ref %} {% content-ref url="/pages/-LuDP2pLFfBf5xMghdVW" %} [Kong](/guide/integrations/kong) {% endcontent-ref %} {% content-ref url="/pages/-LuDQEG\_uTbUdhSUikl9" %} [AWS API Gateway](/guide/integrations/aws-api-gateway) {% endcontent-ref %} Integrations with these API Gateways are implemented using AWS Lambda functions, in which an inbound HTTP method request from a client is mapped to a [Lambda function invocation](https://docs.aws.amazon.com/lambda/latest/dg/API_Invoke.html). By using a *Lambda Proxy Integration,* the entire request is passed wholesale into the Spherical instance. Inbound integrations should be performed ***after*** creating a Spherical instance. ## Outbound Getting events **out of** a running Spherical instance is also a straightforward process. This is done asynchronously using AWS Lambda functions in a deployed **CloudFormation** stack (as discussed in the Summary). Currently there are only two outbound integrations, but it is very simple to implement your own Lambda functions to integrate with your own (or third party) services: {% content-ref url="/pages/-LuDP-kkXBkrdNGEZv3V" %} [Slack](/guide/integrations/slack) {% endcontent-ref %} {% content-ref url="/pages/-LuDOyHtxhmZsjJZevGV" %} [Splunk](/guide/integrations/splunk) {% endcontent-ref %} Outbound integration steps should be performed ***before*** creating a Spherical instance. # Apigee This page documents how to get up and running with the APIGEE API Gateway ## Prerequisites Before using the extension from an API proxy using the [ExtensionCallout policy](https://docs.apigee.com/api-platform/reference/policies/extension-callout-policy?authuser=1), you must Get your Amazon [access key ID and secret access key](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys). You'll need these, along with the name of the region where your functions are deployed, to [configure](https://docs.apigee.com/api-platform/reference/extensions/aws-lambda/aws-lambda-extension-100?authuser=1#configuration_reference) the AWS Lambda extension which plugs into Spherical. {% hint style="info" %} This integration requires Apigee Edge. These instructions are for after you have already created a Spherical EC2 instance (and are largely taken from the instructions [here](https://docs.apigee.com/api-platform/reference/extensions/aws-lambda/aws-lambda-extension-100?authuser=1)). {% endhint %} ## Add and configure Lambda extension 1. [Sign in to Edge](https://apigee.com/edge), click the **Admin** menu, then click **Extensions**. ![](/files/-LuXffL7xvAF34dGiZT6) The **Extensions** page that appears lists all of the extensions that have been configured from extension packages available on the system. If you wanted to view an existing configured extension, you could filter the list by typing filtering terms at the top of the page. In the next steps, you'll add a Lambda Extension. 2\. In the **Extension** page, click **Add Extension**. ![](https://docs.apigee.com/api-platform/images/extensions-add-extension-button.png) The Select an extension page that appears divides the extensions between those that are developed by Google and those developed by other companies. 3\. Under **Select an extension**, locate the **Lambda Extension** corresponding to the external resource you're accessing. 4\. Click the box for the extension package to begin configuring an extension from the package. 5\. In the **Extension Properties** dialog box, enter a name and description for the extension, then click **Create**. 6\. In the **Extension** > *My-Extension-Name* page for the extension you're creating, locate an Apigee Edge environment for which you want to configure the extension. An extension must be deployed to the same environment as the API proxy that will be using it. 7\. Click the environment's row in the list. 8\. In the **Configuration** dialog, select or enter values to configure this extension. Each extension has its own configuration properties. Extension references are listed in [Extensions reference overview](https://docs.apigee.com/api-platform/reference/extensions/reference-overview-extensions). ```markup invoke flow-variable ``` {% hint style="warning" %} Note: Change the functionName above to the qualified name of the Broker. {% endhint %} 10\. Click **Save**. ## Deploy the Proxy # Kong This page documents an inbound integration with your Kong API Gateway. We currently support both Kong OS and Kong Enterprise deployments by the use of a custom plugin. This plugin may be attached to either a Service or a Route within your system, we do not currently support protection of Consumers. ## Install the Spherical Defense plugin To install the custom Spherical Defense plugin, we need to be inside your Kong docker container. Our plugin is hosted on the official `Luarocks` package manager and can easily be installed. Afterwards Kong must be restarted with the plugin enabled. ``` ## get into the kong docker container docker exec -u root -it sh ## install spherical defense (note the luarocks package name!) luarocks install sphericaldefence export KONG_PLUGINS=sphericaldefence,$KONG_PLUGINS ## restart kong kong prepare kong reload ``` ## Adding to your Kong deployment Our plugin can be added to either a `Route` or `Service`. This can be done either via the Kong Enterprise GUI or if using Kong through curl commands, all that is required is the knowledge of the exposed IP address for your Spherical Defence Instance `SPHERICAL_DEFENCE_IP`. Below we list an example of either curl command to add the plugin to your services: ### Adding to a Service Adding a plugin to service is easy. All you need is the exposed IP address `KONG` for the container and the name of the `SERVICE`you send the following command. Note, that Kong defaults the port to 8001 to be open. ``` $curl --request POST \ --url http://{KONG IP}:8001/services/{SERVICE}/plugins \ --form name=sphericaldefence \ --form config.url=http://{SPHERICAL_DEFENCE_IP} \ --form config.blocking=false \ --form config.timeout=10000 \ --form config.keepalive=60000 ``` ### Adding to a Route To add a plugin to a route, we require both the exposed IP address `KONG` for the container and name of the `ROUTE` ``` $curl --request POST \ --url http://{KONG IP}:8001/routes/{ROUTE}/plugins \ --form name=sphericaldefence \ --form config.url=http://{SPHERICAL_DEFENCE_IP} \ --form config.blocking=false \ --form config.timeout=10000 \ --form config.keepalive=60000 ``` ## Additional Documentation Below we've listed the parameters associated with the plugin: | form parameter | default | description | | ------------------ | ------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | `name` | `spherical_defence` | The name of the plugin in use | | `config.url` | | The address of a Spherical Defence instance to which the plugin will validate before proxying the original request *(required)* | | config.blocking | False | Whether malicious requests should be blocked from moving through to the service/route | | `config.timeout` | 10000 | Timeout (milliseconds) for the request to the URL specified above. | | `config.keepalive` | 60000 | Keepalive time (milliseconds) for the request to the URL specified above. | The plugin will execute a JSON `POST` request to the Spherical instance with the following body: | Form Parameter | Description | | -------------- | ----------------------------------------- | | `body_data` | The body of the original request | | `url_args` | The url arguments of the original request | | `headers` | The headers of the original request | | `uri` | The uri of the original request | | `method` | The method of the original request | # AWS API Gateway This page documents how you can use AWS API Gateway with a running Spherical instance. {% hint style="info" %} These instructions are for after you have already created a Spherical EC2 instance (and are largely taken from the instructions [here](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-getting-started-with-rest-apis.html#apigateway-getting-started-rest-step-2)). {% endhint %} ### Create an IAM Administrator User As a best practice, you should also create an AWS Identity and Access Management (IAM) user with administrator permissions. Use that for all work that does not require root credentials. Create a password for console access, and access keys to use command line tools. For instructions, see [Creating Your First IAM Admin User and Group](https://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html) in the *IAM User Guide*. ### Create a REST API in the API Gateway Console In this step, you create a simple REST API in the API Gateway console and attach the Spherical Lambda function to it as a backend. 1. From the **Services** menu, choose **API Gateway** to go to the API Gateway console. 2. If this is your first time using API Gateway, you see a page that introduces you to the features of the service. Choose **Get Started**. When the **Create Example API** popup appears, choose **OK**. If this isn't your first time using API Gateway, choose **Create API**. 3. Under **Choose the protocol**, choose **REST**. 4. Under **Create new API**, choose **New API**. 5. Under **Settings**: * For **API name**, enter `spherical-api`. * If desired, enter a description in the **Description** field; otherwise, leave it empty. * Leave **Endpoint Type** set to **Regional**. 6. Choose **Create API**. 7. Under **Resources**, you'll see nothing but **/**. This is the root-level resource, which corresponds to the base path URL for your API (`https://b123abcde4.execute-api.us-west-2.amazonaws.com/`*`{stage-name}`*). From the **Actions** dropdown menu, choose **Create Method**. 8. Under the resource name (**/**), you'll see a dropdown menu. Choose **ANY** and then choose the checkmark icon to save your choice. 9. In the **/ – ANY – Setup** pane, for **Integration type**, choose **Lambda Function**. 10. Choose **Use Lambda proxy integration**. 11. For **Lambda Region**, choose the Region where you created your Lambda function. 12. In the **Lambda Function** field, type any character and then choose **SphericalDefenceProxy-\*** (where \* stands for the application ) from the dropdown menu. (If the dropdown menu doesn't appear, delete the character you just typed to make the dropdown menu appear.) Leave **Use** **Default Timeout** checked. Choose **Save** to save your choice. 13. When the **Add Permission to Lambda Function** popup appears (saying **"You are about to give API Gateway permission to invoke your Lambda function…"**), choose **OK** to grant API Gateway that permission. Now you'll see a **/ – ANY – Method Execution** pane: ![](/files/-LuSomaK21ofGPn7m7KM) ### Deploy the REST API in the API Gateway Console Once you complete Step 2, you've created an API, but you can't actually use it yet. This is because it needs to be deployed. 1. From the **Actions** dropdown menu, choose **Deploy API**. 2. From the **Deployment stage** dropdown menu, choose **\[New Stage]**. 3. For **Stage name**, enter `prod`. 4. Choose **Deploy**. 5. In the `prod` **Stage Editor**, note the **Invoke URL** at the top. It should be in this format: (`https://`*`b123abcde4`*.execute-api.*`us-west-2`*.amazonaws.com/prod). If you choose the **Invoke URL**, it will open a new browser tab with that URL. # Slack This page documents how you can integrate Slack with a Spherical CloudFormation stack. Spherical integrates nicely with Slack using AWS, and can stream alerts directly to a channel of your choice. The following steps need to happen *before* starting a Spherical instance. To start test driving Spherical with Slack, go onto your Slack dashboard [here](https://api.slack.com/apps) and create a new Slack App. This app will be configured to interact with a Spherical instance. ![](/files/-LuImD8xWSXI6Ae-wM5L) Once you have created the app, go on to the *Incoming Webhooks* link on your sidebar and activate incoming web-hooks. ![](/files/-LuImH7Cbdg8yYl6Ky90) Once activated, click ‘*Add New Webhook to workspace*’, and copy the resultant web-hook URL. ![](/files/-LuImMwMBGXZ91WdUQLY) Once you have the web-hook to your Slack app, launch the Spherical stack on AWS using the CloudFormation button. Add the above web-hook url to the **SlackWebhook** parameter. # Splunk This page describes how to configure a Splunk integration with a Spherical CloudFormation stack. ## Pre-installation configuration Before you install a Spherical stack, you must first set up a **HTTP Event Collector** in your installation of **Splunk**. 1. For **Splunk Enterprise** or self-service **Splunk Cloud**, first go to 'Settings'. 2. Click 'Data inputs' and navigate to 'HTTP Event Collector'. 3. Click 'Global Settings'. 4. Click the Enable button, and then click Save. (For more information, see "Enable HTTP Event Collector" in the Getting Data in the Splunk manual). *Note: For managed Splunk Cloud, submit a support ticket to have the feature enabled.* 5. Create at least one input token. You'll need this token later. ![Settings for Splunk Enterprise and self-service Splunk Cloud.](/files/-LuIakfvk3gLGAutYP4o) ![Data input settings.](/files/-LuIkGDzNQVi1Whz1DP4) ### Token 1. For Splunk Enterprise or self-service Splunk Cloud, click the 'Add New' button. 2. Proceed through the 'Add Data' workflow until you've successfully created a token. \ *(For more information, see 'Create an Event Collector' token in the 'Getting Data In' Splunk manual).* 3. For managed Splunk Cloud, submit a support ticket to create or manage a token. ![Configuring a new token for recieving HT](/files/-LuIbUleCeRS18XPJHI1) ### Splunk URL In addition to an authorisation token, you also need a Splunk URL. This varies depending on your type of Splunk deployment. *Enterprise:* `://:/` \ *Managed Cloud:* `://http-inputs-:/` \ *Self-Service Cloud:* `://input-:/` | Field | Description | | -------- | -------------------------------------------------------------------------------------------- | | Protocol | Either HTTP or HTTPs | | Host | The Splunk instance that runs HEC | | Port | The HEC port number, which is 8088 by default, but you can change in the HEC Global Settings | | Endpoint | The HEC endpoint you want to use. Usually this is the `/services/collector` endpoint | Once you have got both a **Splunk URL** and a **Token**, you can go ahead and create a Spherical stack using CloudFormation. # NGINX (WIP) This page describes how to use the NGINX traffic mirroring module to integrate with a Spherical instance NGINX can integrate with Spherical using the [**mirror** module](http://nginx.org/en/docs/http/ngx_http_mirror_module.html) which appeared in version 1.13.4. # Changelog A catalog of recent changes in the codebase. ## 4.0 - 2019-11-29 ## 3.0 - 2019-11-25 ### Added * A continuous score for each request to enable ranking of requests in accordance with how bad the model thinks it is. ### Fixed * If the amount of data that we are training from is *exactly* 16,000 requests - training will crash and no model will be mounted. * Some 500 responses were still occasionally returned. ## 2.0 - 2019-11-06 ### Fixed * 500 errors that are returned by the ingestion service. ### Changed * Explainability has been improved, and is now more readable. ## 1.0 - 2019-10-03 ### ​Added * First distributable cut. ##