Amazon Machine Image

This page shows how you can get up and running with an Amazon Machine Image

You can use Amazon Machine Images to get the Spherical Defense service up and running, and then craft your own integrations around it.

This installation method gives you the greatest flexibility, and is useful when you wish to train a security model without having to ingest traffic over the standard HTTP interface.

This deployment approach is free for 7 days, before reverting to a $1 per hour pricing model.

Installation

It is straightforward to install a Spherical AMI onto your infrastructure.

1. Go to the Spherical Defense listing here.

2. Click 'Continue to Subscribe'.

3. At this stage you can view our End User License Agreement, before clicking 'Continue to Configuration'.

4. Select your fulfilment options, and click 'Continue to Launch'.

5. On the launch screen, you will be presented with several options:

   1. From 'Choose Action' - select 'Launch from Website'.

   2. From 'EC2 Instance Type' - select 'G3.4xlarge'.

   3. From 'VPC Settings' - select a VPC for the Spherical app, and then either select an existing subnet within this VPC that you already have in place for other service, or let AWS create you one automatically for the deployment.

   4. From 'Security Group Settings' - either create your own, or use an existing one. Ensure that ports 9000 and 8080 are open. Port 8080 is the management port, and needs to be exposed to your local machine. Port 9000 is the ingestion port, and must be exposed to your API gateway VPC.

   5. From 'Key Pair Settings' - select a key pair that you've previously used to access your AWS instances.

6. Click 'Launch'.

7. In the region that you launched in, go to the AWS EC2 console, and verify that the instance launched correctly.

Data Ingestion

A running Spherical instance needs data in order to build a machine learning model to protect you. There are two ways to pass in this data. For longer term protection, a live stream of traffic is necessary - ideally from an API gateway. However, to get up and running quickly you can use an existing dataset.

The format required when ingesting a static dataset is a sequence of JSON objects, separated by a new line. See the following example:

{"method": "POST", "url": "/api/payment", "body": {"expiry": "11/2013", "number": "3112893758824764", "cvv": "718"}, "headers": {"Content-Length": "311", "Accept-Language": "en-US,en;q=0.8", "Content-Type": "application/json,*/*;q=0.1", "Accept": "application/json", "Accept-Encoding": "deflate", "X-Forwarded-Port": "9490", "Cookies": [{"vk": "O226kQr8-WC5P-gVll-bkRc-1aGv3bLW"}]}}
{"method": "DELETE", "url": "/orders/vHgPXnL7C6U4kWPE95ZEQlDKW0ESNTeCk", "query": {"order_id": "vHgPXnL7C6U4kWPE95ZEQlDKW0ESNTeCk"}, "headers": {"Content-Length": "294", "Accept-Language": "en-US,en;q=0.8", "Content-Type": "application/json,*/*;q=0.1", "Accept": "application/json", "Accept-Encoding": "deflate", "X-Forwarded-Port": "7663", "Cookies": [{"vk": "VCkYuJST-ruJX-ghM0-bTFf-GpmnsVBo"}]}}
{"method": "POST", "url": "/deposits/payment-method", "body": {"amount": "9948453.498402445", "currency": "Gbi", "payment_method_id": "RjDyGDw3W3DxvEJzhb9iFCLcchC0ugoEp9"}, "headers": {"Content-Length": "205", "Accept-Language": "en-US,en;q=0.8", "Content-Type": "application/json,*/*;q=0.1", "Accept": "application/json", "Accept-Encoding": "deflate", "X-Forwarded-Port": "6299", "Cookies": [{"vk": "a4Ho12tU-Kn2l-iKEC-jVGt-kEfpNXyD"}]}}
{"method": "POST", "url": "/deposits/account", "body": {"amount": "5595519.921778078", "currency": "tRy", "account_id": "HUVTS4iLTvajacsLEmOEwzFwfbOb0jDjbZdH"}, "headers": {"Content-Length": "291", "Accept-Language": "en-US,en;q=0.8", "Content-Type": "application/json,*/*;q=0.1", "Accept": "application/json", "Accept-Encoding": "deflate", "X-Forwarded-Port": "5453", "Cookies": [{"vk": "B1fWiRp2-PmKS-hb7Q-aPRN-8BQyN8y4"}]}}
...
{"method": "GET", "url": "/accounts/7124518/holds", "query": {"id": "7124518", "account_id": "7UBMT1IhCLyFFnQ2wdkjhGPipTq", "created_at": "a7PdLpBZDmvlhrrT6EGgl", "updated_at": "rCDH3e5ng05b4XvQW2Hx490zGlp", "amount": "4730830", "type": "qgSQgZBZoN", "ref": "WzMqzFrJem37GEt7wxlFdLn6twIet6apAUbgby"}, "headers": {"Content-Length": "227", "Accept-Language": "en-US,en;q=0.8", "Content-Type": "application/json,*/*;q=0.1", "Accept": "application/json", "Accept-Encoding": "deflate", "X-Forwarded-Port": "5367", "Cookies": [{"vk": "A7bpIqdo-PMUP-lUZa-Zkuk-xweqniPM"}]}}

1. Use a program such as SCP to transfer a static dataset file onto the running instance. If you already have existing logs in a different format, first you have to convert them into the correct format.

2. Once the dataset is on the filesystem, execute the ingestion script in the 'ubuntu' user root directory as shown below, after ensuring there are execution permissions in place.

If you don't have a dataset to hand, use the demo file provided at the bottom of this section.

cd ~/
chmod +x charon-upload.sh
./charon-upload <ABSOLUTE_PATH_TO_TRAIN_FILE> continuous CharacterDatabase

Ensure you use the absolute path to the log file or the ingestion process won't work.

This script will take the provided file, ingest it into the internal database, and start training a security model.

8MB

demo.txt

Demo dataset

Training Progress

GET https://<LOCATION_OF_RUNNING_SERVER>:8080/collections/:collection/progress

Once you have ingested a dataset, it's important to ensure that the training process has begun. This can be done by accessing this API endpoint. This method enables you to see the progress of a security model that is being trained on a specific collection.

Path Parameters

Name	Type	Description
Collection	string	The name of the collection. Default to 'continuous'

200 This response gives information about the data ingestion process, i.e whether enough data has been seen by the system, as well as: 1. The predicted accuracy of a running security model. 2. The parameters of the trained security model. 3. The number of different security models that have been trained and evaluated.

{
  "populationProgress": {
    "generation": 0,
    "numberOfIterations": 3000,
    "progress": 0.9951258086449938,
    "parents": 12,
    "popSize": 20,
    "active": true,
    "bestGenome": {
      "score": 0.9951258086449938,
      "genome": {
        "weight_decay": 0.00026861,
        "hidden_size": 105,
        "embedding_size": 201,
        "learning_rate": 0.02487348
      },
      "generation": 0
    },
    "elites": []
  },
  "trainingDataState": {
    "readyToTrain": true,
    "percentageOfDataConsumedForTraining": 1,
    "minimumTrainingBatches": 500,
    "keyDatabasePresent": true,
    "tokenDatabasePresent": true,
    "statsPresent": true,
    "dataSetsCounts": {
      "trainingDataCount": 8748,
      "attackDataCount": 0,
      "classifierFitDataCount": 0,
      "validationTreeDataCount": 0
    }
  }
}

Evaluate Security Model

POST https://<LOCATION_OF_RUNNING_SERVER>:9000/evaluate

Once a security model has been trained, you can start evaluating it by sending JSON objects in the body of the request.

Request Body

Name	Type	Description
	object	A JSON object to be evaluated by the security model

200 This response contains event information regarding the classified request object.

{
    "id": "5dde6f69a7b11b000696952f",
    "collection": "continuous",
    "timestamp": 1574858601646,
    "score": 0.89,
    "mostSuspiciousField": ".body.names.first:felipa",
    "request": "{\"method\":\"PUT\",\"url\":\"/api/Order\",\"id\":\"HVRwQMpb-QBWl-fRj7-aC0u-9wP4qS4H\",\"body\":{\"names\":{\"first\":\"felipa\",\"second\":\"wiggins\"},\"age\":\"67\",\"address\":\"venusst\",\"state\":\"CA\",\"city\":\"San Francisco\"},\"origin\":\"108.119.5.22\",\"status\":\"200\",\"headers\":{\"Content-Length\":\"283\",\"Accept-Language\":\"en-US,en;q=0.8\",\"Content-Type\":\"aplication/json,*/*;q=0.1\",\"Accept\":\"application/json\",\"Accept-Encoding\":\"deflate\",\"X-Forwarded-Port\":\"9697\",\"Host\":\"wwi.microsoft.org\",\"Cookies\":[{\"ASP.NET_SessionID\":\"HVRwQMpb-QBWl-fRj7-aC0u-9wP4qS4H\"},{\"vk\":\"HVRwQMpb-QBWl-fRj7-aC0u-9wP4qS4H\"}]}}",
    "isAnomaly": false
}