basic opentaxii 2.1 docker install and python post

[lcia-projects]

hi, i'm trying to get a basic openTaxii server up.. and get a python script to insert data into that opentaxii server.

i've googled all i know to google.. i've checked the github issues for examples.. with no luck.. so here i am..

here is my basic docker-compose.yml

db:
  image: postgres:9.4
  environment:
    POSTGRES_USER: user
    POSTGRES_PASSWORD: password
    POSTGRES_DB: opentaxii

authdb:
  image: postgres:9.4
  environment:
    POSTGRES_USER: user1
    POSTGRES_PASSWORD: password1
    POSTGRES_DB: opentaxii1

opentaxii:
  image: eclecticiq/opentaxii
  environment:
    OPENTAXII_AUTH_SECRET: secret
    OPENTAXII_DOMAIN: 192.168.3.95:9000
    OPENTAXII_USER: user
    OPENTAXII_PASS: pass
    DATABASE_HOST: db
    DATABASE_NAME: opentaxii
    DATABASE_USER: user
    DATABASE_PASS: password
    AUTH_DATABASE_HOST: authdb
    AUTH_DATABASE_NAME: opentaxii1
    AUTH_DATABASE_USER: user1
    AUTH_DATABASE_PASS: password1
  volumes:
    - ./:/input:ro
  ports:
    - 9000:9000
  links:
    - db:db
    - authdb:authdb

opentaxii2:
  image: eclecticiq/opentaxii
  environment:
    OPENTAXII_AUTH_SECRET: secrettwo
    OPENTAXII_DOMAIN: 192.168.3.95
    OPENTAXII_USER: user1
    OPENTAXII_PASS: pass1
    DATABASE_HOST: authdb
    DATABASE_NAME: opentaxii1
    DATABASE_USER: user1
    DATABASE_PASS: password1
  volumes:
    - ./:/input:ro
  ports:
    - 9001:9000
  links:
    - authdb:authdb

here is my modified data-configuration.yml to allow for stix2.1

---

domain: 192.168.3.95:9000

services:
    - id: inbox
      type: inbox
      address: /services/inbox
      description: Inbox Service
      destination_collection_required: yes
      accept_all_content: yes
      authentication_required: yes
      supported_content:
        - urn:stix.mitre.org:json:2.1
      protocol_bindings:
        - urn:taxii.mitre.org:protocol:http:1.0

    - id: discovery
      type: discovery
      address: /services/discovery
      description: Discovery Service
      advertised_services:
        - inbox
        - discovery
        - collection_management
        - poll
      protocol_bindings:
        - urn:taxii.mitre.org:protocol:http:1.0

    - id: collection_management
      type: collection_management
      address: /services/collection-management
      description: Collection Management Service
      protocol_bindings:
        - urn:taxii.mitre.org:protocol:http:1.0

    - id: poll
      type: poll
      address: /services/poll
      description: Poll Service
      subscription_required: no
      max_result_count: 100
      max_result_size: 10
      authentication_required: yes
      protocol_bindings:
        - urn:taxii.mitre.org:protocol:http:1.0

collections:
  - name: cs2
    available: true
    accept_all_content: true
    supported_content:
      - urn:stix.mitre.org:json:2.1
    service_ids:
      - inbox
      - collection_management
      - poll

accounts:
  - username: community
    password: community123
    permissions:
      cs2: modify

and here is my basic python code trying to insert data into cs2 collection:

from pprint import pprint
import json

from cabby import create_client

client = create_client(
    '192.168.3.95',
    port=9000,
    use_https=False,
    discovery_path='/services/discovery'
)
print (client)
client.set_auth(username='community', password='community123')

services = client.discover_services()

binding="urn:stix.mitre.org:json:2.1"

#example from stix2.1 examples
test_stix21={
    "type": "bundle",
    "id": "bundle--2ac7882f-76a3-4a9b-97b3-811b3af1c7c0",
    "objects": [
        {
            "type": "indicator",
            "spec_version": "2.1",
            "id": "indicator--9299f726-ce06-492e-8472-2b52ccb53191",
            "created_by_ref": "identity--39012926-a052-44c4-ae48-caaf4a10ee6e",
            "created": "2017-02-27T13:57:10.515Z",
            "modified": "2017-02-27T13:57:10.515Z",
            "name": "Malicious URL",
            "description": "This URL is potentially associated with malicious activity and is listed on several blacklist sites.",
            "indicator_types": [
                "malicious-activity"
            ],
            "pattern": "[url:value = 'http://paypa1.banking.com']",
            "pattern_type": "stix",
            "valid_from": "2015-06-29T09:10:15.915Z"
        },
        {
            "type": "identity",
            "spec_version": "2.1",
            "id": "identity--39012926-a052-44c4-ae48-caaf4a10ee6e",
            "created": "2017-02-24T15:50:10.564Z",
            "modified": "2017-02-24T15:50:10.564Z",
            "name": "Alpha Threat Analysis Org.",
            "roles": [
                "Cyber Security"
            ],
            "identity_class": "organization",
            "sectors": [
                "technology"
            ],
            "contact_information": "[email protected]"
        },
        {
            "type": "identity",
            "spec_version": "2.1",
            "id": "identity--5206ba14-478f-4b0b-9a48-395f690c20a2",
            "created": "2017-02-26T17:55:10.442Z",
            "modified": "2017-02-26T17:55:10.442Z",
            "name": "Beta Cyber Intelligence Company",
            "roles": [
                "Cyber Security"
            ],
            "identity_class": "organization",
            "sectors": [
                "technology"
            ],
            "contact_information": "[email protected]"
        },
        {
            "type": "sighting",
            "spec_version": "2.1",
            "id": "sighting--8356e820-8080-4692-aa91-ecbe94006833",
            "created_by_ref": "identity--5206ba14-478f-4b0b-9a48-395f690c20a2",
            "created": "2017-02-28T19:37:11.213Z",
            "modified": "2017-02-28T19:37:11.213Z",
            "first_seen": "2017-02-27T21:37:11.213Z",
            "last_seen": "2017-02-27T21:37:11.214Z",
            "count": 1,
            "sighting_of_ref": "indicator--9299f726-ce06-492e-8472-2b52ccb53191",
            "where_sighted_refs": [
                "identity--5206ba14-478f-4b0b-9a48-395f690c20a2"
            ]
        }
    ]
}

test_stix21=json.dumps(test_stix21)
for service in services:
    print('Service type={s.type}, address={s.address}'
          .format(s=service))
    service_address=service.address
    s_type=service.type
    content_blocks = client.poll(collection_name=s_type)

print (":")
gservices=client.get_services()
for service in gservices:
    print('Service type={s.type}, address={s.address}'
          .format(s=service))
    
# print ("::")
collections=client.get_collections('http://192.168.3.95:9000/services/collection-management')

for collection_item in collections:
    print ("cname",collection_item.name)

# print(":::")
content_count=client.get_content_count('cs2')
print (content_count)

push_result=client.push(test_stix21, binding, collection_names=['cs2'],uri='/services/inbox')

print ("pr:",push_result)
print(":::")
content_count=client.get_content_count('cs2')
print (content_count)

i get no errors... but it doesnt insert into collection either..
any examples .. or tips would be greatly appreciated.

[lcia-projects]

i think i got it.. i at least have stuff going in:

from pprint import pprint
import json

from cabby import create_client

client = create_client(
    '192.168.1.114',
    port=9000,
    use_https=False,
    discovery_path='/services/discovery'
)
print (client)
#client.set_auth(username='community', password='community123')
client.set_auth(username='admin', password='admin')

services = client.discover_services()

#test data
binding="urn:stix.mitre.org:json:2.1"

test_stix21={
    "type": "bundle",
    "id": "bundle--2ac7882f-76a3-4a9b-97b3-811b3af1c7c0",
    "objects": [
        {
            "type": "indicator",
            "spec_version": "2.1",
            "id": "indicator--9299f726-ce06-492e-8472-2b52ccb53191",
            "created_by_ref": "identity--39012926-a052-44c4-ae48-caaf4a10ee6e",
            "created": "2017-02-27T13:57:10.515Z",
            "modified": "2017-02-27T13:57:10.515Z",
            "name": "Malicious URL",
            "description": "This URL is potentially associated with malicious activity and is listed on several blacklist sites.",
            "indicator_types": [
                "malicious-activity"
            ],
            "pattern": "[url:value = 'http://paypa1.banking.com']",
            "pattern_type": "stix",
            "valid_from": "2015-06-29T09:10:15.915Z"
        },
        {
            "type": "identity",
            "spec_version": "2.1",
            "id": "identity--39012926-a052-44c4-ae48-caaf4a10ee6e",
            "created": "2017-02-24T15:50:10.564Z",
            "modified": "2017-02-24T15:50:10.564Z",
            "name": "Alpha Threat Analysis Org.",
            "roles": [
                "Cyber Security"
            ],
            "identity_class": "organization",
            "sectors": [
                "technology"
            ],
            "contact_information": "[email protected]"
        },
        {
            "type": "identity",
            "spec_version": "2.1",
            "id": "identity--5206ba14-478f-4b0b-9a48-395f690c20a2",
            "created": "2017-02-26T17:55:10.442Z",
            "modified": "2017-02-26T17:55:10.442Z",
            "name": "Beta Cyber Intelligence Company",
            "roles": [
                "Cyber Security"
            ],
            "identity_class": "organization",
            "sectors": [
                "technology"
            ],
            "contact_information": "[email protected]"
        },
        {
            "type": "sighting",
            "spec_version": "2.1",
            "id": "sighting--8356e820-8080-4692-aa91-ecbe94006833",
            "created_by_ref": "identity--5206ba14-478f-4b0b-9a48-395f690c20a2",
            "created": "2017-02-28T19:37:11.213Z",
            "modified": "2017-02-28T19:37:11.213Z",
            "first_seen": "2017-02-27T21:37:11.213Z",
            "last_seen": "2017-02-27T21:37:11.214Z",
            "count": 1,
            "sighting_of_ref": "indicator--9299f726-ce06-492e-8472-2b52ccb53191",
            "where_sighted_refs": [
                "identity--5206ba14-478f-4b0b-9a48-395f690c20a2"
            ]
        }
    ]
}
#####
test_stix21=json.dumps(test_stix21)

for service in services:
    print('Service type={s.type}, address={s.address}'
          .format(s=service))
    service_address=service.address
    s_type=service.type
    content_blocks = client.poll(collection_name=s_type)

print (":")
gservices=client.get_services()
for service in gservices:
    print('Service type={s.type}, address={s.address}'
          .format(s=service))

push_result=client.push(test_stix21, binding, collection_names=['cs2'],uri='/services/inbox')
content_count=client.get_content_count('cs2')
print (content_count)
print (push_result)
content_blocks = client.poll(collection_name='cs2')
print (content_blocks)
for item in content_blocks:
    print (item.content)```

[lcia-projects]

alright.. question.. can you push a stix21 bundle into opentaxii?
if so.. how?

[arcsector]

Hi there @lcia-projects - since OpenTAXII is content-agnostic as TAXII v1 as a protocol was meant to be, you should be able to do this however you'd like

[erwin-eiq]

Hi @lcia-projects thank you for your interest in this project.

If I read you last example correctly you're already pushing a stix21 bundle into opentaxii. What's the response you're getting from that code and what's the response you expected?