Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration with 3rd party services #46

Open
1 of 5 tasks
Rafiot opened this issue May 22, 2019 · 3 comments
Open
1 of 5 tasks

Integration with 3rd party services #46

Rafiot opened this issue May 22, 2019 · 3 comments
Labels
New Features

Comments

@Rafiot
Copy link
Member

Rafiot commented May 22, 2019 •
edited
Loading

@Rafiot Rafiot changed the title PSSL & PDNS Integration with 3rd party services May 23, 2019
@Rafiot Rafiot mentioned this issue May 26, 2019
Closed
@Rafiot
Copy link
Member Author

Rafiot commented Jun 6, 2019 •
edited
Loading

Note: getting the IP of each request should be done on splash side: scrapinghub/splash#599

Note2: It seems pretty much impossible to get the IP address from the PyQt code: scrapinghub/splash#599 (comment) - let's just do an extra query on lookyloo's side.

@Rafiot
Copy link
Member Author

Rafiot commented Oct 2, 2019

Example outputs for 3rd party integrations

Passive SSL

Query: IP

{
  "8.8.8.8": {
    "certificates": [
      "7359755c6df9a0abc3060bce369564c8ec4542a3",
      "d6ad07c6675630f57b927f66be8ce1f768f87948",
      "2b1cfcc7a28761057b4916c7f531efefcea0694e",
      "afc4b2ab726ba1be1a08a86fbde5c25bb38438c0"
    ],
    "subjects": {
      "d6ad07c6675630f57b927f66be8ce1f768f87948": {
        "values": [
          "C=US, O=Google Inc, CN=Google Internet Authority G2"
        ]
      },
      "7359755c6df9a0abc3060bce369564c8ec4542a3": {
        "values": [
          "C=US, O=GeoTrust Inc., CN=GeoTrust Global CA"
        ]
      },
      "2b1cfcc7a28761057b4916c7f531efefcea0694e": {
        "values": [
          "C=US, ST=California, L=Mountain View, O=Google Inc, CN=*.c.docs.google.com"
        ]
      }
    }
  }
}

Passive DNS

Query: domain (www.circl.lu)

{
  "count": 989255,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1475825162,
  "rrtype": "CNAME",
  "rrname": "www.circl.lu",
  "rdata": "cpab.circl.lu",
  "time_last": 1540860996
}
{
  "count": 20426,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1298398417,
  "rrtype": "A",
  "rrname": "www.circl.lu",
  "rdata": "194.154.205.24",
  "time_last": 1299264077
}
{
  "count": 23479,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1298398002,
  "rrtype": "CNAME",
  "rrname": "www.circl.lu",
  "rdata": "cpa.circl.lu",
  "time_last": 1329211894
}

Query: IP (194.154.205.24)

{
  "count": 225,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1316156112,
  "rrtype": "A",
  "rrname": "upl.cases.lu",
  "rdata": "194.154.205.24",
  "time_last": 1500285381
}
{
  "count": 401,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1298971806,
  "rrtype": "A",
  "rrname": "www.smile.public.lu",
  "rdata": "194.154.205.24",
  "time_last": 1327518411
}
{
  "count": 86,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1327953923,
  "rrtype": "CNAME",
  "rrname": "www.smile.public.lu",
  "rdata": "cpa.circl.lu",
  "time_last": 1499776884
}
{
  "count": 2,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1310560646,
  "rrtype": "A",
  "rrname": "www.energyefficient.lu",
  "rdata": "194.154.200.102",
  "time_last": 1310560646
}
{
  "count": 3,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1483454623,
  "rrtype": "A",
  "rrname": "www.energyefficient.lu",
  "rdata": "194.154.205.24",
  "time_last": 1483454623
}
{
  "count": 227,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1299530144,
  "rrtype": "A",
  "rrname": "circl.lu",
  "rdata": "149.13.33.4",
  "time_last": 1329124550
}
{
  "count": 61616567,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1475825158,
  "rrtype": "NS",
  "rrname": "circl.lu",
  "rdata": "ns4.eurodns.com",
  "time_last": 1540865989
}
{
  "count": 62812441,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1298398002,
  "rrtype": "NS",
  "rrname": "circl.lu",
  "rdata": "ns1.eurodns.com",
  "time_last": 1540865989
}
{
  "count": 62812431,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1298398002,
  "rrtype": "NS",
  "rrname": "circl.lu",
  "rdata": "ns2.eurodns.com",
  "time_last": 1540865989
}
{
  "count": 2,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1299079885,
  "rrtype": "A",
  "rrname": "circl.lu",
  "rdata": "194.154.205.24",
  "time_last": 1299079885
}
{
  "count": 61616567,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1475825158,
  "rrtype": "NS",
  "rrname": "circl.lu",
  "rdata": "ns3.eurodns.com",
  "time_last": 1540865989
}
{
  "count": 30179,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1475825169,
  "rrtype": "A",
  "rrname": "circl.lu",
  "rdata": "149.13.33.14",
  "time_last": 1540488114
}
{
  "count": 1,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1536246967,
  "rrtype": "A",
  "rrname": "ee.cases.lu",
  "rdata": "185.106.24.57",
  "time_last": 1536246967
}
{
  "count": 4,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1310548268,
  "rrtype": "A",
  "rrname": "ee.cases.lu",
  "rdata": "194.154.205.24",
  "time_last": 1324299334
}
{
  "count": 61,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1298670051,
  "rrtype": "A",
  "rrname": "enisa.cases.lu",
  "rdata": "194.154.205.24",
  "time_last": 1328781064
}
{
  "count": 191,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1298670049,
  "rrtype": "A",
  "rrname": "tools.cases.lu",
  "rdata": "194.154.205.24",
  "time_last": 1500022112
}
{
  "count": 9,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1301591528,
  "rrtype": "A",
  "rrname": "pwdtest.cases.lu",
  "rdata": "194.154.205.24",
  "time_last": 1324299383
}
{
  "count": 1,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1303894059,
  "rrtype": "MX",
  "rrname": "smile.public.lu",
  "rdata": "20",
  "time_last": 1303894059
}
{
  "count": 254,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1301507879,
  "rrtype": "A",
  "rrname": "smile.public.lu",
  "rdata": "194.154.205.24",
  "time_last": 1500045746
}
{
  "count": 1,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1303894059,
  "rrtype": "MX",
  "rrname": "smile.public.lu",
  "rdata": "10",
  "time_last": 1303894059
}
{
  "count": 12,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1323760163,
  "rrtype": "A",
  "rrname": "storage.cases.lu",
  "rdata": "194.154.205.24",
  "time_last": 1328107207
}
{
  "count": 9,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1299256500,
  "rrtype": "A",
  "rrname": "ee.dev.cases.lu",
  "rdata": "194.154.205.24",
  "time_last": 1301395667
}
{
  "count": 41,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1316101572,
  "rrtype": "A",
  "rrname": "bs.cases.lu",
  "rdata": "194.154.205.24",
  "time_last": 1319093317
}
{
  "count": 62,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1314889582,
  "rrtype": "A",
  "rrname": "rdv.cases.lu",
  "rdata": "194.154.205.24",
  "time_last": 1496823450
}
{
  "count": 2,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1317737507,
  "rrtype": "A",
  "rrname": "testio2.cases.lu",
  "rdata": "194.154.205.24",
  "time_last": 1317737507
}
{
  "count": 989255,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1475825162,
  "rrtype": "CNAME",
  "rrname": "www.circl.lu",
  "rdata": "cpab.circl.lu",
  "time_last": 1540860996
}
{
  "count": 20426,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1298398417,
  "rrtype": "A",
  "rrname": "www.circl.lu",
  "rdata": "194.154.205.24",
  "time_last": 1299264077
}
{
  "count": 23479,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1298398002,
  "rrtype": "CNAME",
  "rrname": "www.circl.lu",
  "rdata": "cpa.circl.lu",
  "time_last": 1329211894
}
{
  "count": 21,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1306228053,
  "rrtype": "A",
  "rrname": "pwws.cases.lu",
  "rdata": "194.154.205.24",
  "time_last": 1327068039
}
{
  "count": 24,
  "origin": "https://www.circl.lu/pdns/",
  "time_first": 1310548430,
  "rrtype": "A",
  "rrname": "epass.cases.lu",
  "rdata": "194.154.205.24",
  "time_last": 1327423495
}

VirusTotal

Query types:

@stale
Copy link

stale bot commented Nov 17, 2020

Close call! This issue has been marked as stale because it has not had any recent activity. It should be closed if no further activity occurs. Add a comment or push a commit to keep this issue stay alive and kicking. Thank you for your contribution; it is appreciated.

@stale stale bot added the stale label Nov 17, 2020
@Rafiot Rafiot removed the stale label Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
New Features
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Rafiot