Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Support for additional GeoIP Databases #12562

Open
minars2 opened this issue Jan 12, 2021 · 0 comments
Open

Add Support for additional GeoIP Databases #12562

minars2 opened this issue Jan 12, 2021 · 0 comments

Comments

@minars2
Copy link

minars2 commented Jan 12, 2021

The ask in this feature request is to add support for more databases, specifically my case would be the DB-IP Location + ISP database, but it seems others would like the rest of the paid MaxMind databases as well.

It looks like the GeoIP filter can read the file but it expects a different schema for the data and therefore fails out when trying to use the DB-IP database in the GeoIP filter with the database option.

DB-IP's database schema is available here:
https://db-ip.com/db/format/ip-to-location-isp/mmdb.html

They also link to a MaxMind code library to read MMDB's which when I attempted to read the DB-IP database it wanted to use the Enterprise database reader, which at the moment doesn't appear to be supported by the GeoIP Filter.

I don't have access to a MaxMind Enterprise database so I couldn't check if to see the schema is similar or not, but the library seems to think so.

Also, it looks like the GeoIP filter does not recognize that MaxMind has added additional databases, such as the Enterprise, Anonymous IP, Connection Type, and Domain, all of which are paid MaxMind databases. Info for these databases can be found here: https://dev.maxmind.com/

There have been a few pull requests for adding support for some of these databases to the GeoIP Filter:
Anonymous IP database PR from 2018: logstash-plugins/logstash-filter-geoip#141
Domain database PR from 2020: https://github.com/logstash-plugins/logstash-filter-geoip/pull/162

Finally, here is a sample of data that I was able to pull using MaxMind's Python MMDB file reader inputting the IP 1.1.1.1 to the DB-IP database and Geolite2 City database:

https://github.com/maxmind/MaxMind-DB-Reader-python

Geolit2 Database:

{
    'continent': {
        'code': 'OC',
        'geoname_id': 6255151,
        'names': {
            'de': 'Ozeanien',
            'en': 'Oceania',
            'es': 'Oceanía',
            'fr': 'Océanie',
            'ja': 'オセアニア',
            'pt-BR': 'Oceania',
            'ru': 'Океания',
            'zh-CN': '大洋洲'
        }
    },
    'country': {
        'geoname_id': 2077456,
        'iso_code': 'AU',
        'names': {
            'de': 'Australien',
            'en': 'Australia',
            'es': 'Australia',
            'fr': 'Australie',
            'ja': 'オーストラリア',
            'pt-BR': 'Austrália',
            'ru': 'Австралия',
            'zh-CN': '澳大利亚'
        }
    },
    'location': {
        'accuracy_radius': 1000,
        'latitude': -33.494,
        'longitude': 143.2104,
        'time_zone': 'Australia/Sydney'
    },
    'registered_country': {
        'geoname_id': 2077456,
        'iso_code': 'AU',
        'names': {
            'de': 'Australien',
            'en': 'Australia',
            'es': 'Australia',
            'fr': 'Australie',
            'ja': 'オースト ラリア',
            'pt-BR': 'Austrália',
            'ru': 'Австралия',
            'zh-CN': '澳大利亚'
        }
    }
}

DB-IP Database:

{
    'city': {
        'geoname_id': 2147714,
        'names': {
            'de': 'Sydney',
            'en': 'Sydney',
            'es': 'Sídney',
            'fa': 'سیدنی',
            'fr': 'Sydney',
            'ja': 'シドニー',
            'ko': '시드니',
            'pt-BR': 'Sydney',
            'ru': 'Сидней',
            'zh-CN': '悉尼'
        }
    },
    'continent': {
        'code': 'OC',
        'geoname_id': 6255151,
        'names': {
            'de': 'Ozeanien',
            'en': 'Oceania',
            'es': 'Oceanía',
            'fa': 'اقیانوسیه',
            'fr': 'Océanie',
            'ja': 'オセアニア',
            'ko': '오세아니아',
            'pt-BR': 'Oceania',
            'ru': 'Океания',
            'zh-CN': '大洋洲'
        }
    },
    'country': {
        'geoname_id': 2077456,
        'is_in_european_union': False,
        'iso_code': 'AU',
        'names': {
            'de': 'Australien',
            'en': 'Australia',
            'es': 'Australia',
            'fa': 'استرالیا',
            'fr': 'Australie',
            'ja': 'オーストラリア',
            'ko': '오스트레일리아',
            'pt-BR': 'Austrália',
            'ru': 'Австралия',
            'zh-CN': '澳大利亚'
        }
    },
    'location': {
        'latitude': -33.8688,
        'longitude': 151.209,
        'time_zone': 'Australia/Sydney',
        'weather_code': 'ASXX0112'
    },
    'postal': {
        'code': '1001'
    },
    'subdivisions': [{
            'geoname_id': 2155400,
            'iso_code': 'NSW',
            'names': {
                'en': 'New South Wales',
                'fr': 'Nouvelle-Galles du Sud',
                'pt-BR': 'Nova Gales do Sul',
                'ru': 'Новый Южный Уэльс'
            }
        }
    ],
    'traits': {
        'autonomous_system_number': 13335,
        'autonomous_system_organization': 'Cloudflare, Inc.',
        'connection_type': 'Corporate',
        'isp': 'Cloudflare, Inc.'
    }
}

Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants