i want to test your crawler help me!!!

[DeveloperYoungsuk]

i`m planning to test your crawler against Scrapy

but i don`t have enough example code.

can you provide some example code for me??

thank you for making an awesome software ~~

[essiembre]

What type of code sample are you looking for?

We try to make the HTTP Collector easy to extend and work with for coders, but it is used by most as a stand-alone command-line application.

You can find here how to get started: https://www.norconex.com/collectors/collector-http/getting-started

If you wonder how to handle specific use cases not addressed in the documentation or other tickets, feel free to describe them here.

[DeveloperYoungsuk]

how can i parse json??

[essiembre]

JSON files will get crawled. Can you elaborate on what you are trying to do exactly? If you mean parsing a JSON file so some fields can be stored as metadata fields, for instance, you can do so by implementing your own IDocumentTagger.

If you know the pattern of just a few items to extract, you can use another approach such as with a TextPatternTagger. Alternatively, we can make it a feature request to have JSON parsing options built-in.

[DeveloperYoungsuk]

i`m planning to parse 300 page in premierleague.com like this ↓ {"entity":{"playerId":136935.0,"info":{"position":"F","shirtNum":18.0,"positionInfo":"Centre Striker","loan":true},"nationalTeam":{"isoCode":"GB-ENG","country":"England","demonym":"English"},"birth":{"date":{"millis":8.757504E11,"label":"2 October 1997"},"country":{"isoCode":"GB-ENG","country":"England","demonym":"English"},"place":"London"},"age":"21 years 333 days","name":{"display":"Tammy Abraham","first":"Tammy","last":"Abraham"},"id":13286.0},

[essiembre]

What do you want to do with the information? Store each field/value in a separate field in your database or search engine? Something else?

I suggest you look at available options for manipulating content (e.g. taggers, splitters, etc.) here: https://www.norconex.com/collectors/importer/configuration

Maybe you will find ways to break the JSON the way you want. Otherwise, you will have to create your own or wait for that feature to be implemented.

I am marking this as a feature request to have a tagger that can extract data from JSON (similar to DOMTagger).