Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

skipping records after successful/unsuccessful import #101

Open
ghost opened this issue Apr 23, 2019 · 1 comment
Open

skipping records after successful/unsuccessful import #101

ghost opened this issue Apr 23, 2019 · 1 comment

Comments

@ghost
Copy link

ghost commented Apr 23, 2019

Unable to insert Downloaded example-data.csv from: https://wiki.52north.org/SensorWeb/SosImporter#How_to_Build
Using SOS importer from: https://github.com/52North/sos-importer/releases/tag/v0.4.2

Records are skipped and not inserted. Thanks for help in advance.

Following is the output at the end of wizard module execution. Attaching input file, configuration file, and output file
example-data.zip

11:30:34.102 [main] INFO Feeder - Application started
Implementation-Title: Feeding Module
Implementation-Version: 0.4.0
Archiver-Version: Plexus Archiver
Built-By: jenkins
Implementation-Vendor-Id: org.n52.sensorweb
geotools-version: 9.5
n52-oxf-version: 2.0.0-alpha.4
Manifest-Version: 1.0
Implementation-Vendor: 52°North
Main-Class: org.n52.sos.importer.feeder.Feeder
build-time: 2017-02-13_14-56
Created-By: Apache Maven
Build-Jdk: 1.8.0_74
n52-sos-importer-xmlbeans-version: 0.2
HeapSize Information: max: 7097MB; total now: 479MB; free now: 456MB; used now: 22MB
os.name: Linux; os.arch: amd64; os.version: 4.15.0-47-generic
11:30:34.282 [OneTimeFeeder] INFO OneTimeFeeder - Starting feeding data from file via configuration 'example-data.csv.52n-sos-import-config.xml' to SOS instance
11:30:34.288 [OneTimeFeeder] TRACE SensorObservationService - SensorObservationService(Configuration [file=/home/trupti/Desktop/example-data.csv.52n-sos-import-config.xml])
11:30:34.334 [OneTimeFeeder] DEBUG SimpleHttpClient - executing POST method to 'http://localhost:8080/52n-sos-webapp/service'.
11:30:34.742 [OneTimeFeeder] INFO SensorObservationService - Using 25ms timeout buffer during insert observation requests. Change if required.
11:30:34.743 [OneTimeFeeder] DEBUG SensorObservationService - OperationsMetadata found: org.n52.oxf.ows.capabilities.OperationsMetadata@741eff7a
11:30:34.743 [OneTimeFeeder] DEBUG SensorObservationService - Found all required operations: (RegisterSensor|InsertSensor), InsertObservation
11:30:34.744 [OneTimeFeeder] DEBUG OneTimeFeeder - Check counter file '/root/.SOSImporter/600dd3e7bee371d56c2d30919fa3fb7c'.
11:30:34.744 [OneTimeFeeder] DEBUG OneTimeFeeder - Read already read lines from file
11:30:34.750 [OneTimeFeeder] DEBUG SensorObservationService - Lastline updated: old: 0; new: 6985
11:30:34.750 [OneTimeFeeder] TRACE SensorObservationService - importData()
11:30:34.754 [OneTimeFeeder] TRACE SensorObservationService - Skip CSV line #2: 22.08.1984 15:00, 21.5894, 34.5897, 13.0003
11:30:34.754 [OneTimeFeeder] TRACE SensorObservationService - Skip CSV line #3: 22.08.1984 15:15, 22.4305, 34.2951, 11.8646
11:30:34.755 [OneTimeFeeder] TRACE SensorObservationService - Skip CSV line #4: 22.08.1984 15:30, 25.4943, 33.0418, 7.5475
.
.
.
.
11:30:35.251 [OneTimeFeeder] TRACE SensorObservationService - Skip CSV line #6985: 03.11.1984 09:45, 12.3489, 21.9825, 9.6337
11:30:35.251 [OneTimeFeeder] DEBUG SensorObservationService - Using hunkSize '5000'
11:30:35.253 [OneTimeFeeder] DEBUG SensorObservationService - Timing:
Start File: Tue Apr 23 11:30:35 CEST 2019
Finished importing: Tue Apr 23 11:30:35 CEST 2019
11:30:35.253 [OneTimeFeeder] INFO SensorObservationService - New observations in SOS: 0. Failed observations: 0.
11:30:35.253 [OneTimeFeeder] INFO OneTimeFeeder - OneTimeFeeder: save read lines count: 6985 to '/root/.SOSImporter/600dd3e7bee371d56c2d30919fa3fb7c'
11:30:35.253 [OneTimeFeeder] INFO OneTimeFeeder - Feeding data from file example-data.csv to SOS instance finished.

@ghost
Copy link
Author

ghost commented Apr 24, 2019

I think this issue is related to: #43
I experience that even though once the data is inserted successfully, you cannot insert the same data again from the same path. The records gets skipped! Unsuccessful data insertion also results in skipping records on next try!

@ghost ghost changed the title unable to insert example data from csv skipping records after successful/unsuccessful import Apr 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

0 participants