Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/opg scanning poc #6

Merged
merged 45 commits into from
Nov 15, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
459810e
feat/opg-scanning: initial commit of base structure
Oct 29, 2024
82ea835
feat/opg-scanning-poc: simplification of factory parser format type c…
Oct 30, 2024
0aacb5d
feat/opg-scanning-poc: refactored approach after deeper analysis of d…
Oct 30, 2024
75d44b4
feat/opg-scanning-poc: integrate centralized logging in ingestion queue.
Oct 30, 2024
6aa18e1
feat/opg-scanning-poc: job queue test implementation - added proper p…
Oct 30, 2024
d46969d
feat/opg-scanning-poc: added empty callback to avoid breaking changes..
Oct 30, 2024
0d66214
feat/opg-scanning-poc: fixed typo in readme.
Oct 30, 2024
be438d1
feat/opg-scanning-poc: added context-based managment to worker pool t…
Oct 30, 2024
60b1b9d
feat/opg-scanning-poc: implemented xsd validation / cleaned redundant…
Oct 31, 2024
ffceab1
feat/opg-scanning-poc: bug fixes and prep for test handler of api.
Nov 1, 2024
bee2d3c
feat/opg-scanning-poc: fixed config.go implementation, improved inges…
Nov 1, 2024
bc21772
feat/opg-scanning-poc: first handler test pass for xsd validation.
Nov 1, 2024
7dbd699
feat/opg-scanning-poc: updated handler_test.go.
Nov 1, 2024
53343e0
feat/opg-scanning-poc: better logging and improved config.go / added …
Nov 4, 2024
5b6082d
feat/opg-scanning-poc: removed log dir.
Nov 4, 2024
95ef026
feat/opg-scanning-poc: added parser vaidators based on struct require…
Nov 4, 2024
5cda630
Fix build pipeline
gregtyler Nov 4, 2024
c3cd52c
Set PROJECT_PATH when running tests
gregtyler Nov 4, 2024
9908c7a
feat/opg-scanning-poc: test fix for uncontrolled data used in path ex…
Nov 4, 2024
b1576fa
Merge branch 'feat/opg-scanning-poc' of https://github.com/ministryof…
Nov 4, 2024
a5b6605
feat/opg-scanning-poc: fixed a bug with logger not initalizing in job…
Nov 4, 2024
7c6af09
Fix Trivy and ECR workflow
gregtyler Nov 4, 2024
6114140
Install AWS CLI before configuring ECR
gregtyler Nov 4, 2024
625a707
Assume correct role
gregtyler Nov 4, 2024
f8d03a9
feat/opg-scanning-poc/types-refactor: initial commit of complete lpf1…
Nov 6, 2024
806e7fe
Merge branch 'feat/opg-scanning-poc' of https://github.com/ministryof…
Nov 6, 2024
e02eb8c
feat/opg-scanning-poc: refactored factory to be more extensible and m…
Nov 6, 2024
78bc329
feat/opg-scanning-poc: date validators added / document factory test
Nov 7, 2024
df72866
feat/opg-scanning-poc: clean-up.
Nov 7, 2024
29c5e33
feat/opg-scanning-poc: added alternative xml for test case and extend…
Nov 11, 2024
b99ec2c
feat/opg-scanning-poc: added common validators.
Nov 11, 2024
abd65b4
feat/opg-scanning-poc: bug fixes along with common validators impleme…
Nov 12, 2024
279c65a
feat/opg-scanning-poc: fixed issue with validation and introduced han…
Nov 12, 2024
28a765b
feat/opg-scanning-poc: fixed bug with array structures being returned…
Nov 13, 2024
47534a3
feat/opg-scanning-poc: better coverage for invalid xml example.
Nov 13, 2024
bcc11c8
feat/opg-scanning-poc: fixes against comments in PR - config.yml / RE…
Nov 15, 2024
3c943ff
feat/opg-scanning-poc: added todo to mark docs when validation fails …
Nov 15, 2024
76a3365
feat/opg-scanning-poc: lp1f validation changes for parseDate
Nov 15, 2024
834a24a
feat/opg-scanning-poc: added client.go and API handler changes to han…
Nov 15, 2024
ba26b41
feat/opg-scanning-poc: clean-up comments and provided minor fixes bas…
Nov 15, 2024
6645984
fix UID field name
gregtyler Nov 15, 2024
bf18172
Remove terraform from workflow
gregtyler Nov 15, 2024
4c32985
Fix timeout error message
gregtyler Nov 15, 2024
6706d44
Set default values on config
gregtyler Nov 15, 2024
151f688
Finish removing terraform from workflow
gregtyler Nov 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions service-app/internal/api/client.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
package api

import (
"bytes"
"encoding/json"
"fmt"
"net/http"
"time"

"github.com/ministryofjustice/opg-scanning/internal/types"
)

func CreateStubCase(url string, set types.BaseSet) (*types.ScannedCaseResponse, error) {
var scannedCaseRequest types.ScannedCaseRequest
now := time.Now().Format(time.RFC3339)

if set.Header.CaseNo == "" {
// Check for LPA cases
for _, doc := range set.Body.Documents {
if doc.Type == "LPA002" || doc.Type == "LP1F" || doc.Type == "LP1H" || doc.Type == "LP2" {
// Create a new LPA case
scannedCaseRequest = types.ScannedCaseRequest{
BatchID: set.Header.Schedule,
CaseType: "LPA",
gregtyler marked this conversation as resolved.
Show resolved Hide resolved
ReceiptDate: set.Header.ScanTime,
CreatedDate: now,
}
break
} else if doc.Type == "EP2PG" || doc.Type == "EPA" {
// Create a new EPA case
scannedCaseRequest = types.ScannedCaseRequest{
BatchID: set.Header.Schedule,
CaseType: "epa",
ReceiptDate: set.Header.ScanTime,
CreatedDate: now,
}
break
}
}
} else if set.Header.CaseNo != "" {
// Check for COPORD case with CaseNo
for _, doc := range set.Body.Documents {
if doc.Type == "COPORD" {
scannedCaseRequest = types.ScannedCaseRequest{
CourtReference: set.Header.CaseNo,
BatchID: set.Header.Schedule,
CaseType: "order",
ReceiptDate: set.Header.ScanTime,
}
// Trigger events as per Order::createFromDdc (requires additional implementation)
gregtyler marked this conversation as resolved.
Show resolved Hide resolved
break
}
}
}

return requestCreateScannedCase(url, scannedCaseRequest)
}

func requestCreateScannedCase(url string, reqData types.ScannedCaseRequest) (*types.ScannedCaseResponse, error) {
body, err := json.Marshal(reqData)
if err != nil {
return nil, fmt.Errorf("failed to marshal request data: %w", err)
}

req, err := http.NewRequest("POST", url+"/scanned-cases", bytes.NewBuffer(body))
if err != nil {
return nil, fmt.Errorf("failed to create request: %w", err)
}
req.Header.Set("Content-Type", "application/json")

// Make the request (dummy for testing purposes)
// Placeholder for actual HTTP call
// resp, err := http.DefaultClient.Do(req)
// Instead return a dummy UUID for now
return &types.ScannedCaseResponse{UUID: "dummy-uuid-1234"}, nil
}
22 changes: 18 additions & 4 deletions service-app/internal/api/handler.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package api

import (
"encoding/json"
"fmt"
"io"
"net/http"
Expand Down Expand Up @@ -68,18 +69,31 @@ func (c *IndexController) IngestHandler(w http.ResponseWriter, r *http.Request)
return
}

// Step 4: Queue each document for further processing
// Step 4: Create a case stub in Sirius if we have a case to create
scannedCaseResponse, err := CreateStubCase(c.config.App.SiriusBaseURL, *parsedBaseXml)
if err != nil {
c.logger.Error("Failed to create case stub in Sirius: " + err.Error())
http.Error(w, "Failed to create case stub in Sirius", http.StatusInternalServerError)
return
}

// Step 5: Queue each document for further processing
c.logger.Info("Queueing documents for processing")
for i := range parsedBaseXml.Body.Documents {
doc := &parsedBaseXml.Body.Documents[i]
c.Queue.AddToQueue(doc, "xml", func() {
c.Queue.AddToQueue(doc, "xml", func(processedDocument interface{}) {
c.logger.Info(fmt.Sprintf("%v", processedDocument))
c.logger.Info("Job processing completed for document")
})
c.logger.Info("Job added to queue for document")
}

// Step 6: Send the UUID response
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusAccepted)
c.logger.Info("Ingestion request processed successfully")
json.NewEncoder(w).Encode(scannedCaseResponse)
c.logger.Info("Ingestion request processed successfully, UUID: " + scannedCaseResponse.UUID)

}

func (c *IndexController) CloseQueue() {
Expand All @@ -106,7 +120,7 @@ func (c *IndexController) validateAndSanitizeXML(bodyStr string) (*types.BaseSet

// Validate against XSD
c.logger.Info("Validating against XSD")
xsdValidator, err := ingestion.NewXSDValidator(c.config.ProjectFullPath+"/xsd/"+schemaLocation, bodyStr)
xsdValidator, err := ingestion.NewXSDValidator(c.config.App.ProjectFullPath+"/xsd/"+schemaLocation, bodyStr)
if err != nil {
return nil, err
}
Expand Down
2 changes: 1 addition & 1 deletion service-app/internal/api/handler_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ func TestIngestHandler_SetValid(t *testing.T) {
controller.IngestHandler(w, req)

resp := w.Result()
if resp.StatusCode != http.StatusAccepted {
if resp.StatusCode != http.StatusAccepted { // Expecting 202 for async processing
t.Errorf("expected status %d; got %d", http.StatusAccepted, resp.StatusCode)
}
}
Expand Down
8 changes: 4 additions & 4 deletions service-app/internal/ingestion/job_queue.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import (
type Job struct {
Data *types.BaseDocument
format string
onComplete func()
onComplete func(interface{})
}

type JobQueue struct {
Expand All @@ -31,7 +31,7 @@ func NewJobQueue() *JobQueue {
return queue
}

func (q *JobQueue) AddToQueue(data *types.BaseDocument, format string, onComplete func()) {
func (q *JobQueue) AddToQueue(data *types.BaseDocument, format string, onComplete func(interface{})) {
job := Job{Data: data, format: format, onComplete: onComplete}
q.wg.Add(1)
q.Jobs <- job
Expand Down Expand Up @@ -62,14 +62,14 @@ func (q *JobQueue) StartWorkerPool(ctx context.Context, numWorkers int) {
}

// Process the document
_, err = processor.Process()
parsedDoc, err := processor.Process()
if err != nil {
q.logger.Error("Worker %d failed to process job: %v\n", workerID, err)
return
}

if job.onComplete != nil {
job.onComplete()
job.onComplete(parsedDoc)
}
}()

Expand Down
2 changes: 1 addition & 1 deletion service-app/internal/ingestion/job_queue_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ func TestJobQueue(t *testing.T) {
EmbeddedXML: xml,
}

queue.AddToQueue(doc, "xml", func() {
queue.AddToQueue(doc, "xml", func(processedDocument interface{}) {
atomic.AddInt32(&processedJobs, 1)
})
}
Expand Down
1 change: 1 addition & 0 deletions service-app/internal/ingestion/set_validator.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ func (v *Validator) ValidateSet(parsedSet *types.BaseSet) error {
if parsedSet.Header == nil {
return errors.New("missing required Header element")
}

// if parsedSet.Header.CaseNo == "" {
// return errors.New("missing CaseNo in Header")
// }
Expand Down
22 changes: 22 additions & 0 deletions service-app/internal/types/client.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package types

type ScannedCaseRequest struct {
BatchID string `json:"batchId"`
CaseType string `json:"caseType"`
CourtReference string `json:"courtReference,omitempty"`
ReceiptDate string `json:"receiptDate"`
CreatedDate string `json:"createdDate"`
}

type ScannedCaseResponse struct {
UUID string `json:"uuid"`
j1mb0b marked this conversation as resolved.
Show resolved Hide resolved
}

// For handling error responses according to the OpenAPI spec
type ErrorResponse struct {
Type string `json:"type"`
Title string `json:"title"`
Status string `json:"status"`
Detail string `json:"detail"`
ValidationErrors map[string]string `json:"validation_errors"`
}
Loading