Skip to content

Commit

Permalink
R 2.2.3 (#151)
Browse files Browse the repository at this point in the history 
* Etrips  fix (#73)

* apk updates

* version update

* apk name change

* Etrips  fix (#74)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* Etrips  fix (#75)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* plist fix

* Etrips  fix (#76)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* plist fix

* change cert

* Develop (#77)

* Etrips  fix (#73)

* apk updates

* version update

* apk name change

* Etrips  fix (#74)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* Etrips  fix (#75)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* plist fix

* Etrips  fix (#76)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* plist fix

* change cert

* Various fixes (#82)

* FIX: trip locations modified

* remove print statement

* allow numbers in the column headers for workplans

* partner_portal fix

* add picture upload api again

* add permission classes

* Develop (#83)

* Develop (#65) (#66)

* changed deprecated name for the number (#37)

#118468621

* update build process (#38)

* update build process

[DO NOT MERGE]

* build updates

* small tweaks

* small tweaks

* get around permissions

* update gitignore

* display spinner when waiting for map location search results (#27)

* display spinner when waiting for map location search results

* simplify loading spinner

* remove unneeded Icon file

* fix government ntervention implementation status

[Finishes #105550212]

* update Android / iOS builds (#41)

* set agreement start end dates based on signed dates and result structure (#34)

* set agreement start end dates based on signed dates and result structure

* move start end date auto population from model to form

* updated error messages

* added agreement form tests

* added start date validation and test

* fix-government-in-simple WIP

[WIP] [DO NOT MERGE]

* migration fixes

* fix-circle-ci-build (#50)

* fix-circle-ci-build

updated library fail

* correct spelling error

* small updates

* Added etools/github contribution best practices guide (#40)

* Added etools/github contribution best practices guide

A document introducing new developers to the eTools project workflow,
and best practices when contributing to the GitHub.

* docs(CONTRIBUTING): change file name

* docs(guide): add commit guidelines section 

Add section on committing message best practices
Add section headers dividing concept from practices

PR (#40)

* small tweaks

* HACT dashboard tweaks to show trips for government interventions

* small tweak

* Listing Trip's Action Points in email notification (#30)

* Fix small issues - BY Ali (#44)

* fix distribution items checkbox readonly

* Fix some issues

Result display name
Approve trip by supervisor
Do not send trip notification to PA if no budget owner

* Fix test sending  email to PA with budget owner

* tweaks

* migration add (#56)

* remove existing migration

* fix migration dependency

* gitingore update (#57)

* small tweaks

* add migration

IN DEV ROLL BACK TRIPS MIGRATIONS TO 16 AND RUN MIGRATIONS AGAIN

* ENJOY PERFORMANCE

Fix performance issue on compound queries

* fix locations latency

* planned cash transfers for non-gov

* planned visits for NGOs

* only show government partners if they have interventions

* fix unicode return

* formatting fix

* small tweaks

* refractor(templates/admin): change test environments to have more obvious headers (#53)

* refractor(templates/admin): change test environments to have more obvious headers

Remove "Testing Environment" top bar.
Change banner in localhost, etools-dev, and etools-staging to be red
and display "Local Testing Environment", "Develop Testing Environment",
and "Staging Testing Environment" on banner respectively.

* refractor(#header): obvious testing header for dashboard

Added change from 097f25c to etools dashboard

* docs(guide): new dev setup guide for linux/ubuntu (#54)

* feat(views) add outdated browser page

Add javascript in `base.html` to load `outdated_browser.html`
when browser is IE 10 or older (user-agent contains 'MSIE')

* changed cartodb SQL statement

* changed tiles layer to Esri as Mapquest has changed their api

* fix(views) store browser logos on server

Previously was fetching logo images from remote url's
Now images are stored on server and fetched from there.

* addded comment

* build-fix (#64)

* build-fix

* remove sudo

* add npm

* update

* two run commands

* tweaks

* remove sudo

* remove sudo

* update

* update

* small update

* SMALL TWEAK

* small tweak

* Etrips  fix (#73)

* apk updates

* version update

* apk name change

* Etrips  fix (#74)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* Etrips  fix (#75)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* plist fix

* Etrips  fix (#76)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* plist fix

* change cert

* Develop (#77) (#78)

* Etrips  fix (#73)

* apk updates

* version update

* apk name change

* Etrips  fix (#74)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* Etrips  fix (#75)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* plist fix

* Etrips  fix (#76)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* plist fix

* change cert

* Various fixes (#82)

* FIX: trip locations modified

* remove print statement

* allow numbers in the column headers for workplans

* partner_portal fix

* add picture upload api again

* add permission classes

* fix funding and ram sync (#87)

* Fix Funding Sync for Vision (#88)

* fix funding and ram sync

* fix funding and ram sync

* fix funding sync

* Fix ram sync (#89)

* Etrips  fix (#73)

* apk updates

* version update

* apk name change

* Etrips  fix (#74)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* Etrips  fix (#75)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* plist fix

* Etrips  fix (#76)

* apk updates

* version update

* apk name change

* temporary change of ipa link

* plist fix

* change cert

* Various fixes (#82)

* FIX: trip locations modified

* remove print statement

* allow numbers in the column headers for workplans

* partner_portal fix

* add picture upload api again

* add permission classes

* fix funding and ram sync (#87)

* Fix Funding Sync for Vision (#88)

* fix funding and ram sync

* fix funding and ram sync

* fix funding sync

* celery flower

* small tweak

* celery flower (#91)

* celery flower

* small tweak

* worker long-running tasks setting

* broker visibility timeout in env (#96)

* Various fixes 002 (#97)

* broker visibility timeout in env

* no record

* updated country lat long decimal places and validation (#99)

* update .gitignore to ignore pmp_frontend (#98)

* Fix Agreement start end dates (#94)

* set agreement start end dates based on signed dates and result structure

* move start end date auto population from model to form

* updated error messages

* added agreement form tests

* added start date validation and test

* set end date for PCAs only

* set end date for PCAs only

* Various fixes 002 (#100)

* broker visibility timeout in env

* no record

* locations import fix

* added ability to hide partner records (#69)

* Fix development setup (#90)

* Fix development environment setup

* Move local dependencies to requirements/local.txt

* Add local_base settings so local deps can be removed when CI runs

* partners model changes

[MIGRATION NEEDED]

* vision sync

* expose total cash

* partner sync tweaks

* fake tasks

* fix dev deploy

* allow 30 seconds for time difference between adfs server and etools server

* change to 5 mins

* change to 5 mins

* import Decimal

* Develop (#108)

* Fix locations map 500 error (#107)

* list empty coordinates in locations

* took out extra code

* took out extra code

* took out extra code

* trip exports fix (#102)

* Develop (#111)

* Fix locations map 500 error (#107)

* list empty coordinates in locations

* took out extra code

* took out extra code

* took out extra code

* trip exports fix (#102)

* added 60secs to jwt auth leeway (#110)

* Revert "Develop" (#112)

* Develop (#116)

* Fix locations map 500 error (#107)

* list empty coordinates in locations

* took out extra code

* took out extra code

* took out extra code

* trip exports fix (#102)

* added 60secs to jwt auth leeway (#110)

* Feature results api (#109)

* Fix development environment setup

* Results Model and API changes

* Downgrade Factory boy back to the original 2.5.2, to fix build

* Use UserFactory for tagging users in tests to fix build

* Remove explicit ID definition and use  Meta fields instead

* Add IsAdminUser as permission for Milestone View

* Refactor sections field to M2M to users.models.Section

* Add IsAdminUser persmission class to ResultViewSet, fix permission class on MilestoneViewSet

* Remove unused import

* Remove unused import #2

* Change status labels

* Remove labels field

* Add missing migration file for removed labels field

* Utilize pre-loaded ResultTypes during test

* Refactor milestone field from O2O to M2O

* Resolve database migration conflicts

* remove all migrations that could conflict

* add new migrations

* fix export interventions (#115)

* fix broken conflict

* Develop (#119)

* Fix locations map 500 error (#107)

* list empty coordinates in locations

* took out extra code

* took out extra code

* took out extra code

* trip exports fix (#102)

* added 60secs to jwt auth leeway (#110)

* Feature results api (#109)

* Fix development environment setup

* Results Model and API changes

* Downgrade Factory boy back to the original 2.5.2, to fix build

* Use UserFactory for tagging users in tests to fix build

* Remove explicit ID definition and use  Meta fields instead

* Add IsAdminUser as permission for Milestone View

* Refactor sections field to M2M to users.models.Section

* Add IsAdminUser persmission class to ResultViewSet, fix permission class on MilestoneViewSet

* Remove unused import

* Remove unused import #2

* Change status labels

* Remove labels field

* Add missing migration file for removed labels field

* Utilize pre-loaded ResultTypes during test

* Refactor milestone field from O2O to M2O

* Resolve database migration conflicts

* remove all migrations that could conflict

* add new migrations

* fix export interventions (#115)

* fix vision sync (#118)

partners get total cash even if negative decimal

* add util_scripts

* Develop (#145)

* Fix locations map 500 error (#107)

* list empty coordinates in locations

* took out extra code

* took out extra code

* took out extra code

* trip exports fix (#102)

* added 60secs to jwt auth leeway (#110)

* Feature results api (#109)

* Fix development environment setup

* Results Model and API changes

* Downgrade Factory boy back to the original 2.5.2, to fix build

* Use UserFactory for tagging users in tests to fix build

* Remove explicit ID definition and use  Meta fields instead

* Add IsAdminUser as permission for Milestone View

* Refactor sections field to M2M to users.models.Section

* Add IsAdminUser persmission class to ResultViewSet, fix permission class on MilestoneViewSet

* Remove unused import

* Remove unused import #2

* Change status labels

* Remove labels field

* Add missing migration file for removed labels field

* Utilize pre-loaded ResultTypes during test

* Refactor milestone field from O2O to M2O

* Resolve database migration conflicts

* remove all migrations that could conflict

* add new migrations

* fix export interventions (#115)

* fix vision sync (#118)

partners get total cash even if negative decimal

* Add assumtions fields to Indicator model

* Allow partial updates on API (#121)

* feature add result type subactivity (#124)

* Add new ResultType: SubActivity

* Rename result type

* added 60secs to jwt auth leeway (#125)

good work ;)

* fix Result Model (#130)

* fix Result Model

* fix wbs null constraint on empty strings

* add migrations

* small tweaks

* other small tweaks

* final tweaks

* fix partner sync

* resolve conflict

* Fix - major vision sync improvements (#138)

* fix Result Model

* fix wbs null constraint on empty strings

* add migrations

* small tweaks

* other small tweaks

* final tweaks

* fix partner sync

* partner sync fixes

* Funding Synchronizer fixes

FCs do not duplicate anymore, FC line item expenditures are aggregated
across all records with the same FC ref

Performance improvements on FCs

* allow more funds to be connected to trips

* programme synchronizer performance and stability improvements

* add clean result type script

* skip bad wbs records

* removed cash transfer fields from admin form (#139)

renamed deleted flag

* wbs skipping activity and logging skips

* Fix result structure 001 nik (#140)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* remove fr number dependency on intervention

* remove tests

* Fix result structure 001 nik (#141)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* results cleanup of result structures
change to no cascade on foreign key to result structure

* remove result structure filter

* cleanup util scritps

* only remove duplicate ram indicators

* fix ram synchronizer to check only cropped versions of fields

* Migrations for result structure (#142)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* results cleanup of result structures
change to no cascade on foreign key to result structure

* cascade do nothing migrations

* update migration

* Fix result structure 001 nik (#143)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* results cleanup of result structures
change to no cascade on foreign key to result structure

* cascade do nothing migrations

* added mixin

* RAM results for inteventions

* removed  result_structure.name

* Ram indicator migrations (#144)

* fix Result Model

* fix wbs null constraint on empty strings

* add migrations

* small tweaks

* other small tweaks

* final tweaks

* fix partner sync

* partner sync fixes

* Funding Synchronizer fixes

FCs do not duplicate anymore, FC line item expenditures are aggregated
across all records with the same FC ref

Performance improvements on FCs

* allow more funds to be connected to trips

* programme synchronizer performance and stability improvements

* add clean result type script

* removed cash transfer fields from admin form
renamed deleted flag

* skip bad wbs records

* removed cash transfer fields from admin form (#139)

renamed deleted flag

* fix indicator import

* wbs skipping activity and logging skips

* Fix result structure 001 nik (#140)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* remove fr number dependency on intervention

* remove tests

* results cleanup of result structures
change to no cascade on foreign key to result structure

* Fix result structure 001 nik (#141)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* results cleanup of result structures
change to no cascade on foreign key to result structure

* remove result structure filter

* cleanup util scritps

* only remove duplicate ram indicators

* fix ram synchronizer to check only cropped versions of fields

* cascade do nothing migrations

* Migrations for result structure (#142)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* results cleanup of result structures
change to no cascade on foreign key to result structure

* cascade do nothing migrations

* update migration

* added mixin

* RAM results for inteventions

* removed  result_structure.name

* updated ram indicator

* Develop (#150)

* Fix locations map 500 error (#107)

* list empty coordinates in locations

* took out extra code

* took out extra code

* took out extra code

* trip exports fix (#102)

* added 60secs to jwt auth leeway (#110)

* Feature results api (#109)

* Fix development environment setup

* Results Model and API changes

* Downgrade Factory boy back to the original 2.5.2, to fix build

* Use UserFactory for tagging users in tests to fix build

* Remove explicit ID definition and use  Meta fields instead

* Add IsAdminUser as permission for Milestone View

* Refactor sections field to M2M to users.models.Section

* Add IsAdminUser persmission class to ResultViewSet, fix permission class on MilestoneViewSet

* Remove unused import

* Remove unused import #2

* Change status labels

* Remove labels field

* Add missing migration file for removed labels field

* Utilize pre-loaded ResultTypes during test

* Refactor milestone field from O2O to M2O

* Resolve database migration conflicts

* remove all migrations that could conflict

* add new migrations

* fix export interventions (#115)

* fix vision sync (#118)

partners get total cash even if negative decimal

* Add assumtions fields to Indicator model

* Allow partial updates on API (#121)

* feature add result type subactivity (#124)

* Add new ResultType: SubActivity

* Rename result type

* added 60secs to jwt auth leeway (#125)

good work ;)

* fix Result Model (#130)

* fix Result Model

* fix wbs null constraint on empty strings

* add migrations

* small tweaks

* other small tweaks

* final tweaks

* fix partner sync

* resolve conflict

* Fix - major vision sync improvements (#138)

* fix Result Model

* fix wbs null constraint on empty strings

* add migrations

* small tweaks

* other small tweaks

* final tweaks

* fix partner sync

* partner sync fixes

* Funding Synchronizer fixes

FCs do not duplicate anymore, FC line item expenditures are aggregated
across all records with the same FC ref

Performance improvements on FCs

* allow more funds to be connected to trips

* programme synchronizer performance and stability improvements

* add clean result type script

* skip bad wbs records

* removed cash transfer fields from admin form (#139)

renamed deleted flag

* wbs skipping activity and logging skips

* Fix result structure 001 nik (#140)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* remove fr number dependency on intervention

* remove tests

* Fix result structure 001 nik (#141)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* results cleanup of result structures
change to no cascade on foreign key to result structure

* remove result structure filter

* cleanup util scritps

* only remove duplicate ram indicators

* fix ram synchronizer to check only cropped versions of fields

* Migrations for result structure (#142)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* results cleanup of result structures
change to no cascade on foreign key to result structure

* cascade do nothing migrations

* update migration

* Fix result structure 001 nik (#143)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* results cleanup of result structures
change to no cascade on foreign key to result structure

* cascade do nothing migrations

* added mixin

* RAM results for inteventions

* removed  result_structure.name

* Ram indicator migrations (#144)

* fix Result Model

* fix wbs null constraint on empty strings

* add migrations

* small tweaks

* other small tweaks

* final tweaks

* fix partner sync

* partner sync fixes

* Funding Synchronizer fixes

FCs do not duplicate anymore, FC line item expenditures are aggregated
across all records with the same FC ref

Performance improvements on FCs

* allow more funds to be connected to trips

* programme synchronizer performance and stability improvements

* add clean result type script

* removed cash transfer fields from admin form
renamed deleted flag

* skip bad wbs records

* removed cash transfer fields from admin form (#139)

renamed deleted flag

* fix indicator import

* wbs skipping activity and logging skips

* Fix result structure 001 nik (#140)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* remove fr number dependency on intervention

* remove tests

* results cleanup of result structures
change to no cascade on foreign key to result structure

* Fix result structure 001 nik (#141)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* results cleanup of result structures
change to no cascade on foreign key to result structure

* remove result structure filter

* cleanup util scritps

* only remove duplicate ram indicators

* fix ram synchronizer to check only cropped versions of fields

* cascade do nothing migrations

* Migrations for result structure (#142)

* removed cash transfer fields from admin form
renamed deleted flag

* fix indicator import

* results cleanup of result structures
change to no cascade on foreign key to result structure

* cascade do nothing migrations

* update migration

* added mixin

* RAM results for inteventions

* removed  result_structure.name

* updated ram indicator

* temporarily remove hact dashboard (#149)

* temporarily remove hact dashboard

remove dashboard until fixed

* add a print statement to keep the console alive
  • Loading branch information
@robertavram
robertavram authored Sep 16, 2016
1 parent 8ac8270 commit d1baae5
Show file tree
Hide file tree
Showing 30 changed files with 1,279 additions and 208 deletions.
2 changes: 1 addition & 1 deletion EquiTrack/EquiTrack/urls.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@
url(r'^partnerships', login_required(PartnershipsView.as_view()), name='partnerships_dashboard'),
url(r'^map/$', login_required(MapView.as_view()), name='map'),
url(r'^cmt/$', login_required(CmtDashboardView.as_view()), name='cmt'),
url(r'^hact/$', login_required(HACTDashboardView.as_view()), name='hact_dashboard'),
#url(r'^hact/$', login_required(HACTDashboardView.as_view()), name='hact_dashboard'),

url(r'^locations/', include('locations.urls')),
url(r'^management/', include('management.urls')),
Expand Down
382 changes: 382 additions & 0 deletions EquiTrack/EquiTrack/util_scripts.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,382 @@
from __future__ import print_function
from django.db import connection
from django.db.models import Count
import time
from datetime import datetime, timedelta
from users.models import Country
from reports.models import ResultType, Result, CountryProgramme, Indicator, ResultStructure
from partners.models import FundingCommitment

def printtf(*args):
print([arg for arg in args])
f = open('mylogs.txt','a')
print([arg for arg in args], file=f)
f.close()

def set_country(name):
connection.set_tenant(Country.objects.get(name=name))


def fix_duplicate_indicators(country_name):
if not country_name:
printtf("country name required /n")
set_country(country_name)
printtf("Fixing duplicate indicators for {}".format(country_name))
fattrs = ["ramindicator_set"]

def fix_indicator_code():
printtf('cleaning codes of indicators for country ', country_name)
indicators = Indicator.objects.filter(code__exact='').all()
for ind in indicators:
ind.code = None
ind.save()
time.sleep(3)
fix_indicator_code()

def relates_to_anything(cobj):
for a in fattrs:
if getattr(cobj, a).count():
printtf(cobj.id, cobj, "relates to ", a)
return True
return False
def update_relationships(dpres, keep):
for a in fattrs:
objs = getattr(dpres, a).all()
if len(objs):
for obj in objs:
obj.indicator = keep
obj.save()
printtf("saved obj.id={} obj {} with keepid{} keep {}".format(obj.id, obj, keep.id, keep))

def _run_clean(dupes):
printtf(len(dupes), dupes)
for dup in dupes:
dupresults = Indicator.objects.filter(code=dup['code'], result=dup['result']).all()
delq = []
keep = None
for dpres in dupresults:
if not keep:
keep = dpres
continue
else:
error = False
if relates_to_anything(dpres):
try:
update_relationships(dpres, keep)
except Exception as exp:
printtf('Cannot remove Object {}, id={}'.format(dpres, dpres.id))
error = True
if error:
printtf("ERROR OCCURED ON RECORD", dpres.id, dpres)
continue
delq.append(dpres)
if not len(delq):
printtf("Nothing is getting removed for {}".format(dupes))
else:
# delete everyting in the queue
[i.delete() for i in delq]
printtf("deleting: ", delq)

dupes = Indicator.objects.values('code', 'result').order_by('code', 'result').annotate(Count('pk')).filter(pk__count__gt=1, ram_indicator=True).all()
_run_clean(dupes)

def fix_duplicate_results(country_name):

if not country_name:
printtf("country name required /n")
set_country(country_name)
printtf("Fixing duplicate Results for {}".format(country_name))
# foreign attributes
fattrs = ["governmentinterventionresult_set",
"indicator_set",
"ramindicator_set",
"resultchain_set",
"tripfunds_set"]
fattrs_mapping = {
"governmentinterventionresult_set": "result",
"indicator_set" : "result",
"ramindicator_set": "result",
"resultchain_set": "result",
"tripfunds_set": "wbs"
}
def fix_string_wbs():
results = Result.objects.filter(wbs__exact='').all()
for res in results:
res.wbs = None
res.save()
fix_string_wbs()

def reparent_children(current_object, new_parent):
for child in current_object.get_children():
child.parent = new_parent
printtf( "reparenting child", child.id, child, new_parent.id, new_parent)
child.save()
def relates_to_anything(cobj):
for a in fattrs:
if getattr(cobj, a).count():
printtf(cobj.id, cobj, "relates to ", a)
return True
return False
def update_relationships(dpres, keep):
for a in fattrs:
objs = getattr(dpres, a).all()
if len(objs):
for obj in objs:
if a == "tripfunds_set":
obj.wbs = keep
else:
obj.result = keep
obj.save()
printtf("saved obj.id={} obj {} with keepid{} keep {}".format(obj.id, obj, keep.id, keep))
def _run_clean(prop, dupes):
printtf(len(dupes), dupes)
search_string = prop + '__exact'
for dup in dupes:
dupresults = Result.objects.prefetch_related('result_type').filter(**{search_string: dup[prop]}).all()
delq = []
keep = None
for dpres in dupresults:
if not keep:
keep = dpres
continue
else:
error = False
if dpres.get_children():
try:
reparent_children(dpres, keep)
except Exception as exp:
printtf('Cannot reparent from Object {}, id={}'.format(dpres, dpres.id))
error = True
if relates_to_anything(dpres):
try:
update_relationships(dpres, keep)
except Exception as exp:
printtf('Cannot remove Object {}, id={}'.format(dpres, dpres.id))
error = True
if error:
printtf("ERROR OCCURED ON RECORD", dpres.id, dpres)
continue
delq.append(dpres)
time.sleep(0.3)
if not len(delq):
printtf("Nothing is getting removed for {}".format(dupes))
else:
# delete everyting in the queue
[i.delete() for i in delq]
printtf("deleting: ", delq)
# get all duplicates that have the same wbs
dupes = Result.objects.values('wbs').annotate(Count('wbs')).order_by().filter(wbs__count__gt=1, wbs__isnull=False).exclude(wbs__exact='').all()
dupes = sorted(dupes, key=lambda x: x['wbs'].count('/'), reverse=True)
_run_clean('wbs', dupes)

def cp_fix(country_name):
if not country_name:
printtf("country name required /n")
set_country(country_name)
printtf("Fixing Country Programme for {}".format(country_name))
def get_cpwbs(wbs):
grp = wbs.split('/')
return '/'.join(grp[:3])

results = Result.objects.filter(wbs__isnull=False).exclude(wbs__exact='')
locpwbs = []
for res in results:
cpwbs = get_cpwbs(res.wbs)
if cpwbs not in locpwbs:
locpwbs.append(cpwbs)

today = datetime.now()
for i in range(len(locpwbs)):
today = today + timedelta(days=i)
tomorrow = today + timedelta(days=365)
cp, created = CountryProgramme.objects.get_or_create(wbs=locpwbs[i], name='Country Programme '+str(i), from_date=today, to_date=tomorrow)

time.sleep(5)

for res in results:
cp = CountryProgramme.objects.get(wbs=get_cpwbs(res.wbs))
res.country_programme = cp
res.save()
print(res.name)

def clean_result_types(country_name):
if not country_name:
printtf("country name required /n")
set_country(country_name)
if not country_name:
printtf("country name required /n")

set_country(country_name)
printtf("Fixing duplicate Result Types for {}".format(country_name))
fattrs = ["result_set"]

def relates_to_anything(cobj):
for a in fattrs:
if getattr(cobj, a).count():
printtf(cobj.id, cobj, "relates to ", a)
return True
return False

def update_relationships(dpres, keep):
for a in fattrs:
objs = getattr(dpres, a).all()
if len(objs):
for obj in objs:
obj.result_type = keep
obj.save()
printtf("saved obj.id={} obj {} with keepid{} keep {}".format(obj.id, obj, keep.id, keep))

def _run_clean(dupes):
printtf(len(dupes), dupes)
for dup in dupes:
dupresults = ResultType.objects.filter(name=dup['name']).all()
delq = []
keep = None
for dpres in dupresults:
if not keep:
keep = dpres
continue
else:
error = False
if relates_to_anything(dpres):
try:
update_relationships(dpres, keep)
except Exception as exp:
printtf('Cannot remove Object {}, id={}'.format(dpres, dpres.id))
error = True
if error:
printtf("ERROR OCCURED ON RECORD", dpres.id, dpres)
continue
delq.append(dpres)
if not len(delq):
printtf("Nothing is getting removed for {}".format(dupes))
else:
# delete everyting in the queue
[i.delete() for i in delq]
printtf("deleting: ", delq)

# get all duplicates that have the same wbs
dupes = ResultType.objects.values('name').annotate(Count('name')).order_by().filter(name__count__gt=1).all()
_run_clean(dupes)

def clean_result_structures(country_name):
if not country_name:
printtf("country name required /n")
set_country(country_name)
if not country_name:
printtf("country name required /n")

set_country(country_name)
printtf("Fixing duplicate Result Structures for {}".format(country_name))
fattrs = ["result_set",
"indicator_set",
"goal_set",
"pca_set",
"governmentintervention_set", ]

def relates_to_anything(cobj):
for a in fattrs:
if getattr(cobj, a).count():
printtf(cobj.id, cobj, "relates to ", a)
return True
return False

def update_relationships(dpres, keep):
for a in fattrs:
objs = getattr(dpres, a).all()
if len(objs):
for obj in objs:
obj.result_structure = keep
obj.save()
printtf("saved obj.id={} obj {} with keepid{} keep {}".format(obj.id, obj, keep.id, keep))

def _run_clean(dupes):
printtf(len(dupes), dupes)
for dup in dupes:
dupresults = ResultStructure.objects.filter(name=dup['name']).all()
delq = []
keep = None
for dpres in dupresults:
if not keep:
keep = dpres
continue
else:
error = False
if relates_to_anything(dpres):
try:
update_relationships(dpres, keep)
except Exception as exp:
printtf('Cannot remove Object {}, id={}'.format(dpres, dpres.id))
error = True
if error:
printtf("ERROR OCCURED ON RECORD", dpres.id, dpres)
continue
delq.append(dpres)
if not len(delq):
printtf("Nothing is getting removed for {}".format(dupes))
else:
# delete everyting in the queue
[i.delete() for i in delq]
printtf("deleting: ", delq)

# get all duplicates that have the same name
dupes = ResultStructure.objects.values('name', 'from_date', 'to_date').order_by('name', 'from_date', 'to_date').annotate(Count('pk')).filter(pk__count__gt=1).all()
_run_clean(dupes)


def delete_all_fcs(country_name):
if not country_name:
printtf("country name required /n")
set_country(country_name)
printtf("Deleting all FCs for {}".format(country_name))
fcs = FundingCommitment.objects.all()
fcs.delete()

def dissasociate_result_structures(country_name):
if not country_name:
printtf("country name required /n")
set_country(country_name)
printtf("Dissasociating result structures for {}".format(country_name))
results = Result.objects.all()
for result in results:
if result.wbs and result.result_structure:
result.result_structure = None
result.save()


def all_countries_do(function, name):
for cntry in Country.objects.order_by('name').all():
if cntry.name in ['Global']:
continue
printtf("CALLING {} for all countries".format(name))
function(cntry.name)



def before_code_merge():
# Clean results
all_countries_do(fix_duplicate_results, 'Result Cleaning')

# Clean results structure
all_countries_do(clean_result_structures, 'Result Structure Cleaning')

# clean result types
all_countries_do(clean_result_types, 'Result Types Cleaning')

# Clean indicators
all_countries_do(fix_duplicate_indicators, 'Indicators Cleaning')

# Delete all fcs
all_countries_do(delete_all_fcs, 'Deleting FCs')

print('FINISHED WITH BEFORE MERGE')

def after_code_merge(): #and after migrations

# set up country programme
all_countries_do(cp_fix, 'Country Programme setup')

# disassociate result structures
all_countries_do(dissasociate_result_structures, 'Dissasociate Result Structure')

print("don't forget to sync")
Loading

0 comments on commit d1baae5

Please sign in to comment.