forked from DIRACGrid/DIRAC
-
Notifications
You must be signed in to change notification settings - Fork 0
/
release.notes
7516 lines (5471 loc) · 261 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v7r0-pre12]
*Accounting
CHANGE: (Multi)AccountingDB - Grouping Type and Object loader together with the MonitoringSystem ones.
*WorkloadManagementSystem
NEW: Add JobElasticDB.py with getJobParameters and setJobParameter methods to work with ElasticSearch (ES) backend.
NEW: Add gJobElasticDB variable and indicates the activation of ES backend.
CHANGE: Modify export_getJobParameter(s) to report values from ES if available.
CHANGE: (#3748) Reduced (removed, in fact) interactions of Optimizers with JobParameters, using only OptimizerParameters
NEW: (#3760) Add Client/JobStateUpdateClient.py
NEW: (#3760) Add Client/JobManagerClient.py
CHANGE: (#3760) Use the above Client classes instead of invoking RPCClient()
NEW: Added ES backend to WMSAdministratorHandler to get JobParameters.
NEW: Added separate MySQL table for JobsStatus in JobDB, and modified code accordingly.
CHANGE: ElasticJobDB.py: Modify setJobParameter method to register JobAttributes like Owner, Proxy, JobGroup etc.
CHANGE: ElasticJobDB.py: added getJobParametersAndAttributes method to retrieve both parameters and attributes for a given JobID.
CHANGE: Pilot Watchdog - using python UNIX services for some watchdog calls
CHANGE: (#3890) JobState always connects to DBs directly
CHANGE: (#3890) remove JobStateSync service
CHANGE: (#3873) Watchdog: use Profiler instead of ProcessMonitor
*Core
NEW: (#3744) Add update method to the ElasticSearchDB.py to update or if not available create the values
sent from the setJobParameter function. Uses update_by_query and index ES APIs.
CHANGE: (#3744) generateFullIndexName() method made static under the ElasticSearchDB class.
CHANGE: (#3744) removed unused/outdated stuff from Distribution module
FIX: check for empty /etc/grid-security/certificates dir
NEW: (#3842) Add createClient decorator to add wrapper for all export_ functions automatically
NEW: (#3678) dirac-install can install a non released code directly from the git repository
FIX: (#3931) AuthManager - modified to work with the case of unregistered DN in credDict
*ProductionManagement
NEW: (#3703) ProductionManagement system is introduced
*tests
NEW: Add ChangeESFlag.py script to modify useES flag in dirac.cfg. To be integrated with Jenkins code.
CHANGE: (#3760) Use the above Client classes instead of invoking RPCClient()
NEW: (#3744) Added performance tests for ES and MySQL for WMS DB backends
NEW: (#3744) Added miniInstallDIRAC function for Jenkins jobs
*Docs
FIX: Better dirac.cfg example configuration for web
NEW: (#3744) Add WMS documentation in DeveloperGuide/Systems
NEW: Added script (docs/Tools/UpdateDiracCFG.py) to collate the ConfigTemplate.cfg
files into the main dirac.cfg file
[v6r21p1]
*WorkloadManagementSystem
CHANGE: (#3989) JobDB.py - do not add default SubmitPool parameter to a job description
FIX: (#3989) dirac-admin-get-site-mask - show only sites in Active state
*DataManagementSystem
CHANGE: (#3985) FTS3DB: getActiveJobs, those jobs are now selected that have been monitored the longest time ago. Ensure better cycling through FTS Jobs
FIX: (#3987) check missing file with another string
[v6r21]
*Core
NEW: (#3921) DictCache - allow threadLocal cache
FIX: (#3936) DictCache - Fix exception upon delete
FIX: (#3922) allow Script.py to accommodate specific test calls with pytest
CHANGE: (#3940) dirac-install - instrument to support DiracOS
FIX: (#3945) set DIRACOS environment variable before souring diracosrc
CHANGE: (#3949) Removed unattended dirac-install-client.py
CHANGE: (#3950) File.py - do not follow links when getting files list or size
in directory via getGlobbedFiles and getGlobbedTotalSize
CHANGE: (#3969) Use EOS for installing DIRAC software
*FrameworkSystem
FIX: (#3968) removed the old logging
*ResourceStatusSystem
FIX: (#3921) fix logic of the RSSCache leading in expired keys
*Accounting
CHANGE: (#3933) Change the columns size of the FinalMinorStatus
*WorkloadManagementSystem
CHANGE: (#3923) Clean PYTHONPATH from *rc when installing DIRAC from the pilot
NEW: (#3941) JobDB: getJobParameters work also with list on job IDs
CHANGE: (#3941) JobCleaningAgent queries for job parameters in bulk
CHANGE: (#3941) Optimizers only set optimizers parameters (backported from v7r0)
CHANGE: (#3970) streamlining code in OptimizerModule. Also pep8 formatting (ignore white spaces for reviewing)
FIX: (#3976) fixed Banned Sites matching in TaskQueueDB
FIX: (#3970) when an optimizer agent was instantiating JobDB (via the base class) and the machine
was overloaded, the connection to the DB failed but this was not noticed and the agent was
not working until restarted after max cycles. Now testing JobDB is valid in OptimizerModule
base class and exit if not valid.
*TransformationSystem
CHANGE: (#3946) Remove directory listing from ValidateOutputDataAgent
CHANGE: (#3946) Remove directory listing from TransformationCleaningAgent
FIX: (#3967) TransformationCleaningAgent: don't return error if log directory does not exist
*Interfaces
CHANGE: (#3947) removed old methods going through old RMS
CHANGE: (#3960) Dirac.py - getLFNMetadata returns result for both file and directory LFNs
FIX: (#3966) Dirac: replace the use of deprecated function status by getJobStatus
*DataManagementSystem
FIX: (#3922) Fixes FTS3 duplicate transfers
FIX: (#3982) respect the source limitation when picking source for an FTS transfer
*MonitoringSystem
CHANGE: (#3956) Change the bucket size from week to day.
*Resources
CHANGE: (#3933) When crating a consumer or producer then the error message must be
handled by the caller.
CHANGE: (#3937) MessageQueue log backends is now set to VERBOSE instead of DEBUG
NEW: (#3943) SSHComputingElement - added Preamble option to define a command to be
executed right before the batch system job submission command
NEW: (#3953) Added the possibility to add filters to log backends to refine the
output shown/stored
NEW: (#3953) Resources.LogFilters.ModuleFilter: Filter that allows one to set the
LogLevel for individual modules
NEW: (#3953) Resources.LogFilter.PatternFilter: Filter to select or reject log
output based on words
FIX: (#3959) PoolComputingElement - bug fix: initialize process pool if not yet
done in getCEStatus()
*test
CHANGE: (#3948) integration tests run with unittest now exit with exit code != 0 if failed
*docs
NEW: (#3974) Added HowTo section to the User Guide
[v6r20p26]
*WorkloadManagementSystem
FIX: (#3932) MutiProcessorSiteDirector: get platform is checkPLatform flag is true
*DataManagementSystem
FIX: (#3928) `FileCatalogClient` now properly forwards function docstrings through
`checkCatalogArguments` decorator, fixes #3927
CHANGE: (#3928) `Resources.Catalog.Utilities`: use functool_wraps in `checkCatalogArguments`
*TransformationSystem
CHANGE: (#3934) make the recursive removal of the log directory explicit in the TransformationCleaningAgent
[v6r20p25]
*Core
FIX: (#3909) DISET - always close the socket even in case of exception
*FrameworkSystem
FIX: (#3913) NotificationHandler - bugfixed: changed SMTPServer to SMTP
FIX: (#3914) add extjs6 support to the web compiler
*docs
NEW: (#3910) Added documentation on MultiProcessor jobs
*WorkloadManagementSystem
FIX: (#3910) TaskQueueDB - fixed strict matching with tags, plus extended the integration test
*DataManagementSystem
CHANGE: (#3917) FTS3: speedup by using subqueries for the Jobs table
*TransformationSystem
CHANGE: (#3916) use SE.isSameSE() method
*Resources
NEW: (#3916) Add isSameSE method to StorageElement which works for all protocols
[v6r20p24]
*WorkloadManagementSystem
FIX: (#3904) SiteDirector fixed case with TQs for 'ANY' site>>>>>>> rel-v6r20
[v6r20p23]
*TransformationSystem
NEW: (#3903) do not remove archive SEs when looking at closerSE
*CORE
NEW: (#3902) When the environment variable DIRAC_DEPRECATED_FAIL is set to a non-empty value,
the use of deprecated functions will raise a NotImplementError exception
*ConfigurationSystem
FIX: (#3903) ServiceInterface - fix exception when removing dead slave
*FrameworkSystem
FIX: (#3901) NotificationClient - bug fix
[v6r20p22]
*Core
FIX: (#3897) ObjectLoader returns DErrno code
FIX: (#3895) more debug messages in BaseClient
*ResourceStatusSystem
FIX: (#3895) fixed bug in dirac-rss-set-token script
*WorkloadManagementSystem
FIX: (#3897) SiteDirector: using checkPlatform flag everwhere needed
CHANGE: (#3894) Using JobStateUpdateClient instead of RPCClient to it
CHANGE: (#3894) Using JobManagerClient instead of RPCClient to it
[v6r20p20]
*Core
CHANGE: (#3885) Script.parseCommandLine: the called script is not necessarily the first in sys.argv
*ConfigurationSystem
CHANGE: (#3887) /Client/Helpers/Registry.py: Added search dirac user for ID and CA
*MonitoringSystem
FIX: (#3888) mqProducer field in MonitoringReporter can be set to None, and the comparison was broken.
It is fixed. Also some additional checks are added.
*WorkloadManagementSystem
CHANGE: (#3889) removed confusing Job parameter LocalBatchID
CHANGE: (#3854) TQ matching (TaskQueueDB.py): when "ANY" is specified, don't exclude task queues
(fix with "Platforms" matching in mind)
CHANGE: (#3854) SiteDirector: split method getPlatforms, for extension purposes
*DataManagementSystem
FIX: (#3884) restore correct default value for the SEPrefix in the FileCatalogClient
FIX: (#3886) FTS3: remove the hardcoded srm protocol for registration
FIX: (#3886) FTS3: return an empty spacetoken if SRM is not available
*TransformationSystem
CHANGE: (#3891) ReplicationTransformation.createDataTransformation: returns S_OK with the
transformation object when it was successfully added, instead of S_OK(None)
*Resources
NEW: (#3886) SE - return a standard error in case the requested protocol is not available
[v6r20p18]
*DataManagementSystem
CHANGE: (#3882) script for allow/ban SEs now accepting -a/--All switch, for allo status types
*Core
FIX: (#3882) ClassAdLight - fix to avoid returning a list with empty string
*Resources
FIX: (#3882) Add site name configuration for the dirac installation inside singularity CE
*test
FIX: (#3882) fully activating RSS in Jenkins tests
[v6r20p17]
*Core
CHANGE: (#3874) dirac-create-distribution-tarball - add tests directory to the tar file and fix pylint warnings.
FIX: (#3876) Add function "discoverInterfaces" again which is still needed for VMDIRAC
*ConfigurationSystem
CHANGE: (#3875) Resources - allow to pass a list of platforms to getDIRACPlatform()
*WorkloadManagement
CHANGE: (#3867) SandboxStoreClient - Returning file location in output of getOutputSandbox
CHANGE: (#3875) JobDB - allow to define a list of Platforms in a job description JDL
*ResourceStatusSystem
CHANGE: (#3863) deprecated CSHelpers.getSites() function
*Interfaces
NEW: (#3872) Add protocol option to dirac-dms-lfn-accessURL
CHANGE: (#3864) marked deprecated some API functions (perfect replace exists already, as specified)
*Resources
FIX: (#3868) GFAL2_SRM2Storage: only set SPACETOKENDESC when SpaceToken is not an empty string
*Test
CHANGE: (#3863) Enable RSS in Jenkins
*DataManagementSystem
FIX: (#3859) FTS3: resubmit files in status Canceled on the FTS server
NEW: (#3871) FTS submissions can use any third party protocol
NEW: (#3871) Storage plugin for Echo (gsiftp+root)
FIX: (#3871) replace deprecated calls to the gfal2 API
NEW: (#3871) Generic implementation for retrieving space occupancy on storage
*TransformationSystem
FIX: (#3865) fixed submission of parametric jobs with InputData from WorkflowTask
FIX: (#3865) better logging for parametric jobs submission
*StorageManagamentSystem
FIX: (#3868) Fix StageRequestAgent failures for SEs without a SpaceToken
*RequestManagementSystem
FIX: (#3861) tests do not re-use File objects
[v6r20p16]
*WorkloadManagementSystem
CHANGE: (#3850) the platform discovery can be VO-specific.
*Interfaces
CHANGE: (#3856) setParameterSequence always return S_OK/S_ERROR
*TransformationSystem
FIX: (#3856) check for return value on Job interface and handle it
*ResourceStatusSystem
FIX: (#3852) site may not have any SE
[v6r20p15]
*Interface
FIX: (#3843) Fix the sandbox download, returning the inMemory default.
*WorkloadManagementSystem
FIX: (#3845) late creation on RPC in JobMonitoringClient and PilotsLoggingClient
*DataManagementSystem
FIX: (#3839) Update obsolete dirac-rms-show-request command in user message displayed when running dirac-dms-replicate-and-register-request
*FrameworkSystem
FIX: (#3845) added setServer for NotificationClient
*Docs
NEW: (#3847) Added some info on parametric jobs
[v6r20p14]
CHANGE: (#3826) emacs backup file pattern added to .gitignore
*MonitoringSystem
CHANGE: (#3827) The default name of the Message Queue can be changed
*Core
FIX: (#3832) VOMSService.py: better logging and error prevention
*ConfigurationSystem
FIX: (#3837) Corrected configuration location for Pilot 3 files synchronization
*FrameworkSystem
FIX: (#3830) InstalledComponentDB.__filterFields: fix error in "Component History Web App" when filter values are unicode
*Interface
CHANGE: (#3836) Dirac.py API - make the unpacking of downloaded sandboxes optional
*Accounting
CHANGE: (#3831) ReportGenerator: Authenticated users without JOB_SHARING will now only get plots showing their own jobs, solves #3776
*ResourceStatusSystem
FIX: (#3833) Documentation update
CHANGE: (#3838) For some info, use DMSHelper instead of CSHelper for better precision
*RequestManagementSystem
FIX: (#3829) catch more exception in the ReqClient when trying to display the associated FTS jobs
[v6r20p13]
*FrameworkSystem
FIX: (#3822) obsolete parameter maxQueueSize in UserProfileDB initialization removed
*WorkloadManagementSystem
FIX: (#3824) Added Parameter "Queue" to methods invoked on batch systems by LocalComputingElement
FIX: (#3818) Testing parametric jobs locally now should also work for parametric input data
NEW: (#3818) Parameters from Parametric jobs are also replaced for ModuleParameters,
and not only for common workflow parameters
*DataManagementSystem
FIX: (#3825) FileCatalogCLI: print error message when removeReplica encounters weird return value
FIX: (#3819) ReplicateAndRegister: fix a problem when transferring files to multiple storage
elements, if more than one attempt was needed the transfer to all SEs was not always
happening.
CHANGE: (#3821) FTS3Agent: set pool_size of the FTS3DB
*TransformationSystem
FIX: (#3820) Fix exception in TransformationCleaningAgent: "'str' object not callable"
*ConfigurationSystem
FIX: (#3816) The VOMS2CSAgent was not sending notification emails when the DetailedReport
option was set to False, it will now send emails again when things change for a VO.
CHANGE: (#3816) VOMS2CSAgent: Users to be checked for deletion are now printed sorted and line
by line
NEW: (#3817) dirac-admin-check-config-options script to compare options and values between
the current Configuration and the ConfigTemplates. Allows one to find wrong or
missing option names or just see the difference between the current settings and
the default values.
[v6r20p12]
*Core
FIX: (#3807) Glue2 will return a constant 2500 for the SI00 queue parameter,
any value is needed so that the SiteDirector does not ignore the queue, fixes #3790
*ConfigurationSystem
FIX: (#3797) VOMS2CSAgent: return error when VO is not set (instead of exception)
FIX: (#3797) BDII2CSAgent: Fix for GLUE2URLs option in ConfigTemplate (Lower case S at the end)
*DataManagementSystem
FIX: (#3814) SEManager - adapt to the new meaning of the SE plugin section name
FIX: (#3814) SEManager - return also VO specific prefixes for the getReplicas() and similar calls
FIX: (#3814) FileCatalogClient - take into account VO specific prefixes when constructing PFNs on the fly
*TransformationSystem
FIX: (#3812) checking return value of jobManagerClient.getMaxParametricJobs() call
[v6r20p11]
*Core
FIX: (#3805) ElasticSearchDB - fix a typo (itertems -> iteritems())
[v6r20p10]
*Core
NEW: (#3801) ElasticSearchDB - add method which allows for deletion by query
NEW: (#3792) added breakDictionaryIntoChunks utility
*WorkloadManagementSystem
FIX: (#3796) Removed legacy "SystemConfig" and "LHCbPlatform" checks
FIX: (#3803) bug fix: missing loop on pRef in SiteDirector
NEW: (#3792) JobManager exposes a call to get the maxParametricJobs
*TransformationSystem
NEW: (#3804) new option for dirac-transformation-replication scrip `--GroupName/-R`
FIX: (#3804) The TransformationGroup is now properly set for transformation created with dirac-transformation-replication, previously a transformation parameter Group was created instead.
FIX: (#3792) Adding JobType as parameter to parametric jobs
FIX: (#3792) WorkflowTaskAgent is submitting a chunk of tasks not exceeding the MaxParametricJobs accepted by JobManager
[v6r20p9]
*Core
FIX: (#3794) Fix executeWithUserProxy when called with proxyUserDN,
fixes exception in WMSAdministrator getPilotLoggingInfo and TransformationCleaningAgent
*DataManagementSystem
CHANGE: (#3793) reuse of the ThreadPool in the FTS3Agent in order to optimize the Context use
*WorkloadManagementSystem
FIX: (#3787) Better and simpler code and test for SiteDirector
FIX: (#3791) Fix exception in TaskQueueDB.getActiveTaskQueues, triggered
by dirac-admin-show-task-queues
[v6r20p8]
*ResourceStatusSystem
FIX: (#3782) try/except for OperationalError for sqlite (EmailAction)
*Core
FIX: (#3785) Adjust voms-proxy-init timeouts
NEW: (#3773) New Core.Utilities.Proxy.UserProxy class to be used as a contextManager
FIX: (#3773) Fix race condition in Core.Utilities.Proxy.executeWithUserProxy,
the $X509_USER_PROXY environment variable from one thread could leak to another, fixes #3764
*ConfigurationSystem
NEW: (#3784) Bdii2CSAgent: New option **SelectedSites**, if any sites are set, only those will
be updated
NEW: (#3788) for CS/Registry section: added possibility to define a QuarantineGroup per VO
*WorkloadManagementSystem
FIX: (#3786) StalledJobAgent: fix "Proxy not found" error when sending kill command to stalled job,
fixes #3783
FIX: (#3773) The solution for bug #3764 fixes a problem with the JobScheduling executor, where
files could end up in the checking state with the error "Couldn't get storage metadata
of some files"
FIX: (#3779) Add setting of X509_USER_PROXY in pilot wrapper script,
which is needed to establish pilot env in work nodes of Cluster sites.
*DataManagementSystem
FIX: (#3778) Added template for RegisterReplica
FIX: (#3772) add a protection against race condition between RMS and FTS3
FIX: (#3774) Fix FTS3 multi-VO support by setting VO name in SE constructor.
*TransformationSystem
FIX: (#3789) better tests for TS agents
*StorageManagamentSystem
FIX: (#3773) Fix setting of the user proxy for StorageElement.getFileMetadata calls, fixes #3764
[v6r20p7]
*Core
FIX: (#3768) The Glue2 parsing handles some common issues more gracefully:
handle cases where the execution environment just does not exist, use sensible;
dummy values in this case (many sites);
handle multiple execution environments at a single computing share (i.e., CERN);
handle multiple execution environments with the same ID (e.g., SARA)
CHANGE: (#3768) some print outs are prefixed with "SCHEMA PROBLEM", which seem to point to problems in the published information, i.e. keys pointing to non-existent entries, or non-unique IDs
*Tests
NEW: (#3769) allow to install DIRACOS if DIRACOSVER env variable is specified
*ResourceStatusSystem
CHANGE: (#3767) Added a post-processing function in InfoGetter, for handling special case of FreeDisk policies
*WorkloadManagementSystem
FIX: (#3767) corrected inconsistent option name for pilotFileServer CS option
*TransformationSystem
CHANGE: (#3766) TransformationCleaningAgent can now run without a shifterProxy, it uses
the author of the transformation for the cleanup actions instead.
CHANGE: (#3766) TransformationCleaningAgent: the default value for shifterProxy was removed
FIX: (#3766) TaskManagerAgent: RequestTasks/WorkflowTasks: value for useCertficates to `False`
instead of `None`. Fixes the broken submission when using a shifterProxy for the TaskManagerAgents
[v6r20p6]
*Tests
CHANGE: (#3757) generate self signed certificate TLS compliant
*Interfaces
FIX: (#3754) classmethods should not have self! (Dirac.py)
*WorkloadManagementSystem
FIX: (#3755) JobManager - bug fix in __deleteJob resulting in exceptions
*DataManagementSystem
NEW: (#3736) FTS3 add kicking of stuck jobs
FIX: (#3736) FTS3 update files in sequence to avoid mysql deadlock
CHANGE: (#3736) Canceled is not a final state for FTS3 Files
CHANGE: (#3736) FTS3Operations are finalized if the Request is in a final state (instead of Scheduled)
FIX: (#3724) change the ps_delete_files and ps_delete_replicas_from_file_ids to not lock on MySQL 5.7
*TransformationSystem
CHANGE: (#3758) re-written a large test as pytest (much less verbosity, plan to extend it)
FIX: (#3758) added BulkSubmission option in documentation for WorkflowTaskAgent
*RequestManagementSystem
FIX: (#3759) dirac-rms-request: silence a warning, when not using the old FTS Services
*ResourceStatusSystem
FIX: (#3753) - style changes
[v6r20p5]
*Docs
FIX: (#3747) fix many warnings
FIX: (#3735) GetReleaseNotes.py no longer depends on curl, but the python requests packe
FIX: (#3740) Fix fake environments for sqlalchemy.ext import, some code documentation pages were not build, e.g. FTS3Agent
NEW: (#3762) Add --repo option, e.g. --repo DiracGrid/DiracOS, or just --repo DiracOS, fixes DIRACGrid/DIRACOS#30
*TransformationSystem
FIX: (#3726) If the result can not be evaluated, it can be converted to list
FIX: (#3723) TaskManagerAgentBase - add option ShifterCredentials to set the credentials to
use for all submissions, this is single VO only
FIX: (#3723) WorkflowTasks/RequestTasks: pass ownerDN and ownerGroup parameter to all the submission
clients if using shifterProxy ownerDN and ownerGroup are None thus reproducing the original behaviour
FIX: (#3723) TaskManagerAgentBase - refactor adding operations for transformation to separate function to
ensure presence of Owner/DN/Group in dict entries RequestTaskAgent no longer sets shifterProxy by default.
*Resources
CHANGE: (#3745) Add the deprecated decorator to native XROOT plugin
[v6r20p4]
*DMS
FIX: (#3727) use proxy location in the SECache
*RMS
FIX: (#3727) use downloadVOMSProxyToFile in RequestTask
*TS
FIX: (#3720) TaskManager - pass output data arguments as lists rather
than strings to the parametric job description
Docs:
FIX: (#3725) AdministratorGuide TransformationSystem spell check and added a few
phrases, notably for bulk submission working in v6r20p3
[v6r20p3]
*Framework
FIX: SystemAdministrator - Get the correct cpu usage data for each component
*TS
NEW: new command dirac-transformation-replication to create replication transformation to copy files from some SEs to other SEs, resolves #3700
*RMS
FIX: fix integration tests to work with privileged and non privileged proxies
*RSS
FIX: Fix for downtime publisher: wrong column names. Avoiding dates (not reflected in web app)
[v6r20p2]
*Core
CHANGE: (#3713) Fixes the infamous "invalid action proposal" by speeding up the handshake and not looking up the user/group in the baseStub
*RequestManagementSystem
CHANGE: (#3713) FowardDISET uses the owner/group of Request to execute the stub
CHANGE: (#3713) owner/group of the Requests are evaluated/authorized on the server side
CHANGE: (#3713) LimitedDelegation or FullDelegation are required to set requests on behalf of others -> pilot user and hosts should must them (which should already be the case)
*docs
NEW: (#3699) documentation on Workflow
CHANGE: (#3699) update on documentation for integration tests
*ConfigurationSystem
CHANGE: (#3699) for pilotCS2JSONSynchronizer: if pilotFileServer is not set, still print out the content
*WorkloadManagementSystem
CHANGE: (#3693) introduce options for sites to choose usage of Singularity
*TransformationSystem
FIX: (#3706) TaskManger with bulksubmission might have occasional exception, depending on order of entries in a dictionary
FIX: (#3709) TaskManager - fix the generated JobName to be of the form ProdID_TaskID
FIX: (#3709) TaskManager - check the JOB_ID and PRODUCTION_ID parameters are defined in the workflow
*Interfaces
FIX: (#3709) Job API - do not merge workflow non-JDL parameters with the sequence parameters of the same name
[v6r20p1]
*WorkloadManagementSystem
FIX: (#3697) Ensure retrieveTaskQueues doesn't return anything when given an empty list of TQ IDs.
FIX: (#3698) Call optimizer fast-path for non-bulk jobs
[v6r20]
*Core
NEW: MJF utility added, providing a general interface to Machine/Job Features values.
NEW: DEncode - added unit tests
NEW: JEncode for json based serialization
NEW: Add conditional printout of the traceback when serializing/deserializing non json compatible
object in DEncode (enabled with DIRAC_DEBUG_DENCODE_CALLSTACK environment variable)
NEW: File.py - utility to convert file sizes between different unit
NEW: new flag in dirac-install script to install DIRAC-OS on demand
CHANGE: Removed deprecated option "ExtraModules" (dirac-configure, dirac-install scripts)
CHANGE: dirac-deploy-scripts, dirac-install - allow command modules with underscores in
their names in order for better support for the code checking tools
CHANGE: dirac-distribution and related scripts - compile web code while release
generation
CHANGE: dirac-external-requirements - reimplemented to use preinstalled pip command rather than
the pip python API
FIX: dirac-distribution - fixed wrong indentation
NEW: new command name for voms proxy
FIX: dirac-install default behaviour preserved even with diracos options
New: Add additional check in MJF utility to look for a shutdown file located at '/var/run/shutdown_time'
FIX: The hardcoded rule was not taken into account when the query was coming from the web server
CHANGE: VOMSService - reimplemented using VOMS REST interface
FIX: MJF utility won't throw exceptions when MJF is not fully deployed at a site
*Framework
NEW: WebAppCompiler methods is implemented, which is used to compile the web framework
NEW: add JsonFormatter for logs
NEW: add default configuration to CS: only TrustedHost can upload file
CHANGE: ComponentInstaller - remove the old web portal configuration data
used during the installation
CHANGE: MessageQueue log handler uses JsonFormatter
*Monitoring
CHANGE: fixes for testing in Jenkins with locally-deployed ElasticSearch
FIX: fixes in the query results interpretation
*Configuration
FIX: ConfigurationHandler, PilotCStoJSONSynchronizer - fixes for enabling pilotCStoJSONSynchronizer, and doc
NEW: dirac-admin-voms-sync - command line for VOMS to CS synchronization
NEW: VOMS2CSSynchronizer - new class encapsulating VOMS to CS synchronization
CHANGE: VOMS2CSAgent - reimplemented to use VOMS2CSSynchronizer
*WorkloadManagementSystem
NEW: StopSigRegex, StopSigStartSeconds, StopSigFinishSeconds, StopSigNumber added to JDL, which cause Watchdog to send a signal StopSigNumber to payload processes matching StopSigRegex when there are less than StopSigFinishSeconds of wall clock remaining according to MJF.
NEW: PilotLoggingDB, Service and Client for handling extended pilot logging
NEW: added a new synchronizer for Pilot3: sync of subset of CS info to JSON file,
and sync of pilot3 files
NEW: dirac-admin-get-pilotslogging script for viewing PilotsLogging
NEW: Bulk job submission with protection of the operation transaction
NEW: WMSHistoryCorrector and MonitoringHistoryCorrector classes inheriting from a common BaseHistoryCorrector class
CHANGE: SiteDirector - refactored Site Director for better extensibility
CHANGE: dirac-wms-cpu-normalization uses the abstracted DB12 benchmark script used by the HEPiX Benchmarking Working Group, and the new MJF utility to obtain values from the system and to save them into the DIRAC LocalSite configuration.
CHANGE: Removed TaskQueueDirector and the other old style (WMS) *PilotDirector
CHANGE: TaskQueueDB - removed PilotsRequirements table
CHANGE: TaskQueueDB - added FOREIGN KEYS
CHANGE: Removed gLite pilot related WMS code
FIX: always initialize gPilotAgentsDB object
FIX: JobManager - Added some debug message when deleting jobs
FIX: Job.py - fixing finding XML file
NEW: SiteDirector - added flag for sending pilot3 files
CHANGE: SiteDirector - changed the way we create the pilotWrapper (better extensibility)
NEW: SiteDirector - added possibility for deploying environment variables in the pilot wrapper
*Workflow
CHANGE: Script.py: created _exitWithError method for extension possibilities
*TS
FIX: TranformationCleaningAgent - just few simplifications
*DMS
NEW: FTS3Agent working only with the FTS3 service to replace the existing one
NEW: FTS3Utilities - use correct FTS Server Selection Policy
NEW: StorageElement service - getFreeDiskSpace() and getTotalDiskSpace() take into account
MAX_STORAGE_SIZE parameter value
CHANGE: Adding vo name argument for StorageElement
CHANGE: Fixing rss to fetch fts3 server status
NEW: Add a feature to the DFC LHCbManager to dump the content of an SE as a CSV file
FIX: FTS3DB: sqlalchemy filter statements with "is None" do not work and result in no lines being selected
NEW: FTS3Agent and FTS3DB: add functionality to kick stuck requests and delete old requests
NEW: FTS3Agent - add accounting report
*RMS
FIX: Really exit the RequestExecutingAgent when the result queue is buggy
*RSS
CHANGE: Using StorageElement.getOccupancy()
FIX: Initialize RPC to WMSAdministrator only once
FIX: Using MB as default for the size
FIX: flagged some commands that for the moment are unusable
FIX: fixed documentation of how to develop commands
*Resources
NEW: New SingularityComputingElement to submit jobs to a Singularity container
NEW: Added StorageElement.getOccupancy() method for DIP and GFAL2_SMR2 SE types
CHANGE: enable Stomp logging only if DIRAC_DEBUG_STOMP environment variable is set to any value
*Interfaces
CHANGE: Dirac.py - saving output of jobs run with 'runLocal' when they fail (for DEBUG purposes)
*Docs
CHANGE: WebApp release procedure
FIX: Update of the FTS3 docs
*Tests
FIX: add MonitoringDB to the configuration
FIX: Installing elasticSeach locally in Jenkins, with ComponentInstaller support.
[v6r19p25]
*TransformationSystem
FIX: (#3742) TransformationDB - when adding files to transformations with a multi-threaded agent,
it might happen that 2 threads are adding the same file at the same time. The LFN was not
unique in the DataFiles table, which was a mistake... This fix assumes the LFN is unique,
i.e. if not the table had been cleaned and the table updated to be unique.
[v6r19p24]
*WMS
FIX: (#3739) pilotTools - added --tag and --requiredTag options
FIX: (#3739) pilotCommands - make NumberOfProcessors = 1 if nowhere defined (default)
*Resources
FIX: (#3739) CREAMComputingElement - possibility to defined CEQueueName to be used in the pilot submission command
[v6r19p23]
*TS
FIX: (#3734) catch correct exception for ast.literal_eval
[v6r19p22]
*Core
CHANGE: Backport from v6r20 - fixes the infamous "invalid action proposal" by speeding up
the handshake and not looking up the user/group in the baseStub
RMS:
CHANGE: Backport from v6r20 - FowardDISET uses the owner/group of Request to execute the stub
CHANGE: Backport from v6r20 - owner/group of the Requests are evaluated/authorized on the server side
CHANGE: Backport from v6r20 - LimitedDelegation or FullDelegation are required to set requests on behalf
of others -> pilot user and hosts should must them (which should already be the case)
*API
NEW: Dirac.py - running jobs locally now also works for parametric jobs. Only the first sequence will be run
FIX: Dirac.py - running jobs locally will now properly work with LFNs in the inputSanbdox
*DMS
FIX: DMSHelpers - in getLocalSiteForSE() return None as LocalSite if an SE is at no site
[v6r19p21]
*Configuration
FIX: Bdii2CSAgent - make the GLUE2 information gathering less verbose; Silently ignore StorageShares
*Test
CHANGE: backported some of the CI tools from the integration branch
[v6r19p20]
*StorageManagement
FIX: StorageManagementDB - fixed buggy group by with MySQL 5.7
[v6r19p19]
*Configuration
NEW: BDII2CSAgent - new options: GLUE2URLs, if set this is queried in addition to the other BDII;
GLUE2Only to turn off looking on the old schema, if true only the main BDII URL is queried;
Host to set the BDII host to search
NEW: dirac-admin-add-resources new option G/glue2 , enable looking at GLUE2 Schema,
H/host to set the host URL to something else
[v6r19p18]
*Configuration
CHANGE: Better logging of the Configuration file write exception
*RSS
FIX: SummarizeLogsAgent - fix the case when no previous history
[v6r19p17]
*Framework
FIX: ProxyManager - if an extension has a ProxyDB, use it
*RSS
FIX: CSHelpers.py minor fixes
[v6r19p16]
*WMS
FIX: pilotCommands - cast maxNumOfProcs to an int.
CHANGE: pilotTools - change maxNumOfProcs short option from -P to -m.
[v6r19p15]
*Framework
NEW: ProxyDB - allow FROM address to be set for proxy expiry e-mails
*DMS
CHANGE: FTSJob - FailedSize is now BIGINT in FTSJob
CHANGE: FTSJob - increase the bringonline time
*WMS
FIX: SiteDirector won't set CPUTime of the pilot
FIX: convert MaxRAM inside the pilots to int
*RSS
FIX: SummarizeLogsAgent: comparison bug fix
FIX: Fixed sites synchronizer
[v6r19p14]
*WMS
NEW: pilotCommands/Tools - added possibility to specify a maxNumberOfProcessors parameter for pilots
CHANGE: MultiProcessorSiteDirector - allow kwargs to SiteDirector getExecutable & _getPilotOptions functions
*RMS
FIX: Fix a bug in ReplicateAndRegister Operation preventing files having failed once to be retried
*DMS
FIX: FileCatalogWithFkAndPsDB.sql - Fixes for the DFC to be compatible with strict group by mode
(https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html#sqlmode_only_full_group_by)
*docs
CHANGE: added little documentation for lcgBundles
[v6r19p13]
*WMS
FIX: JobWrapper - added a debug message
FIX: Allow non-processor related tags to match TQ in MultiProcessorSiteDirector.
*Test
CHANGE: improve Gfal2 integration tests by checking the metadata
[v6r19p12]
*Core
CHANGE: QualityMapGraph - change the color map of the Quality plots
*Framework
FIX: Logging - remove the space after log messages if no variable message is printed, fixes #3587
*MonitoringSystem
CHANGE: ElasticSearch 6 does not support multiple types, only one type is created instead.
*RSS
FIX: GOCDBClient - encode in utf-8, update goc db web api URL
FIX: fixed bug in creation of history of status (avoid repetition of entries)
*DMS
FIX: fixed bug in FTSAgent initialization
*WMS
FIX: fix bug in dirac-wms-job-select: treating the case of jobGroup(s) not requested
[v6r19p11]
*Framework:
CHANGE: moved column "Instance" of InstalledComponentsDB.InstalledComponent
table from 64 to 32 characters
*WMS
FIX: JobWrapperTemplate - fix exception handling
CHANGE: dirac-wms-select-jobs - new option to limit the number of selected jobs
CHANGE: returning an error when sandboxes can't be unassigned from jobs (JobCleaningAgent)
*RMS
FIX: RequestDB - add missing JOIN in the web summary query
NEW: dirac-rms-request - add option to allow resetting the NotBefore member even
for non-failed requests
*DMS
FIX: FTSAgent - change data member names from uppercase to lower case
*Interfaces
CHANGE: autopep8 on the API/Dirac module
*docs:
NEW: added some doc about shifterProxy
[v6r19p10]
*Core
FIX: MySQL - catch exception when closing closed connection
*TS
CHANGE: add possibility to get extension-specific tasks and files statuses in TransformationMonitor web application
*RMS
NEW: dirac-rms-request - add option --ListJobs to list the jobs for a set of requests
*Resources
FIX: Use parameters given at construction for SRM2 protocols List
*StorageManagement
FIX: use StorageElement object to get disk cache size
*DMS
FIX: DMSHelpers - fix case when no site is found for an SE
FIX: ReplicateAndRegister - don't try and get SE metadata is replica is inactive
[v6r19p9]
*WMS
CHANGE: DownloadInputData was instantiating all local SEs which is not necessary... Only instantiate those that are needed
CHANGE: JobWrapper - use resolveSEGroup in order to allow defining SE groups including other SE groups
FIX: JobDB - fixed typo in getSiteMaskStatus() method
FIX: Fix getSiteMaskStatus in SiteDirector and MultiProcessSiteDirector
CHANGE: WatchdogLinux - using python modules in instead of shell calls
*DMS
FIX: in DMSHelpers don't complain if an SE is at 0 sites
*Interfaces
CHANGE: Job.py - using the deprecated decorator for 2 deprecated methods
*RSS
FIX: EmailAction considers also CEs, not only SEs
*Resources
FIX: removed a useless/broken method in Resources helper
FIX: marked as obsoleted two methods in Resources helper (FTS2 related)
[v6r19p8]
*Configuration
FIX; Resources - don't overwrite queue tags if requiredtags are set.
*Framework
CHANGE: dirac-proxy-init - increase dirac-proxy-init CRL update frequency
*Accounting
CHANGE: AccountingDB - if the bucket length is part of the selected conditions,
add to the grouping
*WorkloadManagement
FIX: ConfigTemplate.cfg - allow user access to getSiteMaskStatus
*DataManagementSystem
FIX: DMSHelpers - recursive resolution of SEGroup was keeping the SEGroup in the list
*RSS
FIX: CSHelper - getting FTS from the correct location
CHANGE: use the SiteStatus object wherever possible
*Resources
FIX: CREAMComputingElement - added CS option for extra JDL parameters
*Documentation
CHANGE: point README to master and add badges for integration
[v6r19p7]
*WorkloadManagement
FIX: SiteDirector - correct escaping in pilot template
FIX: dirac-wms-get-wn-parameters - added some printouts to dirac-wms-get-wn-parameters
[v6r19p6]
*Core
FIX: SocketInfo - log proper message on CA's init failure.
*Accounting
CHANGE: NetworkAgent - remove support of perfSONAR summaries and add support of raw metrics.
*WMS
FIX: JobDB - don't trigger exception in webSummary if a site with a single dot is in the system
CHANGE: SiteDirector - added logging format and UTC timestamp to pilot wrapper
FIX: JobMonitoring - fix in getJobPageSummaryWeb() for showing correct sign of life for stalled jobs
*TS
FIX: TransformationManager - fix for wrong method called by the Manager