-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGES
165 lines (127 loc) · 9.2 KB
/
CHANGES
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
v1.2.3p0
----------------------------------------------------------------------
Re-tooled the rebuild process. force_resource_rebuild=2 behavior is unchanged. Although
the behavior when 1 is different. When force_resource_rebuild=1 an attempt is made to
recover the DB directly. If this succeeds the usage is then recaclulated by scanning the DB.
This is the default recovery behavior if the depot dies unexpectantly upon restart.
Duration truncation- IF force_resource_rebuild=1 and truncate_durations=1 then the DB is walked
shortening any durations as needed and also removing any expired allocations.
v1.2.2p0
----------------------------------------------------------------------
Added background resource cleanup thread. As a result the freeing of all expired allocations
that used to occur for status and get_version calls has been dropped. If extra space is
needed in create_allocation_resource() attempts are still made to free the space.
v1.2.1p0
----------------------------------------------------------------------
IBP_PUSH - Expanded form of IBP_SEND/IBP_PHOEBUS_SEND, which is used for depot-depot copies.
The new command provides equivalent functionality to IBP_SEND/IBP_PHOEBUS_SEND but also
adds support for specifying an offset for the destination allocation. If "-1" is specified
it defaults to append the same as the IBP_*SEND commands.
IBP_PULL - Same functionality as IBP_PUSH but in this case the data is "pulled" or read from
the remote depot instead of pushed or written. The command supports all the functionality
as IBP_PULL: phoebus/normal tranfers, support for data appending or specific destination
offset. IBP_PULL allows for the transfer of data to depot's located behind firewalls,
private subnets, or NATed. A private depot can normally send or push data to a publically
available depot but the reverse is normally not true since the private depot is not allowed
direct access to the Internet and instead accesses the internet through a proxy or gateway
machine. IBP_PULL gets around this problem by allowing the private depot to initiate the
transfer and pull or read the data from the public depot.
IBP_SPLIT_ALLOCATE - Provides support to atomically split an existing allocation
NOTE: THe data is not split just the allocations size/space is changed.
IBP_MERGE_ALLOCATE - Atomic support to merge the space of 2 existing allocations
NOTE: The data in the 2ndary alloc is lost. Just it's space is merged with the primary.
v1.2.0p0
----------------------------------------------------------------------
***NOTE: This version is not backward compatible with previous version because
the allocation_t type has changed!!!****
ibp_upgrade_v1-2-0 - Routine to upgrade the depot allocations from previous versions.
Big Changes
........................
Activity log support - This feature is not complete yet. Currently it logs each IBP command to an
activity log file which is appended between subsequent ibp_server restarts. It does recover from
hard kills. It does NOT currently send full activity logs to a remote log server. That will be
added soon. The program print_alog processes the compressed activity log files
generated and prints them to the screen.
Recent activity queries - The allocation_t data structure has been modified which breaks backward
compatibility. This change keeps the most recent 16 read, 16 write, and 16 manage operations for each
allocation. This includes operations generated through an alias. This can be disabled in the
config file. There are new IBP commands designed for internal use to access this information
discussed below.
DB reliability changed to ACID compliant - This has not been fully tested so be careful. As a
result fo this change the create allocation rate has dropped significantly to around
~80 creates/sec/resource.
Initial Security Framework - This is definitely just started and not complete. The current
implementation is focused on 2 aspects creating, using, and managing alias allocations along
with controlling IBP command execution.
Alias framework - Think of an alias allocation as a soft link to the primary allocation that
can also be byte restricted. The idea is that only alias allocations are handed out to
clients. Since clients never see the true allocation their access can be revoked at any
time without effecting other users access to the same allocation. With the exception
of making allocation changes via IBP_MANAGE the client has no concept they are using
a alias allocation.
iptables-like control - Each IBP command's execution can be controlled through and iptables-like
framework. Namely each ibp command can be restricted via a subnet mask. This way
you can restrict IBP_ALLOCATE commands to only come from approved hosts but IBP_READ/WRITE
commands can come from anywhere.
Network layer abstraction completely retooled - This was for adding phoebus support and
for future SSL support needed for the security framework. This framework is much more
flexible and also supports monitoring of multiple ports.
Managemnt tools
.....................
print_alog - Used to parse the activity log files.
date_spacefree - Generates a table showing the daily amount of space that comes available.
expire_list - Generates a table listing each allocation in the order they expire.
get_alloc - Retreives the given allocation and prints all the information. This includes the
read, write, and manage recent activity history along with the creation date. Optionally
a byte range of data can be read.
New IBP functionality
..........................
IBP_PHOEBUS_SEND - Performs a depot-depot copy using a phoebus network bath instead of a standard
TCP/IP connection.
IBP_RENAME - Forces the generation of a new set of caps for an existing allocation. The old caps are no
longer valid after execution.
IBP_ALIAS_ALLOCATE - Generates a set of caps that act as an alias for an existing allocation. The alias can
be created using a specific byte range of the existing allocation to restrict access. The primary
allocation is not effected upon deletion of the alias allocation.
IBP_ALIAS_MANAGE - Similar to IBP_MANAGE but for the alias allocation. Requires the primary or master
allocations manage cap to make changes.
INTERNAL_GET_ALLOC - Used to retreive a raw allocation.
INTERNAL_DATE_FREE - Used to retreive the free space vs date table
INTERNAL_EXPIRE_LIST - Used to retreive which allocations are expiring and when.
Bugs fixed
..................
Fixed a remove_allocation bug which resulted in an empty allocation not being completely removed.
v1.1.7p0
------------------------------------------------------------------------------------
Support for installable commands - Replaced the hard coded case statement with a function table to allow
additional commands to added modularly. This added commands.c and install_commands.c and an internal restructutring
in ibp_server.c and server_lib.c. From now on to add a new command you should just have to edit install_commands.c
to add the new command and also modify the makefile.
v1.1.6p2
-------------------------------
read_command() - Fixed bug so that a fragmented line wouldn't return early leading to a mangled command.
server_version() - Added compile flags to string
v1.1.6p1
-------------------------------
make_dportal_connection() - Rearranged things so the net_connect() call occurs outside the lock.
This means that depot-depot copies should be be faster. The old approach meant that all depot-depot copies
were serialzed in making the connection to the remote depot. This isn't a problem if the depot
is up but if it's unresponsize it can cause a *huge* backlog if lots of threads are being used for
depot-depot copies.
depot/depot copy accounting - Modified transfer_stats to support tracking bytes sent to other depots.
The additional info is also reported with get_version. This way it's possible to eliminate the double
counting of bytes if you want to calculate the aggreagate transfer among a collection of depots.
Assuming there are no transfers to depots outside the collection:)
max_size - Added the "max_size" attribute to a resource config file. The parameter defines the maximum
amount of disk space to use for the sum of both hard+soft allocations. "min_free" is still used to make
sure the disk is not accidentally filled up by the depot.
_lookup_id_with_cap_db() - Modified it so that a fatal DB error triggers a shutdown. Also modified
server_loop() to shutdown monitoring port immediately. This way StorCore sees the depot as down even if
there are problems continuing the shutdown.
same_depot_copy() - Fixed bug for append mode. It would instead use an offset of 0 and overwrite.
Also fixed a race condition when multiple threads writing to the same allocation would cause a corruption
of the allocations size.
handle_send() - Same error as above.
network.c - Modified readline_* routines to *not* return the actual newline character.
stack.c - Added link/unlink functionality to help minimize malloc/free calls for certain use cases.
flawfinder/rat - corrected all the potential security issues these tools found. This touched all sorts of files.