Bug fix for extremely large index files

.gitignore

@@ -30,6 +30,7 @@ ipch
 Debug
 Release
 *.lcl
+.vs
 
 # Executables
 *.exe

README.md

@@ -1,55 +1,96 @@
-# mod_mrf
+# mod_mrf [AHTSE](https://github.com/lucianpls/AHTSE)
 
-An apache module that serves tiles directly from a local MRF, 2D or 3D. 
-This module takes two apache configuration directives:
+An apache module that serves tiles directly from a local MRF, 2D or 3D.  
+With the MRF data on a local SSD, this module average tile request latency is .25ms (as measured by httpd), and reaches request rates above 20000 req/sec on a single core.
 
- **MRF On|Off**
-
- Defaults to on if the MRF_ConfigurationFile is provided
+Apache configuration directives:
+
+**MRF_RegExp**
+ Required, only requests matching this pattern are handled.  It can appear multiple times
+ If not provided, the module is inactive
+
+**MRF_Indirect On|Off**
 
- **MRF_ConfigurationFile  Filename**
+ If set, this module will only respond to internal subrequests
 
- Points to a text file that contains lines, where the first word on a line is a directive, followed by parameters
+**MRF_ConfigurationFile  Filename**
+
+ Points to an AHTSE Control text file, where the first word on a line is a directive, followed by parameters
  - Empty lines, lines that start with # are considered comments
  - Unknown directives are ignored
- - Known directives for this module are:
 
-  **Size X Y Z C**
-  - Mandatory, at least x and y, the size in pixels of the input MRF.  Z defaults to 1 and C defaults to 3 (these are usually not meaningful)
+AHTSE Control Directives for this module are:
 
-  **DataFile string**
-  - Mandatory, the path to the data file of the MRF.
+***DataFile path start_offset size***
+ - The path to the MRF data file to serve tiles from. Start and size are optional, by default 
+ a single DataFile is used. At least one DataFile directive is required.  If the path start 
+ with ://, the path is interpreted as an internal redirect to a path 
+ within the same server, starting from DocRoot. Otherwise it is assumed to be a local file 
+ name. May appear multiple times, with different start offset and size values. If the values are 
+ present, read operations within start_offset and start_offset + size are made to the data file, 
+ after the read  offset is adjusted downward by start_offset.
+ If the read offset falls outside the range, the other DataFile entries are searched, 
+ in the order in which they appear in the configuration file. Old style redirects are tested last.
+ The multiple entries allows an MRF data file to be split into multiple parts. Single tiles 
+ cannot be split between sources, but overlapping ranges between source are allowed. Only one read 
+ operation is issued, to the first DataFile entry that matches the range.  If the read fails, 
+ the server will report an error.
+ Start offset and size default to zero. Zero size means that any read above the offset will be 
+ considered present in this data file.
 
-  **RegExp**
-  - Optional, a regular expression that must match the request URL for it to be considered a tile request.  By default all URLs are considered tile requests.  This directive can appear multiple times.  If there are multiple RegExp lines, at least one has to match the request URL.
-
-  **PageSize X Y 1 C**
-  - Optional, the pagesize in pixels.  X and Y default to 512. Z has to be 1 if C is provided, which has to match the C value from size
+***Size X Y Z C***
+ - Mandatory, at least x and y, the size in pixels of the input MRF.
+ Z defaults to 1 and C defaults to 3 (these are usually not meaningful)
 
-  **IndexFile string**
-  - Optional, The index file name.
-  If not provided it uses the data file name if its extension is not three letters.  
-  Otherwise it uses the datafile name with the extension changed to .idx
-
-  **MimeType string**
-  - Optional.  Defaults to autodetect.
+***PageSize X Y 1 C***
+ - Optional, the pagesize in pixels. X and Y default to 512. 
+ Z has to be 1 if C is provided, which has to match the C value from size
 
-  **EmptyTile Size Offset FileName**
-  - Optional.  By default it ignores the request if a tile is missing.
-  First number is assumed to be the size, second is offset.
-  If filename is not provided, it uses the data file name.
+***RetryCount N***
+  - Optional, [0 - 99). If the DataFiles are redirects, how many times to retry a redirected 
+  read that fails to retun the requested data. The Default is 5.
 
-  **SkippedLevels N**
-  - Optional, how many levels to ignore, at the top of the MRF pyramid.
-  For example a GCS pyramid will have to skip the one tile level, so this should be 1
+***IndexFile string***
+ - Optional, the index file name. Can only be provided once.
+  If not provided it uses the data file name if its extension is not three letters.
+  Otherwise it uses the first data file name with the extension changed to .idx
+  It can be a redirect path in the host namespace, if it starts with ://
 
-  **ETagSeed base32_string**
-  - Optional, 64 bits in base32 digits.  Defaults to 0.  
-  The empty tile ETag will be this value but bit 64 (65th bit) is set. All the other tiles
-  have 64 bit ETags that depend on this value.
+***EmptyTile Size Offset FileName***
+ - Optional, provides the tile content to be sent when the requested tile is missing.
+ The file has to be local, since the empty tile is read at start-up
+ By default the request is ignored, which results in a 404 error if a fallback mechanism does not 
+ exist. If present, the first number is assumed to be the size, second is offset. If filename is 
+ not given, the first data file name is used.
+
+***SkippedLevels N***
+ - Optional, how many levels to ignore, at the top of the MRF pyramid. For example a GCS pyramid 
+ will have to skip the one tile level, so this should be 1
 
-  **Redirect path**
-  - Optional, if the data file is on an object store
+***ETagSeed base32_string***
+ - Optional, 64 bits as 13 base32 digits [0-9a-v], defaults to 0. The empty tile ETag will be 
+ this value but 65th bit is set, also the only value that has this bit set. All the other tiles 
+ have 64 bit ETags that depend on this value.
+
+***Dynamic On***
+ - Optional, flags the local files as dynamic, disabling any caching or file handle reuse. To be used
+ when the MRF files are changed at run-time, avoiding stale or even broken content.  MRF in-place
+ modification do not require this flag because the old content is still available
+
+ ***MMapping prefix***
+ - Optional, controls the mapping of the M parameter to data source.  The only value currently 
+ implemented is _prefix_, which means that the M, as a decimal number will be added right in front of
+ the basename of the file, both the Index and Data. The range based data file split still applies, 
+ each part will be prefixed by the M value
+
+
+***CannedIndex On***
+ - Optional, flags the index file as a canned format index, see mrf_apps/can.cpp.  This is a dense 
+ format index that can be much smaller.  Should be used only when needed, and not recommended when
+ Dynamic is also on
+
+***Redirect path start_offset size***
+  *Deprecated*, use the DataFile directive and start path with ://
 
 For better performance on local files, the httpd source against which this module is compiled should include support for random file access optimization. A patch file for libapr is provided, see apr_FOPEN_RANDOM.patch
 

macros.props

@@ -0,0 +1,15 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ImportGroup Label="PropertySheets" />
+  <PropertyGroup Label="UserMacros">
+    <prefix>C:\httpd</prefix>
+  </PropertyGroup>
+  <PropertyGroup />
+  <ItemDefinitionGroup />
+  <ItemGroup>
+    <BuildMacro Include="prefix">
+      <Value>$(prefix)</Value>
+      <EnvironmentVariable>true</EnvironmentVariable>
+    </BuildMacro>
+  </ItemGroup>
+</Project>

mod_mrf.sln

@@ -1,22 +1,29 @@
 
 Microsoft Visual Studio Solution File, Format Version 12.00
-# Visual Studio 2013
-VisualStudioVersion = 12.0.40629.0
+# Visual Studio Version 17
+VisualStudioVersion = 17.10.34928.147
 MinimumVisualStudioVersion = 10.0.40219.1
 Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "mod_mrf", "mod_mrf.vcxproj", "{C7E01836-333F-490D-A3B5-3554A8935040}"
 EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Win32 = Debug|Win32
+		Debug|x64 = Debug|x64
 		Release|Win32 = Release|Win32
+		Release|x64 = Release|x64
 	EndGlobalSection
 	GlobalSection(ProjectConfigurationPlatforms) = postSolution
-		{C7E01836-333F-490D-A3B5-3554A8935040}.Debug|Win32.ActiveCfg = Debug|Win32
-		{C7E01836-333F-490D-A3B5-3554A8935040}.Debug|Win32.Build.0 = Debug|Win32
-		{C7E01836-333F-490D-A3B5-3554A8935040}.Release|Win32.ActiveCfg = Release|Win32
-		{C7E01836-333F-490D-A3B5-3554A8935040}.Release|Win32.Build.0 = Release|Win32
+		{C7E01836-333F-490D-A3B5-3554A8935040}.Debug|Win32.ActiveCfg = Debug|x64
+		{C7E01836-333F-490D-A3B5-3554A8935040}.Debug|x64.ActiveCfg = Debug|x64
+		{C7E01836-333F-490D-A3B5-3554A8935040}.Debug|x64.Build.0 = Debug|x64
+		{C7E01836-333F-490D-A3B5-3554A8935040}.Release|Win32.ActiveCfg = Release|x64
+		{C7E01836-333F-490D-A3B5-3554A8935040}.Release|x64.ActiveCfg = Release|x64
+		{C7E01836-333F-490D-A3B5-3554A8935040}.Release|x64.Build.0 = Release|x64
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
 	EndGlobalSection
+	GlobalSection(ExtensibilityGlobals) = postSolution
+		SolutionGuid = {A5643073-62AA-4472-AEC6-7057F30D5683}
+	EndGlobalSection
 EndGlobal

mod_mrf.vcxproj

@@ -1,87 +1,100 @@
 <?xml version="1.0" encoding="utf-8"?>
-<Project DefaultTargets="Build" ToolsVersion="12.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+<Project DefaultTargets="Build" ToolsVersion="15.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
   <ItemGroup Label="ProjectConfigurations">
-    <ProjectConfiguration Include="Debug|Win32">
+    <ProjectConfiguration Include="Debug|x64">
       <Configuration>Debug</Configuration>
-      <Platform>Win32</Platform>
+      <Platform>x64</Platform>
     </ProjectConfiguration>
-    <ProjectConfiguration Include="Release|Win32">
+    <ProjectConfiguration Include="Release|x64">
       <Configuration>Release</Configuration>
-      <Platform>Win32</Platform>
+      <Platform>x64</Platform>
     </ProjectConfiguration>
   </ItemGroup>
   <ItemGroup>
     <ClCompile Include="src\mod_mrf.cpp" />
   </ItemGroup>
   <ItemGroup>
-    <ClInclude Include="src\mod_mrf.h" />
-  </ItemGroup>
-  <ItemGroup>
+    <None Include=".gitignore" />
     <None Include="apr_FOPEN_RANDOM.patch" />
+    <None Include="LICENSE" />
+    <None Include="mod_proxy_http.patch" />
     <None Include="README.md" />
+    <None Include="src\Makefile" />
+    <None Include="src\Makefile.lcl.example" />
   </ItemGroup>
   <PropertyGroup Label="Globals">
     <ProjectGuid>{C7E01836-333F-490D-A3B5-3554A8935040}</ProjectGuid>
     <Keyword>Win32Proj</Keyword>
     <RootNamespace>mod_mrf</RootNamespace>
+    <WindowsTargetPlatformVersion>10.0</WindowsTargetPlatformVersion>
   </PropertyGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
-  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
     <ConfigurationType>DynamicLibrary</ConfigurationType>
     <UseDebugLibraries>true</UseDebugLibraries>
-    <PlatformToolset>v120</PlatformToolset>
+    <PlatformToolset>v143</PlatformToolset>
     <CharacterSet>NotSet</CharacterSet>
   </PropertyGroup>
-  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
     <ConfigurationType>Application</ConfigurationType>
     <UseDebugLibraries>false</UseDebugLibraries>
-    <PlatformToolset>v120</PlatformToolset>
+    <PlatformToolset>v143</PlatformToolset>
     <WholeProgramOptimization>true</WholeProgramOptimization>
-    <CharacterSet>Unicode</CharacterSet>
+    <CharacterSet>NotSet</CharacterSet>
   </PropertyGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
   <ImportGroup Label="ExtensionSettings">
   </ImportGroup>
-  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
+  <ImportGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="PropertySheets">
     <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="macros.props" />
   </ImportGroup>
-  <ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
+  <ImportGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="PropertySheets">
     <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+    <Import Project="macros.props" />
   </ImportGroup>
   <PropertyGroup Label="UserMacros" />
-  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
     <LinkIncremental>true</LinkIncremental>
     <TargetExt>.so</TargetExt>
+    <OutDir>$(SolutionDir)$(Configuration)\</OutDir>
+    <IntDir>$(Configuration)\</IntDir>
+    <IncludePath>$(prefix)\include;$(prefix)\include\apr-2;$(VC_IncludePath);$(WindowsSDK_IncludePath);</IncludePath>
+    <LibraryPath>$(prefix)\lib;$(VC_LibraryPath_x64);$(WindowsSDK_LibraryPath_x64)</LibraryPath>
   </PropertyGroup>
-  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
     <LinkIncremental>false</LinkIncremental>
+    <IncludePath>$(prefix)\include;$(prefix)\include\apr-2;$(VC_IncludePath);$(WindowsSDK_IncludePath);</IncludePath>
+    <LibraryPath>$(prefix)\lib;$(VC_LibraryPath_x64);$(WindowsSDK_LibraryPath_x64)</LibraryPath>
   </PropertyGroup>
-  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
+  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
     <ClCompile>
       <PrecompiledHeader>NotUsing</PrecompiledHeader>
       <WarningLevel>Level3</WarningLevel>
       <Optimization>Disabled</Optimization>
       <PreprocessorDefinitions>WIN32;_DEBUG;_WINDOWS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <SDLCheck>true</SDLCheck>
-      <AdditionalIncludeDirectories>\Apache24\include</AdditionalIncludeDirectories>
+      <AdditionalIncludeDirectories>
+      </AdditionalIncludeDirectories>
       <CompileAs>Default</CompileAs>
     </ClCompile>
     <Link>
       <SubSystem>Windows</SubSystem>
       <GenerateDebugInformation>true</GenerateDebugInformation>
-      <AdditionalLibraryDirectories>\Apache24\lib</AdditionalLibraryDirectories>
-      <AdditionalDependencies>libhttpd.lib;libapr-1.lib;libaprutil-1.lib;%(AdditionalDependencies)</AdditionalDependencies>
+      <AdditionalLibraryDirectories>
+      </AdditionalLibraryDirectories>
+      <AdditionalDependencies>libahtse.lib;libhttpd.lib;libapr-2.lib;%(AdditionalDependencies)</AdditionalDependencies>
       <AdditionalOptions>/EXPORT:mrf_module,@1</AdditionalOptions>
-      <OutputFile>$(OutDir)$(TargetName)$(TargetExt)</OutputFile>
     </Link>
     <PostBuildEvent>
-      <Command>copy /y $(OutDir)$(TargetName)$(TargetExt) \Apache24\modules</Command>
+      <Command>copy /y $(TargetPath) $(prefix)\modules</Command>
+      <Message>Install to $(prefix)\modules</Message>
     </PostBuildEvent>
   </ItemDefinitionGroup>
-  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
+  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
     <ClCompile>
       <WarningLevel>Level3</WarningLevel>
-      <PrecompiledHeader>Use</PrecompiledHeader>
+      <PrecompiledHeader>NotUsing</PrecompiledHeader>
       <Optimization>MaxSpeed</Optimization>
       <FunctionLevelLinking>true</FunctionLevelLinking>
       <IntrinsicFunctions>true</IntrinsicFunctions>
@@ -93,7 +106,13 @@
       <GenerateDebugInformation>true</GenerateDebugInformation>
       <EnableCOMDATFolding>true</EnableCOMDATFolding>
       <OptimizeReferences>true</OptimizeReferences>
+      <AdditionalDependencies>libahtse.lib;libhttpd.lib;libapr-2.lib;$(CoreLibraryDependencies);%(AdditionalDependencies)</AdditionalDependencies>
+      <AdditionalOptions>/EXPORT:mrf_module,@1</AdditionalOptions>
     </Link>
+    <PostBuildEvent>
+      <Command>copy /y $(TargetPath) $(prefix)\modules</Command>
+      <Message>Install to $(prefix)\modules</Message>
+    </PostBuildEvent>
   </ItemDefinitionGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
   <ImportGroup Label="ExtensionTargets">

mod_mrf.vcxproj.filters

@@ -9,21 +9,31 @@
       <UniqueIdentifier>{93995380-89BD-4b04-88EB-625FBE52EBFB}</UniqueIdentifier>
       <Extensions>h;hh;hpp;hxx;hm;inl;inc;xsd</Extensions>
     </Filter>
-    <Filter Include="DOC">
-      <UniqueIdentifier>{a73e6c46-3010-4cc7-b8fa-4d7c875ede0f}</UniqueIdentifier>
+    <Filter Include="Support Files">
+      <UniqueIdentifier>{76786145-6ccc-4796-bd24-c8edce17e2a3}</UniqueIdentifier>
     </Filter>
   </ItemGroup>
   <ItemGroup>
-    <ClInclude Include="src\mod_mrf.h">
-      <Filter>Header Files</Filter>
-    </ClInclude>
-  </ItemGroup>
-  <ItemGroup>
-    <None Include="README.md">
-      <Filter>DOC</Filter>
+    <None Include=".gitignore">
+      <Filter>Support Files</Filter>
+    </None>
+    <None Include="LICENSE">
+      <Filter>Support Files</Filter>
+    </None>
+    <None Include="src\Makefile">
+      <Filter>Support Files</Filter>
+    </None>
+    <None Include="src\Makefile.lcl.example">
+      <Filter>Support Files</Filter>
     </None>
     <None Include="apr_FOPEN_RANDOM.patch">
-      <Filter>DOC</Filter>
+      <Filter>Support Files</Filter>
+    </None>
+    <None Include="README.md">
+      <Filter>Support Files</Filter>
+    </None>
+    <None Include="mod_proxy_http.patch">
+      <Filter>Support Files</Filter>
     </None>
   </ItemGroup>
   <ItemGroup>