Add frontend and scheduler for Autotest

The attached tarball includes the new Autotest web frontend for creating 
and monitoring jobs and the new scheduler for executing jobs.  We've 
been working hard to get these complete and stabilized, and although 
they still have a long, long way to go in terms of features, we believe 
they are now stable and powerful enough to be useful.
 
The frontend consists of two parts, the server and the client.  The 
server is implemented in Python using the Django web framework, and is 
contained under frontend/ and frontend/afe.  The client is implemented 
using Google Web Toolkit and is contained under frontend/client.  We 
tried to use Dojo initially, as was generally agreed on, but it proved 
to be too young and poorly documented of a framework, and developing in 
it was just too inefficient.

The scheduler is contained entirely in the scheduler/monitor_db Python 
file.  It looks at the database used by the web frontend and spawns 
autoserv processes to run jobs, monitoring them and recording status.  
The scheduler was developed by Paul Turner, who will be sending out some 
detailed documentation of it soon.
 
I've also included the DB migrations code for which I recently submitted 
a patch to the mailing list.  I've included this code because it is 
necessary to create the frontend DB, but it will (hopefully) soon be 
part of the SVN repository.

Lastly, there is an apache directory containing configuration files for 
running the web server through Apache.
 
I've put instructions for installing a full Autotest server, including 
the web frontend, the scheduler, and the existing "tko" results backend, 
at http://test.kernel.org/autotest/AutotestServerInstall.  I've also 
created a brief guide to using the web frontend, with plenty of 
screenshots, at http://test.kernel.org/autotest/WebFrontendHowTo.  
Please let me know if you find any problems with either of these pages.

Take a look, try it out, send me comments, complaints, and bugs, and I 
hope you find it useful!

Steve Howard, and the rest of the Google Autotest team

From: Steve Howard <[email protected]>

apache/conf/all-directives

@@ -0,0 +1,3 @@
+Include "/usr/local/autotest/apache/conf/tko-directives"
+Include "/usr/local/autotest/apache/conf/django-directives"
+Include "/usr/local/autotest/apache/conf/site-directives"

apache/conf/apache-conf

@@ -0,0 +1,24 @@
+NameVirtualHost *
+<VirtualHost *>
+    DocumentRoot /usr/local/autotest/apache/www/
+    <Directory />
+	Options FollowSymLinks
+	AllowOverride None
+    </Directory>
+    <Directory /usr/local/autotest/apache/www/>
+	Options Indexes FollowSymLinks MultiViews
+	AllowOverride None
+	Order allow,deny
+	allow from all
+    </Directory>
+
+
+    ErrorLog /var/log/apache2/error.log
+    LogLevel warn
+
+    CustomLog /var/log/apache2/access.log combined
+    ServerSignature On
+
+    Include "/usr/local/autotest/apache/conf/all-directives"
+
+</VirtualHost>

apache/conf/django-directives

@@ -0,0 +1,15 @@
+Alias /afe "/usr/local/autotest/frontend/client/www/afeclient.ClientMain"
+<Location "/afe">
+    DirectoryIndex ClientMain.html
+</Location>
+
+<Location "/afe/server">
+    SetHandler python-program
+    PythonHandler django.core.handlers.modpython
+    SetEnv DJANGO_SETTINGS_MODULE frontend.settings
+    PythonDebug On
+    PythonPath "['/usr/local/autotest'] + sys.path"
+</Location>
+
+
+Alias /media "/usr/lib/python2.4/site-packages/django/contrib/admin/media"

apache/conf/tko-directives

@@ -0,0 +1,18 @@
+RedirectMatch ^/$ /afe/
+
+Alias /results "/usr/local/autotest/results/"
+<Directory /usr/local/autotest/results/>
+    Options Indexes FollowSymLinks MultiViews
+    AllowOverride None
+    Order allow,deny
+    Allow from all
+</Directory>
+
+Alias /tko "/usr/local/autotest/tko/"
+<Directory /usr/local/autotest/tko/>
+    Options ExecCGI Indexes MultiViews +SymLinksIfOwnerMatch
+    DirectoryIndex compose_query.cgi
+    AllowOverride None
+    Order allow,deny
+    Allow from all
+</Directory>

apache/www/index.html

@@ -0,0 +1 @@
+<META HTTP-EQUIV="Refresh" CONTENT="1; URL=/afe/">

frontend/afe/control_file.py

@@ -0,0 +1,67 @@
+"""\
+Logic for control file generation.
+"""
+
+__author__ = '[email protected] (Steve Howard)'
+
+import os
+import frontend.settings
+
+AUTOTEST_DIR = os.path.abspath(os.path.join(
+    os.path.dirname(frontend.settings.__file__), '..'))
+
+
+KERNEL_INSTALL_TEMPLATE = """\
+def step_init():
+	job.next_step([step_test])
+	testkernel = job.kernel('%(kernel)s')
+	%(kernel_config_line)s
+	testkernel.install()
+	testkernel.boot(args='%(kernel_args)s')
+
+def step_test():
+"""
+
+def kernel_config_line(kernel, platform):
+	if (not kernel.endswith('.rpm') and platform and
+	    platform.kernel_config):
+		return "testkernel.config('%s')" % platform.kernel_config
+	return ''
+
+
+def read_control_file(test):
+	control_file = open(os.path.join(AUTOTEST_DIR, test.path))
+	control_contents = control_file.read()
+	control_file.close()
+	return control_contents
+
+
+def get_kernel_stanza(kernel, platform, kernel_args):
+	return KERNEL_INSTALL_TEMPLATE % {
+	    'kernel' : kernel,
+	    'kernel_config_line' : kernel_config_line(kernel, platform),
+	    'kernel_args' : kernel_args}
+
+
+def get_tests_stanza(tests):
+	return ''.join(read_control_file(test) for test in tests)
+
+
+def indent_text(text, indent):
+	lines = [indent + line for line in text.splitlines()]
+	return '\n'.join(lines)
+
+
+def generate_client_control(tests, kernel=None, platform=None):
+	control_file = ''
+	indent = ''
+	if kernel:
+		control_file = get_kernel_stanza(kernel, platform, '')
+		indent = '\t'
+
+	control_file += indent_text(get_tests_stanza(tests), indent)
+	return control_file
+
+
+def generate_server_control(tests):
+	return get_tests_stanza(tests)