START: Submission Tracking And Review Toolset
Copyright © 1998-2000 Rich Gerber, Jeff Hollingsworth and Adam Porter. All Rights Reserved
This software is released on a controlled basis.
Permission for using START is granted for a single conference or workshop only, unless explicitly obtained for multiple events. The START logo appears on auto-generated forms, reports and HTML pages; this logo may not be removed or altered, unless prior consent is obtained from the authors. The system is released AS IS, without support, and may be modified as desired. Bug reports are solicited, as are suggestions for upgraded functionality. However, we cannot guarantee that the repairs or upgrades will be made. By installing this software, you hereby agree to these terms, and to all other provisions in the general usage license below.
What is START?
START is a Web-based conference management system. It was created to help streamline some of the tedious duties associated with being a conference chair, namely:
- Paper Submissions: Templates are provided for producing HTML submission forms. These forms allow authors to automatically upload papers in postscript or PDF format, optionally compressed via any common MIME-aware code. When an author submits a paper, the START server carries out the following actions: The paper gets assigned a unique numerical ID, which is used to track it through the system; information on the paper gets logged in a tracking record, which enter the paper title, author names, keywords, and other information; then paper gets archived in a private directory. An automatic confirmation message is sent to the author, notifying him or her of the paper ID, and giving information on the schedule for announcing acceptances/rejections.
- Document Conversion: START eases the process of converting documents from postscript to PDF, or vice versa.
- Program Committee Support: START includes modules to help export information to the PC. A private directory for the PC is automatically created, which contains an indexed, hyperlinked list of papers; then members can then download and print the papers on-demand; on-line reports are created for the PC, which categorize the submissions in various formats; a abstracts are automatically transformed to HTML, which can then be perused by the PC. START tools set up exclusive login/password access for PC members, without requiring any help from your system administrators.
- Assigning Reviewer Duties: PC members can automatically report reviewing preferences, via web-based form submission (the START tools pre-format these forms, based on the submissions). Another tool helps to assign papers to PC members, taking into account their preferences, as well as their conflicts of interest.
- Review Collection: PC members submit reviews via WWW form upload. The review module generates customized review forms on-demand, which inputs all PC member/paper information from the START database. START's server processes on-line reviews from PC members, and logs the pertinent information. Also, the server provides member-specific views into the review database, which support viewing and editing previously logged reviews, without involvement of the PC chair.
- Comprehensive Report Generation: START's report module produces multiple collations of the review reports - for the chair's use during the review period, and then for the committee's use during the "vetting stage." These can be used on-line, or at a PC meeting.
- Author Notification: START's tools send accept/reject letters back to the authors, using a form-generation engine. This allows you to write your own letter, and to use fields to for paper-specific data. Shepherding is also supported.
The START software cannot review the papers for you - as usual, your referees have to read the papers, and evaluate them. However, it will substantially reduce the burden involved in handling papers and reviews. You can concentrate more on the technical content in your submitted papers - and leave most of the secretarial drudgery to your computer.
The following terms are used throughout this document.
- External Web Server: This is where you put all public information related to the conference, e.g., your CFP, your submission forms, etc.
- Internal Web Server: This server runs your private conference management software. While the internal and external servers may be the same, you will find it easier to keep them separate. See below for details.
Hardware and Software
The following infrastructure is NECESSARY to run START:
- A Unix system. This can be Solaris, SunOS, FreeBSD, BSD, Linux, etc. You will also need a reasonable amount of free filesystem space, for archiving papers and reviews.
- A version of perl5.0 or higher. If you have the Unix system, chances are you also meet this requirement.
- The conference software. If you are reading this, you have it.
- An Apache Web server. You should install a stand-alone http server on your workstation, over which you have exclusive control. This allows running the server as a user process, and setting it up to expressly handle the conference software. Theoretically, you can run the conference software using your institution's main web server; however, you'll need system administrator's help to set it up, and to maintain it.
The following is tool is OPTIONAL:
- Adobe Acrobat: This tool helps in document conversion. It can convert almost any type of postscript to PDF (or vice versa). It allows processing all papers at once - or alternatively, they can be processed as they come in. It also can convert A4 to USLetter, or back again; it can handle most MS Word control characters, and can scale text to fit any page (so that you can literally force authors to meet pre-specified galley guidelines). It will handle many more variants of postscript than GS and other freeware. While not absolutely necessary, we recommend that you use it. Note that Adobe sells versions for NT, Win95 and MacOS, and some Unix flavors.
Setting up Software
- You have now expanded the conference software tar file. This README file currently resides at an implicit "root" directory, that is, the top-level directory for your conference documents. If you are not happy with the location, move your directory now; its pathname will be used to configure many other files. This directory needs to be accessible from your Unix account, in a directory which you own. Aside from this single constraint, it can go anywhere on your filesystem.
In this document, we refer to your conference's top-level directory as $STARTroot. Unless otherwise specified, we assume you are working in this directory - hence, most other directory references below are considered relative to $STARTroot.
- If you do not have already have a binary image of Apache (version 1.1 or higher), you will have to build one. Get it from Apache, and build it in a separate directory. Here's the URL to get your HTTP server:
Run configure as you normally would,
and the Makefile will get built to accommodate START's directories. Then run "make," and you'll get an httpd image to run for your conference. Finally, put the httpd program where you normally store your binaries.
After you have your httpd binary, the sources used to build it aren't needed for START - including the conf and log files. START generates conf and log files for you, tailored to your conference parameters.
- Now go to the $STARTroot directory, and run the program called setup, which customizes files for your conference. As input, you are requested to enter your conference's name, its year, the acronym, your email address, and other information. Then, setup automatically configures the START software, as follows:
- A master template file is built. This file is called paramters.pl, and it resides in the place as the setup script. It is updated to include the conference name, year, acronym, and other information.
- The START files are changed to account for the location of your directory, i.e., $STARTroot.
- The Web server's three configuration files are generated, and stored in the directory httpd/conf. Modifications reflect the value of $STARTroot, the username (yours), and other fields.
- A sample submission form is created, incorporating your conference-specific information. Part of this process involves creating a list of keywords for paper submissions (see below).
At any time, you can change the conference-specific information entered on the setup command line. To do this, you should first shut down your web server (since setup alters its conf files). Then just re-run setup, your changes will be reflected throughout the system. If desired, you can hand-edit most of the fields in parameters.pl. Note that setup will not override these edits.
You will find a template which automatically-generates your submission form in
Note that setup fills in your conference-specific data, including the hostname/port for your web server, and puts the result in www/submit.html. This file will eventually be copied to your public webserver space.
Since you may be running setup multiple times, we suggest you carry out additional editing on the template - and not on submit.html. Then you can just use setup to re-generate the real form.
A keyword check-off list is also initialized in the setup script. Note that the keyword check-off list is not strictly required to run START. Indeed, it has no semantic interpretation inside the database; if desired, you can eliminate it from submit-template.html, and it won't show up on your submission page. But take it from us - this would not be a wise move. To the contrary, we'd advise including any topic tangentially related to your conference area. One reason is obvious: the keywords can help reviewers select the papers they wish to read. Another reason is a bit less obvious: a long list of topics serves as a sort of advertising tool, since researchers in the identified areas will feel "welcome" at your conference, and perhaps submit their papers to it. Finally, the more keywords you include, the more likely it is that your submission page will get listed in search engine queries.
Setup also asks whether you want PS to PDF conversion. To do this, you'll need some additional software (see below). If you check "NO", then all postscript files will be moved "as is" to the program committee directory, and all PDF files will be treated likewise. If you don't wish to hassle with conversion, this is probably the more pain-free way of running things. However, it does have its liabilities, which we discuss below.
To launch your server for the first time, use the script tools/starthttpd. (Note that the script killhttpd does the opposite - it halts your server.) To verify that it works, bring up the conference submission form, and submit it - without filling in any of the fields. You should receive a START error report as a response. If this does not happen, carefully review the steps enumerated above - chances are you made a small error in configuring the software.
Now configure your cron daemon, to ensure that your httpd always stays up and running, even after a system crash. This is trivial on Solaris (via crontab). Most Unix variants have similarly nice interfaces to cron (check your man pages), i.e., via
- crontab -e
This launches your default EDITOR, with a file containing the current cron schedule for your username. Append a line at the end of the file, which will look like this:
0,15,30,45 * * * * /usr/jim/bin/httpd -f /usr/jim/myconf/httpd/conf/httpd.conf 2 > /dev/null
These items have the following meaning:
- /usr/jim/bin/httpd: This is the absolute pathname for Jim's http binary image, which is running his conference submission process.
- /usr/jim/myconf: This is the value of $STARTroot for Jim's conference.
- 0,15,30,45: The frequency at which the httpd is re-spawned; here it's every 15 minutes. If you want to update the server more frequently - say at 10-minute intervals - just specify a string like 0,10,20,40,50.
Most of START's software is written in the perl language. Scripts handling paper submissions and reviews are spawned by the web server; other scripts are invoked by the conference manager; others are used as subroutines. The organization of the START package is as follows:
Scripts for author-triggered actions.
Scripts for reviewer-triggered actions.
Subroutines used by the PC chair
Shared library files.
If you need to make radical changes to your process, these are the places you'll need to look for the files to change. Unless you wish to change the submission process in a major way, we would advise against this. The important fields get automatically updated via the variables in parameters.pl.
When papers get submitted, the papers and their log files are placed in the directory papers, and the papers themselves are copied into papers/process for pre-processing. The idea is that no files should ever be deleted from your incoming directory - it holds the record of everything which came in.
Before proceeding, you should test the server's functionality. Fire up your web browser once again, and open the submit.html file - which still resides in the private www directory. Submit a few bogus papers to your conference, and see what happens. Check the contents of the abovementioned directories, to make sure everything works.
As papers come in,some post-processing is often needed. We handle this in steps, which should hopefully minimize your problems in handling poorly typeset papers. When a paper gets submitted, it is copied to the directory papers/process. Then it gets transferred to specific locations depending on file type (ps or PDF). For example, for paper number ID, the following actions are taken based on the file's MIME extent.
- ID.pdf: moved to papers/out.
- ID.ps: moved to papers/in (see below).
- ID.*.gz: uncompressed using gunzip, reprocessed via new extent.
- ID.*.Z: uncompressed using uncompress, reprocessed via new extent.
- ID.zip: unzipped, processed again.
At the end of the algorithm, files should be in either papers/in if they're postscript or papers/out if they're PDF - and anything remaining in papers/process should be checked manually.
NOTE: The following section on document conversion explains how postscript files are also moved to papers/out. If you you're only using postscript submissions, you need not read the next section. However, using this alternative is somewhat risky, since it rests on a faulty assumption - that all fonts are printable by everyone, and that no postprocessing is necessary.
When you are ready, copy submit.html to your external web server, for public access, and link your CFP to it. At that point, the system will be ready to accept paper submissions, and the submission form will now be exposed to the entire world.
As the conference manager, chances are you may be running the START scripts many times - and you may wish to do this from any directory. To get this functionality, you should make two edits to your Unix startup files. Assuming that $STARTroot has the value /usr/jim/myconf, you would insert the following line in .cshrc file, or wherever you keep environmental settings:
setenv STARTroot '/usr/jim/myconf'
Also, you'd add the following absolute pathname to your $path variable (probably also in .cshrc):
These settings are optional, but they will save you time.
Conversion to PDF [optional]
In theory, one could handle all electronic submissions in postscript, without using PDF, a method that some conferences use by imposing certain formatting constraints on authors. In practice, this isn't so easy. There are many flavors of postscript around, and some typesetting tools use non-standard subsetting - an enormous problem for generic Type1/Type2 printers. Hence, we recommend converting all postscript papers to PDF; then, you can allow your PC to access papers via either PDF or postscript. There are several ways to generate PDF files:
- Adobe's Acrobat distiller can convert almost all postscript to PDF, in either one-off mode, or as a server.
- A freeware Unix program called ps2pdf can convert postscript files to PDF. The result can sometimes be larger than that which the distiller produces. However, note that this brute-force approach almost always works - even for ill-formed font-subsets, where the Adobe product occasionally fails.
- If authors send you PDF, no conversion is needed.
In the sequel, we'll assume you're using the Distiller as your PDF conversion tool, and that it's running on a Mac or Windows workstation. If you're running the Unix version, much of this information is similar; moreover, many of these instructions can be used for similar tools.
If you are running the Distiller on a NT/Win95/Mac workstation, you'll need read/write accessibility to the files from your Unix machine. This is easy if (i) your Unix system IS also your Win/NT systemó e.g., you run a Unix server process on NT or MacOS; or alternatively, (ii) you have NFS or Samba interfaces between your Win95/NT/Mac machine and your Unix system.
- Launch the distiller, and click on the "Watched Folders" option. A "watched folder" is, in the Acrobat lexicon, the parent of the directories where the postscript and PDF papers are processed.
- Set up the following directory as a "watched folder":
Then, the Distiller uses two sub-directories for its work:
If these directories don't exist, the Distiller creates them for you.
- $STARTroot/papers/in: Postscript versions of papers.
- $STARTroot/papers/out: Converted PDF versions (and ps originals).
Upon arrival, postscript files get transferred into papers/in. If the Distiller is active, it will automatically plow through each one. The resulting PDF (and the postscript originals) are archived in papers/out.
If the distiller cannot handle a postscript file, it will remain in the papers/in directory - along with an error report. These are the papers that will need personal attention - perhaps conversion via ps2pdf or the equivalent.
Regardless of how clearly you state your submission guidelines, there are always a few clowns who will seek out your street address, and then send their papers via surface mail. You may wish to throw them out. However, note that Acrobat has a scan-in function, which will convert scanned bitmaps into PDF. Hence, you may consider just "submitting" these papers yourself - which will then automatically put them on-line the other papers.
Setting up the PC Directory
- The directory www/PC is used for password-protected PC access, and the PC home page is www/PC/index.html, which is the page PC members will see when they login to your server. However, like the submit page, we have also provided a sample PC index page for you, in
- Chances are this template will need editing for your specific conference; however, again we suggest you edit the template, and not the generated HTML. Then re-run setup, and it'll get copied to the right place.
- Generate the lists of papers. If you edited .cshrc as suggested above, you can just type genPaperLists from anywhere - otherwise, go into the tools directory, and run the script:
This script carries out the following actions:
- It moves all papers from papers/out to www/PC/papers - a subdirectory used by PC downloads.
- It generates an HTML report of all paper submissions received thus far. The report includes title and author information, with links to the appropriate postscript and PDF files. Like many of the other scripts, genPaperLists uses the log files stored in the papers directory; hence, it is essential that these do not get moved or deleted.
- Before assignments are made, it will generate a form for PC reviewers to upload their assignment preferences. Note the variable $BidPeriodDone in parameters.pl. If $BidPeriodDone = 0, the form IS generated; if $BidPeriodDone = 1, the form is NOT generated. You should leave the value set to 0 until the reviewing assignments are made.
- It also collates paper abstracts (linked to the papers); these are made accessible to the program committee.
Now you can assign usernames/passwords for the PC members. The script addreviewer can be used for this, e.g., if you run
addReviewer joe crazyfish
It causes a username joe to be added, with password crazyfish. Now your PC member named Joe can logon with his password. The password information is stored in
Test the password feature. Assume your IP address is fish.idiocy.com, that your webserver is configured for port 8080, and that $STARTroot = /usr/jim/myconf. Try opening the following URL:
You should get a login/password box, which should allow you to login as joe (or whatever). Also, make sure that you can't login using an unregistered username, and/or password. This is important!
After setting up your PC directory, you can send mail to each PC member with his/her username/password, along with information on how the reviewing protocols will work (see below)
Assigning Papers to Reviewers
- The conference software is currently set up to accept PC reviewing preference forms (which also indicate their conflicts of interest). The form - paperForm.html - is generated by the above mentioned scripts, and need only get linked to your index page. (If preferred, you can do all the assignments yourself - and in this case, just don't include a link to the form on the home page, and set $BidPeriodDone to 1.)
- After all preference forms are in, the next job is to actually assign papers to reviewers. The database uses the following file to keep track of this:
While we do this automatically, it's a good idea to check out the structure of this file, since you may wish to adjust it to balance the strengths of your PC. Each line consists of a PC member's username, and paper id - denoting that the PC member is assigned that paper to review.
This file is used to by the review module, to restrict access into the database - i.e., to ensure that a paper will only get reviewed by assigned reviewers. So, please head the following warning:
IMPORTANT: If there are any sample assignments in your file, delete them before proceeding further.
- The automatic assignments are generated by the the the script
This script takes into account everyone's preferences, excluding their conflicts, and just assigns papers on a round-robin basis. There are a few items to note here:
- Some PC members invariably forget to submit their forms. You should submit their forms for these members; just log in using the PC member's username/password. You probably know the strengths of your committee well enough to do this; moreover, the software works best if all forms are present.
- The round-robin scheme is ALWAYS over-ridden by the current assignment file, i.e., reviewData/assignments.
You can easily hand-edit the assignments file, to give papers to whomever you wish. Then, in the round-robin scheme, reviewers with pre-assigned papers will wait their turn, until the other PC members have "caught up."
The algorithm produces an assignment report as standard standard output, which you can redirect to a file if desired. (It can be large for a big conference.) The script also produces a "tentative assignments" file, stored in:
Inspect the assingment report; if you're happy with the allocation, just execute:
- mv reviewData/assignments.out reviewData/assignments
Go through a few trial runs until you're satisfied. If your conference has a lot of submissions, you'll probably end up fine tuning the process a bit - either by editing the assignment file, or altering the preference files. The first time this was used - in a conference with about 200 submission, and 25 PC members - tuning the assignments took about 3 hours or so.
The Review Phase
- At this point, reviewing can work automatically, without any intervention by you. All you need to do is to include the following line on your (template) PC home page:
- <a href="get-assigned.cgi"> Your Assignments</a>: A customized list of papers (with review forms) assigned to you.</a>
See the sample pages for reference. The script get-assigned.cgi customizes a "to do" list for each PC member, showing the papers allocated to him/her, with links to review forms for those not done yet. The to-do list also allows editing previously written reviews. In fact, the same PC member can review a paper hundreds of times, with the last submitted review always considered the "current one."
- PC members really like this feature: it allows them to make progress on their assignments, but nobody's committed to their opinions until the start of the vetting period. Even after the paper decisions are made, you'll find people still using this feature - polishing reviews to get sent to authors. For this reason, we've omitted the typical "confidential comments" part of the review form. If a reviewer wants to hold some portion of the comments in confidence, after the decisions are made, he/she can edit that portion out.
- Review forms are produced automatically. Whenever a reviewer needs a form, it gets created and uploaded on demand, with most of the bookkeeping info filled in automatically, by the START tools.
- Reviews are submitted using the Web form-upload protocol. Each review is given a unique ID, and is placed in the directory reviewData.
- Note that these reviews should NOT be hand edited by you; nor should they get moved elsewhere. They are required for online updating/editing, and for generating final reports.
- You can keep tabs on PC productivity by running the script:
which lists the number of reviews done/outstanding for each PC member.
- At some point in the review process, you may have a "vetting process," during which a reviewer of paper X will be allowed to see all other reviews for paper X. This is essential for on-line PC meetings, and it is supported by START. To enable this feature, go to parameters.pl, and set $ViewAll to 1. The result: when PC members check their assignments, they will now have this option.
At any point during or after the review process, you can generate a set of reports summarizing the results. There are two programs for this, both of which are in the tools directory.
- genReports: This is a multi-purpose report program; it does a lot of work, so don't get scared if it takes a long time to finish, e.g., 20 seconds for about 700 reviews, on a Sparc Ultra. The main input you need for this program is the following file:
Due the sensitivity of the review information, genReport will not run at all unless such a file is present. It contains a line-by-line list of papers written by PC members, which genReports uses it to produce a "sanitized" comprehensive report for your committee, which can be handed out at a meeting.
You'll find a sample blockList in the reports directory - delete the lines, and fill in the lines your wish to block. Using this file (and the review log files), genReport produces the following information:
- reports/Summaries/*.html: Individual summaries for all papers, 1 file created for each paper, with all reviews shown.
- reports/pcReport.html: Report for PC, with the PC member papers excluded.
- reports/master.html: Same as pcReport.html, except all papers are shown.
- reports/Conflicts/*.html: Individual summaries for conflicts; while these are in Summaries as well, it is helpful to have these separate, to print copies to hand out at PC meeting.
- reports/summary.html: Summary list of papers & scores, with statistics.
- reports/stats.txt: Colon-separated list of review scores, with avgs, variances - which importable into Excel or Access.
extractScores: This tool is more basic - it simply puts reviewer scores into colon-separated records; all info is included except comments. It is suitable for importing the raw review records into Excel, Access, or other common statistical tools.
NOTE: None of this report information is published on-line. Don't worry about its security - it is private, accessible only by you. If you want to link it to your PC web page, you must copy it to www/PC yourself, and put the appropriate hyperlinks into your PC home page.
Finally, you've accepted your papers for the program. Now it's time to wrap things up.
- Enter the PC results into 2 files:
- reports/acceptList: ALL accepted papers, including shepherded papers, are entered in this file. Put one paper ID on each line, the ID corresponding to that of an accepted paper. Be sure to double-check this list against your results from the PC decisions. Then do it again, and count the number. This is one job you don't want to screw up. If you manage to send a reject letter for an accepted paper, it's not the end of the world - it's a correctable problem. On the other hand, you surely don't want to send an accept letter for a rejected paper; then you'll have a major problem on your hands. So be careful.
NOTE: Any paper not on this list is considered rejected
- reports/shepherdList: If you have shepherded papers, this list contains a subset of the acceptList above. Each line has the following form:
paperID ShepherdFirstName ShepherdLastName ShepherdEmail
This list is used to have the authors and shepherds contact each other. Make sure you edit this list correctly, or just leave it blank if you don't have any shepherding.
Edit the letter templates accept.txt, reject.txt, and shepherd.txt, in templates/letters. (Actually, you may not need to, since the letters are fairly generic.) The following variables can be used
in the letters; see the templates for details.
Run genLetters, and the letters are
produced, with comments and scores appended. For
safety sake, the letters are not sent - rather,
they are put in reports/Letters as files. Check
them, and see if they came out OK. If so, do the
following: go to the directory reports,
where you'll find a 3-line script to mail them all out.
What's done is done.
first name of shepherd, if any
last name of shepherd, if any
contact address for shepherd, if any
PC Chair's name, as signature
Acronym/yr of conference
Finally, run genAccept, which produces an HTML list of accepted papers for external publication. It goes in reports/accepted.html, as well reports/Abstracts, a directory containing all the paper abstracts. Copy both these to your External web directory, and then link them to the conference's public home page - at which point the world can see it.
Run genAuthorList, which produces the formatted author contact
information for your conference's publisher (or for you). This information is placed in reports/authorList.txt,
and you can then send it to IEEE/ACM, etc.
Get out of your chair, stand up, and scream. You're done.
Copyright © 1998-2000. All Rights Reserved
We provide the Submission Tracking And Review Toolset (below described as "START") on an AS IS basis, and do not warrant its validity or performance. We reserve the right to update, modify, or discontinue this software at any time. We shall have no obligation to supply such updates or modifications or any other form of support to you.
This license is for personal use. For such uses, there is no charge. We define "personal use" to mean you may used the software within your organization, for one conference. You may not re-distribute START or parts of START, in any form source or binary (including derivatives), electronic or otherwise, to any other organization or entity without our permission. All warranties, including without limitation, any warranty of merchantability or fitness for a particular purpose, are hereby excluded.
By your use of START, you understand and agree that we (or any other person or entity with proprietary rights in START) are under no obligation to provide either maintenance services, update services, notices of latent defects, or correction of defects for START. Even if advised of the possibility of such damages, under no circumstances shall we (or any other person or entity with proprietary rights in the software licensed hereunder) be liable to you or any third party for direct, indirect, or consequential damages of any character regardless of type of action, including, without limitation, loss of profits, loss of use, loss of good will, or computer failure or malfunction. You agree to indemnify us (and any other person or entity with proprietary rights in the software licensed hereunder) for any and all liability it may incur to third parties resulting from your use of START.
This software is released on a controlled basis. Permission
for using START is granted for a single conference or workshop only, unless explicitly obtained for multiple events. The START logo appears on auto-generated forms, reports and HTML pages; this logo may not be removed or altered, unless prior consent is obtained from the authors. The system is released AS IS, without support, and may be modified as desired. Bug reports are solicited, as are suggestions for upgraded functionality. However, we cannot guarantee that the repairs or upgrades will be made.
14 Jan 2000 at 12:00:00