The selection process has two general phases - review and selection. The review team consists of approximately 10 people with a range of backgrounds, focus areas, and familiarity with Strange Loop. As soon as the CFP closes, the reviewers begin rating talks (we usually have about 400 submissions). Reviewing is done blind - reviewers see only talk info, not speaker info.
Not every reviewer rates every talk but in general, most talks get rated at least 6 times. The result of the review process is a set of ratings for every talk and an aggregate rating. The aggregate ratings are a good general measure of interest and we'll often use that to sort talks early in the process. Reviewer comments are also used throughout as we compare talks.
The rating info is used as fodder for the selection process, which is done by the organizers. Selection involves taking the giant pool of submissions and reducing it to the actual conference program. There are a variety of ways we evaluate talks:
- Invited talks pass through
- Talks below a certain aggregate rating without any high votes are given a no. Talks with a low average but at least one high rating are kept for more consideration.
- Talks ruled as out of scope are given a no regardless of ranking. In some cases, we do these immediately after submission before the review process even begins.
- Talks are reviewed ranked, per track, to rule more talks in or out
- Talks are reviewed per speaker - we typically only accept one talk per speaker. We may leave more than one in for a while as we assess balance.
- Talks are compared based on topic balance (languages, databases, etc)
- Talks are checked per company to avoid too many from a single company
- Overall program is evaluated for a balance of concept talks vs deep technical talks vs fun/inspiring talks.
- Inevitably we narrow to a list of very good talks and start to make the really hard choices to subjectively build the best program.
Some reasons a talk might not be selected include:
- Abstract or comments did not contain enough detail - in this case, we typically say no even if it might be a good talk. Due to the volume of talks and the short time scale we work on, we can't go back to every speaker to request a better abstract. A good abstract should a) capture a reader's interest and b) hint at what you will learn. The reviewer comments should explain to reviewers exactly what topics and technologies you expect to cover (even if they may change by the time you do the talk). If we read both and still don't know what it's about, we're probably going to say no.
- Out of scope - there is a list on the CFP page about which topics we are more or less likely to accept.
- Accepted different talk by the same speaker - as mentioned above, we're probably only going to take one talk from a speaker. The CFP warns not to submit more than two and fortunately most people do follow that advice.
- Speaker spoke last year - sometimes we turn down good talks by good speakers just because they've spoken at Strange Loop recently. This is a judgement call and sometimes a very difficult one but it's important to keep the program fresh.
- Talk is substantially similar to existing one on video - with the frequency of video release for conferences, we try to focus on primarily new or revised content for talks that have not yet been widely released. This is not a hard rule however.
- Talk is an ad for a product - if your talk reads like product marketing, it's much less likely to be accepted than if it covers a new idea or technique.
- Balance - every year we turn down hundreds of excellent talks by excellent speakers in the service of creating a balanced program (by track, topic, language, style, etc). The majority of the talks submitted to Strange Loop are interesting enough to be included, but we just can't include them all.
The review phase is blind, so matters of gender/race/company/sponsor affiliation/speaking experience are not considered. In the selection phase, we do consider speaker ability when comparing talks on a particular topic. We also consider company in the interest of balance. We don't explicitly consider other speaker factors although in a close call we may lean on the side of increased diversity. In general, the majority of the selection decision will be based on the content of the talk.