Data engineering (streaming ETL, a Parquet lakehouse, a join engine), databases (a SPARQL store, SQL change replay), languages (a bytecode VM, function layout) and systems (an embedded graphics renderer)
This sample task is provided for illustration only. Each model's trajectory is its best of eight attempts on this task, so it may not reflect its overall performance within this domain. Domain scores average every task in the domain.
The task
Make an exact join-analytics engine use less CPU time. It reads one database and a batch of conjunctive queries on standard input and writes grouped aggregates to standard output. The deliverable is a static Linux ELF64 executable of at most 256 MiB, at /app/engine. Each database has eight relations of (left, right, payload) rows; duplicates count separately. A query is a connected graph of variables, each with an interval filter, whose edges are labeled by relations. For every joined row, COUNT adds 1 and SUM adds the selected payloads, in unsigned 64-bit arithmetic, into one of 8 buckets set by the grouping variable. Each timed run is a fresh process on one CPU with a 512 MiB address space, so start-up and parsing count. Scoring uses a small and a large private database from each of six data families, and answer checks also cover valid inputs outside the generator. Any wrong answer scores 0; otherwise each database is scored by its speedup over the starter against the target below.
Speedup that earns full credit, by database
Family
Small
Large
Uniform
6.6×
15.2×
Hubs
5.0×
12.3×
Communities
9.2×
22.3×
Correlated
8.3×
23.6×
Selective
20.9×
68.6×
Mixed
6.1×
16.0×
What the agent starts with
The agent gets the working indexed C++ starter, a build helper for optimized static binaries, and a specification of the input and output protocol, the input bounds and each database's scoring target. It also gets the published workload generator and sandbox code for public development data (scoring uses private databases), an evaluator for public feedback on any family, and 4 CPUs, 8 GB, g++ and public internet access.
