Mathematical problems in computation theory