Agent Memory Is the Graph