The Department of Justice's inspector general has declined to investigate the handling of the criminal case against a former Olympian cleared of accusations that he intentionally damaged the Lincoln Memorial Reflecting Pool, according to a letter Wednesday from the man's lawyers.