The short answer is: you should do it at least once a week. The skimmer baskets collect large pieces of foreign objects in the pool, such as leaves, twigs, toys, etc. Especially during fall, you can expect there to be more leaves in your pool.
When left unnoticed for some time, smaller objects like balls or goggles that get trapped in the skimmer baskets reduce the efficiency of your pump. Ideally, it is advisable to clean them out twice a week when it is in use.