The short answer is: you should do it at least once a week. The skimmer baskets collect large pieces of foreign objects in the pool, such as leaves, twigs, toys, etc. Especially during fall, there will be more leaves in the pool.
When left unnoticed for a while, smaller objects like balls or goggles that get trapped in the skimmer baskets reduce the efficiency of your pump. Ideally, it is advisable to clean them out twice a week in the swimming season.