Federated Learning (FL) has been drawing significant attention from both academia and industry working on distributed machine learning. In practice, learning over mutually isolated datasets residing at the network edge, also known as silos, FL clients can suffer from a lack of samples, due to many reasons (e.g., expensive annotation), and this has potentially significant negative impact on FL performance. Few-Shot Learning (FSL) has been considered as a promising solution, but unfortunately cannot be directly applied to practical Cross-Silo Federated Learning (CSFL) systems. In this article, as far as we know, we conduct the first systematic discussion of the specific challenges of FSL in CSFL systems. We extract essential design issues found in Federated Few-Shot Learning (FFSL), and develop a new FFSL method based on Model-Agnostic Meta Learning (MAML). Through experiments using real-world federated datasets, we comprehensively demonstrate our method's advantages over existing FL and FSL methods in different practical CSFL scenarios where hitherto FL and FSL methods failed. We also highlight some promising future research directions.