Upon immune activation, chloroplasts switch off photosynthesis, produce anti-microbial compounds, and associate with the nucleus through tubular extensions called stromules. Although it is well-established that chloroplasts alter their position in response to light, little is known about the dynamics of chloroplasts movement in response to pathogen attack. Here, we report that chloroplasts accumulate at the pathogen interface during infection by the Irish potato famine pathogen Phytophthora infestans, associating with the specialized membrane that engulfs the pathogen haustorium. Chemical inhibition of actin polymerization reduces the accumulation of chloroplasts at the pathogen haustoria, suggesting this process is partially dependent on the actin cytoskeleton. However, chloroplast accumulation at haustoria does not necessarily rely on movement of the nucleus to this interface and is not affected by light conditions. Stromules are typically induced during infection, embracing haustoria and facilitating chloroplast interactions, to form dynamic organelle clusters. We found that infection-triggered stromule formation relies on BRASSINOSTEROID INSENSITIVE 1-ASSOCIATED KINASE 1 (BAK1) mediated surface immune signaling, whereas chloroplast repositioning towards haustoria does not. Consistent with the defense-related induction of stromules, effector mediated suppression of BAK1 mediated immune signaling reduced stromule formation during infection. On the other hand, immune recognition of the same effector stimulated stromules, presumably via a different pathway. These findings implicate chloroplasts in a polarized response upon pathogen attack and point to more complex functions of these organelles in plant-pathogen interactions.