High-dimensional crowdsourced data collected from a large number of users may produc3 rich knowledge for our society but also bring unprecedented privacy threats to participants. Recently differential privacy has been proposed as an effective means to mitigate privacy concerns. However, existing work on differential privacy suffers from the 'curse of high-dimensionality' (data with multiple attributes) and high scalability (data with large scale records). Moreover, traditional methods of differential privacy were achieved via aggregation results, which cannot guarantee local privacy for distributed users in crowdsourced systems. To deal with these issues, in this paper we propose a novel scheme that can efficiently estimate multivariate joint distribution for high-dimensional data with local privacy. On the client side, we employ randomized response techniques to locally transform data from distributed users into privacy-preserving bit strings, which can prevent potential inside privacy attacks in crowdsourced systems. On the server side, the crowdsourced bit strings are aggregated for multivariate distribution estimation. Specifically, we first propose a multivariate version of the expectation maximization (EM) based algorithm to estimate the joint distribution of high dimensional data. To speed up the performance, unlike the EM-based method that needs to scan each user's bit string, we propose to use Lasso regression to obtain the distribution estimation from the aggregation information only once, which can significantly reduce the computation time for multivariate distribution estimation. Extensive experiments on real-world datasets demonstrate the efficiency of our multivariate distribution estimation scheme over existing estimation schemes.