|
| 1 | +--- |
| 2 | +title: Copy data from MongoDB Atlas |
| 3 | +description: Learn how to copy data from MongoDB Atlas to supported sink data stores by using a copy activity in an Azure Data Factory pipeline. |
| 4 | +services: data-factory |
| 5 | +documentationcenter: '' |
| 6 | +author: linda33wj |
| 7 | +ms.author: jingwang |
| 8 | +manager: shwang |
| 9 | +ms.reviewer: douglasl |
| 10 | +ms.service: data-factory |
| 11 | +ms.workload: data-services |
| 12 | +ms.topic: conceptual |
| 13 | +ms.custom: seo-lt-2019; seo-dt-2019 |
| 14 | +ms.date: 09/28/2020 |
| 15 | +--- |
| 16 | + |
| 17 | +# Copy data from MongoDB Atlas using Azure Data Factory |
| 18 | + |
| 19 | +[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)] |
| 20 | + |
| 21 | +This article outlines how to use the Copy Activity in Azure Data Factory to copy data from a MongoDB Atlas database. It builds on the [copy activity overview](copy-activity-overview.md) article that presents a general overview of copy activity. |
| 22 | + |
| 23 | +## Supported capabilities |
| 24 | + |
| 25 | +You can copy data from MongoDB Atlas database to any supported sink data store. For a list of data stores that are supported as sources/sinks by the copy activity, see the [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats) table. |
| 26 | + |
| 27 | +Specifically, this MongoDB Atlas connector supports **versions up to 4.2**. |
| 28 | + |
| 29 | +## Prerequisites |
| 30 | + |
| 31 | +If you use Azure Integration Runtime for copy, make sure you add the effective region's [Azure Integration Runtime IPs](azure-integration-runtime-ip-addresses.md) to the MongoDB Atlas IP Access List. |
| 32 | + |
| 33 | +## Getting started |
| 34 | + |
| 35 | +[!INCLUDE [data-factory-v2-connector-get-started](../../includes/data-factory-v2-connector-get-started.md)] |
| 36 | + |
| 37 | +The following sections provide details about properties that are used to define Data Factory entities specific to MongoDB Atlas connector. |
| 38 | + |
| 39 | +## Linked service properties |
| 40 | + |
| 41 | +The following properties are supported for MongoDB Atlas linked service: |
| 42 | + |
| 43 | +| Property | Description | Required | |
| 44 | +|:--- |:--- |:--- | |
| 45 | +| type |The type property must be set to: **MongoDbAtlas** |Yes | |
| 46 | +| connectionString |Specify the MongoDB Atlas connection string e.g. `mongodb+srv://<username>:<password>@<clustername>.<randomString>.<hostName>/<dbname>?<otherProperties>`. <br/><br /> You can also put a connection string in Azure Key Vault. Refer to [Store credentials in Azure Key Vault](store-credentials-in-key-vault.md) with more details. |Yes | |
| 47 | +| database | Name of the database that you want to access. | Yes | |
| 48 | +| connectVia | The [Integration Runtime](concepts-integration-runtime.md) to be used to connect to the data store. Learn more from [Prerequisites](#prerequisites) section. If not specified, it uses the default Azure Integration Runtime. |No | |
| 49 | + |
| 50 | +**Example:** |
| 51 | + |
| 52 | +```json |
| 53 | +{ |
| 54 | + "name": "MongoDbAtlasLinkedService", |
| 55 | + "properties": { |
| 56 | + "type": "MongoDbAtlas", |
| 57 | + "typeProperties": { |
| 58 | + "connectionString": "mongodb+srv://<username>:<password>@<clustername>.<randomString>.<hostName>/<dbname>?<otherProperties>", |
| 59 | + "database": "myDatabase" |
| 60 | + }, |
| 61 | + "connectVia": { |
| 62 | + "referenceName": "<name of Integration Runtime>", |
| 63 | + "type": "IntegrationRuntimeReference" |
| 64 | + } |
| 65 | + } |
| 66 | +} |
| 67 | +``` |
| 68 | + |
| 69 | +## Dataset properties |
| 70 | + |
| 71 | +For a full list of sections and properties that are available for defining datasets, see [Datasets and linked services](concepts-datasets-linked-services.md). The following properties are supported for MongoDB Atlas dataset: |
| 72 | + |
| 73 | +| Property | Description | Required | |
| 74 | +|:--- |:--- |:--- | |
| 75 | +| type | The type property of the dataset must be set to: **MongoDbAtlasCollection** | Yes | |
| 76 | +| collectionName |Name of the collection in MongoDB Atlas database. |Yes | |
| 77 | + |
| 78 | +**Example:** |
| 79 | + |
| 80 | +```json |
| 81 | +{ |
| 82 | + "name": "MongoDbAtlasDataset", |
| 83 | + "properties": { |
| 84 | + "type": "MongoDbAtlasCollection", |
| 85 | + "typeProperties": { |
| 86 | + "collectionName": "<Collection name>" |
| 87 | + }, |
| 88 | + "schema": [], |
| 89 | + "linkedServiceName": { |
| 90 | + "referenceName": "<MongoDB Atlas linked service name>", |
| 91 | + "type": "LinkedServiceReference" |
| 92 | + } |
| 93 | + } |
| 94 | +} |
| 95 | +``` |
| 96 | + |
| 97 | +## Copy activity properties |
| 98 | + |
| 99 | +For a full list of sections and properties available for defining activities, see the [Pipelines](concepts-pipelines-activities.md) article. This section provides a list of properties supported by MongoDB Atlas source. |
| 100 | + |
| 101 | +### MongoDB Atlas as source |
| 102 | + |
| 103 | +The following properties are supported in the copy activity **source** section: |
| 104 | + |
| 105 | +| Property | Description | Required | |
| 106 | +|:--- |:--- |:--- | |
| 107 | +| type | The type property of the copy activity source must be set to: **MongoDbAtlasSource** | Yes | |
| 108 | +| filter | Specifies selection filter using query operators. To return all documents in a collection, omit this parameter or pass an empty document ({}). | No | |
| 109 | +| cursorMethods.project | Specifies the fields to return in the documents for projection. To return all fields in the matching documents, omit this parameter. | No | |
| 110 | +| cursorMethods.sort | Specifies the order in which the query returns matching documents. Refer to [cursor.sort()](https://docs.mongodb.com/manual/reference/method/cursor.sort/#cursor.sort). | No | |
| 111 | +| cursorMethods.limit | Specifies the maximum number of documents the server returns. Refer to [cursor.limit()](https://docs.mongodb.com/manual/reference/method/cursor.limit/#cursor.limit). | No | |
| 112 | +| cursorMethods.skip | Specifies the number of documents to skip and from where MongoDB Atlas begins to return results. Refer to [cursor.skip()](https://docs.mongodb.com/manual/reference/method/cursor.skip/#cursor.skip). | No | |
| 113 | +| batchSize | Specifies the number of documents to return in each batch of the response from MongoDB Atlas instance. In most cases, modifying the batch size will not affect the user or the application. Cosmos DB limits each batch cannot exceed 40MB in size, which is the sum of the batchSize number of documents' size, so decrease this value if your document size being large. | No<br/>(the default is **100**) | |
| 114 | + |
| 115 | +>[!TIP] |
| 116 | +>ADF support consuming BSON document in **Strict mode**. Make sure your filter query is in Strict mode instead of Shell mode. More description can be found at [MongoDB manual](https://docs.mongodb.com/manual/reference/mongodb-extended-json/index.html). |
| 117 | +
|
| 118 | +**Example:** |
| 119 | + |
| 120 | +```json |
| 121 | +"activities":[ |
| 122 | + { |
| 123 | + "name": "CopyFromMongoDbAtlas", |
| 124 | + "type": "Copy", |
| 125 | + "inputs": [ |
| 126 | + { |
| 127 | + "referenceName": "<MongoDB Atlas input dataset name>", |
| 128 | + "type": "DatasetReference" |
| 129 | + } |
| 130 | + ], |
| 131 | + "outputs": [ |
| 132 | + { |
| 133 | + "referenceName": "<output dataset name>", |
| 134 | + "type": "DatasetReference" |
| 135 | + } |
| 136 | + ], |
| 137 | + "typeProperties": { |
| 138 | + "source": { |
| 139 | + "type": "MongoDbAtlasSource", |
| 140 | + "filter": "{datetimeData: {$gte: ISODate(\"2018-12-11T00:00:00.000Z\"),$lt: ISODate(\"2018-12-12T00:00:00.000Z\")}, _id: ObjectId(\"5acd7c3d0000000000000000\") }", |
| 141 | + "cursorMethods": { |
| 142 | + "project": "{ _id : 1, name : 1, age: 1, datetimeData: 1 }", |
| 143 | + "sort": "{ age : 1 }", |
| 144 | + "skip": 3, |
| 145 | + "limit": 3 |
| 146 | + } |
| 147 | + }, |
| 148 | + "sink": { |
| 149 | + "type": "<sink type>" |
| 150 | + } |
| 151 | + } |
| 152 | + } |
| 153 | +] |
| 154 | +``` |
| 155 | + |
| 156 | +## Export JSON documents as-is |
| 157 | + |
| 158 | +You can use this MongoDB Atlas connector to export JSON documents as-is from a MongoDB Atlas collection to various file-based stores or to Azure Cosmos DB. To achieve such schema-agnostic copy, skip the "structure" (also called *schema*) section in dataset and schema mapping in copy activity. |
| 159 | + |
| 160 | +## Schema mapping |
| 161 | + |
| 162 | +To copy data from MongoDB Atlas to tabular sink, refer to [schema mapping](copy-activity-schema-and-type-mapping.md#schema-mapping). |
| 163 | + |
| 164 | +## Next steps |
| 165 | +For a list of data stores supported as sources and sinks by the copy activity in Azure Data Factory, see [supported data stores](copy-activity-overview.md#supported-data-stores-and-formats). |
0 commit comments