You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
| inputFilePath | Function | The path to the CSV file to stream from. |
15
-
| transformer | Function | <p>The transformer function to apply to each row.</p><p>Note that each row of the CSV file is passed as a DataFrame with a single row to the transformer function, and the transformer function is expected to return a transformed DataFrame.</p> |
15
+
| transformer | Function | <p>The transformer function to apply to each row.</p><p>Note that each row of the CSV file is passed as a DataFrame with a single row to the transformer function, and the transformer function is expected to return a transformed DataFrame.</p>|
16
16
| options | object | <p>Configuration options for the pipeline. These include:</p><ul><li><code>outputFilePath</code> The local file path to write the transformed CSV file to.</li><li><code>customCSVStreamWriter</code> A custom CSV stream writer function. This is applied at the end of each transform. If not provided, a default CSV stream writer is used, and this writes to local storage.</li></ul><ul><li><code>inputStreamOptions</code> Configuration options for the input stream. Supports all Papaparse CSV reader config options.</li><li><code>outputStreamOptions</code> Configuration options for the output stream. This is only applied when using the default CSV stream writer. Supports all <code>toCSV</code> options.</li></ul> |
17
17
18
18
**Returns:**
19
19
20
-
> return A promise that resolves when the pipeline transformation is complete.
20
+
> A promise that resolves when the pipeline transformation is complete.
21
21
22
22
The streamCsvTransformer can be used to [incrementally transform](https://en.wikipedia.org/wiki/Stream\_processing) a CSV file. This is done by:
23
23
24
24
* Streaming a CSV file from a local or **remote** path.
25
25
* Passing each corresponding row as a DataFrame to the specified transformer function.
26
-
* Writing the result to an output stream.  
27
-
28
-
26
+
* Writing the result to an output stream.
29
27
30
28
## **Stream processing a local file**
31
29
32
-
In the example below, we stream a local CSV file (titanic.csv), applies a transformer function, and write the output to the `titanicOutLocal` file.
30
+
In the example below, we stream a local CSV file (titanic.csv), apply a transformer function, and write the output to **`titanicOutLocal.csv`**.
33
31
34
-
The transformer takes each `Name` column, splits the person's title, and creates a new column from it. 
32
+
The transformer takes each `Name` column, splits the person's title, and creates a new column from it.
If you need custom control of the output writer, then you can provide a pipe-able custom writer. See [https://www.freecodecamp.org/news/node-js-streams-everything-you-need-to-know-c9141306be93/](https://www.freecodecamp.org/news/node-js-streams-everything-you-need-to-know-c9141306be93/)
123
121
124
-
In the example below, we add a custom writer that logs each row. You can extend this to upload each chunk to a database, or any other function you need. 
122
+
In the example below, we add a custom writer that logs each row. You can extend this to upload each chunk to a database, or any other function you need.
Copy file name to clipboardExpand all lines: examples/migrating-to-the-stable-version-of-danfo.js.md
+4-4
Original file line number
Diff line number
Diff line change
@@ -13,7 +13,7 @@ The following list summarizes some of these updates:
13
13
***Typescript support:** This new version has been completely re-written using Typescript. This means we now have well-defined types that increase the developer experience.
14
14
***Standard naming convention:** Functions, methods, classes, and variable names have been standardized to follow JavaScript best practices.
15
15
* Standardize function argument: Functions and methods have been updated to accept arguments and parameters intuitively resulting in improved developer experience.
16
-
***New features**: We added a couple of new features which users have been requesting for. For example:
16
+
***New features**: We added lots of new features which users have been requesting for. For example:
17
17
* Stream and process large CSV data
18
18
* General bug fixes and improvements
19
19
* Better error messages
@@ -28,19 +28,19 @@ Update your function and/or method names to use camelCase instead of snake\_case
28
28
29
29
```javascript
30
30
read_csv ==> readCSV
31
-
to_json ==>toJSON
31
+
to_json ==>toCSV
32
32
drop_duplicates ==> dropDuplicates
33
33
```
34
34
35
-
**Note:** that your code editor will generally suggest the new function or method names, so it will be easier to update to the new names. If using Typescript, then it is even easier as the TS compiler will show you warnings.
35
+
**Note:** that your code editor auto-complete will general suggest the new function or method names, so it will be easier to update to the new names. If using Typescript, then it is even easier as the TS compiler will show you warnings.
36
36
37
37
### Functions and Methods argument structure
38
38
39
39
Another major breaking change of v1, is that the structure of arguments/parameters in most functions has been updated to be more intuitive.
40
40
41
41
In general, it is important to understand our thought process behind this, so, here goes:
42
42
43
-
Assuming we take the method called _**rename**_, which takes the required argument **mapper**, as well as, an optional configuration argument. In pre-v1 version of Danfojs, the function signature is as follows:
43
+
Assuming we take the method called _**rename**_, which takes required object mapper, as well as, optional configuration argument. In pre-v1 version of Danfojs, the function signature is as follows:
0 commit comments