Development feature build with localPath functionality #175
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…elease. remove unsigned nupkg from artifacts.
…FabricData_Public. set develop pipeline to always publish to Tools/CollectServiceFabricData
…FabricData_Public. set develop pipeline to always publish to Tools/CollectServiceFabricData
…cSource in nuspec. tested
…lity dependency updates configuration updates for jarvis security vulnerability dependency updates ado pipeline updates
… days CheckForUpdates. default 30 days. #25
…istencePath and LogPath
…nt in response to PR comments
… of https://servicefabricsupport.visualstudio.com/Tools/_git/CollectServiceFabricData into t-vchin-UpdateConfigLocalIngest
… azure client id as it is not supported block net462 and kusto federated authentication without azure client id as it is not supported remove runtime property IsARMValid when saving configuration add additional vscode launch configurations for net462 and net48
…rol commands are limited by ingest cluster capacity add lock for kusto endpoint CommandAsync as ingest control commands are limited by ingest cluster capacity https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/capacity-policy#management-commands-throttling seen throttled exception in ingest command for kusto emulator "code": "Too many requests", "message": "Request is denied due to throttling. " "@type": "Kusto.DataNode. Exceptions.ControlCommandThrottledException
I made changes to the CollectServiceFabricData tool to support local ingestion. My first main changes involve adding 2 new configuration field options, "LocalPath" and "OverwriteTable." "LocalPath" is the field that contains a path to a folder where the original, unmodified trace files are put. If this has a value, then that indicates that the user wants to do local ingestion (depending on what we decide with time stamp filtering, this could change). "OverwriteTable" is a boolean indicating whether to append or replace the data in the table the user is ingesting traces into. A value of "true" indicates that the table will be cleared before ingesting new data. A value of "false" (default) will just add new traces to the table without replacement. For local ingestion, I am utilizing the preexisting "KustoCluster" field to take in a local web server url with a database name (example: http://localhost:8080/MyDBName). To enable local ingestion, you must provide a url like the example in the "KustoCluster" field and a path in the "LocalPath" field in your collectsfdata.options.json file. I created/edited functions in KustoConnection.cs and KustoEndpoint.cs to support connecting to the Kusto emulator, creating a new database/table if they don't exist, and ingesting traces after they've been formatted. Rather than making use of an ingestion queue, my local ingestion process executes KQL commands to directly ingest data into a table. At the end, you should be able to use the Kusto Explorer to view and evaluate traces. What I tested: - local ingestion with one short trace file (5 rows) with 'OverrideTable' set to false - local ingestion with one short trace file (5 rows) with 'OverrideTable' set to true  (image shows example of ingested data in table) - local ingestion with 2-10 trace files (the short one from above and a large one (CreateSelfReconfiguringService.test.trace) with 35,622 rows) "OverrideTable" set to true. Clears table and uploads correct number of rows. - local ingestion with the 2-10 trace files with "OverrideTable" set to false. Successfully adds to the table - local ingestion with 2-10 trace files with no DB / table already existing. They are created successfully, and ingestion continues - local ingestion with 5 trace files with no "overrideTable" field in config file - sucessfully ingests into new table - Attempting local ingestion without providing a "cachelocation" value in my configuration file - successfully gave the user an error and in the message says to provide a value. Safely exits the program. - Attempting local ingestion with a localhost url that doesn't match my pattern in the 'KustoCluster' field of my configuration file. Ex: "http://localhost:808000/MyDBName". The program safely exits and provides appropriate...
I added changes primarily in Collector.cs (UploadLocalData() function and other helper functions) to grab files from the LocalPath field location specified in the configuration file. Then go through those files and filter out those whose 'LastModified' time does not fit the StartTimeStamp/EndTimeStamp times. Then, with the files that fit the time range, I created a copy of those files and put them in the CacheLocation path specified in the configuration file. The files in the CacheLocation are used later for formatting and local ingestion. I added changes to the other files to support the LocalPath field and allow you to test the code I put in the Collector.cs file. Those changes are the same as what is in my prior PR but doesn't include all of them. I only added what was necessary to reach the new functionality I described above and test that the file copies were created in the CacheLocation. You won't be able to actually completely do local ingestion with my changes in this PR, only in the prior PR.
github-actions
bot
added
documentation
dbucce
approved these changes
Jul 29, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Development feature build with localPath functionality
remove unused dependencies
update dependencies for security vulnerabilities
updates for jarvis compatibility