Skip to content

feat: add built-in caching #2082

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
bafa5f0
Remove 'vector_math' dependency
JaffaKetchup Apr 30, 2025
cfd187b
Initial caching implementation
JaffaKetchup May 3, 2025
6007304
Improved performance of persistent registry writer & format efficiency
JaffaKetchup May 4, 2025
6fe737c
Refactored to re-enable web support
JaffaKetchup May 4, 2025
25ebb25
Improved startup performance by not waiting for caching instance
JaffaKetchup May 4, 2025
b79cb07
Exposed `MapTileCachingManager` & `CachedMapTileMetadata`
JaffaKetchup May 4, 2025
92c37cb
Added `MapCachingOptions.cacheKeyGenerator`
JaffaKetchup May 4, 2025
985ba4c
Merge branch 'master' into caching
JaffaKetchup May 9, 2025
c1a8bb6
Reverted to waiting for manager instance to be created before returni…
JaffaKetchup May 10, 2025
bd5dfe7
Added size monitor
JaffaKetchup May 14, 2025
2c38eb8
Major refactoring & renaming
JaffaKetchup May 15, 2025
e9c4465
Fixed linting issue
JaffaKetchup May 15, 2025
e67f7e4
Improved speed and efficiency of size limiter
JaffaKetchup May 15, 2025
36d7268
Removed leftover debugging tools
JaffaKetchup May 15, 2025
5fe9133
Improved documentation
JaffaKetchup May 17, 2025
192860c
Switch to FlatBuffers instead of JSON
JaffaKetchup May 18, 2025
9321d0c
Return number of tiles loaded from `isInitialised`
JaffaKetchup May 19, 2025
fd7a79b
Added call to `WidgetsFlutterBinding.ensureInitialized` internally
JaffaKetchup May 19, 2025
0997c70
Drop default cache size limit to 800 MB
JaffaKetchup May 19, 2025
5547146
Minor general improvements
JaffaKetchup May 19, 2025
db50851
Replace `catch (_)` with `on Exception`
JaffaKetchup May 19, 2025
92452f6
Switched back to JSON from FlatBuffers
JaffaKetchup May 20, 2025
92a60b1
Fixed size limiter
JaffaKetchup May 20, 2025
3fe5adc
Minor improvement
JaffaKetchup May 20, 2025
6078fe2
Fixed minor bug in example app
JaffaKetchup May 22, 2025
37da3d4
Improved network tile image provider
JaffaKetchup May 23, 2025
c4b2bf4
Merge branch 'master' into caching
JaffaKetchup May 25, 2025
2548269
Replace the JSON registry with a mechanism to store metadata within t…
JaffaKetchup May 27, 2025
56bb2a6
Discard changes to example/lib/main.dart
JaffaKetchup May 27, 2025
5c4899b
Fixed bug
JaffaKetchup May 27, 2025
53e584f
Improved resilience to corruption
JaffaKetchup May 28, 2025
16c1827
Improved resilience to corruption
JaffaKetchup May 29, 2025
5f85c71
Fix unintended change
JaffaKetchup May 29, 2025
3b350dc
Fixed minor bug
JaffaKetchup May 31, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions example/lib/misc/tile_providers.dart
Original file line number Diff line number Diff line change
@@ -1,10 +1,15 @@
import 'package:flutter_map/flutter_map.dart';
import 'package:flutter_map_cancellable_tile_provider/flutter_map_cancellable_tile_provider.dart';
import 'package:http/http.dart';
import 'package:http/retry.dart';
//import 'package:flutter_map_cancellable_tile_provider/flutter_map_cancellable_tile_provider.dart';

final httpClient = RetryClient(Client());

TileLayer get openStreetMapTileLayer => TileLayer(
urlTemplate: 'https://tile.openstreetmap.org/{z}/{x}/{y}.png',
userAgentPackageName: 'dev.fleaflet.flutter_map.example',
// Use the recommended flutter_map_cancellable_tile_provider package to
// support the cancellation of loading tiles.
tileProvider: CancellableNetworkTileProvider(),
// TODO: change
tileProvider: NetworkTileProvider(httpClient: httpClient),
);
13 changes: 10 additions & 3 deletions example/lib/widgets/drawer/menu_drawer.dart
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,13 @@ class MenuDrawer extends StatelessWidget {
return Drawer(
child: ListView(
children: <Widget>[
DrawerHeader(
Container(
padding: const EdgeInsets.fromLTRB(16, 32, 16, 16)
.add(EdgeInsets.only(top: MediaQuery.paddingOf(context).top)),
margin: const EdgeInsets.only(bottom: 8),
decoration: BoxDecoration(
border: Border(bottom: Divider.createBorderSide(context)),
),
child: Column(
mainAxisAlignment: MainAxisAlignment.center,
children: [
Expand All @@ -65,11 +71,12 @@ class MenuDrawer extends StatelessWidget {
textAlign: TextAlign.center,
style: TextStyle(fontSize: 14),
),
const SizedBox(height: 8),
if (kIsWeb)
const Text(
Text(
_isWASM ? 'Running with WASM' : 'Running without WASM',
textAlign: TextAlign.center,
style: TextStyle(fontSize: 14),
style: Theme.of(context).textTheme.bodySmall,
),
],
),
Expand Down
58 changes: 57 additions & 1 deletion example/pubspec.lock
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,14 @@ packages:
url: "https://pub.dev"
source: hosted
version: "1.19.1"
crypto:
dependency: transitive
description:
name: crypto
sha256: "1e445881f28f22d6140f181e07737b22f1e099a5e1ff94b0af2f9e4a463f4855"
url: "https://pub.dev"
source: hosted
version: "3.0.6"
dart_earcut:
dependency: transitive
description:
Expand Down Expand Up @@ -89,6 +97,14 @@ packages:
url: "https://pub.dev"
source: hosted
version: "7.0.1"
fixnum:
dependency: transitive
description:
name: fixnum
sha256: b6dc7065e46c974bc7c5f143080a6764ec7a4be6da1285ececdc37be96de53be
url: "https://pub.dev"
source: hosted
version: "1.1.1"
flutter:
dependency: "direct main"
description: flutter
Expand Down Expand Up @@ -255,6 +271,30 @@ packages:
url: "https://pub.dev"
source: hosted
version: "1.9.1"
path_provider:
dependency: transitive
description:
name: path_provider
sha256: "50c5dd5b6e1aaf6fb3a78b33f6aa3afca52bf903a8a5298f53101fdaee55bbcd"
url: "https://pub.dev"
source: hosted
version: "2.1.5"
path_provider_android:
dependency: transitive
description:
name: path_provider_android
sha256: d0d310befe2c8ab9e7f393288ccbb11b60c019c6b5afc21973eeee4dda2b35e9
url: "https://pub.dev"
source: hosted
version: "2.2.17"
path_provider_foundation:
dependency: transitive
description:
name: path_provider_foundation
sha256: "4843174df4d288f5e29185bd6e72a6fbdf5a4a4602717eed565497429f179942"
url: "https://pub.dev"
source: hosted
version: "2.4.1"
path_provider_linux:
dependency: transitive
description:
Expand Down Expand Up @@ -380,6 +420,14 @@ packages:
url: "https://pub.dev"
source: hosted
version: "1.10.1"
sprintf:
dependency: transitive
description:
name: sprintf
sha256: "1fc9ffe69d4df602376b52949af107d8f5703b77cda567c4d7d86a0693120f23"
url: "https://pub.dev"
source: hosted
version: "7.0.0"
stack_trace:
dependency: transitive
description:
Expand Down Expand Up @@ -500,8 +548,16 @@ packages:
url: "https://pub.dev"
source: hosted
version: "3.1.4"
uuid:
dependency: transitive
description:
name: uuid
sha256: a5be9ef6618a7ac1e964353ef476418026db906c4facdedaa299b7a2e71690ff
url: "https://pub.dev"
source: hosted
version: "4.5.1"
vector_math:
dependency: "direct main"
dependency: transitive
description:
name: vector_math
sha256: "80b3257d1492ce4d091729e3a67a60407d227c27241d6927be0130c98e741803"
Expand Down
1 change: 0 additions & 1 deletion example/pubspec.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ dependencies:
proj4dart: ^2.1.0
shared_preferences: ^2.3.4
url_launcher: ^6.3.1
vector_math: ^2.1.4

dependency_overrides:
flutter_map:
Expand Down
13 changes: 9 additions & 4 deletions lib/flutter_map.dart
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,16 @@ export 'package:flutter_map/src/layer/tile_layer/tile_coordinates.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_display.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_image.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_layer.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_provider/asset_tile_provider.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_provider/asset/provider.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_provider/base_tile_provider.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_provider/file_providers/tile_provider_stub.dart'
if (dart.library.io) 'package:flutter_map/src/layer/tile_layer/tile_provider/file_providers/tile_provider_io.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_provider/network_tile_provider.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_provider/file/stub_tile_provider.dart'
if (dart.library.io) 'package:flutter_map/src/layer/tile_layer/tile_provider/file/native_tile_provider.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_provider/network/caching/built_in/built_in_caching_provider.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_provider/network/caching/caching_provider.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_provider/network/caching/disabled/disabled_caching_provider.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_provider/network/caching/tile_metadata.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_provider/network/caching/tile_read_failure_exception.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_provider/network/tile_provider.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_update_event.dart';
export 'package:flutter_map/src/layer/tile_layer/tile_update_transformer.dart';
export 'package:flutter_map/src/map/camera/camera.dart';
Expand Down
2 changes: 1 addition & 1 deletion lib/src/geo/latlng_bounds.dart
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import 'dart:math';

import 'package:flutter_map/src/misc/deg_rad_conversions.dart';
import 'package:latlong2/latlong.dart';
import 'package:vector_math/vector_math_64.dart';

/// Data structure representing rectangular bounding box constrained by its
/// northwest and southeast corners
Expand Down
2 changes: 1 addition & 1 deletion lib/src/gestures/map_interactive_viewer.dart
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ import 'package:flutter/gestures.dart';
import 'package:flutter/services.dart';
import 'package:flutter/widgets.dart';
import 'package:flutter_map/flutter_map.dart';
import 'package:flutter_map/src/misc/deg_rad_conversions.dart';
import 'package:flutter_map/src/misc/extensions.dart';
import 'package:latlong2/latlong.dart';
import 'package:vector_math/vector_math_64.dart';

part 'package:flutter_map/src/gestures/compound_animations.dart';

Expand Down
9 changes: 3 additions & 6 deletions lib/src/layer/tile_layer/tile_layer.dart
Original file line number Diff line number Diff line change
Expand Up @@ -362,20 +362,17 @@ class _TileLayerState extends State<TileLayer> with TickerProviderStateMixin {
false && (kReleaseMode || kProfileMode) && !_unblockOpenStreetMapUrl;
void _warnOpenStreetMapUrl() {
if (!_isOpenStreetMapUrl || !kDebugMode || _unblockOpenStreetMapUrl) return;
Logger(printer: PrettyPrinter(methodCount: 0)).e(
Logger(printer: PrettyPrinter(methodCount: 0)).w(
'''\x1B[1m\x1B[3mflutter_map\x1B[0m
flutter_map wants to help keep map data available for everyone.
We use the public OpenStreetMap tile servers in our code examples & demo app,
but they are NOT free to use by everyone.
In an upcoming non-major release, requests to 'tile.openstreetmap.org' or
'tile.osm.org' will be blocked by default in release mode.
Please review https://operations.osmfoundation.org/policies/tiles/ to see if
your project is compliant with their Tile Usage Policy.
For more information, see https://docs.fleaflet.dev/tile-servers/using-openstreetmap-direct.
It describes in additional detail why we feel it is important to do this, how
you can unblock the tile servers if your use-case is acceptable, the timeframes
for this new policy, and how we're working to reduce requests without any extra
work from you.''',
you can disable this warning, the timeframes for this new policy, and how we're
working to reduce requests without any extra work from you.''',
);
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,7 @@ class FileTileProvider extends TileProvider {
@override
ImageProvider getImage(TileCoordinates coordinates, TileLayer options) =>
throw UnsupportedError(
'The current platform does not have access to IO (the local filesystem), and therefore does not support `FileTileProvider`');
'The current platform does not have access to IO (the local '
'filesystem), and therefore does not support `FileTileProvider`',
);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
import 'package:flutter_map/flutter_map.dart';
import 'package:flutter_map/src/layer/tile_layer/tile_provider/network/caching/built_in/impl/stub.dart'
if (dart.library.io) 'package:flutter_map/src/layer/tile_layer/tile_provider/network/caching/built_in/impl/native/native.dart'
if (dart.library.js_interop) 'package:flutter_map/src/layer/tile_layer/tile_provider/network/caching/built_in/impl/web/web.dart';

/// Simple built-in map caching using an I/O storage mechanism, for native
/// (non-web) platforms only
///
/// Stores tiles as files identified with keys, containing some metadata headers
/// followed by the tile bytes, alongside a file used to track the size of the
/// cache.
///
/// Usually uses HTTP headers to determine tile freshness, although
/// `overrideFreshAge` can override this.
///
/// This is enabled by default in flutter_map, when using the
/// [NetworkTileProvider] (or cancellable version).
///
/// For more information, see the online documentation.
abstract interface class BuiltInMapCachingProvider
implements MapCachingProvider {
/// if a singleton instance exists, return it, otherwise create a new
/// singleton instance (and start asynchronously initialising it)
///
/// If an instance already exists, the provided configuration will be ignored.
///
/// See individual properties for more information about configuration.
factory BuiltInMapCachingProvider.getOrCreateInstance({
/// Path to the caching directory to use
///
/// This must be accessible to the program.
///
/// Defaults to a platform provided cache directory, which may be cleared by
/// the OS at any time.
String? cacheDirectory,

/// Preferred maximum size (in bytes) of the cache
///
/// This is applied when the internal caching mechanism is created (on the
/// first tile load in the main memory space for the app). It is not an
/// absolute limit.
///
/// Defaults to 1 GB. Set to `null` to disable.
int? maxCacheSize = 1_000_000_000,

/// Override the duration of time a tile is considered fresh for
///
/// Defaults to `null`: use duration calculated from each tile's HTTP
/// headers.
Duration? overrideFreshAge,

/// Function to convert a tile URL to a key used in the cache
///
/// This may be useful where parts of the URL are volatile or do not
/// represent the tile image, for example, API keys contained with the query
/// parameters.
///
/// The resulting key should be unique to that tile URL. Keys must be usable
/// as filenames on all intended platform filesystems.
///
/// Defaults to generating a UUID from the entire URL string.
///
/// The callback should not throw.
String Function(String url)? cacheKeyGenerator,

/// Prevent any tiles from being added or updated
///
/// Does not disable the size limiter if the cache size is larger than
/// `maxCacheSize`.
///
/// Defaults to `false`.
bool readOnly = false,
}) {
assert(
maxCacheSize == null || maxCacheSize > 0,
'`maxCacheSize` must be greater than 0 or disabled',
);
assert(
overrideFreshAge == null || overrideFreshAge > Duration.zero,
'`overrideFreshAge` must be greater than 0 or disabled',
);
return _instance ??= BuiltInMapCachingProviderImpl.createAndInitialise(
cacheDirectory: cacheDirectory,
maxCacheSize: maxCacheSize,
overrideFreshAge: overrideFreshAge,
cacheKeyGenerator: cacheKeyGenerator,
readOnly: readOnly,
);
}

static BuiltInMapCachingProviderImpl? _instance;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# `BuiltInMapCachingProvider` storage spec

The `BuiltInMapCachingProvider`, referred to as just 'built-in caching', is implemented using the filesystem for storage on native platforms.

Cached tiles & their metadata are stored as individual keyed files. An additional file is used to improve the efficiency of tracking and reducing the cache size, called the 'size monitor'.

## Tiles

Tiles are stored in files, where the filename is the output of the supplied `cacheKeyGenerator` given the tile's URL. This defaults to a v5 UUID. Files have no extension.

Also stored alongside tiles is metadata used to perform caching, namely:

* `staleAt`: The calculated time at which the tile becomes 'stale'
* (optionally) `lastModified`: The time at which the tile was last modified on the server, based on the HTTP header
* (optionally) `etag`: A unique string identifier for the current version of that tile, using the 'etag' HTTP header

The file format is as follows:

1. The header containing the tile metadata
2. The tile image bytes (as responded by the server), no longer than 4,294,967,295 bytes

The format of the header is as follows:

1. 8-byte signed integer (Int64): the `staleAt` timestamp, represented in milliseconds since the Unix epoch in the UTC timezone
2. 8-byte signed integer (Int64)
* Where provided, the `lastModified` timestamp, represented in milliseconds since the Unix epoch in the UTC timezone, which must not be 0
* Where not provided, the integer '0'
3. 2-byte unsigned integer (Uint16)
* Where provided, the length of the ASCII encoded `etag` in bytes
* Where not provided, the integer '0'
4. Variable number of bytes
* Where provided, the ASCII encoded `etag` (where each character is 7 bits but stored as 1 byte) with no greater than 65535 bytes
* Where not provided, no bytes
5. 4-byte unsigned integer (Uint32): the length of the tile image bytes

## Size monitor

Contains an 8-byte unsigned integer (Uint64), representing the size of all tiles (including metadata) stored in the cache in bytes.

Named as 'sizeMonitor.bin'.
Loading